How to reach all keys from Storage API

Dear Atlassian!

I need an urgent help.
Our Forge Custom UI application with Storage API designed to store users.
From the structure boundaries of Storage API, we want to store these keys as like this:
Each user is under a single key.

“user1”: { some user data… }
“user2”: { some user data… }

“user500”: { some user data… }

As a limitation of a query, we can get only 20 entries from the Storage.
And here comes the cursor, which seems a good thing, but apparently doesn’t capable to reach all ~500 users from the Storage.

const query = storage.query()
.where(‘key’, startsWith(‘user’))
.limit(20)

if (cursor) {
  query.cursor(cursor)
}
return query.getMany()

What is the correct implementation of a query with cursor, to get all ~500 user keys from the Storage with a single call, before the application load?

Thanks for the help in advance!
Unfortunately, the documentation doesn’t help with this :frowning:

Warning: not the solution, but…
I know this is not what you want, but consider storing users as a single object entity with an array:

{
   users: [ { user1 }, { user2 }, ... ]
}

then use compressed.json to store this entity object, which dramatically increases the amount of elements the object can have.

Then you can fetch all elements with one call.

Thanks for your answer, but this is the current way we use the Storage API, and a single key can contain a maximum of 128 KB of data. Even with compressed.json, it’s simply not enough space.

We need to switch to one user - one key method, I hope Atlassian Staff will give the information we need.

Hi @kornel.leitner – I’m not sure if your snippet is representative of your actual code, but if it is, you might not be updating your query reference with the results of the .cursor() method.

So, instead of this:

if (cursor) {
  query.cursor(cursor)
}
return query.getMany()

Maybe something like this:

if (cursor) {
  query = query.cursor(cursor)
}
return query.getMany()

Note: I didn’t test this. Just guessing based on this snippet from the API docs:

// Fetch a page of 10 results
const { nextCursor } = await storage.query().getMany();

// Fetch the next 10 results
await storage.query().cursor(nextCursor).getMany();

Hi Aaron! :slight_smile:

Yes, it was just an example snippet. Here is my snippet, which is working, there is no problem with the cursor, but there is with the limitations of Forge.

const getUsers = async ({ payload: { cursor } }) => {
  let query = storage.query()
    .where('key', startsWith('user'))
    .limit(20)
    .cursor(cursor)

  return await query.getMany()
}

And I call it with invoke recursively.

const getUsers = (nextCur = '') => {
  invoke('getUsers', { cursor: nextCur }).then(({ results, nextCursor }) => {
    if (nextCursor) {
      getUsers(nextCursor)
    }
  })
}

But this recursion simply not working, I have to use setTimeout to make this query workable.
It would be good with setTimeout, but in that case, I couldn’t put the new users in each query to a common state array. (In the body of setTimeout, I couldn’t operate with the current value of useState.)

So here comes my question to the Atlassian Staff, how can I use this query with cursor to solve a real problem in normal time. How can I perform a SELECT * FROM users with the Storage API?
Is this query capable to this operation consider the limitations?

Thanks for the answers in advice.

Hi @kornel.leitner

From how your code is looking I’m guessing you’re using Custom UI and making multiple calls from the frontend. Your backend then passes the call to Forge Storage.

If you’re only serving 500 users you should be able to move this “bulking” logic to the backend instead of in the frontend.

I ran a quick test fetching stub data using two methods. The getUsersRecursive takes 20 seconds while getUsersBulk takes 2.23 seconds.

Was this what you were looking for?

I’ve provided the code I’ve tested with below.

// src/index.js
import Resolver from '@forge/resolver';
import { storage, startsWith } from '@forge/api'

const resolver = new Resolver();

resolver.define('setUsers', async ({ payload: { count } }) => {
  await Promise.all([...Array(parseInt(count)).keys()].map(async (id) => await storage.set(`user${id}`, "user info")));
});

resolver.define('flush', async ({ payload: { count }}) => {
  await Promise.all([...Array(parseInt(count)).keys()].map(async (id) => await storage.delete(`user${id}`)));
});

resolver.define('getUsers', async ({ payload: { cursor } }) => {
  let query = storage.query().where('key', startsWith('user')).limit(20).cursor(cursor);
  return await query.getMany();
});

resolver.define('getUsersBulk', async ({ payload: { cursor, limit } }) => {
  const iterations = Math.ceil(limit / 20);
  let users = [];
  for (let i = 0; i < iterations; i++) {
    let query = storage.query().where('key', startsWith('user')).limit(20).cursor(cursor);
    const res =  await query.getMany();
    cursor = res.nextCursor;
    users = users.concat(res.results);
  }

  return { results: users, nextCursor: cursor };
});

export const handler = resolver.getDefinitions();
// static/hello-world/src/App.js
import React, { useState } from 'react';
import { invoke } from '@forge/bridge';

const getUsersRecursive = async (nextCur = '') => {
  const { results, nextCursor } = await invoke('getUsers', { cursor: nextCur })
  if (nextCursor) {
    return [results, ...(await getUsersRecursive(nextCursor))];
  }
  return results;
}

function App() {
  const [data, setData] = useState(null);

  return (
    <div>
      <div>
        {data ? data : 'Loading...'}
      </div>
      <button onClick={async () => {
        setData(null);
        await invoke('setUsers', { count: 500 });
        setData("Set Users complete!");
      }}>
        Set Users
      </button>

      <button onClick={async () => {
        setData(null);
        await invoke('flush', { count: 500 });
        setData("Flush complete!");
      }}>
        Flush Users
      </button>

      <button onClick={async () => {
        setData(null);
        setData(JSON.stringify(await getUsersRecursive()));
      }}>
        Get Users
      </button>

      <button onClick={async () => {
        setData(null);
        setData(JSON.stringify(await invoke('getUsersBulk', { limit: 500 })));
      }}>
        Get Users Bulk
      </button>
    </div>
  );
}

export default App;

Thank you, Joshua for the answer!

It is a bit closer to what I would like to implement and it works well, but how about to try it with only 1500 users.
(As the pricing calculates with up to 35000.)
I got this error during the setUsers method:

Here comes the optimization? Should I delay the calls?

Yeah this is likely a result of our limits

It ain’t pretty but this solution will go up to 1500.

resolver.define('setUsers', async ({ payload: { count } }) => {
  const chunkSize = 1000;
  const total = parseInt(count);

  for (let i = 0;;) {
    const remaining = Math.min(chunkSize, total - i);
    await Promise.all([...Array(remaining).keys()]
      .map((idx) => idx + i) // idx starts at 0
      .map(async (userId) => {
        await storage.set(`user${userId}`, "user info")
      }));

    i += chunkSize;
    if (i < total) {
      await sleep(21000);
    } else {
      break;
    }
  }
});

There is a 25 second limit to invocation so there isn’t much time after the 20 seconds to generate more users. At that point I would start looking at getting the frontend to start recursively calling as you’ve originally done.

How do you define the sleep function for this example?

There are a few examples online on how to create such a sleep function

This is what I normally use

function sleep(ms) {
    return new Promise(resolve => setTimeout(resolve, ms));
}