Hi there, I’m Rodolfo, an engineer at Atlassian. Our team is working on improvements to the Forge platform’s hosted storage capabilities. We’re planning to introduce the ability to create complex queries through custom entities in Forge and we’d love your feedback on it.
How Forge hosted storage works right now
At the moment, Forge hosted storage only allows you to perform query operations on the key
defined by you, the developer. It doesn’t let you query based on the JSON values stored against each key.
As a result, you can only perform basic CRUD operations on hosted storage. In addition, Forge hosted storage query only supports the startsWith
condition.
What we’re planning to do
We’re planning to introduce new ways of structuring data and indexes based on how you want to query it.
We’ll let you define up to 5 custom entities per app. Each entity will have:
- Up to 50 attributes
- Up to 5 indexes
After defining these entities and indexes, you’ll then have the ability to perform complex queries . This means you can query data through our storage API based on the indexes pointing to your data value, and not just the key
field.
We’re also planning to add new conditions to support complex queries – more on this in the below section.
How this will affect you
Because this new capability will coexist with the existing untyped key/value storage, there’s no impact on your app’s data.
However, with this new ability, you’ll be able to start defining custom entities in your manifest.yml
file. Each entity will be identified by name and will have a list of attributes. The indexes section allows you to define your access patterns.
app:
id: <app id here>
storage:
entities:
- name: books
attributes:
- title: string
- author: string
- publishedYear: float
indexes:
- author
- publishedYear
- name: by-author-and-published-year
partition: author
range: publishedYear
Attribute types
You can define up to four data types:
string
float
boolean
any
Note, the data type any
is not supported for queries, so it can’t be used in any part of the indexes definition.
Indexes
You can specify two kinds of indexes:
- Simple index
- Named index
A simple index is based on a single attribute on which you can apply range conditions when you query on this index (see below for details).
A named index can take an additional attribute that will be used to partition the data before any query conditions are applied.
We expect that most use-cases will be solved with simple indexes. However, for large datasets, the partitions in named indexes will allow you to reduce the number of entities queries traverse; thereby providing performance improvements and advanced access patterns.
The existing API for untyped key/value will still work as normal, but we will extend it to allow you to specify which entity
your operation is for, like this:
// Set
await storage.entity("books").set("book-1656551689", {
title: "The Shining",
author: "Stephen King",
publishedYear: 1977
});
// Get
await storage.entity("books").get("book-1656551689");
// Delete
await storage.entity("books").delete("book-1656551689");
// Query on simple index
await storage.
.query()
.entity("books")
.index("author")
.range(isEqualTo("Stephen King"))
.limit(20)
.getMany();
// Query on named index
await storage.
.query()
.entity("books")
.index("by-author-and-published-year")
.partition("Stephen King")
.range(isGreaterThan(2010), 'DESC')
.filter("title", contains("au"))
.limit(20)
.cursor('')
.getMany();
Partition
The partition()
method is only required for named indexes; otherwise, it should not be used. The query will return only results where the partition attribute of the index exactly matches the argument value.
Range
The range()
method is the recommended way to filter your data with the new conditions listed below. The second parameter defines the order of the results as the range attribute is used for sorting.
- ASC (default)
- DESC
Filter
The filter()
method provides further filtering capabilities for all other typed attributes (excluding any
).
New conditions
We’re planning to support the following conditions. Some of them will be only supported in certain parts of the query.
- Supported for range:
- Equal to
- Less than / Less than or equal to
- Greater than / Greater than or equal to
- Between
- Begins with (already supported as
startsWith
)
- Supported for filtering:
- all the conditions supported for range and;
- Not equal to
- Contains / Not contains
- Exists / Not exists
Known limitations
Filter operations
The response of queries using filter()
might result in pages with fewer items than the limit you specify in the query (or even no items at all) even though there may be more items remaining. You can access these items with additional calls using the cursor.
You can tell when there really are no items by checking if the attribute nextCursor
is undefined
or not. This may happen because filtering happens in memory after the range operations return the data from the database. This behaviour can result in the response payload with items or empty pages and is less performant.
Feedback
We would love to hear your feedback about these plans, so we can adjust our solution as needed. In particular, we’re interested in:
- Whether your app needs higher limits than our planned ones – 5 different entities, 50 attributes each, 5 indexes.
- The type of query conditions that will be most useful to your development plans.
- Whether you have more complex needs than what we’re planning to provide.
- Any clarifications you need in addition to the information above to understand the feature. This will help us ensure we can provide the right level of documentation and support.
Feel free to comment on this post and let’s keep the conversation going!
You can also submit feature requests here.
- Yes, this will address my pain points in using Forge-hosted storage
- No, this will not address my pain points using Forge-hosted storage (please provide more details in the comments)
0 voters