Feedback request: Forge complex queries through custom entities

That’s a great question Sven, thank you for that!

At this point we are not going to provide support for breaking changes (ie. removing attributes, changing types, removing entities), but your concern is under our radar and we are going to iterate to provide a more robust solution that should address this.

We will share more details about this with the community when we have planned a solution for it.

Hi @RodolfoCandido,

Is 2023 correct? It seems a bit too far away.

Best regards,
Sven

Just to be sure: 2023 and not 2022 ? :sweat_smile:

Apologies, it’s Oct-Nov 2022 :blush:

(I was going to refer to Q2 FY23 but then decided to share based on calendar year to make easier to understand).

1 Like

On the surface, this looks promising. I am concerned about the limits though:

We’ll let you define up to 5 custom entities per app. Each entity will have:

  • Up to 50 attributes
  • Up to 5 indexes

Why limit us to 5 custom entities? This means that you’re thinking that we might only be storing 5 types of json objects in Forge Storage? Once you start with configs and “schema” type of things - the number of object types gets large quite fast (all of which really needs to be queriable if they’re attached to a project, issue etc).

Because of this limitation - I’m not sure if we would be able to make use of this feature. :slightly_frowning_face: Any chance of use being able to have that limitation be larger than 5 entities? Ideally there is no limit on the number of entities that can be indexed, but if there has to be a limit - for large functionality apps - I would imagine 20 wouldn’t be too far off (we’d be at 10 today - so I’m future proofing myself). Of course the work around is for us to create indexing entities in forge storage to work around this limitation (but that seems like a fragile approach).

7 Likes

The initial number of up to 5 entities was defined to make sure we can handle all existing apps and their environments, and maintain our service reliability.

We do intend to increase those limits in future releases (after the EAP), but we will need to assess what those new limits are going to be. We will definitely take your use case of up 20 entities into consideration.

On another note, would the limit in the number of attributes also be a problem for your use case?

Thank you very much for your feedback, it’s very important to us!

1 Like

Hi everyone, I wanted to provide a quick timeline update on this project.

Our team is still progressing with development work, however due to some technical challenges and dependencies we are trying to sort out, the EAP release will be delayed. At this stage, we are still firming up the timelines (looking like March end), however I will provide an update in January on how we are going / if any changes to the plan.

I apologise for any inconvenience caused due to this shift in dates. Please let me know if you have any questions.

Thanks,
Sushant

Hey @SushantBista,

It is a pity that there haven’t been any updates to the Forge Storage API this year. It’s really been the largest roadblock for me and others when it is about building more complex Forge apps. We started a new Forge app with egress to its own database a few days ago and I would have loved to avoid this.

If there are any updates or other insights on this please keep us posted. I would also love to hop on a call with the team to give feedback and to see what features to expect.

Happy holidays

Julian

1 Like

Hey @JulianWolf, Happy New Year! Apologies for the delayed response. I was on leave.

I apologise for the delays on updates to Forge storage API. We are actively working on it, however there have been some challenges we are trying to sort out. I understand this is a big roadblock and it is one of our top priorities to address.

Yes, it would be great to have a call with you on this. I will DM you to organise a time.

Thanks,
Sushant

2 Likes

Hi Atlassian Team,

Any ETA for this feature? We are working on a Forge App and can really really use this feature. We are working towards our first release around May 31st 2023 and would like to know what your timeline is, so we can plan for either leveraging this feature or working around it.

If you want us to connect so we can explain our use case, and load / query scenarios, we would be happy to meet you and explain.

thanks

Nathan

I hate to sound so negative because I’m sure the team has put a lot of work into this already.

I think this is the completely wrong direction. It feels like a terrible mashup between document based storage and SQL based storage.

If it’s sql based storage, you should host a real db like Postgres or MySQL and just impose limitations and give the developer root access to their own database and step back. It’s been done successfully for decades by hundreds of hosting providers.

If it’s document-based storage, the storage should be schema-less because this is what everyone uses nowadays. Also, declaring entities introduces a migration problem. What happens to the data on v2 of your add-on when you need to add a field to your entity. It feels like you are creating more complexity than required for the developer

Schema-less has been the state of the art in JSON based document storage for years. You can do indexes, queries, etc. on schema-less JSON documents. I know you guys are on Amazon. Please look at their Document DB product. If you are already using something like that for Forge storage, why are you making it more complicated than it needs to be? Just create a thin layer on top of an existing MongoDB-like product. This is pretty much the defacto standard.

Hi @NathanReddy, great to hear this feature will be useful for your app. In terms of ETA, we are currently still targeting March end for an EAP. However, we are still working through some technical challenges, there is a possibility this will get pushed a bit. We will be able to confirm once we progress more with our internal testing. Please also note, as an EAP release, it will not be recommended for production deployment. We would love to connect further to better understand your use-cases and how we can support in the interim.

Thanks,
Sushant

1 Like

Hi @ryan, we really appreciate the feedback, and will be looking closer to how we can improve this. When deciding on the direction for Storage API, we considered multiple options, and the decision was based on multiple factors including addressing concerns from wider set of developers, not just around API functionality, but also impact to compliance, scalability and security. With this EAP, we will be further assessing the value of this feature, and how we can improve moving forward. We also plan to offer additional DB solutions in future to provide more flexibility and options to our developers. It would be great to discuss in more detail with you to better understand your pain points and how we can help to address them.

Thanks,
Sushant

Hi @SushantBista, thanks for taking the time to respond. As a developer, I realize I can’t see the meticulous research and thought that goes into these types of decisions. I only want to give an outside perspective. I just hope I come across as respectful. I appreciate the work and passion you and the rest of the team are putting into the Forge platform.

I don’t have a specific use case right now but I am looking at migrating some of my old plugins to Forge. A lot of where I’m coming from is a general frustration with Forge. It feels like every corner of the Forge platform is filled with compromises that aren’t being addressed as the platform matures. It’s been a couple of years since Forge left beta and I haven’t seen a lot of progress. I’m sure you guys have put a lot of thought into this particular feature but it feels like this is another compromise being added to the pile for reasons I mentioned in my original post. Multi-tenant cloud-hosted storage is a problem that has been solved. I wish you guys would better leverage those existing solutions. It seems like it would be less work for you guys and it would do more to meet developer expectations.

1 Like

Hi @SushantBista,

do you think the end of March timeline is realistic? In my view it would make sense to also just release some of the features, e.g. adjusting the sort ordering (ascending, descending).

In my assumption, Forge storage uses DynamoDB under the hood? Then passing that sort option through is a single boolean.

As a side note: I find the plan to “offer additional DB solutions” a bit worry some. It feels that you are reinventing a lot of existing, polished and stable tooling. You might deliver interesting features, but they will take a very long time to release. I just got started with forge, but I am already feeling a couple of topics where it would just be easier for me to go with a Connect app.

Best,
Peter

Hi @ryan, please know that we really appreciate your feedback, and definitely acknowledge your frustration.

We will definitely take your feedback on board while we are assessing features and improvements to Forge storage. I agree on taking more learnings from similar services to better meet developer expectations.

Hi @PeterJuras, thanks for your feedback.

We are close to EAP release. A team member will provide an update on this tread soon (likely next week).

In regards to sorting - as part of upcoming custom entities release, we will support sorting (ascending, descending) for indexable attributes.

I just want to clarify my comment on “offer additional DB solutions", by which I meant that we understand the need from wider developer community on accessing DB solutions outside of key-value. We are currently exploring these in order to provide more options and flexibility to users, while balancing other security, performance and scalability requirements of a multi-tenant platform.

Thanks,
Sushant

Hi everyone,
I wanted to provide a quick update for Complex Queries through Custom Entities.

Firstly, we have locked in April 2023 as a target release for the EAP.

Secondly, there are a few changes; however the experience remains close to what we shared earlier. The changes are:

  • We are removing the partition method in favor of passing partition as an optional parameter to the index method
  • The Range method has been renamed Where
  • Sorting is no longer a parameter, but a function named sort()
  • We have added support for the data type Integer
  • The Float type is currently not supported for Indexing operations. We expect to solve this by the EAP release in April; if not, we will rectify this soon after with a release
  • Partition can now be multi keyed
  • filter() is now broken down for convenience:
    • orFilter()
    • andFilter()
  • The objects you can store as a custom entity can have a maximum depth of 31
  • The attributes you can use with custom entity can have a maximum size of 240KB
  • The format for defining attributes has changed slightly and now looks like this:
1entities:
2      - name: 'product'
3        attributes:
4          brand:
5            type: 'string'
6          price:
7            type: 'float'
8        indexes:
9          - 'price'                     # 'simple' index on attribute 'price'
10          - name: 'by-brand'            # 'named' index on attribute 'brand'
11            range:
12              - 'brand'
13          - name: 'by-brand-and-price'  # 'named' index with partition
14            range:
15              - 'price'
16            partition:
17              - 'brand'
18

We will be sharing more details and any additional changes as we move closer to the EAP release.
Please keep an eye on this space to get more details about how to participate.
We are really excited to get this in your hands soon and start receiving your valuable feedback.
Thank you!

2 Likes

Hi everyone,

We have opened the EAP for complex queries through custom entities.

We are accepting participants as of this Wednesday, April 12th

We have been continuously working on solving for your pain points and are very excited to get you started. We have been sharing updates on the feature set and changes that we made along the way in this topic, however you can find a quick re-cap here. We have decided to keep the participation process very simple for convenience , more details below.

How do you participate?

Opt-in via this form. We will add you to the relevant community topic, where you will be able to access announcements, updates, documentation, feedback and other resources. We will be enabling the feature asap post receiving the request, and keep you updated.

Thank you!

3 Likes