It’s great to hear that the proposed solution will solve your problems Clément!
Our main focus at the moment is validating the solution, to ensure everything’s on the right track.
We’re hoping to do an early access program around Oct-Nov 2023, but we will keep the community updated on the progress.
Your entire dataset will first be sorted (in this case by the date attribute) and then the range is applied, meaning that if there’s more data to be retrieved (through pagination) the order will be kept.
I hope it clarifies things, thank you for your question!
At this point we are not going to provide support for breaking changes (ie. removing attributes, changing types, removing entities), but your concern is under our radar and we are going to iterate to provide a more robust solution that should address this.
We will share more details about this with the community when we have planned a solution for it.
On the surface, this looks promising. I am concerned about the limits though:
We’ll let you define up to 5 custom entities per app. Each entity will have:
Up to 50 attributes
Up to 5 indexes
Why limit us to 5 custom entities? This means that you’re thinking that we might only be storing 5 types of json objects in Forge Storage? Once you start with configs and “schema” type of things - the number of object types gets large quite fast (all of which really needs to be queriable if they’re attached to a project, issue etc).
Because of this limitation - I’m not sure if we would be able to make use of this feature. Any chance of use being able to have that limitation be larger than 5 entities? Ideally there is no limit on the number of entities that can be indexed, but if there has to be a limit - for large functionality apps - I would imagine 20 wouldn’t be too far off (we’d be at 10 today - so I’m future proofing myself). Of course the work around is for us to create indexing entities in forge storage to work around this limitation (but that seems like a fragile approach).
The initial number of up to 5 entities was defined to make sure we can handle all existing apps and their environments, and maintain our service reliability.
We do intend to increase those limits in future releases (after the EAP), but we will need to assess what those new limits are going to be. We will definitely take your use case of up 20 entities into consideration.
On another note, would the limit in the number of attributes also be a problem for your use case?
Thank you very much for your feedback, it’s very important to us!
Hi everyone, I wanted to provide a quick timeline update on this project.
Our team is still progressing with development work, however due to some technical challenges and dependencies we are trying to sort out, the EAP release will be delayed. At this stage, we are still firming up the timelines (looking like March end), however I will provide an update in January on how we are going / if any changes to the plan.
I apologise for any inconvenience caused due to this shift in dates. Please let me know if you have any questions.
It is a pity that there haven’t been any updates to the Forge Storage API this year. It’s really been the largest roadblock for me and others when it is about building more complex Forge apps. We started a new Forge app with egress to its own database a few days ago and I would have loved to avoid this.
If there are any updates or other insights on this please keep us posted. I would also love to hop on a call with the team to give feedback and to see what features to expect.
Hey @JulianWolf, Happy New Year! Apologies for the delayed response. I was on leave.
I apologise for the delays on updates to Forge storage API. We are actively working on it, however there have been some challenges we are trying to sort out. I understand this is a big roadblock and it is one of our top priorities to address.
Yes, it would be great to have a call with you on this. I will DM you to organise a time.
Any ETA for this feature? We are working on a Forge App and can really really use this feature. We are working towards our first release around May 31st 2023 and would like to know what your timeline is, so we can plan for either leveraging this feature or working around it.
If you want us to connect so we can explain our use case, and load / query scenarios, we would be happy to meet you and explain.
I hate to sound so negative because I’m sure the team has put a lot of work into this already.
I think this is the completely wrong direction. It feels like a terrible mashup between document based storage and SQL based storage.
If it’s sql based storage, you should host a real db like Postgres or MySQL and just impose limitations and give the developer root access to their own database and step back. It’s been done successfully for decades by hundreds of hosting providers.
If it’s document-based storage, the storage should be schema-less because this is what everyone uses nowadays. Also, declaring entities introduces a migration problem. What happens to the data on v2 of your add-on when you need to add a field to your entity. It feels like you are creating more complexity than required for the developer
Schema-less has been the state of the art in JSON based document storage for years. You can do indexes, queries, etc. on schema-less JSON documents. I know you guys are on Amazon. Please look at their Document DB product. If you are already using something like that for Forge storage, why are you making it more complicated than it needs to be? Just create a thin layer on top of an existing MongoDB-like product. This is pretty much the defacto standard.
Hi @NathanReddy, great to hear this feature will be useful for your app. In terms of ETA, we are currently still targeting March end for an EAP. However, we are still working through some technical challenges, there is a possibility this will get pushed a bit. We will be able to confirm once we progress more with our internal testing. Please also note, as an EAP release, it will not be recommended for production deployment. We would love to connect further to better understand your use-cases and how we can support in the interim.
Hi @ryan, we really appreciate the feedback, and will be looking closer to how we can improve this. When deciding on the direction for Storage API, we considered multiple options, and the decision was based on multiple factors including addressing concerns from wider set of developers, not just around API functionality, but also impact to compliance, scalability and security. With this EAP, we will be further assessing the value of this feature, and how we can improve moving forward. We also plan to offer additional DB solutions in future to provide more flexibility and options to our developers. It would be great to discuss in more detail with you to better understand your pain points and how we can help to address them.
Hi @SushantBista, thanks for taking the time to respond. As a developer, I realize I can’t see the meticulous research and thought that goes into these types of decisions. I only want to give an outside perspective. I just hope I come across as respectful. I appreciate the work and passion you and the rest of the team are putting into the Forge platform.
I don’t have a specific use case right now but I am looking at migrating some of my old plugins to Forge. A lot of where I’m coming from is a general frustration with Forge. It feels like every corner of the Forge platform is filled with compromises that aren’t being addressed as the platform matures. It’s been a couple of years since Forge left beta and I haven’t seen a lot of progress. I’m sure you guys have put a lot of thought into this particular feature but it feels like this is another compromise being added to the pile for reasons I mentioned in my original post. Multi-tenant cloud-hosted storage is a problem that has been solved. I wish you guys would better leverage those existing solutions. It seems like it would be less work for you guys and it would do more to meet developer expectations.
do you think the end of March timeline is realistic? In my view it would make sense to also just release some of the features, e.g. adjusting the sort ordering (ascending, descending).
In my assumption, Forge storage uses DynamoDB under the hood? Then passing that sort option through is a single boolean.
As a side note: I find the plan to “offer additional DB solutions” a bit worry some. It feels that you are reinventing a lot of existing, polished and stable tooling. You might deliver interesting features, but they will take a very long time to release. I just got started with forge, but I am already feeling a couple of topics where it would just be easier for me to go with a Connect app.
Hi @ryan, please know that we really appreciate your feedback, and definitely acknowledge your frustration.
We will definitely take your feedback on board while we are assessing features and improvements to Forge storage. I agree on taking more learnings from similar services to better meet developer expectations.
We are close to EAP release. A team member will provide an update on this tread soon (likely next week).
In regards to sorting - as part of upcoming custom entities release, we will support sorting (ascending, descending) for indexable attributes.
I just want to clarify my comment on “offer additional DB solutions", by which I meant that we understand the need from wider developer community on accessing DB solutions outside of key-value. We are currently exploring these in order to provide more options and flexibility to users, while balancing other security, performance and scalability requirements of a multi-tenant platform.