Hi @ryan, we really appreciate the feedback, and will be looking closer to how we can improve this. When deciding on the direction for Storage API, we considered multiple options, and the decision was based on multiple factors including addressing concerns from wider set of developers, not just around API functionality, but also impact to compliance, scalability and security. With this EAP, we will be further assessing the value of this feature, and how we can improve moving forward. We also plan to offer additional DB solutions in future to provide more flexibility and options to our developers. It would be great to discuss in more detail with you to better understand your pain points and how we can help to address them.
Hi @SushantBista, thanks for taking the time to respond. As a developer, I realize I can’t see the meticulous research and thought that goes into these types of decisions. I only want to give an outside perspective. I just hope I come across as respectful. I appreciate the work and passion you and the rest of the team are putting into the Forge platform.
I don’t have a specific use case right now but I am looking at migrating some of my old plugins to Forge. A lot of where I’m coming from is a general frustration with Forge. It feels like every corner of the Forge platform is filled with compromises that aren’t being addressed as the platform matures. It’s been a couple of years since Forge left beta and I haven’t seen a lot of progress. I’m sure you guys have put a lot of thought into this particular feature but it feels like this is another compromise being added to the pile for reasons I mentioned in my original post. Multi-tenant cloud-hosted storage is a problem that has been solved. I wish you guys would better leverage those existing solutions. It seems like it would be less work for you guys and it would do more to meet developer expectations.
do you think the end of March timeline is realistic? In my view it would make sense to also just release some of the features, e.g. adjusting the sort ordering (ascending, descending).
In my assumption, Forge storage uses DynamoDB under the hood? Then passing that sort option through is a single boolean.
As a side note: I find the plan to “offer additional DB solutions” a bit worry some. It feels that you are reinventing a lot of existing, polished and stable tooling. You might deliver interesting features, but they will take a very long time to release. I just got started with forge, but I am already feeling a couple of topics where it would just be easier for me to go with a Connect app.
Hi @ryan, please know that we really appreciate your feedback, and definitely acknowledge your frustration.
We will definitely take your feedback on board while we are assessing features and improvements to Forge storage. I agree on taking more learnings from similar services to better meet developer expectations.
We are close to EAP release. A team member will provide an update on this tread soon (likely next week).
In regards to sorting - as part of upcoming custom entities release, we will support sorting (ascending, descending) for indexable attributes.
I just want to clarify my comment on “offer additional DB solutions", by which I meant that we understand the need from wider developer community on accessing DB solutions outside of key-value. We are currently exploring these in order to provide more options and flexibility to users, while balancing other security, performance and scalability requirements of a multi-tenant platform.
Hi everyone,
I wanted to provide a quick update for Complex Queries through Custom Entities.
Firstly, we have locked in April 2023 as a target release for the EAP.
Secondly, there are a few changes; however the experience remains close to what we shared earlier. The changes are:
We are removing the partition method in favor of passing partition as an optional parameter to the index method
The Range method has been renamed Where
Sorting is no longer a parameter, but a function named sort()
We have added support for the data type Integer
The Float type is currently not supported for Indexing operations. We expect to solve this by the EAP release in April; if not, we will rectify this soon after with a release
Partition can now be multi keyed
filter() is now broken down for convenience:
orFilter()
andFilter()
The objects you can store as a custom entity can have a maximum depth of 31
The attributes you can use with custom entity can have a maximum size of 240KB
The format for defining attributes has changed slightly and now looks like this:
We will be sharing more details and any additional changes as we move closer to the EAP release.
Please keep an eye on this space to get more details about how to participate.
We are really excited to get this in your hands soon and start receiving your valuable feedback.
Thank you!
We have opened the EAP for complex queries through custom entities.
We are accepting participants as of this Wednesday, April 12th
We have been continuously working on solving for your pain points and are very excited to get you started. We have been sharing updates on the feature set and changes that we made along the way in this topic, however you can find a quick re-cap here. We have decided to keep the participation process very simple for convenience , more details below.
How do you participate?
Opt-in via this form. We will add you to the relevant community topic, where you will be able to access announcements, updates, documentation, feedback and other resources. We will be enabling the feature asap post receiving the request, and keep you updated.
Hello everyone!
We are thrilled to announce that complex queries through custom entities will soon be transitioning to Preview (planned for the last week of June). To accommodate this change, we have decided to close the EAP on June 21, 2023 IST. We will continue to accept participants until the given date for those interested in testing, however please note that effective from this date, access to data generated from the EAP phase testing will no longer be available.
We will provide further details as part of the Preview release. If you have participated in the EAP and already finished testing and would like to share feedback with us, we would love to hear from you on how we can improve and offer more value.
Please do help by filling this feedback form here, this will help us work more effectively towards solving for your pain points.
Complex queries through custom entities is moving into Preview effective today i.e June, 26, 2023 IST.
(Please have a look at the changelog entry for more details)
Here is a quick summary:
How do you participate?
Going into the preview release, we have removed the sign-up requirement. Now, you can simply start by defining custom entities in your manifest file. Here are the steps:
Start defining custom entities in the manifest to start testing (Here is a sample app to get you started)
What has changed?
We are very excited to get you started. We have been sharing updates on the feature set and changes introduced, however here is a quick re-cap. Moving into preview, we have introduced further changes to improve the experience:
What has changed?
You can finally deploy apps into production environment :celebrate:
CLI experience has improved, for example - you can now easily check the status of indexing processes
Now you have more flexibility with increase of indexes from 5 → 7 and entities from 5 → 20
Known limitations
Index and schema update/ deletion is not supported at the moment
The objects you can store as a custom entity can have a maximum depth of 31
The objects you can store as a custom entity can have a maximum size of 240KB per object
Empty keys are not allowed
Upper case not allowed in entity names
We are currently exploring how and when we can increase/solve for the limits and will keep you posted.
More details on known limitations, validation checks and changes are covered extensively in the documentation linked in the changelog.
That’s all!
Looking forward to receiving you valuable feedback!