RFC-72 : Forge Object Store

[EckhardMaass]

Hello @SunandanGokhroo ,

We have a couple of Apps, so it is a bit hard to squeeze all into one answer, so I try to mainly take the vantage point of our Survey App (Multivote & Enterprise Survey).

@SunandanGokhroo

• What specific scenarios or applications would benefit from an Object Store solution, such as handling large-scale data storage or media content management?

The current storage size is quite low, so we can run into problems for storing definition and responses to our surveys - I know DC clients with long definitions that would fail in the Cloud. But we are talking more in the range of a couple of hundred Kbs, maybe some Mbs.

• What security measures do expect to be implement to ensure safe and controlled access to stored objects, considering encryption and access policies?

I would expect it to behave like the before mentioned storage systems.

• Do you perform any administrative or maintenance tasks on stored objects, and would features like automatic expiration (TTL) be beneficial for managing data lifecycle?

We don’t, but we are about to release a possibility for users to do it. As the lifespan of a survey is driven by the users (ie, they decide how long it should run, and how long they want to store the raw data), it might even be detrimental.

• What is your projected data storage requirement per customer over the next 12 months? How often do you anticipate these objects will be read or written in your application?

This is really hard to tell, but I don’t see many peaks. For reading: there might be peaks during the hot phase of a vote (which then could maybe cached), afterwards I would anticipate roughly something like 2, 3 times a month max.

• Are there scenarios in your application where multiple users or processes might attempt to update the same object simultaneously? If so, how do you currently handle such concurrent updates?

Yes, when updating a definition. As the storage is eventually consistent, we just live with this now and it is inherently racing. No atomic updates is really some kind of pain - I would love to have at least the possibility to only update when the previous version has some quality (like version number).

• Does your use case require versioned storage and retrieval of objects, and how would you utilise object versioning to manage data changes?

See above - we live with the ray behaviour. With versioned storage, we could more gracefully handle this, but also easily add features like undos.

• What additional functionalities or operations would you expect from the Object Store solution to support your application needs, such as metadata management or data replication?

Right now, I don’t see much. The very tight limits (invocation size and time wise) and the consistency without any way to interact are the major factors. This is also hinders upgrading the data for new versions when a new version rolls out.

Additionally, we would want to have a way to migrate from the other storage solutions to the new more easily - Custom entity would have been nice for us, but as there is no good migration path: you have to be really careful with the tight limits to react on an App update trigger; but then we cannot force older version to be upgraded, see RFC-71: Improving Connect to Forge and Forge Platform Upgrades - #21 by EckhardMaass , so the codebase becomes a mess…

• How might the proposed storage limits and quotas implementation impact your application, and what strategies would you employ to optimise resource usage?

Right now, if it is too much, it won’t work. In another App, to temporarily store something, we split up the file and stored the splits. This is a eally ugly workaround!

Best
Eckhard