Another reason - Custom Entity Store collection having reference to either work item, project or board within one property and kind in another, like:
- name: note
attributes:
....
sourceType: // board | project | work item
type: string
sourceId:
type: string // id of board or project or work item
I would rather go not with declarative way for conversion, but rather imperative. Like your migration could call a module
function convertIdForCloud2Cloud(collectionname, collectionItem, translationService);
Anyway, like Remie said every vendor can have its own view on how to design data structure for his apps - this means that solution A does not really cover 90% of cases but rather heavily less.
Just to be sure, what you are calling custom field type identifiers ? The key we set in the Forge manifest ?
And custom field identifiers are like customfield_11288 ?
Does your app have any Atlassian-issued IDs in text, string or object typed fields?
I am not sure to understand your question.
Customers of our App can specify that our custom fields must change/reset their values when others custom fields change their values. Those others custom fields are specified like customfield_11288. And right now, this info is both stored in our backend storage outside of Forge, and, for performance reason, in the context config of our custom field.
So obviously, if this identifier change, the migration process will need to handle the mapping to the new identifiers, in all places where it is present. And therefore, I have a feeling that we should be in control of these migrations.
Thanks @SilvreLestang - that is very clear. It seems that you have a requirement for any solution to support mapping identifiers within the text of the context configuration (as well as a way to make remote calls to update the mapping).
Hey @AndrewMiller1 I’ve collected some feedback from across Appfire and I’ve aggregated it below. Thanks for all of your work on this topic!
Appfire position:
We believe that Option B is required - there are too many cases where Option A is not sufficient (including app-generated ids, customer-supplied ids that need to change, app that have different storage methods for different clouds). However, we would prefer to have both Option A (with full jsonpath support) and Option B available so we can, where possible, use the simpler approach. It is likely that a single app will have cases that match both - so we would like to be able to use each in the same application. It’s even possible that a single dataset would mostly support A but need B for a few cases, in which case we would like to be able to apply both to that dataset in order to reduce both migration time and cost. Having both options available would be the most performant option.
TL;DR We need B but both options would be nice. Option A is simpler but this would not work for our more complicated apps.
Option A (with full jsonpath support) for those cases where we can use it
Simpler where it applies
Fewer moving parts overall
Option B for those cases where we cannot use Option A
Each migration would allow invocation of both
Some data / values are mapped with A
The rest with B
Scope
We would need a solution that works for the following use cases:
merge (many instances into one)
partial migration
all-at-once migration
move vs copy migrations
Option A does NOT cover the merge use case
Option A - Nice to haves
Custom JSON path and Regex support
Custom serialization
Use case: LaTex Math
Assumptions:
We’re not going to run into API limits
“we could let app partners define a Forge function that’s invoked with a batch of records, which could call an API to retrieve mappings.” Can I assume this means that we would be able to call an arbitrary function in a batch?
Questions:
We have a variety of questions and may have more questions once we have more details about how this will actually be implemented. Please reach out if you’d like to discuss in person. This is a topic we have a vested interest in getting right.
Execution
Lifecycle events available so partners/vendors can track and understand migration state?
We would like these events for all migrations, successful and unsuccessful.
Progress reporting to customers - It’s unclear how this is expected to work
Error reporting and recovery
How are failures handled?
What options are expected to be available to customer admins?
Restart from clean?
Rollback for merge?
Restart from failure point?
If these options aren’t available to the customer admins directly, are they available through other avenues?
What are partners expected to support?
What are the reporting / retry options available for each option?
What is the customer’s opportunity to disrupt the migration process, and what is the expected app behavior in each case?
What are users allowed to do during this migration process?
How do we control for automated processes during automation?
What kind of bookkeeping is done? Does the app have access to earlier batches? Will we be able to identify conflicts?
Are additional limits (other than those already published) envisioned for this process?
In particular, Option B may encounter various API and processing limits.
Option B: Are there limits on the function(s) that can be defined?
For example, is container-based code allowed?
There’s a description of how Option A would fit into a migration process. What is the description for Option B?
Verification - I don’t see discussion of a verification pass, or anything of that nature. Has this been considered?
Is either option expected to log before / after of the mappings? Where will those logs be stored and who has access to them?
Testing
It is critical that this performs correctly - data corruption is unacceptable but one possible outcome of migration errors (bad/broken mapping, missing mapping, etc.). We need the ability to test these migrations and test them at enterprise-scale.
Partner testing: We need to be able to repeatedly, reliably, and inexpensively test c2c migrations.
Partner testing: We need to be able to create both source and destination test instances
Partner testing: We need to be able to automate local unit / integration testing as well as e2e regression testing for merges - this indicates API-based invocation capabilities
Customer testing: Customers will test migrations - how do we understand the cost generated by this activity?
Opt-in
Apps indicate support for migration through
Entry in the manifest
Additional documentation
How does an app indicate ‘no migration required’ separately from ‘migration required but not supported in-app’?
Definition
Mapping should be a controlled artifact
Removing them from the manifest does help keep the manifest clean
However, mappings are tightly bound to app version and need to move with the app (rather than being separately deployable)
Other Questions
An assumption seems to be that merges are like-for-like - that both source and destination have the same configuration. Is this a valid assumption? What happens if this is not the case?
Application global configuration - is that part of a migration?
If so - what happen in the case of a merge? Configuration may be different.
Is it possible to have a user defined end result?
Can mapping be applied to Confluence content properties, space properties or application properties?
Can they be applied to KVS objects that are not Custom Entities?
How will identifiers stored in places like macro configurations, post-function configurations, and other places be migrated?
Thank you @rachelminer for your detailed reply; it was very helpful to helping us identify next steps internally.
We don’t have answers to your questions immediately, but as we drill down to the next level of detail, we’ll aim to ensure that any solution has a path to giving a clear answer to partners on the key questions.
Thank you everyone for participating in this RFC - all your answers helped give us clarity on what will or will not work for partners.
We’ve decided, based on your input, that long term we’ll need to implement both Solution A and Solution B (including with good support for Forge Remotes). We will prioritise this work and deliver incrementally, with Solution A first, and assess the details of subsequent steps as we progress.
I acknowledge there was also helpful feedback given about other related topics in this RFC - for example, about providing partners the ability to opt-in or opt-out from supporting automated cloud-to-cloud migrations. We haven’t come to a decision on these details yet, but your comments have been noted for consideration as we implement cloud-to-cloud app migrations.