No attachments accessible via REST directly after migration

DanielAlile · October 31, 2023, 8:22am

Hello,

I’m currently developing an automatic migration path for one of our Confluence plugins. The objective is to modify certain XML files after a successful migration. Our app retrieves the XML Attachments via the REST API, searches for specific strings (e.g., server Confluence page links), updates them using mapping data to their new migrated counterparts (e.g., Cloud Confluence pages), and then uploads them to the cloud. Surprisingly, this process doesn’t work on the first run but completes successfully on the second attempt:

First Execution: The corresponding REST API request for attachments returns no attachments, even though they are accessible on the cloud.
Second Execution: The corresponding REST API request for attachments returns all attachments as expected.

The current code functions as follows:

The cloud app listens for the listener-triggered and app-data-uploaded events of its server counterpart.
The cloud app queries all attachments and filters them based on specific file types.
The cloud app queries mappings with different namespaces.
The cloud app utilizes the mappings to update strings within the XML attachments.
The cloud app pushes the updated attachments to Confluence.

Example:

    private async setMigrationNotification(req, res) {
        try {
            console.log(`Received listener-triggered event`);

            // wait for event since app-data is not uploaded yet
            if (req.body.eventType === "app-data-uploaded" && !this.appDataUploaded) {
               console.log(`Received app-data-uploaded event`);

                this.appDataUploaded = true;

                this.transferId = req.body.transferId;
                this.clientKeyId = req.body.migrationDetails.confluenceClientKey;
                this.cloudUrl = req.body.migrationDetails.cloudUrl + "/wiki";

                if (await this.getAppData("attachment-data")) {
                    console.log(`App data was uploaded successfully. We can start the migration process now...`);

                    // TEST:
                    const testAttachments = await this.confluenceApiService.getAllAttachments(this.getHttpClient(this.clientKeyId));
                    if (testAttachments) {
                        console.log(`Got ${testAttachments.length} Attachments`);
                    }
                    await this.updateProgress(this.transferId, MigrationStatusEnum.IN_PROGRESS, 0, "Starting migration process.");
                    await this.startAttachmentMigration();
                }

                console.log(`Returning status code 200 to migration server.`);
                return res.status(HttpStatusCodeEnum.OK).json(true);
            }

            return res.status(HttpStatusCodeEnum.OK).json(true);
        } catch (err) {
            console.log(`Error ${err}`);
        }
    }

Is there something wrong with the code, or am I missing something else?

Thanks!

jrichards · October 31, 2023, 11:33pm

Hi @DanielAlile,

Thanks for your post. We recommend returning HTTP 2xx right away when you receive the message and then starting the work to process the files. This is because we expect a response within 5 seconds

Webhooks and events

You should queue the work for the actual processing with a slight delay (up to 60 seconds) as well to make sure the uploaded has completed in the AWS S3 bucket. Most of the back-end system are asynchronous and sometimes if you request the data right away it won’t be ready.

I know this isn’t ideal, but app migration ins’t a race to the end.

James.

DanielAlile · November 10, 2023, 2:30pm

Hello @jrichards,

Thank you for your quick reply. I have made the changes you suggested. Unfortunately, the error persists. I am using the REST endpoint api/v2/attachments to retrieve all attachments from the migrated cloud site. Although I now wait 60 seconds before starting the migration process of our application, I can only retrieve the attachments of the default demonstration space.

Observations:

Surprisingly, when I request the endpoint via cURL with the administrator’s credentials, I can retrieve all attachments, while the app cannot.
If I perform the migration steps separately (first normal space and user migration without app migration and then the app migration without space and user migration), the app can retrieve all migrated attachments.

Thanks for your help.

jrichards · November 15, 2023, 11:47pm

Hello @DanielAlile,

Can we see the code for this.confluenceApiService.getAllAttachments(this.getHttpClient(this.clientKeyId)); and this.startAttachmentMigration(); because that’s really where the action is happening.

James.

DanielAlile · November 17, 2023, 8:28am

Hi @jrichards,

this.confluenceApiService.getAllAttachments:

public async getAllAttachments(reqOrClientKey: string): Promise<ConfluenceAttachment[]> {
    let results: ConfluenceAttachment[] = [];
    let url = "api/v2/attachments";
                                                                                             
    while (url) {
        try {
            const resp = await this.asyncCloudApiRequestGET(url, reqOrClientKey);
            const attachmentData = JSON.parse(resp);
                                                                                             
            if (attachmentData.results) {
                for (const result of attachmentData.results) {
                    const attachment: ConfluenceAttachment = result as ConfluenceAttachment;
                                                                                             
                    results.push(attachment);
                }
                url = attachmentData._links?.next;
                                                                                             
                // cutoff first "/wiki/"
                if (url) {
                    const urlRegex = /(?<=\/wiki\/).*$/g;
                    const urlMatch = url.match(urlRegex) ?? "";
                    url = urlMatch[0];
                }
            } else {
                url = "";
            }
        } catch (error) {
            return [];
        }
    }
                                                                                             
    return results;
}

asyncCloudApiRequestGET

private async asyncCloudApiRequestGET(url, reqOrClientKey): Promise<any> {
    const httpClient: HostRequest = this.getHttpClient(reqOrClientKey);
                                                                           
    return new Promise((resolve, reject) => {
        httpClient.get(
            {
                url,
                headers: {
                    "X-Atlassian-Token": "no-check",
                },
            },
            async (error, callbackData, data) => {
                if (error) {
                    reject(error);
                } else {
                    resolve(data);
                }
            }
        );
    });
}

startAttachmentMigration is quite extensive and works. Therefore, you do not need to look at this part of the code.

Thank you!

DavidYu · November 21, 2023, 5:04am

Hello @DanielAlile,

I am not sure the root cause but what has been described sounds like perhaps the app does not have the authorization to retrieve the migrated attachments at that point in time but when you remigrate it manages to gain it later on.

What do you think?

DanielAlile · November 21, 2023, 8:27am

Hi @DavidYu,

That could be, but is this behavior intentional or is it a bug in our implementation?

DavidYu · November 21, 2023, 10:42pm

I would need to reproduce this issue to know but currently it doesn’t sound like an ideal experience. We could try re-authenticating which may create a JWT with the right permissions.

DanielAlile · November 24, 2023, 10:43am

To further investigate the issue, I’d like to clarify the steps to reproduce the problem. Here are the required steps:

Start app migration with multiple spaces, pages, and attachments
Query api/v2/attachments as app during app migration

If you require additional information or specific configurations, please let me know.

You suggested to re-authenticate to potentially generate a JWT with the correct permissions. Could you please provide the exact steps for re-authentication?

Thank you.

DavidYu · November 25, 2023, 3:02am

Could you please provide the exact steps for re-authentication?

I see you are probably using an Atlassian library which handles this so it would be difficult for you to do.

Let me try reproduce this so we have a common understanding are you just using something like this Bitbucket and what is your atlassian-connect.json in particular the scopes field?

DanielAlile · November 27, 2023, 8:54am

Yes, our project is based on the example you provided. Here is our atlassian-connect.json.

DavidYu · December 3, 2023, 6:54am

Hello apologies for the delay,

I attempted to reproduce your issue but was unsuccessful. The steps I took were:

Used your code making small edits to make it work and did not use pagination part.
Created a new space added a page and added an attachment
Ran the migration and saw the new attachment being returned in the cloud app

The issue described sounds like a bug but if it is one with our platform I would have expected more people to report it.

Here is the code I used:


    app.post('/migration', addon.authenticate(), async function (req, res) {

        await setMigrationNotification(req, res);
        return res.json({}, 200)
    })


    async function setMigrationNotification(req, res) {
        try {
            console.log(`Received listener-triggered event`);

            // wait for event since app-data is not uploaded yet
            if (req.body.eventType === "app-data-uploaded") {
                console.log(`Received app-data-uploaded event`);

                let clientKey = req.body.migrationDetails.confluenceClientKey;
                console.log(`App data was uploaded successfully. We can start the migration process now...`);

                // TEST:
                const testAttachments = await getAllAttachments(clientKey);
                console.log(JSON.stringify(testAttachments, null, 2))
                if (testAttachments) {
                    console.log(`Got ${testAttachments.length} Attachments`);
                }
            }

        } catch (err) {
            console.log(`Error ${err}`);
        }


        async function getAllAttachments(clientKey) {
            let results = [];
            let url = "api/v2/attachments";


            try {
                const resp = await asyncCloudApiRequestGET(url, clientKey);
                const attachmentData = JSON.parse(resp);


                for (const result of attachmentData.results) {
                    const attachment = result;

                    results.push(attachment);
                }

            } catch (error) {
                console.error(error)
            }

            return results;
        }

        async function asyncCloudApiRequestGET(url, clientKey) {
            const httpClient = addon.httpClient({ clientKey: clientKey });

            return new Promise((resolve, reject) => {
                httpClient.get(
                    {
                        url,
                        headers: {
                            "X-Atlassian-Token": "no-check",
                        },
                    },
                    async (error, callbackData, data) => {
                        if (error) {
                            reject(error);
                        } else {
                            resolve(data);
                        }
                    }
                );
            });
        }
    }

DanielAlile · December 4, 2023, 7:58am

Hi @DavidYu,

I am using a test dataset with multiple spaces, pages, and attachments. During migration, the app can receive a minority of the attachments, but most are not accessible as previously described. I would guess that you need to test with more than one attachment. The test dataset I used contains more than 50 attachments.