Trashed custom content appears in search results, breaking expansions

3 separate customers have reported this to us today and yesterday, so it seems to be a new issue on Cloud (or a big coincidence), even though this issue has existed on Server for many years. I can also only reproduce it on one instance but not on another, which speaks for it being a new problem.

Steps to reproduce:

  1. Create a page A
  2. Create a custom content object B that is attached to page A
  3. Add a JSON content property to B
  4. Trash page A

What happens now is that B still appears in CQL searches, although with "status": "trashed". Whenever B appears in the CQL results, expansions of content properties don’t work for any custom content objects in the results, even for non-trashed ones.

Page A is shown in the trash, but object B is not. When trashing object B on its own before trashing page A, still only page A appears in the trash, but the problem does not occur and object B does not appear in search results.

The workaround for our customers is to purge the trash, or at least to purge those specific pages that had custom content attached. However, this is not an acceptable solution because:

  • Purging the trash is only possible for space admins
  • One user trashing a page might break other pages for other users without noticing, those users will not know what is causing the problem and how to solve it.

Concrete example

Our app attaches custom content objects of type ac:k15t-scroll-document-versions-for-confluence:document to pages. These custom content objects need a content property K15tDocsMetadata to work properly.

In my space TT I have two pages, Page 1 and Page 2. These pages each have a custom content object of the same title attached to them.

Searching for these custom content objects and expanding the content property (/rest/api/search?cql=space=TT%20and%20type="ac:k15t-scroll-document-versions-for-confluence:document"&expand=content.metadata.properties.K15tDocsMetadata) yields this correct result:

{
    "results": [
        {
            "content": {
                "id": "445317227",
                "type": "ac:k15t-scroll-document-versions-for-confluence:document",
                "status": "current",
                "title": "Page 2",
                "metadata": {
                    "properties": {
                        "K15tDocsMetadata": {
                            "id": "445317232",
                            "key": "K15tDocsMetadata",
                            "value": {
                            }
                        }
                    }
                }
            }
        },
        {
            "content": {
                "id": "445087765",
                "type": "ac:k15t-scroll-document-versions-for-confluence:document",
                "status": "current",
                "title": "Page 1",
                "metadata": {
                    "properties": {
                        "K15tDocsMetadata": {
                            "id": "445087770",
                            "key": "K15tDocsMetadata",
                            "value": {
                            }
                        }
                    }
                }
            }
        }
    ],
    "start": 0,
    "limit": 25,
    "size": 2,
    "totalSize": 2,
    "cqlQuery": "space=TT and type=\"ac:k15t-scroll-document-versions-for-confluence:document\"",
}

Now, I trash Page 2 (through the Confluence UI). Doing the same request as above now yields this result:

{
    "results": [
        {
            "content": {
                "id": "445317227",
                "type": "ac:k15t-scroll-document-versions-for-confluence:document",
                "status": "trashed",
                "title": "Page 2",
                "metadata": {
                    "_expandable": {
                        "currentuser": "",
                        "comments": "",
                        "simple": "",
                        "frontend": "",
                        "labels": "",
                        "likes": ""
                    }
                }
            }
        },
        {
            "content": {
                "id": "445087765",
                "type": "ac:k15t-scroll-document-versions-for-confluence:document",
                "status": "current",
                "title": "Page 1",
                "metadata": {
                    "_expandable": {
                        "currentuser": "",
                        "comments": "",
                        "simple": "",
                        "frontend": "",
                        "labels": "",
                        "likes": ""
                    }
                }
            }
        }
    ]
}

As you can see, the object attached to Page 2 still appears in the results as trashed. However, the metadata.properties expansion is missing for both objects and is not even listed in _expandable.

If I modify the query in any way that only Page 1 is returned, for example by searching only for its ID, the expansion works properly again. If Page 1 was the first result, I could also achieve this with a limit=1, but because Page 2 is the first result, I cannot achieve that in this case (because the start parameter is not supported anymore).

I tried to find a way to exclude trashed items from the results, but haven’t succeeded yet. In the documentation, a contentStatuses field is mentioned for the cqlcontext parameter, but I couldn’t manage to get it to have any effect on my results at all.

4 Likes

We started experiencing very similar issue today - some custom content searches return trashed content that breaks app’s functionality, exactly as @candid described. So it seems that more apps are affected.

1 Like

One new detail about this issue that I noticed is that when querying the custom content without any expansions, it is shown as current. Only when adding some expansions such as content.space, content.container or content.metadata, the status appears as trashed.

Because this issue means for us that our app is breaking for customers when there is one trashed page somewhere in the instance, and it is not acceptable for us to ask our customers to navigate to every single space and purge the whole trash there, we ask them to run this script on a Confluence page through the dev console:

fetch('/wiki/rest/api/content/search?expand=space,container&limit=200&cql=type+%3D+%27ac%3Ak15t-scroll-document-versions-for-confluence%3Adocument%27').then((r) => r.json()).then((r) => { for(const d of r.results) d.status === 'trashed' && console.log(`${d.space.name} (${d.space.key}): ${d.container.title}`) }).catch((e) => { console.error(e); })

It lists all the page titles and space keys of those trashed pages with broken custom content. Maybe someone else who is experiencing the same problem finds this useful.

3 Likes

Hi @candid,

I haven’t had much experience with custom content so I’ve created a minimal app to try to explore this issue. Would you mind repeating the following steps to see if we observe the same behaviour:

  1. Install the app at URL https://dx-custom-content.glitch.me/connect.json
  2. Create a new page.
  3. Insert dynamic content macro ‘DX Custom Content Macro’.
  4. Publish the page.
  5. Record the content ID of the page by locating it in the URL.
  6. Click the ‘Create custom content’ link.
  7. Click the link shown under the “Search query” heading. Keep this browser window open.
  8. Verify the search results show the custom content was created with attached property keyed by ‘dx-custom-content-metadata’.
  9. Record the content ID of the custom content from the returned JSON: results[0].content.id
  10. Visit the link shown under the “Custom content page” heading to validate the content exists. Keep this browser window open.
  11. Archive the page that was created at step 4.
  12. Reload the browser window that was opened at step 7 and observe if the search results still shows the custom content.
  13. Reload the browser window that was opened at step 10.
  14. Record the status of the of the custom content in the returned JSON: results[0].content.status
  15. Deleted the archived page.
  16. Reload the browser window that was opened at step 7 and observe if the search results still shows the custom content.
  17. Reload the browser window that was opened at step 10.

I continue to get search results for the custom content after the page has been archived and deleted. Since the custom content is attached to a page, I would have expected the search for this content to find no results for pages that have been archived or deleted. Is this what you expect?

I haven’t observed the status of the custom content changing to trashed as you noted. Maybe I don’t understand your scenario.

Regards,
Dugald

1 Like

Maybe could be helpful to know that for my app, this issue is (or started to be) transient one. “trashed” content that breaks metadata expansions sometimes appears in search results up to few seconds just after content is deleted. In our case it means that users that are redirected to the list of custom content (immediately after deletion of one of it) get errors caused by this issue, although waiting couple of seconds and refreshing the page helps at least for this case.

Hey @dmorrow,

Thanks for the detailed investigation.

As far as I can tell, the custom content object that you are creating is attached to the space rather than the page. This can be seen by expanding content.container in the CQL request. So the behaviour that you are seeing, that the custom content doesn’t get deleted along with the page, is expected.

I see that you are trying to attach the custom content to the page using the ancestors property. As far as I know, only pages can have ancestors, custom content can only have a container. So if you change the payload to container: { id: contentId, type: "page" }, you should be able to reproduce the problem.

1 Like

I just tried this again and couldn’t reproduce it anymore. I heard some rumours that a fix for this was rolled out on the 18th of March. However, the fix appears to have solved the problem only for custom content that is deleted from now on. But on my instance I still have some custom content in the trash for which the problem persists.

1 Like

Hi @candid,

OK, for the moment we’ll assume the issue you identified is fixed. However, if you have existing content in customer tenants that needs to be tidied up, then I think you’ll have to work with your customers to raise support cases.

Regards,
Dugald