Confluence search API does not expose a mechanism for parallel processing

Hi devs,
We’ve been following the solution suggested in this thread

We needed to utilize the start and limit parameters to process requests asynchronously without depending on “_links.next” link but unfortunately start parameter seems to be missing in this API

The question, is there a way to use search API to search for pages in an instance in a parallel way and knowing the totalSize of results as well?

6 Likes

:thinking: Sir, I am not very clear with your question.
Do you wish to search for pages in an instance based on a given parameter?
And you wish to receive the totalSize of the search result?
or the totalSize of all the pages?

So the goal is to filter all pages across one instance by spaceKey, and if not provided just to bring all pages in that instance.
The totalSize parameter that we are looking for is the one provided in the API documentation and that refers to the total size of search results.

Hi @hazem.gharib,

The content search API provides the concept of a cursor to page through results. The cursor to the next page is provided as the next field in the response from a previous page of results.

The API is not really designed to be called with a high degree of concurrency. If you do so, there will be an increased likelihood of being rate limited. A better design might be to use webhooks to keep your app in sync with the content.

Further to the above, it may be tempting for you to extract the cursor query parameter from the next URL returned in a search response and use this to construct next URLs with different start query parameters and call them concurrently, however, I think the next URL is intended to be opaque and so there is no guarantee about its format. The design of the API is primarily intended to aid a user experience, rather than some kind of robot interaction. See also HATEOAS. In summary, I wouldn’t recommend this approach.

Regards,
Dugald

1 Like