RFC-61: Evolving Search Capabilities: Addressing Scalability with a New, Enhanced Search API

General Request for Comments (RFC)
, , ,
21

Just a thought: is there any chance to have no limit access as a paid service? I understand that Atlassian has to cut costs of running infrastructure with gazillion requests from apps. But I think we as vendors can come up with a balance between fees we have to pay for fast performance and our pricing.

22

Hi, thanks for sharing this initiative!
Could you please clarify if POST /rest/api/3/search is affected when posting a JQL in the body (as opposed to /rest/api/3/search/jql)?
Sorry for the naive question.
Thank you!

23

This is quite a significant change that will take our focus away from other initiatives, such as registering on and migrating to Forge. What is the priority for Atlassian for us to focus on?

24

I have an additional concern for external data synchronization in a pull-like model via JQL search due to the eventual consistency.

Let’s take an example with some base query, eg PROJECT = ASD

With current APIs, we are able to request PROJECT = ASD and then pull new changes with query like PROJECT = ASD AND UPDATED >= PREV_UPDATE_TS AND UPDATED < CURRENT_UPDATE_TS.

CURRENT_UPDATE_TS will obviously be very close to the timestamp of “now”, so now we cannot be sure that we will retrieve all issues updated within specified timeframe due to eventual consistency.

Some services provide in their APIs field/parameter like nextSyncToken/syncToken in order to allow incremental and consistent retrieval of the data which was updated since previous search.

1 Like
25

We have concerns covered by your first ask:

Any challenges that the proposed search endpoint fails to address, especially when your application demands immediate search functionality following write consistency.

Our application makes edits to Jira configuration (at scale) as part of clean-up activities. We present the issue count associated with configuration items (e.g., custom fields) to indicate their usage. If a user performs a clean-up activity to a configuration item, then one of the ways they verify the change is to consult the issue count for that item’s new usage. With your proposed changes, we either lose this ability (because only an approximate count is provided and won’t be timely updated) or the experience is degraded (because the user will have to wait until the search becomes write consistent and the changes are reflected in the count). Overall, we are concerned that there will be a loss of accuracy in our JQL searches when this has been relied upon as authoritative until now.

We have the following clarification questions:

  • What is the duration after a change (i.e., write) where search consistency is assured?
  • Can we get an exact definition/better explanation of what ‘approximate’ means in the new approximate issue count endpoint?
  • If you will not provide an endpoint to recover the exact issue count for a JQL query, then we will have to paginate as you have described to recover it. What is the maximum value for maxResults on /search/jql? But even this search endpoint will have approximate results, so won’t this be no different to just using the approximate issue count endpoint?
26

It seems there is no way to request strong consistency for a new expression api. If it uses new search api and the list of issues is small, then why it’s not possible to reconcile them.