The expanded options you referred to are exclusively accessible in the Get Issue endpoint. Could you kindly provide us with more details about your use case involving editmeta and versionedRepresentations?
can You ensure that asking for ex for 100k issues ids using new API on large client jira instance will be fast enough to fetch issue details later in the time we have left after fetching ids (in comparison with old issue search API)??
We cannot test every possible scenario on each individual tenant. Nevertheless, our data indicates that the new search engine outperforms the old one. For instance, 99% of searches are completed within 500 ms using the new search engine, compared to 94% with the old one. The larger the tenant, the more pronounced the benefit.
Additionally I think that some pugins which performs on smaller client instances, do not encountered any performance issues, so maybe can You only mark them as deprecated, but dont remove them?
Regrettably, retaining both endpoints will be a source of confusion for API users and would require maintaining both services, which is not feasible.
- Removal of startAt parameter: In the current setup, we use the startAt parameter for pagination. While the introduction of nextPageToken can replace this, the lack of a total count of issues creates challenges for handling pagination. Without knowing the total number of issues, we cannot accurately determine how many pages of results we need to fetch. This would complicate the pagination logic.
Two methods are available to determine the number of pages:
- The total count of matching issues can be obtained using the
approximate-countendpoint. - The
nextPageTokenwill indicate that you have reached the last page when it is null.
Loss of editmeta in expand: Currently, we retrieve editable field metadata in a single request for 50 issues. With the new API, we would need to send a separate request for each issue to get the editmeta information. This would increase the number of requests from 1 to 51 for every 50 issues, creating considerable overhead and reducing system performance
Additional questions: Is there any alternative approach planned for retrieving editable field information for bulk issue processing? This is critical for our use case, where we need to check if specific fields are editable by the current user.
Could you kindly explain how you utilize the editmetadata information?
Thanks!
Thank you for your response.
In our current implementation (Spreadsheet Issue Field Editor plugin), we rely on the expand=editmeta parameter in the POST /rest/api/{2|3|latest}/search endpoint to retrieve the editable fields metadata in bulk. This includes important information for each issue, such as the field’s name, key, allowedValues (if applicable), and whether the field is editable, which we determine by checking if the operations array contains the value set.
This allows users to edit multiple issue fields directly in a spreadsheet interface, without the need to navigate to each individual issue. The bulk retrieval of editmeta in a single request allows us to avoid making a separate request to /rest/api/3/issue/{issueIdOrKey}/editmeta for every issue.
Sure, thank you.
Our product integrates with Jira and heavily uses the search api to bulk import the issues.
Below is the info about usage of editmeta and versionedRepresentation expand options.
EditMeta
When fetching Jira issues via the Search API, we rely on the editMeta information to obtain details about custom fields—such as their names—which allows us to retrieve specific custom field values from the issue payload. currently we are using epic-link (com.pyxis.greenhopper.jira:gh-epic-link) and sprint (com.pyxis.greenhopper.jira:gh-sprint) and more field will be supported in future.
For example, if the import process requires the Jira sprint custom field value from the issue payload, we use the editMeta data to filter for the custom field associated with “sprint” and use its name to extract the corresponding value from issue payload.
VersionedRepresentation
VersionedRepresentation contains more data, i.e. for epic links version 1 contains only issue key for the Epic, version == 2 contains id, key, summary etc. If versionedRepresentation expand param is not supported in the new bulk fetch api, we will have to load the required information by calling get issue request separately for each issue. It will drastically increase the amount of requests to jira and take more time. this means it will cause more load to Jira side as well.
We are planning to use the bulkfetch api to replace with issue search api. Therefore, having access to editMeta data within the bulk-fetch API is crucial for us. Without it, we would be forced to use the “Get Issue” API, which significantly reduces performance and would slow down the process, ultimately leading to a poor user experience.
Thank you @glewandowski for your answer.
We are after some performance tests with one of our biggest client, and I would like to inform You that new API does not perform very well… Client has about 6 - 7 million issues on their JSM and new API still needs 25-30 seconds to return issues ids. I know that such instance is large, but such response time is not acceptable for our big clients.
Are your dev teams still working under performance improvements on new API?
Hi there,
Thanks a lot for this RFC @glewandowski and let us know previously the changes you have been preparing for the search API. I believe it can be one of the most used sections of the Jira API, so it may impact lots of Marketplace partners & apps.
In fact, three of our Cloud apps are using it in different use cases, but specially one of it is using it massively.
I have a big concern about the performance: with the current API, when getting lots of issue pages for the same JQL using the search API, the performance was getting worse and worse. Can we expect better performance with this new approach when accessing large amount of data?
Another concern about the new approach will be related to accessing issue pages in parallel. Have you though about the use cases that will perform badly because of this is not possible anymore?
On the other hand, I will really love to be able to have a bigger page size than 100 for the search API. There are lots of apps accessing lots of issues for different use cases and it may improve a lot their performance and reaching API usage limits will not be so easy. Maybe open it for Marketplace partners after some circumstances/consideration will be an option here not to open to everyone.
Have a nice day!
There are typically two main scenarios:
- Load only relevant data: For example, load only the latest comment for a specific set of known issues.
- Efficient batching: For a set of issues, use an expression to load all associated comments. The expression API has a limit of 1000 comments, which is sufficient for most issues. For any issues that exceed this limit, they are identified, and we use a fallback to the regular API.
Hey, thank you for sharing these outcomes with us. Yes, we are still tuning the performance of architecture supporting the new APIs, so you can expect much better performance when we transition from Experimental to Generally Available.
I have a big concern about the performance: with the current API, when getting lots of issue pages for the same JQL using the search API, the performance was getting worse and worse. Can we expect better performance with this new approach when accessing large amount of data?
Another concern about the new approach will be related to accessing issue pages in parallel. Have you though about the use cases that will perform badly because of this is not possible anymore?
Although we cannot test every possible scenario for each customer, we have discovered that the new service, when utilizing a continuation token, does not exhibit this behavior. Fetching subsequent pages should be just as swift as retrieving the initial one. If you need to hydrate data related to issues, you can continue to utilize a parallel approach with bulk fetch issues endpoint.
On the other hand, I will really love to be able to have a bigger page size than 100 for the search API. There are lots of apps accessing lots of issues for different use cases and it may improve a lot their performance and reaching API usage limits will not be so easy. Maybe open it for Marketplace partners after some circumstances/consideration will be an option here not to open to everyone.
At this time, we are unable to provide a larger batch size for hydration.
Please note that random page access is not supported with the continuation token API. Instead, you can implement either linear or scrolling navigation for a smoother user experience. If you prefer to maintain paginated navigation, we recommend caching the search results within your application to facilitate this functionality.
To which degree of parallelism can we aim at when loading issues via bulkfetch in large-ish datasets per single action/request?
@glewandowski, I have the same scenario as @lexek-92. Our app typically performs also the same queries:
With current APIs, we are able to request
PROJECT = ASDand then pull new changes with query likePROJECT = ASD AND UPDATED >= PREV_UPDATE_TS AND UPDATED < CURRENT_UPDATE_TS.
We’re already combining our approach for some parts with webhooks but this isn’t feasible for all scenarios. Is there a way I can retrieve guaranteed all issues without using webhooks?
Assume this scenario:
- 10:05: My app performs a search ‘PROJECT = A’ and updated >= 10:00
- 10:06: Issue A-1 and A-2 get updated
- 10:10: My app performs a search ‘PROJECT = A’ and updated >= 10:05
- Due to eventual consistency only A-1 will be returned
- 10:15: My app performs a search ‘PROJECT = A’ and updated >= 10:10
A-2 won’t be returned since it doesn’t qualify the search criteria anymore
What would you recommend to get consistency in this search example? I don’t care too much about receiving all issues immediately, but I want to guarantee that I’m getting all updated issues in the end.
PS: The times in the example are only to illustrate the problem. I know that the consistency is designed more around seconds instead of minutes.
@glewandowski what do you think about the mentioned usage of these expand options editmeta and versionedRepresentations.
would it still possible to support in the new issue APIs, I see plenty of people coming with the use cases of editmeta with the need of bulk.
Hey @glewandowski what’s the latest update on this RFC?
The last comment was 22 days ago and the date for “resolving” the RFC was 24 days ago, but there’s not been an official deprecation announcement on the Developer Changelog yet.
I can see the new APIs are available though.
Hey Jon! I posted an update to this RFC a little while back. We’re currently working on incorporating some feedback we’ve received, and the deprecation has been pushed back from the original timeline. The new target date for the deprecation is now the end of this month.
I’d suggest the team write code to consume the API to pickup on these flaws:
nextPageTokenis inconsistent with how pagination is handled in ALL other API endpoints.- The simplest way to consume this would be to set
nextPageToken=nullornextPageToken=''then use that in the first request and update if new property exists. Instead that throws an error.
For the sake of basic DX, one would hope there’s an org-wide document for consistent APIs. On pagination alone this is what we developers have to deal with…
Confluence v1: start and limit and _links.next
Confluence v2: cursor and limit and _links.next (with duplicate /wiki in next links lol)
Jira v2: startAt and maxResults and total
Jira v3: nextPageToken and maxResults
I’ve banged on about this before too, but I wish Atlassian would learn how to develop modern APIs that don’t need to be broken every version. Stripe APIs from 13 years ago still work without breaking developer’s apps: APIs as infrastructure: future-proofing Stripe with versioning
The deprecation has been announced on the changelog now.
This section mentions in regards to my question:
How do I reach consistency when periodically polling with search?
Depending on your needs, we recommend the following:
- Polling with overlapping periods of 5 minutes if you do not rely on non-organic updates.
- Polling with overlapping periods of 25 minutes if you do depend on non-organic updates, or if you are depending on results of operations that impact numerous issues.
To increase the level of assurance, we recommend combining the above tactics with the consumption of Webhook events.
Wow, an overlapping polling with a 25 minutes timeframe means we get a lot of issues which we’ve already got with the previous search. That probably means for our app that we query for a lot of issues which we’ve already processed (and wouldn’t need to query and require Atlassian computing resources for the query again). That doesn’t feel like a great solution to me 
The changelog is out now and both issues have not been addressed:
- The Connect scope required for Bulk fetch issues is still
WRITE - The documented limit for Bulk fetch issues is still 100
@glewandowski please address these issues as you have announced before. We cannot start with any code migration before these uncertainties have been clarified.
Thanks!
On top of the very important concerns mentioned by @matthias and @BenRomberg which weren’t properly addressed, we are also not seeing any solution for removal of editmeta expand in search APIs.