Update
Thank you so much for your feedback! We really appreciate you taking the time to share your thoughts with us.
At this moment, we can’t offer larger batch sizes. We know that eventual consistency might be a bit confusing for some of you. We’ll put together some helpful tips on how to work with it. We’re also looking into the possible options for expand in the new APIs. Thanks for your understanding!
The deprecation announcement will be slightly delayed, and you can expect to hear from us about it in mid to late October. After a 6-month deprecation period, the old APIs will be phased out.
Summary of Project:
The use of JQL has grown in scale and complexity over the years, and we’ve witnessed increased scalability and reliability issues. For that, our team has been working on an architectural refresh, which will allow our JQL capabilities to scale and perform for our growing customer and vendor needs. This RFC is here to gather feedback for the proposed API changes that will give you access to the benefits of our architectural refresh.
The proposed API changes introduces a new pagination API pattern (via nextToken) and eventual consistency behaviours. We believe these two API pattern changes deserve your attention.
- Publish : Sep 03, 2024
- Discuss : Sep 17, 2024
- Resolve : Sep 21, 2024
Problem
Over the years, Jira has grown its customer base, and its users are using more of Jira, in larger scale and more complex ways. JQL is one of the core capabilities of Jira, and we’ve noticed that JQL APIs, which used to work very well, are now being challenged by the scale and complexity of how users use the APIs. JQL needs to evolve to stay reliable, performant, and become more scalable to serve bigger, more complex use cases.
The problems we’re witnessing largely stems from the access patterns of the APIs.
Today, API users are able to use JQL to access paginated results randomly. To provide random access pagination, an exact number of previous issues must be found. For instance, if issues between 10000 to 10025 are requested, JQL filtering and ordering need to be performed on the prior 10000 issues to accurately return the results. This is slow for the user, costly for us, and not scalable for the JQL result sizes that we’re seeing.
API users are also able to get the total count of issues of a JQL query. To provide a total count of matched issues on every page, it requires us to calculate the number of issues matching the JQL query repeatedly. We’ve observed performance degradation for this use case, particularly when the JQL query is run across large data sets.
Given how our APIs return much of the issue’s data by default, the payloads returned by our APIs are getting bigger, to the point where we see OOM (out of memory) errors. The growing payloads returned by default are impacting the reliability of all of the systems involved.
The scope of API changes in this RFC is to address the reliability, performance, and scalability concerns outlined above. Please read on and give us feedback on the proposed API changes.
Proposed Solution
The proposal involves these API changes.
- New endpoints:
- Removal of these endpoints:
- graduate the following API from “experimental” to “generally available”
- POST
/rest/api/{2|3|latest}/jql/parse→ link
- POST
- Changes to the amount of data returned per issue when fetching multiple issues. Read more on this below.
We don’t plan any changes to:
Changes to Issue & field data returned in the new APIs
These changes will only impact the following new APIs which return Issue field data across numerous issues:
- Bulk Fetch Issues
/rest/api/{2|3|latest}/issue/bulkfetchGET/rest/api/{2|3|latest}/search/jqlPOST /rest/api/{2|3|latest}/search/jqlThe existing Get Issue API (
/rest/api/{2|3|latest}/issue/{issueIdOrKey}) will not be changed.
To improve the performance and stability of APIs which return data for many issues, we are proposing changes to the amount of data available & returned per issue for these new endpoints. These changes aim to encourage integrations to only fetch the data needed to power their experiences, and looks to avoid excessive payload sizes when returning data from Jira Cloud.
The proposed changes are:
- Changes to which fields are included by default
/rest/api/{2|3|latest}/issue/bulkfetchwill return allnavigablefields by default, instead of all fields used in the single issue Get Issue API./rest/api/{2|3|latest}/search/jqlwill return just theidfield by default- In both cases, fields can still be explicitly included and excluded, and both will also still accept
*navigable&*allas options. However, fewer issues may be returned in cases where request payloads sizes become excessively large.
- When the
commentfield is requested, we will return a maximum of 20 comments per issue.- To fetch more comments for an issue, please use the Get comments API
- When the
changelogexpand parameter is provided, we will return a maximum of 20 changelogs per issue.- To fetch more changelogs for an issue, please use the Get changelogs API.
- Removal of API options which are less useful within the context of working with multiple issues and/or drastically increase response payload sizes when working across large numbers of issues with large numbers of fields.
- The Bulk Fetch Issue API
/rest/api/{2|3|latest}/issue/bulkfetchwill not include the theupdateHistoryparameter used in the single issue Get Issue API to flag if the request should treat the user as “viewing” the issue for the context of the Last Viewed field. This is to match the behaviour of other search & multi-issue views of Issues across Jira Cloud products. - The following
expandoptions will not be provided on the new APIsoperationsReturns all possible operations per issue.transitionsReturns all possible transitions per issueseditmetaReturns information about how each field can be edited.versionedRepresentationsReturns a JSON array for each version of a field’s value, with the highest number representing the most recent version. Note: When included in the request, thefieldsparameter is ignored.
- To use these parameters, please use the single issue Get Issue API
- The Bulk Fetch Issue API
API definition
Search API
POST /rest/api/3/search/jql
Main changes relative to the current endpoint:
- By default, the endpoint will only return Issue IDs. Unless requested, all other issue fields will be omitted in the Issue objects. You can request them by using the
fieldsproperty. - The endpoint doesn’t provide immediate search after write consistency. Search results may not incorporate recent changes made by users. If such behaviour is important for your case, please refer to the migration path at the end of this page.
- The
validationModeparameter is removed. - We’ll replace random page access with a continuation token API. This means you won’t be able to get multiple pages at the same time with parallel threads.
startAtparameter will be replaced withnextPageToken. You can find usage instructions at the end of this page. - We will only return a maximum of 20 comments and 20 changelog items. If you require more, please refer to the migration guide at the end of this page.
- The endpoint won’t accept unbounded JQL query and will return 400 if an unbounded JQL query is provided.
Documentation available here
Approximate count
POST /rest/api/3/search/approximate-count
Documentation available here
Expression Eval
POST /rest/api/3/expression/evaluate
The evaluate Jira expression endpoint is complex and it accepts many parameters. The changes we’re proposing are related to how Jira builds the issues.jql context. The main changes are:
- The endpoint won’t accept unbounded JQL query and will return 400 if an unbounded JQL query is provided.
- The maximum number for
maxResultsis 5000. The actual number of returned issues may be lower, depending on the complexity of the request. - We’ll replace random page access with a continuation token API. This means you won’t be able to get multiple pages at the same time with parallel threads. startAt parameter will be replaced with nextPageToken. You can find usage instructions at the end of this page.
- No JQL validation will be available.
- The endpoint won’t return the total count of matched issues.
Documentation available here
Bulk Fetch Issue
POST to /rest/api/{2|3|latest}/issue/bulkfetch
Documentation available here
Example scenarios & FAQ
We have identified a number of typical scenarios where current APIs are used and have came up migration paths for the new endpoints.
I want to fetch data from lots of issues at once
This operation may be achieved by utilizing two Jira’s endpoints:
- First get issue IDs via
/rest/api/3/search/jql
curl --location 'https://example-jira.atlassian.net/rest/api/latest/search/jql' \
--header 'Content-Type: application/json' \
--header 'Accept: application/json' \
--data '{
"jql":"project in (FOO, BAR)"
}'
Response:
{
"issues": [
{
"id": "10068"
},
{
"id": "10067"
},
{
"id": "10066"
}
],
"nextPageToken": "CAEaAggD"
}
- Then hydrate issues’ fields with Bulk Fetch API to
/rest/api/{2|3|latest}/issue/bulkfetch
curl --location 'https://example-jira.atlassian.net/rest/api/latest/issue/bulkfetch' \
--header 'Content-Type: application/json' \
--header 'Accept: application/json' \
--data '{
"issueIdsOrKeys": ["FOO-1","10067", "BAR-1"],
"fields": ["priority", "status", "summary"]
}'
I need to hydrate data for known issue ids
You can use Bulk Fetch Issue endpoint
POST to /rest/api/{2|3|latest}/issue/bulkfetch
Example request:
curl --location 'https://example-jira.atlassian.net/rest/api/latest/issue/bulkfetch' \
--header 'Content-Type: application/json' \
--header 'Accept: application/json' \
--data '{
"issueIdsOrKeys": ["FOO-1","10067", "BAR-1"],
"fields": ["priority", "status", "summary"]
}'
I want to know how many issues match my JQL
You can use approximate count endpoint
POST /rest/api/3/search/approximate-count
curl --location 'https://example-jira.atlassian.net/rest/api/latest/search/approximate-count' \
--header 'Content-Type: application/json' \
--header 'Accept: application/json' \
--data '{
"jql":"project in (FOO, BAR)"
}'
Response
{
"count": 3
}
I need to validate query
You can use JQL Parse endpoint
POST /rest/api/3/jql/parse
curl --location 'https://example-jira.atlassian.net/rest/api/latest/jql/parse' \
--header 'Content-Type: application/json' \
--header 'Accept: application/json' \
--data '{
"queries":["project in (FOO, BAR)"]
}'
Response
{
"queries": [
{
"query": "project in (FOO, BAR)",
"structure": {
"where": {
"field": {
"name": "project",
"encodedName": "project"
},
"operator": "in",
"operand": {
"values": [
{
"value": "FOO",
"encodedValue": "FOO"
},
{
"value": "BAR",
"encodedValue": "BAR"
}
],
"encodedOperand": "(FOO, BAR)"
}
}
}
}
]
}
I require immediate search after write consistency
You can use /rest/api/3/search/jql endpoint. Firstly, you need to specify issue id’s for which you require higher consistency guarantees. Example request would look like
curl --location 'https://example-jira.atlassian.net/rest/api/latest/search/jql' \
--header 'Content-Type: application/json' \
--header 'Accept: application/json' \
--data '{
"jql":"project in (FOO, BAR)",
"fields": "key, id",
"reconcileIssues": [10068]
}'
Response:
{
"issues": [
{
"expand": "operations,versionedRepresentations,editmeta,changelog,renderedFields",
"id": "10068",
"self": "https://example-jira.atlassian.net/rest/api/latest/issue/10068",
"key": "FOO-1"
},
{
"expand": "operations,versionedRepresentations,editmeta,changelog,renderedFields",
"id": "10067",
"self": "https://example-jira.atlassian.net/rest/api/latest/issue/10067",
"key": "BAR-2"
},
{
"expand": "operations,versionedRepresentations,editmeta,changelog,renderedFields",
"id": "10066",
"self": "https://example-jira.atlassian.net/rest/api/latest/issue/10066",
"key": "BAR-1"
}
],
"nextPageToken": "CAEaAggD"
}
-
Typical scenario would involve Application listening to Webhooks.
-
Application subscribes to WebHook Events
-
New issue event arrives
{
"issue": {
"id": "10068",
...
"fields":[
...
"updated": "2024-08-19T11:40:30.010+0200",
]
}
}
Application requests JQL search with higher consistency guarantee for issue 10012
{
"jql": "project in (FOO, BAR)",
"reconcileIssues": [10068]
}
I need to paginate over large set of results
POST /rest/api/3/search/jql
Use next page token returned on each request to get further results
curl --location 'https://example-jira.atlassian.net/rest/api/latest/search/jql' \
--header 'Content-Type: application/json' \
--header 'Accept: application/json' \
--data '{
"jql":"project in (FOO, BAR)",
"maxResults": 1
}'
Response:
{
"issues": [
{
"expand": "operations,versionedRepresentations,editmeta,changelog,renderedFields",
"id": "10068",
"self": "https://example-jira.atlassian.net/rest/api/latest/issue/10068",
"key": "FOO-1"
}
],
"nextPageToken": "CAEaAggB"
}
Request
curl --location 'https://example-jira.atlassian.net/rest/api/latest/search/jql' \
--header 'Content-Type: application/json' \
--header 'Accept: application/json' \
--data '{
"jql":"project in (FOO, BAR)",
"maxResults": 1,
"nextPageToken": "CAEaAggB"
}'
Response
{
"issues": [
{
"expand": "operations,versionedRepresentations,editmeta,changelog,renderedFields",
"id": "10067",
"self": "https://example-jira.atlassian.net/rest/api/latest/issue/10067",
"key": "BAR-2"
}
],
"nextPageToken": "CAEaAggC"
}
I need more than 20 comments for an issue
GET /rest/api/3/issue/{issueIdOrKey}/comment
curl --location 'https://example-jira.atlassian.net/rest/api/latest/issue/BAR-1/comment' \
--header 'Content-Type: application/json' \
--header 'Accept: application/json' \
--data ''
Response
{
"startAt": 0,
"maxResults": 100,
"total": 3,
"comments": [
...
]
}
I need more than 20 changelog items for an issue
Use Get changelogs endpoint
GET /rest/api/3/issue/{issueIdOrKey}/changelog
curl --location 'https://example-jira.atlassian.net/rest/api/latest/issue/BAR-1/changelog' \
--header 'Content-Type: application/json' \
--header 'Accept: application/json' \`
Response:
{
"self": "https://example-jira.atlassian.net/rest/api/2/issue/BAR-1/changelog?maxResults=100&startAt=0",
"maxResults": 100,
"startAt": 0,
"total": 3,
"isLast": true,
"values": [
...
]
}
I want to evaluate my Jira Expression
You can use new evaluate expression endpoint POST /rest/api/3/expression/evaluate
Example request for the first page
curl --request POST \
--url 'https://example-jira.atlassian.net/rest/api/3/expression/evaluate' \
--header 'Content-Type: application/json' \
--header 'Accept: application/json' \
--data '{
"context": {
"issues": {
"jql": {
"maxResults": 5,
"query": "summary ~ \"task\"",
}
}
},
"expression": "issues.map(i => i.summary)"
}'
Response:
{
"value": [
"task 7",
"task 6",
"task 5",
"task 4",
"task 3"
],
"meta": {
"issues": {
"jql": {
"nextPageToken": "CAEaAggF"
}
}
}
}
Example request for subsequent pages
curl --request POST \
--url 'https://example-jira.atlassian.net/rest/api/3/expression/evaluate' \
--header 'Content-Type: application/json' \
--header 'Accept: application/json' \
--data '{
"context": {
"issues": {
"jql": {
"maxResults": 5,
"query": "summary ~ \"task\"",
"nextPageToken":"CAEaAggF"
}
}
},
"expression": "issues.map(i => i.summary)"
}'
Response:
{
"value": [
"task 2",
"task 1",
"test task"
],
"meta": {
"issues": {
"jql": {}
}
}
}
I need to run unbounded JQL query
We do not recommend running unbounded JQL queries. These queries are slow for users and costly for every system involved. As our customers grow in number and size over time, we see customers with millions of Jira issues and this trend will continue.
We strongly recommend you bound your JQL queries to give our users the best performance and reliability. Consider bounding your queries like:
updated > -1m
How long the next page token is valid for?
Next Page Token will expire after 7 days of issuing.
Timeline
We’re sharing a timeline for these changes effective:
- Sep 03, 2024 → We’re publishing this RFC seeking your feedback and making new experimental endpoints publicly available
- Sep 17, 2024 → We’ll close this RFC for discussion
- Sep 21, 2024 → We’ll resolve this RFC and share any planned updates with you based on your feedback
- Sep 30, 2024 (tentative) → Depending on feedback, we’re targeting this date to launch the new APIs, and mark the existing JQL APIs as deprecated. Rework may push this date out.
- After deprecaction notice ends → Deprecated JQL APIs are removed from service.
Asks
While we would appreciate any reaction you may have to this RFC, we’re especially interested in:
- any challenges that the proposed search endpoint fails to address, especially when your application demands immediate search functionality following write consistency.
- any aspect of the search API that may have been overlooked and is not covered by any of the newly suggested endpoints
- any feedback on the proposed changes to issue data returned in the new endpoints detailed on this page