Introducing OpenSearch for Jira

Jira Development Jira Data Center
, , ,
21

I see 10.4 is out but the search upgrade guide is 404 at the moment

1 Like
22

Hi Reece. Apologies — it looks like the page didn’t get unlocked correctly during go-live. I’ve now made the guide available for all to view: Search API upgrade guide | Administering Jira applications Data Center 10.4 | Atlassian Documentation

23

Thanks for the documentation, the part about the org.apache.lucene.search.Collector is a bit concerning.
The main reason to use the Collector was the high performance when you need to collect only a small part of the indexed documents (e.g. only issue ids or keys). Would that be possible using the suggested aggregation?
I don’t think that there are more than a few plugin vendors out there that had anyting to do with OpenSearch and it would be helpfull if you could at least provide a few examples for the migration

1 Like
24

Thanks for reviewing the docs @m.herrmann.
The purpose of the Aggregation will only be for usages where you’re using the Collector to aggregate field values.
If you only need specific fields, we suggest using the com.atlassian.jira.search.index.IndexSearcher#scan(SearchRequest request, Function<Document, Boolean> callback) function.
We’ll update the docs to include examples as requested.

3 Likes
25

We tested our Jira app with Jira 10.4’s new Search API and found a discrepancy in how FieldIndexers are used in com.atlassian.jira.issue.customfields.searchers.information.CustomFieldSearcherInformation:

Situation

  • FieldIndexer
    • old and deprecated: com.atlassian.jira.issue.index.indexers.FieldIndexer
    • new: com.atlassian.jira.search.issue.index.indexers.FieldIndexer
  • CustomFieldSearcherInformation:
    • it now uses a Builder to construct it → is has 2 methods for FieldIndexers
      • relatedIndexers(...) that expects a list of old FieldIndexers
      • indexers(...) that expects a list of new FieldIndexers

Problem
When constructing a CustomFieldSearcherInformation with its Builders build() method, it requires both indexers(...) and relatedIndexers(...) having been called with non-empty Lists. Or in other words: It requires both old/deprecated FieldIndexers and new FieldIndexers. Otherwise the searcher cannot be set to customfields and the following error is logged:

com.atlassian.jira.util.dbc.Assertions$EmptyArgumentException: relatedIndexers should not be empty!
	at com.atlassian.jira.util.dbc.Assertions.notEmpty(Assertions.java:50)
	at com.atlassian.jira.issue.customfields.searchers.information.CustomFieldSearcherInformation.<init>(CustomFieldSearcherInformation.java:40)
	at com.atlassian.jira.issue.customfields.searchers.information.CustomFieldSearcherInformation$Builder.build(CustomFieldSearcherInformation.java:124)
	at de.communardo.jira.plugins.userprofile.customfield.searcher.MultipleSelectSearcher.init(MultipleSelectSearcher.java:74)
	at de.communardo.jira.plugins.userprofile.customfield.searcher.MultipleSelectSearcher.init(MultipleSelectSearcher.java:36)
	at com.atlassian.jira.issue.fields.ImmutableCustomField.loadCustomFieldSearcher(ImmutableCustomField.java:1888)
    ...

Questions
Is that intended? Do we as an app really have to maintain both types of FieldIndexers, one of which is marked as deprecated via this very Search API change? That seems like an oversight.
If we really have to provide both FieldIndexers, then how does Jira decide which one to use? Is it actually using the new ones, or is it still hard-wired to use the old/deprecated ones?

26

Hey @AndreasEbert this was an oversight. We detected it too late to be able to fix it in 10.4 There will be a fix for it in 10.4.1.

With 10.4.1, you’ll be able to provide only the search API indexer. We don’t expect you to implement both.

1 Like
27

@GabrielWeyer, ah, perfect. Good to hear. We’ll skip 10.4.0 then, and have another look with 10.4.1 :+1:

Another question, concerning backwards compatibility:
Seeing that the new Search API classes don’t exist prior to Jira 10.4, we can’t use them in app-versions that also support older Jira-versions (at least not without using Java Reflection). For some similar cases in the past, Atlassian provided a compatibility-helper library (or whatever such things are called), that apps could use to achieve backwards compatibility. Any plans for providing such a library here in this case?
Otherwise, we’ll probably wait till Jira 11+ and release a non-backwards compatible app-version.

28

Hi @AndreasEbert, we don’t have any plans to provide a helper library. If maintaining two versions is costly, I’d suggest the “wait till Jira 11” approach.
The primary purpose of the Search API is to decouple Search and Indexing from Lucene, allowing Jira to switch over to OpenSearch in the future. We understand there will be a significant amount of rework for many vendors and customers which is why we’re releasing it incrementally before Jira 11.
OpenSearch will only be generally available in Jira 11+, so I think waiting until Jira 11 is fine.

1 Like
29

The IssueIndexManager previously provided methods to hold and release indexing operations. However, in Jira Data Center 10.4+, neither com.atlassian.jira.search.index.IndexAccessor nor com.atlassian.jira.issue.index.IssueIndexer offer equivalent functionality to pause or resume indexing. As a result, there is currently no direct API support for programmatically holding or releasing indexing in the latest Jira version.

Would you like to include possible alternatives or workarounds?

30

Hey @denispasha, could you please describe your use case for calling hold and release? We’re currently looking at how we’ll replace internal calls to hold and release and would benefit from additional context.

1 Like
31

Hi @GabrielWeyer,
We are importing a large number of issues, and during this process, we want to temporarily hold reindexing to avoid unnecessary performance overhead and potential indexing inconsistencies. Once the import is complete, we would then release indexing to ensure all imported issues are indexed correctly in one go. This helps prevent partial or inconsistent search results during the import and improves efficiency by reducing the number of reindexing operations.

Previously, we achieved this using hold() and release(), but with the deprecation of these methods, we are looking for an alternative approach to maintain indexing control during such operations.

3 Likes
32

Thanks for explaining your use case @denispasha. We agree that this is a nice feature for optimising performance regardless of whether indexing is done by Lucene or OpenSearch.

From Jira 10.5, hold() and release() functionality will be available on the com.atlassian.jira.issue.index.IssueIndexingService, which is part of the public API and will be supported through Jira 11+.

2 Likes
33

We appreciate the clarification @ChristoferBowles ! It’s great to know that hold() and release() will be officially supported
Also, in the documentation, it’s mentioned that we should use IndexAccessorRegistry to get an IndexAccessor , which provides an IndexSearcher . The search method of IndexSearcher requires a SearchRequest , which must be built using a builder. However, the builder requires a documentType , and the documentType() method is deprecated. Will there be a new method or an alternative way to construct SearchRequest without using a deprecated method?
Last but not least currently, even though IndexAccessor is implemented by OpenSearch at runtime, we still receive a LuceneIndexAccessor . Is there any plan to introduce a flag or another mechanism to explicitly use OpenSearch instead of Lucene?

34

Hi @denispasha ,

Will there be a new method or an alternative way to construct SearchRequest without using a deprecated method?

SearchRequest.documentType is a concept that we are planning to remove, but it is still needed while we support Lucene. The alternative is OpenSearch specific, so we recommend to continue using documentType.

For Jira 10.5, we’ll remove the Deprecated annotation from SearchRequestBuilder.documentType and SearchRequest.getDocumentType.

Is there any plan to introduce a flag or another mechanism to explicitly use OpenSearch instead of Lucene?

Yes, it will be possible to configure the search platform through jira-config.properties. When configured to use OpenSearch, OpenSearch implementations of the Search API interfaces will be provided at runtime e.g. OpenSearchIndexAccessor rather than LuceneIndexAccessor.

We’ll provide details about this configuration once OpenSearch is ready.

2 Likes
35

Hi @WillYasvoin,

We’ve recently started researching OpenSearch, and my main concern, as highlighted by @AndreasEbert, is backward compatibility.

Currently, our application supports and is compatible with Jira 8, 9, and 10. Of course, the codebase remains the same, with differentiation based on the build/artifact profile for each respective Jira version.

From my initial understanding:

  • OpenSearch is not backward compatible with versions prior to Jira 10.4.
  • If we want to migrate to OpenSearch incrementally, we would need to maintain a separate codebase/flow of our application, to support/for the older Jira versions.
  • Alternatively, we could wait for Jira 11 and implement the migration in one go.

If we decide to wait for Jira 11, we assume there will be ample time before the final deprecation of the current search/Lucene library, correct?

Looking forward to hearing your thoughts. Thanks!

1 Like
36

Hi @NikhilDiwan,

Thank you for starting look into the Search API changes. I’d like to clarify that OpenSearch is not yet supported in Jira. Jira 10.4 however has the Search API, which is the new abstraction layer between Jira and Lucene. This should be used in place of the direct Lucene usage.

Lucene will still be supported at least until Jira 12, but we will no longer provide access to Lucene specific APIs through the Jira APIs (eg direct access to Lucene index readers) from Jira 11.0.

We plan to release support for OpenSearch during the lifetime of Jira 11, we’ll be sharing more details on the timeline soon.

Out of interest, how do you currently support platform 6 and 7 compatibility with a single binary?

Looking forward to hearing from you.

37

Hi @WillYasvoin ,
Thank you for the response and clarification!

I understand that OpenSearch will be incrementally available starting from Jira 11, and we can begin migrating to the new search API introduced in Jira 10.4.

We have separate binaries for platform 6 & 7, built using different java versions.

Could you please confirm the following:

  • Even if we migrate to the new search API introduced in Jira 10.4, it will not be backward compatible. If that’s the case, we would need to maintain separate application code branches for Jira versions <10.4 and Jira versions >=10.4, correct?
  • Here is the list of deprecated search api that I found, can you share if you have list of new search api list that would be replacing above deprecated api?
  • Also, this section of release notes page seems having some issue, page throwing an error, can you please check.

Thanks again for the support!

38

Hi @NikhilDiwan,

  • Correct, you’ll need separate application code for Jira versions >= 10.4 (assuming you’re using deprecated API e.g. SearchProvider).
  • The link which you shared for the deprecated search API will be updated as new versions of Jira are released.
  • Thanks for raising the broken link, we’ll update the page to fix those

Thanks
Will

1 Like
39

Greetings.
We use both the index of Jira itself and parallel to it we maintain our own.

Next we are talking about our own, internal one. It is stored in a separate directory belonging to our plugin.
We use direct access via IndexWriter / IndexReader / IndexSearcher.
How do we migrate to use OpenSearch api considering the changes? The migration process descriptions don’t specify this.
Is there any alternative to maintain our own index in parallel?

From what I can see so far, the changes are related to the use of the internal Jira index.

1 Like
40

Hi @NikolayShmakov,

Thanks for looking into this. We won’t be supporting custom indexes in OpenSearch in the first iterations. This will most likely be included after OpenSearch is GA.
I assume that you currently control the replication of documents which you write to your index? Is the data stored in this index an extension of the Jira Issue data or is it separate?

Thanks
Will