I assume that you currently control the replication of documents which you write to your index?
Yes. Because of the way our index is structured, we can replicate quite simply.
Is the data stored in this index an extension of the Jira Issue data or is it separate?
It is currently a separate index. We do not interfere in any way with the existing index in Jira.
However, our plans are to start using an internal index as well.
So far, our question is more about the fact that we are actually using the dependency to lucene as provided by Jira.
Will the dependency also still be provided,
or can we bring it in on our own,
or will it be considered to be blocked?
In the current Jira’s implementation we can join our indexes from different directories to perform a search.
Will an alternative to this join be presented?
Do I understand correctly that Lucene packages will be available until Jira 11? The introduction of the new API in Jira 10.4 does not cause the old API to stop working and this will only happen with the introduction of Jira 11?
You are correct. New API does not affect existing API. Lucene packages will still be available in Jira 11 however, all existing API within Jira which references Lucene, will be removed. Please refer to the Search API upgrade guide for more details.
Hello!
Version 11.0.0-m0001 is out and it has OpenSearch.
I would like some documentation on how to customize OpenSearch.
How do I switch indexing to it?
I see that there are parameters that it expects to see, but I can’t understand how the switching is managed.
Parameters:
jira.opensearch.http.url
jira.opensearch.username
jira.opensearch.password
Also in the code I’ve noticed a connection to OpenSearch via AWS.
I would like to know what it is for. Connecting a DC version of Jira to AWS service for OpenSearch looks strange at least. Or this is some copy-paste code from Cloud version of Jira?
Thanks for taking the time to look at the EAP. We will publish a guide on using OpenSearch with Jira in the next couple of weeks and I’ll link it to this thread when it’s ready.
Connecting a DC version of Jira to AWS service for OpenSearch looks strange at least
This is for customers running their Jira instance in AWS, so this is intentional. It is also a supported configuration for Confluence DC (see Option 2: authenticate to Amazon…). Switching to this configuration allows for AWS request signing.
We are in the process of implementing the new Search API, using Jira 11 EAP 1, to know early if all our search functions will be able to migrate.
One thing our team noticed is that when adding new index fields, we are required to restart the whole Jira instance for it to work.
We add a new index field in the code (name example customfield_10101_date)
Package the *.jar
Upload the app
Run indexing again
Doing this will throw the following exception:
com.atlassian.jira.search.exception.InvalidDocumentFieldException: Document contains invalid fields [customfield_10101_date]
at com.atlassian.jira.search.field.SchemaValidatedFieldValueCollector.lambda$validate$0(SchemaValidatedFieldValueCollector.java:35)
at java.base/java.util.Optional.orElseThrow(Optional.java:403)
at com.atlassian.jira.search.field.SchemaValidatedFieldValueCollector.validate(SchemaValidatedFieldValueCollector.java:35)
at com.atlassian.jira.search.field.SchemaValidatedFieldValueCollector.add(SchemaValidatedFieldValueCollector.java:28)
at com.atlassian.jira.search.field.FieldValueCollector.add(FieldValueCollector.java:18)
at com.atlassian.jira.search.field.FieldValueCollector.add(FieldValueCollector.java:22)
We have to restart the Jira instance for the indexing to work again.
This is not great to iterate quickly while developing, but we are also concerned if this behavior also affect customers when they update an app.
Our team has raised another potential limitation that I wish to discuss.
The API requires the indexed fields to be described once in FieldIndexer.getFields.
Some of our indexed field names are based on the custom field data. With this new limitation, we would not be able to store indexed fields with dynamic name since their name must be described/known beforehand.
Let’s imagine an app that stores the following value in a single custom field:
As someone that’s recently been trying out OpenSearch via the Confluence helm chart, found some interesting things, by default the index content is only spread evenly over 2 Opensearch nodes, but the default is 3 replicas, essentially meaning you have a shard sitting there with 20% of the data, if replica 0 goes down, you then dont have the indexer data on the other nodes, unless you tweak it using opensearch api functionality. Not great compared to regular ol lucene, hope the helm charts get these configs build in by default or just default config in the opensearch template file.
A large portion of the value added by our app comes from regex searches. Could you please provide the corresponding com.atlassian.jira.search.query version of the RegexpQuery?
public class DefaultRegexpQuery implements RegexpQuery {
private final String field;
private final String value;
public DefaultRegexpQuery(final String field, final String value) {
this.field = requireNonNull(field);
this.value = requireNonNull(value);
}
@Override
public String field() {
return field;
}
@Override
public String value() {
return value;
}
}
Create a corresponding LuceneQueryMapper which converts the agnostic Query to Lucene e.g.
public class RegexpQueryMapper implements LuceneQueryMapper<RegexpQuery> {
@Override
public RegexpQuery map(final RegexpQuery query) {
return new org.apache.lucene.search.RegexpQuery(new Term(query.field(), query.value()));
}
}
Register the mapper above e.g.
final LuceneQueryMapperRegistry registry = ComponentAccessor.getComponent(LuceneQueryMapperRegistry.class);
registry.registerMapper(RegexpQuery.class, this);
Finally you’ll be able to use the query when searching e.g.
final IndexAccessorRegistry registry = ComponentAccessor.getComponent(IndexAccessorRegistry.class);
final IndexAccessor indexAccessor = registry.getIssuesIndexAccessor();
final SearchResponse response = indexAccessor.getSearcher().search(
SearchRequest.builder()
.query(new DefaultRegexpQuery("summary", summary))
.documentType(DocumentTypes.ISSUE)
.build(),
PageRequest.of(0, 100));
I’ll add the following to the Search migration guide and provide an example in the Jira reference plugin.
Thanks a lot for the detailed answer! It made it very easy for us to give it a go.
We were able to add a custom LuceneQueryMapper and at first glance it works - the mapper is applied to the query.
There is, however, a problem with the lifecycle of these mappers. There is only the LuceneQueryMapperRegistry#registerMapper method, but there’s no corresponding one to unregister.
When an app is disabled and enabled, registering throws an exception, because the key class is already there in com.atlassian.jira.search.AbstractQueryMapperRegistry#mappers. We could catch this exception, but that’s ugly, isn’t it?
When an app is re-installed, the class is loaded again and added to the map. Over time this map grows, holding references to multiple instances of the same class (but from different class loaders) that should have been forgotten.
Are you planning to work on the LuceneQueryMapperRegistry to properly handle the lifecycle of apps?
BTW, LuceneQueryMapper is marked for removal in 12.0. Does it mean Jira will say a final goodbye to Lucene at that point?
