Reindex fails for custom fields

Jira Development Jira Server
1

Hi everyone!
I have some scripted custom fields in a Jira test instance and I have some troubles with them when trying to reindex.

The scripted field searches through a project for a certain issue based on another custom field’s value.
The scripted field runs smoothly in normal use and the info I am trying to display is present when I open the issues where the field is configured.
The bit of code where I search for the certain issue looks like:

//Step 3. Get the corresponding issue (project) for processing and process the PREG issue and retrieve info from custom fields
PagerFilter pagerFilter = PagerFilter.getUnlimitedFilter();
com.atlassian.jira.issue.search.SearchResults searchResults = null;

try {
    //execute the query and retrieve results
    threadLocalSearcherCache.startSearcherContext();
    searchResults = searchService.search(user, query, pagerFilter);
    //log.debug ("Total issues: ${searchResults.total}")
} catch (SearchException e) {
    e.printStackTrace();
    return null ;
} finally {
    threadLocalSearcherCache.stopAndCloseSearcherContext();
}

//store the results into a list
Collection theIssues = searchResults.getResults()

I found a thread on the community with an idea where this all could come from - starting and stopping the searcher manually - and implemented that:

As I said, all runs smoothly when displaying the issues, but when I try to reindex the entire instance, I get the below errors in the logs:

2020-02-11 15:39:23,224 IssueIndexer:thread-20 ERROR username 933x111x1 1pigcih 2001:db8:8586:11:409d:3c6e:67c7:af4 /secure/admin/IndexReIndex!reindex.jspa [c.o.scriptrunner.customfield.GroovyCustomField] *************************************************************************************
    Script field failed on issue: INT_PROJ_ENH_2019-1193, field: WorkRequest proj. prop.
com.atlassian.jira.issue.index.SearchUnavailableException
	at com.atlassian.jira.issue.index.DefaultIndexManager.getEntitySearcher(DefaultIndexManager.java:885)
	at com.atlassian.jira.issue.index.DefaultIndexManager.getIssueSearcher(DefaultIndexManager.java:865)
	at com.atlassian.jira.config.component.SwitchingInvocationHandler.invoke(SwitchingInvocationHandler.java:38)
	at com.sun.proxy.$Proxy8.getIssueSearcher(Unknown Source)
	at com.atlassian.jira.issue.search.SearchProviderFactoryImpl.getSearcher(SearchProviderFactoryImpl.java:17)
	at com.atlassian.jira.issue.search.providers.LuceneSearchProvider.getIssueSearcher(LuceneSearchProvider.java:130)
	at com.atlassian.jira.issue.search.providers.LuceneSearchProvider.searchDocuments(LuceneSearchProvider.java:443)
	at com.atlassian.jira.issue.search.providers.LuceneSearchProvider.searchDocuments(LuceneSearchProvider.java:450)
	at com.atlassian.jira.issue.search.providers.LuceneSearchProvider.runSearch(LuceneSearchProvider.java:439)
	at com.atlassian.jira.issue.search.providers.LuceneSearchProvider.getHits(LuceneSearchProvider.java:218)
	at com.atlassian.jira.issue.search.providers.LuceneSearchProvider.search(LuceneSearchProvider.java:362)
	at com.atlassian.jira.issue.search.providers.LuceneSearchProvider.search(LuceneSearchProvider.java:135)
	at com.atlassian.jira.issue.search.providers.LuceneSearchProvider.search(LuceneSearchProvider.java:140)
	at com.atlassian.jira.bc.issue.search.DefaultSearchService.search(DefaultSearchService.java:118)
	at com.atlassian.jira.bc.issue.search.SearchService$search.call(Unknown Source)
	at Script5.run(Script5.groovy:81)
2020-02-11 15:39:23,292 IssueIndexer:thread-12 ERROR username 933x111x1 1pigcih 2001:db8:8586:11:409d:3c6e:67c7:af4 /secure/admin/IndexReIndex!reindex.jspa [c.a.j.issue.index.DefaultIndexManager] Wait attempt timed out - waited 30000 milliseconds
com.atlassian.jira.issue.index.IndexException: Wait attempt timed out - waited 30000 milliseconds
	at com.atlassian.jira.issue.index.DefaultIndexManager.obtain(DefaultIndexManager.java:831)
	at com.atlassian.jira.issue.index.DefaultIndexManager.access$600(DefaultIndexManager.java:93)
	at com.atlassian.jira.issue.index.DefaultIndexManager$IndexLock.tryLock(DefaultIndexManager.java:1173)
	at com.atlassian.jira.issue.index.DefaultIndexManager.getIndexLock(DefaultIndexManager.java:817)
	at com.atlassian.jira.issue.index.DefaultIndexManager.getEntitySearcher(DefaultIndexManager.java:884)
	at com.atlassian.jira.issue.index.DefaultIndexManager.getIssueSearcher(DefaultIndexManager.java:865)
	... 2 filtered
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.atlassian.jira.config.component.SwitchingInvocationHandler.invoke(SwitchingInvocationHandler.java:38)
	at com.sun.proxy.$Proxy8.getIssueSearcher(Unknown Source)
	at com.atlassian.jira.issue.search.SearchProviderFactoryImpl.getSearcher(SearchProviderFactoryImpl.java:17)
	at com.atlassian.jira.issue.search.providers.LuceneSearchProvider.getIssueSearcher(LuceneSearchProvider.java:130)
	at com.atlassian.jira.issue.search.providers.LuceneSearchProvider.searchDocuments(LuceneSearchProvider.java:443)
	at com.atlassian.jira.issue.search.providers.LuceneSearchProvider.searchDocuments(LuceneSearchProvider.java:450)
	at com.atlassian.jira.issue.search.providers.LuceneSearchProvider.runSearch(LuceneSearchProvider.java:439)
	at com.atlassian.jira.issue.search.providers.LuceneSearchProvider.getHits(LuceneSearchProvider.java:218)
	at com.atlassian.jira.issue.search.providers.LuceneSearchProvider.search(LuceneSearchProvider.java:362)
	at com.atlassian.jira.issue.search.providers.LuceneSearchProvider.search(LuceneSearchProvider.java:135)
	at com.atlassian.jira.issue.search.providers.LuceneSearchProvider.search(LuceneSearchProvider.java:140)
	at com.atlassian.jira.bc.issue.search.DefaultSearchService.search(DefaultSearchService.java:118)
	at com.atlassian.jira.bc.issue.search.SearchService$search.call(Unknown Source)
	at Script5.run(Script5.groovy:81)
	at org.codehaus.groovy.jsr223.GroovyScriptEngineImpl.eval(GroovyScriptEngineImpl.java:321)
	at org.codehaus.groovy.jsr223.GroovyScriptEngineImpl.eval(GroovyScriptEngineImpl.java:159)
	at javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:233)
	at javax.script.ScriptEngine$eval.call(Unknown Source)
	at com.onresolve.scriptrunner.runner.AbstractScriptRunner.runScriptAndGetContext(AbstractScriptRunner.groovy:106)
	at com.onresolve.scriptrunner.runner.AbstractScriptRunner$runScriptAndGetContext$1.callCurrent(Unknown Source)
	at com.onresolve.scriptrunner.runner.AbstractScriptRunner.runStringAsScript(AbstractScriptRunner.groovy:226)
	at com.onresolve.scriptrunner.runner.AbstractScriptRunner$runStringAsScript$0.callCurrent(Unknown Source)
	at com.onresolve.scriptrunner.runner.AbstractScriptRunner.runScript(AbstractScriptRunner.groovy:218)
	at com.onresolve.scriptrunner.runner.ScriptRunner$runScript$7.call(Unknown Source)
	at com.onresolve.scriptrunner.customfield.GroovyCustomField.getValueFromIssue(GroovyCustomField.groovy:299)
	at com.atlassian.jira.issue.fields.ImmutableCustomField.getValue(ImmutableCustomField.java:350)
	at com.atlassian.jira.issue.index.indexers.impl.SortableTextCustomFieldIndexer.addDocumentFields(SortableTextCustomFieldIndexer.java:43)
	at com.atlassian.jira.issue.index.indexers.impl.SortableTextCustomFieldIndexer.addDocumentFieldsSearchable(SortableTextCustomFieldIndexer.java:35)
	at com.atlassian.jira.issue.index.indexers.impl.AbstractCustomFieldIndexer.addIndex(AbstractCustomFieldIndexer.java:40)
	at com.atlassian.jira.issue.index.indexers.FieldIndexerWithStats.addIndex(FieldIndexerWithStats.java:54)
	at com.atlassian.jira.issue.index.DefaultIssueDocumentFactory$Builder.add(DefaultIssueDocumentFactory.java:93)
	at com.atlassian.jira.issue.index.DefaultIssueDocumentFactory$Builder.addAll(DefaultIssueDocumentFactory.java:83)
	at com.atlassian.jira.issue.index.DefaultIssueDocumentFactory.apply(DefaultIssueDocumentFactory.java:51)
	at com.atlassian.jira.issue.index.DefaultIssueDocumentFactory.apply(DefaultIssueDocumentFactory.java:32)
	at com.atlassian.jira.issue.index.DefaultIssueIndexer$DefaultDocumentCreationStrategy.get(DefaultIssueIndexer.java:618)
	at com.atlassian.jira.issue.index.DefaultIssueIndexer$IndexIssuesOperation.perform(DefaultIssueIndexer.java:450)
	at com.atlassian.jira.issue.index.DefaultIssueIndexer$IndexIssuesOperation.perform(DefaultIssueIndexer.java:438)
	at com.atlassian.jira.issue.index.DefaultIssueIndexer.lambda$null$2(DefaultIssueIndexer.java:365)
	at com.atlassian.jira.index.SimpleIndexingStrategy.apply(SimpleIndexingStrategy.java:7)
	at com.atlassian.jira.index.SimpleIndexingStrategy.apply(SimpleIndexingStrategy.java:5)
	at com.atlassian.jira.index.MultiThreadedIndexingStrategy$1.call(MultiThreadedIndexingStrategy.java:33)
	at com.atlassian.jira.index.MultiThreadedIndexingStrategy$1.call(MultiThreadedIndexingStrategy.java:31)
	at com.atlassian.jira.util.concurrent.BoundedExecutor$2.call(BoundedExecutor.java:68)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Also, when some reports are exported that contain these custom fields, I see the following logs:

2020-02-05 06:13:47,174 g2gcrm-69914:Cron Service ERROR anonymous     [c.o.scriptrunner.customfield.GroovyCustomField] Script field failed on issue: XXXX-2127, field: Custom Field21
java.lang.IllegalStateException: Incorrect usage of JIRA/lucene search API. You can only create/use: ManagedIndexSearcher inside a context (request or Jira-Thread-Local). Check: JiraThreadLocalUtils for details.
	at com.atlassian.jira.index.ManagedIndexSearcherFactory.createFrom(ManagedIndexSearcherFactory.java:15)
	at com.atlassian.jira.issue.index.ThreadLocalSearcherCache$Cache.retrieveEntitySearcher(ThreadLocalSearcherCache.java:116)
	at com.atlassian.jira.issue.index.ThreadLocalSearcherCache.getSearcher(ThreadLocalSearcherCache.java:39)
	at com.atlassian.jira.issue.index.DefaultIndexManager.getEntitySearcher(DefaultIndexManager.java:888)
	at com.atlassian.jira.issue.index.DefaultIndexManager.getIssueSearcher(DefaultIndexManager.java:865)
	at com.atlassian.jira.config.component.SwitchingInvocationHandler.invoke(SwitchingInvocationHandler.java:38)
	at com.sun.proxy.$Proxy12.getIssueSearcher(Unknown Source)
	at com.atlassian.jira.issue.search.SearchProviderFactoryImpl.getSearcher(SearchProviderFactoryImpl.java:17)
	at com.atlassian.jira.issue.search.providers.LuceneSearchProvider.getIssueSearcher(LuceneSearchProvider.java:130)
	at com.atlassian.jira.issue.search.providers.LuceneSearchProvider.searchDocuments(LuceneSearchProvider.java:443)
	at com.atlassian.jira.issue.search.providers.LuceneSearchProvider.searchDocuments(LuceneSearchProvider.java:450)
	at com.atlassian.jira.issue.search.providers.LuceneSearchProvider.runSearch(LuceneSearchProvider.java:439)
	at com.atlassian.jira.issue.search.providers.LuceneSearchProvider.getHits(LuceneSearchProvider.java:218)
	at com.atlassian.jira.issue.search.providers.LuceneSearchProvider.search(LuceneSearchProvider.java:362)
	at com.atlassian.jira.issue.search.providers.LuceneSearchProvider.search(LuceneSearchProvider.java:135)
	at com.atlassian.jira.issue.search.providers.LuceneSearchProvider.search(LuceneSearchProvider.java:140)
	at com.atlassian.jira.bc.issue.search.DefaultSearchService.search(DefaultSearchService.java:118)
	at com.atlassian.jira.bc.issue.search.SearchService$search.call(Unknown Source)
	at Script22.run(Script22.groovy:83)

Could you guys and girls give me another idea what could I do to get rid of this error and have a successful reindex in a shorter time?

If you need more info, just let me know.
Thanks!

Gabriel

2

Hi @GabrielUdvar,

you run into a typical problem when using Scripted Fields that access the search index. This reindex will work without any problems if it is a background index. In this case, the search index is available and searching for issues works as expected.

In the case that you have a locking index the search index is not available. Hence, you receive a SearchUnavailableException. Additionally, the reindex will not continue until running into some timeout such as 30 seconds. This will happen for every issue and as a result you end up with a slow reindex.

How do you fix the existing problem with SearchUnavailableException:

  • The dumb but quick way would be to disable ScriptRunner while performing the locking index - I’ve seen locking indexes speed up significantly (e.g., 2 weeks with ScriptRunner, 15 minutes with disabled ScriptRunner).
  • The smarter way is to fix the scripted field: Get an instance of the IssueIndexManager IssueIndexManager (Atlassian JIRA - Server 8.5.2 API) and check if the index is available with the isIndexAvailable method. If it is not - then do not perform a search and just return some static value, e.g. null, inside the Scripted Field.

How should you perform a locking reindex with Scripted Fields that perform issue searching - to the best of my knowledge:

  • Run a locking index
  • Run an additional background index after the locking index completes.

The second index ensure that Scripted Fields are indexed correctly as they could not be updated in the locking index.

3

Hi @dennis.fischer,
Many thanks for taking the time to answer my questions in such detail.
I’ve ran some tests in the past days and got to your conclusion: disable scriptrunner and do a full reindex.
Luckily all my scripted fields are located in two certain projects, which makes it easy just to do a project reindex afterwards.
I will most certainly implement your suggestion with the isIndexAvailable method, which looks like an elegant and smart way to do it.

Have a great day!

Cheers,
Gabriel

4

Hi @GabrielUdvar,

please run a background index afterwards - otherwise Scripted Fields are not in the search index.

5

Hi @dennis.fischer,
I was able to filter issues based on the custom fields values after the Project reindex, which I take as an indication that the fields are in the index.
Am I missing something? Is there something extra being made in the background reindex?

Thanks!
Gabriel

6

Just a note - these were the key lines of code I have successfully implemented:

//IMPORT THE CLASS:
import com.atlassian.jira.util.index.IndexLifecycleManager

//CHECK FOR INDEX BUILD:
if (ComponentAccessor.getComponent(IndexLifecycleManager).isIndexAvailable().toString()==“true”) {
// code here if available
}