Migrating apps to Jira 8: be careful when using JQL from background threads

migration
index
jira-server
jira-80

#1

Hi everyone!

I would like to share ideas about a problem we had faced when we were making our app, Structure, compatible with Jira 8. I think this can be helpful to other app developers.

As you probably know, Jira 8 comes with the new Lucene engine. The binary Lucene files are not compatible between Lucene used in Jira 7 and Jira 8. As a result, Jira 8 will automatically remove the old index files and start full reindexing on Jira start. (See this article: Preparing for Jira 8.0)

That means Lucene index is going to be absent/empty for some time after Jira 8 (and all apps) have started. Which, in turn, means that any JQL will return an empty result.

This doesn’t affect most of Jira and app functionality where code is executed in response to a user’s request, because Jira is unavailable for interaction during complete reindexing. But, in our product there are also background jobs (synchronizers) that start with the application and that potentially can make changes in Jira database based on the results of a user-configured JQL query.

Executing such background tasks with an invalid index is extremely undesirable and may be harmful to Jira state, stored in the database.

To resolve this problem, we needed a mechanism for detecting absent or corrupt index state. We would then postpone executing the background jobs until the index is back to normal.

Thanks to the Atlassian team, we learned about method IndexLifecycleManager.isIndexConsistent() in the Jira API. Under the hood, this method counts issues in the database and compares it with count of issues in the index. So we can check this method and delay background work while index is still not valid.

Calling this method is not cheap, but it’s not super-expensive either. We used a guideline to cache a negative result (index inconsistent) for 5 seconds and a positive result (index consistent) for a minute or more.

If your app has background tasks that rely on JQL and make any kind of mutations based on the JQL result, we strongly advise you to review your code and implement a similar solution.

Here is example of a small component that we’ve implemented. Feel free to grab this code if you need it.

Click to expand a component code

public class IndexConsistencyChecker {
  private static final long SUCCESS_EXPIRATION = TimeUnit.MINUTES.toNanos(2);
  private static final long FAILED_EXPIRATION = TimeUnit.SECONDS.toNanos(5);

  // implementation of IndexLifecycleManager
  private final IssueIndexManager indexManager; 
  // last result of consistency check
  private volatile ConsistencyState currentState; 

  public IndexConsistencyChecker(IssueIndexManager indexManager) {
    this.indexManager = indexManager;
  }

  public boolean isConsistent() {
    if (!indexManager.isIndexAvailable()) {
      return false;
    }
    long now = System.nanoTime();
    ConsistencyState state = currentState;
    if (state != null && !state.isExpired(now)) {
      return state.consistency;
    }
    synchronized (this) {
      ConsistencyState state1 = currentState;
      if (state1 == state) {
        boolean isConsistent = indexManager.isIndexConsistent();
        currentState = state1 = new ConsistencyState(isConsistent, now);
      }
      return state1.consistency;
    }
  }

  private static class ConsistencyState {
    final boolean consistency;
    final long timestamp;

    ConsistencyState(boolean consistency, long timestamp) {
      this.consistency = consistency;
      this.timestamp = timestamp;
    }

    boolean isExpired(long now) {
      long expiration = consistency ? SUCCESS_EXPIRATION : FAILED_EXPIRATION;
      return now - timestamp > expiration;
    }
  }
}

Thanks again to Jira and Premier Support teams, Andriy Yakovlev, Łukasz Wlodarczyk, Syed Masood, Adam Jakubowski and others for helping with this!

Best regards,
Egor Lyashenko
ALM Works


#2

Thanks for sharing !


#3

Thank you @egor.lyashenko!


#4

Thank you, @egor.lyashenko!


#5

Very useful I may well need this in my app at some point


#6

@egor.lyashenko thank you for this post and for outlining the pattern to make plugins more aware of the index availability. We’re happy that we can move forward with the community to make better products.

The pattern outlined in this blog was developed to handle the specific case of missing index in the 7.x to 8.0 upgrade process. Thanks to your hard work, we’ll be able to support our customers through this upgrade better. Unfortunately, the Atlassian Jira Team has focused on the upgrade scenario, failing to catch the problem this pattern can cause in Jira 7.x. We are sorry this has hit our customers and only further analysis revealed the problems described in JRASERVER-68900 and JRASERVER-68901.

We are planning to fix this vulnerability in the future 7.6.x and 7.13.x versions. But this means that there is little we can do from here to allow your apps to be built for the entire range of 7.x versions to leverage this mechanism. A possible workaround is to use this capability as optional, and gracefully degrade for versions where it is not available.

@egor.lyashenko, can I kindly ask you to edit your blog post to caution the readers that this recommendation applies to Jira 8+ only? This will allow us to set proper expectations for the readers of this blog. And again, sorry about the caused problems.


#7

Important: please use this solution for Jira 8+ only. Some problems have been discovered with the usage of this approach in Jira 7.x. Thanks to Lukasz Wlodarczyk for clarification. Please take a look at his reply for details.


#8

Lukasz, thanks for your detailed reply. Unfortunately I can no longer edit this post but I’ve added clarifying comment.


#9

Nicely done! Thank you.


#10

Hi,
could this explain why in Jira8 I am having

java.lang.IllegalStateException: Incorrect usage of JIRA/lucene search API. You can only create/use: 
ManagedIndexSearcher inside a context (request or Jira-Thread-Local). Check: JiraThreadLocalUtils for details.
at com.atlassian.jira.index.ManagedIndexSearcherFactory.createFrom(ManagedIndexSearcherFactory.java:15)

when my thread tries to execute search ?


#11

After upgrading to Jira 8, plugin “JQL functions collection” is not compatible with Jira8.

Our devs don’t use JQL functions, but other plugins installed might do.

Should I disabled plugin or leave it enabled?