Experimental API for custom field values waiting for your feedback

[DamianKedzierski]

As mentioned in the previous blogpost, we marked the new API as Experimental, because we want to make sure it will be useful for you before we will make it public. Please feel free to share your suggestions for changes with us.

Here is what you would need to do to benefit from optimizations:

1. Return fields having values

First, you need to allow Jira to recognize the value of your field and establish which fields have actual values assigned to them. To do that, implement the following method: CustomFieldType#getNonnullCustomFieldProvider():

public class LabelsCFType extends AbstractCustomFieldType<Set<Label>, Label> {
  
  private final LabelManager labelManager;
  
  @Override
  public NonnullCustomFieldProvider getNonnullCustomFieldProvider() {
      return new LabelCustomFieldProvider(labelManager);
  }
}

And here is how your implementation of NonnullCustomFieldProvider may look like:

public class LabelCustomFieldProvider implements NonnullCustomFieldProvider {
    // persister that you use for obtaining custom fields values
    private final LabelManager labelManager;
    
    @override
    public Collection<String> get(long issueId) {
        return labelManager.getCustomFieldLabels(issueId).stream()
                    .map(x -> CustomFieldUtils.CUSTOM_FIELD_PREFIX + x.getCustomFieldId())
                    .collect(Collectors.toSet());
    }
    
    @override
    public Object getIdentity() {
        return labelManager.getIdentity();
    }
}

To make it work correctly with Spring/OSGI proxies, your value persister will need to implement IdentifiableComponent interface:

public class LabelManager implements IdentifiableComponent {
    public Set<Label> getCustomFieldLabels(long issueId) {
        // your logic to retrieve values
    }
}

2. Mark your indexer to skip indexing null values

Next, you need to set your indexer to skip null values. To do that it will need to override the following method to return true:

@Override
public Boolean skipsIndexingNull() {
  return true;
}

3. Perform full re-index

[a.belostotskiy]

Hi, The mentioned blogpost is linked to internal Atlassian instance, and I can’t access it.

[DamianKedzierski]

@a.belostotskiy I updated the link - thanks!

[linklefebvre]

Hi @DamianKedzierski,

Thanks for the documentation!

I noticed in your code snippet you are implementing NonnullCustomFieldValueProvider instead of NonnullCustomFieldProvider and I believe might be a mistake. Class name might have changed during development?

To be sure, is the implementation of CustomFieldType.getNonnullCustomFieldProvider mandatory in order for the FieldIndexer.skipsIndexingNull (returning true) to do anything? Or does it use some default null condition otherwise?

EDIT: I am currently looking at our existing Indexer’s code, and we already have code that returns when the Custom Field value is empty (so literally nothing is added to the document). Would that new implementation be redundant for us? I am not sure if there’s a lot of additional overhead that would be avoided by going through the API?

[DamianKedzierski]

@linklefebvre I fixed the typo - thanks!
In the current design, you will need to extend both methods: FieldIndexer.skipsIndexingNull and CustomFieldType.getNonnullCustomFieldProvider.

Jira will need a way to get list of fields having non-null values for the issue. On the other side, Jira built-in field could be used by apps with a custom indexer, so we exposed the other method to allow indexers to control it. Do you have any suggestions on how could we improve it?

From our tests, even calling indexers is expensive, as context and visibility will be checked inside of each field indexer (or most of them). When filtering out indexers without any value, we limit the number such checks.

[linklefebvre]

@DamianKedzierski, thanks for the answer.

So the overhead we avoid is verifying the Context and Visibility.

For testing purposes, I’ve created a NonnullCustomFieldProvider that returns an empty array, so I expected the Indexer to never run. What I’ve noticed is that it still runs if our Custom Field has CustomFieldValue records in the database. Is that an intended behavior?

I believe it might be because of the DefaultNonnullCustomFieldProvider which returns the list of all CustomFieldIds which has value, even the vendor ones. (For clarifications this is not an issue in our case, but could it be one for some vendors?)

[DamianKedzierski]

@linklefebvre Thanks for sharing your feedback!

This behaviour was intended, as we assumed it will not be problematic for any apps.
If there will be any vendors suffering from that we will add a capability to limit it.

[david2]

Hi @DamianKedzierski,
we have a single artifact (app version) that supports all versions of Jira back to Jira 7.0.11. How can we maintain backward compatibility if we implement these changes?

Usually, we use one of these approaches to support breaking changes:

• use introspection to call new methods that didn’t exist in Jira 7.0.11
• in some cases, when there are a lot of changes, we write adaptor classes and put them in a separate artifact that we compile against the new Jira version and conditionally call from our main artifact

We can somewhat easily add the NonnullCustomFieldProvider implementation in a separate artifact built against Jira 8.10. However, I don’t see how we can override a new method in our implementation of AbstractCustomFieldType. We can’t do two implementations in separate artifacts, since there can only be one implementation of the custom field class, which is declared in the atlassian-plugin.xml file. And we can’t override the method without the @override annotation (thus simply adding a new method for versions of the base class that didn’t declare the method), because the return type doesn’t exist either in older versions of Jira.
Any idea?
David

[BartoszKwiatek]

Hey, please take a look at this article.

[david2]

Hi @BartoszKwiatek,
thanks for the pointer, I didn’t notice the last bullet point. I guess we could try to compile just the custom field implementation classes against Jira 8.10 (we can’t compile the whole project against Jira 8.10 as there were many breaking changes in Jira 8 that would make the resulting code incompatible with Jira 7). I supposed the import won’t break until we actually call the method, and since Jira < 8.10 will never call it, we should be fine, right?

[DamianKedzierski]

We will release a new version of this API, which allows your apps to be compiled and executed with older Jira versions. We kindly recommend you to not use the current version and wait till 8.12.0.

[GianlucaTedeschi]

hi, it’s Gianluca!

we had developed a plugin with own “custom fields”, it’s worked before Jira 8.10, after not and the error in Atlassian-jira.log is:
Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name XXXXXXXX: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.NoSuchBeanDefinitionException: No qualifying bean of type ‘com.atlassian.jira.issue.customfields.persistence.CustomFieldValuePersister’ available: expected at least 1 bean which qualifies as autowire candidate. Dependency annotations: {@com.atlassian.plugin.spring.scanner.annotation.imports.ComponentImport(value=)}

could be related at this new experimental API?

NO compile error using jira-core 8.10.0 maven dependency…

It’s possible to have an empty (working) custom field plugin project sample?

thanks!

[rolyanos]

Hi @DamianKedzierski ,

I have faced with problem during indexation of own custom field.

I have added own NonNullCustomFieldProvider

public class TestCFProvider implements NonNullCustomFieldProvider {

  private final IssueManager issueManager;

  private final CustomFieldValuePersister customFieldValuePersister;

  private final CustomFieldManager customFieldManager;

  public TestCFProvider(IssueManager issueManager, CustomFieldValuePersister customFieldValuePersister, CustomFieldManager customFieldManager) {
    this.issueManager = issueManager;
    this.customFieldValuePersister = customFieldValuePersister;
    this.customFieldManager = customFieldManager;
  }

  @Override
  public Map<Long, Map<String, CustomFieldPrefetchedData>> getCustomFieldInfo(List<Issue> issues) {
    Map<Long, Map<String, CustomFieldPrefetchedData>> data = new HashMap<>();
    for (Issue issue : issues) {
      Map<String, CustomFieldPrefetchedData> dataMap = new HashMap<>();
      List<CustomField> cfs = customFieldManager.getCustomFieldObjects(issue);
      cfs = cfs.stream().filter(cf -> cf.getCustomFieldType().getKey().equals(Constants.MY_CF_TYPE)).collect(Collectors.toList());
      for (CustomField cf : cfs) {
        Collection<Issue> selectedIssues = ((IndexedCF) cf.getCustomFieldType())
                .getLinkedIssuesList(cf, issue);
        dataMap.put(cf.getId(), new CustomFieldPrefetchedData(selectedIssues));
      }
      data.put(issue.getId(), dataMap);
    }
    return data;
  }

  @Override
  public Object getIdentity() {
    return customFieldValuePersister.getIdentity();
  }
}

And here is the indexer

public class FieldIndexer extends AbstractCustomFieldIndexer implements FieldIndexer {
  private final CustomField customField;

  protected FieldIndexer(final FieldVisibilityManager fieldVisibilityManager, CustomField customField) {
    super(fieldVisibilityManager, customField);
    this.customField = customField;
  }

  public Boolean skipsIndexingNull() {
    return true;
  }

  private void addOwnIndex(Document doc, Issue issue) {
    Collection<Issue> selectedIssues = ((IndexedCF) customField.getCustomFieldType())
        .getLinkedIssuesList(customField, issue);

    String idIndexValue;
    String keyIndexValue;
    String keyIndexValueFolded;
    if (selectedIssues.isEmpty()) {
      idIndexValue = NO_VALUE_INDEX_VALUE;
      keyIndexValue = NO_VALUE_INDEX_VALUE;
      keyIndexValueFolded = NO_VALUE_INDEX_VALUE;

      indexField(doc, idIndexValue, keyIndexValue, keyIndexValueFolded, issue);
    } else {
      for (Issue referredIssue : selectedIssues) {
        idIndexValue = referredIssue.getId().toString();
        keyIndexValue = referredIssue.getKey();
        keyIndexValueFolded = getKeyFoldedValue(referredIssue.getKey());

        indexField(doc, idIndexValue, keyIndexValue, keyIndexValueFolded, issue);
      }
    }
  }

  /**
   * 
   * @param doc
   * @param idIndexValue
   * @param keyIndexValue
   * @param keyIndexValueFolded
   */
  private void indexField(Document doc, String idIndexValue, String keyIndexValue, String keyIndexValueFolded,
      Issue issue) {
    if (isFieldVisibleAndInScope(issue)) {
      doc.add(new StringField(getIdFieldId(customField), idIndexValue, Field.Store.YES));
      // store the unchanged key, index the folded one for consistent searching
      doc.add(new StoredField(getKeyFieldId(customField), new BytesRef(keyIndexValue)));
      doc.add(new StringField(getKeyFoldedFieldId(customField), keyIndexValueFolded, Field.Store.NO));

      //for gadgets
      doc.add(new SortedSetDocValuesField(getIdFieldId(customField), new BytesRef(idIndexValue)));
      doc.add(new SortedSetDocValuesField(getKeyFieldId(customField), new BytesRef(keyIndexValue)));
    } else {
      doc.add(new StoredField(getIdFieldId(customField), new BytesRef(idIndexValue)));
      doc.add(new StoredField(getKeyFieldId(customField), new BytesRef(keyIndexValue)));
    }
    SecurityIndexingUtils.indexPermissions(doc, issue, getIdFieldId(customField), idIndexValue);
  }

  public static String getIdFieldId(CustomField customField) {
    return customField.getId();
  }

  private static String getKeyFieldId(CustomField customField) {
    return customField.getId() + "_key";
  }

  private static String getKeyFoldedFieldId(CustomField customField) {
    return customField.getId() + "_key_folded";
  }

  private static String getKeyFoldedValue(String key) {
    return CaseFolding.foldString(key);
  }

  @Override
  public void addDocumentFieldsNotSearchable(Document doc, Issue issue, com.atlassian.jira.issue.customfields.vdi.CustomFieldPrefetchedData data) {
    addOwnIndex(doc, issue);
  }

  @Override
  public void addDocumentFieldsSearchable(Document doc, Issue issue, CustomFieldPrefetchedData data) {
    addOwnIndex(doc, issue);
  }

Here is the problem:

• Not global context (have set one or more issue type(s))
  JQL:

project = crm and cf[10021] is not EMPTY 

returns all out of context issues + correctly filtered in context issues.

• Global context
  same JQL returns correctly filtered issues only those that have not empty value in CF.

Maybe I something miss or made some mistake in indexer?

Custom field store list of issues as comma separated string.