Custom field implementation with new API

As an addition to the posts listing on our custom field optimizations (Changes to Custom Fields’ implementation requiring action from you and Experimental API for custom field values waiting for your feedback) we want to elaborate on the custom fields’ implementation. In this brief article, we walk you through the steps necessary to successfully implement a custom field and index/cache it.

CustomFieldType

In Jira 8.10 we introduced a new approach to custom field values’ indexing. The general idea is that a custom field indexer should only be called when there is a value assigned to the custom field and the field is visible. The custom field is considered to be visible when it is not hidden in the Field Configuration menu.

Note: Screens are not used to calculate fields’ visibility - the field may not be assigned to any screen, but is still considered visible.

This optimization decreases reindexing times. The new experimental API in CustomFieldType gives you the possibility to define a NonnullCustomFieldProvider.

@ExperimentalApi
default NonnullCustomFieldProvider getNonnullCustomFieldProvider() {
    return null;
}

When the new API is not implemented the custom fields’ behaviour is the same as in the previous versions of Jira. Namely, the custom field indexer’s default indexing method is called, which could lead to long indexing times for particular custom fields. That’s why it’s recommended to apply the new API for better indexing results.

NonnullCustomFieldProvider

NonnullCustomFieldProvider provides the list of custom field IDs that have values. It can also include their pre-fetched values in the returned map. It defines two methods that should be implemented:

  • Map<String, CustomFieldPrefetchedData> getCustomFieldInfo(Issue issue)
  • Object getIdentity()

The getCustomFieldInfo method returns a map, where the keys are a collection of IDs of custom fields that have values defined for a given issue. Only the custom fields handled by the specific provider need to be considered.

CustomFieldPrefetchedData contains arbitrary data that will be later made available to the indexer of this custom field.

The getIdentity method returns the identity of this provider. The identity will be used for equality check to ensure that any dynamic proxy of a provider will be equal to that provider.

An example implementation of NonnullCustomFieldProvider can look like this:

private static class LabelCustomFieldProvider implements NonnullCustomFieldProvider {
    private final LabelManager labelManager;

    private LabelCustomFieldProvider(LabelManager labelManager) {
        this.labelManager = labelManager;
    }

    @Override
    public Map<String, CustomFieldPrefetchedData> getCustomFieldInfo(final Issue issue) {
        final Map<Long, List<Label>> allCustomLabels = labelManager.getCustomFieldLabels(issue.getId()).stream()
                .collect(Collectors.groupingBy(Label::getCustomFieldId));

        return allCustomLabels.entrySet().stream()
                .collect(Collectors.toMap(e -> toFieldId(e.getKey()), e -> {
                    final TreeSet<Label> sortedLabels = new TreeSet<>(LabelComparator.INSTANCE);
                    sortedLabels.addAll(e.getValue());
                    return new CustomFieldPrefetchedData(sortedLabels);
                }));
    }

    private static String toFieldId(final Long customFieldId) {
        return CustomFieldUtils.CUSTOM_FIELD_PREFIX + customFieldId;
    }

    @Override
    public Object getIdentity() {
        return labelManager.getIdentity();
    }
}

FieldIndexer

We’ve also introduced the new experimental API in the FieldIndexer interface.

default Boolean skipsIndexingNull() {
    return null;
}

It gives you the possibility to explicitly declare whether the indexing method of your custom field indexer should be called or not for a custom field with a null value. This API was introduced to keep backward compatibility with the custom field indexers that are writing to Lucene document even when the custom fields had no value. However, we generally don’t recommend writing to Lucene document when custom fields have no value.

There are two ways to access CustomFieldPrefetchedData in custom field indexer that extends AbstractCustomFieldIndexer.

Overriding new addIndex method:

public void addIndex(Document, Issue, CustomFieldPrefetchedData)

Overriding new addDocumentFieldsSearchable and addDocumentFieldsNotSearchable methods:

protected void addDocumentFieldsSearchable(Document, Issue, CustomFieldPrefetchedData)

protected void addDocumentFieldsNotSearchable(Document, Issue, CustomFieldPrefetchedData)

These new methods are called by Jira custom field indexing mechanism starting with version 8.10. To maintain backward compatibility, default implementation of each of these new methods calls their old representatives. Thus, in some cases fetching the custom field’s value twice. First time to verify if the value is not null and indexer should be called and second time to add the value to the Lucene document.

To avoid duplicated custom field’s value fetching, either new addIndex method or addDocumentFieldsSearchable and addDocumentFieldsNotSearchable methods should be implemented in your custom field indexer.

For custom field indexer extending FieldIndexer the new addIndex method should be implemented.

Accessing data from CustomFieldPrefetchedData in the addIndex() method could look like this:

final Set<Label> labels = (Set) customFieldPrefetchedData.getData().orElseGet(() -> customField.getValue(issue));
// Do something with the value

Summary

In version 8.10 changes have been introduced to Jira custom field indexing mechanism. Backward compatibility is supported. However, it is strongly advised that you implement the new API in your custom fields to further decrease indexing times. The custom field value is fetched once and then passed on to custom field indexer when it is not null or custom field indexer does not skip indexing nulls and custom field is visible.

FAQ

  • Do I need to implement new API when my indexer does not do logic for null values? Yes. For custom fields with null values Jira will skip checking context and visibility of that custom field, thus improving reindexing times.

  • How should I implement methods addDocumentFieldsSearchable() and addDocumentFieldsNotSearchable()? addDocumentFieldsSearchable() should be the only method ued to add data to the Lucene document. addDocumentFieldsNotSearchable() exists only for backward compatibility purposes and should not be used as it might be removed in the future.

  • Will my custom field values be calculated twice and decrease performance? Only if the getNonnullCustomFieldProvider() method is implemented and new addIndex() method is not, thus fetching the value before adding it to the Lucene document.

  • How do I maintain single plugin codebase for Jira versions before and after the changes? Plugins that implement the old API will work in old and new Jira versions without issues. However, older Jira versions will not be able to instantiate classes of plugins that implement both old and new API. We are investigating for a solution.

1 Like

Have you released the new API yet? We don’t seem to find a new EAP for it.

Hi Bartozs,

Thank you for the updates on the new API. Keeping the prefetched value seem to make a lot of sense.

Like @yvesriel said, I don’t believe you released a new EAP for those changes yet, is that correct?

You are not demonstrating how we would access the prefetched value. You will need to still support the current indexing function, so I am curious how you will achieve this.

The “Summary” of your post is a little bit confusing as it states that this is how we implement a new Custom Field, but I suspect you might mean how to implement a new “Nonnull Custom Field Provider”. Will that new implementation be mandatory for all Custom Fields?

The new EAP will be released soon.

I have updated the post to reflect your suggestions, please take a look.

It’s not mandatory at the moment - but at some point we will start tracking API implementation through health check system (which may result in a warning displayed to the admin).

Hi @pbruski,

… but at some point we will start tracking API implementation through health check system (which may result in a warning displayed to the admin).

This is where I have some issues. For us implementing the new API would break compatibility but moreover we don’t want to implement the API because it would end up in more indexing time since our use case is different than the vast majority of other custom fields. Now, by taking the right decision for our customers we would be penalised by having a warning displayed beside our field :frowning:

If you do that, I would like you to consider the following points:

  1. Since this is a braking API change, only add the system check on a new major version (e.g. 9.x). At least then we could have a version for 9.x+ and one for 8.x and below.
  2. Include a mechanism where add-ons are given an approval if they prove you that not implementing the API is faster and make sure that they are not marked in the health check.

I’m really happy that you are making efforts to decrease indexing time but there are some use cases that you are not covering and vendors should not be penalised for it.

Cheers,

The warning would only be displayed if the plugin contributes to the total indexing time significantly AND does not implement the API. For all health checks, we already carefully monitor the amount of alerted instances and adapt thresholds accordingly.
If a plugin is hurting customers, we need to react, vendor’s preference to maintain a single binary deployable has to take a back seat.

Since this is a braking API change

This is not a breaking API change as far as we define ‘API breaking’ . I understand that it’s harder to run the same code across different Jira versions, but it does not constitute API breakage. Our guarantee is only on forward compatibility.

Thanks! That makes sense and indeed we do not want to hurt customers. I do understand that from Jira’s standpoint it’s not an API breaking change but it does have major impacts on vendors. So since it’s an ecosystem, it would be nice to lessen the impact for us when it’s possible.

We used to have different versions at some point. For strange reasons, customers don’t check that new versions are compatible when they want to perform an upgrade and we got a lot of support calls for that. It all stopped when me made the effort to have one artifact that supports multiple majors versions :man_shrugging:

1 Like

New API keeping backward compatible will be ready soon: Experimental API for custom field values waiting for your feedback.

The new API guidelines are documented here: Implementing new Custom Field APIs in a backward-compatible manner