Weekly GDPR API status development update - February 20th

Apps are required to migrate to Atlassian AccountID and remove legacy user references (username and user key) by 29 March 2019. For more background on why - please see our first post .

Understanding migration status

We are tracking use of the apiMigrations flag in the Connect App descriptor as a way to understand developer readiness to deprecate username and user key from our REST APIs. This flag serves as both a mechanism to test the new API behaviors (when set to gdpr:true), and a communication tool to signal when/where you are blocked (when set to gdpr:false).

Today, we have 38 apps that have signaled that they are ready for the deprecation, 2 that have signaled they are blocked, and ~700 that have not indicated their status. In order for us to understand where you are with migration to accountID and use that information to effectively communicate with our product teams we need you to signal your status using the “opt In mechanism”.

Atlassian Marketplace Requirements

We’ve fielded a number of questions from the community regarding the new Atlassian Marketplace requirements. As a reminder, these requirements are to signal data storage practices, provide a privacy policy, and provide either a customer terms agreement or end user license agreement. These are required for all apps regardless of deployment (cloud/server/data center).

For cloud apps that indicate that they are storing personal data, there is an additional requirement to implement the personal data reporting API. We’ve updated the messaging in Atlassian Marketplace to make it clearer that this is a requirement for apps with cloud versions only.

Recently closed issues

  • Jira and Confluence webhook bodies now contain accountID and account type.
  • Jira and Confluence have new migration APIs that are unaffected by the 29 March 2019 deprecation. (See here for Jira: /rest/api/3/migration) We will provide a link to the Confluence documentation soon.
  • Testing Profile Visibility Controls - If you are ready to start testing profile visibility controls, please raise a ticket in our Developer Support Help Desk here . We are not quite ready but will provide you with more details on when testing will become available and what you should expect to be able to test.

Items in progress

  • Conversion of stored CQL containing legacy user references to accountID - The Confluence team is aware that this is an issue, however, have de-prioritized this work due to the amount of remaining work to prepare for profile visibility controls. We recommend using the new migration API provided by Confluence to transform CQL containing old user references to accountID.

  • Email address hidden from public by default - Email address will be hidden from public by default which means that the email address will no longer be provided to you through our product APIs for the majority of users. Users will need to manually adjust their profile visibility control settings in order to provide access to email. We recognize that for some of you this restricts core functionality. We intend to enable access to email for approved use cases only. We are working on guidelines and a Service Desk system to allow you to apply for access to an API that will provide access to email address. The API is being worked on now and should be available 15 March 2019.

  • Regarding the issue with the Jira PD Cleaner recognizing issue entity property aliases, the Jira team is currently working on this issue and expects the work to be complete on 28 February 2019.

  • Regarding the issue with ACJIRA-1625 , the Jira team is working on this and expects to ship next week.

  • Jira does not have an API to edit workflows (ACJIRA-1718) - the Jira team is working on this and expects the API to ship next week.

  • JQL when transformed using the PD cleaner becomes un-readable - We are working on a new UI component that will display the user’s current name in JQL rather than the accountID. We will provide more information on when this expects to ship next week.

Thank you

We realize this is a large amount of work to complete in a short timeframe. We appreciate you working with us through this process to deliver a more trusted experience for our customers.

If we’ve missed anything preventing you from completing the migration, please comment below.

1 Like

Hey Alex,

Thanks for the update. We chatted about this previously, but just thought I’d comment here too:
We wont be turning the apiMigrations flag on before the date, since this will break any un-migrated smart-values users might be using in our Automation for Jira (e.g. {{assignee}})

We are prompting users over the next few weeks to upgrade as many of them as possible to accountIds, but we need to give them as much time as possible before the switch-over before we turn off usernames/keys.

Cheers,
Andreas

Are you only looking at descriptors in MPAC? This isn’t taking into account what # of people have a dev descriptor in place for testing purposes right? Running a query against all Developer instances to see how many of their apps have this descriptor in place may give you a better gauge on where the community is.

1 Like

Repeating my reply from Weekly GDPR API status development update - February 13th as the previous update threads seem to get closed:

Hi @akassab,
thanks for the clarification.

I suppose you require apps to do an explicit request for this data so you can better record what app is accessing it. Would you consider making these two explicitly requested context parameters (not sent by default, but only if asked for in the URL template string in the app descriptor)? This way you could still record it.

With the current state we’re forced to either block displaying the UI until the request is loaded or to change locale (and this means UI i18n) after our UIs got the response. This would result in either a slow UI or in a UI that briefly after loading suddenly changes its language.
Using the browser’s information from the Accept-Language header and the JS Date object would be possible but would result in app iframe UIs behaving different from the Confluence UI - after all not every user has their language settings set to ‘auto-detect’.

This is also a performance issue for static macros.

I will repeat my question from the previous thread as well.

Hi,
We have another question. We are wondering whether it is possible to retrieve our own Addon’s accountId from a reliable source?

As you mentioned, the accountId is the same across all Jira Cloud instances. Currently in our first drafts of GDPR compliance we make use of the /myself endpoint authorized by the JWT method which seems to be properly returning the accountId of our Addon user. The problem is, however, that we can only do this when the first instance sends us a request so that we can ask this particular instance for those details.

This of course does not feel right, is error-prone, unreliable and random. Furthermore, it forces us to work around proper multitenant architecture internally. We would love to see a single resource/microservice or generally something Jira Cloud instance-independent that we could ask about our Addon’s accountId in runtime. Is it possible? What would you recommend?

Hi @akassab,

I have a GDPR related question – not specific to my app, but hoping you can still point me in the right direction.

Has Atlassian released any information at this point about how Jira data imports from CSV will be affected by the migration? Currently, user fields (e.g. assignee) are imported by specifying the username. I’m assuming this will stop working after the March 29 deadline, and the importer will expect the accountId to be provided instead?

If this assumption is correct, what is the recommended method for a Jira administrator to obtain accountIds for each of their users? As far as I know, accountId is not displayed anywhere in the Jira or Site Administration UI. There is an “Export Users” feature in Site Administration, but the exported file contains a “username” column and not accountId. It’s of course possible to retrieve the accountId via the Jira API, but this will be way over the head of most of the Jira admins I work with. I’m hoping there is a more accessible solution in the works so I can advise my customers of the change prior to March 29.

Thanks in advance for your help!

Cheers,
Mark

I am currently investigating some use cases for issue import.

In the situation where you want the data you are importing to match existing users, how would the columns containing user fields currently be populated?

My thoughts (just thoughts at this stage) are that there may be use cases where that data comes from some other system, and you generate this data on a regular basis, in which case it may be possible to specify an account ID for a user. I think it would be useful to support an exact match on account ID.

For cases where this data has some kind of name for an existing user, what use cases might need this kind of functionality, and how might it work? My thought is that supporting some kind of name match (against any personal data that this person allows to be visible) might be useful. Suppose the person makes their email address visible, and their email address contains “bkelley”. In that case it might be useful when importing data with “bkelley” for a user for it to match the user that has an email address containing “bkelley”.

If Jira allowed a partial name match against visible user fields, what should it do if the name matched multiple users, or matched no users?

Hi @bkelley,

Thanks for the quick response.

I have one particular use case that deeply impacts approx. 25 of my consulting clients, all of which are marketing agencies that use Jira to manage their service delivery. They plan their marketing activities on a quarterly or monthly basis, and then use the “External System Import” feature to bulk import hundreds of Epics and Stories that represent this work. The CSV files are generated from a Google Sheet that essentially acts as a template, with data validation and other controls. We maintain an import configuration file that they pass to the importer, which takes care of field mappings and all of the other import details. This method is probably 10x more efficient for their use case than creating the same data in Jira via the UI, and if this capability is lost, they’ll be in for some pretty significant operational pain.

For this use case, they map four user fields to Jira: Assignee, plus three other custom user fields. All of the Jira users already exist, and the template maintains a mapping of people in their organization to their corresponding Jira usernames. Data validation in the import template ensures that only valid usernames are present in CSV file when it gets imported. Therefore, the matching is always exact. The equivalent setup post-GDPR would be for us to simply match based on accountId rather than username – the only tricky part would be actually getting the accountIds from Jira so they can maintain the mappings, as this is not exposed anywhere in the Jira UI.

The ability to match on other values such as Full Name or Email Address would be a nice-to-have that would marginally improve their import experience, but obviously the implementation would be a lot more complex in order to handle the cases you outlined.

At the end of the day, I really just want to:

  1. Ensure that CSV import isn’t going to go unnoticed and stop working when the GDPR switch gets flipped a month of now.
  2. Have a solid understanding of how it will work post-GDPR so I can help my clients update their import templates and re-train them appropriately to minimize impact to the business.

Happy to jump on a call and walk you through how we use it in more detail if you would find that helpful, just let me know.

Cheers,
Mark

2 Likes

Hi @akassab,
I thought the addition of the account type to webhooks would also apply to post-function /triggered calls (since they’re technically webhooks), but apparently that is not the case. When will we get the account type in post-functions?

Furthermore, despite the gdpr:true flag, the /triggered REST call still contains the user_id and user_key query parameters.

Cheers,
David

Hi. Thanks for the detailed description of the use case. That’s helpful. It seems very likely that we would support an exact match on account ID at a minimum.

There are REST endpoints that can return account IDs. The Find Users endpoint (documentation at https://developer.atlassian.com/cloud/jira/platform/rest/v3/#api-api-3-user-search-get) can match on email address. The user object returned includes the account ID.

In terms of migrating user names to account IDs, there is an endpoint that lets you look up multiple user names. (Documentation at https://developer.atlassian.com/cloud/jira/platform/rest/v3/#api-api-3-user-bulk-migration-get)

The work for the account type is still in progress. You can track progress by watching this ticket: [ACJIRA-1674] - Ecosystem Jira

Thanks @bkelley. Glad to hear exact match will be supported at minimum. Do you know if there is a ticket I can track for implementation?

There are REST endpoints that can return account IDs.

I was really hoping for an option for getting these from the UI, as having to make an API call to determine the accountId when they onboard a new employee will frankly be a significant amount of friction for many of the Jira admins I work with. If an accountId column were to be included in the “Export Users” feature in Site Administration, that would be sufficient. Username is currently included, so I’m hoping that will just be swapped out for accountId when the time comes.

Cheers,
Mark

can you provide a link to that REST API? I just checked the documentation site and couldn’t find it

In the migration guide are the following points stated:

  • Operations that currently accept a username and/or userKey will also accept an Atlassian account ID. ( accountId ). After the deprecation period ends, only an accountId will be accepted.

  • Operations that currently use a username and/or userKey as a search clause (for example, JQL queries) will also use an Atlassian account ID ( accountId ). After the deprecation period ends, only an accountId may be used. To help you update your JQL queries, we’ve introduced a new operation: POST /rest/api/3/jql/pdcleaner which takes one or more JQL queries with user identifiers and converts them to equivalent JQL queries with account IDs.

At the end there is mentioned on how to test the new APIs:

  • If you want to test your changes, you can force the REST APIs to only use GDPR-compliant functionality by including the x-atlassian-force-account-id: true header in any REST API call.

Doing a Postman call to the search issue API with this header set, still returns personal data.
(like emailAddress and timeZone)
Same goes for a jql call like: assignee=daniel

As mentioned above, these shoudln’t work anymore or will at least return GDPR-ready data.
The header does not work the way it is meant to work.
Is there any other way to enforce (and test) the APIs so that we can migrate in time?

Cheers, Daniel

@jbevan - I’m expecting docs this week and will update with a link once one becomes available. I know the endpoint name (if that helps). It’s /wiki/rest/api/user/bulk/migration?username= .

1 Like

@david2 - I will follow up with the Jira team on the post-function in account type issue and include in the next round of weekly updates.

In terms of the opt in mechanisms, the gdpr:true flag only modifies API behaviors for Connect APIs. In addition to the gdpr:true flag you will also need to add the x-atlassian-force-account-id: true header to all of your Jira REST Calls.

@andreas - Setting the flag to gdpr:false should not have any impact on your apps because it will not modify the API behaviors during an active migration period. I’ve updated our documentation to better describe the behaviors of these active migration flags.

@jens - We may consider this, however it is not currently in our near term plans. I will provide more information in this week’s update.

@akassab Thanks for the update. BTW, I have created [ACJIRA-1722] - Ecosystem Jira for adding the account type to the /triggered call.

As for the gdpr:true flag, I understand it only impacts Connect, but the post-function /triggered call is part of the Connect API (just in the other direction), so it should impact it as well. I believe it’s an omission.
And of course x-atlassian-force-account-id: true header is required to force the Connect REST API to return only accountIds - for example in issue change history.

1 Like