Weekly GDPR API status development update - February 28th

[akassab]

Apps are required to migrate to Atlassian AccountID and remove legacy user references (username and user key). For more background on why - please see our first post .

Update on deprecation date

Previously we’ve communicated that apps are required to complete their migration to Atlassian AccountID and remove legacy user references (username and user key) by March 29, 2019. Due to the amount of remaining open issues raised by you, the Developer Community, we’ve decided to extend the deprecation notice period by 30 days. The new target deprecation date for Jira Cloud, Confluence Cloud, Bitbucket Cloud and Connect REST APIs is April 29, 2019.

Understanding migration status

We are tracking use of the apiMigrations flag in the Connect App descriptor as a way to understand developer readiness to deprecate username and user key from our REST APIs. This flag serves as both a mechanism to test the new API behaviors (when set to gdpr:true) and a communication tool to signal when you are blocked (when set to gdpr:false).

Today, we have 53 apps that have signaled that they are ready for the deprecation, 5 that have signaled they are blocked, and ~650 that have not indicated their status.

In order for us to understand where you are with migration to accountID and use that information to effectively communicate with our product teams we need you to signal your status using the “opt In mechanism”.

Learn more about how to properly use this flag.

Recently closed issues

No new issues closed this week.

Items in progress

• AccountID in JQL statusby clause does not work - Jira is working on this and have a fix in place but not released yet. This may be resolved before our next weekly status update. See ACJIRA-1625. An additional issue was raised specifying componentsLeadByUser() method. The work to fix this bug has not started yet. See ACJIRA-1704.

• Jira user picker REST API missing account type - Same as above (fix in place but not released yet). See ACJIRA-1686.

• Jira webhooks missing accountId and account type - This topic was reopened. I apologize for the miscommunication from last week. The Jira team is currently working on adding both accountID and account type to webhooks. See ACJIRA-1674.

• Jira post function /triggered calls missing accountID and account type - This issue was split from the above. The Jira team has not started on this work yet nor provided estimates. See ACJIRA-1722.

• Jira incoming webhooks missing accountId - Same as above. See ACJIRA-1692.

• Jira does not recognize issue entity property aliases - The Jira team is currently working on this issue and expected it to ship this week. Unfortunately, the work is still in progress. We are getting new estimates for the ship date. See ACJIRA-1723.

• Jira does not have an API to edit workflows - Same as above. See ACJIRA-1718.

• JQL transformed to remove legacy user references and replaced with accountID are un-readable - We’re building a JQL Editor Dialog component that will display the user’s current name in JQL rather than the accountID. This work is in progress and we expect this to ship in 2 weeks. See ACJIRA-1710.

• Email address hidden by default - Email address will no longer be provided to you through our product APIs for the majority of users. We recognize that for some of you this restricts core functionality. We intend to enable access to email for approved use cases only. We are working on guidelines and a Service Desk system to allow you to apply for access to an API that will provide access to email address. The API is being worked on now and should be available on March 15, 2019. See ACJIRA-1726

• User mentions in Wiki Markup not working with expected format - User mentions in Jira require a prefix prior to accountID in order to work correctly. New user references will be saved in this new format, however, historical user references haven’t been migrated yet. We do not have estimates for how/when historical user references will be migrated. See ACJIRA-1715

• Jira PD Cleaner converting user strings that are not exact usernames to ‘unknown’ - This issue was raised in ACJIRA-1712 but now is being tracked in ACJIRA-1725. We do not have estimates for how/when this will be fixed.

• Confluence documentation for new REST endpoint for migrations - Jira and Confluence have new migration APIs that are unaffected by the 29 April 2019 deprecation. (See here for Jira: /rest/api/3/migration ) The endpoint for Confluence is ‘/wiki/rest/api/user/bulk/migration?username=’. Documentation is not currently available. We will provide a link as soon as it becomes available.

Testing Profile Visibility Controls

If you are ready to start testing profile visibility controls, please raise a ticket in our Developer Support Help Desk here . We are not quite ready to set up developer environments but this guide will describe how testing will work and what you can expect once privacy controls are turned on.

Thank you

We realize this is a large amount of work to complete in a short timeframe. We appreciate you working with us through this process to deliver a more trusted experience for our customers.

If we’ve missed anything preventing you from completing the migration, please comment below.

[christoffer]

Hi @akassab,
Just felt the urge to let you know that all of us at K15t, but I think I can also speak for the vendor community, are aware of (and excited by) the great work you are doing for Atlassian and the vendors. You get a lot of complaints and frustration from us - sorry for that. I hope you and your teammates feel valued and respected :all the time:.
It’s outstanding how you have been keeping track of all the open ends and making these and the progress transparent to us outside Atlassian.
Have a great weekend and keep up this great work, Chris

[andreas]

Thanks for posting this! I wholeheartedly agree with Christoffer - I know I’ve definitely lately been more in the ‘complaining’ bucket. We here at Code Barrel definitely appreciate all the hard work around this project inside Atlassian as well and (having been inside myself) know that it’s tricky to get something that may break things across the board shipped given all the conflicting internal priorities and projects that are always going on at Atlassian. These weekly updates of late have been fantastic so thanks for sharing them so openly Alex (and the rest of the team at Atlassian).

At the end of the day we do appreciate the improvements this will bring around customer privacy.

Cheers,
Andreas

[jmort]

I wholeheartedly echo Christoffer and Andreas. I cannot imagine the amount of cat herding necessary to do something like this. The weekly posts are much appreciated, the transparency goes a long way to reducing our frustration.
Jon

[brendan]

Thank you for this update!

We are making great strides towards compliance. We’ve actually had to rewrite more than most as the REST APIs did not exist when we started and still had a few old rpc calls to replace, some of for which there are yet no REST calls.

I just received the note below and had not heard the “March 15th” date before. Does the ‘March 15th’ date actually mean anything specific?

You are being contacted because you are listed as a contact for an app that has not yet used the mechanism to signal readiness for the new API behavior. We are asking you to update your app descriptor to signal readiness for the deprecation by 15 March 2019.

Thank you!

[akassab]

Thanks @christoffer; your note definitely brightened my weekend. Your sentiments mean a lot to me (and the rest of the team here at Atlassian). This project has been a process and - while I regret not starting this series sooner - I’m glad that the team here landed on something that seems to be well received. Its important to all of us here that we ship together so finding a way to communicate with and bring the developer community along was critical to the success of this project.

[akassab]

Thanks @andreas for this note and being so active in the ACJIRA (also for updating your app descriptors to signal that you’re still working on the migration). Your feedback is truly valuable in helping us identify the things that we’ve missed or have not explained well. While it would have been ideal to have solved everything before starting the deprecation period (for both our teams here at Atlassian and for the Developer Community), thats often not how our deprecation notice periods or migrations happen. We often rely on the developer community for feedback and I appreciate your efforts and patience with supporting us through this process.

[akassab]

Thanks @jmort! There are definitely a lot of teams involved and more than a few moving parts.
I’m glad these weekly posts are helping to provide more transparency.

[akassab]

Hi @brendan - Thanks for the update. The email you received was sent to you because you haven’t yet set your app descriptor(s) to signal readiness for the deprecation of username or user key.

As a reminder: if you are ready, set the apiMigrations flag to gdpr:true , if you are still working on the migrations please set the apiMigrations flag to gdpr:false. This blog post explains a bit more.

The March 15, 2019 date mentioned in the email is an internal milestone we’ve set to discuss developer readiness for the deprecation. As mentioned earlier in this post we have low signaling from the developer community (~ < 10% have indicated either gdpr:true or gdpr:false in their app descriptors). We’re hoping to get > 50% signaling by end of next week which is why we reached out via email.

[david2]

Hi @akassab, I didn’t get a chance yet to express my support for the incredibly hard work you’ve been doing on GDPR over the past 6 months (at least), so here it is And I hope Atlassian gives you a 6-month paid vacation to recover, once this project is over!

I do have a question though: is there any other way to communicate our GDPR status than releasing a new app version? This creates a new version on the Marketplace that is kind of confusing to users if we don’t have any new feature or bug fix to offer at the same time…

Thanks!
David

[rwhitbeck]

To be transparent, in the note that @brendan described above, we sent the following email to 419 emails who are listed as contacts for apps listed in the Marketplace that have not published an app descriptor using the apiMigrations property and have set the gdpr property to either true or false. As @akassab explained in her reply we’re looking for indication by March 15th of app readiness (either ready gdpr:true or not ready/blocked gdpr:false). Again this blog post explains what we’re looking for.

Are you ready for the GDPR API deprecation?

Immediate action required

We are making changes to our cloud products in order to provide users with more control over their profile information — both in response to recent changes to data privacy legislation (i.e. GDPR) and as part of a broader effort to improve customer trust.

These changes require apps to migrate their app and data stores to a new global identifier — the Atlassian Account ID, and remove legacy user references (username and user key).

In October, we published migration guides signaling the start of a 6-month deprecation notice period. These migration guides include details about which Jira Cloud, Confluence Cloud, Bitbucket Cloud, and Atlassian Connect REST APIs are changing. The deprecation notice period ends on 29 April 2019.

In addition to the migration guides, we’ve implemented a new mechanism in Atlassian Connect which allows developers to signal readiness for the deprecation. The mechanism when opted in also allows developers to test their apps with the new API behaviors.

We track use of this mechanism to understand developer readiness in order to facilitate actions to unblock and provide weekly reports on our status of those actions in the Developer Community.

You are being contacted because you are listed as a contact for an app that has not yet used the mechanism to signal readiness for the new API behavior. We are asking you to update your app descriptor to signal readiness for the deprecation by 15 March 2019.

Learn how to update in 3 easy steps

Since the blog post was published last week we have seen an uptick in apps with published descriptors with the apiMigrations flag and we want to say thank you to all of you that have sent your readiness signal to us. By sending the above email we want to ensure we are reaching everyone, including those that might not be reading the Developer Community regularly.

[rwhitbeck]

As it was first reported in this weekly update we’ve pushed out the deprecation deadline for migrating username and user key to accountID to April 29, 2019 instead of March 29, 2019.

We’ve just published this blog post to announce the new date.

I’ll be working over the next couple of days to make sure all of our documentation has the updated deadline date as well.

[brendan]

@rwhitbeck and @akassab,

Wow thank you for the quick responses!

Just to let you know 3 of our 4 apps are completed and marked as GDPR compliant. We are scrambling to complete the 4th, but have not pushed a release including that “apiMigrations flag” for our last one. We will be compliant within the next week or two.

Not directly related to your effort but have also been slowed down by open and ongoing issues like this one.
Additionally, the new Confluence V2 editor is causing massive problems that were known well before roll wide roll out but it was rolled out anyway.
Just FYI, sometimes the headwinds are VERY strong for this platform. AND sometimes they are not. For some reason, they have been for the past couple of months.
I literally only mention these things to share the perspective that time which we would have invested in completing these changes weeks ago has been consumed by those new issues with the platform which affected hundreds of just our customers…narrowly avoided it being thousands.

Cheers,
Brendan

[david2]

Hi @akassab,
for the next update, could we please have a clarification on the exact effects of:

• the x-atlassian-force-account-id’:true header
• the gdpr:true apiMigration flag

I learnt yesterday that the latter has zero effect on Jira’s behavior, and only the header changes what Jira returns from REST calls. Is that true? For example, does the api migration flag change what Jira passes to the app on workflow post-function calls? Or does it change anything in user objects passed and/or returned by Jira?

As for the http header, it was initially created to force the /issue API to return accountIds in the changelog, but is it now what forces usernames and user keys out of any response?

Our concern is with gradual migration - if we release our GDPR-ready version before April 29, we’d like to ensure everything continues to work seamlessly - thus we need to make sure we still receive usernames and user keys from Jira, while still receiving accountIds since our code now uses accountIds internally. Therefore, we can’t really remove the http header, and we’re not sure what gdpr:true will do either…

Thanks in advance,
David

[rwhitbeck]

@david2,

Great question, in writing the blog post last week on the proper use of the apiMigration flag I needed to truly understand what was happening as well.

If you look at this page, you notice a bunch of APIs that are affected by GDPR. All of them except REST APIs involve Atlassian Connect. As the apiMigration docs state the gdpr flag is only updating the Atlassian Connect APIs (Inbound Auth, Outbound Auth, App iFrames, App Lifecycle, Webhooks, and JavaScript API) while the header/query parameter only affects the REST APIs as you can use the REST API outside of an app.

I hope that makes things more clear.

[akassab]

Thanks @david2 , after this I think we all deserve a vacation .

In terms of other ways to communicate GDPR status other than releasing a new app version - no I’m afraid not. You may want to use the release notes feature on Atlassian Marketplace to communicate to user’s watching your app that you’re working on changes for the cloud version and suggest ways they could help - for example, reviewing and re-saving their historical post function configurations to assist with the in product data storage problem we’ve sync’d on previously. (I’m not suggesting that be the solution to the in product data storage problem given that not all customers watch apps or action release notes - merely offering a suggestion for reducing customer confusion about the version release).

[charlie]

One aspect I haven’t seen covered here or in any of the ACJIRA issues is how Jira project page URLs should be constructed once user.id and user.key are removed by AC-2417.

Our app has a project admin panel, and when you click on it in the Jira navigation it takes us to a path like this that contains user data:

/plugins/servlet/ac/com.thinktilt.proforma.add-on/project-forms?user.key=admin&project.key=SERVICE&project.id=11215&servicedesk.serviceDeskId=35&user.id=admin

The question is what is going to happen to that URL after the migration period? If Jira simply drops the user data from the URL then navigation is likely to break. That’s what happens if we do that now… if we manually change the URL to this:

/plugins/servlet/ac/com.thinktilt.proforma.add-on/project-forms?project.key=SERVICE&project.id=11215&servicedesk.serviceDeskId=35

Then our page displays but the navigation is broken:

This is a problem for us because we have internal navigation between parts of our app — we have links from other pages to this project page. We will have to remove the user key from the links we generate, but if we do that now the navigation will be broken so we don’t want to do it yet!

Ideally Jira could give us the links so we don’t have to generate them ourselves and deal with the user problem. However the AP.navigator.go API which should do that is currently broken so we can’t use that either (we raised ACJIRA-1678 on that a few weeks ago).

So my question is what links should we be using for project pages? And will the navigation bar keep working after the GDPR migration removes user data from URLs, or is that a bug which hasn’t been dealt with yet?

[rwhitbeck]

This weeks update has been posted Weekly GDPR API status development update - March 7th