Weekly GDPR API status development update - March 7th

akassab · March 7, 2019, 7:59pm

Apps are required to migrate to Atlassian AccountID and remove legacy user references (username and user key) by April 29, 2019 (Europe: Paris), April 29, 2019 (America: Los Angeles). For more background on why - please see our first post .

Understanding migration status

We are tracking use of the apiMigrations flag in the Connect App descriptor as a way to understand developer readiness to deprecate username and user key from our REST APIs. This flag serves as both a mechanism to test the new API behaviors (when set to gdpr:true) and a communication tool to signal when you are blocked (when set to gdpr:false).

Today, we have 66 apps that have signaled that they are ready for the deprecation, 18 that have signaled they are blocked, and ~700 that have not indicated their status.

In order for us to understand where you are with migration to accountID and use that information to effectively communicate with our product teams we need you to signal your status using the “opt In mechanism”.

Learn more about how to properly use this flag.

Additional note on opt’ing into the new API behaviors

The flag described above controls Atlassian Connect behaviors. When opt’d in gdpr:true only the APIs related to Atlassian Connect will change this includes: Inbound Auth, Outbound Auth, App iFrames, App Lifecycle, Webhooks, and JavaScript API. To change the behaviors of Jira and Confluence REST APIs you will need to pass additional header/query parameters on every REST call. The header for Jira APIs is x-atlassian-force-account-id’:true. This header is also required to transform change logs from username to accountID. The query parameter for Confluence is privacyMode=true.

Recently closed issues

Jira webhooks missing accountId and account type - Jira has communicated that this is rolling out this week and should be complete by end of week. See ACJIRA-1674.
User mentions in Wiki Markup not working with expected format - User mentions in Jira require a prefix prior to accountID in order to work correctly. New user references will be saved in this new format, however, historical references not stored in Jira will need to be migrated. Jira will not commit to facilitating this migration prior to the deprecation of username and user key. See ACJIRA-1715
Email address hidden by default - Check out our newly released Guidelines for requesting access to email address. The Email API is still being worked on (see In Progress).

Items in progress

AccountID in JQL statusby clause does not work - Jira is working on this and have a fix in place but not released yet. This may be resolved before our next weekly status update. See ACJIRA-1625. An additional issue was raised specifying componentsLeadByUser() method. The work to fix this bug is in progress. We do not yet have an estimated fix date yet. See ACJIRA-1704.
Jira user picker REST API missing account type - Same as above (fix in place but not released yet). See ACJIRA-1686.
Jira post function /triggered calls missing accountID and account type - This issue was split from the above. The Jira team has is preparing to work on this issue. The estimated fix date is March 20, 2019 (Europe: Paris), March 20, 2019 (America: Los Angeles). See ACJIRA-1722.
Jira incoming webhooks missing accountId - Jira is planning to add a separate event that will use accountIds. The work on this issue has not started yet nor do we have estimates for when this will be fixed. See ACJIRA-1692.
Jira does not recognize issue entity property aliases - The Jira team is currently working on this issue and estimates this work to be complete next week. See ACJIRA-1723.
Jira does not have an API to edit workflows - The Jira team is working on this and expects the work to be complete March 19, 2019 (Europe: Paris), March 19, 2019 (America: Los Angeles). See ACJIRA-1718.
JQL transformed to remove legacy user references and replaced with accountID are un-readable - We’re building a JQL Editor Dialog component that will display the user’s current name in JQL rather than the accountID. This work is in progress and we expect this to ship March 15, 2019 (Europe: Paris), March 15, 2019 (America: Los Angeles). See ACJIRA-1710.
Email address hidden by default - The Email API is being worked on now and should be available on March 15, 2019 (Europe: Paris), March 15, 2019 (America: Los Angeles). See ACJIRA-1726
Jira PD Cleaner converting user strings that are not exact usernames to ‘unknown’ - This issue was raised in ACJIRA-1712 but now is being tracked in ACJIRA-1725. We do not have estimates for how/when this will be fixed.
Confluence documentation for new REST endpoint for migrations - Jira and Confluence have new migration APIs that are unaffected by the 29 April 2019 deprecation. (See here for Jira: /rest/api/3/migration ) The endpoint for Confluence is ‘/wiki/rest/api/user/bulk/migration?username=’. Documentation is not currently available. We will provide a link as soon as it becomes available.

Testing Profile Visibility Controls

If you are ready to start testing profile visibility controls, please raise a ticket in our Developer Support Help Desk here . We are not quite ready to set up developer environments but this guide will describe how testing will work and what you can expect once privacy controls are turned on.

Thank you

We realize this is a large amount of work to complete in a short timeframe. We appreciate you working with us through this process to deliver a more trusted experience for our customers.

If we’ve missed anything preventing you from completing the migration, please comment below.

rwhitbeck · March 7, 2019, 8:20pm

tim · March 7, 2019, 10:40pm

Thanks @akassab! These weekly updates have been really helpful.

We’ve got a bug blocking our GDPR journey. We had to rollback our descriptor from true to false after receiving an influx of customer support tickets regarding the Jira issues macro in Confluence.
https://jira.atlassian.com/browse/CONFCLOUD-65773

joshua.galea · March 7, 2019, 10:55pm

Hi @akassab
One question about all this, are the emails for users already meant to be coming through in cql searches with the new privacy mode on? or is that still coming? I’m still unable to get user emails to be returned through cql queries regardless of my account settings.

charlie · March 7, 2019, 11:10pm

Thanks @akassab from us too!

We are still blocked by the Jira navigation breaking when user key is removed from project administration page URLs, see my comment on last week’s post for details. Though it’s not a critical problem, it’s enough for us to not want to deploy our code with user references removed. We don’t know if the navigation problem is being dealt with, so we’d appreciate hearing something about it.

akassab · March 7, 2019, 11:25pm

Thanks @tim - This issue isn’t one that we’ve been tracking as of yet. I’ll confer with the Confluence team to see what they’d recommend.

natashbar · March 7, 2019, 11:52pm

Hi @charlie,
We are looking into it, will update you shortly.

natashbar · March 8, 2019, 3:45am

Update:
We have a fix on the way that removes those personal data related context parameters (like user.key) from the project admin tab links. Initial verification shows that the navigation bar also works fine.
However, in order for this to work properly the app needs to be opted in to ‘gdpr’ via API migration declaration in the descriptor.
We will update this thread when the fix is in production, so that you can start testing. If it is not possible for you to opt-in with your production app (due to blockers, etc), you can always test that out with a sample app.
Regards,
Norman

jbevan · March 8, 2019, 9:33am

Thanks for the update, I’ve lost track of where the CQL PD Cleaner API got to. Is this done or planned?

We are unable to use the Confluence user migration REST API to migrate stored CQL unless we implement a CQL parser which is a lot of extra effort for vendors to go to when I’m assuming there’s already a CQL parser somewhere in Confluence

shesse · March 8, 2019, 3:58pm

@akassab We haven’t opted-in with gdpr:true for our app so far, because we were unsure of future changes to the Connect APIs which might break our code in production. Today I was testing again the behavior with gdpr:true (due to your mails and weekly requests to set this flag ) and I have noticed that setting it to true didn’t change anything in the webhook data we’re receiving. I mean, the username, user key and email address are still present in the webhook data. (Of course I have updated the installation of my app in my Jira developer instance, I also uninstalled + installed it, no change) However, I’m pretty sure that this was different when I was testing this flag the last time (sometime in January I guess) where this user data was removed after opting in. Did you recently receive similar feedback also from someone else? Or am I confusing something? Based on your description and my (sometimes bad) memory I’d say the user data should not be there anymore with gdpr:true.

rwhitbeck · March 8, 2019, 5:49pm

This was the update @akassab provided two weeks ago.

Conversion of stored CQL containing legacy user references to accountID - The Confluence team is aware that this is an issue, however, have de-prioritized this work due to the amount of remaining work to prepare for profile visibility controls. We recommend using the new migration API provided by Confluence to transform CQL containing old user references to accountID.

akassab · March 8, 2019, 6:49pm

Sorry @rwhitbeck and @jbevan for the confusion. I did communicate that last week but have since heard from Confluence that they are in fact working on it now. @sluthra mentioned this in an email thread we have with the team at Adaptavist. I will include the status in next week’s post.

akassab · March 8, 2019, 7:03pm

@shesse - Setting the descriptor will only modify Connect API behaviors. You will also need to pass a header/query parameter on all of your REST calls to modify product API behavior. For Jira this would be a header x-atlassian-force-account-id:true and for Confluence this would be query parameter privacyMode=true. We missed adding accountID (and account type) to webhooks at the start of the deprecation notice period but have recently added them. Confluence webhooks should already have them and Jira was rolling out webhooks this week. Can you confirm if you’ve tried using the header/query parameter method?

akassab · March 8, 2019, 7:25pm

@joshua.galea - I don’t think email address was ever returned for a user with CQL. CQL is not a database query language, unlike JQL. You can only add query parameters to filter the list of returned content.

shesse · March 8, 2019, 9:31pm

Of course I need to set the header when calling the REST API to not receive usernames anymore (this is working fine btw), but that was not my concern. My concern was about the webhook data we receive, for example for jira:issue_updated. Setting the gdpr:true flag should influence the webhook data, because this flag is related to the Connect API behavior (as you’ve mentioned), right? The webhook data is not coming into our app by us calling an API in Jira, but instead Jira/the Connect API is calling an endpoint on our side. So, when I set gdpr:true, I assume I should only receive webhook data which does not contain username or user key information anymore, right? Or am I confusing this with the HTTP header x-atlassian-force-account-id ?

For example, a webhook (with gdpr:true in my case) now still contains the following data:

{
            "timestamp": 1552059883865,
            "webhookEvent": "jira:issue_updated",
            "issue_event_type_name": "issue_updated",
            "user": {
                "self": "...",
                "name": "addon_com.k15t.backbone.backbone-issue-sync",
                "key": "addon_com.k15t.backbone.backbone-issue-sync",
                "accountId": "557058:...",
                "emailAddress": "com.k15t.backbone.backbone-issue-sync@connect.atlassian.com",
                "avatarUrls": {...},
                "displayName": "Backbone Issue Sync for Jira",
                "active": true,
                "timeZone": "Europe/Berlin"
            },
            "issue": {...}
}

I was assuming that setting the flag will remove name, key and emailAddress (and I guess some more) from this data. This example is now for an app/system user, but I’ve seen the same for normal users as well. Is my assumption correct or am I wrong here?

natashbar · March 10, 2019, 10:11pm

Update #2:
Further testing reproduces the problem with the sidebar, and we have raised this issue for fixing it: https://ecosystem.atlassian.net/browse/ACJIRA-1736
We are holding on to the changes in Connect that removes those context parameters (user.id and user.key). We will verify and release that change once the ACJIRA ticket above is fixed.
cc: @charlie

charlie · March 12, 2019, 6:08am

Thanks for the update @natashbar! We will watch ACJIRA-1736.

f.demir · March 12, 2019, 10:59am

I am unable to retrieve an avatarUrl when calling the Jira endpoint /rest/api/3/groupuserpicker while having showAvatar=true and x-atlassian-force-account-id header set to true. Setting the x-atlassian-force-account-id header to false does return an avatarUrl as expected.

acalantog · March 13, 2019, 3:43am

Hi @f.demir,

Please track progress for this issue in ACJIRA-1732

Cheers,
Anne Calantog

shesse · March 14, 2019, 7:30am

I haven’t got an answer to my previous reply in this thread so far, but Ivan commented on this issue (https://ecosystem.atlassian.net/browse/AC-2433) that it seems I was right with my assumption that webhooks still contain the name and key even when opted in. Just wanted to let you know about it.

// Edit: Okay, I understood that comment & issue in a wrong way. It is about the query parameters, not the webhook payload data. So maybe it’s not related. Anyway, it would be interesting to get an answer about my question above.