Getting 403 on 3LO token exchange for sizeable number of random customers as of this week

We have a 3LO app that we have a number of users authenticated with. We started noticing an increasing trend as of around last Tuesday where a large (random) number of our customers can no longer perform successful token rotations to obtain fresh tokens and are getting 403 responses, seemingly having their existing grants all of a sudden invalidated.

This is essentially breaking our app for these users and the automations that they have created in it (until the users go and completely re-authenticate with each of their connections again), and is causing all sorts of frustration and confusion. The refresh tokens are regularly rotated and aren’t expired, and we haven’t yet upgraded to granular scopes (for which the deprecation is now cancelled).

We have a large chunk of users who aren’t affected by this, but for users that are it seems to be affecting everybody within their organization who have made connections to the same domain. We also haven’t updated our 3LO app configuration during the time.

I am just wondering if anybody else who is using 3LO with customers is seeing similar behaviour amongst them? Although I think it’s unlikely, could it possibly be somehow related to the outages from earlier in the week? If I am the only one experiencing this/it isn’t anything on Atlassian’s side, then I’m curious if anybody has any ideas about what could be the problem?

Thanks much appreciated

2 Likes

From what I can tell about the currently on-going incident, 3LO clients for those customers were disrupted. We are continuing to work with affected customers & vendors to restore service.

There is still a possibility your problem is unrelated. Since the above incident is engaging most of Atlassian right now, I doubt you will get a diagnosis here in the community. Could you please open a developer support ticket?

Thanks for the response @ibuchanan. Helpful to know that 3LO clients were disrupted as well due to the incident. I will follow up with a support ticket to further verify this is indeed the root cause of it for us. Cheers.