Hey @szn,
Here are a few thoughts/ideas on what could have gone wrong:
But sometimes this object is missing the key
refresh_tokenproperty.
At this stage, the OAuth connection may be broken. Either because on the next token refresh, your old refresh token is not accepted anymore and you get the “Unknown or invalid refresh token” error, or because you overwrite the existing refresh token with the token response that is lacking the refresh token. In this case, on the next token refresh, you would not have a refresh token. I cannot comment on why you sometimes do not get a refresh token back, but maybe someone from Atlassian can help you out with this.
Another potential issue could be that you run your program in a clustered environment, in which case you would need to synchronize the token refresh between your running instances. Otherwise, you may get lost updates when multiple instances are trying to refresh the same access token at the same time. Unfortunately, this is a massive complexity introduced by the rotating refresh token approach and seemingly ignored by Atlassian thus far.
The general idea to mitigate issues with concurrent token refreshes in the Auth0 rotating refresh tokens implementation (on which Atlassian’s is based) is to have a reuse interval in which older (previously rotated) refresh tokens can still be used (currently configured in the Atlassian implementation to 10 mins). This method is a way to mitigate the problem of concurrent attempts to refresh a token, however, it does not protect you from lost updates in a clustered environment. If you are using Redis, then Redis dist locks may be able to help with building a cluster lock to synchronize a token refresh.
I hope this helps.