We recently updated our Jira Cloud app and migrated from Connect 1.5.0 to Connect 2.0.4 and SpringBoot 1.5.3 to SpringBoot 2.2.4.
Everything seems fine. The app passed all tests but since the update (for the last two days) we are seeing unusual numbers of JWT Expired errors in service logs. Similar to this one:
JWT expired at Wed Apr 22 02:15:26 UTC 2020 and time is now Wed Apr 22 03:30:42 UTC 2020 (30 seconds leeway is allowed)
Of course the exception is thrown by Atlassian’s authentication filters on the server side and the customer gets HTTP 401. The errors are not specific to any of our endpoints, happens all around. In fact this error even happens when the customer opens our app page in the first place. I mean, it is customer’s Jira that is making that request and It is customer’s Jira that is attaching the JWT to the request. We don’t know how it can be Expired.
Anyone wih experience with a similar case?
@emre.toptanci the code that checks the expiration time hasn’t changed since version 1.5.0, and uses plain Java APIs (so wouldn’t change with the Spring Boot upgrade).
Please provide some more details.
- Is the expiration time always off by so much? It’s over an hour in your example.
- For how many tenants have you seen these errors (one, a few, all)?
- At the time of the errors, is your server time reasonably correct? (This would suggest that the JWTs you receive are too old.)
1 Like
At the time of the errors, is your server time reasonably correct? (This would suggest that the JWTs you receive are too old.)
^^ Yes, please check the system time. If there’s not a regularly scheduled time update (i.e. cron with ntpdate), clock drift (even by just a few seconds) might be contributing to this.
2 Likes
Hello,
System time was a promising suspect but I compared the clocks of the server, the load balancer and www.timeanddate.com. The inconsistency among all these is less than 2 sec.
So far we see this error in 8 out of 147 distinct tenants that used our app since the aforementioned upgrade and for this 8, not all of their requests fail, only around 1% of them. As I said, the fails are not specific to an endpoint of ours.
In almost all of errors, the time difference in the error message (time between expire time and current time) ranges from 3 minutes to 90 minutes. Only in one case, it is almost 8 hours.
Yesterday, we changed the UI code and now making all REST requests to our server via “AP.context.getToken()”. This should make sure we get a fresh and valid token everytime but did not change anything. We were getting JWT Expired errors before this change and still getting them after the change.
This being caused by Connect or SpringBoot update does not make sense to me either but it is a symptom. The error is either caused by them or another change that was caused by their update.
We discovered some weird behavior while digging, that we cannot make sense of.
We tried this:
- Open our app page (so the user is past Jira authentication stage)
- Open another tab while the first page is open.
- From the new tab make a direct request to one of our endpoints without a JWT (a request that is normally made by the page via JS).
The steps listed above ends with a JWT Required error on Atlassian Connect 1.5 but returns the results fine with Connect 2.2. Our understanding so far was that every request from the UI to our app service had to include a JWT attached to it so Atlassian Connect filters would catch and process it but apparently that is not the case. Is Atlassian Connect keeping some kind of Session ?
And here is why this is relevant: In our app we are using LogAppender’s to collect error messages from container logs to a common data structure for reporting. That is where we actually track the errors. We believe we were already getting the same “JWT Expired” errors wih Connect 1.5 but (due to some irrelevant technical implementation details) LogAppender was also throwing an exception and the errors were never written to the common data structure. Probably we were getting these errors for a long time and did not know about it.
Now, this does not get us any closer to finding the cause of JWT Expired errors but at least it might explain why we started seeing them after switching to Connect 2.2.
Do you have a front end serving component (perhaps a cdn or nginx server etc)? If so does it do anything with query strings (or perhaps ignore it when it comes to cache keys?)
We see a lot of this, but have been ignoring this error so far:
Some examples just from today:
JWT expired at Fri Apr 24 10:16:03 UTC 2020 and time is now Fri Apr 24 10:36:09 UTC 2020 (30 seconds leeway is allowed)
JWT expired at Fri Apr 24 08:55:00 UTC 2020 and time is now Fri Apr 24 09:57:10 UTC 2020 (30 seconds leeway is allowed)
JWT expired at Fri Apr 24 08:55:00 UTC 2020 and time is now Fri Apr 24 09:57:13 UTC 2020 (30 seconds leeway is allowed)
JWT expired at Fri Apr 24 01:44:08 UTC 2020 and time is now Fri Apr 24 08:23:05 UTC 2020 (30 seconds leeway is allowed)
We observe many similar errors on app based on atlassian-connect-express too(about 40 today):
(401): Authentication request has expired. Try reloading the page.
Is it something usual? I remember 2-3 years ago being told that the token is refreshed automatically by the framework.
@danielwester
Nope, our containers run only behind AWS Load Balancer and everything is served from these containers.
@epehrson, @nmansilla
New idea…
Is it possible these JWT Expired errors are happening because the customer is putting his/her computer to sleep while the app page is open, waking it up the next day and trying to use the page without refreshing it?
1 Like
Is it possible these JWT Expired errors are happening because the customer is putting his/her computer to sleep while the app page is open, waking it up the next day and trying to use the page without refreshing it?
@emre.toptanci I imagine so, yes. That sounds reasonable 
1 Like
@epehrson
After thinking of “computer in sleep” scenario, we started sending all requests from UI via “AP.context.getToken()”. This method claims to get a fresh and valid JWT everytime, yet we are still getting these errors.
I was thinking “maybe the customer’s clock is wrong” but the are facing this on many different customers and the time discrepency is not mere seconds but minutes even hours. It is not reasonable to think that so many customers have so inaccurate system times.
Is Atlassian Connect keeping some kind of Session?
@emre.toptanci, I don’t have an explanation for the original issue you raised, but I looked into this report and raised it as the bug ACSPRING-108. Please upgrade to atlassian-connect-spring-boot 2.0.6 for the fix.
1 Like
Hello @epehrson,
I think we discovered something important about this.
First, let me give you some context: Our Jira Cloud app adds a new tab to issue view screen (which is defined in atlassian-connect.json). When the user opens that tab, the browser sends a request to our service asking for the tab contents and the request contains a JWT that is added by Jira. That JWT is set to expire in 15 minutes. So far, no problem.
If the user switches to another tab on issue view screen, waits for 15 minutes or more, then switches back to our app’s tab; the browser asks for the contents of the tab again using the same old JWT, which in turn causes a JWT expired exception in our service. The browser was expected to make this request with a fresh JWT.
We have no control over how the tab content request is constructed so we suspect this to be a bug in Jira Cloud. (We tested with Chrome on Windows 10)
Can you check this behavior?
Thanks…
Anybody else with an issue view screen tab in their apps is welcome to report their findings as well.
@emre.toptanci could you please raise a bug report for that, with steps (ideally minimal) to reproduce, including a sample app descriptor?
I realize the instructions there insist on only reporting critical bugs, which this hardly is, but I would like to capture a reproducible configuration.
Hello @epehrson,
While working on this, we identifed another case in Connect Framework that I hope can be improved.
When sending requests from our cloud app service to Jira Cloud .authenticatedAsHostActor(), Connect generates a JWT to authorize as the end-user. For subsequent requests, Connect uses the same JWT unless it expires. The JWT is set to expire in 15 minutes and Connect seems to be making an exact time check so even if it is 1ms before expire date, the token qualifies. When expired, Connect generates a new JWT on the fly and uses it.
The case is: When our service is sending a request to Jira Cloud at a time very very close to the expiry date of the connect token, occasionally the token is considered “not expired” by Connect in our service but Jira Cloud returns the following error:
Connect Session Auth Token is invalid. It has the following problems sessionClaimSet.expirationTime must be in the future
We think this happens due to very small time differences between our server and Jira Cloud instance, and also caused by the time lost because of network latency. As I said, the token qualifies even it is 1ms before expiry date but these time losses cause the token to be received by Jira Cloud after the expire date.
An easy solution would be to evaluate the token as expired (and force a refresh) if the expiry date is, let’s say, less that 30 seconds away.
We are now using the following versions in our app:
<spring.boot.version>2.2.6.RELEASE</spring.boot.version>
<atlassian-connect-spring-boot.version>2.0.6</atlassian-connect-spring-boot.version>
<dep.spring.version>5.2.5.RELEASE</dep.spring.version>
The aforementioned checks are all performed by deprecated springframework classes. I don’t know if this issue was addressed in newer released of spring framework but this causes a problem for Connect, especially for app services with high request volumes.
1 Like
@emre.toptanci, I have raised ACSPRING-109 to track this.
1 Like
Very good news, thank you.
I’m getting the same error if I Iet my Jira issue page sit idle long enough. Nothing else, just sit idle. I stay on the computer so it doesn’t go to sleep but regardless I always get this error. I’m using Jira Software Cloud and hosting my addon in Azure App Service.
I assume that it’s checkValidToken() that’s throwing the error.
- Why is the jwt not automatically being refreshed?
- How can I get the return value from checkValidToken() and advice the user to refresh the page?
Is there any update on this topic? We are facing the same behaviour, expecially:
Is it possible these JWT Expired errors are happening because the customer is putting his/her computer to sleep while the app page is open, waking it up the next day and trying to use the page without refreshing it?
Thank you in advance.
Cheers,
Andrea