Big increase in request timeouts

I’ve recently noticed a large spike in request timeouts.

In addition I’m noticing this error pop up in my logs (though I’m not sure if it’s connected to the request timeouts as it will pop up at different times).

errorMessage: Error: connect ETIMEDOUT 
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1148:16) {
errno: -110,
code: 'ETIMEDOUT',
syscall: 'connect',
address: '[xx.xxx.xx.xxx]',
port: 443,
}

I’m running an ACE app and the request timeouts happen when I call my own API which then requests some data from Jira’s API. The request to my API will timeout and often a few seconds later another request will be initiated and succeed. I’ve searched my server logs and no error seems to occur when it times out.

I’m confused by this behaviour as it happens only occasionally and I’ve only been able to replicate it once by accident. I’ve tried reverting recent changes but this continues to happen at an increasing frequency.

node v14.7.15
atlassan-connect-express v7.4.1
pg v8.6.0

Has anyone been experiencing something similar? I’m running out of ideas.

Hi @RhysDiab ,

There is an incident (not publicly visible) relating to AWS PG DB connection timeouts that the Jira SRE team are looking into. I don’t know if this is related though.

Regards,
Dugald

2 Likes

Thanks for the response @dmorrow

That might be it since I’m using a PG DB hosted on heroku (which uses AWS under the hood).

Can I subscribe to this somewhere?

Hi @RhysDiab ,

There’s no public ticket and the team looking into this issue believe the issue only relates to a single tenant. Sorry, it looked like the might be a common underlying cause.

Regards,
Dugald

1 Like

We are seeing the same errors as @RhysDiab .

1 Like

Lots of 503 errors from Jira APIs recently… probably related
So it is definitely not about one tenant @dmorrow

1 Like

We saw quite a lot of Jira API timeouts yesterday (for multiple host URL’s). Today looking better but still occasionally happening.

1 Like