Forge App Requests to External System Returning 503

Forge
, , ,
1

Hello,

We’ve encountered a strange issue with our Forge app. Nothing has changed in our implementation, and it was working perfectly until recently.

Now, in the vast majority of cases, when our deployed Forge app sends a request to our external system, it returns a 503 error. However, when running the app via Forge Tunnel, everything works fine. This happens across all Forge environments.

Does anyone know what could be causing this?

Have there been any recent changes on the Forge platform that might affect outgoing requests?
Or is this more likely an issue with the external system’s configuration?

Any insights would be greatly appreciated!

Adding the response here as it might help:

{
  status: 503,
  statusText: 'Service Unavailable',
  headers: Headers {
    date: 'Thu, 13 Feb 2025 10:10:36 GMT',
    'content-type': 'text/html;charset=utf-8',
    'content-length': '3113',
    connection: 'keep-alive',
    'strict-transport-security': 'max-age=63072000',
    server: 'envoy',
    'mime-version': '1.0',
    'x-squid-error': 'ERR_CONNECT_FAIL 110',
    vary: 'Accept-Language',
    'content-language': 'en',
    'forge-proxy-upstream-latency': '5477',
    'x-envoy-upstream-service-time': '5494'
  },
  body: ReadableStream { locked: true, state: 'closed', supportsBYOB: true },
  bodyUsed: true,
  ok: false,
  redirected: false,
  type: 'basic',
  url: 'https://vpce-0474ec11bf52fb8bb-1ezyg8yy.vpce-svc-01191597db9a71f9f.eu-west-1.vpce.amazonaws.com/egress'
}"
2

Hi @MartinFischer, I can do some digging, are you able to help me out with the following:

  • How do you make the request to your external system? (are you using global fetch or our fetch client?)
  • Do you have any dates on when you started seeing the error?
  • Can you provide invocation traceIds that include failed requests?
2 Likes
3

Hello @BoZhang, thank you for your help.

  • We are using the @forge/api fetch making our request calls
  • We have noticed it doing the last couple of days (during this week, we noticed this on 12.2.)
  • Invocation traceId of a failed request - 68367245710d4561a8b381dc6c80e210-83b7340ebb7f1503
  • Invocation traceId of a successful request - 15d7c4ded59f4319bd7029e929189532-01d17ac53f151c3c (they happen really sporadically now)
4

Hi @MartinFischer, thanks for providing me with the details above.

So I did a bit of digging and I can indeed see a large number of errors occurring within the last 2 days for your app, but it seems to be in proportion to the amount of network traffic from your app, which seems to be elevated over the last couple of days.
I also had a look at the logs over the last 30 days and I can see this error occurring throughout the period, so it seems like this has been a persistent issue.

The particular error that seems to be occurring is specific to our proxy not being able to setup a TCP connection with your server (note the value x-squid-error response header is ERR_CONNECT_FAIL 110).

I will see if there has been any changes to our proxy configuration, which may cause TCP connections to fail. In the meantime, can you please check the configuration on your side and see if there is any possibility that your server is rejecting incoming TCP connections.

5

Thanks for looking into this and for the insights @BoZhang!

Just to clarify, the errors seen throughout the logs were mostly caused during our testing, where we intentionally checked how the app behaves when the service is unavailable. However, in the last couple of days, we’ve noticed these errors even when the service is available—some requests are still succeeding, which makes this more unusual.

Also, when running the app via Forge Tunnel, everything works fine, and the server responds as expected. The issue seems to happen only in deployed environments. I’ll check our server configuration on our end and will update you.

Edit:
I checked it with my colleagues, and it seems nothing has changed on our side.

6

Hi @MartinFischer, I spent some time with the team debugging this today.
What we found from our logs and testing is that:

  • It seems like we cannot connect to your external service at all now, from what it seems, there is no listening web server. Did you make some changes recently to firewall rules or the service itself?
  • One of the theories as to why it worked sometimes but not others is due to regional routing, where we attempt to reach our service from proxies with different IP addresses and some of them might be blocked. Can you please make sure that you have whitelisted all of the IP addresses listed here: IP addresses and domains for Atlassian cloud products | Atlassian Support
1 Like
7

Hello @BoZhang, I apologize for the inconvenience.

  • The testing server needs to be manually started and automatically shuts down every night. It is only active when required for testing. I can ensure that it remains available for most of the time during your further testing, specifically from 6 AM to 8 PM Central European Time (CET).

  • Regarding access, the test server should be open to all connections, and as I previously mentioned - according to my colleagues, there should be no configuration changes. However, I will personally check today to confirm this, and if I find any changes, I will update you accordingly.

Please let me know if you need any further details.

1 Like
8

Thanks @MartinFischer, I will try repeat some of my tests tomorrow with your server up.

In terms of further details, if you could let me know of how you are conducting the tests on your side and if there are any specific patterns with the failures, that would be useful for me to know.

9

Awesome, thank you!

I checked the server configuration, and at first glance, nothing seems off. However, when I tested it today, most of the requests went through successfully—though not all. Running the app via Forge Tunnel still works as expected.

Since I’m not entirely sure what’s causing this inconsistency, I suspect the issue might be on our side, even though we haven’t encountered this problem before.

If you don’t find anything unusual on your end, I’ll proceed with investigating this further on our side and escalate it accordingly.

10

Hi @MartinFischer, I simulated traffic hitting your external service via our proxy earlier, sorry if this caused any noise on your side. Unfortunately, I wasn’t able to reproduce the issue.
If you could, can you please try create another script which does the same thing as your Forge app but runs out side our network to see if you can reproduce the same issue? Other than that I think the next step if you need us to investigate this is to create a support ticket with sample code to hit your service so that we can reproduce it on our end.

2 Likes