Hey @PeterVelosy and @Margus_SoftComply,
Have you tried using the new native Node.js runtime?
This deploys your code also on EU servers and might improve performance.
We have done this and found that everything is a bit faster (although I can’t prove it).
@ppasler I am using the new native Node.js runtime, there is some slight improvement in comparison with the old runtime, but it is unfortunately still extremely slow.
I have collected some measurements in this post: https://community.developer.atlassian.com/t/forge-bridge-functions-are-very-slow-geographic-latency-in-europe
Any updates here?
I have read the announcements about Forge Data Residency and Multi-Region Compute (https://developer.atlassian.com/platform/forge/data-residency/). Since, however, data residency seems not to be available for free Jira licenses (Move product data to another location | Atlassian Support), I can’t do a test on my development instance whether migrating the host app and the Forge apps to Europe makes function invocations any faster.
Please use an instance created through https://go.atlassian.com/cloud-dev. These are Standard tier instances provided free for developers; instead of Free Tier instances. There are user limits (5 users for JSW + Confluence, 1 agent for JSM) and the instance will be converted to Free tier if you exceed them. But I was able to find all the data residency settings in admin.atlassian.com as described in the docs you linked.
I successfully moved my development Jira to EU (pinned status). I installed the app afterward, but it is not eligible. I have read somewhere that It may not be eligible because my instance is already in the EU, but how can I verify that? I don’t see any residency app status.
Same here. I moved my Jira (created through the link @ibuchanan has posted) to the Germany region, but I get a “not eligible” status for my Forge app. Maybe because it does not use Forge hosted storage (all the data is stored within Jira)?
At this stage, I unfortunately haven’t noticed any performance improvement of my Forge app since moving my Jira to Germany… Is migrating the app back to Connect really the only way out?
@ibuchanan May I ask for some input on this one, please? I am seriously considering migrating everything back to Connect, since the performance I am seeing is unacceptable in any production scenario. Thank you!
As it is worryingly silent in this thread, let me illustrate my point with a video.
Here is how an trivial Forge Jira IssueContext (running on a development instance generated as described by @ibuchanan above) that does not do anything but invoke a trivial resolver endpoint (as it is generated initially by Forge CLI 10.0.1) loads from Vienna, Austria: https://www.youtube.com/watch?v=NYiKARWL_UY . The code is running remotely, no tunneling is involved.
Speed-wise, this is unacceptable. No software tester would let it out into production and no customer would pay a cent for it.
One of my previous posts above contains a link to concrete performance measurements (from a more complex application where it becomes obvious that the overhead is not caused by addon code but by Forge platform calls).
Please, Atlassian, show us a way ASAP, how this can be improved, or issue a clear statement that application developers targeting the European market should stick with Connect for the time being.
Thanks for your persistence on this topic and for your thorough data gathering. I fear that some context might be getting lost from the duration of this thread and the Forge changes that have happened in the meantime. For those on this thread, or folks having invocation performance problems, let me summarize a few points.
Atlassian’s first engagement on this topic acknowledged that latency would be expected in the case of Forge Tunneling. At that time, FRGE-1098 “Improve performance of Forge app execution when using Forge tunnel” was already open. That issue remains in “Gathering Interest”. Maybe that should be interpreted as “Gathering data” with more data still needed to understand how best to solve for tunneling performance. Subsequent data points on this thread, like @klaussner quick comparison of with/without tunnel showed that tunneling is slow, but it doesn’t fully explain high latency.
Later, @HeyJoe acknowledged that Forge compute was geographically locked in 1 place. With CHANGE-1484 “Action required to avoid performance impacts and to offer data residency for Forge hosted storage apps in beta”, Multi-region compute became available. With multi-region storage and compute, there are a set of migration steps needed to take advantage of those improvements. The Forge Native Node runtime helped with overall performance. If you had a deployed app in or before the Feb-Apr 2024 timeframe, please follow the indicated steps for migration and redeploy your app.
The fresh deployment of a new trivial app was a good way to avoid all the complexities of those changes. Since we still have a problem, I opened FRGE-1435 “Forge invoke has high latency (>1s) in Europe (maybe other geographies)”. I regret that we didn’t have have that issue sooner, so I’ll take the opportunity to remind the community not to treat threads like this as formal feature requests (which can be opened in our open Jira (JAC) in the ECO project) or bugs (which can be filed with developer support). Now that we have an issue to track, please “pile on” by watching, voting, and commenting with more data points. I’ll engage the Forge team internally to see if we can get comments from “the source”.
The Forge team would like to investigate what we can see in our logs. Can you supply App ID and, if possible, Install ID for your trivial Forge App?
@ibuchanan Certainly! The appId is “ari:cloud:ecosystem::app/d39be3ad-6989-45f9-b6a4-23762ef5ef0f” and the installationId is “59648682-a879-4c73-bfbd-a9f1a57d17d8”.
@ibuchanan I don’t see any updates in the public ticket yet. What did your colleagues see in the logs and what is the way forward?
@ibuchanan , similar issue - i created an app to find me number of open bugs for each day for the for past 2 weeks…I am calling JQL for each day to get the list of open bugs for that particular date and the response is way too slow. Any suggestions?
Please open a developer support case so you can share details of your app and have the logs analyzed. Unlike @PeterVelosy’s case, I don’t think your case sufficiently isolates Forge. Complex queries like what you describe have been known to tax JQL. If your app is timing out from trying to wait for those results synchronously, you’ll probably need to use Forge Async Events.
What the team saw was high latency in AWS Lambda cold starts and creating TCP connections. We couldn’t come up with any recommendations that you can take on your side. What the team is doing about it or recommends will take more time. Thanks for your persistence in following up.
Just to be clear, Forge Async Events are infrastructure for queuing functions, which isn’t the same as JavaScript’s internal support for async functions. It won’t make requests go faster, but it will help avoid function invocation timeouts. It helps but doesn’t completely remove them.
Yes, proposing you performance problem as a bug should work. I highly recommend doing what @PeterVelosy has done by creating a “simplified test case”: a code example that isolates the problem as best as you can. That’s going to help the support engineers reproduce the problem and engage with the Forge team.
Got it. Thank you.
I have uploaded a snapshot of my backend function. Will add more info as needed.
@IbrahimSaid In which country are you located?
@ibuchanan In addition to the already ongoing research, could you perhaps also tag the Atlassian teams in Poland and the Netherlands in the respective ticket so that they also execute some performance tests from their locations? Also, do you have a way to monitor the performance of Forge API calls per country so that you can arrive at a map of all affected geographical locations? If you currently have no central way to do so, perhaps it would make sense to introduce an opt-in performance telemetry solution for Forge applications deployed to development environments.
Noida, India. It’s next to Delhi.