usually, PaaS or IaaS Providers not only provide a way to inspect the current usage and resulting costs, but one can also set alerts & hard cost limits.
This helps to prevent unexpected cost explosions, possibly ruining a vendor, due to mistakes made during development or unusual or even mis-usage by customers.
I can not find anything like that in the Developer Console yet.
Am I missing something?
How do you manage costs? There is also no API yet as far as i know. Just take a look at the “Usage and charges” tab in the browser regularly?
Has Atlassian said anything about alerts and / or limits?
We emit platform alerts when chargeable resources reach 50,75,90 and 100% of thresholds - https://developer.atlassian.com/platform/forge/usage-alerts/. Exception for alerts is LLM and Containers as it has zero threshold and alerts for these elements are not supported as of now from the platform side.
@ChandanaMeka would it be possible to adjust this page to specify what “specific thresholds” means in this context?
To help you proactively manage your apps, the Forge platform sends email alerts when your app’s resource usage crosses specific thresholds. These alerts give you early warning as your app approaches its resource limits, so you can investigate, optimize, or upgrade before hitting a hard limit.
I cannot determine from this page whether or not this is a fixed resource allocation, or if this is something that developer can adjust somewhere in the console. If it is the latter, please also document how developer can adjust these thresholds. If it is the former, please document the max. resource thresholds that trigger these alerts.
In addition, please note that apparently this page did not surface for @PaulLunkenheimer when specifically looking for settings budget alerts. May I suggest to consider this a hint that the developer console might need some adjustment to point developers to the documentation on how to set budget alerts. This seems like a pretty important topic to me, so personally I would take this feedback very seriously.
Hi @ChandanaMeka – I also hope Atlassian is prioritizing the “hard cost limits” part of this subject. Most developers care more about the configurable limits than about the alerts, and it’s a concern we’ve repeatedly raised for quite a long time now. Who cares if Forge sends an alert while I’m sleeping if I’m already bankrupt by the time I wake up?
And it’s in Atlassian’s best interest, too, which is why I’m surprised it hasn’t been acted upon yet. If a runaway Forge spike bankrupts a developer, then Atlassian will be left holding the bag with the AWS fees on one hand and potentially angry customers on the other. That can’t be good for business.
P.S. - I understand you’re probably not personally responsible for this feature set. I’m just hoping you have the ears of the people who are.
Hi @remie, w.r.t usage alerts, we provide usage alerts at standard percentage numbers as mentioned earlier. Configurable % threshold usage based alerts are not present in the developer console yet. I have created a suggestion ticket for the same https://ecosystem.atlassian.net/browse/FRGE-2252
Usage and the Commerce APIs can be leveraged for these insights till the native solution is built
On the cost control side @AaronMorris1, we provide feature flags capability. I know developers can control in what sites should further invocations or logs as an example be encouraged or not. Could you let me know on the gaps here ?
What signals from our side would help you better setup the enforcement behaviour for your app and particular app installs on your side ? Happy to discuss this in a call as well.
We’re now two weeks out from warning messages being appended to all Connect apps fyi.
Usage caps and alerts have been solved for decades. Atlassian simply needs to copy any of the major PaaS providers.
In my opinion there are three levels that developers may want to control:
portfolio: cumulative $ usage across all apps
app: cumulative $ usage for specific app across all installs
install: total $ usage per customer instance
Those should all be in the Developer Console with API endpoints to manage them.
And there should probably be sensible defaults to these caps so that if developers never create caps/alerts they’ll be protected automatically from a cost blowout.
Also developers surely don’t care about any of these alerts except maybe the invocation errors. There’s no ability to set an alert based on $ usage…
With respect to cost/usage caps and enforcements, could you share on the expectations on what should happen for your app once you set a cost cap at dev space or app level
You don’t want the app to work across all installs of your app post the usage/cost cap ?
You prefer to own your own logic on degradation of certain type of installs post the usage/cost cap ?
If you could share what is your expectation once the limit is reached that would help.
I feel like an actual fool for doing this, but to help Atlassian understand this issue I actually logged in to our GCP billing console as well as our AWS billing console.
In firebase, I can set billing alerts with a single modal:
What we are asking, as developers, is for Atlassian to:
Allow us to set the maximum monthly budget (per app)
Sent us emails when X, Y and Z% of that budget is reached
This really isn’t that difficult and Atlassian would have been able to solve this on her own if it actually took some time to investigate how more mature PaaS operators have implemented this.
I have earlier evaluated platforms for cost based alerts and I have a decent idea on them, as mentioned earlier, it is a dependent platform limitation. I will raise this again w.r.t budget based notifications to the concerned team.
I am looking for insights on my other two questions from the community.
Yes. Remie already highlighted the biggest gap. We need a configurable threshold. Setting the threshold is much more important than setting the alert percentages. Every developer/vendor and every app has a different threshold of what is dangerously “too much”.
This is my personal wishlist…
Must have: A configurable dollar ($) amount I can enter that determines what the threshold percentages mean. This should be configurable on a per-application basis.
Nice to have:
– The current free-tier notifications in addition to the configured threshold notifications.
– Multiple configurable thresholds per app.
– Configurable threshold percentages.
– Configurable thresholds at the account (Developer Space) level.
– Configurable thresholds at the installation level.
– Configurable thresholds at the environment level.
For me, the minimum would be to automatically kill all the application’s non-production environments when the threshold is exceeded.
In practice, the threshold alerts will be sufficient for most cost problems in Production. The runaway spikes are much more likely to happen in a developer’s personal environment or a testing environment, etc. I believe new code changes and buggy automations are more dangerous than production users when it comes to resource usage.
I also understand that this would not be real-time control, and there would be latency before the usage monitor discovers a spike. That’s okay. If I set a threshold at $10,000, and the spike isn’t killed until $15,000, then that will suck, but I’ll live another day. The nightmare is when the spike isn’t killed until $50,000+ is accrued after the $10,000 threshold is exceeded.
Alerts
The minimal solution to our current challenge would be a way to configure alerts when certain budgets in USD are reached.
This is what I was looking for and what lead me to open this topic.
For a first and quick MVP, I would not need:
percentages of that budget (I can just configure those percentages as another alert by myself)
alerts about resource usage (I can use billing as an indicator for resource usage)
configuration per app or environment (I can investigate that myself, once I have been alerted about breaching a certain budget overall)
All of the above would be nice to have, but configurable budget alerts are simply necessary to have some control of the possibly bankrupting event of an extreme usage spike.
Limits
I can sleep even better (and will not have to keep somebody looking out for alerts), if we can set hard budget limits.
Preferrably, I can configure what will be shut down in the event of that limit being breached. MVP should be a choice between:
shut down everything (I would only use this option for an absolutely absurd budget limit possibly ruining our company)
shut down everything except production (keep billing down to the absolute minimum while keeping our customers happy)
I hope we see something like this soon!
We really depend on it as this is risk management 101.
I think when implementing limits, it would be great for affected vendors to have a way to react within their app.
Lets say the vendor can define a limit where the app starts running in a reduced maintenance mode where most features still work that do no inflict a high cost. To do so we would need a bridge function to check for limits or have it within the context information if the set limit is exceeded. Both per customer and global would be great to make sure that one customer may break the whole app
Idea:
If Atlassian provides a way to read the custom metric via the forge bridge, a plugin vendor would be able to increment custom metrics for their own functions that they identify to have a high risk to be costly. This way they could reduce their app features for specific customers when they identify way to much usage and inform their customers / write it into logs
