as there have been pricing changes announced today, I checked the usage on our first Connect on Forge apps (one of the smaller ones we have) and I’m a bit concerned regarding log usage.
To make sure we already have a fresh app token available we have a remote function that is being invoked every five minutes. This follows the pattern described in the RFC (still unreleased but planned).
Currently the app has around 1200 installations, whereas all our apps combined (currently all moving to Connect on Forge) have at least 10x this combined.
Now, we are already seeing massive log usage, even though these seem to be system logs of Forge for the single, five minute scheduled trigger, which we cannot turn off, nor configure. I’m just wondering if there are any plans to address this before January, because if every remote invocation of our app will be logged in Forge, this will create endless, unnecessary logs which we do not need… It would be unacceptable if we’d need to pay hundreds or thousand dollars per month for logs we do not use.
We have plans to provide resource usage controls at the app level to developers. This would aid developers in configuring what the maximum resource usage is that they would like the app to consume in a month, and beyond that, further resource usage will not be allowed. We are yet to finalise the experience side of things. Would this solve the problem of unexpected costs of an app resource in this example - logs?
I do not know about Tobias, but I do not believe this would be an acceptable resolution. Developers may very well want to log things, as well as to have access to said logs, so turning logging off after a limit would not be acceptable for this particular problem.
What developers do not want to do is to pay massive sums for obligatory log lines that are generated by Atlassian and which are of no use to developers.
Two solutions would be:
exclude all Atlassian-generated logging from the quota (which seems the most fair), or
allow all Atlassian-generated logging to be disabled
Logs are an extremely important part of the ability to provide support, detect (malicious) abuse and ensure reliability and stability of a product.
Partners should not be encouraged to disable logs or penalized for trying to provide exceptional service, especially not when we are forced to pay for entries made on our behalf over which we do not have control.
Thanks for raising @tobias.viehweger, this is exactly the type of feedback we’re looking to get over the coming months.
I’ll have a chat with the team about what we can do with these system logs. It’s certainly not our intention to make you pay for logs you don’t want or need.
Echo others: logs should be free. Just auto-delete after 90 days or something.
Debugging marketplace apps is a blackbox exercise. We developers obviously can’t access customer instances or data.
So if a customer reports an issue which you cannot replicate in dev then your only option is either logging in prod or awkwardly asking the customer to send console logs.
Additionally there should be a Forge package and effort to negate the need for external logging tools like Sentry.
Just exploring if there is a possibility to control log levels for forge logging. Say my forge app uses console.debug, console.info, console.error, console.warn etc -
Is there way to tell d.a.c forge logging to log only warning and errors to reduce the log volume.
If investigating an issues I want to set a different log level, if possible for a particular customer instance.
Edit: needless to say that I don’t want to deploy a new version of the app to changing log levels.
I do not know about Tobias, but I do not believe this would be an acceptable resolution. Developers may very well want to log things, as well as to have access to said logs, so turning logging off after a limit would not be acceptable for this particular problem.
What developers do not want to do is to pay massive sums for obligatory log lines that are generated by Atlassian and which are of no use to developers.
Two solutions would be:
exclude all Atlassian-generated logging from the quota (which seems the most fair), or
allow all Atlassian-generated logging to be disabled
Yes, that sounds good actually, as others mentioned almost like log-level / log-group like configurations. I a dream world these would also be togglable by an API, so we could turn on more granular Forge debug logs (like these shown in my screenshot), on a case by case basis for troubleshooting. One can dream
Thanks @tobias.viehweger for raising this. We have the same issue: > 100 GB of logs with 0 Forge functions and 0 Logs that we log ourselves.
We can’t even see the logs at the moment due to ECO-929, but good to know that it’s probably due to our scheduledTrigger.
Can I suggest there is a page in the Developer Console where it is possible to disable access of a particular site when it generates excessive loads/logs? Ideally, it can be toggled either manually or automatically when a configured threshold is exceeded. And notifications can be sent to both the marketplace vendor as well as the technical contacts for the Cloud site.
This will help to avoid disruption to other sites when 1 site goes haywire.
Otherwise the entire app mayhave to be shut down to avoid bill shock.
Got an additional and related questions about log pricing. Since some customers can disable log access for the vendor, will we still get charged for logs that we cannot access?
Currently, even if customers disable log access, Atlassian continues to ingest the logs into the system. This is due to the possibility that customers may encounter an issue and temporarily grant log access to developers. After resolving the issue, customers subsequently disable access.
To avoid charging developers, we would need to cease log ingestion, which could pose challenges for debugging for enterprise customers issues in the future. There is a trade-off involved.
Please do share how frequently the customer issue scenario occurs with you and how the community feels about stopping log ingestion for disabled access.
Thanks for your reply and explanation. I understand that point of view.
Logs are valuable to us. They can give us an insight into app usage, can highlight errors and warnings early and sometimes even give us the chance to proactively talk to a customer about a potential problem. And of course they help troubleshooting during support.
What prompted my question is a scenario where a customer suddenly starts producing a lot of log volume, with us having no way to access these logs. We would still pay for those logs and with the way usage is distributed, an individual customer might make up a very large part of overall log volume and thus a sizeable chunk of the costs.
Roughly 11-12% of our customers opt-out of log sharing – whether that is a lot or very little compared to the overall Forge customer base I cannot say. But that number is pretty consistent across our products. In my mind, it’s not a huge number.
So yes, I would feel positive about log ingestion being stopped for disabled access. But I can see how there’s extra effort for Atlassian to implement this. And I can understand that some customers might like to have access to the logs, but not wanting to share them. I guess it comes down to a question of who owns the logs.
It seems reasonable to cease log ingestion for customers who have disabled logs (or at least make this behavior configurable by the vendor). If customers want help, they would need to re-enable their logs and reproduce the issue again.
The problem with any alternative is that, if a customer disables log access but the logging process runs haywire and generates tons of logs for that particular customer, the vendor cannot even tell what needs to be fixed in order to stop the customer’s instance from generating so much log data (so the vendor still has to pay for it until the end of time, with no way to stop it).
The problem with usage based Forge pricing is that app revenues are seat based, while costs are usage based. Vendors have very limited insight into usage per specific seat, and so can’t throttle seats which use disproportionately high usage.
In the end that can lead to spiked cloud bills from an app when e.g. a customer sets an instance public and at once hundreds of thousands of (anonymous, unpaid) users come in.
That can bankrupt an app vendor who e.g. sold a video embedding app to a 10 person PR agency, which then embeds one public page with the video embedding app in their Shakira concert page. Boom, you go from $50/day cost to $50,000/day cost.
How will Atlassian handle such spikes? Given the structure of Forge, Atlassian should shoulder the risk for this.
Thanks for the inputs @osiebenmarck and @scott.dudley, very helpful. I will get back soon on an update about how we will go ahead with instances that have disabled log access.
@marc and others, thanks for raising a valid concern of noisy customer scenario. Remie also raised this point earlier A preview of future Forge pricing - #26 by remie. We are brainstorming on different possibilities.
How do partners handle this in Connect side ? I would like to learn from different strategies adopted by partners in the Connect world.
Currently in Connect, our hosting costs are fixed. We have a fixed number of VMs running our app. In some sense this is overprovisioned, and we can handle spikes in traffic quite well. In case traffic gets extremely high, latency gets larger and we get an alert, and in theory could provision more VMs rapidly (never needed this since 2019). However we are in control of the cost.
Our hosting provider will also queue up requests on their load balancer (up to a limit) if our VM’s get overwhelmed (also has never been necessary).
Our database costs are also fixed, we use a managed Postgres instance.
Logging costs are also on a fixed plan. We did have occurrences when this plan was not sufficient. These cases where caused by some Confluence instances hammering our app due to automations/backups going wild. We did solve this by contacting Atlassian support and dropping affected logs. Unfortunately this seems not possible in Forge.
The mistake was building Forge on top of AWS which charges for egress, and has the cold start problem with Lambda which is why all Forge invocations are visibly slow. They also now charge for that cold start time and broadly have much higher prices than other providers.
Our Connect apps run on Google Cloud. Generally speaking, for us, logging is cheap enough that we do not have to worry about it. Based on a quick search, it appears Atlassian is charging about 2x of what the big Cloud providers charge.
Quick example for 50GB of logs (screenshots below): Atlassian: $49.24 (1 GB/month free tier) AWS: $25.23 (5GB/month free tier) Google Cloud: $0.50 (50 GiB/project/month free tier - without free tier ~$25)
This makes me think that it may be cheaper and more future-proof for us to build a log solution that directly exports them via the analytics endpoint, rather than logging them through the Forge platform. The customer can still disable analytics and choose to suspend log exports in this way. If they decided to disable analytics, they would also not incur additional logging costs for us.
There’s the cost factor that others have already pointed and that @tbinna did a good calculation for. But I want to add another aspect, that’s not been as prominent here so far and that’s risk.
With all usage-based pricing, there’s the risk of suddenly increased usage or unexpected usage pattern leading to huge spikes in costs. I’m sure everyone has at least seen one or two memes by now of people waking up to amazingly large cloud bills. This can of course happen in Connect as well – there’s always some infra somewhere.
The difference with Forge is, that we as vendors have less control over what’s driving the costs. Staying with the logs example: If a customer has a sudden spike in log volume and they didn’t share logs with us, we would have no way of finding out until they show up in the top 20 list. (Or do we? Let me know if that’s the case.)
And once we’ve identified who is responsible for that spike in log volume – there’s little we can do. We could reach out to the customer – they might react or they might not.
So in my nightmare scenario, we’d be sitting there, seeing our bill climb, or customer oblivious or unwilling to care and we would have to reach out to Atlassian for help. Because we have no way to investigate, to interfere or to make it stop. If the customer is on an active license, would Atlassian be willing to cut off the logs for them because the vendor says so? (Or uninstall/disable a licensed app if the spike is yet another resource like storage?)
The same general risk exists for Connect apps, but we can have an acceptable use policy in our T&C and we can enforce it on our infrastructure, because we own it. We cannot enforce it the same way on Forge, even if we have the same T&C. So we have better mitigation options with Connect.
In the end, vendors will do some risk calculations and include that in their architectural choices and prices.
