It’s time to start being open to the idea that Forge may have been a mistake

[remie]

Disclaimer: this post is not intended to bash Atlassian, the Forge team or anyone else for that matter. The purpose is to start an honest and open conversation about the future of Forge.

This week, Atlassian published a blog post to celebrate the one year anniversary of Forge. The post included some milestones that have been achieved. Obviously, these numbers drew the attention of the Atlassian Marketplace community as they are an indication of Forge adoption.

At first, the numbers look pretty decent. However, this is a very thin veil to conceal a more distressing reality.

The number of apps created seems impressive, until you look at a different number shown in one of the talks at Atlassian Dev Day ‘22 (BONUS TALK - Forge on Forge: How the Forge team builds Forge apps at Atlassian)

Out of the 5600 apps created, 2844 were created by Atlassian. That is a little over 50%. In addition, out of all these apps created by Atlassian the [..] conversion from forge create app to anything useful is ~ 6% even in house. That leaves us with 2756 apps that were created by external developers.

Another number that is interesting to Atlassian Marketplace Partners is the number of apps available on the Atlassian Marketplace. The blog post mentions 200+ apps available on the Atlassian Marketplace. Let’s dig a little deeper into this number.

The total number of Forge apps published to the Atlassian Marketplace on July 12 was 239. Out of those 239, there are 6 apps that are created by Atlassian. So that leaves us with 233 apps created by Atlassian Marketplace Partners. Out of 2756 Forge apps created by external developers, only 233 are published on the Atlassian Marketplace (~8.5%).

117 apps are free apps, 116 are paid
The 116 paid apps have been installed into 10,081 instances, reaching 8,613,148 users*
Out of 116 paid apps, 94 have less than 100 installs

Forge has been in development by Atlassian since early ‘19, with the first announcement at AtlasCamp‘19 in Vienna. Atlassian has been pouring resources into this framework ever since, and Atlassian Marketplace Partners have been as well. It is not a stretch to say that diverting those resources has resulted in less investments in improvements of both Atlassian products and Atlassian Marketplace Partner apps.

With roughly 3 years of development and being GA for one of those, looking at the actual adoption of the Forge platform is reason for concern. Especially given the amount of goodwill from the Atlassian Marketplace Partner community. Most Atlassian Marketplace Partners are very eager to work with Forge, spending lots of developer time on it, giving feedback, participating in Codegeist, joining calls with Atlassian, contributing to Dev Day ‘22 and publishing the apps to the Atlassian Marketplace.

So here is my take on what may have contributed to the current state in terms of adoption, what is left of the initial goals and why doubling down on Forge is not the most obvious way forward.

How did we get at the current state

This is not an exhaustive list, and there are many more reasons why adoption may have been slow, but I think in terms of discussions on CDAC these are a few topics that have been popping up most frequently.

Forge UI versus Custom UI
Initially, Forge shipped with Forge UI support only. This is a limited set of server-side rendered front-end components using a similar syntax as React/JSX. Although it allows for rapid prototyping, Atlassian Marketplace Partners soon discovered that, apart from the required investment into a different UI library, Forge UI also could not meet their standards for creating appealing user interfaces.

It took a while until Custom UI became available in November ‘20 (⚡️ Introducing Custom UI: Static HTML/CSS/JS frontends on Forge) and even longer for it to support more advanced features like routing, code reusability, etc.

Forge Storage API
Having the ability to store data within the Atlassian infrastructure has been a popular request ever since the implementation of GDPR. This is an important asset for Forge, but unfortunately also proved to be a huge blocker for most app developers. The initial size limit of 32kb, API limitations with regards to querying data, overall performance and quotas meant that either a significant engineering investment was required or the app would require alternative storage outside of Atlassian.

Lack of parity with Connect
Up until recently, the number of integration points for Forge apps have been very limited. Important products like JSM were out of scope even at GA, and popular integration points that many developers are using in Connect were unsupported. Getting Connect parity in terms of integration points has been coming in waves, with priority choices that were not always clear or relatable to the Atlassian Marketplace Partner community. A lot of the default building blocks that are part of almost all Connect apps just weren’t available to Forge for a long time.

Unrealistic quotas
Up until this year, most of the quota’s for Forge did not make sense with regard to the needs of large and complex apps. A execution timeout of 10s for functions, a max package size of 256MB and 32KB of storage per entity just simply isn’t enough. It’s fine for Hello World playground type of projects, but the amount of engineering required to support complex apps within these boundaries made it unattractive to invest in Forge when developing any serious app for the Atlassian Marketplace.

Overall developer experience
In short, the developer experience has not been great. The lack of multi-ownership, an unstable CLI with random errors, the granular scopes debacle, lack of communications of changes (like the fact that Forge Storage API file size limitations were increased), the lack of ability to migrate Connect apps to Forge on the Atlassian Marketplace, slow performance and priorities that are alienating to the Atlassian Marketplace Partner community have resulted in a lot of frustration and friction, preventing mass adoption.

What is left of the initial goals

When Forge was introduced, there were a couple of important pillars that have been repeated over and over again as a justification for the platform and as a reason why Connect will eventually be deprecated. Now that we are 3 years in, let’s look at what is left of these goals.

Here is how it was announced:

The Forge platform is a reimagining of cloud app development for our ecosystem, composed of three advanced components:

1. A serverless Functions-as-a-Service (FaaS) hosted platform with Atlassian-operated compute and storage for app developers
2. A flexible declarative UI language (Forge UI) that enables developers to build interactive user experiences across web and devices with just a few lines of code
3. A state-of-the-art DevOps toolchain, powered by the intuitive Forge Command Line Interface (CLI).

[…]

Forge helps solve for a whole host of cloud considerations, including:

1. Trust: As personal information goes digital, privacy, transparency, and security are more important than ever before. With Forge UI, developers and app consumers benefit from built-in, best-in-class security for apps, by default. Plus, thanks to Atlaskit, when Atlassian makes an update, your apps won’t break.
2. Running anywhere: Atlassian customers want experiences that are consistent across their products and apps, and across their devices and web. Forge’s UI enables users to build once for both web and mobile.
3. Moving up the stack: In general, developers aren’t concerned with where their code is running – they simply want to spend less time on implementation details of the code, and more time on providing customer value. Forge’s serverless FaaS model enables developers to write single functions rather than building entire web applications.

It is often great to look back at what was initially conceived and how it panned out. Just looking at this quote you can already see that the focus started to shift quickly after Forge was introduced. For instance, one of the things that became obvious was that Atlassian-operated compute and storage was not going to work out because Atlassian would not be providing every type of solution that app developers might need. The same applies to Forge UI. Support for mobile is still in the works and as an Atlassian Marketplace Partner it is unclear how this will work and whether it will fulfill the promise of “build once for both web and mobile”. The DX of both the CLI and overall stack shows that at this point, Forge is actually a greater investment for Atlassian Marketplace Partners. One year after GA the platform is not delivering on any these promises.

But more importantly, let’s do a deep dive into the biggest selling point of Forge: trust / security. Here are the main argument for using Forge over Connect with regard to trust / security:

It runs on Atlassian Infrastructure
As previously mentioned, this is not entirely true. Atlassian will not be able to, and does not have to, provide all the services required for creating complex apps. This means that a lot of Forge apps will continue to use 3rd party services and transfer data outside of Atlassian Infrastructure for processing.

Due to the limitations of Forge, a lot of Atlassian Marketplace Partners will require external storage (database), compute (for specific use cases) and even means to contact the Atlassian API externally (which will be made available as part of the Connect on Forge plan).

This also means that a lot of Atlassian Marketplace Partners will still need to be considered data processors separate from Atlassian, and security audits will still be required for most customers.

Forge users are more informed about data egress & permissions
Yes, this is true. Forge apps are required to list all external services to which data will be shared and when granular scopes are implemented this will allow for more detailed control over permissions. But that is not specific to Forge. This could have been implemented for Connect as well.

Forge has out-of-the-box support for data residency
Yes, but… Atlassian Marketplace Partners are more than happy to implement support for data residency in Connect except are currently blocked by Atlassian due to stalled implementation in Connect.

Forge has better authentication / The current JWT / iframe solution of Connect is outdated
There is no reason why Connect cannot switch to industry standards for authentication, like OAuth. Using iframes is not per definition a security problem, a lot of other SaaS products that support 3rd party integrations use iframes. This is the industry standard for in-app extensions. Just like having a bridge between the iframe and host product. If security concerns for the current Connect implementation are the main driver, creating the Forge infrastructure was and is not the only option available.

Why doubling down on Forge is not the most obvious way forward

With Atlassian doubling down on Forge, there also comes a real incentive to artificially beef up the numbers of apps. This is a path that Atlassian is already taking with the Connect on Forge initiative, which is meant to create the ability for Atlassian Marketplace Partners to migrate existing Connect apps to Forge.

It is becoming increasingly clear that the goal of moving Atlassian Marketplace Partners to Forge comes at a great cost with regard to the initial plans of Forge.

By accepting the fact that complex apps will need data egress, compute, storage and API access outside of Forge, the goal to address the security concerns that Forge was meant to resolve is watered down.

Another way that Atlassian is doubling down on Forge is by deprecating Connect features and making them Forge only, by not adding support for Connect for new products (like Atlas and Compass) and by only making new integrations to existing products available on Forge. Unfortunately, due to all the previously mentioned frictions, this currently only means that either existing apps lose functionality, cannot make use of new functionality and that the new functionality will probably not be used due to lack of Forge adoption. There is no audience for what Atlassian is building.

While there are many great use cases for Forge, and there is definitely a place for Forge within the Atlassian Ecosystem, doubling down on Forge as a replacement for Connect may not be the best decision.

• It erodes the initial goals of Forge
• It creates friction & frustration within the Atlassian Marketplace Partner community
• It distracts developers from delivering value to customers

Instead of spending a lot of time and effort in trying to increase Forge adoption by artificially pretending that Connect apps are now “on Forge”, Atlassian can also decide to spent that engineering time on improving security of Connect and leave that platform available to Atlassian Marketplace Partners for the more advanced features.

I really hope that now that we have celebrated the one year anniversary of Forge and are preparing to wind down during the well deserved holiday season, there is also room to reflect on the decision that have been made, the impact that they have had on both Atlassian and the Atlassian Marketplace Partner community and to reconsider the path forward.

——
*) the total number of installs & users is skewed by the fact that some forge apps also have Server/DC deployments and the distribution numbers are combined for all platforms

[remie]

The Forge app numbers on the Atlassian Marketplace as mentioned in the post have been taken on July 12th 2022 from public resources like the documented Marketplace API and the undocumented Atlassian GraphQL API that powers the listing pages on the Atlassian Marketplace.

The forge apps were identified by the fact that their Cloud deployment type is MarketplaceCloudAppDeployment instead of MarketplaceConnectAppDeployment.

For those interested, the list with some of the properties from the API is in this CSV file (with txt extension, because csv is not allowed on CDAC):

2022-07-12 - List of Forge apps.txt (24.5 KB)

[jack]

I love the following data mentioned in this post:

With roughly 3 years of development
116 are paid
Out of 116 paid apps, 94 have less than 100 installs

And for this:

@remie

The initial size limit of 32kb, API limitations with regards to querying data, overall performance and quotas meant that either a significant engineering investment was required or the app would require alternative storage outside of Atlassian.

It is the main blocker for us to migrate to Forge. We store tens of Gb of customer data, and we need to query it within milliseconds. It is something that we will never get from Atlassian I guess.

Well written @remie

[boris]

I’ve tried to write this 4 separate times and I get stuck each time because I feel like Atlassian will misunderstand anything I write. If you’re an Atlassian, please read this as if it’s not talking about your baby of a project. This isn’t a personal attack on you, it’s a critique of systemic failures of a multi billion dollar organization that hasn’t improved in these areas in a literal decade.

• I disagree with Remie and firmly believe Forge is the right path forward for Atlassian and for Atlassian customers
  • SFDC and Force.com prove the point
• I believed in Forge so much that I delayed building NAFJ Cloud for a year. That was a mistake. We trusted Atlassian to ship more, better, faster. This cost us a lot of money in opportunity cost. 3 years later, it’s still not even close.
• I continue to believe in Forge, so we try and build on Forge whenever we do anything new. This has led to tens of thousands of dollars in wasted dev hours. The comments from Joe’s Dev Day talk about the Compass team hitting the same pain points internally seem to show that it’s not only us. Forge today is still a child. It needs to be treated like a child. It will eventually grow into a mature platform. It’s nowhere near there.
• Prioritization in Atlassian as a whole is a mess. But since Forge is what we care about here, the Allow Access thing is still there 3 years later. Vendors flagged this on day 1. You’re shipping new features that surpass what Connect can do, but the core UX of Forge is broken. 100% of our support tickets for Forge apps are end users being stuck on this UX. God knows how much of our churn rate is caused by this.
• Huge numbers of apps being created, but few being published? It’s nothing complex, it’s the lack of any coherent devex for multiple devs on a single project. 5 devs? 6 apps registered. This is why the numbers look so weird for Forge.
• Communication around Forge has also been what I term … Atlassian quality, or what I expect out of a child. Forge went GA with no backups? Comms to vendors about Forge being the future have led to the above waste I mentioned. These are not small issues. Atlassian is causing significant harm by continuing to erode vendor trust in a time when the partnership relationship satisfaction has hit an all time low.
• $0.02 from my solution partner hat, one of our Cloud Enterprise customers couldn’t believe how useless Forge was to them given the limits on it. They’re currently planning on using Connect.

I want to re-iterate here that I still believe in Forge, I will continue to believe in Forge long term, and that I want the Forge team to be successful. But please think of what you’re doing in the ecosystem. The things you say, the ways you say them, and how that actually maps back to the actions taken isn’t trending in the right direction.

[Johnny]

Thanks, Remie, for your OCNBS and for representing the interests of our developer and partner community. This post echoes many of the important conversations and strategy decisions we are in the process of making internally.

We started on the Forge journey in 2019, and we have learned a tremendous amount from our developers and partners, large and small, and we have more data to inform our decisions today than we did at that point in time. Shifts in our product delivery that you have picked up on are a reflection of how we have adapted to that evolved understanding. We acknowledge cloud app development is a complex topic without a one-size-fits all solution, which is why shades of gray are needed as we work together to find the best solution for serving our customers, long-term, as they continue to adopt cloud.

We are proud of the progress we have made with Forge and how it has lowered the barrier for new developers to join the ecosystem. At the same time, we know we are on a multi-year journey to serve our existing Marketplace Partners who have mature apps in cloud better with our platform capabilities.

All of that being said, I know I have not offered a clear message or a solution to the challenges you’ve raised. I would love to propose we start to evolve how we are working with interested Connect developers, finding a way to bring you into our thought process earlier, and ensuring that we are clear in both the Forge<>Connect strategy and in how we communicate about it with you.

Johnny Ferguson
Head of Product Management - Atlassian Ecosystem

[HeyJoe]

Hey @remie

Just offering a note on the number of apps creating by Atlassians vs. external developers: 5600+ refers to the number of apps built by customers and partners. The 2,844 apps built by Atlassians I mentioned in my talk are in addition to that number, not a subset. While this may seem like a very large number in proportion to the Forge apps listed in the Marketplace, it is fairly consistent with what we’ve seen in server and Connect. Private apps and apps built for experimentation and learning far outnumber those that enter the Marketplace as a commercial product.

Cheers,
Joe

[remie]

Elton John - Sorry Seems To Be The Hardest Word

[remie]

Thank you @Johnny for acknowledging that there are indeed internal discussions regarding the future of Forge & Connect. It might seem obvious to you, but a lot of the things you might be talking about internally do not even reach the inner circle of Atlassian Marketplace Partners. At least not publicly.

Atlassian created an Ecosystem by opening up their products to apps (first on Server, then on Cloud). That Ecosystem thrived and created hundreds, or even thousands of jobs for app developers. There are business built solely on top of your Ecosystem. Families are depending on it.

With great power comes great responsibility
Voltaire

Whatever happens next with Forge or Connect, one of the first things Atlassian should be doing is spent time healing the wounds it created.

You might come a long way by now also acknowledging that this great learning path you were on with Forge had (unintended) consequences for the Atlassian Marketplace Partner community (as also detailed in the post by @boris), and maybe, perhaps, even consider an apology.

[george1]

@remie thanks for bringing this up. Here are my 5 cents: Atlassian as a FaaS provider doesn’t seem right to me conceptually. There are Cloud providers (AWS, Azure, GCloud) and other FaaS players that are optimized for such use cases and offer unique capabilities. With full respect, I think Atlassian took an overly ambitious route with Forge and entered the race with other providers with low chances of success (feature-parity, speed of innovation, ecosystem of integrations, etc.). Same with storage, same with DevOps – other best-in-breed tools are way ahead, why would you compete with them?

Don’t get me wrong, I like the idea, perhaps it is a good solution for starter apps so devs don’t have to worry about the infrastructure to make it easier. However, not a good fit for complex products in my opinion. Connect gives the flexibility to build any kind of app and leverages a well-known platform – HTTP & HTML so the vendor can decide what is the best suited technology for the backend and frontend. HTML runs on mobile too, btw.

Customers need more privacy and security? Build a more granular access scopes for API, introduce more transparency on what data is consumed by apps (GDPR reporting is a great example), add more controls to govern data flows, and so on. Why do you need to build a whole compute and storage infrastructure for this? (well, it is a rhetoric question after a few years now )

At the end of the day, my biggest concern is the future of Connect and more importantly the flexibility that comes with it. Do you want awesome products in your ecosystem? Give us the room to innovate.

[MichalVrany]

My 2c,
as a former atlassian developer I would say that it’s apparent that atlassian decided 3-4 years ago to “close” the platform for themself… which is quite obvious to me after the server closure and the forge platform and now they want to take another form of freedom from the ecosystem (connect apps).

It’s a reason why I stopped 1.5 years ago being atlassian developer and currently developing spring boot apps for insurance company…

[RomanStoffel]

My 2 cents: For small apps forge apps is fine, even awesome. Especially if you want to create a ‘one-off’ small app to scratch your own need, which you do not want to baby-sit or sell.

However, for more complicated apps Forge adds headaches,
while the ‘benefits’ get smaller. Here are some of my head-aches.

• We were surprised that a Forge apps which did a lot of request to the Atlassian endpoint ran slower deployed on forge than locally. We expected that the ‘co-locating’ the app reduces the network latency and therefore speeding up the app.
• Maybe it is an issue with our app: However, here the ‘closed platform / re-invent your own wheel’
  starts to bite. All the standard performance tooling we would use does not work with Forge.
  Usually we would run some profiler / system monitoring along with our app to point out the issue.
  For now, we just ‘ignored’ the issue =)
• If you have a complex app, you very likely end up with your own backed. A that point ‘Forge’ becomes UI/Frontend framework, where everything is a bit more cumbersome.
• For our ‘biggest’ cloud apps, I don’t see that we ever rewrite them to Forge as is. Even if they fit ‘invoke-a-function’ style. Biggest blocker there is that its not worth rewriting battle-hardened
  code from the JVM (or other platforms) to NodeJS.

Anyway, back to the future. Forge is here. Here are my hopes for its future, some realistic,
some are pipe-dreams:

• Keeping the data hosted by Atlassian is great. Giving away part of the burden of the data security is great.
  • However, the data stores are rudimentary at the moment.
  • I wished for some raw AWS data stores (RDS, etc).
    I do not want yet another ‘data store’ API, but rather the ‘standard AWS’ stuff, ‘secured by Atlassian ™’.
  • A ‘data debug’ story. Ideally I as a vendor can not access the clients data by default.
    However, if there is a rare bug in the data, it would be great to to allow the customer to ‘opt-in’ to allow access the data. Somewhat similar to the logs.
• Mix of Connect with Forge: Afaik that is work in progress
  • Like, we would like the have the Forge UI kit capabilities of the Forge Macros,
    but keep the actual macro a Connect IFrame.
• More guides on tooling:
  • How do I get profiling data our of production instances?
    Can maybe run perf or something like that? Or maybe some nodejs profiler?
• Long term I wish for a future where your function can be shipped as ‘any Docker/tar.gz/what-ever’ format, and you can choose your runtime. This would allow us for example to ship existing tested JVM code.
  I was surprised to learn that the network security is actually implemented in the custom NodeJS runtime, so this is not something going to happen soon.
• Tooling to run ‘Forge’ without Forge. Or document the protocols so well you can do it yourself.
  • An example, something we did for our Connect apps: We needed to do some load testing.
    Since Connect is protocol based, we wrote a load test which does the Connect HTTP
    requests for us. This way we could generate a lot of requests, without running
    actually calling through the Atlassian products.
  • Same goes for testing. We easily can check the output of a Connect app against
    a reference, because we can directly call the Connect app.
    Not sure how I would do that with Forge.

I probably forgot some wishes / head-aches. As most of my developer time goes to our existing Connect apps.

[PabloBeltran]

I am feeling this too. In fact, we’ve put our eye on alternative Marketplaces and our last and by far our best app (launch within a few months) has been built with portability to other platforms in mind as a top priority in design. I see a lot of business risks in developing apps just for Atlassian. And Forge, definitely does not help to change this.

[ademoss]

Hey @Johnny - You were talking about clearer communications and all that, but then kinda disappeared for the past 2 months. Have you abandoned this thread, or will there be more communications from you/your team?

[remie]

On the off chance that this post might actually become the 5th Great Topic on CDAC, would that encourage Atlassian to give a more meaningful response?

[AliRaza1]

Ouch

[amoerchen]

To extend: Atlassian forgot about small teams relying on server apps… they basically throw us out of the market… DC certification is really hard, new knowledge needed, a lot of resources needed. Support, not really existent…

Migration to cloud? Another very high invest, as this is a totally different stack, with way less feature (it isn’t really possible to build the same apps). Not only you need a lot of new knowledge/resources too, you even need to become a web hosting company (security, privacy,… )

[eagle.xiao]

8 months later. No meaningful communication and no meaningful improvement.

I am starting a new application. The “allow access thing” is a deal breaker. That is core UX!

However, I really want to context menu which is only available on Forge.

[EdgarsBatna]

Yes, it’s like code smell, just so bad that it already smells on the outside. Is Forge just another platform that will be shot down once Atlassian decides that they could make more money by forcefully monopolizing their own product with active customers? The fact that they say “Forge will remain free through 2023” is partly supporting the notion here. If you jump on it, you will end up on the hook just like many other users once, all of a sudden, you’ll pay per user per API request. If you don’t jump on it, then you will constantly feel like you’re missing out.

Too much revenue optimization. Just give us some planning security… It’s not too hard to make money with the products Atlassian offers without jumping ship every few minutes. If only Atlassian worked on actual feature requests that have been open for years and years…

[KCWong]

The fact that they say “Forge will remain free through 2023” is partly supporting the notion here

Yes, you only need to take one look at Automation to realize that.