RFC-47: Evolving the Developer Canary Program & associated Developer Assistant Apps

General Request for Comments (RFC)
1

RFCs are a way for Atlassian to share what we’re working on with our valued developer community.

This RFC is will be a bit unique. Not only are we sharing our plans about the technical solution for the future of the Developer Canary Program, we are also making an appeal and building awareness. The most important aspect of this RFC is providing your feedback on the existing program and using the Developer Assistant Apps.

Please respect our community guidelines: keep it welcoming and safe by commenting on the idea not the people (especially the author); keep it tidy by keeping on topic; empower the community by keeping comments constructive. Thanks!

Summary

We’re making changes to the Developer Canary Program (DCP) and the Developer Assistant App that manages enrollment. We’re also seeking wider participation, your advocacy, and your feedback to help us continue to expand and evolve the program and the app.

  • Publish: 25 April 2024
  • Discuss: 14 May 2024
  • Resolve: 20 May 2024

Context

DCP empowers developers to “go to the front of the line” for Atlassian’s latest APIs & newest features. The Developer Assistant App is a self-service tool giving developers control & visibility for their site’s enrollment in a “feature flag segment” intended to receive all feature flags before wide-scale roll-out. We provide this program via an app so that developers can test how their own apps perform on a consistent set of features. You can learn more about DCP at: Developer Canary Program

Problem

The proximal cause is internal changes to our feature flag systems that require us to make an update to the Developer Assistant App.

We’re taking the opportunity to rethink how the DCP fits within Atlassian. Looking to onboard more products & push more changes through the DCP. Alas, low participation via the app creates a “chicken and egg” problem where it’s hard to convince Atlassian teams they should participate.

  • ~200 installs vs ~500 enrollments. DCP was already a reboot of the earlier “Cloud First Releases initiative”, where sites were enrolled manually after requesting access in a Google Form. The Dev Asst App made it completely self-service and self-observable.
  • ~200 installs vs nearly that number of new developers in the community every month.
  • ~200 installs vs over 2000 developers using Forge every month. Using this as our metric, we’ll characterize this as 10% participation. Even as we use Forge developers as our benchmark, we know there are many non-overlapping Connect developers. And with individual developer sites, plus team environments, we estimate the actual participation is below 5%.

For us, this low participation is not sufficient to justify the continued investment in this program & the app, let alone meaningful improvements. We’re setting a long-term goal that every Forge developer would also use the Developer Assistant App. In the next quarter, we’re targeting an increase in participation to 25% and target 50% by end of the year.

We’re also setting new targets for Atlassian participation. We believe the value of a Canary program is only as good as the features that flow through it. We need a high % of features rolled-out through DCP so that external developers see most of the features before customers do. We know that more than 60% of features are flowing through DCP now but noise in our current use of feature flags makes it hard to know if the remaining flags are really for progressive deployments or other configuration purposes. You wouldn’t see our efforts here as they would involve internal education and refinement in our instrumentation.

Furthermore, we’re taking the opportunity to rethink what DCP should mean for the community. A common problem we see in community posts & developer support tickets is how feature roll-outs can make app troubleshooting confusing. We want DCP to become the standard support baseline to help isolate the effect of Atlassian feature flags on developer apps.

Proposed Solution

We’ve already spent a few weeks in refreshing this initiative, both understanding the program and app requirements so that we disrupt existing users as little as possible. We are building API first with a GraphQL endpoint that developers could access, if they had needs for “bulk enrollment”. Please let us know if this would be interesting to you. We are also “dogfooding” on Forge to learn with you about what it means to build & operate a real-world Forge app. We’ve already reported a number of new observations to our Forge team.

Following are what you can expect in a sequence that would complete over the next 2-6 weeks, depending on what we learn from this RFC and as we build.

Phase 1: Minor version update to the Developer Assistant App with built-in timebomb

We’ll release one more version of the Developer Assistant App that prompts developers to upgrade to the latest major version. Please “pardon the construction” if the new major version takes some hours before it is available. We’re working on parallel branches but release through Marketplace may not be perfectly coordinated.

Phase 2: Major version update to the Developer Assistant App with dual-state

We’ll be switching between “state management strategies”. In the current version, the source of truth for enrollment in DCP is a 3rd party feature flag management tool. We’ll be moving enrollment & integration with our feature flag tooling into a back-end service. Our new architecture will isolate the app from such back-end changes in future. The change may require a change of scopes that will bump the major version number. We’re working on mitigating this.

Phase 3: Timebomb for the legacy DCP state management

Timebomb: 1 December 2024

On this day, both versions of the app will cease to interact with the legacy DCP state management strategy.

Developers who have the old version of the app will see a prompt to upgrade to the latest version. The DCP toggle will no longer appear.

Developers who have the new version of the app will be fine.

At this time, anyone who installs the app may enroll, or return to their enrolled state by toggling the DCP state.

Phase 4: Remove the legacy DCP state management

At some point, we will update the app to remove the dead code paths that handled the legacy DCP state management. We would expect this to be a minor version and have no impact.

Asks

  • Do you value the ability to see & manage your enrollment in DCP? If so, please tell us why this is valuable to you. In the event that we had to invest in a different solution, your input about the value will help us keep what matters.
  • If you had not installed the Developer Assistant App & activated DCP, why not? Even if this is the first time hearing about Developer Assistant App & DCP, can you please try out the overall solution and let us know if it solves your problem. Is the app a good fit?
  • What can we do to help improve the solution to make it more compelling and help reach our participation goals?
  • As stated in the pre-amble, this RFC is a bit different. It’s as much asking for feedback as participation. Please share this RFC widely and help us drive awareness. With the safe transition we’ve planned, anyone can join at anytime: Developer Canary Program
3 Likes
2

Hi @ibuchanan ,
Thank you for this RFC.

I do have a background question: Is there a connection between the Developer Canary Program and EAP programs?

3

Seeing and changing the enrollment status in the Developer Assistant App is very useful for testing. I have used the enrollment switch in the past to test the compatibility of our apps with and without new features.

What would help me the most is a list of all feature flags and the ability to enable and disable them individually. In the current version of the app, users don’t see what happens when they enroll in the program unless they are looking for a specific feature that they have heard about elsewhere. I think that more transparency and customizability could help to increase the DCP participation rate.

6 Likes
4

Atlassian offers free development instances. Why shouldn’t those instances all automatically include the DCP app? This seems to fit the purpose of the app quite well… You can still let people disable it when required, but this is a better default and would surely see increased adoption.

Fwiw my own experience with the program wasn’t great. I noticed an issue with the REST API /fields/search in instances with the app enabled. I reported it in a support case. It was just ignored until a couple weeks later when the problem hit all instances.
I think you need better avenues for reporting problems when using the app. It should not be treated as an excuse to ignore problems.

10 Likes
5

Historically, not much. Under the hood both are operated by feature flags. In the lingo of feature flag tools, DCP is a segment or a target which all feature flags should use. Meanwhile, EAPs are each their own segment/target, which have to be managed separately, and rather manually.

But your question is quite insightful because we are already putting these 2 programs together in our service infrastructure. We fully expect to use the Dev Asst App as the front-end to satisfy RFC-20.

1 Like
6

Thanks for the frank feedback. I’m working very closely with developer support in this revision. I’m expecting to build education, standard operating procedures, and measurement to help dev support be more engaged with DCP. Would you mind sharing (in DM is probably best) your support issue so I can learn from it as a “case study”?

7

Fully agree on the DCPs usefulness with @klaussner, I’ve always been expecting that you’ll evolve this to the one stop place to manage all these programs, which would automatically make adoption rates higher. So given none of the motivating problems will go away, I wonder why the chicken and egg problem applies here, shouldn’t this just solve itself over time?

From that angle, both @AdamAhmed’s smart proposal and your own EAP integration plans should go a really long way already, so I’m irritated why you have to set such laudably ambitious yet almost impossible sounding goals?

Can you clarify whether those 2000 developers are only ecosystem developers, or does this include third-party in-house staff? From my experience, the latter have close to zero incentive to proactively maintain their apps against new Atlassian releases, Enterprises roll slowly after all. So other than Marketplace partners with commercial apps, I would expect them to look for and use such an app only in a reactive fashion, i.e. when a Forge based process is broken. So you could exclude them from your metrics, and voila, adoption rate significantly increased :upside_down_face:

Exemplary major version problem aware rollout plan btw. (even though that really shouldn’t matter that match given the developer audience), kudos!

3 Likes
8

More than half are in-house staff. And I appreciate the insights (reminder?) on the enterprise developers. It’s good reason to weigh your words about ambitious goals carefully. We are considering “tricks” like @AdamAhmed’s pre-install idea.

Thanks for the encouragement @sopel.

1 Like
9

We use the Canary program on a single instance across our entire team. The instance is exclusively used for regression testing for us to know when upcoming changes will break our code.

I don’t any reason for us to use canary functionality on a regular basis for anything else as it exists today. In 99% of scenarios, we don’t want to use new features and we don’t want to use new APIs. We want to get the same experience as a customer. Even without using this functionality, we are in different environments, regions, feature flag cohorts, etc.

The thing I want from this tool is to help enable support and dev activities that matter day to day for us, not when we happen to be building against a net new feature / api. This means:

  • Ability to disable all marketing in app. This means we don’t see popups / UI guides / etc. This would reduce our integration tests failure rate by 20% or even more depending on a given week.
  • Given a site URL, adopt identical feature flags. This ensures we’re reproducing customer issues like for like.
  • Ability to reset various states / configs / etc that are typically not end user controllable for whatever reason. Help us not have to contact support.
  • Access to any kind of elevated log sharing from the system for why something may be failing.
5 Likes
10

I don’t think this should be your metric. Most of these are probably customers themselves, solution partners developing one-off solutions or very small developers. Since there seem to be around 400 payouts every month (i.e. 400 vendors above $500 monthly revenue), and in most countries, you need more than that to exclusively live from Atlassian apps, I’d assume there’s less than 200 vendors in (what I would consider) your target group: Developers making a living from Atlassian apps, working on them for > 50% of their working hours. Those have a realistic chance to catch a bug being rolled out by a feature flag early, or invest enough time to have canary tests where the app is installed.

Also, don’t know where the ~200 installs is coming from, but I’m counting ~500:

Screenshot 2024-04-26 at 08.51.52
Screenshot 2024-04-26 at 08.51.521312×514 50.5 KB

Most (especially smaller) vendors focus only on one of the products. In our case, we do have Confluence apps, but mostly use the Developer Assistant for Jira app. Which is great, by the way, please continue to invest in it :slight_smile:

3 Likes
11

Reiterating this - All our new dev hires create an dev instance via https://go.atlassian.com/cloud-dev => I don’t see any reason why those dev instances should not be enrolled automatically - this would increase the adoption significantly.

Besides this, an overview of active feature flags (when possible regarding NDA topics), would be extremely helpful - or even a notification when a new feature flag has been activated. Going through the whole Jira instance only to find anything that might be different is not scalable for us, even though we’d be happy to test certain changes. Kind of a changelog inside of that app to see what to look out for.

3 Likes
12

Reiterating the importance of:

  • Having more visibility and control over the list of feature flags that hidden behind the DCP flag as mentioned by @Klaussner
  • Seamlessly integrate DCP with developer env without requiring the installation of AAP app or manual toggling on/off, as is the case with the current version as mentioned by @AdamAhmed

Additionally, it would greatly benefit us to include all new RFCs under the DCP umbrella, either by default or through their own set of feature flags, as we believe they are currently not included?

For example, we have DCP enabled on one of our testing environments for regression testing purposes, as well as on some other local instances across developers. However, none of these instances were able to capture the changes introduced by, for instance, RFC-33

Given the usual tight window for RFCs (~one month), we see lots of values of having them enabled by default on the development instances, while also allowing them to be toggled on/off individually and as needed.

13

It is good to know that this app exists. We have just learned about it. Looks like the app needs some more publicity :slight_smile:

I have just set it up on two tenants which we use for daily regression testing.

2 Likes