Vendor support response to slow DC review times

Hi all,

By now I’m sure this is a familiar message but I know that there has been a lot of frustration surrounding the data center app reviews, so I thought it was time for Atlassian to explain what has been happening behind the scenes (@bmagro can also offer his thoughts). Fair warning - I am verbose.

TLDR:

  • Crummy start to our review process
  • DC apps take longer than average
  • Larger interest in the DC app program than expected
  • Wasn’t reactive enough to fix the big process issues

Solutions?

  • Contractors to work full time on DC app reviews
  • Additional vendor support staff
  • Prioritizing people who have been waiting critically long (read the bottom)

Context: In addition to the standard app review where the app is installed/reviewed in a product instance, data center vendors also submit a technical checklist. This is the questionnaire that everyone fills out and submits to the DCHELP project. The technical checklist is supposed to open a dialogue between vendors and Atlassian’s product teams in hopes of discussing technical limits and solutions of developing at enterprise scale. It also helps the product team member get a better understanding of the developer community.

The vendor support team initially didn’t have anything to do with the technical portion of the DC app review. We’re not product experts (unless that product is Marketplace) so our insight into scaling apps for enterprise is limited.

What we do know is how to poke around an app and look for flaws or missing features. We also know the Marketplace.

We realized that after the technical review is finished that we still need to make sure the apps work in products. That’s when my team, Vendor support, was added to the review workflow - a few months after it started accepting applications.

We figured the product teams knew what they needed for an app to be “data center ready”, but they didn’t know enough about how Marketplace works. We had the opposite problem. Together we could use our strengths to make the DC app review process work smoothly.

So what happened?

Right away we noticed that vendors were creating data center versions of their apps before they had even started in the technical checklist portion of the review. That put us in an awkward position of having to ignore DC submissions because we were waiting on the green light from the technical review team. The DC reviews kept piling up and we kept waiting for our moment to start chiseling away, but in the meantime we still had other work to do - support tickets, non-dc app reviews, Summit, and dev work.

Eventually the technical review team started to move much quicker through the questionnaires. By the time they had reached full momentum we realized that not only had we accrued a serious backlog of issues, but that testing them was taking much longer than a standard review. On average, it takes about 45m to review an app. DC apps take 1h 10m on average, not mention setting up the testing environment which can sometimes take up to an hour.

Where we are at today?

I’m sure I don’t have to tell you that the review wait times are drastically higher than what we aim to deliver (5 business day turn around for a review). Lately even non-DC apps have been taking longer, but it stems from the same problem.

Since the DC app launch we have seen ~700 DC app review tickets come into the support queue. That’s on top of our regular queue work like support and reviews (which have grown substantially as well). Admittedly, we haven’t done enough to accommodate the increased workload.

We saw the DC backlog growing larger than our ability to handle, so we set up weeks where we would focus on doing just reviews. When that would end we would see the support queue was looking bloated, so we’d focus on that for awhile and then the reviews queue would get worse. We would juggle responsibilities but it never seemed like we were making progress. It’s caused a serious deal of frustration for ourselves but most especially for the vendors. I’d like to apologize for the delay and offer my gratitude for your patience while we’ve worked on getting to a better place.

So what are you doing to fix it?

Now I get to bring up the good news!

2 additional people were brought in to focus on the marketplace side of data center app reviews. Iryana and Olha, who you may have already worked with on the technical review, are now also helping on the other side to get things reviewed faster.

Additionally, you might have seen @ayarazarvi both in AMKTHELP tickets and in this forum. He’s a great guy and a full time hire to the vendor support team. He just started in March and has already closed 280 tickets since then!

During our FY20 planning we talked how best to tackle the growing number of support and review issues that our small team deals with. As the the workload has gone up our ability to spend time on automation has gone down, but it’s not so much a matter of having the time as being smarter about using what we have. We’re working with teams across Atlassian to create a vendor support roadmap where we have initiatives dedicated to improving the review process. One of our biggest is simplifying the data center reviews.

How can I help?
I know a lot of you are waiting on your app to be reviewed. We are working very hard to move through all of them, but some have had to wait a very long time already. Creating additional support tickets will not help, it isn’t healthy for the community and only makes getting to critical issues slower.

If you’ve had to wait longer than 2 weeks for your DC app to be reviewed, leave a comment with your AMKTHELP ticket and I will personally make sure it is prioritized.

That’s all. Thanks again for working with us during this time and for being an awesome community.

28 Likes

It’s good to hear some official feedback on the situation, thank you!

1 Like

Thanks for the update

1 Like

Thanks for the updated and open words, Jamie!

1 Like

Thank you for being transparent!

1 Like

We appreciate the update @jgaard!

1 Like

Also glad to read this!

1 Like

Question: how long does it usually take for the AMKTHELP tickets to be finally processed after the DCHELP ticket is marked “Approved” ? (In other words, how long before the DC versions appear on marketplace.atlassian.com?)

My DCHELP ticket was marked “Approved” over 1 week ago, but the associated AMKTHELP tickets have remained silent. By coincidence this week a few customers asked me what the status was with PR-Booster becoming DC approved and I don’t really know what to say.

Here are the pertinent AMKTHELP tickets:
https://ecosystem.atlassian.net/servicedesk/customer/portal/9/AMKTHELP-17424
https://ecosystem.atlassian.net/servicedesk/customer/portal/9/AMKTHELP-17425

Here is the approved DCHELP ticket:
https://ecosystem.atlassian.net/browse/DCHELP-695

It’s a Bitbucket app.

@sylvie Here’s some reference material:

These are all apps with an existing server version, the market place submission was just the DC version and in all cases with a DCHELP issue in status APPROVED.

Jira app 1:

  • submitted 19 Jun 2019 11:40 AM
  • approved 01 Jul 2019 11:27 PM
  • lead time 13 days

Jira app 2:

  • submitted 05 Jul 2019 4:27 PM
  • approved 09 Jul 2019 4:43 PM
  • lead time 4 days

Jira app 3:

  • submitted: 26 Jun 2019 4:28 PM
  • approved 16 Jul 2019 4:15 PM
  • lead time: 20 days
2 Likes

Please look into AMKTHELP-17537. The data center version of app pending review on marketplace for 3 weeks. No AMKTHELP ticket has been created yet.

Thanks

It’s pretty variable. The reason there are 2 stages are because in the DCHELP issue we are examining the questionnaire to decide if your app is designed to be be run at enterprise scale. The 2nd half - AMKTHELP ticket is then to install it into an environment and test that it actually works as expected. When an app is approved on the DCHELP side, it means we can proceed with AMKTHELP ticket but realistically it means entering into another queue where you are behind all the other apps that have had their DCHELP tickets approved.
Luckily, we have a team of people dedicated to only reviewing DC apps now. We are moving through them steadily, though it’s still true that they take longer than other apps to review and we can only do maybe 10 - 15 a week.
I’ll take a look at your issues. Let’s continue the conversation over there.

This is really useful information - thanks @jgaard ! Especially useful to know that roughly only 10-15 DCHELP --> AMKT approvals for new DC apps happen per week (and probably fewer when that week includes July 4th!).

Starting to suspect the Data Center Certification process is why Codegeist is no longer a thing: the AMKTHELP group are just completely maxed out. Codegeist woud kill them!

p.s. My DC-linked AMKTHELP tickets were processed yesterday, so it took 2 weeks after the DCHELP ticket was approved in my case.