Hi all,
By now I’m sure this is a familiar message but I know that there has been a lot of frustration surrounding the data center app reviews, so I thought it was time for Atlassian to explain what has been happening behind the scenes (@bmagro can also offer his thoughts). Fair warning - I am verbose.
TLDR:
- Crummy start to our review process
- DC apps take longer than average
- Larger interest in the DC app program than expected
- Wasn’t reactive enough to fix the big process issues
Solutions?
- Contractors to work full time on DC app reviews
- Additional vendor support staff
- Prioritizing people who have been waiting critically long (read the bottom)
Context: In addition to the standard app review where the app is installed/reviewed in a product instance, data center vendors also submit a technical checklist. This is the questionnaire that everyone fills out and submits to the DCHELP project. The technical checklist is supposed to open a dialogue between vendors and Atlassian’s product teams in hopes of discussing technical limits and solutions of developing at enterprise scale. It also helps the product team member get a better understanding of the developer community.
The vendor support team initially didn’t have anything to do with the technical portion of the DC app review. We’re not product experts (unless that product is Marketplace) so our insight into scaling apps for enterprise is limited.
What we do know is how to poke around an app and look for flaws or missing features. We also know the Marketplace.
We realized that after the technical review is finished that we still need to make sure the apps work in products. That’s when my team, Vendor support, was added to the review workflow - a few months after it started accepting applications.
We figured the product teams knew what they needed for an app to be “data center ready”, but they didn’t know enough about how Marketplace works. We had the opposite problem. Together we could use our strengths to make the DC app review process work smoothly.
So what happened?
Right away we noticed that vendors were creating data center versions of their apps before they had even started in the technical checklist portion of the review. That put us in an awkward position of having to ignore DC submissions because we were waiting on the green light from the technical review team. The DC reviews kept piling up and we kept waiting for our moment to start chiseling away, but in the meantime we still had other work to do - support tickets, non-dc app reviews, Summit, and dev work.
Eventually the technical review team started to move much quicker through the questionnaires. By the time they had reached full momentum we realized that not only had we accrued a serious backlog of issues, but that testing them was taking much longer than a standard review. On average, it takes about 45m to review an app. DC apps take 1h 10m on average, not mention setting up the testing environment which can sometimes take up to an hour.
Where we are at today?
I’m sure I don’t have to tell you that the review wait times are drastically higher than what we aim to deliver (5 business day turn around for a review). Lately even non-DC apps have been taking longer, but it stems from the same problem.
Since the DC app launch we have seen ~700 DC app review tickets come into the support queue. That’s on top of our regular queue work like support and reviews (which have grown substantially as well). Admittedly, we haven’t done enough to accommodate the increased workload.
We saw the DC backlog growing larger than our ability to handle, so we set up weeks where we would focus on doing just reviews. When that would end we would see the support queue was looking bloated, so we’d focus on that for awhile and then the reviews queue would get worse. We would juggle responsibilities but it never seemed like we were making progress. It’s caused a serious deal of frustration for ourselves but most especially for the vendors. I’d like to apologize for the delay and offer my gratitude for your patience while we’ve worked on getting to a better place.
So what are you doing to fix it?
Now I get to bring up the good news!
2 additional people were brought in to focus on the marketplace side of data center app reviews. Iryana and Olha, who you may have already worked with on the technical review, are now also helping on the other side to get things reviewed faster.
Additionally, you might have seen @ayarazarvi both in AMKTHELP tickets and in this forum. He’s a great guy and a full time hire to the vendor support team. He just started in March and has already closed 280 tickets since then!
During our FY20 planning we talked how best to tackle the growing number of support and review issues that our small team deals with. As the the workload has gone up our ability to spend time on automation has gone down, but it’s not so much a matter of having the time as being smarter about using what we have. We’re working with teams across Atlassian to create a vendor support roadmap where we have initiatives dedicated to improving the review process. One of our biggest is simplifying the data center reviews.
How can I help?
I know a lot of you are waiting on your app to be reviewed. We are working very hard to move through all of them, but some have had to wait a very long time already. Creating additional support tickets will not help, it isn’t healthy for the community and only makes getting to critical issues slower.
If you’ve had to wait longer than 2 weeks for your DC app to be reviewed, leave a comment with your AMKTHELP ticket and I will personally make sure it is prioritized.
That’s all. Thanks again for working with us during this time and for being an awesome community.