[RESOLVED] RFC-3: Enhanced reports visualisation

General Request for Comments (RFC)
1

RFCs are a way for Atlassian to share what we’re working on with our valued developer community. Please respect our community guidelines: keep it welcoming and safe by commenting on the idea not the people (especially the author); keep it tidy by keeping on topic; empower the community by keeping comments constructive. Thanks!

Summary of Project:

In May 2022, we attempted to improve the partner reports experience through a redesign that was initially rolled out to a small group of partners (5%). Shortly thereafter, we started receiving feedback on the redesign. Partners were not pleased with the new experience. Thus, in order to minimise the detrimental impact on report usage and partner productivity; we consciously decided to completely roll back the redesign.

This update provides an overview of the next steps and the future of reports.

  • Author: 7 Feb 2023
  • Publish: 9 Feb 2023
  • Discuss: 24 Feb 2023
  • Resolve: 10 March 2023

Problem

Let’s start with first understanding the rationale behind the redesign; its potential impact, and importance in the hindsight.

Why Redesign?

At Atlassian, we are upgrading our tech stack to build a reliable and robust foundation for the marketplace partner portal. This will help us enable,

  • The implementation of the “Atlassian Design System” to ensure uniformity and consistency in all the marketplace touch points.
  • Enhanced analytics and instrumentation capabilities to facilitate faster learning and the creation of tailored user experiences.
  • Additionally, improvement in the overall performance and page loading speed.

What does this mean for partners ?

  • Consistent partner experience
  • Speedy experiential enhancements and delivery of newer capabilities.
  • More insights and functionalities going forward (Search words/benchmarking insights)

Proposed Solution

Upon careful examination of the current experience (control), the feedback we received, and a comparison of control vs. the initial redesign (V1), it appears that in the first launch, we missed…

  • Giving sufficient emphasis to the opt-in onboarding experience
  • Retaining existing functionalities - Navigation hierarchy, filter multi-selection, contextual viewing & comparing more details of the transaction/licenses/feedback details reports.
  • Retaining information parity, usage behaviour & experiential models.
  • Accessibility concerns of the charts - Dates, color usage, etc.

along with other issues.

Going forward

In response to the ongoing feedback and your concerns from the last rollout, we have redesigned reports with a fresh look. The new design updates the UI to be more consistent with other Atlassian products, while still maintaining the familiarity of the existing workflows and usage behaviours of partners.

Introducing the new reports experience:

Image1
Image12714×4096 468 KB

Here are some of the key highlights of the new design:

  1. Onboarding experience:
    We have designed an intuitive onboarding experience to help partners smoothly transition to the new experience. During the trial period, partners can switch back and forth between the two at any time.

  2. End-to-End interface:
    Partners now have access to the entire screen to view the insights due to the end-to-end interface.

  3. Hierarchy in navigation and page structure:
    Retaining and enhancing the information hierarchy of the page layout.

    Image2
    Image24032×1657 284 KB

  4. Retaining & enhancing the filter usage & functionalities:
    a. In the first launch, we missed the multi-selection functionality of filters. This time we are being thorough with the functionality and experiential behaviours.
    b. We are bringing out “App & Hosting type” out of the more filters bucket. The rationale here is to make the most commonly used filters easier to find and use. We will update the popular filters in accordance with the partner usage data in future.

    Image3
    Image32560×1960 291 KB

  5. Progressive disclosure of information:
    Partners can learn more about a specific license/transaction using an accordion(expand/collapse) interaction, similar to the current workflow. By clicking on “More details”, they can access all the relevant information about the lic

    Image4
    Image42560×1960 266 KB

    ense/transaction in a new tab.

  6. More clarity & structure to charts:
    Introduction of titles into the charts aids in boosting the clarity of the data and its readability. Including legends at the bottom of the charts for visual reference. Overall visual improvements to the charts.

  7. Accessibility compliant colors for charts:
    Ensuring that the colors used in charts follow accessibility standards and provide at least a 3:1 contrast ratio against light background.

  8. Improved Overview section:

  9. Addition of the “Total value” in the overview. (For example, All sales/Evaluations/Feedback)

  10. Including a summary line at the top of the overview section, and making the tone of voice more conversational throughout the experience.

  11. Addition of a persistent feedback loop:
    We are including a feedback section at the bottom of every report page to get ongoing feedback.

  12. By default, tables will have a maximum of 50 rows. If there are more than 50 rows, pagination will be enabled.

In addition to the above, you may observe visual improvements throughout the experience and could even uncover some hidden surprises. You never know!:face_with_hand_over_mouth:

Not part of this redesign:

  • One of the feedback we received during the first launch was to enable sorting in Transactions/Licenses/Feedback Details tables. Unfortunately, we cannot accommodate this request in the upcoming rollout. We’ll try to consider this in the next round of improvements.
  • For this release, the major emphasis is on optimising the desktop experience while keeping the mobile platform’s functionalities similar.

Actions

Release timelines & roll out plan

We are planning to roll out the new experience on an opt-in basis. Partners will have the choice to enable the new experience via two in-product touch-points. They can either do this through a dismissible “Messaging card” at the top of the reports page, or through a “Fixed CTA” located in the left navigation of the existing report.

For the first 3 months following the launch, the new experience will remain optional. This will give partners the opportunity to use, compare, become accustomed to, and offer feedback on the new experience. Following the trial period, it will become the default experience for all partners.

Timing: The estimated launch date for the new reports experience is tentatively mid March 2023

We are seeking input on

  • Do you see any serious flaws in the new experience? We request partners to share their feedback on the new designs by responding to the comments on this post.We’re also attaching a " Figma Prototype" in this communication for partners to explore the new experience. Since it’s a prototype, not all features may be functioning as intended, but it should give a glimpse of our intentions. Partners can also submit their feedback directly on the prototype so that it is contextual to the experience.

Note: All the attached references above illustrate the new experience, though there may be slight variations when it is finally rolled out.

  • We are happy to set up 1:1 conversations if you wish to provide detailed feedback. Please leave a comment on the post and we will follow up with you to block some time.

We would like to reiterate that this is the first significant step in enhancing the quality & usefulness of reports. We are striving to get the foundation right. From here on out, we will build upon this.

7 Likes
2

Hi @MalathiVangalapati ,
Thank you for the overview. I like the new design much more than the “old new” design.

Not sure if it falls within the scope, but it would be great to be able to filter on monthly/annual licenses in the tables and charts.

2 Likes
3

Hi @MalathiVangalapati ,

I don’t think that I have read it in the blog (might have missed it too) but something that was also missing and extremely problematic for me last time was that, from the license view, clicking on a SEN number would then load the transaction view with the SEN number already in the filter.

That’s really useful and I hope that you kept it this time.

Otherwise, nice job this time and looking forward to try it out.

2 Likes
4

Hi @MalathiVangalapati,

This would be a great improvement. One thing I would really like to have in sale type is monthly vs annual.
Currently, there is no way to see how much of sales is monthly or annual. This is one of the main reasons we have maintained our own reporting in addition to Marketplace reports.

2 Likes
5

I’m going to play devils advocate here :smiling_imp:

I would like to assert that the Marketplace Partner community does not consider a makeover of the reports section a high priority, especially because this makeover does not resolve the main reasons why Marketplace Partners over the years have chosen to create custom solutions for dealing with transaction data.

Let’s go over all the reasons listed as “why”

The implementation of the “Atlassian Design System” to ensure uniformity and consistency in all the marketplace touch points

To be completely honest with you, this isn’t really important to me as a Marketplace Partner. I would even argue that the lack of understanding of AtlasKit within the Atlassian Marketplace developer team has decreased functionality of the MPAC vendor pages. You failed to understand how Dynamic Table works with regard to sorting, you defaulted to 10 page entries (which suddenly added the requirement of paging that we didn’t have before). The half-baked implementation of AtlasKit/DS components have not been an improvement to MPAC.

In addition, the proposed solution does not ensure uniformity and consistency. You are still not implementing ADG3 because in the proposed solution you are still keeping the ADG2 top bar navigation which is no longer used by Atlassian in this way.

The proposed The implementation of the “Atlassian Design System” to ensure uniformity and consistency in all the marketplace touch points is A) undesirable and B) simply not true as it does not apply to all of the MPAC vendor pages.

Enhanced analytics and instrumentation capabilities to facilitate faster learning and the creation of tailored user experiences.

This is a huge oversell of the proposed solution. Reading this I would expect a Stripe/Chargebee/Profitwell type of dashboard. I would expect graphs with trend analysis, comparing data with previous timeframe (last month, last year, etc). I would expect MRR/ARR calculations.

Instead, what I’m getting is a tweaked version of the existing filters, perhaps with some more filters. This is not going to persuade me from moving away from my custom solution. Again, there is no added value in this redesign for me as a Marketplace Partner.

Additionally, improvement in the overall performance and page loading speed.
This is a generic statement. I will never be able to measure this. What I do know is that I have no objection to the current page loading speed.

There are a lot of features that I would want the Atlassian Marketplace development team to work on instead of a cosmetic redesign of the reports pages.

To only reason I would be in favour of these changes is because of the initial sentence we are upgrading our tech stack to build a reliable and robust foundation for the marketplace partner portal., but only because that implies that this will enable you to further enhance the features of these reports pages.

However, because this RFC is not about we are making intermediate changes X to be able to accommodate the real features Y, but instead is trying to sell something that will not improve my overall experience of the MPAC reporting pages and will definitely not persuade me to abandon my own solutions for trying to get valuable business information out of those transaction/license reports, I have no other option than to say I disapprove of this feature and consider it a waste of time.

2 Likes
6

@remie,

Maybe “too soon” for RFCs? Please understand that @MalathiVangalapati is the brave first author (who didn’t also create the “RFC play”). Yes, over your many posts, I can agree that Atlassian needs to figure out how to “level up” our engagement with developers but is this the right time and place for this leveling up? Can we get a little slack while we figure out this new practice? I ask because you eventually point out RFC-3 (this RFC)…

Maybe you can let a few of these early “no-ops” just go? Especially when, by your own admission, they won’t affect you.

I accept your critique that we might be “dressing up” a technical chore as a feature. Or, that with a little more consideration, it could have been something more than a technical chore. But it doesn’t seem right to be so critical when we are starting this RFC practice. Meanwhile, here in the community and in other channels, you have rightly pointed out that we should assume that UI changes will affect developers. Here we are trying to signal that kind of a change, to which you react:

I can’t tell if this is meant as a “serious flaw” in the RFC. I don’t have enough understanding of Atlaskit myself to know what is the misuse. Or are you proposing that we should not use Atlaskit at all in this effort? It seems like you want to make this a conversation about Atlaskit but you already know that’s off-topic.

I also realize you are suggesting the “serious flaw” is prioritization and effort spent (wasted, from your perspective) on features that have no value (to you). While I want to be able to have that discussion (and we can try in other categories), I’m going to call priorities “off topic” for RFCs (I’ll clarify in RFC-1, and you can disagree with me over there, if I’m wrong).

Overall, I recognize some genuine engagement in there, but at the same time, this doesn’t feel like a great start to a new kind of community engagement. Was your comment a good example of “keeping it welcoming and safe”? Or was it a simple misstep as both authors and commenters alike learn how to work together?

3 Likes
7

Hi @marc ,

Thanks for the feedback, Can you please elaborate on the ask?
Are you referring to having an option to filter the current licenses and transactions based on monthly and annual subscriptions?

8

Hi @yvesriel ,

Thanks for flagging this, The issue with SEN, is already being addressed in the upcoming rollout.

2 Likes
9

Thanks @chhantyal , we have added this feedback to our backlog. Will update this thread as we make progress.

1 Like
10

keeping it welcoming and safe

This is a two way street @ibuchanan. As you might have digested from my comment, I do not consider the initial RFC to be welcoming nor safe.

Can we get a little slack while we figure out this new practice?

Actually no, Atlassian can’t. At least not from me. Although I fully support the format of RFC, it does not exist in a void, nor does it mean that the unequal power dynamics between Atlassian and Marketplace Partners is no longer applicable.

Please understand that @MalathiVangalapati is the brave first author (who didn’t also create the “RFC play”).

If Atlassian truly wants to make this safe & welcoming, please note that the author for any RFC will forever and always be Atlassian, with @MalathiVangalapati (or any other staff member) solely being the representative for the company.

It is unfair to put the burden of “figuring out” this practice on the Atlassian Marketplace community, especially given the history of both the state of communication and specifically the contents of this RFC.

The burden of guidance for Atlassian staff with regard to working with RFC lies solely with Atlassian. Atlassian could have reviewed the RFC prior to publishing or chosen a different (smaller, less problematic) topic for RFC.

But it doesn’t seem right to be so critical when we are starting this RFC practice.

Asking the Atlassian Marketplace community to hold back on their critique with regard to these RFCs event though they have a direct impact on their business, within the context of the unequal power dynamics of our relationship, is bordering emotional blackmail.

Now with regard to the contents of the RFC:

Of course I will be impacted. It means that I, and many other vendors, will have to continue to support my own haphazard solutions. Do you really think I want to maintain internal tooling for sales reporting? I get a far better sales dashboard from Stripe for a fraction of the revenue share I pay them.

No, my point is was that the RFC is saying that the overall experience will be improved because of the use of ADG3/AtlasKit. However, this is a false statement, as the vendor pages will not entirely be converted to ADG3 and past experiences have showcased a lack of understanding of how to work with AtlasKit components by the Atlassian Marketplace team. The serious flaw is the disconnect between the statement The implementation of the “Atlassian Design System” to ensure uniformity and consistency in all the marketplace touch points. will not be achieved by the proposed solution.

Yes, I think we should have this discussion in RFC-1 because I do wish to argue my case that we should be able to discuss prioritisation as part of an RFC.

Again, I really like the process of using RFCs, but if they are only going to be used to sugar-coat the underlying communications problems between Atlassian and the Atlassian Community, we might as well not spent any time on this. Unless Atlassian is genuinely interested in listening to and acting on feedback (even if this does not suit you), we are still left screaming in the void.

We are coming from a place of zero confidence. I can already guarantee you that whatever anyone says the Atlassian Marketplace team will continue moving forward with this RFC, and given that we still have no means to escalate this, why should we even bother to comment?

3 Likes
11

Just to clarify, I vote to reject this RFC based on the following serious flaws:

1. The problem statement does not actually describe a problem

The only possible hint of a problem I can find is that this is the mention of upgrading our tech stack to build a reliable and robust foundation for the marketplace partner portal.

However, this does not describe a problem? It does mention that upgrading to a newer tech stack will allow Atlassian to enable

  • The implementation of the “Atlassian Design System” to ensure uniformity and consistency in all the marketplace touch points.
  • Enhanced analytics and instrumentation capabilities to facilitate faster learning and the creation of tailored user experiences.
  • Additionally, improvement in the overall performance and page loading speed.

However, there is no problem statement describing the necessity for these three enhancements. Which problem does migrating to ADG3 solve? Which problem does enhanced analytics an instrumentation capabilities solve? Which problem does overall performance and page loading speed solve?

These statements are just put there in a matter of fact way as if they are self-explanatory and desirable. However, there is no mention of any customer demand for any of these three bullet points.

2. The proposed solution does not relate to the problem statement

The proposed solution mentions that it resolves feedback from the previous attempt to rollout this experience. Many of the features mentioned (like a trial period with the ability to switch back/forth) are not mentioned in the problem statement itself. It tries to sell the initial redesign (that was rolled back) based on it’s features and how it addresses the previous feedback.

However, because the problem statement never mentions any of these features to be an actual problem of the current experience, the redesign in and of itself is presented again as matter of factly and not something up for debate. Again, what problem does the proposed solution of the redesign try to resolve?

3. The actions do not clarify the process of gathering feedback
Although the actions call for partners to give feedback to the new redesign, with a 3 month trial period, there is no mention on how the Atlassian Marketplace team will respond to this feedback, whether it will be made public, if we can vote on it, etc. Nor does the team mention anything with regard to either extending the trial period or even rolling back the entire experience based on the feedback. As it is currently stated, the feedback loop is completely gratuitous. Atlassian does not commit in any way to doing anything with that feedback.

4 Likes
12

Hi @MalathiVangalapati ,
Yes, I would like to see which licenses and transactions are annual, and filter on annual and monthly.
At this moment we can see which transaction is monthly/annual by clicking at the details view, but it is not possible to e.g. list all monthly/annual transactions or list all monthly/annual licenses.

Similar, it would be great if there is a filter in the sales graph, which allows to show monthly/annual sales (similar in principle on how you can view sales by region, but using monthly/annual instead of regions).

Thanks for following up!

2 Likes
13

Hi Remie,

Thank you for the detailed feedback!

Our focus is primarily on upgrading our tech stack to build a reliable and robust foundation for reports. In the process, we see a merit in enhancing the user experience.
There is wasted effort in trying to recreate older user experience on the new stack hence we prioritised the partial makeover of experience along with the tech stack upgrade.

We’ve had limitations with the old tech stack and upgrading the experience will aid us in delivering value faster to the partners.In this process, we are incrementally migrating the experience to adhere to Atlaskit guidelines and hence you will see a few components that are still a part of the old guideline.

With these changes, we are enhancing our internal analytics and instrumentation to facilitate faster learning to be able to serve partners better with tailored experiences.

Again to reiterate, this is just a FIRST STEP and not the end of Reports enhancement.

On the point of ARR/MRR, a RFC is in works and I will be publishing it in ~4weeks.

6 Likes
14

Morning @MalathiVangalapati - appreciate you publishing this RFC.

Just to clarify, would this redesign accommodate any changes in the underlying data and insights? Specifically a shift from cash to accrual reporting?

For instance, we are leveraging SaaS metrics on an accrual basis (MRR, Logo Churn, $ Churn, etc) to smooth out seasonality. Today that understanding is provided in our own systems, so I’m not fussed either way, yet I wonder if that is the direction Marketplace is heading.

Thank you Malathi, have a good day,
Nick Muldoon
Easy Agile

3 Likes
15

What kind of “warranty” commitment is Atlassian making during this period? 3 months and nothing being improved and then forced out to prod is something we have experienced in the past.

Will there be any ability to toggle from old to new and back if the new one has critical flaws that impact us?

1 Like
16

Unclear whether I will still be able to:

  • easily see the monthly sales sum by hosting type,
  • back to 2 years or more,
  • up to the immediate time (It was a drawback of the last redesign),
  • and save that URL so I don’t have to select all boxes all the time,
  • Plus, I gave up hoping for quality polish of UIs,

Therefore, if one asked me to vote, I’d vote against.

If they want to upgrade, they could have just created a new chart type and see whether people select that chart type by default.

17

Further to @nick’s comments, the existing Marketplace charts are not particularly useful for subscription revenue. Cloud is the future, so dressing up the old charts by adding titles and legends does not make them any more useful. For cash-based reporting, having (say) the net sales number for a specific month is helpful, but that is best expressed as a tabular report and not a chart.

What’s the whole point of displaying any of this information graphically? To visualize trends. But the Marketplace reports don’t help you visualize trends, because the hodgepodge of monthly and annual purchases do not let you see MRR or any vaguely-equivalent metric that might actually expose a trend.

Let me give you an example. Here is what Marketplace output looks like for Cloud sales:

not-helpful
not-helpful1412×230 14.7 KB

This is not particularly helpful. If I want to use it to calculate cashflow, I need the actual numbers (so again, we’re talking about a table) rather than a pretty chart. And the line does not tell any story, other than cashflow being inconsistent.

Here is what that exact same data looks like when the sales are broken down to express MRR and to allocate subscription revenue to the month in which it belongs:

helpful
helpful1084×486 9.5 KB

This is actually useful and actionable. This is also why I go elsewhere for Marketplace data reporting. Investing time in making the charts look nicer doesn’t help unless you are charting relevant data.

6 Likes
18

@scott.dudley - thanks for the detailed examples, you explained why it is important to have the accrual reports. Champion!

19

Hi @nick ,

This RFC is currently not addressing any changes to the underlying data.
That being said , totally agree with you on the need for surfacing more SaaS metrics .
This is on our roadmap and we will publish an RFC as we make progress.

Best,
Malathi

1 Like
20

@scott.dudley , appreciate the detailed examples.
We will share more updates on the SaaS metrics as we make progress.

Best,
Malathi

2 Likes