I’m planning to develop Forge app on the Marketplace. Before that, we would like to investigate two use cases:
1.) Analytics collection support about our app usage
2.) A/B Testing launching multiple versions (with different features)
These features would help us as a developer to make informed product decisions in the future to see if we are solving for real problems for our users.
Example of analytic data to collect:
How many users installed the App?
How much users are using our App?
How many users are using our App daily & weekly?
Application pages visited per user
Features used by the user (e.g. settings are changed)
Stats regarding our Forge functions (number of execution, time, …)
Some data can only be collected in the “front” part (CustomUI), while other are retrievable from Forge function executions.
Analytics Questions
Q1 is collecting that kind of data OK regarding Atlassian policy? We don’t plan to retrieve information about users.
Q2 Is there a recommended way to collect that kind of data? Is any official supported tool or method by Atalassian we should using to achieved this goal?
A/B Testing Questions
Q3 Would Jira marketplace allow us as developer to release new version of our app to certain percentage of users (with a new feature)? Is this a option?
Q4 If Q4 Yes; then Is there a recommended way to conduct A/B Testing? Is any official supported tool or method by Atalassian we should using to achieved this goal?
Any thoughts or insights on this matter is much much appreciated.
We don’t have any specific recommendations for capturing analytics, Forge doesn’t provide any user-based analytics out-of-the-box but it’s OK to integrate a service of your choice in your Custom UI app.
Just make sure they’re in a non-personally identifiable form as you mentioned.
You will need to declare it as data egress which will be visible to admins during installation - but it’s your choice if you want to make that trade off.
@AdamMoore@rcsr Thank you both for your advice, very helpful
@rcsr - Sorry perhaps this is going off-topic, how do you find using Segment? Does create report and graphs for you to review your data? Or do you have export the data to create your own graphs/reports?
@AdamMoore If you can confirm your findings on A/B Testing either way, that would be good to understand the options. Thanks again
@ChrisJones1 I think Segment acts more like a middle-man. You send all of your analytics to Segment and then they have various connectors to other services like Mixpanel, Amplitude, BigQuery etc. These services would receive the data from Segment to generate reports, dashboards and whatever else you want. I don’t believe Segment offers reports/graphs for you out of the box (at least not that I’ve seen).
Integrating a Forge app with Segment should be pretty straight forward. They offer an SDK but we tend to just handle it ourselves by sending a POST request to their REST API.
@rcsr Thank you for that. Very useful insight. What does your company use along with Segment? Does your company have product/data people review the data and create and share reports with the company to highlight trends & user behavior?
@AdamMoore - Have you had any chance to speak with your colleague as yet in regards of A/B testing options?
Hey @ChrisJones1, yep have confirmed with the marketplace team that there are no rules preventing you from A/B testing and rolling out new features to subsets of customers.
We don’t have any specific frameworks we recommend or anything built into the platform currently. It is something we hope to someday offer on Forge but no timelines in mind yet.
@ChrisJones1 Our company used to use Amplitude but we’ve migrated over to MixPanel. Our product team are mainly the ones who use it and they’ve set up dashboards and charts to monitor user adoption and on a granular level how frequently a specific module or feature is being used.
Considering the soon new badge “Runs on Atlassian”, will Forge provide at some point tools for analytics similar to, for example, Segment? As these tools provide a lot of insights into business decisions, we would have to choose to either get the badge and make decisions blinded or to be left out of this extra trust system for customers in order to provide a better product.
In the short to medium term for Runs on Atlassian, apps will be able egress to analytics tools of their choice but customers will be able to control whether they want to share analytics or not as mentioned in the roadmap ticket here. We will be exploring Atlassian Analytics next year. After thorough analysis, we realised that for the short term, customer consent based analytics egress would be the best way forward. We will additionally be exploring ways to show some common metrics to all forge developers like like Daily active users etc in developer console or in Atlassian analytics.
