Is there any official Analytics & A/B Testing tooling/method for Jira Forge Apps? So I can make better product decisions on user behaviour

Hi,

I’m planning to develop Forge app on the Marketplace. Before that, we would like to investigate two use cases:

1.) Analytics collection support about our app usage
2.) A/B Testing launching multiple versions (with different features)

These features would help us as a developer to make informed product decisions in the future to see if we are solving for real problems for our users.

Example of analytic data to collect:

  • How many users installed the App?
  • How much users are using our App?
  • How many users are using our App daily & weekly?
  • Application pages visited per user
  • Features used by the user (e.g. settings are changed)
  • Stats regarding our Forge functions (number of execution, time, …)

Some data can only be collected in the “front” part (CustomUI), while other are retrievable from Forge function executions.

Analytics Questions

Q1 is collecting that kind of data OK regarding Atlassian policy? We don’t plan to retrieve information about users.

Q2 Is there a recommended way to collect that kind of data? Is any official supported tool or method by Atalassian we should using to achieved this goal?

A/B Testing Questions

Q3 Would Jira marketplace allow us as developer to release new version of our app to certain percentage of users (with a new feature)? Is this a option?

Q4 If Q4 Yes; then Is there a recommended way to conduct A/B Testing? Is any official supported tool or method by Atalassian we should using to achieved this goal?

Any thoughts or insights on this matter is much much appreciated.

Thanks,

Chris

3 Likes

Hey Chris,

We don’t have any specific recommendations for capturing analytics, Forge doesn’t provide any user-based analytics out-of-the-box but it’s OK to integrate a service of your choice in your Custom UI app.

Just make sure they’re in a non-personally identifiable form as you mentioned.

You will need to declare it as data egress which will be visible to admins during installation - but it’s your choice if you want to make that trade off.

The developer console has some really good metrics on invocations and API calls that you do get for free. You can also export them to a tool of choice.

There’s Forge logs as well which can also be exported (in EAP currently), but note that admins can opt out of log sharing if they want.

I’ll try and find a colleague to answer your A/B testing question, hope the other answers help though.

4 Likes

Regarding analytics, we use Segment for our apps and it’s also a good choice if you want to capture marketplace data since Atlassian offers the integration natively. See: https://developer.atlassian.com/platform/marketplace/marketplace-integration-with-segment/

3 Likes

@AdamMoore @rcsr Thank you both for your advice, very helpful :slight_smile:

@rcsr - Sorry perhaps this is going off-topic, how do you find using Segment? Does create report and graphs for you to review your data? Or do you have export the data to create your own graphs/reports?

@AdamMoore If you can confirm your findings on A/B Testing either way, that would be good to understand the options. Thanks again

@ChrisJones1 I think Segment acts more like a middle-man. You send all of your analytics to Segment and then they have various connectors to other services like Mixpanel, Amplitude, BigQuery etc. These services would receive the data from Segment to generate reports, dashboards and whatever else you want. I don’t believe Segment offers reports/graphs for you out of the box (at least not that I’ve seen).

Integrating a Forge app with Segment should be pretty straight forward. They offer an SDK but we tend to just handle it ourselves by sending a POST request to their REST API.

2 Likes

@rcsr Thank you for that. Very useful insight. What does your company use along with Segment? Does your company have product/data people review the data and create and share reports with the company to highlight trends & user behavior?

@AdamMoore - Have you had any chance to speak with your colleague as yet in regards of A/B testing options?

Hey @ChrisJones1, yep have confirmed with the marketplace team that there are no rules preventing you from A/B testing and rolling out new features to subsets of customers.

We don’t have any specific frameworks we recommend or anything built into the platform currently. It is something we hope to someday offer on Forge but no timelines in mind yet.

3 Likes

@AdamMoore Thank you for your effect on this :slight_smile: That is really useful for know.

@ChrisJones1 Our company used to use Amplitude but we’ve migrated over to MixPanel. Our product team are mainly the ones who use it and they’ve set up dashboards and charts to monitor user adoption and on a granular level how frequently a specific module or feature is being used.