Hi,
For those of you who still have DC apps (you’re a dying breed) that still need to do the annual DC testing and who have been doing this manually, I would like to offer you some convenience methods of automating parts of your task.
We’ve been incorporating support for the Data Center App Performance Toolkit (APT) in our unofficial Atlassian Data Center Plugin Development CLI (DCDX). Here are some steps on how you can use DCDX to prepare and run your DC performance tests
Prerequisites
Install DCDX
In order to start, please install DCDX on your local machine. You will need NodeJS and NPM. Once you have installed those, run:
npm install -g dcdx@next
The reason the @next tag is selected is because the APT support is currently in BETA and only available on the nightly build. This may sound scary, but we’ve been using this for several months now and it is actively supported by us.
Get AWS credentials
To run the DC performance tests, you will need to have an AWS account and get CLI credentials. You will be asked for AWS access key ID and AWS secret access key during every step in the process.
Manually operate the cluster
The AWS cluster will be automatically created when you are running the tests with DCDX, but if you need to have a cluster (for instance, for developing the required Selenium UI, Locust and/or JMeter tests), you can also manage it manually using the following commands.
Provision a cluster on AWS
In order to perform the DC performance testing, you will need to provision a cluster on AWS. You can do this by running the following command:
dcdx apt provision --product <jira|confluence|bitbucket|bamboo> --environment <name>
Choose the host product that you want to install a cluster of and provide a name. Be careful, the name needs to be lower case and preferably be short due to a limitation in S3 bucket names. For instance, we use ffj when we provision a cluster for testing our Figma for Jira app.
The CLI will ask you for a path to install the Data Center App Performance Toolkit. You can use an existing location or use a new location. If you choose a new location, the CLI will download the toolkit for you.
Afterwards it will ask you for the AWS credentials. Once provided it will start provisioning the cluster. When it is ready, you will see the URL of the new cluster in the terminal output on your screen:
It will also save this information to a JSON file on disk, in the location where you told the CLI to install the DC App Performance Toolkit, in the file ./app/util/k8s/outputs.json
For more options, run dcdx apt provision --help
Restart a cluster on AWS
If required, you can restart the host product on the cluster by running the following command:
dcdx apt restart --product <jira|confluence|bitbucket|bamboo> --environment <name>
Select the same host product and name of the cluster as the one you provisioned earlier. You will again be asked for the location of the Data Center App Performance Toolkit and you AWS credentials.
For more options, run dcdx apt restart --help
Terminate a cluster on AWS
Once you are done testing, it is advised to terminate the cluster on AWS by running the following command:
dcdx apt teardown --product <jira|confluence|bitbucket|bamboo> --environment <name>
Select the same host product and name of the cluster as the one you provisioned earlier. You will again be asked for the location of the Data Center App Performance Toolkit and you AWS credentials.
For more options, run dcdx apt teardown --help
IMPORTANT: if you forget to terminate the cluster, you will incur AWS costs as the cluster will continue to consume resources
Running the tests
Depending on the product and requirements, you can run the re-index test, the performance tests (run 1 & run 2), the scalability tests (run 3, run 4, run 5), the dependency test (maven only) and the static code analysis (SCA) test.
Not all products require all the steps. To ensure that you only run the required tests, it is recommended to run the performance and scalability commands as they will determine which run is required for the chosen host product.
Lucene Index Timing
For Atlassian Jira, you will need to run a Lucene Index Timing test (reindex) with the app installed. To do this automatically, run
dcdx apt reindex --environment <name> --appKey <key>
This will create the cluster, install the app and run the reindex test. It will download the latest version of the app from the Atlassian Marketplace based on the appKey (the key you provided in atlassian-plugin.xml). If you need to run the test on a specific version of the app, or if the app is not yet on the Atlassian Marketplace, you can use --archive <path> to use a specific JAR/OBR file.
For more options, run dcdx apt reindex --help
Performance test
The performance test usually consists of two runs: a test run without the app for baseline results, and the same test run with the app installed. If you run
dcdx apt performance --product <jira|confluence|bitbucket|bamboo> --environment <name> --appKey <key>
it will provision a cluster, run the baseline test, install the app, run the comparison test with the app installed. It will download the latest version of the app from the Atlassian Marketplace based on the appKey (the key you provided in atlassian-plugin.xml). If you need to run the test on a specific version of the app, or if the app is not yet on the Atlassian Marketplace, you can use --archive <path> to use a specific JAR/OBR file.
For more options, run dcdx apt performance --help
Report
Once you’ve completed the performance tests, you will need to generate the report that has to be attached to the ECOHELP ticket. Make sure to collect the tests results of the two performance test runs, and run:
dcdx apt report --type performance --product <jira|confluence|bitbucket|bamboo> --resultsDir1 <path_to_results> --resultsDir2 <path_to_results>
This will generate the report for the performance test.
For more options, run dcdx apt report --help
Scalability test
The scalability test usually consists of three runs and includes the app specific Selenium UI, Locust and/or JMeter tests. In addition, it will run the same test with 1 node, 2 nodes and 4 nodes.
To do this, you will have to make sure that you download the Data Center App Performance Toolkit and adjust the correct files (extension_ui.py, extension_locust.py and/or <product>.jmx).
Once you have made sure that your app specific tests are in the correct location, run
dcdx apt scalability --product <jira|confluence|bitbucket|bamboo> --environment <name> --appKey <key>
it will provision a cluster, install the app, use the app specific tests and run the test with 1 node, 2 nodes and 4 nodes respectively. It will download the latest version of the app from the Atlassian Marketplace based on the appKey (the key you provided in atlassian-plugin.xml). If you need to run the test on a specific version of the app, or if the app is not yet on the Atlassian Marketplace, you can use --archive <path> to use a specific JAR/OBR file.
For more options, run dcdx apt scalability --help
Report
Once you’ve completed the scalability tests, you will need to generate the report that has to be attached to the ECOHELP ticket. Make sure to collect the tests results of the three performance test runs, and run:
dcdx apt report --type scalability --product <jira|confluence|bitbucket|bamboo> --resultsDir1 <path_to_results> --resultsDir2 <path_to_results> --resultsDir3 <path_to_results>
This will generate the report for the performance test.
For more options, run dcdx apt report --help
Dependencies
You will also need to provide a dependency graph. You can do this by running the following command (Maven only):
dcdx apt dependencies --groupId <groupId> --artifactId <artifactId> --appKey <key>
This will generate the dependency file in the correct format that can be included in the ECOHELP ticket. It will download the latest version of the app from the Atlassian Marketplace based on the appKey (the key you provided in atlassian-plugin.xml). If you need to run the test on a specific version of the app, or if the app is not yet on the Atlassian Marketplace, you can use --archive <path> to use a specific JAR/OBR file.
For more options, run dcdx apt dependencies --help
Static Code Analysis (SCA)
Another artifact that you need to provide is a Static Code Analysis report. To run this test, use
dcdx apt sca --appKey <key>
This will generate the static code analysis report in the correct format that can be included in the ECOHELP ticket. It will download the latest version of the app from the Atlassian Marketplace based on the appKey (the key you provided in atlassian-plugin.xml). If you need to run the test on a specific version of the app, or if the app is not yet on the Atlassian Marketplace, you can use --archive <path> to use a specific JAR/OBR file.
For more options, run dcdx apt sca --help
!! IMPORTANT !!
Although the cluster is automatically provisioned when running one of the tests, it is not automatically deleted. You will need to run the
dcdx apt teardowncommand manually to make sure that the cluster does not continue to incur costs on your AWS account
Automation
The next step is to automate the execution of your DC app performance tests in your CI/CD environment. We use CircleCI and have created a pipeline that does ever test concurrently, and automatically generates a single ZIP file that we can attach to the ECOHELP ticket. All of this is open source, and I will write another topic on how we do this soon!
Still not feeling it?
Although the above steps should improve the life of any DC developer tasked to perform the annual DC app approval, and should allow you to even automate it, there is also the option of letting us do it for you! If you want to offload the burden, you can create a support request here and we can help run the tests for you!