Hello,
I was trying to port our last year’s test config to the current version of DC App Performance Toolkit, but when I ran it locally against Jira DC 10.0.0, one of the default Locust tests and all of the Selenium tests failed. I have then tried running the toolkit without our tests and only minimal modifications, but it failed in the same way.
Below is my jira.yml. The only changes compared to default are the connection configuration, replacing JMeter with Locust, toggling WEBDRIVER_VISIBLE, and slightly bumping the chromedriver version, as I was unable to find the exact version of Chrome that was pre-configured.
---
settings:
artifacts-dir: results/jira/%Y-%m-%d_%H-%M-%S
aggregator: consolidator
verbose: false
check-updates: false # disable bzt check for updates
env:
application_hostname: localhost # Jira DC hostname without protocol and port e.g. test-jira.atlassian.com or localhost
application_protocol: http # http or https
application_port: 2990 # 80, 443, 8080, 2990, etc
secure: False # Set False to allow insecure connections, e.g. when using self-signed SSL certificate
application_postfix: /jira # e.g. /jira for TerraForm deployment url like `http://a1234-54321.us-east-2.elb.amazonaws.com/jira`. Leave this value blank for url without postfix.
admin_login: admin
admin_password: admin
load_executor: locust # jmeter and locust are supported. jmeter by default.
concurrency: 200 # number of concurrent virtual users for jmeter or locust scenario
test_duration: 45m
ramp-up: 3m # time to spin all concurrent users
total_actions_per_hour: 54500 # number of total JMeter/Locust actions per hour
WEBDRIVER_VISIBLE: True
JMETER_VERSION: 5.6.3
LANGUAGE: en_US.utf8
allow_analytics: Yes # Allow sending basic run analytics to Atlassian. These analytics help us to understand how the tool is being used and help us to continue to invest in this tooling. For more details please see our README.
environment_compliance_check: True # Pre-test environment compliance validation. Set to "False" to skip it.
# Action percentage for Jmeter and Locust load executors
create_issue: 4
search_jql: 11
view_issue: 34
view_project_summary: 3
view_dashboard: 10
edit_issue: 5
add_comment: 2
browse_projects: 9
view_scrum_board: 8
view_kanban_board: 7
view_backlog: 6
browse_boards: 1
standalone_extension: 0 # By default disabled
# Custom dataset section.
custom_dataset_query: # Write JQL query to add JQL output to the app/datasets/jira/custom-issues.csv, e.g. "summary ~ 'AppIssue*'"
services:
- module: shellexec
prepare:
- python util/pre_run/environment_checker.py
- python util/pre_run/environment_compliance_check.py jira
- python util/data_preparation/jira_prepare_data.py
shutdown:
- python util/post_run/jmeter_post_check.py
- python util/jtl_convertor/jtls-to-csv.py kpi.jtl selenium.jtl
post-process:
- python util/analytics/analytics.py jira
- python util/post_run/cleanup_results_dir.py
- module: pip-install
packages:
- selenium==4.25.0
execution:
- scenario: ${load_executor}
executor: ${load_executor}
concurrency: ${concurrency}
hold-for: ${test_duration}
ramp-up: ${ramp-up}
- scenario: selenium
executor: selenium
runner: pytest
hold-for: ${test_duration}
scenarios:
selenium:
script: selenium_ui/jira_ui.py
locust:
script: locustio/jira/locustfile.py
jmeter:
script: jmeter/jira.jmx
properties:
application_hostname: ${application_hostname}
application_protocol: ${application_protocol}
application_port: ${application_port}
application_postfix: ${application_postfix}
# Workload model
total_actions_per_hr: ${total_actions_per_hour}
perc_create_issue: ${create_issue}
perc_search_jql: ${search_jql}
perc_view_issue: ${view_issue}
perc_view_project_summary: ${view_project_summary}
perc_view_dashboard: ${view_dashboard}
perc_edit_issue: ${edit_issue}
perc_add_comment: ${add_comment}
perc_browse_projects: ${browse_projects}
perc_view_scrum_board: ${view_scrum_board}
perc_view_kanban_board: ${view_kanban_board}
perc_view_backlog: ${view_backlog}
perc_browse_boards: ${browse_boards}
perc_standalone_extension: ${standalone_extension}
modules:
consolidator:
rtimes-len: 0 # CONFSRVDEV-7631 reduce sampling
percentiles: [] # CONFSRVDEV-7631 disable all percentiles due to Taurus's excessive memory usage
jmeter:
version: ${JMETER_VERSION}
detect-plugins: true
memory-xmx: 8G # allow JMeter to use up to 8G of memory
plugins:
- jpgc-casutg=2.10
- jpgc-dummy=0.4
- jpgc-ffw=2.0
- jpgc-fifo=0.2
- jpgc-functions=2.2
- jpgc-json=2.7
- jpgc-perfmon=2.1
- jpgc-prmctl=0.4
- jpgc-tst=2.6
- bzm-random-csv=0.8 # not used default jmx file
system-properties:
server.rmi.ssl.disable: true
java.rmi.server.hostname: localhost
httpsampler.ignore_failed_embedded_resources: "true"
selenium:
chromedriver:
version: "130.0.6723.116" # Supports Chrome version 130. You can refer to https://googlechromelabs.github.io/chrome-for-testing
reporting:
- data-source: sample-labels
module: junit-xml
bzt.log (partial, as the full thing is too long, apparently):
[2024-12-27 11:36:40,088 INFO Engine.final-stats] Request label stats:
+------------------------------------------------+--------+---------+---------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| label | status | succ | avg_rt | error |
+------------------------------------------------+--------+---------+---------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| locust_add_comment:open_comment | OK | 100.00% | 27.966 | |
| locust_add_comment:save_comment | OK | 100.00% | 129.918 | |
| locust_browse_boards | OK | 100.00% | 0.089 | |
| locust_browse_projects | OK | 100.00% | 0.073 | |
| locust_create_issue:fill_and_submit_issue_form | FAIL | 62.50% | 75.379 | for url: http://localhost:2990/jira/secure/QuickCreateIssue.jspa?decorator=none&pid=10001&issuetype=10101&atl_token=BWP3-NZB2-6EDY-6C7K_7f77bdd1318a4ee8ca298a4ceb12eb20fa7dfe5b_lin&formToken=5936409779c3856d21046e650d266306a73baf83&summary=Locust%20summary%20LT8CwJzwge&duedate=&reporter=performance_uylwdtsdae&environment=Locust%20environment%20aTiSpWMkg7&description=Locust%20description%208MVmSVkRoo3PuDNmuMRF&timetracking_originalestimate=&timetracking_remainingestimate=&is_create_issue=true&hasWorkStarted=&fieldsToRetain=project&fieldsToRetain=issuetype&fieldsToRetain=summary&fieldsToRetain=description&fieldsToRetain=priority&fieldsToRetain=labels&fieldsToRetain=attachment&fieldsToRetain=issuelinks&fieldsToRetain=assignee&fieldsToRetain=customfield_10014&fieldsToRetain=customfield_10015 |
| | | | | similar urls keep repeating
| locust_create_issue:open_quick_create | OK | 100.00% | 118.152 | |
| locust_edit_issue:open_editor | OK | 100.00% | 119.154 | |
| locust_edit_issue:save_edit | OK | 100.00% | 245.550 | |
| locust_login_and_view_dashboard | OK | 100.00% | 0.895 | |
| locust_search_jql | OK | 100.00% | 337.823 | |
| locust_view_backlog | OK | 100.00% | 0.153 | |
| locust_view_dashboard | OK | 100.00% | 0.091 | |
| locust_view_issue | OK | 100.00% | 127.571 | |
| locust_view_kanban_board | OK | 100.00% | 0.143 | |
| locust_view_project_summary | OK | 100.00% | 0.196 | |
| locust_view_scrum_board | OK | 100.00% | 0.120 | |
| selenium_a_login | FAIL | 0.00% | 230.744 | Exception: ('Failed measure: selenium_login - Exception', Exception('Failed measure: selenium_login:open_login_page - TimeoutException - timeout: Timed out receiving message from renderer: 299.971 (Session info: chrome=130.0.6723.116)', TimeoutException())) |
| | | | | Exception: ('Failed measure: selenium_login - Exception', Exception("Failed measure: selenium_login:login_and_view_dashboard - TimeoutException - ('id' 'dashboard')<function visibility_of_element_located.<locals>._predicate at 0x7f5dd6df2d40>", TimeoutException())) |
| selenium_create_issue | FAIL | 0.00% | 300.040 | selenium.common.exceptions.TimeoutException: Message: timeout: Timed out receiving message from renderer: 299.998 |
| | | | | (Session info: chrome=130.0.6723.116) |
| | | | | Stacktrace: |
| | | | | #0 0x556ed43df6da <unknown> |
| | | | | #1 0x556ed3ef3f80 <unknown> |
| | | | | #2 0x556ed3edc5e3 <unknown> |
| | | | | #3 0x556ed3edc2d2 <unknown> |
| | | | | #4 0x556ed3eda35c <unknown> |
| | | | | #5 0x556ed3edab0f <unknown> |
| | | | | #6 0x556ed3ee9fb7 <unknown> |
| | | | | #7 0x556ed3efffc4 <unknown> |
| | | | | #8 0x556ed3f0531b <unknown> |
| | | | | #9 0x556ed3edb21e <unknown> |
| | | | | #10 0x556ed3effe24 <unknown> |
| | | | | #11 0x556ed3f83902 <unknown> |
| | | | | #12 0x556ed3f64953 <unknown> |
| | | | | #13 0x556ed3f3372e <unknown> |
| | | | | #14 0x556ed3f3479e <unknown> |
| | | | | #15 0x556ed43aaf1b <unknown> |
| | | | | #16 0x556ed43aeeb8 <unknown> |
| | | | | #17 0x556ed439842c <unknown> |
| | | | | #18 0x556ed43afa37 <unknown> |
| | | | | #19 0x556ed437cfef <unknown> |
| | | | | #20 0x556ed43cdad8 <unknown> |
| | | | | #21 0x556ed43cdca0 <unknown> |
| | | | | #22 0x556ed43de556 <unknown> |
| | | | | #23 0x7ffba7481ac3 <unknown> |
+------------------------------------------------+--------+---------+---------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
I have switched Chrome to be visible, and the tests appeared to get stuck on the “Welcome to Jira” page with all the language selections (and later on just a white page, probably the result of the previous tests failing?), even though the default language is set to English.
If I understand the Selenium config right, it should look for the Continue button by ID and click it, but this seems to fail, even though I have confirmed the presence of the correct ID on the rendered button on the Webdriver Chrome. I have even tried disabling the Welcome page, but no dice. It’s eventually supposed to get to the main dashboard and look it up by ID, but it either doesn’t get there, or the lookup fails, even though the element with the ID is again present.
Has anyone else encountered this and figured out a solution?