Cloud Fortified - low installation callback success rate

Hi

Lately, we have noted a decrease in the installation callback success rate reported for one of our plugins. We have tried to identify the cause of the problem but with no success. We tried several installation tests - all successful; we inspected our logs - and nothing suspicious was found; we verified our architecture - everything looks and works ok.

We have not received any tickets from the clients, reporting any problems with installation. However, It is our major priority to make sure that our plugin can be installed without problems.

We have also noticed that the beginning of the errors seems to coincide with our latest release, in which we have upgraded ‘atlassian-connect-spring-boot’ to the latest version 4.0.7. It is also the only differentiator compared to our other plugins - as other plugins have not been upgraded to the newest
version yet. According to the docs, ACSB is responsible for handling lifecycle events - including /installed. We suspect it to be the cause of the problem.

Have any of you encountered a similar problem in the past? If so, what would be the cause and solution?

5 Likes

I’m currently working on the upgrade of ACSB myself so am curious to know what causes the issue you are facing, not in production yet.

In my deployment I have heavily customized the ACSB configuration and also needed to debug some issues related to the lifecycle events, in my case security was blocking requests.

Logging is limited, but if you enable the logger com.atlassian.connect.spring.internal.lifecycle then you should see some info, warning, or error messages come by.
If the app itself is working you should see messages like Saved installation for host ... for installations and Saved uninstallation for host ... for uninstallations. In other cases it doesn’t always log errors or warnings, but with log some and throw others.

I have in the past deployed temperate Filters to log any issues or unexpected responses.

Maybe this in combination with access logs of the app will give some insight.

2 Likes

@markrekveld thanks for the info!
We will continue to investigate the problem and will gladly share our findings here.

2 Likes

Hi @MichaTkacz
Please reach out to the CFA team via “General question” on the Developer and Marketplace Support service desk(Jira Service Management)

@opavliuk thank you for the reply.
We have reached out to the support on Monday but unfortunately got no answer yet.

In the meantime, we are investigating the lifecycle logging.

Hi @MichaTkacz,
If you listen to AddonInstalledEvent in your app, pay attention whether this code is executed in new thread. Something may have changed in your app (especially in the TaskExecutors definitions) when upgrading to the latest versions of Spring/ACSB.

Hi @PeterAppsvio
Thank you for your input.

To give an update on the situation, we have learned from Atlassian Support that the problems with the installation callback reported in our Cloud Fortified developer console coincided with some major issues reported at that time in Atlassian Marketplace.

Also, we were given the information that most often the problem with installation success rate is caused because of “the endpoint (/installed) could not be reached”. Ultimately, we have changed nothing on our side and so far all our installations have been successful and stable for over two weeks, so we assume, that the problem was outside of our infrastructure and was not connected to upgrading ACSB. Of course, we continue to monitor the reports.

Just to document our issue with the Cloud Fortified installation success rate here:

If you run the app in development with the same key as production it will run into the Cloud Fortified metrics even if the /install callback points to your local backend.

Debugging the install callback leading to an timeout? → Counts as fail
Backend Server not being started at all → Counts as fail

That is our common reason to break the installation success SLA

1 Like