"Incoming webhook" type automation triggers changed their base URL and have a new undocumented "Secret" parameter (randomly?!)

One of our customers sent us a screenshot about the “Incoming webhook” trigger config in one of his Jira Cloud automation rules. His shot looks a bit different compared to our triggers, because apart from smaller differences (links moved, text changed):

  1. His base URL does not start with https://automation.atlassian.com, but with https://api-private.atlassian.com?!
  2. He has a “Secret” field?!

I tried to figure out what’s going on, but we don’t have those changes. The official Jira Automation documentation doesn’t mention anything, either.

His config form looks like this:

While mine looks like this:

Then I picked one of my playground sites to see if I can reproduce the problem, and created a dummy rule with this trigger type. Now I can see that my config form is “old style”, but the rule component’s preview is already “new style”?!

It seems that Atlassian is making some changes, but it is not documented anywhere, not rolled out globally and even on the same site, it is “half-working”.

Atlassian, could you please clarify?

4 Likes

Hey @aron.gombas,

As part of our continuous focus to uplift the security and reliability of Atlassian automation, we’re routing all rules containing incoming webhook triggers through a more secure endpoint. For further details on this update, please refer to our community article: Update on Incoming Webhooks Trigger for Atlassian Automation Rules(Update on Incoming Webhooks Trigger for Atlassian ... - Atlassian Community).

The changes are rolled out gradually and may not be on your site just yet. All new rules featuring incoming webhook triggers will be routed through the new endpoint within the next 4 to 5 business days, barring any significant issues. Additionally, our Atlassian Cloud changes blog(https://confluence.atlassian.com/cloud/blog) is published weekly, and you can expect to see information about this change in the upcoming blog post.

We apologise for any confusion this may have caused.

Thanks for the reply, @DhanapalMohanasamy! I missed that article on CAC, thanks for the link, too!

It is not clear to me that there is a change that is already gradually rolled out, but it is not documented anywhere how to use it. I mean that the new trigger is already available on many sites, but I can’t see the documentation how to call the URL, more specifically how to use the “Secret” value? Should this be passed in a request header?

Without this information, this trigger type cannot be used if someone wants to build a new automation rule.

Is the documentation available anywhere?

3 Likes

I’m seeing a similar issue. Actually I created an incoming webhook automation in the last couple days and now the automation is completely missing. I tried recreating it, but I get the new incoming trigger type and have no idea how to use the secret value. I assume some sort of authorization header, but as Aron pointed out there is no documentation. Can you please direct us on how to use this new value, or is that a secret?

Ahh, @aron.gombas I found this article here: Check if the incoming webhook trigger is working correctly in your automation rule | Automation | Atlassian Support which states that the secret should be sent as X-Automation-Webhook-Token

Check using cURL

curl -X POST -H 'Content-type: application/json' -H 'X-Automation-Webhook-Token: <Add you secret here>' \ <Insert_webhook_URL>

That seems to work for me. Hope that helps.

3 Likes

Hi @DhanapalMohanasamy. Several of our apps (e.g. eSign for Jira) support integration with automation webhooks and for security reasons the webhook URL is restricted to automation.atlassian.net hostnames.

This security change that was rolled out to production without notice has negatively impacted our customers who are seeing this new alternate webhook URL (api-private) for webhook triggers.

PLEASE release sufficient information immediately so that we can plan and develop the changes to our app to work with the new security infrastructure.

Sincerely
Chris C
Digital Rose

5 Likes

Hi @Chris_at_DigitalRose ,

My name is Diego and I’m a developer with the Automation team.

I’m sorry to hear that some of your apps are having difficulties triggering the new webhook URLs. Thank you for being security conscious. As @DhanapalMohanasamy has said, we are rolling out this change in order to uplift the security and reliability of the Automation webhooks. Atlassian uses a number of domains, and my advice would be to relax the restriction. Restricting your apps to only talk to a specific domain may cause problems with existing Atlassian Cloud features like custom domains, and ongoing developments like FedRamp. If completely relaxing the restriction is not a viable solution for you, I would recommend to at least allow traffic to *.atlassian.com domains.
I hope that helps,
Diego

Thank you for getting back to us Diego. We will make changes to relax the domain restrictions. The new secret key with custom header is also going to require a UX change in our apps configuration screens.

PS. It would have been nice to get some notice before this significant webhook change was released to production. Now we have to deal with customers who don’t understand why they can’t connect webhooks when they were able to last week.

It doesn’t make either of our companies look very good when all we can do is apologize and say Atlassian surprised us with this change that was rolled out without notice. This may be an opportunity for improvement.

Sincerely
Chris Cairns
Digital Rose

8 Likes

Noticed the change by an accident today. We released support of incoming webtrigger last week. All was working like a charm and no nothing works anymore.
Really unpleasant how this was announced. Nothing in developers changelog about that in near past. Incoming webtrigger is the only option for Forge in some usecase where the app needs to send some custom notifications or allow users their custom extensions.
What is the old rules (no tokens/no security) deprecation policy by Atlassian? Today my old automation rule works with no token. Gateway is just different. If I add new rule, then i have another automation with additional security and most likely it also works. Thus when and how the working rules will be shut down - or whats the strategy here?

Margus
SoftComply

4 Likes

This should be part of the Incoming Webhook Trigger description, thanks for sharing.

Btw the documentation has a typo so be aware of not copy and paste because it might not work.

3 Likes

Hi Chris,
Glad to hear you can relax the domain restrictions. Regarding the custom header, I appreciate that may require some extra effort, but we believe that is an important change in order to uplift security for our customers.
Regards,
Diego

1 Like

Hi @ItaloModusCreate ,
Good catch! Yes, there’s a typo in that document. The correct header name is X-Automation-Webhook-Token. We will fix that typo soon.
Regards,
Diego

2 Likes

Hi Atlassian team. Currently the new webhook returns 200 with an invalid token, and nothing in the automation audit log. Will make troubleshooting difficult with our customers.

Would you consider returning a 401 from the webhook if the auth token is invalid? Note: webhook does return 400 if the auth token header is totally missing, which is good.

Thank you
Chris
Digital Rose

PS. We noticed a minor UX issue in the new Automation UI. The secret key “Copy to Clipboard” button only works if the key is unmasked first. If the bullet masking is on the copy button says “Copied” but the clipboard is still empty.

1 Like

Hi @Chris_at_DigitalRose ,
Thank you for the feedback!
We have taken note of the problem with the “Copy to clipboard” button and we will be looking at it shortly.
Regarding the status code, as you have noticed the endpoint returns 400 in case the header with the secret token is missing. However, we made the decision to return 200 OK if the token is present but invalid. This is for security reasons, to prevent credential stuffing and brute force attacks. We understand this makes it a bit harder to troubleshoot some legit problems, but it is necessary in order to keep customers safe.
Regards,
Diego

1 Like

Uhhhhm. This is not how security works. I think we need to run this by your security team - I fear you are actually making things worse here.

  1. Assuming I have a botnet capable of sending you 100 million requests per second (unlikely) it would still take 1.2584325753214312e+41 years (eternity, for the arguments sake) to brute force a 40 character API key. How is this a justification to break standard API status codes? No other APIs in Atlassian do this, or is this something that will also be rolled out to other API endpoints?

  2. You are prioritizing a theoretical security risk for an actual, operational one. Now that you cannot find out if a trigger is actually still authenticated correctly, rotating the secret becomes problematic, because affected pipelines or scripts will silently fail. This is not good and hampers the ability to rotate the secret. We can also not show customers if a trigger is still working, because we just don’t know, so our audit log message basically now needs to say “We tried to execute the trigger, but we don’t know if it worked :)”

  3. You are apparently storing the secret in plain text and not hashed, because you can view the secret after the rule has been created. This does not seem very secure to me, to be honest, and far from security best practices. So it’s a bit hard for me to understand that you are not sending proper HTTP codes but decided to not hash the actual API key - or at least not show it forever once the trigger has been created? Best practice btw. would also be to allow multiple secrets (with a name, like, actual Atlassian API keys), so you can rotate them without causing a downtime.

11 Likes

Hi @tobias.viehweger ,
Thank you for the feedback! Please rest assured that we have considered the aspects you mention, and that we are following expert advice when making tradeoffs. The changes we have just rolled out are part of our ongoing initiative to uplift security, and they put webhooks in a better place to eventually deliver further improvements with better tradeoffs and more aligned with how other Atlassian APIs behave. In terms of troubleshooting misconfigured webhooks, the new webhook URL behaves like the old one did.
Regards,
Diego

Hi Diego,

Thanks for the reply. To me it seems to not behave the same, because there is now a secret and we have no idea if we are sending the correct one? I don’t understand how this would be considered the same?

Thanks for the clarification!
Tobias

3 Likes

I would also like to point out to others that you might need to fix your logs/sanitization logic to avoid leaking this into your log files. We have a rather exhaustive audit log for outgoing requests, which also logs x-headers, but due to this using a custom X-Header instead of the standard Authorization header, this secret leaks into all our log files.

4 Likes

Can you explain the reasoning for your consideration of the threat of credential stuffing attacks etc? What you describe is typically possible when credentials are poorly constructed. Is Atlassian telling us that the API keys are not properly opaque, and that rather than creating keys with a satisfiable cardinality/algorithm+key size, the solution is security through obscurity? I would appreciate links to research articles, industry standards or other sources that support the reasoning behind this decision, because it is unclear to me how this improves security. This comes at the expense of basic expectations of semantic HTTP responses from an API that damages stability of integrations, without any obvious reduction of security risk.

5 Likes

Just adding to this that it has completely broken integration with Call Events from Dialpad to JSM.

They can’t just put in the x-automation into the header and there is no way to blat the secret from the automation node.

Everything is failing now.

{
“errorMessage”: null,
“message”: null,
“warningMessages”: ,
“errorMessages”: [
“Missing token”
],
“errors”: {},
“status”: 400
}

What would be ideal is a toggle or something to include the secret or not as an expectation for the incoming webhook.