Sorry for hijacking the thread Pablo lol
Remi, you stating your deposit figures proves my point mate: our required deposit was $15,000.
That’s down from the initial $1,810,000 they wanted, followed by a second deal that wanted $25,000-$35,000 or $258,000-$362,000 (lacked clarification). Those figures, ticket numbers and quotes are all documented below.
So why were we screwed and what other deals are vendors getting? I asked multiple times to know what deals other comparable vendors with large app portfolios were getting. Seemed only fair to pay the same they did. I specifically asked what deal you were getting and did not get an answer.
“Waste of money” = we’ve paid out $700 of $15k since these bounties went live in Sep 2022.
“Waste of time” = the 434 days it took dealing with the bureaucracy of it all (100s of hours in total).
Combined = unfair deposit requirement compared to other vendors and the untold loss in clicks (thus revenue) while waiting an unfair amount of time for those badges to be applied.
Regarding the anti-competitiveness this whole program brings to the marketplace: I’m not saying bug bounties are bad. I’m saying it’s financial inequality (as in unequal opportunity) for new vendors that only serves to entrench the older vendors. Unequal opportunity when it comes to install and review count is one thing. It’s another to have unequal opportunity purely on the basis of ability to pay.
$5,000 is nothing to you and I at this stage of our businesses (and good god I don’t want to be here as long as you have - my ambitions go far beyond selling enterprise widgets) but in the early days I believe it took me over two years before I saw that much cumulative revenue.
So the problem is you have a pay-to-play incentive which boosts entrenched vendor apps with giant “THIS APP IS SECURE AND GOOD” badges while any new vendors get royally f*cked in the search results.
The marketplace should be structured in a way that sees the best products organically rise to the top. It’s not setup that way in any shape or form.
It’s in both yours and my best business interests to exclude new vendors and adopt any and all of these dumb Atlassian programs that build an artificial moat around threats from any new competitors. But it’s also not right.
I know you’ve complained about the unfairness of older vendors getting to keep all of their obviously fake reviews. The security badge program is no different to that. It’s yet another policy (of many I can document) that creates an anti-competitive marketplace.
Apps with more reviews and more installs in search results = more clicks = more installs = more $.
Apps with giant “THIS APP IS SECURE AND GOOD” badges = more clicks = more installs = more $.
I don’t know how you can possibly go one way with your opinion on reviews and the other with your opinion on security badges. The only explanation I can think of is that you forget how much $5k is and thus can’t empathise with new vendors.
If I were making the decisions on the marketplace I’d remove badges, installs and reviews from search results. Keep them in the individual app listing but make the search results view an equal playing field with relevant results randomised. Obviously the old guard would fight tooth and nail to prevent this because they know full well these things give them unfair competitive advantage.
None of those figures are genuine indicators anyway of whether one app is better than the other. Most installs = oldest app. Most reviews = oldest app with fake reviews they got to keep before Atlassian started enforcing it. Security badges = vendors who can afford it.
I’d also radically change the homepage since it’s mostly a static view that promotes the same old apps.
…
Part Deuce: The Details
You want the full badge journey with exact dates, figures, quotes and ticket numbers. Here you go…
All of this is documented in emails and tickets if anyone at Atlassian is intrigued to investigate.
When we first enquired on 16th December 2021 (AMKTHELP-41562) the wording of the program quite literally said the bug bounty deposit was “$5000 PER APP” (a wayback link because they later changed the wording).
At the time we had 362 apps though I explained most of these are built via a monorepo so the total repos across the portfolio is very small (thus any security risks were limited). Even today with 403 apps we only have 14 repos for those frontends. With the exception of our Forge apps they’re all static frontend apps, storing data exclusively inside Confluence/Jira with authentication set to “none” in the descriptor. We don’t store customer’s secret keys to external tools like some vendors do.
I knew from the beginning the apps were performant and probably secure. The only bounties we paid out ($700 total) in the first 24 hours were insignificant XSS oversights.
I enquired about the program, clearly explaining the situation and was told this exact quote:
To participate in the bug bounty program, you will be requested to fund a minimum of $5,000 USD per app.
So they wanted a $1,810,000 deposit (362 * $5k).
I said that was dumb and got back this offer from the actual PM this time, again an exact quote:
We could potentially ask Bugcrowd to break your apps into various programs (maybe 5-7?), and then have a starting reward pool of $5/k per program.
So depending on how that’s interpreted (they never clarified despite questions) they either wanted $25,000 to $35,000 or $258,000 to $362,000
I explained that this second offer was still bloody ridiculous, went on a bit of a rant, said I want all my apps to go into a single $5000 deposit pool. Asked if that could happen, and was then ghosted by the project manager for the Cloud Security program. That ticket alone dragged on from 16th December 2021 until 3rd March 2022. Never got a reply. Look up AMKTHELP-41562 for Atlassian’s who want to verify this one.
13th June 2022 I figured f*ck it, I’ll try again and just split the apps into three distinct pools with $5,000 each (ECOHELP-368, ECOHELP-369, ECOHELP-370). That worked.
I did ask at some point what the lowest I could go on the deposit pool was and the BugCrowd rep told me, quote:
This is the minimum for all Atlassian engagements on the Bugcrowd platform. Since we will be launching 3 separate engagements the total would be 15,000USD.
Those bounties didn’t go live until 23rd September 2022. That is 102 days from first submission or 281 days from the first enquiry about joining the program.
We then immediately applied for Cloud Fortified on 1st November 2022 (ECOHELP-6358) which was also a bureaucratic nightmare. Those badges were not applied until 23rd February 2023: 114 days later.
So from first unbadged contact to Cloud Fortified badges it was 434 days exactly.
And like I said, apart from fixing two minor XSS oversights, nothing about the security or performance of the apps was changed in those 434 days. They would have qualified for Cloud Fortified status from the moment they went live on the marketplace. I’m sure that’s the case with most new apps on the marketplace (particularly Forge apps or any with authentication: none in the descriptor) which further suggests this program is virtue signalling at best and anti-competitive at worst.
Again, bug bounties are useful and important. They helped us catch some minor things. But if I were making the policy and the goal was actually to secure apps on the marketplace: I’d start by removing the badges from search results, and then enrol ALL apps by default into bounties.
Keep it simple: one pool per vendor regardless of app count. If they voluntarily want to add more pools, so be it. Charge vendors with higher revenues a pool premium and use that to subsidise all other vendors. Smaller vendors would be assigned small researcher pools and perhaps lower bounty payouts. As soon as smaller vendors have the revenues to cover their own deposit pool, they do so.
