We at re:solution are also very concerned with the changes proposed by Atlassian.
Not exporting embedded crowd would break almost all our DC apps (18 apps) including SAML SSO and UserSync, with a heavy influence on our company and business (just look at the install counts).
Our DC apps rely heavily on this dependency to work with users and groups. There is no alternative to that (apart from interacting with the DB tables directly - which is something you do not wish to do …). The product-specific APIs also do not help with working with directories.
It is not only bad for us as a company, but also for our customers. SAML SSO Jira and UserSync (but also API tokens and all our other apps) are essential to many of the biggest companies worldwide. They rely on our apps for syncing users, working with groups, or interacting with directories. This is made possible by embedded crowd. Without it, no chance.
I know that Atlassian wants to clean up some things, but this and all the other changes feel rushed and not well prepared.
The communication strategy also feels strange, it is very hard to keep up with the different articles, community threads, etc. - especially since Atlassian does breaking changes for Confluence and Jira at the same time.
As written there, the 8.8 beta release notes state that user directory passwords will be encrypted, and since we need the password to access LDAPs that require authentication, I wonder where to obtain information about how to handle this?
First of all, I wanted to thank you for your detailed reply. It is becoming an increasingly rare occurrence for the ecosystem to be able to communicate directly with people who are making platform decisions, and the fact that you took the time to reply here (and are open to listening) is fantastic.
I also wanted to emphasize that I understand the need to make breaking changes in platform upgrades, and that the platform team has a herculean task of trying to upgrade a decade’s worth of code cruft. Thanks for your efforts in making that happen.
For context, I probably pay more attention than most people to the EAP posts here on CDAC because I have a product that is (for better or for worse) tightly intertwined with a bunch of Confluence internals. I have also been shipping multiple products in the Atlassian ecosystem over the last 14 years, and in that time, I have not even once been forced to branch code or ship multiple JARs to simultaneously support different major host product versions. If at all possible, I would like to keep it that way.
With regards to the WRM, my specific concern relates to changing the public signature of types that are required at plugin load time. (After load time, we can find other ways to manage modified signatures, so those are less of a concern, but we don’t have control at load time.)
That a bunch of module descriptors types are “just” moving to a different package may or may not be a big deal: if Atlassian will support both the old and the new signatures, and you can commit to supporting the old signature for roughly two years of product releases, then there is no problem. If support for the old signature is going away soon, then I do have a concern.
This is intertwined with the deprecation issue and the lack of vendor certainty about exactly what is happening. Historically, in platform releases, a bunch of code has been marked as deprecated in the initial x.0 release of the platform, to be removed in the next major (x+1).0, which usually takes a year or two. Now, vendors are first getting the deprecation signaling at the last or close-to-last minor release of x.y, with (x+1).0 to be released in only a couple of months. The difference is significant due to the timing of support commitments to our mutual customers.
The confusion is fueled by multiple, competing Atlassian sources with slightly different views of what may or may not be going away:
There have been around three or four different lists published on DAC/CDAC with OSGi packages supposedly being removed from exports in Confluence 9.0.
The product itself ships with a .YAML file describing which packages will actually not be exported, which results in warnings in the logs at plugin load time (that may be different from the lists above).
The product platform POM was modified to remove transitive dependencies, which generate a set of undefined references at compile time. This list is also slightly different from the above.
And it seems that every time I compile a new build of the latest 8.x series, a new pile of warnings about @Deprecated methods turns up (which is how I discovered the WRM changes).
The messaging has also shifted from the initial “we are removing certain 3rd-party libraries from export” to “we are removing 3rd-party libraries and also a ton of Atlassian packages”, so there is a bit of a feeling that the goalposts are getting moved.
As noted earlier, the fact that sources are not readily available means that the @Deprecated detailed annotations are not easily visible. Without any further information, the conservative approach has to be to assume that all of these methods/classes are being nuked for 9.0. (If we stick our heads in the sand and assume that everything is going to be OK, but Atlassian actually does ship a product that has removed one critical package on which our product depends, then it’s too late and the cat does not get put back in the bag.)
Knowing that the WRM package is open-source and that sources can be found elsewhere is helpful (thanks!), but for context, most vendors also have a list of a dozen other things that need to be fixed. Perhaps they are open-source too, but then I (and every other vendor) need to go figure out if they are really open source, go hunting in the Maven repo or track down the correct bitbucket repository, etc. I know that you can’t do anything about the source distribution yourself, but I wanted to emphasize why it is important.
I also recognize that the overall problem you are dealing with is very difficult: it’s a huge ecosystem with many moving parts, and Atlassian is trying to put the horse back into the barn for APIs that it would rather not have vendors use. I further recognize that there have been a dozen CVEs issued since the “grey api” topic was first mentioned and that there are probably multiple competing pressures from the security team and your boss’s boss’s boss to clean this stuff up sooner rather than later. Communicating changes of this magnitude is not easy and I don’t know if I could have done a better job.
All of this is to say that I still have no idea of where the WRM changes fit into this scheme. To ask the question point-blank, will the WRM methods/classes marked as @Deprected in Confluence 8.8 be completely removed in whatever WRM version ships with Confluence 9.0?
If we only need two release branches, but there are (say) 2,000 plugins out there, then we have just created 2,000 new branches and a bunch of testing burden that (in an ideal world) does not necessarily need to be there.
I am still unclear about what is happening to the deprecated module descriptor types in Confluence 9.0, as mentioned above. I would like to propose three possible ways to address the concern without forcing every vendor to maintain two branches:
Agree to support both the old and the new signatures for long enough that vendors can achieve cross-product support (shipping in the product for about two years).
Consider that the existing module types are part of the long-used public API. Refactor whatever you need to do in the WRM, but don’t move the package or signature of the types that are part of that API (where possible, perhaps with exceptions where third-party types need to be removed). The package where those types have always lived is now effectively the API, and anything that’s not part of the API can be easily moved out of those packages. You still have API vs implementation separation, but it’s just that you haven’t changed the package names on which everyone else depends.
Remove the old signatures immediately in the host product, but introduce new module types in the atlassian-plugin.xml that are used exclusively for the new signatures, allowing a single plugin to support both. I believe I saw this approach used recently (in either Conf or Jira, I think?). For example, <web-item> would require the old com.atlassian.plugin.web.descriptors signature and <web-item2> would use the new com.atlassian.plugin.web.api.descriptors signature. Older host products would ignore <web-item2> and newer products would ignore <web-item>, so apps can ship both modules and it will work magically.
I guess one could also argue that none of this is necessary, and that vendors should just suck it up and maintain two branches. But doing this will save literally thousands of other people from having to do thousands or tens of thousands of hours of extra work.
I can also point out that shipping multiple versions of a product with different compatibility version ranges is not great from a customer UX perspective on MPAC. This is compounded by the fact that the last server-supported LTS releases have become catch-all kitchen sink versions, abandoning semver by allowing breaking changes to be regularly introduced into point releases. This means that an app might actually support Confluence 8.0.0-8.5.4 (but not 8.5.5 - 8.5.x due to a breaking change) and also 8.6.0-8.7.x, yet there is no way to mark this gap on MPAC.
If we end up in the situation where one of these signature changes from the “just create two branches” model is ever backported into (say) 8.5.5, then we need a new release that somehow supports 8.5.5-8.5.x as well as 8.8.0+, but not 8.6 or 8.7. Now we have to make at least four different releases on MPAC to specify contiguous version ranges for one software release (8.0.0-8.5.4 with old sigs, 8.5.4-8.5.x with new sigs, 8.6.0-8.7.x with old sigs, 8.8.0+ with new sigs), and possibly even create a third branch, depending on the extent of the changes. Nobody wants that.
Before you say “never”, I should add that this type of LTS backporting is exactly what has been happening with Struts changes over the last four months. Luckily, the majority of the Struts changes have been shoehorned into the product in a way that does not change the signatures.
I also further recognize that these semver-related problems are outside your control or even that of the Confluence team, and that everyone is just dealing with a (ehrm) confluence of decisions that did not intersect well. I assume that this situation is the result of the marketing decision not to allow Server customers to use anything past the last LTS, combined with the security team’s requirements to still get stuff merged into something that Server customers can actually use.
Is there any chance of getting more details about these other changes you mention, inasmuch as they will impact vendors?
Thanks @scott.dudley, for your well-written and detailed posts! I think it’s feedback like yours that really help Atlassian see what impact changes like the Gray API removal has. Much appreciated
To add Communardo’s perspective:
We definitely see the reasoning behind Gray API removal in general. Also, we appreciate that Atlassian has to make some hard decisions here. I wouldn’t want to be in your shoes.
But coming to specifics: Similar to @scott.dudley, @chrschommer and @BjrnDhler I also find that one of our most popular Confluence apps (~900 installs) will we broken if com.atlassian.crowd.embedded.api.* is unavailable without replacement.
Technical details:
In that app, we try to use javax.naming.* were possible, but afaik for some Confluence-specific stuff there is no app-available alternative to com.atlassian.crowd.embedded.api.* (Pretty please correct me, if I’m wrong!) Specifically for us, we use:
class Directory: isActive(), getName(), getAttributes(), getType(), getValue()
enum DirectoryType
I don’t see any alternatives to those APIs (except direct database queries). I’m open for suggestions.
Which ones in Confluence for the examples above?
Additionally to this Marketplace app, we also have at least a handful of customer-individual Confluence apps, that also use com.atlassian.crowd.embedded.api.*. They would also be broken. We foresee that we will have a hard time explaining to those customers why those apps won’t work with Confluence 9.x anymore.
To summarize: @Atlassian, are you really, really double-sure, that you want to hide com.atlassian.crowd.embedded.api.* for 3rd-party apps? Considering the substantial amount of broken apps?
I understand that you’re specifically referring to the signature of types. Yes, plugins will fail to load if their OSGi package-import requires the type instead of optionally importing during the resolution stage. Let me repeat what I said before, there are other changes that guarantee a plugin cannot be engineered cross-major-compatible which makes this discussion moot.
I know this is hard, but we (collectively) do not have time. The security posture of the DC products must improve, this is non-negotiable with customers.
I understand, I’ve raised this internally.
Yup, totally aware, was only meant to provide some workaround and perhaps a little reassurance. Let me repeat something though; once you’d added Atlassian’s maven registries in your IntelliJ settings it can grab the source JARs for most of our platform.
I’m not in control of the source build distribution, I’ve raised it again with the Confluence team. The only way I know for vendors to get source code ATM is to raise a support ticket.
Let me answer even more explicitly this time: it depends.
Most things are just changing packages.
Guava and Dom4J are removed but have Java or Atlassian replacements.
Code for dead features of the WRM are being removed; prebake (failed), failed-AMD support, (failed) BigPipe, Internet Explorer support.
Legacy transformers are also removed (UrlReading variants are safe). Web fragment conditions will not work on web-resources, but again UrlReading variants will work fine.
The curl web-resource is removed
Google closure compiler is removed (was only used in products)
A bunch of methods that have had API replacements for 5-10 years
Some product specific hacks #clearCache and #temporaryWayToGetGlobalsDoNotUseIt
That’s everything off the top of my head. The epic where all the work is tracked is already linked above, I work in public.
Ok, we might be needing an escalation path here, because although the security posture of DC products must improve, I do think we also all agree that BREAKING EVERY MAJOR APP ON THE ATLASSIAN MARKETPLACE is also non-negotiable with customers?
If you look closely at what @scott.dudley is arguing here is that apart from the nuisance of maintaining multiple branches / versions of apps (which we can overcome) there are specific situations in which Atlassian Marketplace will not allow us to cater to all the different situations that may arise because of this:
This is not a hypothetical DX issue, this is a real world problem with maintaining backwards compatibility for customers.
The problem I see is that the grey api change actively reduces the security posture of DC. It makes vendors responsible for patching, going from getting patches from a single vendor (Atlassian) to multiple vendors (all of the app vendors you’ve installed apps from the marketplace). I’ve not seen any argument put forward as to why this is a good thing for customers.
I second this. During the whole Log4J vulnerability debacle, we were able to tell customers that once they updated their DC instance, they would not need to update our apps because we used the provided dependency. With the removal of the grey API, this would mean that customers would need to identify vulnerabilities in each individual app, wait for vendors to fix it, and install the newest versions (which might not even be possible if the customer is not on the latest & greatest version of DC)
Unfortunately, I’m afraid we’ve already passed this station and this thing is going to happen anyway. But at least, hopefully, we can still make sure that Atlassian takes enough time (and gives us vendors enough time) to do this right.
I’m actually not referring to the OSGi-related issues since I know that those can be easily managed with optional imports. Let me rephrase my earlier question in an even more precise manner, using two specific examples:
Starting in Confluence 8.8, the com.atlassian.webresource.api.data.WebResourceDataProvider class is marked as @Deprecated. Will this class be removed in Confluence 9.0? (If you are “moving” it to a different package, that still counts as removal because the fully-qualified class above is gone.)
And the same question for com.atlassian.plugin.webresource.condition.UrlReadingCondition.
I am referring to the “loading” scenario because the first example is the required interface for <data> elements inside <web-resource>s, and the referenced class is unconditionally loaded during plugin startup before we could have a chance to do anything about it.
That seems like a pretty big deal that has not (until now) been announced anywhere. Is it possible to let us vendors know what those changes are? I poked around in the WRM projects but nothing jumped out at me as being impossible.
If you are indeed correct (I am not willing to give up quite yet though ), you implied that OBRs could help solve the distribution issue. Is there any documentation or even a sample for how to create a multi-version-targeting OBR? The existing documentation on OBRs seems scant and potentially obsolete (it still refers to the “Plugin Exchange” and P1 plugins), and it does not describe this use case.
I want, for example, to be able to create an OBR that will deploy variant1.jar on Confluence [8.0,8.5.4) and variant2.jar to Confluence [8.5.4,]. Is this possible?
I’m using Spring Java Configuration and I noticed that org.osgi:org.osgi.framework is not on the list in that doc but it has been removed. I needed to add an explicit dependency:
<dependency>
<groupId>org.osgi</groupId>
<artifactId>org.osgi.framework</artifactId>
<!-- not in confluence-plugins-platform-pom so we need to specify a version -->
<version>1.10.0</version>
<scope>provided</scope>
</dependency>
It is working fine in my build against 8.7.1, I need it for org.osgi.framework.ServiceRegistration so that I can do things like register/export a DiscoverableListener (for Cloud migration) in my Spring Java Configuration class.
Over the years, using Spring Java Configuration in my Confluence plugins, I have noticed that none of the Atlassian examples and few of the docs consider Spring Java Configuration. It makes me nervous that we might be going down this path to 8.7, 8.8, 9 without Spring Java Configuration in plugins being considered.
It is like Spring Java Configuration was the new thing so I switched to it to get ahead in the game, and then it became a bastard child and I was the only one on Earth who had switched to it. I get stuck all the time trying to figure out how to get something like a migration listener to work with Spring Java Configuration instead of Spring Scanner (because the examples always use Spring Scanner).
I am afraid of needing to back out Spring Java Configuration and go back to Atlassian Spring Scanner. Can we make sure that it doesn’t get missed during all of this?
That would be great, except the list is incomplete. Case in point, com.atlassian.fugue (as mentioned on the 9.0 thread) has been removed but it’s not listed in the link above.
EDIT: Upon rereading, I see that @turehoefner_appfire was referring to the list of transitive POM dependencies and not the to-be-blocked package exports, so perhaps the list in the post is complete with respect to those packages removed from the POM transitive list.
Regardless of whether the packages are still visible from the POM or not, some packages not listed there are not going to be available in the runtime of 9.0, so it’s a concern either way.
There are a lot of these lists floating around with variations, and it would be great if Atlassian could point vendors towards one central, authoritative list that will be maintained until release and always kept up-to-date.
Thank you for raising this, I’ll raise this internally with the Confluence team (sorry again, I don’t own this).
Certainly our documentation and replies here on CDAC have been supportive of moving to it over all the other alternatives. We use it a lot internally and we love it, that’s why we’ve been adding new features to it.
A quick update - there was an error in Confluence 8.7 where the com.atlassian.crowd.embedded.api package got wrongly moved to the deprecated list. We’re addressing this in the next 8.7 bugfix (if there is one), and it’ll be resolved in Confluence 8.8 as well.
But Confluence no longer exports the com.atlassian.json.marshal package (which I flagged two months ago). Furthermore, since apps now have to bundle the Guava dependency, there will be a class conflict on the Supplier superclass, even if we were able to access the Jsonable class.
This means that many <web-resource>s are broken and I would love to know what the replacement should be.
(I tried looking in various atlassian-bundled-plugins to see how Atlassian’s own code does it, but it looks like they use the old interface…and they have some sort of cheat code to allow them to access OSGi packages that are prohibited for third-party plugins.)
I am getting a new error now in my unit tests. I want to mock com.atlassian.confluence.xhtml.api.MacroDefinition (as I have always done) and get some fugue errors.
Regardless of what dependencies I add with scope=test this error stays. Also some other error is about the pool size and not being able to borrow a pool …
UPDATE: I got all three things resolved now. The first+second via my own compat lib.
The third one by setting this in the pom.xml:
This allows power mock reflection stuff in Java9+ and in the test classes I needed to add this class annotation:
@RunWith(PowerMockRunner.class)
@PrepareForTest({AuthenticatedUserThreadLocal.class, MacroUtils.class, MacroContextTransformHelper.class})
@PowerMockIgnore({ "javax.security.*", "javax.management.*"}) // <-- NEW NEW
public class MyTest {
Now compile wise + unit test wise everything runs.
