Include/exclude list for RFC-24 Data Center grey API removal

Continuing the discussion from RFC-24: Data Center Grey API Removal:

Hi @MateuszMiodek ,

I am hoping that everything is still on track to publish the list of excluded libraries in the near future.

In the meantime, in the latest Confluence 8.7 beta, I see that new log messages labeled with “osgi.hook.dmz.DmzResolverHook” appear when loading plugins. These messages seem to note which imported packages will be removed from the OSGi exports, such as the following:

2023-11-27 14:18:09,380 WARN [UpmAsynchronousTaskManager:thread-4] [osgi.hook.dmz.DmzResolverHook] filterMatches Package com.atlassian.crowd.embedded.api
is deprecated and will be made unavailable for export to plugin <PLUGIN> in a future release

Should we expect the current code to already contain the full and authoritative source of what will eventually be removed, or should we depend on the list that you are going to provide here on CDAC?

Thanks!
Scott

4 Likes

The irony is that it is not even possible to build a .JAR to remove some of the dependencies that the host is complaining about, even with the most recent version of AMPS that isn’t documented.

Using plugin SDK 8.2.7 + the seemingly-latest AMPS 8.12.0, there is a hardcoded list inside the amps-maven-plugin JAR (in class com.atlassian.maven.plugins.amps.BannedDependencies) of packages that you are not allowed to bundle, which include…the things that Confluence is now warning us we should bundle.

For example, Confluence says this:

2023-11-27 14:18:09,417 WARN [UpmAsynchronousTaskManager:thread-4] [osgi.hook.dmz.DmzResolverHook] filterMatches
Package org.apache.commons.lang3 is deprecated and will be made unavailable for export to plugin <PLUGIN> in a future release

But if you try to actually bundle it, Maven gives up this error:

[WARNING] Rule 0: org.apache.maven.plugins.enforcer.BannedDependencies failed with message:
make sure platform artifacts are not bundled into plugin
Found Banned Dependency: org.apache.commons:commons-lang3:jar:3.12.0
Use 'mvn dependency:tree' to locate the source of the banned dependencies.

Perhaps someone could also look into fixing the tooling?

1 Like

Hello Scott,

This project has been taken over by my colleague, @MalathiVangalapati. I shared your feedback with her.

Best regards,
Matesz

1 Like

Why is there no mention of this in the “Preparing for Confluence 8.7” documentation? Are these new warnings intended to be seen by vendors in this release?

A quick test with ScriptRunner gives me the following warnings:

Package com.atlassian.confluence.plugins.rest.manager is deprecated and will be made unavailable for export to plugin com.onresolve.confluence.groovy.groovyrunner in a future release
Package bucket.user is deprecated and will be made unavailable for export to plugin com.onresolve.confluence.groovy.groovyrunner in a future release
Package com.atlassian.crowd.embedded.api is deprecated and will be made unavailable for export to plugin com.onresolve.confluence.groovy.groovyrunner in a future release
Package com.atlassian.json.marshal is deprecated and will be made unavailable for export to plugin com.onresolve.confluence.groovy.groovyrunner in a future release
Package com.opensymphony.util is deprecated and will be made unavailable for export to plugin com.onresolve.confluence.groovy.groovyrunner in a future release
Package org.apache.commons.collections is deprecated and will be made unavailable for export to plugin com.onresolve.confluence.groovy.groovyrunner in a future release
Package org.apache.commons.collections.comparators is deprecated and will be made unavailable for export to plugin com.onresolve.confluence.groovy.groovyrunner in a future release
Package org.apache.commons.collections.iterators is deprecated and will be made unavailable for export to plugin com.onresolve.confluence.groovy.groovyrunner in a future release
Package org.apache.commons.collections.keyvalue is deprecated and will be made unavailable for export to plugin com.onresolve.confluence.groovy.groovyrunner in a future release
Package org.apache.commons.collections.map is deprecated and will be made unavailable for export to plugin com.onresolve.confluence.groovy.groovyrunner in a future release
Package org.apache.commons.lang3 is deprecated and will be made unavailable for export to plugin com.onresolve.confluence.groovy.groovyrunner in a future release
Package org.apache.commons.lang3.concurrent is deprecated and will be made unavailable for export to plugin com.onresolve.confluence.groovy.groovyrunner in a future release
Package org.apache.commons.lang3.exception is deprecated and will be made unavailable for export to plugin com.onresolve.confluence.groovy.groovyrunner in a future release
Package org.apache.commons.lang3.math is deprecated and will be made unavailable for export to plugin com.onresolve.confluence.groovy.groovyrunner in a future release
Package org.apache.commons.lang3.text is deprecated and will be made unavailable for export to plugin com.onresolve.confluence.groovy.groovyrunner in a future release
Package org.apache.commons.lang3.tuple is deprecated and will be made unavailable for export to plugin com.onresolve.confluence.groovy.groovyrunner in a future release
Package org.apache.commons.pool is deprecated and will be made unavailable for export to plugin com.onresolve.confluence.groovy.groovyrunner in a future release
Package org.apache.commons.pool.impl is deprecated and will be made unavailable for export to plugin com.onresolve.confluence.groovy.groovyrunner in a future release
Package org.apache.oro.text is deprecated and will be made unavailable for export to plugin com.onresolve.confluence.groovy.groovyrunner in a future release
Package org.apache.oro.text.regex is deprecated and will be made unavailable for export to plugin com.onresolve.confluence.groovy.groovyrunner in a future release
Package org.cyberneko.html.parsers is deprecated and will be made unavailable for export to plugin com.onresolve.confluence.groovy.groovyrunner in a future release

Most of these seem easy enough to resolve by bundling Apache Commons.

I echo the initial question from @scott.dudley, when can we expect the scope of these upcoming changes to be communicated in full, including a list of packages destined to no longer be OSGi exported.

3 Likes

For reference, the current full list of deprecated packages for which warnings are issued can be found in confluence/WEB-INF/lib/com.atlassian.confluence_confluence-8.7.0-beta1.jar, in the deprecated section of public-api.yaml. (I don’t even want to know: why YAML?!)

deprecated:
  - bucket.*
  - com.atlassian.activeobjects.confluence.*
  - com.atlassian.config.util.*
  - com.atlassian.confluence.ext.code.*
  - com.atlassian.confluence.extra.webdav.*
  - com.atlassian.confluence.importexport.*
  - com.atlassian.confluence.notifications.batch.*
  - com.atlassian.confluence.plugins.collaborative.content.feedback.*
  - com.atlassian.confluence.plugins.content_report
  - com.atlassian.confluence.plugins.edgeindex.*
  - com.atlassian.confluence.plugins.gadgets.refimpl
  - com.atlassian.confluence.plugins.jirareports
  - com.atlassian.confluence.plugins.hipchat.*
  - com.atlassian.confluence.plugins.mobile.analytic
  - com.atlassian.confluence.plugins.remotepageview.jwt
  - com.atlassian.confluence.plugins.rest.*
  - com.atlassian.confluence.plugins.softwareproject
  - com.atlassian.confluence.plugins.soy.*
  - com.atlassian.confluence.upgrade.*
  - com.atlassian.confluence.userstatus.*
  - com.atlassian.confluence.vcache.*
  - com.atlassian.confluence.velocity.*
  - com.atlassian.confluence.warming
  - com.atlassian.crowd.embedded.*
  - com.atlassian.db.*
  - com.atlassian.dragonfly.*
  - com.atlassian.h2.*
  - com.atlassian.favicon.*
  - com.atlassian.hazelcast.*
  - com.atlassian.hsqldb.*
  - com.atlassian.json.*
  - com.atlassian.migration.agent.*
  - com.atlassian.plugins.roadmap.upgradetask
  - com.atlassian.mywork.*
  - com.atlassian.plugin.notifications.spi
  - com.atlassian.sal.confluence.*
  - com.atlassian.threadlocal.*
  - com.atlassian.velocity
  - com.google.gson.*
  - com.ibm.wsdl.*
  - com.opensymphony.module.*
  - com.opensymphony.oscache.*
  - com.opensymphony.provider.*
  - com.opensymphony.sitemesh.*
  - com.opensymphony.util.*
  - com.rometools.*
  - com.sun.imageio.*
  - com.sun.media.*
  - com.sun.syndication.*
  - javax.json.*
  - javax.jws.*
  - javax.xml.soap.*
  - net.sf.cglib.*
  - org.apache.axis.*
  - org.apache.batik.*
  - org.apache.commons.collections.*
  - org.apache.commons.collections4.*
  - org.apache.commons.compress.*
  - org.apache.commons.digester.*
  - org.apache.commons.discovery.*
  - org.apache.commons.httpclient.*
  - org.apache.commons.io.*
  - org.apache.commons.jrcs.*
  - org.apache.commons.lang.*
  - org.apache.commons.lang3.*
  - org.apache.commons.math3.*
  - org.apache.commons.pool.*
  - org.apache.fontbox.*
  - org.apache.jackrabbit.*
  - org.apache.oro.*
  - org.apache.pdfbox.*
  - org.apache.regexp
  - org.apache.xmlrpc.*
  - org.apache.xmpbox.*
  - org.bouncycastle.*
  - org.cyberneko.html.*
  - org.jdom.*
  - org.jdom2.*
  - org.slf4j.bridge.*
  - org.slf4j.impl.*
2 Likes

Hi @scott.dudley , thanks for checking in! I’ve been taking over this piece of work from @MateuszMiodek and will share an update in the coming weeks on the list of libraries we’re looking to remove. The plan is to keep the partner community posted whenever we have a list, even if it’s not the final one.

That said, here are some of the libraries we’re considering for removal - it’s not set in stone yet, but it’s an initial pass on the main Java packages we’re aiming to get rid of long with some reasons behind it. :

  • com.google.common (guava): Our preference lies in leveraging Java replacements to streamline our dependencies.
  • com.google.inject - google.inject is not used in a wholistic way in all our products and is sparsely used
  • com.rometools: We perceive limited value in managing and sustaining this particular library.
  • com.sun.jersey: This component is antiquated and no longer maintained.
  • org.apache.commons.digester
  • org.codehaus: Similarly outdated and lacking maintenance.
  • org.dom4j: Utilized within a specialized context, to be replaced by our proprietary code.
  • org.joda.time: Java offers a native substitute. We aim to simplify our dependencies.
  • com.opensymphony.module.propertyset
  • com.opensymphony.module.sitemesh
  • com.opensymphony.sitemesh
  • org.apache.commons.lang
  • org.apache.commons.pool
  • org.apache.log4j
  • org.jdom
  • org.tuckey.web.filters.urlrewrite

Reiterating an earlier point from the RFC, our initiative involves reducing dependencies to fortify our security posture.

As I understand, the purpose of these alerts is to provide advance notice of forthcoming changes, affording vendors sufficient time to prepare. I can guarantee that there will be no removal of packages until the release of the next major versions of each of the products- such Jira 10, Bitbucket 9 ,Confluence 9 in this case . Any removals will be specifically carried out in conjunction with a major release and will be preceded by advance communication.

Regarding your other questions

Should we expect the current code to already contain the full and authoritative source of what will eventually be removed, or should we depend on the list that you are going to provide here on CDAC?

Currently, we’re informing partners about the modifications in the public API of shared components used in our various products. We’re collaborating with the product teams to create a comprehensive list of changes to the libraries in different releases. However, it’s important to stay updated by checking the release notes for each specific product (like Jira, Confluence, etc.) to understand any specific tweaks they might introduce.

Perhaps someone could also look into fixing the tooling?

We apologize for the inconvenience caused. You’re correct in pointing out that we need to update the tooling (AMPS) to align with the anticipated changes in dependencies. As a temporary fix, you can add those dependencies to the exclude list. More details here - More Info on AMPS Banned Plugin Dependency - #6 by aswan

Why is there no mention of this in the “Preparing for Confluence 8.7” documentation? Are these new warnings intended to be seen by vendors in this release?

Once more, sorry for the inconvenience. As previously stated, we’re actively working with the product teams to compile an extensive list of changes to the libraries across various releases. We’ll share this list in the upcoming weeks. For now, here’s a preliminary list of libraries that will be deprecated in Confluence versions 8.7 to 8.9. Rest assured, we’ll continue to keep the community informed as we gather more specific details.

  • com.addonengine.*
  • com.atlassian.activeobjects.confluence.*
  • com.atlassian.config.util.*
  • com.atlassian.confluence.ext.code.*
  • com.atlassian.confluence.extra.webdav.*
  • com.atlassian.confluence.importexport.*
  • com.atlassian.confluence.notifications.batch.*
  • com.atlassian.confluence.plugins.collaborative.content.feedback.*
  • com.atlassian.confluence.plugins.content_report
  • com.atlassian.confluence.plugins.edgeindex.*
  • com.atlassian.confluence.plugins.gadgets.refimpl
  • com.atlassian.confluence.plugins.jirareports
  • com.atlassian.confluence.plugins.hipchat.*
  • com.atlassian.confluence.plugins.mobile.analytic
  • com.atlassian.confluence.plugins.remotepageview.jwt
  • com.atlassian.confluence.plugins.rest.*
  • com.atlassian.confluence.plugins.softwareproject
  • com.atlassian.confluence.plugins.soy.*
  • com.atlassian.confluence.upgrade.*
  • com.atlassian.confluence.userstatus.*
  • com.atlassian.confluence.vcache.*
  • com.atlassian.confluence.velocity.*
  • com.atlassian.confluence.warming
  • com.atlassian.crowd.embedded.*
  • com.atlassian.db.*
  • com.atlassian.dragonfly.*
  • com.atlassian.fugue.*
  • com.atlassian.h2.*
  • com.atlassian.favicon.*
  • com.atlassian.hazelcast.*
  • com.atlassian.hsqldb.*
  • com.atlassian.json.*
  • com.atlassian.migration.agent.*
  • com.atlassian.plugins.roadmap.upgradetask
  • com.atlassian.mywork.*
  • com.atlassian.plugin.notifications.spi
  • com.atlassian.sal.confluence.*
  • com.atlassian.threadlocal.*
  • com.atlassian.util.concurrent.*
  • com.atlassian.velocity
  • com.ctc.wstx.*
  • com.google.common.*
  • com.google.gson.*
  • com.ibm.wsdl.*
  • org.joda.time.*
  • com.octo.captcha.*
  • com.opensymphony.ejb.*
  • com.opensymphony.module.*
  • com.opensymphony.oscache.*
  • com.opensymphony.provider.*
  • com.opensymphony.sitemesh.*
  • com.opensymphony.util.*
  • com.rometools.*
  • com.sun.imageio.*
  • com.sun.media.*
  • com.sun.syndication.*
  • graphql.schema.*
  • io.atlassian.fugue.*
  • javax.json.*
  • javax.jws.*
  • javax.xml.messaging.*
  • javax.xml.rpc.*
  • javax.xml.soap.*
  • net.sf.cglib.*
  • org.antlr.*
  • org.aopalliance.*
  • org.apache.axis.*
  • org.apache.batik.*
  • org.apache.bcel.*
  • org.apache.commons.*
  • org.apache.el.*
  • org.apache.felix.*
  • org.apache.fontbox.*
  • org.apache.html.*
  • org.apache.http.*
  • org.apache.jackrabbit.*
  • org.apache.oro.*
  • org.apache.pdfbox.*
  • org.apache.regexp
  • org.apache.tools.*
  • org.apache.wml.*
  • org.apache.xalan.*
  • org.apache.xerces.*
  • org.apache.xml.*
  • org.apache.xmlcommons.*
  • org.apache.xmlgraphics.*
  • org.apache.xmlrpc.*
  • org.apache.xmpbox.*
  • org.apache.xpath.*
  • org.bouncycastle.*
  • org.cyberneko.html.*
  • org.dataloader.*
  • org.eclipse.gemini.*
  • org.hibernate.validator.*
  • org.ietf.*
  • org.j3d.*
  • org.jdom.*
  • org.jdom2.*
  • org.json.*
  • org.jsoup.*
  • org.slf4j.bridge.*
  • org.slf4j.impl.*

Thanks,
Malathi Vangalapati

3 Likes

Thanks @MalathiVangalapati this is a great help.

Will anybody from Atlassian involved in these changes be at AtlasCamp next week? We have some details specific to ScriptRunner apps that would be good to discuss.

Primarily we’re looking at adding analysis to the app to measure the potential customer impact of these changes, as customer scripts have access to anything from the host app classpath.

@rlander - @MarekTokarski and I will be at the DC booth in Atlas Camp.

2 Likes

Hi @MalathiVangalapati

Thanks for the list. I took a quick glance at the proposed list and here is some preliminary feedback:

com.atlassian.crowd.embedded.api is an API (it even has “API” in the name :wink: ). Is the intent to remove this from use? If so, what is the suggested alternative for performing (for example) user searches?

com.atlassian.json.marshal.Jsonable is the return type in the API contract of WebResourceDataProvider classes.

com.atlassian.velocity.htmlsafe.HtmlSafe is a public annotation used when writing Velocity helpers. We also use some other parts of this package unrelated to annotations.

3 Likes

Hi Scott. Thanks for the feedback. Part of of this “gray api” removal process is to identify those components that we think shouldn’t be part of the API, but for which there’s no acceptable provided alternative, so there may so back and forth before we get it right.

To address your specific examples, the Embedded Crowd API is very much intended as an internal-only API, to be used by Confluence to talk to an external Crowd server. While it is accessible to plugins, this was never the intention, and in fact could return some odd results if used in that fashion.

The recommended alternative is to use PersonService and Group service from confluence-java-api. If these don’t provide the functionality you need, that’s always helpful know so we can address it.

As for WebResourceDataProvider and Jsonable and HtmlSafe, you’re right, there’s a mismatch there. We’re still in the process of finding and fixing such “API leaks”, and we’ll continue to do so leading up to Confluence 9.0.

Obviously, we don’t want to break any such functionality by accidentally hiding any packages needed to make these public APIs work. The package list will be in flux up until then, and all feedback around this is very helpful.

Please see the new published blog post on this topic, Get your apps ready for Gray API removal

3 Likes

Hi @kmacleod

Thank you very much for your response, as well as for publishing the official exclusion list.

I have not yet had time to digest the blog post, but with regards to switching from the embedded Crowd service to PersonService or GroupService:

I should start by mentioning that PersonService, Expansion and a bunch of their dependencies are still marked as @ExperimentalApi, even in Confluence 8.6. I imagine that this is an oversight?

Assuming that this is the correct path forward, how would one perform a substring match based on partial user displayname or group name? The use case is a user picker or group picker. The Crowd equivalent is:

Query<User> query = QueryBuilder.queryFor(
        User.class,
        EntityDescriptor.user()
    )
    .with(
        Restriction.on(UserTermKeys.DISPLAY_NAME)
            .containing(searchTerm)
    )

and

GroupQuery<Group> groupQuery = new GroupQuery<>(
    Group.class,
    GroupType.GROUP,
    Combine.allOf(
         Restriction.on(GroupTermKeys.NAME).containing(searchTerm),
         Restriction.on(GroupTermKeys.ACTIVE).exactlyMatching(true)
    ),
    0,
    MAX_SEARCH_RESULTS
);
5 Likes

Seeing that this topic has some traction on thread Confluence 8.8 release EAP available now, I’d like to point your attention to the vendor’s feedback there. (I think this thread here is more suited for such feedback, but for whatever reason the feedback’s coming in there)