NoClassDefFoundError for Spring JDBC classes since Confluence 7.1.1

Hi all,

Since Confluence 7.1 we’re having a few customers of an app report of a NoClassDefFoundError for SingleConnectionDataSource (part of Spring JDBC).

In my pom.xml, I’m on Confluence 7.2 and AMPS 8.0.2. We are doing this in our code:

DataSource dataSource = new SingleConnectionDataSource(connection, false);

This is where we sometimes receive a NoClassDefFoundError. I could reproduce this once on my test system. But it seems that it’s flaky because now it works again. What is interesting is that it seems like this does not work in case the NoClassDef is thrown:

Class.forName("org.springframework.jdbc.datasource.SingleConnectionDataSource")

However, this does work:

ClassLoaderUtils.loadClass("org.springframework.jdbc.datasource.SingleConnectionDataSource", this.getClass());

Somehow this must be a classloader conflict topic. I have also created a Maven dependency tree and it reveals a conflict within the Confluence 7.2 dependency with Spring JDBC 2.x and 5.x:

Can anybody help?

3 Likes

I created a minimal Confluence macro named “Spring JDBC Analysis”. Here’s a zip with the (Confluence Plugin) source code: https://cloud.scandio.de/s/eoAXgXYA2GJWesj

Here’s the jar file:

The macro have a few macro parameters with connection parameters for a SQL connection:

The above screenshot is a scenario that works. If it doesn’t work, it’ll be a macro execution error with NoClassDefFound for SingleConnectionDataSource (and other spring JDBC classes).

1 Like

The core functionality of our app is to connect to external databases and display results. Having this uncontrollable flakiness is really annoying for some of our customers. We already had a few support tickets about this.

We were able to solve this problem for some customers by simply telling them to re-install the apps. But currently there’s one customer for which we just can’t get it to work at all. We do not want other customers to encounter this problem as well.

Note that all customers who had this problem upgraded from Confluence 6.13+ -> 7.1.1+.

I hope the minimal macro + source code linked by @felix.grund makes it as easy as possible to reproduce this. (Even though it is super flaky and I still couldn’t get the error to happen on my own machine)

@nmansilla if you could point us to some Server folks who might know how to help with this we would be super grateful.

I’ve pinged the team to see if they can chime in on this issue.

2 Likes

Awesome @nmansilla! Thanks! And also thanks @sven.schatter for following up!

In a convo with one of the engineers, the advice was that you should use confluence-plugins-platform-pom <depdendencyManagement> which helps ensure you’re using the exact versions published for you… that this a runtime class loader issue, rather than a buildtime dependencies problem, and it might depend on what parts of your plugin initialize in what order.

I hope that provides some guidance. If not, please follow up, and I’ll do my best to get the engineer to engage here on this thread. He’s not currently on the dev community forums. #tskTsk

1 Like

@nmansilla, our app uses JdbcTemplate and it cannot work on Confluence 7.1+
In the log it shows: java.lang.NoClassDefFoundError: org/springframework/jdbc/core/JdbcTemplate
Do you have any hint or suggestion for me?
Thank you very much!

I tried using confluence-plugins-platform-pom but it doesn’t make a difference. Right now I have two ways that both work around the issue successfully, but are both pretty ugly.

  1. Use Maven Shade Plugin to compile my o
  2. Use a reflection fallback if a NoClassDefFound error occurs

Right now I’m sticking to approach 2. I’ll just paste the JavaDoc I just wrote. It’s also very weird to me that this issue appears in a bugfix version (7.1.1) for the first time. I’m wondering what has changed in that version?

/**
 * This class was implemented as a follow-up workaround for PQCSRV-10, associated with a few support tickets where
 * Spring JDBC classes could not be loaded. There was also a Community post created here:
 * https://community.developer.atlassian.com/t/noclassdeffounderror-for-spring-jdbc-classes-since-confluence-7-1/34241
 *
 * The problem started with Confluence 7.1.1 and a very flaky NoClassDefFoundError for SingleDataSourceConnection. I
 * was able to reproduce the problem after updating Confluence from 7.1.0 to 7.2.0. However, sometimes a plugin
 * reinstall solves the problem. I was able to get to the point where I had two running Confluence instances, one with
 * the problem and one without. I digged quite deep into the world of OSGI, Felix, and Apache Service Mix, but still
 * couldn't find what the difference between those two instances was. The answers on Community and on Marketplace
 * Vendors Slack channel didn't help either.
 *
 * This class has a workaround solution for the problem: it checks whether all required Spring JDBC classes can be
 * loaded with the current class loader (Class.forName). This is the switch defining whether the NoClassDef occurs.
 * If Class.forName fails (and the Spring JDBC classes cannot be obtained from the current class loader), a reflection
 * process using the parent class loader is invoked.
 *
 * There is one alternative approach that we have found would work: using a shaded Spring JDBC dependency. But this has two major issues:
 * 1) It compiles all the required sources of Spring JDBC into the jar file which will end up with an overhead of ~3MB.
 * 2) We cannot be sure whether the actually compiled version of Spring JDBC will cause any conflicts with the
 *    Confluence environment.
 *
 * I know that this approach for sure isn't a great way to solve this issue, but it seems to work quite well in practice.

Thanks for your suggestion also @nmansilla. Unfortunately no luck, though.

@elitesoftsp you are also on 7.1.1 and not 7.1.0, right? I just double checked with the support tickets we received and every customer that had this issue seems to be on 7.1.1 as well.

I’m not 100% sure, but could this maybe even have been introduced in 7.1.0->7.1.1? Right now it looks like this issue is not happening on 7.1.0 and below. (The list of issues solved in 7.1.1 doesn’t seem to contain anything that could be associated with this though…)

@nmansilla Maybe it’s indeed time for the mentioned engineer to get on the Community… :stuck_out_tongue_winking_eye:

@sven.schatter you’re correct. We tested this issue with 7.1.1 and 7.2.0

1 Like

Have just tested and see that it works well on Confluence 7.1.0

1 Like

Additional info (thanks to @scott.dudley):

7.1.1 upgrades atlassian-plugins-osgi from 5.3.2 to 5.3.4, which seems to include changes in how transformed plugins are scanned and processed (particularly in ScanDescriptorForHostClassesStage.java)

One other possibly-relevant point is that the class does seem to be accessible normally from a plugin built with Spring Scanner, based on one test in which a transformed plugin was failing but the spring-scanner-based .JAR was loaded successfully. I admit I don’t know if this was just a fluke or if it points to something deeper.

1 Like

The problem is that it is really hard to tell about such things because it is flaky - most of the time it also works without Spring scanner. Personally, I have a strong feeling that a change from XML-based component declaration to Spring scanner wouldn’t solve this issue.

1 Like

Here is an instance of another vendor (@aragot) getting hit by the same or at least a very similar issue: https://playsql.atlassian.net/browse/PLAYSQL-288

1 Like

Superthanks, Sven, so I’m not crazy! We did test with Confluence 7.0.1 and next ones! It could also be due to what DB is being used because Spring JDBC might be activated slightly differently depending on whether a direct JDBC or a JNDI-throught-Tomcat connection is made.

Anyway, I bypassed the problem by avoiding using Spring JDBC, so thanks for doing the inventory on this problem. (and I feel like my release notes are watched closely, now ;))

1 Like

Yes! We actually had the same reaction. Of course we tested our apps with the newer versions as well and were unable to reproduce this problem during that phase. I really hope that this doesn’t create a situation where we can’t trust our own tests anymore as we will only find out about flakiness like this once it hits a customer.

We also thought about that and we found our own workaround as mentioned before. However, this can’t be the solution, right? Right now it seems to be only Spring (JDBC) dependencies. But what if this problem appears with other dependencies as well? That would be quite concerning. How are we supposed to know what dependencies “work reliably in dev” but “are flaky in production”? :frowning_face:

I would love to get to the bottom of this and understand what’s causing this problem. As this obviously seems to impact several vendors, it would be awesome to get confirmation on whether Atlassian is still trying to look into this (I think we all understand that this would take some time of course) or if we are expected to find our own workarounds.

1 Like

We now had it fail completely on a few more customers and rolled out our own workaround.

However, it would still be great to get an official statement on this of some sort. :confused:

We are facing other OSGi-related problems in a Jira app. The symptoms are different, but the root cause can be the same.

I reported this to Atlassian as: https://ecosystem.atlassian.net/servicedesk/customer/portal/14/DEVHELP-4024

If anyone is affected by problems that are related to OSGi and classloading in the Confluence versions released in 2019 Dec, drop me a message and I can add you to that ticket. I guess the more vendors affected, the higher priority it will get.

1 Like

My team mate @gergely.juhasz found an interesting workaround! If you downgrade one of the OSGi-related JARs shipped with Jira, the problem is gone! So it seems we manage to narrow down the cause to a JAR and even to a single method (related to transforming the apps)!

(It was inspired by @scott.dudley’s findings. Thanks, Scott!)

See the details in the comments at: https://ecosystem.atlassian.net/servicedesk/customer/portal/14/DEVHELP-4024

Of course, it is not a practical to ask the end-user to stop his Jira, replace a JAR, then restart Jira. Plus, it is difficult to communicate this as “compatibility”, but even with that it is a workaround that could eventually save some time until the problem gets fixed.

We hope this helps others.