PSA - Confluence moves to Struts in 8.0 - EAP

Hello everyone,

We are excited to announce that our team is working on moving to Struts 2 from Webwork/XWork in Confluence 8.0.

Let’s first look at a brief history, Webwork2 is an unmaintained library with fairly no public release in more than a decade(last released 2007) and hence we are vulnerable to new exploits and also have to maintain it ourselves. However, on the contrary Struts2 basically, a fork of Webwork2 and an extension get far more active contribution and active security releases. Hence, we are making the move to Struts2.

We have drafted a page to maintain all the updates to the Struts work in 8.0: Struts 2 upgrade. For EAP, 8.0.0-struts-m020 has been released as an 8.0 milestone with Struts.

For full 8.0 EAP updates, please watch Preparing for Confluence 8.0

Note: We will release Struts EAPs as well as regular 8.0 EAPs until Struts work is merged with 8.0.

Thanks,
Ganesh
Senior Engineer, Confluence DC

2 Likes

Hi @ggautam

in version 8.0.0-struts-m20 on every administration page I get errors like in a file error.txt (137.6 KB)


ATLAS Version:    8.2.7
ATLAS Home:       /Applications/Atlassian/atlassian-plugin-sdk
ATLAS Scripts:    /Applications/Atlassian/atlassian-plugin-sdk/bin
ATLAS Maven Home: /Applications/Atlassian/atlassian-plugin-sdk/apache-maven-3.8.6
AMPS Version:     8.1.2
--------
Executing: /Applications/Atlassian/atlassian-plugin-sdk/apache-maven-3.8.6/bin/mvn --version -gs /Applications/Atlassian/atlassian-plugin-sdk/apache-maven-3.8.6/conf/settings.xml
Apache Maven 3.8.6 (84538c9988a25aec085021c365c560670ad80f63)
Maven home: /Applications/Atlassian/atlassian-plugin-sdk/apache-maven-3.8.6
Java version: 11.0.15, vendor: Azul Systems, Inc., runtime: /Library/Java/JavaVirtualMachines/zulu-11.jdk/Contents/Home
Default locale: en_PL, platform encoding: UTF-8
OS name: "mac os x", version: "12.5", arch: "aarch64", family: "mac"

pom.xml

...
            <plugin>
                <groupId>com.atlassian.maven.plugins</groupId>
                <artifactId>confluence-maven-plugin</artifactId>
                <version>${amps.version}</version>
                <extensions>true</extensions>
                <configuration>
                    <productVersion>${confluence.version}</productVersion>
                    <productDataVersion>${confluence.data.version}</productDataVersion>
                    <jvmArgs>-Xms2048m -Xmx6072m -Duser.language=en -Duser.country=US -Dconfluence.browser.language.enabled=false</jvmArgs>
                    <enableQuickReload>true</enableQuickReload>
                    <instructions>
                        <Spring-Context>*</Spring-Context>
                    </instructions>
                </configuration>
            </plugin>
...
<amps.version>8.6.0</amps.version>

Cheers
Adam

2 Likes

Hi @adam.labus,

Thanks for reporting it. Its a known issue: [CONFSERVER-79610] Fix Classcast (class java.math.BigInteger cannot be cast to class java.lang.String) - PluginResourceDownload.serveFile - Create and track feature requests for Atlassian products.. We plan to work on it in next sprints. I would recommend you to watch this issue for further updates.

Thanks,
Ganesh

3 Likes

Hi @ggautam, thanks a lot for the heads-up, but this is a bit out of the blue tbh.

Will there be a compatibility library for this? Several of our apps use classes from the packages mentioned in the upgrade guide (especially ServletActionContext which is used in task threads to re-establish context used by other 3rd party apps etc.).

Of course I understand the need for updating libraries and frameworks in major versions, but given that these classes were never marked deprecated (obviously, because they aren’t from Atlassian) I suspect that more vendors are using them in their apps.

It would be nice if this could be upgraded in a backwards compatible way (for example using a shim library delegating to the new classes), because otherwise you put a lot of effort on each vendor who still needs to support Confluence 7 - which is likely each of us. The effort we’d be facing would be either in re-implementing a compatibility lib, using reflection hacks, or in the worst case the need to maintain two code branches for Confluence 7 & 8.

Looking forward to your response.
Jens

7 Likes

+1 on a backwards-compat solution for migration.

IMO, forking plugin source code to support 7.x and 8.x is not a fair requirement to put on partners.

Please reconsider the migration plan and see if it would be possible to provide a compat-lib that we can use for a backwards compatible code refactor.

Thanks!

3 Likes

Hi @jens / @turehoefner_appfire,
Thanks for providing the valuable feedback. We are committed to minimise vendor friction.

We have raised CONFSERVER-79630 to provide a layer in confluence-compat-lib to cover >90% plugin cases for compatibility.
We would provide an update in this forum and Jira once we have a working version of the compat library.

Meanwhile, we would still recommend plugin devs to evaluate their plugin with changed class on 8.0.0-struts-m020 just to see the scale of issues(if any) with the upgrade.
Please reach out here if you face any issues with this upgrade.

Thanks,
Ganesh

3 Likes

This is great @ggautam, thank you for responding to vendor feedback.

We only have a couple of simple Webwork actions in our app so this change likely won’t impact us, but there are many apps on marketplace that I’m sure would have felt a lot of compatibility pain.

2 Likes

That is fantastic news @ggautam !!!

I think a 90% target for the compat lib functionality coverage is a good target. The last 10% is probably impractical and super difficult to implement and we wouldn’t want to slow the 90% solution with that last little piece.

Thank you, I had a nightmare last night related to forking our code… stuff like this keeps me awake. I will sleep better now!

1 Like

Thanks @ggautam, that sounds promising! I’ll be looking forward to the compatibility layer :slight_smile:

Given that we’ll need to support Confluence 7 I don’t currently see a way to test compatibility until the compatibility layer is available, though. At least not without migrating to struts classes, but then the apps won’t run on Confluence 7 anymore, which defeats the purpose of this whole discussion :wink:

Btw:
Did you consider bundling the compatibility classes directly with Confluence rather than in confluence-compat-lib? That way they could follow the regular Confluence deprecation cycle AND 3rd party apps wouldn’t need to pull in confluence-compat-lib as a bundled dependency.

Cheers,
Jens

1 Like

Hi @jens,

Thanks for the suggestion. We did look at bundling compat layer, but the whole idea is to bring balance between tech-debt and modernisation. Unfortunately, exposing old Xwork/Webwork based packages delegating to new classes would be not just problematic with IP things(which can be worked with), but also long-term maintenance required for that layer. However, with compat-lib, we get the liberty to make breaking changes with lesser friction and impact.

Cheers,
Ganesh

2 Likes

Hi @ggautam,

It has been a couple of weeks–are there any Struts milestone releases planned in the near future? I see a couple of things that are particularly broken, such as the fact that the UPM doesn’t work at all (at least for me), which makes it difficult to even test a plugin .JAR (because the upload fails).

I was able to work around this by compiling QuickReload, uploading the QR JAR into a different version of Confluence, and then copying the database row for QR from the PLUGINDATA table in the old Confluence DB to the 8.0 DB, restarting 8.0, and then finally using QR to reinstall my test JAR via the filesystem…but it would be nice if there were an easier way.

Scott

Hi @ggautam,

Question 1 (to the community):

Is anyone else actually using the struts-m020 build? Has anyone been able to register an xwork action from a P2 context (versus an Atlassian-bundled plugin) and get it to work?

Question 2 (to Atlassian):

Is it possible to correct the sources published for the struts fork? I downloaded the sources labeled as “8.0.0-struts-m020” from MAC (link) but these do not appear to correspond to what is shipped.

For example, I was trying to figure out why none of my app’s xwork actions were recognized, so I started chasing down error messages like these that occur during plugin install:

2022-08-26 14:52:08,988 WARN [QuickReload - Plugin Installer] [confluence.plugin.descriptor.XWorkModuleDescriptor] buildAllowedMethodNames Could not find allowed methods specified for action pulp in namespace /pulp defined in plugin pulp

The XWorkModuleDescriptor in the shipped Confluence source archive does not contain any reference to this message, and that class still imports the legacy com.opensymphony.xwork.* packages, which did not seem right.

But if I look inside the Confluence .JAR actually shipped in the 8.0.0-struts-m020 milestone release, there is a completely-different implementation of XWorkModuleDescriptor that references xwork2.* and which has the warning I was hunting for.

Question/note 3 (to Atlassian):

The {attachments} macro is broken and it does not render at all…I did not see this in the list of already-tracked issues.

Scott

2 Likes

Hi @scott.dudley,

Thanks for reaching out with feedback.

The first build was most initial one for EAP and it had a lot of bugs. We are preparing another release scheduled for tomorrow and we will post an update here once its out. I have verified UPM working properly in the next potential EAP. Please find answers for questions asked below:

  • Question#2
    GG - we will look into the source path issue for custom EAP milestone as mentioned by you.
    For
  • Question #3
    GG - The new EAP milestone comes with a lot of fixes including fix for editor and attachments.
    Please stay tuned for tomorrow’s updates.

Thanks,
Ganesh

1 Like

Hi everyone,
We have published a new milestone for struts upgrade with major fixes. Please find the full highlights on Struts 2 upgrade page.

@scott.dudley - we have raised two issue in relation to the questions raised by you:

Please let us know if you face any issues or bugs with the milestone.

Thanks,
Ganesh

Hi @ggautam,

Thanks for the updated milestone! Thanks for your team’s work in fixing so many issues.

Here are the new issues discovered to date with struts-m027:

1- Is there any chance of fixing the plugin reinstall problem quickly (CONFSERVER-79847)? There are a ton of moving parts with 8.0, and the requirement to restart the entire instance to test any plugin change severely handicaps the dev loop. I am sensing nostalgia for P1 and Confluence 2.x…

2- The Space Tools->Content Tools link goes to the wrong place (it drops you somewhere in /pages/templates2, without the tabs and chrome, rather than in the expected part of space settings).

3- There is an undocumented breaking change in the declaration format of Xwork action method names. In <= 7.x, it was acceptable to write <action class="Whatever" method="foo"> without the “do” prefix, and Confluence would call Whatever.doFoo(). The declaration now requires the precise method name, so this must be changed to <action class="Whatever" method="doFoo">.

4- There is something flaky happening with VelocityUtils.getVelocityEngine(). It is inconsistently logging errors and returning a fresh engine. The implementation of this in 7.x was trivial and consistent (obtains from static object), but the new 8.x implementation tries to find the Struts dispatcher, then find the linked container, then get the VelocityManager, then grab a specific instance…and if all that doesn’t work, which it sometimes does not, it’s fishing things out of the ServletActionContext. And then it finally gives up with the behavior described above.

I can’t figure out a precise set of steps to reproduce, but it’s flaky. It might possibly be related to the plugin reinstall issues above. I also note that it works less frequently when it’s not on a HTTP thread context (ie. when there’s no fallback to fetching from the SAC). It also doesn’t work during system shutdown. This is perhaps less concerning because the system is shutting down, but the log errors in that case are ugly:

java.lang.IllegalStateException: The configuration manager shouldn't be null
	at org.apache.struts2.dispatcher.Dispatcher.getContainer(Dispatcher.java:1002)

5- The Attachments macro can now be used for general purposes, but if you double-click on the macro from within the editor to edit its configuration, the preview panel shows a NPE (something related to the ServletActionContext).

6- The logs get spammed with hundreds of warnings about allowed XWork action methods, referencing all Xwork-containing plugins installed. This happens any time the system starts, and even any time you (re)install a plugin that contains Xwork modules:

2022-08-30 12:49:24,483 INFO [Catalina-utility-1] [atlassian.plugin.manager.DefaultPluginManager] logTime Plugin system lateStartup ended
2022-08-30 12:49:24,642 WARN [lifecycle:thread-20] [expose.jmx.schedule.JmxInstrumentSchedulerImpl] onStart atlassian-instrumentation-jmx expose scheduler started.
2022-08-30 12:49:24,644 INFO [synchrony-interop-executor:thread-1] [plugins.synchrony.bootstrap.DefaultSynchronyProcessManager] startupSynchrony Starting Synchrony and enabling Collaborative Editing
2022-08-30 12:49:30,805 WARN [Catalina-utility-1] [confluence.plugin.descriptor.XWorkModuleDescriptor] buildAllowedMethodNames Could not find allowed methods specified for action applinks-space-admin in namespace /spaces defined in plugin applinks-space-admin
2022-08-30 12:49:30,806 WARN [Catalina-utility-1] [confluence.plugin.descriptor.XWorkModuleDescriptor] buildAllowedMethodNames Could not find allowed methods specified for action tinymce in namespace /plugins/tinymce defined in plugin tinymce

Scott

Hi everyone,
We have dropped another EAP(8.0.0-struts-m39) for Struts2, Please find the full highlights on Struts 2 upgrade page.

@scott.dudley please find the answers to each of your point below:

  1. CONFSERVER-79847([CONFSERVER-79767] Fix Space Tools, Templates page - Create and track feature requests for Atlassian products.) was a higher priority for us as well given its impact on plugins. Please let us know if you still face any issues with it.
  2. We have CONFSERVER-79767 in progress for this, please watch it for the updates.
  3. It is great observation, fortunately, we had a task for the same in Confluence core before we pushed the first milestone. We would update the Struts upgrade to accomodate this soon.
  4. We see one Velocity related item which touches upon dispatcher and the engine. We will try to reproduce your problem in that, please reach out if you find any concrete steps for reproduction. There are some quirks around ServletActionContext as ServletDispatcher was a Servlet in Webwork, while Struts’s StrutsPrepareFilter/StrutsExecuteFilter are ServletFilters which clears the Context and states.
  5. Please follow CONFSERVER-79907 to track AttachmentMacro feedback.
  6. We have just handled the logSpam, we would add some directions on allowed-methods parameter for XWork module with the next milestone. This is optional and is not needed for actions using execute or has method wiring.

Thanks,
Ganesh

@ggautam
I’m worried that there is an EAP release out there in the world (8.0.0-struts-m39) that breaks some of our plugins that are using xwork and does not provide the compat-lib that we need to unbreak them: [CONFSERVER-79630] Create compat-lib to handle basic compat API for Webwork/Struts - Create and track feature requests for Atlassian products.

One concern is that CONFSERVER-7936 is Priority=Low and I can imagine a world in which Atlassian goes ahead with an 8.x release that does not include the compat-lib because it is low priority. IMO, not delivering the compat-lib soon could damage partners.

Can somebody with the power please increase the priority of CONFSERVER-7936 and commit to delivering it ASAP? We have a Confluence EAP release that we are not testing for some of our plugins… because they fail to deploy… and we don’t have the time/resources to fork our code to write a backwards-incompatible solution that is testable on 8.0.0 but is not ever going to be delivered to our customers.

During this time that the EAP release is not testable by us there could be other problems piling up that we cannot detect. This could be risky.

Thanks!

Hi @turehoefner_appfire,

8.0.0-struts-m39 is specifically a Struts breaking change EAP and the whole point of the EAP is to allow developers to provide early feedback and also see the changes happening. Even thought the priority is lower with respect to other Struts items, I would want to assure you that its a struts-blocker as labelled in the ticket.

Compat lib with Struts work is in progress and there are internal PRs under review for the same. We are targeting to release it pretty soon.
I would like to highlight that we would try to cater to >90% of plugin’s cross-compatibility requirements.

Thanks,
Ganesh

Hi @ggautam,

Any change of escalating CONFSERVER-79848? I checked a recent source milestojne download (8.0.0-struts-m39) and there are no references whatsoever to “xwork2” anywhere in the entire source tree. Something seems not tagged correctly? Third-party developers really need access to certain parts of the Confluence source to understand how things work, so this is a significant impediment.

Thanks!
Scott

Hi @scott.dudley
We looked at CONFSERVER-79848 and it is caused by some infra issues which is slowing us down. We have taken your feedback and keeping it WIP in parallel, but given that we targeting merging of Struts to main EAP. This problem should soon go away.
Since the feature branch EAPs shown this issue, we will still fix the issue but with a lower priority.

Please let me know if that suits.

Thanks,
Ganesh