Race condition with bundled WRM breaks Confluence

Hi,

I’m facing an issue with WRM caused by atlassian-webresource-webpack-plugin.

We use the webpack plugin to bundle our resources and generate the WebResource definitions for our P2 plugins. By default, the plugin has a list of built-in provided dependencies:

exports.builtInProvidedDependencies = new Map()
    .set('wrm/require', (0, exports.buildProvidedDependency)('com.atlassian.plugins.atlassian-plugins-webresource-rest', 'web-resource-manager', 'WRM.require', 'wrm/require'))
    .set('wrm/context-path', webresourceDep('context-path', 'WRM.contextPath', 'wrm/context-path'))
    .set('wrm/data', webresourceDep('data', 'WRM.data', 'wrm/data'))
    .set('wrm/format', webresourceDep('format', 'WRM.format', 'wrm/format'))
    .set('wrm/i18n', webresourceDep('i18n', 'WRM.I18n', 'wrm/i18n'));

This list is added regardless of what we do. In the generated resources XML, the following dependencies are automatically added:

<dependency>com.atlassian.plugins.atlassian-plugins-webresource-plugin:context-path</dependency>
<dependency>com.atlassian.plugins.atlassian-plugins-webresource-rest:web-resource-manager</dependency>

We include our resources using the PageBuilderService:

PageBuilderService.assembler().resources().requireContext(ResourcePhase.DEFER, "context")

I would expect that the PageBuilderService would remove duplicates and filter out the WRM stuff because it would already be used by the page. However, this is not the case. The generated batched javascript file includes the WRM stuff on top. More specifically, it includes the following context path override:

/* module-key = 'com.atlassian.plugins.atlassian-plugins-webresource-plugin:context-path', location = 'js/data/context-path.js' */
;(function() {
    var contextPath = null;

    function getContextPath() {
        if (contextPath === null) {
            contextPath = WRM.data.claim('com.atlassian.plugins.atlassian-plugins-webresource-plugin:context-path.context-path');
        }
        return contextPath;
    }

    if (typeof define === 'function') {
        define('wrm/context-path', () => getContextPath);
    }

    // Export a global variable
    WRM.contextPath = getContextPath;
    // This used to be part of AUI, so preserve the legacy.
    AJS.contextPath = getContextPath;
}());

As you can see, this code sets the global functions WRM.contextPath() and AJS.contextPath().

This should normally not be an issue, but sometimes there is a race condition on the page load in which the same code is also executed through inclusion from other web resources (or, more notably, the host application itself). Since our code is listed as deferred, this is usually the case.

The issue is that WRM.data.claim() only allows a specific key to be “claimed” once. When the key is claimed, the claim() function returns the value of BAD_RETURN:

claim(key, successcb, failurecb) {
                if (!successcb && !failurecb) {
                    if (globalHolder._claimedData.has(key)) {
                        console.error(`Data with key '${key}' has already been claimed`);
                        return BAD_RETURN;
                    }

BAD_RETURN is never defined in code, which means that it returns undefined. This means that as soon as our code is executed, AJS.contextPath() will now return undefined, which results in the following broken XHR requests on the Confluence page:

https://localhost/undefined/rest/wrm/2.0/resources

I would love to be able to exclude the WRM code from my bundles, but at this point I don’t know how to achieve that other than running some XSLT during resource generation.

To be honest, I was expecting the PageBuilderService to know that the WRM stuff was already included on the page and remove it from our bundled code, or some other fancy trick. But that doesn’t seem to be the case.

@mkemp is this something you can help me with?

1 Like

What is a bit weird, is that there seems to be some understanding that sometimes a data claim can be executed multiple times:

* Data for a specific key is retrieved only once. Subsequent requests for the same key will return no data.
* If the data is required by multiple callers, the suggested pattern here is to wrap an API eg
* WRM.contextPath around the WRM.data.claim method and cache inside that external API - see context-path.js
* in this directory for an example.

(source: Bitbucket)

But if you look at the code from the context-path.js, there doesn’t seem to be any specific code that deals with this:

function getContextPath() {
        if (contextPath === null) {
            contextPath = WRM.data.claim('com.atlassian.plugins.atlassian-plugins-webresource-plugin:context-path.context-path');
        }
        return contextPath;
    }

One would expect that for instance, the safe method of doing this would be to make sure that the globals are not overridden:

// Export a global variable
WRM.contextPath = WRM.contextPath || getContextPath;
// This used to be part of AUI, so preserve the legacy.
AJS.contextPath = AJS.contextPath || getContextPath;

(source: Bitbucket)

This would prevent a duplicate claim (as the getContextPath method would not be called multiple times) and would prevent the value to be set to undefined

Yeah

The actual problem is upstream like you say, WRM’s contract is that it delivers a web-resource exactly once. This is the bug that we should actually fix because with side-effect filled code it will cause many other weird and wacky bugs that others wouldn’t be able to understand.

Reproduction is hard, there’s quite a few things that it can be along the way, all of the plugins that ship with Confluence we’ve tested and are working so we don’t have something to play with.

Would you be able to make a very minimal reproduction plugin with atlas-create-confluence-plugin? It doesn’t need to have Webpack setup, just manually defining a dependency on com.atlassian.plugins.atlassian-plugins-webresource-plugin:context-path.context-path.

Also could you say which page of Confluence this happens on? (it might only appear on a subset)

Which version of Confluence is this?

I’m currently writing an XSLT which will remove the dependencies from the XML after generation, as this is blocking a customer from using our app. For all purposes in which this code is used, it is safe to assume wrm/context-path will be present.

But once that is done, I will see if I can give you a reproduction path. To be honest, I think this is even happening right now in our production code for the Figma for Confluence app in any Confluence version higher than 8.5.0 As soon as a macro is added to the page, the issue can happen.

1 Like

@mkemp I really think the issue is twofold here. Yes, there is a problem with the WRM with regard to the bundling and not removing the duplicate dependencies from our bundle.

But there is also an issue with the atlassian-webresource-webpack-plugin with regard to the opinionated built-in provided dependencies list. There should be an option to override or disable this list. For instance, either by setting providedDependencies option to false, or by exposing the built-in list as an optional include in the providedDependencies map.

  • if no providedDependencies option is set, use the built-in provided dependencies
  • if the providedDependencies option is set to false, do not include any dependencies
  • if the providedDependencies option is set to a map, allow users to inject the built-in dependencies into that map by exposing them as an export
1 Like

We’re somewhat limited by the Webpack hooks and the architectural limitations of running inside a DC product. The Webpack runtime we insert to turn Webpack’s require calls into something useful requires the WRM clientside so it’s very correctly declaring those dependencies so the WRM ensures those are loaded first so there isn’t a JS error at runtime.

There is an option called noWRM, but we don’t use it much internally, it could help you hack something together quickly. It should avoid those provided dependencies and modifying the runtime (I haven’t tested that) while still defining the web-resource XML, if it does not, I think that would be a bug.

I’m sorry, I did try this on 8.6.0, but I think there’s Enterprise controls preventing my PAT from working

I don’t think the noWRM option has any impact on adding the built-in dependencies. Here is the code that re-populates the provided dependencies list. It is called in the constructor:

this.options.providedDependencies = this.ensureProvidedDependenciesAreUnique((0, options_parser_1.toMap)(this.options.providedDependencies));

The ensureProvidedDependenciesAreUnique() method does nothing with the noWRM option:

    ensureProvidedDependenciesAreUnique(providedDependencies) {
        const result = new Map(provided_dependencies_1.builtInProvidedDependencies);
        for (const [name, providedDependency] of providedDependencies) {
            if (result.has(name)) {
                continue;
            }
            result.set(name, providedDependency);
        }
        (0, logger_1.log)('Using provided dependencies', Array.from(result));
        return result;
    }

So after the plugin was instantiated, the this.options.providedDependencies option includes the built-in provided dependencies, regardless of whether the noWRM option was set.

As far as I can tell, the noWRM option only influences whether or not the code is added that allows asynchronous loading of required modules.

1 Like

Hi @mkemp

This issue hit me recently too, and I think it is actually an issue with the host app’s WRM interface versus an issue with the wrm-webpack plugin. I believe this behavior is new to Confluence 9.

Even if using the noWRM flag when building the <web-resource> in the webpack plugin, and even if the resulting <web-resource> is bare, Confluence 9 is now dumping a pile of other libraries (AUI, jQuery, etc) into any <web-resource> referenced from Velocity.

I gather that this is not your team’s code, but in summary, by using something like this in Velocity:

$webResourceHelper.getResourceTags("my.plugin:my-module")

This ends up invoking Confluence’s VelocityFriendlyPageBuilderService, which calls:

assembler.assembled().drainIncludedResources().writeHtmlTags(writer, UrlMode.AUTO);

which then calls DefaultAssembledResources#drainIncludedResources:

    public WebResourceSet drainIncludedResources() {
        return this.builder.enableCleanUpAfterInclude().enableAdditionOfWebResourceJavascriptApiDependencies().enableSuperbatch().build();
    }

which (in) DefaultWebResourceSetBuilder then does this (based on the prior call of enableAdditionOfWebResourceJavascriptApiDependencies):

static final WebResourceKey WEB_RESOURCE_MANAGER_RESOURCE = new WebResourceKey("com.atlassian.plugins.atlassian-plugins-webresource-rest:web-resource-manager");

this.requestState.getRawRequest().includeFirst(superbatchConfiguration.getResourcePhase(), WEB_RESOURCE_MANAGER_RESOURCE);

This finally causes the following deps from web-resource-manager-rest to be stuffed into each resource:

    <web-resource key="web-resource-manager">
        <dependency>com.atlassian.plugins.jquery:jquery</dependency>
        <dependency>com.atlassian.auiplugin:aui-core</dependency>
        <dependency>com.atlassian.plugins.atlassian-plugins-webresource-plugin:context-path</dependency>
        <dependency>com.atlassian.plugins.atlassian-plugins-webresource-plugin:data</dependency>
        <dependency>${project.groupId}.${project.artifactId}:curl</dependency>
</web-resource>

Any idea if something could be done about this?

Thanks!
Scott

2 Likes

Thank you @scott.dudley :pray: As always you know far better how to pinpoint and articulate the actual problem!

I think this also happens in other situations in which the assembler.assembled().drainIncludedResources() is called. We call it in our Atlassian Connect Polyfill and not from a velocity context.

Okay, the problem is a bit nuanced and this exposes a few more.

With VelocityFriendlyPageBuilderService#getResourceTags it appears Confluence is try just get the script/link/etc tags for the specified web-resources. Even if this method was working properly, it’s got great potential to be a footgun, use with extreme caution. This is because it’s ejecting from all of the safety mechanisms the WRM provides, chiefly: not loading resources twice.

If you can, you want to rely on the Confluence/Product decorators writing the tags at the end. If you have a totally custom page/document then you ideally have the tags written once.

There’s a few times you need to write HTML tags mid document, really want to avoid that because of the potential impact on the whole browser waterfall. Not sure what/if Confluence provides for this, I’ll do some digging.

What can you do today without having to wait for a bugfix version? Basically reimplement that Confluence helper.

I’ll raise this broken and undocumented helper with the Confluence team and I’ll see if we can make it unnecessary by adapting the WRM API so it’s easier to write cross product.

By ‘you’ I really mean any software engineer working on DC stuff

Hi @mkemp

Thanks for providing the context. I think the use cases for this feature are largely outside the context of Confluence-rendered pages, but in cases where vendors still want to use the WRM for declaration of resources.

For example, I believe Remie’s example is used for generating iframes from a servlet that should not have any Confluence boilerplate JS loaded.

My use case is rendering a separate page (outside of main Confluence pages, without any chrome). The changes to this functionality in Confluence 9 broke a particular feature, because the page first loads a vendor version of jQuery with a specific function extension enabled. The new scripts forced by C9 will defer-load their own copy of jQuery, which blows away the first jQuery (effectively removing the function extension), which breaks later page rendering.

@remie might be able to fix his specific problem by calling drainIncludedSyncResources instead of drainIncludedResources, because the former seems to disable the stuff-in-extra-JS piece. Unfortunately, no such option seems to be easily accessible from Velocity.

I am hoping to avoid having to replicate the DefaultWebResourceAssembler in vendor code (this seems like it will be fragile and hard to keep working properly cross-major, among other things). For now, I have settled on calling the helper from Java rather than Velocity and then filtering out the resources I do not want from the returned String, but the underlying function would ideally be reverted back to pre-C9 behavior.

Yeah, those seem like valid use cases. When operating within an iframe or a full custom page like you say, just disabling the superbatch and using the rest of the assembler API will allow the WRM to provide as much safety as possible (also avoiding deduping loading of resources). Go for it

Sorry I meant specifically doing something like you say

webResourceAssemblerBuilder.resources()
    .requireWebResource(moduleCompleteKey1)
    .requireWebResource(moduleCompleteKey2)
webResourceAssemblerBuilder.assembled()
    .drainIncludedSyncResources().writeHtmlTags(writer, UrlMode.AUTO);

Again, if using the same webResourceAssembler;

  • The WRM should avoid the double loading of the sync batch and the context-path web-resource
  • Ideally the HTML tags are written once for performance

That’s all Java API. We’ll scream, shout, and analyse usage before we [intentionally] change behaviour.

I quickly poked around with a debugger running

WebResourceAssembler webResourceAssembler =
    webResourceAssemblerFactory.create().build();
StringWriter htmlWriter = new StringWriter();
webResourceAssembler.resources().excludeSuperbatch();
webResourceAssembler.assembled()
    .drainIncludedResources().writeHtmlTags(htmlWriter, UrlMode.RELATIVE);
htmlWriter.toString();

AFAICT Confluence 9.1 no longer has anything left in the _sync batch, and I didn’t see anything. If it does, you may want to exclude it, particularly in older versions.

EDIT:
The internals of WRM allow disabling the superbatch from either end. What’s still being included though is some of the WRM runtime. It seems we’re missing a public API for specifically turning that off.

I also haven’t yet had time to see why this has only changed in Confluence 9? All the code in the WRM looks effectively the same.

Once I have a free moment I’ll analyse all of the entry points I can find, see where they’re inconsistent, when they lie to you, etc.

1 Like

The reason I use drainIncludedResources is because drainIncludedSyncResources is marked deprecated :man_shrugging: