Developer Preview of Referentiality within Confluence Released!

[GTD]

Atlassian Partner Developers,

The developer preview of Referentiality has been released! This developer preview provides developers* like you in our Ecosystem App Developers Group insight into how we are thinking of solving the issues with nested bodied macros and how you can solve them with Referentiality. In our sneak peak post, we explained the why we’re tackling this issue and in this post, we are going to go over the how and what: how to use a Connect app for Referentiality and what is happening within Confluence to enable this feature.

Native Table Macro to Connect Macro

Let’s start off with a Connect Macro sourcing its data from a Native Confluence Macros, such as a Native Table Macro**. This solves one of our most common use cases of extending the functionality of Native Table Macro within the page. To achieve this, we’ve introduced a customizable refDataSchema property for Dynamic Content Macros which provides support as a Target for Native Table Macro with table-adf. This property allows for different outputTypes and inputTypes which Confluence uses to identify the different compatible types for an existing macro. When a subscribing macro is set with table-adf as its inputType, it allows for a user to append it under a Native Table via the ContextualToolbar.

Connect Macro to Connect Macro

Another common use case is for our users to use 2nd party Connect dynamic content macros to be able to share, chain data, and have the macros react to incoming changes in realtime. What was possible in the Server with nesting, we’re enabling Cloud users to connect and chain macros. And as Confluence discovers and links all compatible macros by the refDataSchema at runtime, the Target macro would require the same inputType property as the Source macro’s outputType property for a link to be established. Support for chaining macros is also made possible should the chained macros have compatible refDataSchemas.

Other Macro Type Connection

Another connection we have enabled is for macros to target the Native Chart macro. Wouldn’t it be nice if your macro can extend this commonly used macro? With Referentiality enabled, it now can. You can access the same Native Chart as the Native Table can by updating your macro’s outputType to table-adf.

Forge Macros as a Target

In addition to Native Chart and Connect macros as Targets, we are also supporting Forge macros. All Custom UI and UI-Kit macro modules are supported as Targets.

Referentiality Connection type Matrix

And here is a table of all the different combinations between macro types, and which can be a Source and Target macro.

	Native	Connect	Forge
Native to …	Native Table → Native Chart	Native Table → Compatible Connect Macros with table-adf inputType	Native Table → UI Kit & Custom UI Compatible Forge Macros with table-adf inputType
Connect to …	Compatible Connect Macros with table-adf outputType → Native Chart	Any Compatible Connect Macros that share similar refDataSchema	Any Compatible Connect and Forge Macros that share similar refDataSchema

Setting Up Referentiality

As developers, we know the best docs are well-written docs that come with a companion app. So to demonstrate the uses of Referentiality, we have released a demo companion Connect app and a Forge app that will allow you to locally run and test this feature. You can find all of the code used in the examples here in the Connect app repo and Forge app repo.

In order for Referentiality within Confluence to work, a few things must be wired up:

• A Source macro must publish data using the emitToDataProvider api
• A Source macro must have an outputType define within its Descriptor
• A Target macro must have an inputType that is of a compatible outputType as the Source macro
• A Target macro must register to listen for the data_provider topic

Macro Matching

It’s a match given source macro’s outputType is same with target macro’s inputType. When it’s a match, the target macro will be listed in the ContextualToolbar dropdown of the source macro.

Macro Validation

Currently, there are 2 supported types for output and input that are validated before passing the data to the target:

1. table-adf: expected to be a valid json that conforms with table-adf schema.
2. table-json: expected to be a two-dimensional json array, that can include any valid json primitive or object.

If a macro defines table-adf or table-json as their inputType then they can expect that the incoming data will be valid objects. If the data validation fails, then an error message is passed back to the source macro’s callback function and nothing is passed to the target macro.

Upcoming Changes

We plan to work with developers, address concerns that you may have regarding this feature and release this feature to end users in the next few months. And as part of our Open culture, we want to share with you the upcoming changes we plan to make.

Improved chaining user experience - a single viewport, and navigation

Now you may be asking how this 1:1 relationship between macros solves the needs for a more complex chained list of macros. Within the same page, you can currently chain a list of supported macros that share a common schema; that is, they both include the same outputType and inputType. We know some of you might like it stacked atop each other, while others prefer a singular viewport, so we are working on improving this user experience to allow chaining multiple macros together within the same Viewport.

Concept showing how chained macros share the same viewport, in Confluence. Additionally, a portion of the RightContextPanel shows the full chaining relationship and allows navigation between chained macros.

Partner macros will be shown separately from native macros inside toolbar, for an improved browsing experience

A dedicated dropdown within the ContextualToolbar for partner macros. The current iteration of this design allows for developers to easily access their macros within the ContextualToolbar while testing their changes; and while this is optimal for developers, the experience may be difficult for end users to navigate. We plan on adding an additional dropdown to hold a list of Target macros, as the concept below illustrates. Each option will showcase a macro’s icon and name, as described within the manifest. The manifest will be updated to include an optional icon property or will be substituted with a default icon.

Concept showing an extensible toolbar for a macro in Confluence. Native apps are shown in the main toolbar body, and partner macros are shown in a dropdown menu.

Easier viewing and modifying Source and Target macros, using the RightContextPanel

Configuration options within the RightContextPanel to set Source and Target macros for the selected macros. This would be a natural extension of the existing macro configuration that users can customize.

Concept showing a macro’s connections (a Source macro, and a Target macro) inside the RightContextPanel.

A new way to flexibly create and use referenced elements in a page

A new identifier will be introduced to enable referencing multiple ADF nodes. This would expand the pool of references by allowing users more flexibility to create references; as currently, single content block nodes can be used for references.

Extending your apps with Referentiality

We are releasing this feature so early adopters can begin to experiment with your modified macros in a test environment before we enable to the general public. After this main release, we are planning on iterating on our designs and releasing more frequently.

We are actively working on making this developer preview better and are open to your feedback and concerns. We are committed to sharing updates and engaging with the Developer Community. We can’t wait to see what you build with Referentiality.

* If you’re not a part of our Ecosystem App Developers Group, please visit Confluence Cloud: Developer-First Rollouts for Ecosystem App Developers .

** Yes, we’re deliberately calling the Native Table Element a macro because it flows better in the Nested Bodied Macros narrative.

[thomas2]

Neat, we have a few use cases where this might come in handy.

A few things:

• I can’t access the Forge example (not yet made public?). The connect link works fine though.
• From the connect examples I gather that this all works in the users browser via the AP interface. What are the plans for making this work in static renderModes like PDF or Word? Without that a lot of our apps (Scroll PDF/Word Exporters, Scroll Viewport) couldn’t make use of this at all.

[PhilipFeldmann]

As a vendor, I am very concerned.

Will this completely remove the old body macros?
Our apps heavily rely on mixing different macros using this technique. We have all sorts of styling elements that can be used together in body macros. For example with our app, you could do something like creating a custom panel, put some images, text and multiple buttons inside of it.
Can you guarantee that we will be able to offer the same functionality with referencing multiple ADF nodes?
Even if this is the case, the user experience of this is TERRIBLE. I know you think it’s great, but take a custom panel macro for example. The panel is the wrapper around that body, so in the editor, you put the content exactly where you’d expect it to be rendered: Inside the panel body. Now you have to put it outside of the panel and link it, the content will then magically be shown inside the panel, instead of outside, where the user has to put it.

What about custom UI
I’ve raised this question before but I’ve been completely ignored. Our app NEEDS custom ui to configure its macros. Things like a color picker, templates etc are just not available in your default UI. How is this supposed to play together with the new sidebar editor and referentiality?

[candid]

@GTD

As developers, we know the best docs are well-written docs that come with a companion app.

As a developer, I disagree with this statement. Having access to a working piece of source code can be helpful if the documentation is incomplete (or only consists of some animated GIFs), but a proper documentation is always the better and more efficient way to understand an API.

Usage (Connect)

Here is a rough overview how it seems that this functionality is supposed to be used in Connect:

Emitting data

To emit data, macros use AP.dataProvider.emitToDataProvider(). This has to be run whenever the data changes, but I strongly assume that it also has to be called when the macro renders initially. Here is an example:

AP.dataProvider.emitToDataProvider({
    eventData: {
        'table-json': [
            ["Year", "Web", "TV", "Print"],
            [2012, 8, 153, 121],
            [2013, 41, 75, 124]
        ]
    }
});

In atlassian-connect.json, a property "refDataSchema": { "outputType": "table-json" } has to be defined for macros emitting data. The possible output types are table-json and table-adf. My guess is that the table-json format can be used for simple data, while the table-adf format has to be used for rich-text content. The data passed to AP.dataProvider.emitToDataProvider() has to have the output type that is defined for the macro.

Listening to data

To start listening to data, a macro has to run AP.register({ data_provider: (data) => { } }). The data parameter will apparently have a slightly different shape than the emitted object: The emitted table will be available as data.eventData[0]['table-json'] rather than data.eventData['table-json']. I’m not sure if the eventData array may contain multiple objects or always just one.

To listen to data, the following has to be added to the macro definition in atlassian-connect.json: "refDataSchema": { "inputType": "table-json" }. My understanding is that macros can only be linked with other macros whose outputType matches their inputType.

Open questions

I haven’t had the time to try this out yet and was just trying to understand how this is supposed to be used from the source code that you have provided. The following questions have come up for me:

• What if I want to write a macro that wants to support both table-json and table-adf as input formats?
• What if the emitting macro emits data before the subscribing macro is rendered? Will the subscribing macro receive the history of emitted events as soon as it is rendered?
• Will this mechanism work both inside the editor as well as on the view page?
• Is it possible to access the emitted data also from a macro editor? Since the macro always emits a whole table, I assume there will be many use cases where a user will have to configure which rows/columns the subscribing macro should handle in what way.
• How will this work in static render modes?
• How is the relationship between the macros persisted in the storage format of the page?

Overall, this looks like a very useful feature that will enable many interesting features. I think the chart example that you have given is a really good example for imagining how powerful this feature can become. However, I don’t understand how this is supposed to solve or how it is even related to the nested macro use case.

Since I am currently working on a macro that renders complex tables, I have quite a good use case for this functionality. One thing that worries me a bit is the work load that comes from the fact that this feature basically means implementing another render mode. Right now I already have to implement the dynamic render mode (React app) and the static render mode (storage format), and now I will have to implement an ADF render mode to support this. It would be much less work if I could either emit storage format here or use ADF for the static render mode.

[marc]

@GTD That feature looks interesting!

My question is: How does it work during e.g. PDF export?

[GTD]

@thomas2

• I can’t access the Forge example (not yet made public?). The connect link works fine though.

Can you try access the Forge example again? We’ve updated the settings recently.

@thomas2

• From the connect examples I gather that this all works in the users browser via the AP interface. What are the plans for making this work in static renderModes like PDF or Word? Without that a lot of our apps (Scroll PDF/Word Exporters, Scroll Viewport) couldn’t make use of this at all.

Static renderModes like PDF or Word are configured on the macro side, and since the macros can be configured to subscribe to and store incoming data, the data will be available for any of the views to utilize and render. It would be similar to what vendors are currently doing to support the renderModes now: take their underlying data, transform and display it.

[GTD]

@PhilipFeldmann

Will this completely remove the old body macros?

What about custom UI
I’ve raised this question before but I’ve been completely ignored. Our app NEEDS custom ui to configure its macros. Things like a color picker, templates etc are just not available in your default UI. How is this supposed to play together with the new sidebar editor and referentiality?

Thank you for your feedback and sharing your concern. We appreciate your insight as it allows us work with the Developer Community to build a better end user experience. The team is committed to help Vendors like you with the Cloud migration process and is aware of the use case you mentioned.

This release contains the ContextualToolbar experience for dynamic content macros linking, which provides multiple linked macros on the page. The custom-panel-macro use case you mentioned is still achievable on Cloud and will remain so. It will be exactly as you expect. This is opt-in enhancement would not remove the old body macros (nor custom UI configuration) and would not remove existing functionality from any of the existing macros; on the contrary, it would extend the current use case and would provide your macros with the option to chain other dynamic content macros, something not currently possible in the Cloud.

@PhilipFeldmann

Can you guarantee that we will be able to offer the same functionality with referencing multiple ADF nodes?

While the current release does not allow for referencing multiple ADF nodes, there are working towards allowing multi-node referencing in future releases.

I hope that alleviates your concerns, if not, please feel to reach out directly and I would be more than happy to assist you.

[GTD]

Thank you the feedback and sharing your thoughts on this. You’re spot on describing the uses and mechanisms of Referentiality. You’ve had a few outstanding questions, so let me try to answer them here.

@candid

What if I want to write a macro that wants to support both table-json and table-adf as input formats?

I’m glad you asked because we’ve had multiple discussions and it’s been at the top of our minds. It is not currently supported, but we are working thru the different scenarios to allow broad support with minimal developer effort to support multiple input formats.

@candid

What if the emitting macro emits data before the subscribing macro is rendered? Will the subscribing macro receive the history of emitted events as soon as it is rendered?

The subscriber will only receive after the connection is made and when the publisher emits. We don’t support burst events firing as there aren’t any frontend caching built in.

@candid

Will this mechanism work both inside the editor as well as on the view page?

Yes. Referentiality works for both the Editor and the Renderer!

@candid

Is it possible to access the emitted data also from a macro editor? Since the macro always emits a whole table, I assume there will be many use cases where a user will have to configure which rows/columns the subscribing macro should handle in what way.

Yes it is possible to access it from a macro editor but not directly as the Target macro would be the main macro (not the macro editor). The main Dynamic Content Macro would need to listen for to data_provider topic for the data, and share it with the macro editor.

@candid

How will this work in static render modes?

Similar to the macro editor case, the main macro would need to share the data with the other routes that handle the static render modes.

@candid

How is the relationship between the macros persisted in the storage format of the page?

The data being emitted between macros is ephemeral and is not being stored anywhere. It is contingent upon the Target macro’s logic to store the data needed for its function.

@candid

Overall, this looks like a very useful feature that will enable many interesting features. I think the chart example that you have given is a really good example for imagining how powerful this feature can become. However, I don’t understand how this is supposed to solve or how it is even related to the nested macro use case.

Since I am currently working on a macro that renders complex tables, I have quite a good use case for this functionality. One thing that worries me a bit is the work load that comes from the fact that this feature basically means implementing another render mode. Right now I already have to implement the dynamic render mode (React app) and the static render mode (storage format), and now I will have to implement an ADF render mode to support this. It would be much less work if I could either emit storage format here or use ADF for the static render mode.

Regarding the nested macro use case: dynamic content macros that are bodied are able to nest within the Server which enabled data sharing, but are unable to within the Cloud. One of the features of Referentiality is that it serves to bridge that gap within the Cloud by allowing macros to link and share data with the bodied macro.

The two predefined outputTypes are flexible and can be extended with any type such as storage-format. Tho Target macros would need to be able to ingest this data type. Table extensibility is also a pain point we are tackling with Referentiality, hence the support for Native Table as a Source. Other than configuring the app descriptors and configuring how the macro would handle incoming data, there should be no need to implement another render mode. I’d like to learn more about how you plan on using Referentiality for your macros, can you please elaborate more on this macro use case and provide the Atlassian Descriptor for the app? Feel free to message me directly and we can set up a time to discuss.

[GTD]

@marc

My question is: How does it work during e.g. PDF export?

Marc
Static renderModes like PDF or Word are configured on the macro side, and since the macros can be configured to subscribe to and store incoming data, the data will be available for any of the views to utilize and render. It would be similar to what vendors are currently doing to support the renderModes now: take their underlying data, transform and display it.

[ademoss]

@GTD - I am assuming that with Referentiality in place, more apps will support macros, and users will end up putting more macros in pages. How are you going to solve the current limitations where Confluence pages break when they contain too many macros?

[candid]

Thank you for your clarifications.

The following things are still unclear to me:

@GTD

The subscriber will only receive after the connection is made and when the publisher emits. We don’t support burst events firing as there aren’t any frontend caching built in.

Does this mean that the emitting macro needs to always emit its data every few seconds, regardless of whether there are any changes or any subscribers?

@GTD

@candid

How will this work in static render modes?

Similar to the macro editor case, the main macro would need to share the data with the other routes that handle the static render modes.

I don’t understand this one yet. A static macro returns storage format, it does not run any JavaScript that could make any AP calls. How can a static macro emit data, and how can it subscribe to data? I think this would have to happen in the backend of the app. This is relevant for static macros as well as static render modes of dynamic macros (for example during PDF export).

@GTD

@candid

How is the relationship between the macros persisted in the storage format of the page?

The data being emitted between macros is ephemeral and is not being stored anywhere. It is contingent upon the Target macro’s logic to store the data needed for its function.

What I meant is how the relationship is persisted, not how the data is persisted. I’m imagining it could be an additional attribute on the <ac:structured-macro/> element that contains the macro ID of the macro that it is subscribed to.

[marc]

@GTD Somehow I don’t understand how this is working. How can a static render macro for PDF or Word subscribe? It is backend only.
Can you give an example of how it is supposed to work?

[PhilipFeldmann]

Thanks @GTD that is helpful.

You’ve said that:

this is opt-in enhancement would not remove the old body macros (nor custom UI configuration)

As far as I can see currently there is no way to create a body macro or custom ui configuration in forge. Since all apps have to transition to forge earlier or later, will forge allow custom ui and macro bodies as well?

[GTD]

Looking at the previous ticket [CONFCLOUD-35359] Too many nested macros break page - Create and track feature requests for Atlassian products. where there are too many nested macros breaking a page, it looks like an edge case we decided not to pursue to focus on our roadmap for all Confluence users. All systems have limits and we have strived to set our limits high enough to satisfy the needs of almost all but a fraction of our users while balancing the other needs of the user. Thanks for bringing this up, we will continue building with this issue in mind and look into how we can optimize for this with Referentiality.

[ademoss]

@GTD I am not talking about nested macros.

If I place 80 simple macros, that simply render “hello world” in a single page, the page breaks.
Sometimes the page won’t render at all, sometimes some macros render but others don’t. It’s a cornucopia of different failures, where each page refresh shows different macros succeeding/failing.
I’ve seen cases where this happens with as few as 30 macros on a page, and going as high as 80-100.

And before you tell me, well, don’t place 80 macros on a page (because that’s what support told us/customers), realize that what you’re building is inherently designed to encourage customers/developers to use more macros.

The relevant ticket for this would be: [CONFCLOUD-71761] Failed rendering of pages with a large number of static macros - Create and track feature requests for Atlassian products.

[GTD]

@candid

Does this mean that the emitting macro needs to always emit its data every few seconds, regardless of whether there are any changes or any subscribers?

The source macro would receive a confirmation status message for each fire from Confluence; the source macro can then retry if unsuccessful.

@candid

I don’t understand this one yet. A static macro returns storage format, it does not run any JavaScript that could make any AP calls. How can a static macro emit data, and how can it subscribe to data? I think this would have to happen in the backend of the app. This is relevant for static macros as well as static render modes of dynamic macros (for example during PDF export).

Yes that’s right, the static content macro would not be able to emit or listen for any of the data. As all the data is being brokers on the client side, the backend wouldn’t have any access to the shared data unless the non-static-content target macro stores data within the Confluence similar to what this app is doing.

@candid

What I meant is how the relationship is persisted, not how the data is persisted. I’m imagining it could be an additional attribute on the <ac:structured-macro/> element that contains the macro ID of the macro that it is subscribed to.

The relationship between the source and target marcro is persisted within the ADF using a dataConsumer attribute.

Have you had the chance to experiment with Referentiality using the example macros?

[GTD]

@marc Yes that’s right, the static content macro would not be able to emit or listen for any of the data. As all the data is being brokers on the client side, the backend wouldn’t have any access to the shared data unless the non-static-content target macro stores data within the Confluence similar to what this app is doing.

[GTD]

@PhilipFeldmann Yes, Forge does have support for custom ui, but will likely not support macro bodies due to the security concerns that were mentioned earlier. However, the Forge team is aware of the data sharing needs of macros that were enabled by nested bodied macros and are working on a solution.

[marc]

Hi @GTD , for some customers it is a worrying trend that PDF and word export does not render a complete page. Many of our customers in regulated industries need to archive pages in e.g. PDF. Already Forge macros are not rendered in PDF, and this adds to the problem.

[candid]

Thank you for your attempt to answer my questions. Based on your responses, I have some feedback.

@GTD

@candid

Does this mean that the emitting macro needs to always emit its data every few seconds, regardless of whether there are any changes or any subscribers?

The source macro would receive a confirmation status message for each fire from Confluence; the source macro can then retry if unsuccessful.

This does not really answer my question. Imagine you have an emitting macro A and a subscribing macro B on a page. I open the page, the different UI elements of the page load, at around 20 seconds, macro A renders and emits the data, and at around 30 seconds, macro B renders and starts listening. Now macro B missed the data emission of macro A. I am assuming that the only way to work around this problem is to let macro A emit its data every 5 or 10 seconds or so. I believe this is also how it is done in the example that you have shared.

@GTD

The relationship between the source and target marcro is persisted within the ADF using a dataConsumer attribute.

I am assuming that the source macro is referenced by its macro ID. I tried to set up the example app to verify this, but unfortunately my Confluence instance does not seem to show the UI elements necessary to set up macro referentiality.

---

Overall, I think this feature is a really interesting idea and has a lot of potential. But the technical implementation that you have chosen is not really thought out. My hope and advice would be that you change the implementation before going public with this.

As the developer of a source macro, implementing this will require a lot of unnecessary additional effort. At the moment, I have to implement two rendering modes, a dynamic one (rendering a React app in the frontend) and a static one (returning storage format XHTML). With this new feature, I have to implement an additional rendering mode (emitting the data as ADF). This is unnecessary because ADF and storage format XHTML basically represent the same thing, just in a different format (as far as I know). It would be much better if I could either emit the data as storage format XHTML, or if I could return ADF for the static macro render mode.

As the developer of a target macro, I have to accept 3 different data formats as well. I have to publish two different versions of my macro, one that accepts the JSON format and one that accepts the ADF format. In addition, in both macros I have to implement a static render mode that will use user impersonation to fetch the Confluence page (in storage format XHTML or ADF), look up the source macro there, and extract the data from its parameters and body. It would be much better if the target macro could decide in which format it wants to receive the data, and it would be automatically transformed regardless in which format it was emitted.

A big problem that I also see is that in most cases, the source macro will not simply emit its macro body unchanged, but it will generate the data that it emits from different sources (for example from the macro body and parameters, from content properties or custom content attached to the page, or from external data sources). In dynamic render mode, the source macro is responsible for generating the data from the content, but in static render mode, the target macro is responsible for generating the data from the content. This inconsistency will make it very tricky to develop macros that support referentiality with macros from another app. Imagine I have published a macro that renders and emits a table whose content is generated from data stored in a custom content object attached to the page. Now you want to develop a macro that listens to my data. For the static render mode of your macro, you will have to replicate all the logic of my macro of generating the table from the custom content. In some more complex use cases, it will even be impossible to replicate that logic because you will not have access to the data that I’m generating my content from.

Here are some concrete proposals how to solve the problems that I have raised:

• Add an additional render mode to dynamic macros, it could be called something like refData. This render mode would return the macro data in the format defined in the refDataSchema.outputType.
• Add an additional placeholder such as {macro.refData} to the url property of static macro definitions. When rendering a target macro in static mode, fill this placeholder with the data returned by refData render mode of the source macro, converted to the format defined in refDataSchema.inputType of the target macro. Like the {macro.body} placeholder, the data can be truncated if renderingMethod is not POST.
• Introduce an additional inputType/outputType called table-xhtml that uses storage format XHTML.
• Allow hooking up target macros of any inputType with source macros of any outputType. Automatically convert the data to the desired data format.
• Add an additional method AP.dataProvider.requestData() for the target macro to request the data from the source macro. The source macro would receive a request_data event. This way the source macro would only have to emit its data when it first renders, when its data changes and when receiving a request_data event, rather than continuously every few seconds.