Developer Preview of Referentiality within Confluence Released!

[PhilipFeldmann]

@GTD That was my original question. It doesn’t really help if macro bodies are continued to be supported in connect, because all vendors have to move to forge eventually. For us, it’s less about data sharing between macros. Macro bodies allow our users to build compositions with different macros and Atlassian built-ins such as headlines and paragraphs. It’s still very unclear to me if this is going to be supported in the future. I want to raise awareness that a lot of vendors, especially in the content formatting category, rely on exactly this functionality. How is a custom panel supposed to work in forge without macro bodies?

[GTD]

@candid

I am assuming that the source macro is referenced by its macro ID. I tried to set up the example app to verify this, but unfortunately my Confluence instance does not seem to show the UI elements necessary to set up macro referentiality.

Which is the instance are you currently testing on? It might need to have certain flags enabled. Feel free to DM me the cloudId or instance id.

@candid

It would be much better if I could either emit the data as storage format XHTML, or if I could return ADF for the static macro render mode.
It would be much better if the target macro could decide in which format it wants to receive the data, and it would be automatically transformed regardless in which format it was emitted.

Yes, technically you can emit the data in any format you wish as long as a Target accepts such a format. Tho you do bring up a good point about having to deal with the different formats; we will be exploring different data transformers to help with this. Stay tuned!

Thank you for sharing your feedback; you’ve made some really great points and I’ll share this information with the rest of the development team to look into how we can incorporate some of suggestions provided.

[GTD]

Thanks for raising awareness on the content formatting category. We are aware of the limitations that Referentiality has regarding content formatting and that it is not being addressed in this instance. We are ramping up efforts on the Forge side to increase the number of Extension Points within Confluence that apps can leverage.

[TonyGoughAdaptavist]

@GTD Like the above poster this feature is not working for me - I have PM’d you with my EAP enrolled instance.

[GTD]

@TonyGoughAdaptavist Got it, thanks; I’ll respond to your PM.

[TonyGoughAdaptavist]

@GTD

@candid

What if the emitting macro emits data before the subscribing macro is rendered? Will the subscribing macro receive the history of emitted events as soon as it is rendered?

The subscriber will only receive after the connection is made and when the publisher emits. We don’t support burst events firing as there aren’t any frontend caching built in.

Actually according to my testing (admittedly limited so far), a subscribing macro need not have registered its listener before the publishing macro publishes its data. If I put the window.AP.register call in a setTimeout to fire 5 seconds after page load, (and the emitToDataProvider is fired immediately on page load) it still receives the data when it finally fires. This would seem to indicate some cacheing is happening ?

This would be a small improvement on the AP.events functionality.

:EDIT: Note that if multiple events were fired, it does only get the most recent one.

[akhaneev]

How can I enable the UI menu to select macros to connect with? I see AP.dataProvider is available on my Confluence instance, but I can connect two macros to play with the referentiality feature.

[TonyGoughAdaptavist]

Having tested this early preview I’ll post my findings here.

This feature essentially seems to be a pub/sub event mechanism to enable data sharing between macros. Like AP.Events, but more restrictive, as I’ll detail further below.

@GTD

Regarding the nested macro use case: dynamic content macros that are bodied are able to nest within the Server which enabled data sharing, but are unable to within the Cloud. One of the features of Referentiality is that it serves to bridge that gap within the Cloud by allowing macros to link and share data with the bodied macro.

Perhaps the most important point to start with is one raised by others here - data sharing is not nesting, and a data sharing solution like this or AP.Events is not a nesting solution.

For Confluence users, nesting means having a rich text macro and working inside the body area, using the native page composition tools (including adding macros) of the Confluence editor to compose content - take the following basic example:

There are virtually unlimited possibilities for nesting native and third-party macros in various arrangements inside rich-text macros - offering significant value to the user.

Consider now how would we try to implement nesting using this new feature:

You insert the target rich-text macro from a source macro and… Then what? Even assuming the feature is developed further to support multiple source macros, how are we to infer what the user wants to do with these macros inside the rich-text body, or where to place them in relation to the other body content? A data sharing mechanism such as this one is really not designed to support the complex nesting functionality that users have enjoyed thus far.

On Connect this does not have a huge impact because we have rich-text macros and can implement our own nesting solutions without this new feature. As others have pointed out though, fast-forward to a future with native Forge modules seemingly not supporting rich-text macros and this becomes a significant feature absence on that platform, and significant loss of value delivered to the users.

Based on this preview, I feel like the feature is not going to live up to what was sold in the original blog as addressing the nesting gap on cloud.

Another point to make relates to the following:

@GTD

In order for Referentiality within Confluence to work, a few things must be wired up:

• A Source macro must publish data using the emitToDataProvider api
• A Source macro must have an outputType define within its Descriptor
• A Target macro must have an inputType that is of a compatible outputType as the Source macro
• A Target macro must register to listen for the data_provider topic

What should be made explicit here is the additional requirement that you must insert target macros using the new context toolbar UI element found on source macros. Let’s consider the following scenario:

Often when macros work together, they are not necessarily placed directly beside each other. Many apps contain such macros - an example pair of macros from Content Formatting are Footnote and Footnote Display. With Referentiality, in such a scenario the user would no longer be able to insert macro2 freely at any point on the page. They would need to insert it using the new UI element on macro1, then drag it to where they would like it to be. This is a far less convenient user experience than simply inserting the macro where the user wants it, which AP.Events supported through data sharing which was not hindered by restrictions such as this.

The one new piece of functionality introduced here which was not available using AP.Events is the introduction of published data from native macros such as Table. It should be noted however that the data emitted could already have been retrieved by a custom macro using the content API - establishing the correct relationships could also be done this way, based on page placement of the macros.

Some additional restrictions from my testing:

• Does not work if the source macro is inline
• Does not work if the source macro is rich-text

Is this because Forge will not support inline and rich-text macros, or are these planned?

• The target macro can be rich-text, though when you perform the macro insertion it renders as a bodiless macro initially. Seems like a bug.
• The restrictiveness of Referentiality’s data sharing model prohibits use cases supported by AP.Events where only a single macro is involved, such as sharing data between a dialog and the primary macro iframe, or a dialog and a custom editor. We have this kind of functionality in Content Formatting, which AP.Events is useful for.
• The Native Chart functionality was not there so I could not verify this. Based on the matrix in the original post, Chart should be visible on Table’s context toolbar even before adding any custom macros, but on my instance it was not:
  
  

Considering Forge, of course we know that in native Forge modules, we have lost access to the Connect AP library. As Referentiality presents functionally as the replacement for AP.Events going forward it’s worth asking: are there any plans on recreating AP.Events more fully in Forge, eg without the restrictions mentioned above?

@PhilipFeldmann

Thanks @GTD that is helpful.

You’ve said that:

this is opt-in enhancement would not remove the old body macros (nor custom UI configuration)

As far as I can see currently there is no way to create a body macro or custom ui configuration in forge. Since all apps have to transition to forge earlier or later, will forge allow custom ui and macro bodies as well?

I would echo this - when planning ahead for Forge we need to acknowledge that it currently does not support bodied macros and unless I am mistaken we haven’t had an indication that it will. Having established that Referentiality is not a nesting solution, it’s becoming increasingly difficult to be an advocate for Forge adoption.

@candid

In atlassian-connect.json, a property "refDataSchema": { "outputType": "table-json" } has to be defined for macros emitting data. The possible output types are table-json and table-adf. My guess is that the table-json format can be used for simple data, while the table-adf format has to be used for rich-text content. The data passed to AP.dataProvider.emitToDataProvider() has to have the output type that is defined for the macro.

In case this wasn’t clear from other posts I can confirm from my testing:

• Any string value can be used for the outputType and inputType properties
• Macros are matched based on matching values in these properties

So what is special about table-adf and table-json?

1. Automatic validation is provided when these particular values are specified, based on the criteria in the original post. The emit call will fail if the validation does not pass.
2. Atlassian are adding these to native macros (eg outputType: table-adf on Table macro), allowing for example a custom macro to specify inputType: table-adf to pair with the Table.

[akhaneev]

Does it really work now? I tried it in our own app as well as I tested the connect example app, but none macro receives data using the AP.register({data_provider}).

I see that events are published in the logs, but nothing gets to subscribers.

[OleksandrBeztsinnyi]

Hey @akhaneev, thanks for pointing this out.

It appears that one of FFs was not enabled for referentiality to the beta group cohort. We enabled it and now it should work. Could you please confirm?

Sorry for the inconvenience.

[akhaneev]

Hi @OleksandrBeztsinnyi , it works now! Thank you!

[akhaneev]

I have some feedback. In general, referentiality looks interesting in some cases. I suggest adding an attribute to the macro definition to make it not selectable from the list of macros when inserting it. Macros-consumers, that support referentiality, have to be bodiless thus there is no sense to insert them in the page editor as we do with regular macros. There is only one to create a macro - from the source macro. Having these macros in the Insert menu will lead users to misunderstand and ask numerous questions about how to use them.
I want to hide these macros from the Insert macro UI and keep it in the source macros only.

[akhaneev]

I found an issue: if I update the page using REST API PUT /rest/api/content/898465793, then the

<ac:adf-mark key="data-consumer"><ac:adf-data-consumer-source>046ac8ab-de13-4f1a-a05f-473773a34812</ac:adf-data-consumer-source></ac:adf-mark>

is removed and the macro is not linked with the source provider.
Here is the payload:

{
  "id": "898465793",
  "type": "page",
  "title": "Referentiality",
  "body": {
    "storage": {
      "value": "<table data-layout=\"default\" ac:local-id=\"046ac8ab-de13-4f1a-a05f-473773a34812\"><colgroup><col style=\"width: 206.0px;\"/><col style=\"width: 140.0px;\"/><col style=\"width: 207.0px;\"/><col style=\"width: 207.0px;\"/></colgroup><tbody><tr><th><p><strong>Name</strong></p></th><th><p><strong>Value</strong></p></th><th><p><strong>b</strong></p></th><th><p><strong>c</strong></p></th></tr><tr><td><p>DD</p></td><td><p>1</p></td><td><p>78</p></td><td><p>3</p></td></tr><tr><td><p>Ф</p></td><td><p>2</p></td><td><p>Ы</p></td><td><p>В</p></td></tr><tr><td><p>X</p></td><td><p>3</p></td><td><p>sdkf[sdpkfk[psdk]pf</p></td><td><p>6</p></td></tr><tr><td><p>A</p></td><td><p>4</p></td><td><p></p><p class=\"media-group\"><ac:structured-macro ac:name=\"view-file\" ac:schema-version=\"1\" ac:macro-id=\"0636ca5a-6f10-4cbc-8905-c08459276451\"><ac:parameter ac:name=\"name\"><ri:attachment ri:filename=\"SAS Hygiene Issues_Screenshots.pptx\" ri:version-at-save=\"1\"></ri:attachment></ac:parameter></ac:structured-macro></p><p></p></td><td><ac:image ac:align=\"center\" ac:layout=\"center\" ac:original-height=\"886\" ac:original-width=\"1346\"><ri:attachment ri:filename=\"image.png\" ri:version-at-save=\"1\"></ri:attachment></ac:image><p></p></td></tr></tbody></table><ac:structured-macro ac:name=\"table-chart-referentiality\" ac:schema-version=\"1\" data-layout=\"default\" ac:local-id=\"d56c1793-bf3e-47e3-9843-5d775a7684d6\" ac:macro-id=\"bc3f56a7-1247-4b62-95d5-bd403fe28e3d\"><ac:parameter ac:name=\"legend\">right</ac:parameter><ac:parameter ac:name=\"align\">Center</ac:parameter><ac:parameter ac:name=\"datepattern\">undefined</ac:parameter><ac:parameter ac:name=\"separator\">undefined</ac:parameter><ac:parameter ac:name=\"hidecontrols\">false</ac:parameter><ac:parameter ac:name=\"ganttArrows\">%7B%7D</ac:parameter><ac:parameter ac:name=\"innerLabels\">Percentage</ac:parameter><ac:parameter ac:name=\"lineSettings\">undefined</ac:parameter><ac:parameter ac:name=\"is3d\">true</ac:parameter><ac:parameter ac:name=\"interpolation\">undefined</ac:parameter><ac:parameter ac:name=\"line-settings\">undefined</ac:parameter><ac:parameter ac:name=\"pieKeys\">DD‚Ф‚X‚A</ac:parameter><ac:parameter ac:name=\"colorColumns\" /><ac:parameter ac:name=\"colors\">#8eb021,#d04437,#3572b0,#f6c342</ac:parameter><ac:parameter ac:name=\"barColoringType\">mono</ac:parameter><ac:parameter ac:name=\"hide\">true</ac:parameter><ac:parameter ac:name=\"tfc-width\" /><ac:parameter ac:name=\"tfc-height\" /><ac:parameter ac:name=\"type\">Pie</ac:parameter><ac:parameter ac:name=\"aggregation\">Value</ac:parameter><ac:parameter ac:name=\"column\">Name</ac:parameter><ac:parameter ac:name=\"formatVersion\">3</ac:parameter><ac:parameter ac:name=\"macroMarker\">1644937994860_944</ac:parameter><ac:adf-mark key=\"data-consumer\"><ac:adf-data-consumer-source>046ac8ab-de13-4f1a-a05f-473773a34812</ac:adf-data-consumer-source></ac:adf-mark></ac:structured-macro><p></p>",
      "representation": "storage"
    }
  },
  "version": {
    "number": 45
  }
}

[OleksandrBeztsinnyi]

Hi @akhaneev,

It looks like a bug. I raised the issue with the team responsible for this API. In the meantime, I believe using the ADF format instead should work.

  "body": {
    "atlas_doc_format": {
      "value": "#THE BODY IN ADF#",
      "representation": "atlas_doc_format"
    }
  },

Thanks for reporting the issue.

[GTD]

Hi @TonyGoughAdaptavist

Thanks for your feedback. I’ve responded to your questions inline.

@TonyGoughAdaptavist

What should be made explicit here is the additional requirement that you must insert target macros using the new context toolbar UI element found on source macros.

Yes, thanks for noticing. That’s the current state as released to allow for vendors like yourself to initially adopt Referentiality. However we are working on other methods to insert a macro utilizing the contextual toolbar that will allow source macros to be freely inserted within the page, and targeted for later use as shown within the RightContextPanel section. Developer Preview of Referentiality within Confluence Released!

@TonyGoughAdaptavist

The one new piece of functionality introduced here which was not available using AP.Events is the introduction of published data from native macros such as Table. It should be noted however that the data emitted could already have been retrieved by a custom macro using the content API - establishing the correct relationships could also be done this way, based on page placement of the macros.

Some additional restrictions from my testing:

Does not work if the source macro is inline
Does not work if the source macro is rich-text
Is this because Forge will not support inline and rich-text macros, or are these planned?

Yes, the content could have been made available with existing apis for each individual macro; however, the macros would not have been chainable by our end users. The next milestone that uses the Right Context Panel will allow for the selection of macros that could not be done with existing apis.

That’s right. It wouldn’t work for outputType inline and bodyType rich-text macros as these do not have an iframe to trigger receipt or transmit data.

@TonyGoughAdaptavist

Any string value can be used for the outputType and inputType properties
Macros are matched based on matching values in these properties
So what is special about table-adf and table-json?

Yes, that’s intentional to support the dynamic use cases of macros that wish to share data. These two types were created to support the different uses Native and Connect cases defined above. This allows third party macros to target data from Native Tables, which could then transform and extend the existing data within the Table.

@TonyGoughAdaptavist

The Native Chart functionality was not there so I could not verify this. Based on the matrix in the original post, Chart should be visible on Table’s context toolbar even before adding any custom macros, but on my instance it was not.

We’ve flip the flag to enable it, can you try now?

[akhaneev]

One more feature request: make the AP.register({ data_provider }) working in the custom macro editor. For now, I can’t get the source data when editing macro to populate custom inputs with the available data.

[akhaneev]

@GTD , can we expect an option to hide macros from the insert menu and keep them in the referentiality panel available to be developed?

[GTD]

@akhaneev

One more feature request: make the AP.register({ data_provider }) working in the custom macro editor. For now, I can’t get the source data when editing macro to populate custom inputs with the available data.

@akhaneev

can we expect an option to hide macros from the insert menu and keep them in the referentiality panel available to be developed?

Of course, I’ll add these to the backlog for the team to discuss and prioritize as needed. From the user’s point of view, it can be confusing to have macros available in the toolbar but not with within the insert menu.

[akhaneev]

Thank you for adding to the backlog. I think that from the user’s point of view, it can be confusing to be able to insert macros and not be able to set the data source. Most of the users won’t even know what the data source is. They will just create numerous support tickets asking how to configure the macros that can’t be configured in this way.