RFC-23: Confluence Databases

[MatthewMachuca]

RFCs are a way for Atlassian to share what we’re working on with our valued developer community.

It’s a document for building shared understanding of a topic. It expresses a technical solution, but can also communicate how it should be built or even document standards. The most important aspect of an RFC is that a written specification facilitates feedback and drives consensus. It is not a tool for approving or committing to ideas, but more so a collaborative practice to shape an idea and to find serious flaws early.

Please respect our community guidelines: keep it welcoming and safe by commenting on the idea not the people (especially the author); keep it tidy by keeping on topic; empower the community by keeping comments constructive. Thanks!

Summary of Project:

We are planning to launch Confluence Databases to our customers. Currently a closed set of Early Access Program (EAP) customers are testing Databases and are receiving incremental updates. In one of the upcoming releases, we’ll be integrating Databases as a new content type within Confluence that will exist within the current navigation and creation structures, including the content tree. This update would impact API responses for some of our existing APIs, and we’re mindful of the potential effects on your apps. We deeply value our partnership with you and would like to invite you to validate these changes in a staging environment to assess any impact on your apps. Your collaboration and feedback are essential to us as we work together to deliver a seamless experience to our mutual customers.

• Publish : 30 August 2023
• Discuss : ~20 September 2023
• Resolve : ~6 October 2023

Problem/Opportunity

We were very excited to announce Databases coming to Confluence in a community post earlier this year. Databases are structured collections of information organized in tables, where advanced functionality makes it easy for you to change data and content as often as your work does.

Databases will exist as a native content type within Confluence, with functionality and behaviors similar to what pages have today. These include but are not limited to existing in the tree, being able to be created from the create button, and permissions.

Proposed Solution

In this post, we will provide important details about this upcoming change and what you need to know as an ecosystem developer.

What is changing in upcoming release?

In one of the upcoming releases, Databases will be added to Confluence’s current navigation structure. If you are familiar with the changes associated with the introduction of Whiteboards to Confluence, you’ll notice these changes are very similar ( previous post on Whiteboards RFC-10: Confluence Whiteboards ).

Here are the most relevant upcoming changes:

Databases in the Content Tree (currently known as Page Tree)

• The “Page” Tree will be renamed to “Content” Tree [Note: This change also came when Whiteboards were released to Beta]
• Databases will co-exist in the content tree with other content types such as Pages and Whiteboards, and they will be able to be parent nodes as well as child nodes

Content Actions (in the content tree) available for Databases

• Content actions including Rename, Get Link, Copy, Move Archive, and Delete will be available for Databases similarly to Pages

Database Creation

• Databases can be created similarly to pages’ creation mechanism

• Databases can be created both through the top navigation ‘Create’ button
  

• Databases can be created through the “+” button next to Content in the sidebar.

Database Permissions

• The Database permissions model would be similar to the pages model i.e.
  • “Anyone can view and edit”
  • “Anyone can view, some can edit”
  • “Only specific people can view or edit”

• Databases will also inherit permissions from the parent in the content tree, if applicable.

When will this change take place?

These changes will affect a cohort of EAP customers who are currently testing Databases, and will eventually affect all users who opt-in to Beta at launch.

• EAP Release Timeline:
  • The above changes will be released to some of our EAP customers beginning in late October or early November.
• Databases Beta Release:
  • Once the Database Beta becomes available for all customers (exact timeline TBD but tentatively December onwards), Confluence site administrators will have the option to opt-in and enable Databases for their users.
  • Along with Databases, the changes to the tree and create navigation will also change for those who opt-in to beta
  • It’s important to note that customers who choose not to opt-in for the Database Beta will not see the above changes until Databases are officially released in General Availability (GA) later next year. [Note: The changes to the tree, will update prior GA of Databases when Whiteboards releases to GA early next year.]

Actions

How it might this affect you?

With the upcoming changes, some of our APIs would be updated to include Database-related information in the response. Examples of such APIs are:

1. Rest APIs
   a. Endpoints that return children/descendants and ancestors for a Content that support the following expand parameters will be affected. E.g.
   i. ancestors - will now return Databases in the response.
   ii. childTypes.all - this will not be impacted immediately but will include Databases in the future (when we GA)
   b. Endpoints under Content - children and descendants will not be impacted immediately but we will include Databases in the response when we GA.

   /wiki/rest/api/content/{id}/child

   /wiki/rest/api/content/{pageId}/move/{position}/{targetId}

   /wiki/rest/api/content/{id}/child/{type}

   /wiki/rest/api/content/{id}/descendant

   /wiki/rest/api/content/{id}/descendant/{type}

   /wiki/rest/api/content/{id}/pagehierarchy/copy

   /wiki/rest/api/content/{id}/copy

   c. Content body expand will be empty for Databases /wiki/rest/api/content/{id}/expand=body.storage

2. GraphQL APIs
   i. ConfluencePage.ancestor

How can you help?

Apps that directly use content tree (fka page tree) in their functionality are at high risk. We would highly urge those app developers to proactively test when the changes are available through DCP in your test tenants.

As we prepare to go live with the upcoming changes for some of our Early Access Program (EAP) customers, we would like to seek your collaboration in validating and mitigating any potential impact that these changes may have on your applications.

• To facilitate this process, we will provide you with all required changes in your Developer Canary Program. This will allow you to thoroughly test your applications and report back any impact or issues that you may encounter.
• The timing for providing access is currently being finalized. We anticipate that it will be available in October. This post will be updated with additional details once we have the staging environment ready for testing.

If your application is impacted by the changes, we are looking forward to working together with you to resolve any issues before releasing the Database Beta to our shared customers.

[marc]

Hi @MatthewMachuca ,
Thank you for the announcement.
As I understand, there will be API changes because the databases will be introduced as a new content type.
Will there be also new APIs to access the databases, i.e. read write values/records into the database?

[scott.dudley]

Hi Matthew! I see that the majority of the API endpoints mentioned in this RFC are already deprecated and scheduled to be removed two months after this feature is set to be launched in beta.

Can you please:

(1) identify how (and where) databases will be represented in the REST API V.2, and

(2) refrain from launching this feature for anyone until you actually provide a way for apps to at least see this content in the REST API V.2 API?

This feature introduces a new type of content, potentially as an ancestor to other content that apps will be directly manipulating. This breaks any app that needs to (for example) determine all ancestors for any direct or indirect child of a database. Apps need to walk up the content tree, but V.2 does not allow fetching content of arbitrary types, and this creates broken links in the chain.

Developers are already under the gun to migrate from the REST API V.1 to V.2 on a very short timeline, and with a V.2 API that is not quite ready. Launching new functionality that supports only deprecated APIs creates an insane amount of logistical headaches for developers.

We really need Atlassian to respect the six-month deprecation policy as a minimum. And if you truly can’t build the V.2 APIs before launching, please coordinate with the other teams to reset the deprecation of the V.1 get-content-with-expansions API until at least six months after you actually do launch the V.2 APIs.

As a completely separate question, will databases support comments, labels, likes, custom content, content properties, or attachments?

[david]

Hi @MatthewMachuca

Some questions:

• Will you be respecting the atl.footer location so that we can hook into Confluence Databases content?
• What webhooks will be available for us to use?
• Do we still get the contentcreate, contentedit, contentview, & other content* webhooks?

Same questions for Confluence Whiteboards too – It looks like I missed the RFC for that.

Also, it seems a bit weird having to revert my app’s REST API use back to v1 for interacting with either Confluence Databases or Confluence Whiteboards.

Would you recommend just abandoning v2 REST API until it is finished?

[david]

Hi @MatthewMachuca

Some things to consider.

When I call the following JS in a connect app in the atl.footer position:

AP.navigator.getLocation((location) => console.info(location));

I get the following for a page:

{
    "target": "contentview",
    "context": {
        "contentId": "655363",
        "contentType": "page",
        "spaceKey": "EX"
    },
    "href": "https://hiya.atlassian.net/wiki/spaces/EX/pages/655363/Page+title"
}

For a Confluence Whiteboard, while atl.footer is supported (), I get the rather less useful:

{
    "target": "unknown",
    "context": {},
    "href": "https://hiya.atlassian.net/wiki/spaces/EX/whiteboard/33095682"
}

…no context is given on what the thing is, so I have no idea what I have other than by exploding the href and hoping.

What happens for Confluence Databases if we assume that they support the atl.footer extension point?

Please fill the context object with something useful.

[EmileGivental]

Hi Marc,

Operations on the cells or data within the database will not be supported for the EAP or BETA that are mentioned in the post.

[EmileGivental]

Hi david,

Thanks for the questions, I’ll answer them in order.

Will you be respecting the atl.footer location so that we can hook into Confluence Databases content?

Neither whiteboards nor databases support custom content as of now, so the atl.footer location will not be populated with meaningful information.

What webhooks will be available for us to use?

As of now, neither whiteboards nor databases will have webhooks available. We may bring support in the future.

Do we still get the contentcreate, contentedit, contentview, & other content* webhooks?

These will not be supported on day 1 of EAP or BETA.

Also, it seems a bit weird having to revert my app’s REST API use back to v1 for interacting with either Confluence Databases or Confluence Whiteboards.

Our V1 REST API will not be adding any new endpoints related to whiteboards or databases, but some existing endpoints will be impacted because of the organizational structure of the tree. The mitigations that we are offering should enable developers to ignore the changes if they choose the mitigation of replacing whiteboards/databases with pages, or the mitigation of having whiteboards/databases appear instead as null. We will add support for whiteboards/databases in V2 REST APIs before adding any endpoints in V1 (or adding any into V1 at all perhaps), so switching is still a good investment.

[scott.dudley]

Hi @EmileGivental ,

@EmileGivental

Will you be respecting the atl.footer location so that we can hook into Confluence Databases content?

Neither whiteboards nor databases support custom content as of now, so the atl.footer location will not be populated with meaningful information.

I have a parellel question for the atl.general location, but I imagine the answer is the same.

I don’t think us developers really care if databases support custom content or not. The issue is that some apps specify web-panels in these locations, and currently, those web-panels do display on whiteboards (or at least atl.general does). I assume this will be the same for databases.

But Confluence won’t tell us the context in which the panel is displayed, and this seems like a significant error.

If Atlassian is going to display atl.footer/general webpanels in those places, the context should be properly populated to at least help us identify it (so we know to ignore it). If Atlassian cannot do that for whatever reason, then these panels should not be displayed at all.

The mitigations that we are offering should enable developers to ignore the changes if they choose the mitigation of replacing whiteboards/databases with pages, or the mitigation of having whiteboards/databases appear instead as null

Are these mitigations described somewhere else that I missed? If not, can you explain how this will work? And will it apply to the REST V.2 API as well as V.1?

[EmileGivental]

Hi scott,
Thanks for the questions, and I’ll answer them in order.

(1) identify how (and where) databases will be represented in the REST API V.2

This purpose of this RFC is to discuss how existing V1 endpoints will impacted by these changes and offer solutions to developers that may have written in expectations on the content types and the fields in them.

(2) refrain from launching this feature for anyone until you actually provide a way for apps to at least see this content in the REST API V.2 API?

For EAP and Beta, there is no planned V2 API support at launch. As we build support and endpoints in V2, we will continue to discuss with the community and marketplace partners as they become available.

As a completely separate question, will databases support comments, labels, likes, custom content, content properties, or attachments?

On day 1 of EAP and BETA, we will not support these functionalities, and APIs related to them will not work on Databases. We will, however, add functionalities over time, though there is no guarantee that any specific functionality on this list will be added to databases.

[scott.dudley]

Hi @EmileGivental,

@EmileGivental

(1) identify how (and where) databases will be represented in the REST API V.2

This purpose of this RFC is to discuss how existing V1 endpoints will impacted by these changes and offer solutions to developers that may have written in expectations on the content types and the fields in them.

Thanks for your response. I would have to say that this differs from the Summary of Project mentioned above, as well as the “How can you help” section. In response to the question of how we can help, I am telling you (as are at least the 11 other developers who liked my earlier post) that we want to be able to at least identify database content types using the V.2 REST API, because moving to V.2 is what other parts of Atlassian are telling us we need to be doing.

Atlassian is telling developers on one hand that the V.1 REST API is going away in mere months and that we have to migrate everything over to V.2, meanwhile launching multiple new products in the same short window that do not support V.2 and have no stated timeline to support V.2, which will cause breakage in certain apps that try to use only V.2.

I understand that you have customers to please, with internal deadlines and OKRs to meet. I infer that nobody thought of the V.2 API for this project because V.2 work seems to be occurring in a separate silo. But this is giving off a significant “left hand, please meet right hand” aura.

Please try imagining that you built your database product using internal REST APIs. Now management has dictated that you have to rewrite 100% of your code to use (say) GraphQL by January and they will turn off all REST API access then, but you need to interoperate with another Atlassian team who is currently building some other new feature using only REST APIs that don’t yet have any GraphQL endpoints. The other siloed team hasn’t started working on the GraphQL API and they won’t tell you when they might schedule the work, but their APIs will get turned off like all of the others in January (and any new GraphQL API they might eventually produce will have a completely different API, with different shapes of response objects and a different information taxonomy). How do you plan for anything in this context?

In any event, could you please talk with the other teams, and if that fails, escalate to a higher level, figure out where to find the person-hours, and come up with an answer that doesn’t force more pain on the developer community? Like: push out the database/whiteboards launch, push out the V.1 deprecation date until you get the V.2 APIs sorted, prioritize figuring out how to provide minimal V.2 support for databases and ship something soon, or explain the previously-mentioned “mitigations” and assure us how they will apply to V.2 in a way that doesn’t break apps?

[MatthewMachuca]

Hi @scott.dudley

I apologize for the delayed response but I wanted to chat internally with a handful of people.

We’ve taken note of the concerns surrounding the v2 API. Our Ecosystem (API) team is diligently working on integrating v2 API support for both whiteboards and databases. All updates will be available on our changelog at https://developer.atlassian.com/changelog.

Support for both whiteboards and databases in v2 will be available before v1 deprecation to ensure no disruption to app functionality.

[scott.dudley]

Thanks @MatthewMachuca !

In the eventuality that you are not already doing so, I hope you will be able to talk with the Atlassian groups who are organizing the REST API v2 deprecation dates (here) to ensure that there is a consistent policy across potentially-siloed teams.

For example, I would love to see if “support for whiteboards/database in v2 will be available before v1 deprecation” could be written as “support for whiteboards/database in v2 will be available 6 months before v1 deprecation”.

[MatthewMachuca]

Important Update:

For those who have the Developer Canary Program App installed and enabled on your tenants, you will begin to see the new Database features and accompanying updates. This includes Databases in the Content Tree.

Note: Database functionality is being incrementally updated and you may not see 100% of functionality with the feature at this time. The v1 endpoint response changes detailed in the original post are available for testing and should be functional.

[MatthewMachuca]

Thank you to everyone who posted comments on the above RFC, your inputs and concerns surrounding the above worked has initiated some great conversations externally and internally within Atlassian.

As previously mentioned here please check out the Developer Canary Program to see incremental changes to Databases.

Additionally please keep an eye out on the Confluence Cloud changelog as updates regarding v2 APIs support will be available there.

I appreciate the community actively engaging and I encourage all to continue to do so.