New Optional Include Field Parameters in the Confluence Cloud v2 REST APIs

Confluence Development Confluence Cloud
, ,
1

Hey all!

Confluence Cloud recently released a set of include field parameters to our Get page by id and Get blog post by id endpoints (see changelog here).

One of the recurring themes in the feedback we’ve received when migrating from the to-be deprecated v1 APIs to the v2 APIs is the sheer number of requests needed to retrieve the same amount of information in v2. We’ve added the optional include fields to help address this issue and cut down on the number of calls needed.

For example, clients can now retrieve labels associated with a page via /wiki/api/v2/pages/{page-id}/include-labels=true.

A few characteristics of the include fields:

  • the included fields are metadata of the main entity (e.g. labels, properties, operations) and thus will fall under the same OAuth scope
  • pagination, custom sorting, and filtering of the included fields are not supported

We’d also like to take this opportunity to thank the developers and teams who’ve started to migrate over to the v2 APIs. For those who are still planning, we hope this addition helps make the transition easier. We continue to encourage teams to report any gaps they’re facing to developer support as we’re constantly monitoring for gaps and suggestions.

7 Likes
2

Cross posting another question that was asked:

I am glad to see the recent addition of “include-xxxxx” query parameters in CHANGE-1420 . Are there plans to bring this change to custom content? (As well as possibly databases and whiteboards?)

If an application needs to be able to deal with all of the various primary content types, it is quite complicated if there are “shortcuts” to fetch this information with certain content types but not the others. For example, if “include properties” or “include operations” are available only for pages and blogposts, then developers still need to write and test the code to fetch these items separately anyway. This is no less work than before, operations become oddly slower for only one specific content type, and it introduces another code path that needs to be tested separately.

Answer:

…yes! We’re planning to bring the changes to more content types. We’ve released the change for pages and blog posts first so users can have a look and see if it fits their use case. Let us know of your thoughts in that thread.

3 Likes
3

It’s really nice that expand has been brought to v2 as include. Thanks for that.

I’ve noticed that pages (and maybe databases, blogposts and whiteboards) now have owners so that we can understand who is responsible for the content ownership. This is also very cool.

Can the ownerId be added to AP.getContext() and/or AP._data.options in Connect?
That would be really handy so that we don’t have to make an extra API call.

4 Likes
4

Those developers who’ve felt pressured by unrealistic deprecation dates (we’re on the 3rd or 4th extension) have already migrated to v2 and are suddenly realising that was a waste of time as there are now much better ways to fetch the same data. I’ve already seen such posts in the forums.

Why would anyone migrate at this point in time when in this very post you’re saying that more changes and features are coming?!

For the past 18 months the community has consistently voiced some very basic asks:

  1. 100% parity with ALL v1 endpoints.
  2. 100% completion of ALL additional v2 endpoints and features.
  3. ONLY THEN do you set a SINGLE 6 month deprecation date for ALL v1 endpoints.

Any chance the v2 team can deliver on these asks?

2 Likes
5

Hi @HughWu,

Thanks again for this update! A clarification:

You mentioned that “pagination” is not supported, but the REST docs for include-properties say the following:

The number of results will be limited to 50 and sorted in the default sort order. A meta and _links property will be present to indicate if more results are available and a link to retrieve the rest of the results.

Does this mean:

  1. No more than 50 total results will be returned, and they will all be returned at once.
  2. No more than 50 total results will be returned, but they may require multiple calls to fetch using the cursor.
  3. There is no limit to the number of expansion results returned, but they will be delivered in batches of 50, and subsequent batches can be fetched using the cursor.

Also, this is not defined in the REST docs: what sort of response object shape should we expect for the pagination results when following the link with the cursor? Will we receive a copy of the original page/blog content again, simply with a different set of results in the “properties” object? (I guess that’s OK, but it’s a little weird.) Or will it point exclusively to the paginated data?

At least for the content-property include, it would be super handy if this query parameter could actually support filtering. Unlike most of the other “include” expansions, apps are usually only interested in one or two specific content properties (eg. include-properties=myprop1,myprop2), and it sucks to have to iterate through a larger list just to pluck one out. It also curiously makes my app twice as slow if other apps happen to stuff 50 unrelated content properties into an object (assuming that #3 above is correct).

It would be great if these changes could be propagated to the get-attachment endpoint too, in case that wasn’t already on the list.

Thanks!
Scott

2 Likes
6

Hey @david :wave: I do want to clarify that the expand as we know in the v1 APIs will not be in the v2 APIs. For example, we do not want to allow nested includes, including a resource that is considered a separate entity altogether (e.g. no include-space to fetch the entire space object associated with a page).

Re: the ownerId for the AP.getContext(), let me get back to you on that. It may be something that can be brought up as a Suggestion and we can gather interest in that change.

@nathanwaters :wave:
The parity has been the biggest work our team has been working to address. We’re also constantly making changes to our v2 APIs based on feedback and what we’re able to take on, so future features and improvements like these can always come in.

With that said, I understand the migration and the last 18 months has been frustrating with a lot of unknowns, and I’m really sorry for that. The intent with this post and changelog is to give partners a heads up in the direction we’re headed and really just help with plans to migrate.

@scott.dudley :wave:
So the includes will return a maximum of 50 items.

For example, let’s say a page had exactly 100 content properties. The include-properties will return 50 properties, and also include a cursor that can be used in the endpoint in the _links (e.g. Get content properties for page). The link simply points to an API that is dedicated to bulk fetching the included field (so in this case the Get content properties for page), so the response object shape will be same as the shape in that API.

Calling the same Get page by id with include-properties twice in a row will not paginate through the properties.

At least for the content-property include, it would be super handy if this query parameter could actually support filtering.

Funny enough, we actually did get this feedback in a different forum :sweat_smile: And it’s a good point, we’ll definitely keep it in mind, but at the moment of this post there were no plans for filtering include fields.

It would be great if these changes could be propagated to the get-attachment endpoint too, in case that wasn’t already on the list.

Yes, we’ll be aiming to add it to our other content types as well. We wanted to release this initial change so devs could think how best to incorporate it into their own apps.

Let me know if that answers all your questions, happy to clarify further if it didn’t!

2 Likes
7

Hi,

this is a really good addition. And partly addresses my concerns that I have described in: Deprecation of V1-Api Confluence – Major concerns.

One question I would like to ask if there are already any plans for a V2 replacement of the search endpoint. Will there be still something like cql in the future? Can we do search across different content types (pages, blog posts and comments). We currently heavily rely on searching for custom content properties.

All the Best

Sebastian

PS: Just wanting to say I’m glad that things are moving along and I’m also very happy with the improvement of communication.

1 Like
8

@david :wave: Here is the ticket for the suggestion: https://jira.atlassian.com/browse/CONFCLOUD-77722

@SebastianGellweiler :wave: I’m so glad to hear that!

One question I would like to ask if there are already any plans for a V2 replacement of the search endpoint .

There are no defined plans for a v2 replacement of the search endpoint. For the future of CQL search and the current search functionality, we’ll need to follow up with some other folks on that. Sorry I can’t give a direct answer on this end.

We currently heavily rely on searching for custom content properties.

If you want to provide more context on how you’re leveraging search we can also pass that information along to the team that actively work on that feature.

9

Hi @HughWu,

apologies for my late reply. So I started on working to porting our app to V2 so that I get a better grip on what features that we need are still missing. And I will let you know once I made progress on that. Including on how we use the search endpoint.

However one thing I noticed when scanning for code that uses deprecated apis. Are you aware that the atlassian-connect-express project still uses the /rest/api/content endpoint: Bitbucket.

All the Best

Sebastian

10

Hey @SebastianGellweiler

No worries! I appreciate all the work and feedback you’ve given so far.
Re: the deprecated usage in the ACE repository, I’ll bring it up with the team :+1:

2 Likes
11

Hi @HughWu,

so to give some context the app we’re currently porting is mailto.wiki. It’s an email integration for Confluence.

So we’ve partly upgraded to the V2 api. The configuration service now uses the V2 api where possible but other services still use V1. Overall the transformation was easier than I thought. I don’t like that the content api is now replaced by 3 apis (pages, blog posts, comments) but because they share a lot of properties we can sort of easily built abstractions around that without needing to change to much of our internal logic.

What is annoying when switching is the fact that we now have to store space key and space id along side each other because there are still a lot of V1 apis we have to use. The fact that all numeric space keys are allowed, makes this even more error prone as a simple is it all digit check can’t be used to distinguish between space keys and space ids.

Here are the V1 apis that we couldn’t find a replacement for:

/rest/api/search/
We use this endpoint to search for spaces. To propagate select2 select boxes.

/rest/api/content/search/
We use this to search for pages by title.

But the bigger issue for us is the following. Our app maps pages, blog posts and comments to emails. We use custom content properties to store metadata about emails (subject, messageid, from, to, etc.). We then use the search endpoint to match replies to existing pages, blog posts and comments. Are there any plans for V2 search endpoint that will be able to search pages, blog posts and comments simultaneously and to search for custom content properties? We’re currently thinking about using an external index for this. This would be a viable solution for us. However we need backwards compatibility for at least a year, to find old content that users created in the past so if we switch to an external search index we still need a grace period.

/rest/api/group/ and /rest/api/user/
We us this to search for groups and users by name and to get the email addresses of users.

Besides of our backend services still using V1 content apis our end to end tests also still use V1 apis.

The configuration service has now been using the V1 content api for a couple of weeks. Given this I think we can realistically switch to the new V2 apis where they are available within the next half year. But the search endpoint is the one that has me worried the most.

The deprecation headers also help us as we now have build in logging for deprecation notices which makes it a lot easier to monitor deprecations.

We will also have to make a lot more api calls. As the old api allowed us to do a lot of things within a single call (like getting a page and all it comments). Which should decrease performance. But this is in our case not much of an issue.

This is our experience and current state of migration.

All the Best

Sebastian

1 Like
12

Hi @HughWu ,

I have updated our backend APIs to the new versions where available, removing all currently deprecated APIs except in tests.

The whole update took us nearly 100 man-hours, as tracked in our system, which was a significant investment for porting just one app. Our app is probably more complex and deeply integrated into the Confluence API than others.

As expected, we now have to make more API calls due to the separation of APIs, which complicates writing transactional safe code (example).

We also have a lot of abstraction code to handle the separate APIs for pages, blog posts, and comments. We manage this with inheritance to map these new content types.

However, we still rely on many V1 APIs, including:

  • @POST("rest/api/content/{contentId}/child/attachment")
  • @GET("rest/api/content/{contentId}/child/attachment")
  • @POST("rest/api/contentbody/convert/{to}")
  • @POST("rest/api/content/{contentId}/label")

It’s especially surprising that page creation is deprecated in V1, yet there’s no V2 solution for creating labels and attachments.

Overall, the migration process was challenging, but at least we are no longer using deprecated APIs. However, the new APIs feel significantly less powerful and flexible.

On a positive note, I appreciate the less nested data structures and the new operations endpoints for querying permissions.

I hope the missing APIs will be added soon and that the new API will remain stable for a long time.

All the Best,

Sebastian Gellweiler