Upcoming changes to modernize search REST APIs

Hi @SimonKusterer, you are correct. We’ve created a bug to track this here: https://jira.atlassian.com/browse/CONFCLOUD-70462

I see these changes on our EAP server now. Unfortunately, I am seeing some very squirrelly CQL searches now.

Given a search like this:

cql=type in (page) and space = "IT1" and title ~ "Decision" order by title asc

We are getting results like this:

{
  "results": [
    {
      "content": {
        ...
        "title": "@@@hl@@@Decision@@@endhl@@@ log",
        ...
      },
      "title": "@@@hl@@@Decision@@@endhl@@@ log",
      ...
    }
  ],
  "start": 0,
  "limit": 25,
  "size": 1,
  "totalSize": 1,
  "cqlQuery": "type in (page) and space = \"IT1\" and title ~ \"Decision\" order by title asc",
  "searchDuration": 52,
  "_links": {
    ...
  }
}

Notice all of the bizarro @@@hl@@@ and @@@endhl@@@ annotations.

What’s also odd is that your (Atlassian’s) Tree Search app (https://micros-cts-addon.services.atlassian.com/atlassian-connect.json) seems to be broken. When looking at the errors in the browser console, I see these:

bundle.js:formatted:46710 TypeError: Cannot read property 'replace' of undefined
    at t.i.transformDescription (bundle.js:formatted:65010)
    at i.calculateDescWidth (bundle.js:formatted:65029)
    at Ta (bundle.js:formatted:48252)
    at Na (bundle.js:formatted:47873)
    at Sa (bundle.js:formatted:47830)
    at Ea (bundle.js:formatted:47764)
    at $i (bundle.js:formatted:47684)
    at Object.enqueueSetState (bundle.js:formatted:45798)
    at t.x.setState (bundle.js:formatted:43032)
    at s.updateSearchTree (bundle.js:formatted:20939)
yi @ bundle.js:formatted:43335
n.callback @ bundle.js:formatted:43335
co @ bundle.js:formatted:43335
lo @ bundle.js:formatted:43335
Ta @ bundle.js:formatted:43335
Na @ bundle.js:formatted:43335
Sa @ bundle.js:formatted:43335
Ea @ bundle.js:formatted:43335
$i @ bundle.js:formatted:43335
enqueueSetState @ bundle.js:formatted:43335
x.setState @ bundle.js:formatted:42971
s.updateSearchTree @ bundle.js:formatted:19861
(anonymous) @ bundle.js:formatted:49709
success @ bundle.js:formatted:49709
n @ all.js:99
b._handleResponse @ all.js:61
d._receiveMessage @ all.js:21
all.js:16 [Simple-XDM] Cannot read property 'replace' of undefined TypeError: Cannot read property 'replace' of undefined
    at t.i.transformDescription (https://micros-cts-addon.services.atlassian.com/bundle.js:136:281849)
    at i.calculateDescWidth (https://micros-cts-addon.services.atlassian.com/bundle.js:136:282184)
    at Ta (https://micros-cts-addon.services.atlassian.com/bundle.js:120:90516)
    at Na (https://micros-cts-addon.services.atlassian.com/bundle.js:120:84268)
    at Sa (https://micros-cts-addon.services.atlassian.com/bundle.js:120:83769)
    at Ea (https://micros-cts-addon.services.atlassian.com/bundle.js:120:82848)
    at $i (https://micros-cts-addon.services.atlassian.com/bundle.js:120:81763)
    at Object.enqueueSetState (https://micros-cts-addon.services.atlassian.com/bundle.js:120:47344)
    at t.x.setState (https://micros-cts-addon.services.atlassian.com/bundle.js:112:1666)
    at s.updateSearchTree (https://micros-cts-addon.services.atlassian.com/bundle.js:56:19220)

Inspecting the affected line in the dev console yields this code:

            i.transformDescription = function(e) {
                return e.replace(/@@@hl@@@/g, "<strong>").replace(/@@@endhl@@@/g, "</strong>")
            }

Clearly there’s a connection here, and Atassian seems to be anticipating the presence of these odd annotations, but I can’t quite figure out what’s going on.

Please, please, please, take a closer look at the CQL changes before you ship what’s currently on the EAP servers!

At the very least, please enlighten us as to what we should do to react to this new oddity.

3 Likes

@rwhitbeck @DavidRizzuto,

Hey Guys,

We are all for modernizing APIs, but it looks like things are pretty broken at the moment.

Please push the date back a week AFTER you think you’ve fixed all the bugs.

Atlassian’s own Tree Search app is broken which is probably related.

Looks like the team might need to spend a bit more time writing tests. In fact it might be handy if you shared your test coverage with the vendor community.

Our efforts to talk Atlassian into giving Cloud devs EAP servers just paid off big time (for both vendors and Atlassian) otherwise this would have caused another massive outage and three alarm fire. Thank you for the EAP servers.

Its late in the day here so looking forward to engaging more tomorrow.

Thank you!
Brendan

3 Likes

Just some more quick info. Atlassian’s own Tree Search app currently (before the new API) works great:

However when run on an EAP server with the new API the app errors out, likely with backend errors but this in the console:

TypeError: e is undefined
    transformDescription https://micros-cts-addon.services.atlassian.com/bundle.js:136
    calculateDescWidth https://micros-cts-addon.services.atlassian.com/bundle.js:136
    Ta https://micros-cts-addon.services.atlassian.com/bundle.js:120
    Na https://micros-cts-addon.services.atlassian.com/bundle.js:120
    Sa https://micros-cts-addon.services.atlassian.com/bundle.js:120
    Ea https://micros-cts-addon.services.atlassian.com/bundle.js:120
    $i https://micros-cts-addon.services.atlassian.com/bundle.js:120
    enqueueSetState https://micros-cts-addon.services.atlassian.com/bundle.js:120
    setState https://micros-cts-addon.services.atlassian.com/bundle.js:112
    updateSearchTree https://micros-cts-addon.services.atlassian.com/bundle.js:56
    startTreeLoadProgressivelyByCQL https://micros-cts-addon.services.atlassian.com/bundle.js:136
    success https://micros-cts-addon.services.atlassian.com/bundle.js:136
    n https://connect-cdn.atl-paas.net/all.js:99
    _handleResponse https://connect-cdn.atl-paas.net/all.js:61
    _receiveMessage https://connect-cdn.atl-paas.net/all.js:21
bundle.js:120:66351
[Simple-XDM] e is undefined t/i.transformDescription@https://micros-cts-addon.services.atlassian.com/bundle.js:136:281840
t/i.calculateDescWidth@https://micros-cts-addon.services.atlassian.com/bundle.js:136:282184
Ta@https://micros-cts-addon.services.atlassian.com/bundle.js:120:90516
Na@https://micros-cts-addon.services.atlassian.com/bundle.js:120:84268
Sa@https://micros-cts-addon.services.atlassian.com/bundle.js:120:83771
Ea@https://micros-cts-addon.services.atlassian.com/bundle.js:120:82848
$i@https://micros-cts-addon.services.atlassian.com/bundle.js:120:81763
enqueueSetState@https://micros-cts-addon.services.atlassian.com/bundle.js:120:47344
x.prototype.setState@https://micros-cts-addon.services.atlassian.com/bundle.js:112:1666
t/s.updateSearchTree@https://micros-cts-addon.services.atlassian.com/bundle.js:56:19220
t.startTreeLoadProgressivelyByCQL/</<@https://micros-cts-addon.services.atlassian.com/bundle.js:136:288480
success@https://micros-cts-addon.services.atlassian.com/bundle.js:136:292401
n@https://connect-cdn.atl-paas.net/all.js:99:431
2 Likes

I have done a but more research this morning and found that the primary difference I can identify is that the value of results[].content.title have changed on the EAP release to include the @@@(end)?hl@@@ values when they do not in production.

Looking at the Confluence Server REST API docs I see that there is an excerpt query param that accepts a value of none but defaults to a value of highlight. This query param is not documented in the Confluence Cloud REST API docs. Adding excerpt=none to the search query URL in Cloud does seem to revert the API to its current production behavior of not including these highlight annotations in the search result content titles.

In production, the excerpt=none flag also works to suppress the annotations in results[].title field, and the default appears to be excerpt=highlight. So the behavioral change that appears to be happening now in the new search API in the EAP is that the results[].content.title now respond to excerpt=* values, when previously they did not.

While easy to work around now that I reverse engineered the secret sauce, this looks to me like a backward-incompatible behavioral change on the EAP, and should be reverted before shipping to avoid breaking unsuspecting apps and API consumers.

Please Atlassian, respond before the July 24th rollout date so we can properly set our expectations as to whether or not you plan to fix this. In any case, please document the existence, effect and behavior of the secret excerpt query param in the search API docs.

I’m not exactly sure why the AtlassianLabs Tree Search app is broken with the EAP search API, but am not going to dig into that any further myself.

10 Likes

It seems that this has been shipped to production now. That’s quite fucked up.

I did some more digging and it turns out that this only breaks our app in some places, not in others. The reason seems to be that the content title only gets messed up when there is no content-related expansion. Adding even a bogus expansion like expand=content.title or expand=content.asdf fixes the broken content titles. expand=content alone is not enough though.

Atlassian, please overthink your bugfixing policy! Please provide us with a proper way to report bugs, and implement a policy to fix all bugs, not just a few ones that seem to be critical to you. Your strict no-bugfixing policy is making our jobs less and less fun every day. Our customers are complaining to us that our software is not working, when it is in fact Confluence underneath that is broken. In some weeks I spend more than half of my time investigating and finding workarounds for Confluence bugs, and some of those would be so easy to solve that I would be able to open a pull request to fix them in half the time, had I access to the Confluence source code.

You are pushing us to get more customers on Cloud, but then make it almost impossible to make good software for Cloud because it is so broken. Some of our customers are even migrating from Cloud to Server because of how broken Cloud is.

5 Likes

I’m not sure, but I suspect that recent work on the search API may have resulted in some unexpected changes in behavior.

We are receiving reports of customers who are getting empty/zero results for simple CQL searches like
type in (page) and space = “CAM” order by title asc
For a space w/ key CAM. They know there are pages in that space that they have access to and the search returns an empty result.

I could not reproduce that on my instance but while investigating I ran into odd behavior for searching for spaces on the Space Directory page. For example, I have a space “Ture Space” and it is found if I type “Ture” or “Ture Space” into the filter input but if I type “Ture S” it is NOT found. I thought that the Space Directory search used to work but maybe it didn’t?
From the Network tab in my browser dev tools I can watch the search API work (and fail?).

Trying to find “Ture Space” by typing “Ture S” into the filter:

https:///wiki/rest/spacedirectory/1/search?query=Ture+S+OR+Ture+S*&status=current&limit=24&start=0&_=1595605…

That search finds nothing:
{“results”:[],“start”:0,“limit”:24,“size”:0,"_links":{“base”:"https://…

This search finds the “Ture Space”:
https:///wiki/rest/spacedirectory/1/search?query=Ture+Space+OR+Ture+Space*&status=current&limit=24&start=0&_=159560…

2 Likes

@TureHoefner Might you be experiencing this problem? View-restricted pages cannot be found through AP.request('/rest/api/search') anymore

Thank you @candid. Yes, indeed, we have customers who have reported problems with the search API that they say started recently and we determined that root cause was related to the Ap-Client-Key request header you mention in that other thread.

I will follow/comment over there.

Have you tried switching from /rest/api/search to /rest/api/content/search ?

@candid - I completely understand your frustration here and apologize for the headache this has caused. I know this is not the best of outcomes with this issue.
Across all of Atlassian’s teams that engage with our ecosystem and partners, we are currently undergoing a holistic review and revamp of our current processes for addressing bugs and feature requests. We know we need a more systematic and structured approach to address these issues, and our aim is to be more responsive and more proactive with these issues so to minimize the frustration it is causing our partners and their customers.
In the meanwhile, we really appreciate your honest feedback, please continue to let us know how we can do better.

1 Like

Hi @JasonPhan,

Thanks for engaging.

Let me ask one single question:
What very specifically is going to change to prevent this next time?

Multiple devs discovered and told Atlassian about this breaking change (breaking customers, apps, AND EVEN breaking Atlassian’s apps). It seemed inconceivable that it wouldn’t be delayed+fixed, then…once again, it just rolled out, like a curling stone, but gliding right off a metaphorical cliff.

image

Useful answers:

  • “Vendors can now go to page https://stopbreakingchanges.atlassian.com and can halt the rollout. Click a button and halt it, describing the problem.” #SOLUTION
    or
  • “Atlassian is building the above page and it will take three weeks. Then vendors with a commercial cloud app can use it.” #SOLUTION
    or
  • Here is a way that would have prevented the last 20 breaking changes we are implementing and here are the details of what and when we are implementing. #SOLUTION
    or
  • Send smoke signals from the top of Uluru, literally. (If that worked I’d book a flight for my intern right now.) #SOLUTION
    or
  • Connect and Confluence Cloud is not a platform Atlassian takes seriously. That’s why Atlassian hasn’t built a single non-trivial feature using it (integrations don’t count here) in 8 years or EVER (except kind of Confluence Questions which is panned by most reviews). Even though we say we care about Cloud apps and migrating customers, we have no intention of seriously supporting this platform or putting resources to it. #AnHonestAnswer

Non-useful answers:

  • “We care” (but give no specific actions. Caring is amazing, but those words without action cause a zero’ing out of trust and undermining goodwill Atlassian had built up).
  • “We are discussing this internally and urgently” (but give no specific actions).
  • “We know we need to do better.” ( You’re smart, we know you know, but there is a huge disconnect from knowing to action here. Things are getting less reliable not better.)

please continue to let us know how we can do better.

I love the question. Please show vendors there is a reason for us to answer the question, backed by action. Have a look at the last three incident reports. I’m looking forward to the incident report for this one.

When vendors say ‘this new change we see on EAP will break customers, vendors and in this case even Atlassian apps’ then just very very simply please don’t roll out that change.

The bar is pretty low. I wish I had a large team of talented developers (vendors) that paid ME (marketplace) so they could find bugs I created. I know I’d listen and leverage their expertise. To do otherwise squanders a valuable resource.

The fact that this happened last week and happens quite often since 2018 shows just how far away the Confluence Cloud and ecosystem teams are from demonstrating interest or ability to provide platform stability. Words are a great start, but it’s now been years with continuing degradation from the ecosystem perspective for Cloud.

Right now Atlassian is absolutely on track to follow SalesForce : “Salesforce was rated the Most Dreaded Tech tool for the Year in 2015”. It might take a couple more years of this trajectory for Cloud app dev (not server), but it is on track and imminent. I think the teams causing this chaos know it too. What I’m trying to figure out is who, if anyone, really cares?

SF turned their title. How? Real effort, leadership, resources and especially dogfooding, none of which is apparent to vendors by Atlassian currently. And the dogfooding specifically is non-existent for Confluece Cloud. In fact it is decreasing (I’m happy to back all of this up with specifics, I’ve already done so but will limit the novella for a later time and depending on certain outcomes).

I’ve been a huge Atlassian fan, supporter longer than anyone on this thread. I’ve ‘probably’ written more code for free, used by more Confluence customers than anyone(not the most, just the most used that no one ever paid for - especially once Atlassian bundled some of my tools, for free, into Confluence). I care about Atlassian a lot. It feels I care more than many Atlassians.

I care so much I will GIVE Atlassian a fork of a very serious, very profitable app that took thousands of hours to write and recreate every time Atlassian breaks more of the platform (Fabric as one perfect example). And I would do this so Atlassian will have one single real Connect app on Confluence to call their own. I believe that could be a first step to making the platform stable, richer, giving customers the quality they deserve while promoting actual partnership and caring.

Every two or three years now almost everyone we vendors interface with at Atlassian rotates out, leaves or is promoted, etc. Similar messages come and go from different people to vendors that predated them all. Before hitting repeat it might be useful to search back for how many dozens of times the same things have been said since 2012. Or even just read the most recent Atlassian incident reports, but double check the comments which are required for accuracy.

I think most all the people on these teams are smart, talented, honest, hardworking team members. Some are even truly excellent. However, your teams are clearly either understaffed or unmotivated to affect improvement from our perspective. Otherwise, I think someone would be writing integration tests and running them? And it is not because Atlassian’s share of vendor revenue from the MarketPlace vendors can’t pay for the whole party.

I’ll end this on a positive note. I do my best to create solutions. I’ll never ask anyone to improve without sharing ideas, some potential roadmap or brainstorming together. To circumvent the above issue on its ~hundredth re-occurrence I created the Cloud EAP idea (not rocket science I know, rather obvious) and opened a ticket, quickly supported by dozens of vendors. After being told by Atlassian and other vendors it wouldn’t happen, Shrey and a few other Atlassians that cared actually pushed it through. https://jira.atlassian.com/browse/CONFCLOUD-65807

So thank you for that win Confluence Cloud team and ecosystem team! It was a huge improvement. Now let’s together follow up that singular significant improvement with a second one that is meaningful to the same degree. I’ll cross my fingers for a third, but let’s just start with one.

If anything I’ve commented on above is inaccurate, please feel free to correct me with specifics. I’d love to be wrong about almost all of it.

Cheers,
Brendan

25 Likes

I have some more concrete wishes:

  • Implement an internal policy that your aim is to fix all bugs. Open your bug tracker to us so that we can report bugs there. Right now the procedure when we find a bug is that we contact dev support, someone only responds if it is a bug that we couldn’t already find a workaround for, then they spend hours to reproduce the issue and find a workaround for us. Only if they can’t find a workaround, a bug is created in the bug tracker, and that is never fixed, unless a serious amount of customers are affected by it and someone manages to convince their personal contact at Atlassian to make some noise there.
  • Give us access to your source code repository and allow us to open pull requests. We have the skill, motivation and time to fix your bugs for free. Some bugs would take 1 hour to fix, whereas finding a workaround takes 3 days. With Confluence Server, it was a huge advantage that we had access to the source code and could find the source of the bugs ourselves. Although sometimes the fix was a one-liner and Atlassian still hasn’t implemented it for years (but at least on Server, we could implement it ourselves because apps had the power to do so).
  • Announce every change you deploy on Cloud beforehand, deploy it on the EAP instances at least 4 weeks before production, and don’t deploy it on production if any bugs have been found in it.
  • Optimise the path of communication between your developers and us. Actively involve your developers in dev support and the dev community. I don’t know how much this is the case right now, but sometimes it feels like I’m communicating with people whose job it is to help me deal with a broken Confluence, rather than with people whose job it is to fix a broken Confluence. I also often feel when I’m reporting a technical defect or even a security hole that I’m communicating with someone who doesn’t know much about the technical internals of Confluence.
13 Likes

@candid, Excellent, specific, actionable ideas!

If I were able to add only one it would be:

  • specific, detailed and time-lined plan to dogfood APIs and platform in ways that are meaningful and transparent. If Atlassian doesn’t feel the customer pain from their breaking updates, removing features, etc. its unlikely to improve, a near certainty in fact.

maybe one more:

  • Align and publish metrics for some Atlassian team members that align with MarketPlace success. We don’t know what an internal ‘good review’ looks like, but there currently seems to be little or limited synergy. To quote another conversation “they’re trying but their not measured on the right things.” Since Shrey left who is the new advocate, is anyone incentivized in that role?

To your point about announcing changes, it seems this is a step in the right direciton although I’m not clear yet if it offers complete coverage: Check out the new Confluence Cloud Changelog

Props to @rwhitbeck and team for that.

2 Likes

Hi Brendan

Thank you for your response. We absolutely value your feedback.

I want to focus on answering your main question, which is a valid one:

What very specifically is going to change to prevent this next time?

You know this better than I do, but:

  • Right now the communication channels between our partners and us is not clear: Our partners raise requests to Atlassian team members through multiple streams. Issues and requests may not be appropriately prioritized and they sometimes get lost
  • In addition, once we receive feedback, we are not transparent enough to provide updates on the status of these requests

I am actively focused on improving our marketplace partner communication channels. The Atlassian team is currently working through a better process for this, which we hope to communicate in the coming weeks. This process will include:

  • Laying out a more transparent process for how to submit a request / issue / bug.
  • Increased transparency around the status of a request - e.g., whether it will or will not be prioritized and a high-level timeline

Again, we will share details in the coming weeks, but this will include:

  • A single place to submit and view requests / issues / bugs
  • A designated point of contact for each request
  • SLAs for triaging and progressing these requests
  • Clear status labels for each requests, including whether it is planned or not planned (and if there is a recommended workaround), and high-level timelines, if relevant. @candid - We cannot aim to fix every bug; however, we will aim to at least be clear and transparent around what will be addressed (and by when) and what won’t be (and why)

In addition, we are also working on a clearer public roadmap of what our Ecosystem teams are working on, so that all of our marketplace partners have more clarity around what we are focused on to help improve the experience of our developers

Finally, Brendan: I know you have been a fan of Atlassian for years and we appreciate all that you have done for the community. Your constructive feedback very helpful and exactly what we need to improve. If you (or others here) have time and feedback, I would love to hear them. Please reach out to me at jphan@atlassian.com and I would like to grab some time to chat more.

Thank you,

Jason

Note: Related to this post, we are working to ensure that the bugs reported here will be addressed. As of this posting, this is the latest status:

Major issues

  • “Impersonation for apps have incorrect permissions” - ETA for fix: Aug 4
  • “Nested Page properties report macro” - Eta for fix: Aug 5
  • “Mobile search API requests still using old pagination” – Requires further investigation

Other issues

  • “Highlighting of search text inconsistent with out API” - Eta for fix: Aug 4
  • “Wildcard searches not working correctly” - Requires further investigation
  • “Failing searches via graphQL” - Eta for fix: Aug 7
  • “403 rate in production” - Eta for fix: Aug 7
  • “Create page dialog occasional 500s” - Requires further investigation
  • “Invalid pagination cursors” - Requires further investigation
6 Likes

Hey @JasonPhan,

Thank you for the thoughtful response and engagement. The specifics you provided are excellent and encouraging!

I agree specific, actionable, reliable, time aware communications are one of the best places to initiate what I hope is an ecosystem renaissance away from what feels like a dark age. That said there is a lot road for relatively quick wins and hopefully this will be one.

The following is meant to underscore and augment the power and positive impact of the rest of your post.

Ambiguities:
hope → vendors would like to see ‘will’ or ‘definitely will’
coming weeks → when exactly? August 20th 2020 or August 2021?
communicate → I assume you will post to CDAC but not sure

I want to provide positive re-reinforcement and absolutely applaud your specifics. The above ambiguous phrase thrusts the other wonderful, meaningful, actionable, specific, significant statements into limbo.

If dates given slip, that’s completely understandable. They can be reset at that time.

This might feel like an over-reaction but if you consider the context I know you’ll see the value in that clarity. I am excited about the rest of the post.

I’ll reach out to you tomorrow by 5p.m MST to schedule a time to chat next week or when you’re available.

I hope you’ll continue to consider the rest of the requests above by @candid seriously.

Cheers,
Brendan

image

3 Likes

Please also consider following query that broke recently (urldecoded for readability), not sure if you are aware of it
/rest/api/search?cql=type=space and (title ~ 'A*' or space.key = 'ABC')
it returns empty resultset

the OR part breaks it, as separate queries are working OK and each of them returns nonempty resultset
/rest/api/search?cql=type=space and title ~ 'A*'
/rest/api/search?cql=type=space and space.key = 'ABC'

5 Likes

I’m getting the wrong results with this…

  1. Initial load: /rest/api/content/search?limit=1&cql=${foobar} => contentX
  2. Load next: /rest/api/content/search?limit=1&next=true&cursor=[etc]&cql=${foobar} => contentY
  3. Load previous: /rest/api/content/search?limit=1&prev=true&cursor=[etc]&cql=${foobar} => contentY

Step 3 should be loading contentX.

Do we need to pass specific parameters in the initial load or is this a bug?

2 Likes

This also affects our app and customers. The only work around that we see is to make two separate calls which is relatively slower. This s not the most elegant solution but at least it works.

It will be great if it can be fixed by the Confluence team @JasonPhan

“Wildcard searches not working correctly” - Requires further investigation

@JasonPhan Any news on the status of investigating this and some of the other CQL issues raised in this thread? Is there a good place for us to track that, such as in a CONFCLOUD issue?