Complex Search not working

Forge
,
1

I’m using Custom Entities with Complex queries in my Forge app. When I save one entity and search for it by either normalizeName or UUID, the correct result comes back (one record). I then save 20 additional records, all with different names and UUIDs. After that, I once again perform the SAME search for the first record. No results come back. What’s going on here? I’d like to think I’m doing something wrong, but these feels like a bug. Adding additional records should not affect whether my original record is returned. In total, I have 21 records saved -a very small amount. I plan on having 10K-20K records saved. I hope this can scale to this volume.

  storage:
    entities:
      - name: system
        attributes:
          name:
            type: string
          normalizedName:
            type: string
          uuid:
            type: string
          domain:
            type: string
        indexes:
          - name
          - normalizedName
          - uuid

Here is how I do a parallel search across both indexes. It still fails when i just limit this to a single search over one index (in my testing).

async function searchSystems(searchVal: string) {
    let systems = [];
    console.log("Searching Systems for " + searchVal);
    try {
        const systemsByNamePromise = storage
            .entity("system")
            .query()
            .index("normalizedName")
            .andFilter("normalizedName", FilterConditions.contains(searchVal.toLowerCase()))
            .sort(SortOrder.DESC)
            // .limit(5)
            .getMany()

        const systemsByUuidPromise = storage
            .entity("system")
            .query()
            .index("uuid")
            .andFilter("uuid", FilterConditions.contains(searchVal))
            .sort(SortOrder.DESC)
            // .limit(5)
            .getMany()

        await Promise.all([ systemsByNamePromise, systemsByUuidPromise]).then(searchResults => {
            console.log("handling search results: " + JSON.stringify(searchResults));
            let allResults;
            for(let i = 0; i < searchResults.length; i++) {
                const resultObj= searchResults[i];
                systems.push(...resultObj.results);
            }
        });

    } catch (e) {
        console.log("problem loading system entities: "+ e);
    }
    console.log("Returning found systems: " + JSON.stringify(systems));
    return systems;
}

Also, if I include a LIMIT in the queries it can affect whether a result comes back -even though there is only a single record that matches!!! Something very strange is going on and right now it’s not reliable enough to use.

1 Like
2

I’ll add… I’ve been doing some ‘teardown’ and ‘rebuild’ with my dataset. So I’ll delete all 100 records and later in the day re-add the same 100 records. Could this be causing a problem with the indexes? I dunno.

3

Wait a minute… does this have to do with filtering happening in memory… and I may need to page through multiple pages of empty results until I find a page with filtered results? I’ll test and let you know.

— UPDATE —
It’s still not working…

        const oneOffSystem = await storage
            .entity("system")
            .query()
            .index("uuid")
            .andFilter("uuid", FilterConditions.contains(searchVal))
            .sort(SortOrder.DESC)
            .cursor(null)
            .getMany()
        console.log("oneOffSystem: " + JSON.stringify(oneOffSystem));

No cursor is returned in the response, so there are no additional pages to go through and filter.

4

If I remove the sort() function it can sometimes allow me to get results back. But if a use a different search value with sort() removed, I get no results and no cursor. But then I add sort() back in the query and now I get a result with a cursor. There is clearly something going on with sort() and limit() that are affecting whether I get any results and a cursor. But one thing is consistent, if I don’t get any results, then I also don’t get a cursor.

In short, I don’t think the andFilter()/orFilter() is working properly. When I use the the where() clauses it works as expected. But since there is no WhereCondition.contains() function, I can’t use a where() for my search. I’m searching for strings here and I’d like to allow people to search for values in the middle of the name/string and not just ‘beginsWith()’.

5

I raised an issue for the Atlassian engineers. ECOHELP-28945

6

Hi @AndrewSchoewe can you help me verify few things, when you are using a andFilter/orFilter are you getting a cursor back in the results?

Also there was a disclaimer mentioned in filter methods https://developer.atlassian.com/platform/forge/runtime-reference/storage-api-query-complex/#filtering-methods

the andFilter and orFilter methods are in-memory filters. Using them can sometimes produce pages with no results, with cursor pointing to the next page where actual results exist.

So, can you help me verify this?

7

I am NOT getting a cursor back. And I think this is the root of the problem. I now realize the filtering is done in memory. But in this case, since I don’t have a where clause, I would expect it to return ALL of the records (paged of course), and then the filtering would be applied with each new page of results. But in this case, no cursor is returned, so filtering does not happen on subsequent pages of results.

Realizing how this will work, it’s very inefficient if I have 20K records and it pages through them 20 records at time while applying the filter. Are there any plans to offer ‘contains()’ functionality in the WhereCondition? This would have the search happen at the DB level, I imagine, but be much more efficient.

1 Like