Which assets import tutorial is the most up to date recommendation for using async queues?

Forge
1

There are two tutorials having to do with assets import:

They both tell you to use the same forge template: jira-service-management-assets-import-type

However, the template seems to be directly related to the former tutorial. The later tutorial similarly suggests two separate queues, but defines different responsibilities for each queue. The former tutorial has an older modification date and is listed on the later tutorial as a dependency of sorts in the Before you begin section:

This tutorial assumes you’re already familiar with developing an Assets Import app on Forge and the Imports REST API for Assets. If not, see Import third party data into Assets and Imports REST API Guide.

So my hunch is that the guidance provided in the later tutorial is the current best practice for async queues associated with importing external assets, but the template just hasn’t been updated to reflect that and you are on your own to adapt the template. Can anyone confirm this?

2

@LucasTheisen,

I’ve been working on that area with some SaaS partners and trying to improve the surrounding materials. So far, I’ve been able to work up a full example app:

The concrete use case isn’t all that useful. Nobody needs a list of Products from DummyJSON. Indeed, that use case is simple enough that out-of-the-box JSON import can handle it. Moreover, for the scale of importing fewer than 200 assets doesn’t justify the complexity of using the 2 queue scenario.

That said, I think real-world usage often justifies the 2 queue configuration because it helps get around the Forge invocation limits. Here are scenarios where I would recommend the 2 queue setup:

  • Very large data sets >1000 assets. Because 1 or more of the following will inevitably apply.
  • Many items without pagination. Because you may need to introduce wait time between requests to avoid rate limiting, which will eat up a single invocation’s 25s limit.
  • Many items, many pages. Similar reason where rate limits may apply per page.
  • Slow APIs. Similar to wait times, slow APIs can eat up 25s without producing much. Esp if 1 page of data would take more than 25s, Async Events have 900s limit.

Even with async queues, there is still an expectation of “paging” (or more precisely chunking) the data.

After the sample app, I’ve been working on reworking the docs but I’m stuck interleaving that with other work so progress has been slow. If you run into specific question about assets-import please do at-mention me so I can jump in with what I know.

3

Thank you for the example. I’m really glad y’all are starting (at least in this example) migrate over to typescript, and more importantly, i really appreciate that your example has tests unlike the current documentation or template projects. That said, i do have a question about your example. You configure typescript for ES2023:

    "lib": ["ES2023"]

but when i try that on my project i get:

$ forge deploy
Deploying your app to the development environment.
Press Ctrl+C to cancel.

Running forge lint...

Error: Argument for '--lib' option must be: 'es5', 'es6', 'es2015', 'es7', 'es2016', 'es2017', 'es2018', 'es2019', 'es2020', 'es2021', 'es2022', 'esnext', 'dom', 'dom.iterable', 'webworker', 'webworker.importscripts', 'webworker.iterable', 'scripthost', 'es2015.core', 'es2015.collection', 'es2015.generator', 'es2015.iterable', 'es2015.promise', 'es2015.proxy', 'es2015.reflect', 'es2015.symbol', 'es2015.symbol.wellknown', 'es2016.array.include', 'es2017.object', 'es2017.sharedmemory', 'es2017.string', 'es2017.intl', 'es2017.typedarrays', 'es2018.asyncgenerator', 'es2018.asynciterable', 'es2018.intl', 'es2018.promise', 'es2018.regexp', 'es2019.array', 'es2019.object', 'es2019.string', 'es2019.symbol', 'es2020.bigint', 'es2020.date', 'es2020.promise', 'es2020.sharedmemory', 'es2020.string', 'es2020.symbol.wellknown', 'es2020.intl', 'es2020.number', 'es2021.promise', 'es2021.string', 'es2021.weakref', 'es2021.intl', 'es2022.array', 'es2022.error', 'es2022.intl', 'es2022.object', 'es2022.sharedmemory', 'es2022.string', 'esnext.array', 'esnext.symbol', 'esnext.asynciterable', 'esnext.intl', 'esnext.bigint', 'esnext.string', 'esnext.promise', 'esnext.weakref'.

Is there some secret setting to allow ES2023 with forge? If i switch to ES2022, it does seem to deploy fine…

Also, i noticed that the swagger.v3 json doc for Assets (downloaded from the Assets API page) has bugs. Specifically, i am attempting to use the /objectschema/{id}/objecttypes endpoint and it says the 200 response for application/json type is:

                                "schema": {
                                    "type": "object",
                                    "properties": {
                                        "entries": {
                                            "type": "array",
                                            "items": {
                                                "$ref": "#/components/schemas/ObjectType"
                                            }
                                        }
                                    }
                                }

But the response i get back (and the example) is just an array, not an object with an entries property that is an array. This causes the code generation to be incompatible with the response. Is there a standard way to report bugs like this? is there a fair chance they get resolved or should i use a hand modified version of that json to generate that client:

#!/bin/bash

### Generates assets-api.d.ts from the official json schema.
###
### Usage:
###   src/lib/gen_assets_api.sh
###
### Remarks:
###   Assumes openapi-typescript is previously installed:
###     npm install openapi-typescript --global


set -e

# if you navigate to the API documentation page:
#   https://developer.atlassian.com/cloud/assets/rest/api-group-importsource
# this is the download link for OpenAPI
readonly API_VERSION=v3
readonly VERSION=1.135.74
readonly JSON_SCHEMA="https://dac-static.atlassian.com/cloud/assets/swagger.${API_VERSION}.json?_v=${VERSION}"

readonly ROOT_DIR="$(dirname "$(readlink --canonicalize-existing "$0")")"

readonly file="${ROOT_DIR}/assets-api.swagger.${API_VERSION}.${VERSION}-patched.json"
curl --location --silent "${JSON_SCHEMA}" \
  | jq '.paths."/objectschema/{id}/objecttypes".get.responses."200".content."application/json".schema = {
      "type": "array",
      "items": { "$ref": "#/components/schemas/ObjectType" }
    }' \
  > "${file}"
openapi-typescript "${file}" --output "${ROOT_DIR}/assets-api.d.ts"
4

No. I think I was exploring TS6 and did not fully test the reversion to TS5.9. Sorry I missed that.

I’m working on reproducing the issue and I’ll get this one filed (and fixed, if I can). Normally, the path is through:

5

ECO-1474

Working internally to decide whether the eng team wants to fix the API itself, or I will fix the OpenAPI spec.

6

@LucasTheisen,

My PR for changing the OpenAPI spec was just approved and merged over the weekend (relative to my time). Hopefully that prevents any intermediate patching. Thanks for your extended patience and sustained feedback.

8

@ibuchanan , i was able to see that update, but have since run into more issues with the schema (or at least the missing parts of the schema). I have actually been starting to patch the schema during ts code generation with json patch the best i can, but for some, i cannot see the possible values so i’m not sure what to expect. Specifically with the importsource/{importsourceid}/executions/status endpoint, the example bears no resemblance to the response i get in practice:

{
  status: 'CANCELLED',
  executionId: '4617b230-f2b4-4eeb-8d61-6162b5b0c93c'
}

or

{
  status: 'INGESTING',
  executionId: 'cb540465-b395-44aa-b713-f75d77158285'
}

But also, this is clearly an enum of values, but its unclear what the complete set would be. your project has something, but it doesn’t seem to be an authoritative source, and its not referred to by anything so i am not sure if it is indeed intended to be part of the status response (paired with executionId and possible other values?).

9

@LucasTheisen,

Your links seem to link to a private repo. Would you mind publishing?

Yes, those enums are currently a “known issue” with regard to the docs. Working on it even if slowly.

10

@ibuchanan , sorry, forgot it was private (i usually dont publish my learning projects), but it has come far enough as to be useful even if for just example code. So i just made it public, go ahead and take a look now.