Using alternate version of TypeScript

Hi Forge team,

Is there any supported mechanism planned for using an alternate version of TypeScript? I have an existing codebase running on TS 4.4 that I am trying to get to build on Forge, but it is…not easy. I was able to hack it into a working state, but I wouldn’t want to ship anything with this amount of duct tape.

This creates a blocker for anyone who has a project built with a newer version of TS than yours, but who cannot (or doesn’t want to) backport the entire project syntax to an earlier version of TS.

I was able to hack this by replacing the TypeScript versions used in the package.json for @forge/{bundler,cli,cli-shared,tunnel}, referencing those directories as local packages from my own project’s package.json, and then running “./node_modules/.bin/forge tunnel” instead of using the globally-installed “forge” command. This was not entirely straightforward because the source for these @forge/* packages does not seem to be published anywhere (could this be done?).

Ask #1: What about the idea of moving to (say) peer dependencies of typescript (and @typescript-eslint) in all @forge packages, or some other mechanism of your choice, so that we can bring whatever version of typescript we want?

After doing this, I thought I had everything working, but then I found that “forge tunnel” didn’t work, since it turns out that “tunnel” actually compiles in the container (rather than on the host) and it has its own version of typescript baked into the Docker image.

I was finally able to get this to work after I discovered the “dev mode” for Forge Tunnel, which allows you to replace the /tunnel/node_modules directory in the container by invoking it like this:

# Copy the existing image's public tag and name it according to what the dev mode expects
docker tag atlassian/forge-tunnel:latest local/forge-tunnel:test

# Run the old version of the tunnel for reference, grab its node_modules, and stop the tunnel
forge tunnel &
mkdir -p ../forge-tunnel/hardcopied_node_modules
docker cp forge-tunnel-docker:/tunnel/node_modules ../forge-tunnel/hardcopied_node_modules
kill %1

# Now replace ../forge-tunnel/hardcopied_node_modules/node_modules/typescript
# with the version you need

# Start the tunnel in dev mode, which will mount ../forge-tunnel/hardcopied_node_modules/node_modules into /tunnel/node_modules
FORGE_DEV_DOCKER_TUNNEL=true ./node_modules/.bin/forge tunnel

Ask #2: The above method for “forge tunnel” is particularly hacktastic. Can you publish the build source for the Docker image somewhere? Even better, the perfect solution would allow us to provide (say) some sort of command-line parameter to “forge tunnel” that could be passed as a series of packages and versions to “npm install” in the container, which would get run the first time the container is started? And/or allow us to specify our own custom startup script that is run before starting the tunnel?

Ask #3: While on the subject of porting real apps to Forge, we need the ability to provide an explicit path/filename for tsconfig.json, as well as an option to override or append arbitrary properties into the final webpack configuration (perhaps let us supply a config transformer?). For example, our project uses aliases for module resolution (eg. ‘import “Foo/test”’ instead of ‘import “…/…/…/services/foo/test”’) and this doesn’t seem to work at all with your hardcoded webpack config.

Ask #4: Could there be a way to specify full paths for the function handler entrypoints in the manifest.yml? It looks like we are forced to dump every single entrypoint into the top-level “src” directory. This is presumably fine for simple apps, but less so for complex projects that are built for multiple environments (of which Forge is just one).

Thanks!
Scott

5 Likes

Hi @scott.dudley,

Thank you for your detailed query, we’re sorry for not getting back to you earlier.

I hope this isn’t oversimplifying things, but a few of us in the team were thinking about your very valid use cases, and were wondering for Asks 1-3 whether you would be able to use your own build process (i.e. compile and bundle your own TypeScript files into a plain JS file) and then point the manifest handler at the compiled/bundled JS file. This way, you would be able to use all your own custom TypeScript and Webpack configuration/versions, and the Forge CLI would not need to interfere with that (since all it sees is a JS file).

For Ask 4, we’ve just tested this, and it looks like specifying full paths (e.g. handler: path/to/dir/index.run for a file in src/path/to/dir/index.js) in the manifest file actually does work as you might expect. There is currently a linting rule that does not account for this possibility and will fail the deployment with an error message. We will fix this soon, thank you for raising this with us. In the meantime, you can deploy using the --no-verify flag (e.g. forge deploy --no-verify ) in order to bypass the linting.

Hi @kchan,

Thanks for the feedback!

Doing a separate TS/webpack build is certainly something that crossed our minds, but this seems error-prone. To start with, it looks like we’d have to replicate all of the logic in @forge/bundler (which is also undocumented). There are some very specific configurations in your webpack config, and you have a bunch of polyfills and other special things that are done with module resolution, some of which presumably change over time.

While the solution I described in the first post is full of duct tape, it is not clear if this approach would require any less duct tape given the above. Plus, we then end up having to run both a “tsc --watch” on top of a “forge watch” process to make it all work together.

Regardless, if this configuration were supported by Atlassian (ie. you implement this in one of your sample Forge boilerplate apps and you keep it up-to-date), I suppose it could work.

From the outside, I assume the problem is complicated primarily because the bundling is done within the Docker container, and synchronizing the TS versions and build config between the host and the container is not entirely trivial.

Given your suggestion that we bundle our TS into a single .js file from the host, is there any reason why Atlassian cannot adopt that model and do its bundling on the host too? It seems like that would reduce the complexity of the problem.

Alternatively, what about a commitment to upgrading to the most recent TypeScript version (say) every quarter so that you are never tracking that far behind?

Scott

2 Likes

It looks like we’d have to replicate all of the logic in @forge/bundler

What I’m suggesting isn’t replacing the Forge bundler, but rather adding another layer on top, with your custom webpack and TS configurations which will end up bundling your code into something the Forge bundler can understand. This shouldn’t need to replicate the Forge bundler at all, although your custom bundler must bundle the code into something the Forge bundler understands (which as far as I’m aware is pretty standard and shouldn’t be an issue).

Plus, we then end up having to run both a “tsc --watch” on top of a “forge watch” process to make it all work together.

This is true. We will look into ways to make the existing bundler more flexible / allow you to specify your own config/versions. In fact, we already have an existing ticket for this. However, adding your own separate bundling on top should be an adequate workaround in the meantime.

From the outside, I assume the problem is complicated primarily because the bundling is done within the Docker container, and synchronizing the TS versions and build config between the host and the container is not entirely trivial. Given your suggestion that we bundle our TS into a single .js file from the host, is there any reason why Atlassian cannot adopt that model and do its bundling on the host too? It seems like that would reduce the complexity of the problem.

I’m not quite sure I understand. The bundler watches your code from your file system, so when your custom compilation/bundling process updates the JS file that the manifest file points to, the bundler (regardless of where its running) will see those changes, run its bundling process on the new code and then the tunnel will point to the new (twice bundled) code.

Hi @kchan,

Thanks for the feedback. Is this something that Atlassian would be willing to provide somewhere as a sample/supported pattern? Even if we don’t need to replicate all of the @forge/bundler logic, it looks like we still need to replicate some of it.

For example, @forge/bundler overrides 20+ module imports with custom components. To implement our own bundler, we’d need to: (a) keep track of this presumably-shifting list, and (b) for each import above, modify our webpack config to spit out an additional layer of corresponding, unresolved "import"s or "require"s that will then be processed by your webpack instead of ours. I did not investigate in depth, but perhaps there are additional dependencies we’d need to consider.

While all of this is possible, it is a bunch of legwork to work around the problem of “you don’t support newer TS versions”. I also do not have high confidence that the situation with sourcemaps would end well.

I guess, in short, while it is a solution, it seems only slightly less fragile than my tests (that of patching your typescript versions in various modules). It has downsides (presumably, the sourcemaps will be all wrong), and it too will break over time (as the module list shifts).

If Atlassian could provide a public repo with a template for this pattern, that you update whenever a required due to changes in the Forge bundler, and if you figure out how to rewrite the sourcemaps, that would be useful. But maybe it’s a harder problem than just making the TypeScript versions configurable?

From the outside, I assume the problem is complicated primarily because the bundling is done within the Docker container, and synchronizing the TS versions and build config between the host and the container is not entirely trivial. Given your suggestion that we bundle our TS into a single .js file from the host, is there any reason why Atlassian cannot adopt that model and do its bundling on the host too? It seems like that would reduce the complexity of the problem.

I’m not quite sure I understand. The bundler watches your code from your file system, so when your custom compilation/bundling process updates the JS file that the manifest file points to, the bundler (regardless of where its running) will see those changes, run its bundling process on the new code and then the tunnel will point to the new (twice bundled) code.

The gist of the question is: Atlassian is running the final bundler inside the Docker container. Is there any reason why you (or we) cannot do this final bundling on the host?

If out-of-the-box configurable TS versions are not possible, at least if you were to make a public repo somewhere containing the @forge/bundler sources (with some semblance of a documented API), we could invoke or patch it directly, while still being able to pull in your latest changes when needed. We could then just run this and dump a single forge.js into the output directory as you suggested, without having to worry about exactly what is or is not imported.

If this method is possible, it would appear to be more manageable long-term than having to worry about the knockout list of resolved imports and/or deal with sourcemap confusion.

Scott

1 Like

Your points are very valid. My suggestions are definitely workarounds and not long term solutions. You’ve pointed out some valid flaws/complications. I’ve chatted with the team and we are planning to look into making webpack config and TS versions configurable, but I can’t provide any timelines yet.

The gist of the question is: Atlassian is running the final bundler inside the Docker container. Is there any reason why you (or we) cannot do this final bundling on the host?

I checked with the team, and we are also planning to move the bundling to outside the Docker container at some point. It is in our backlog.

I can try ask the team to prioritise these if this issue is blocking your app’s development?

3 Likes

Hi @kchan,

Yes, it would be great if this could be prioritized. Thanks!

Scott

1 Like