Using alternate version of TypeScript

Hi Forge team,

Is there any supported mechanism planned for using an alternate version of TypeScript? I have an existing codebase running on TS 4.4 that I am trying to get to build on Forge, but it is…not easy. I was able to hack it into a working state, but I wouldn’t want to ship anything with this amount of duct tape.

This creates a blocker for anyone who has a project built with a newer version of TS than yours, but who cannot (or doesn’t want to) backport the entire project syntax to an earlier version of TS.

I was able to hack this by replacing the TypeScript versions used in the package.json for @forge/{bundler,cli,cli-shared,tunnel}, referencing those directories as local packages from my own project’s package.json, and then running “./node_modules/.bin/forge tunnel” instead of using the globally-installed “forge” command. This was not entirely straightforward because the source for these @forge/* packages does not seem to be published anywhere (could this be done?).

Ask #1: What about the idea of moving to (say) peer dependencies of typescript (and @typescript-eslint) in all @forge packages, or some other mechanism of your choice, so that we can bring whatever version of typescript we want?

After doing this, I thought I had everything working, but then I found that “forge tunnel” didn’t work, since it turns out that “tunnel” actually compiles in the container (rather than on the host) and it has its own version of typescript baked into the Docker image.

I was finally able to get this to work after I discovered the “dev mode” for Forge Tunnel, which allows you to replace the /tunnel/node_modules directory in the container by invoking it like this:

# Copy the existing image's public tag and name it according to what the dev mode expects
docker tag atlassian/forge-tunnel:latest local/forge-tunnel:test

# Run the old version of the tunnel for reference, grab its node_modules, and stop the tunnel
forge tunnel &
mkdir -p ../forge-tunnel/hardcopied_node_modules
docker cp forge-tunnel-docker:/tunnel/node_modules ../forge-tunnel/hardcopied_node_modules
kill %1

# Now replace ../forge-tunnel/hardcopied_node_modules/node_modules/typescript
# with the version you need

# Start the tunnel in dev mode, which will mount ../forge-tunnel/hardcopied_node_modules/node_modules into /tunnel/node_modules
FORGE_DEV_DOCKER_TUNNEL=true ./node_modules/.bin/forge tunnel

Ask #2: The above method for “forge tunnel” is particularly hacktastic. Can you publish the build source for the Docker image somewhere? Even better, the perfect solution would allow us to provide (say) some sort of command-line parameter to “forge tunnel” that could be passed as a series of packages and versions to “npm install” in the container, which would get run the first time the container is started? And/or allow us to specify our own custom startup script that is run before starting the tunnel?

Ask #3: While on the subject of porting real apps to Forge, we need the ability to provide an explicit path/filename for tsconfig.json, as well as an option to override or append arbitrary properties into the final webpack configuration (perhaps let us supply a config transformer?). For example, our project uses aliases for module resolution (eg. ‘import “Foo/test”’ instead of ‘import “…/…/…/services/foo/test”’) and this doesn’t seem to work at all with your hardcoded webpack config.

Ask #4: Could there be a way to specify full paths for the function handler entrypoints in the manifest.yml? It looks like we are forced to dump every single entrypoint into the top-level “src” directory. This is presumably fine for simple apps, but less so for complex projects that are built for multiple environments (of which Forge is just one).

Thanks!
Scott

8 Likes

Hi @scott.dudley,

Thank you for your detailed query, we’re sorry for not getting back to you earlier.

I hope this isn’t oversimplifying things, but a few of us in the team were thinking about your very valid use cases, and were wondering for Asks 1-3 whether you would be able to use your own build process (i.e. compile and bundle your own TypeScript files into a plain JS file) and then point the manifest handler at the compiled/bundled JS file. This way, you would be able to use all your own custom TypeScript and Webpack configuration/versions, and the Forge CLI would not need to interfere with that (since all it sees is a JS file).

For Ask 4, we’ve just tested this, and it looks like specifying full paths (e.g. handler: path/to/dir/index.run for a file in src/path/to/dir/index.js) in the manifest file actually does work as you might expect. There is currently a linting rule that does not account for this possibility and will fail the deployment with an error message. We will fix this soon, thank you for raising this with us. In the meantime, you can deploy using the --no-verify flag (e.g. forge deploy --no-verify ) in order to bypass the linting.

1 Like

Hi @kchan,

Thanks for the feedback!

Doing a separate TS/webpack build is certainly something that crossed our minds, but this seems error-prone. To start with, it looks like we’d have to replicate all of the logic in @forge/bundler (which is also undocumented). There are some very specific configurations in your webpack config, and you have a bunch of polyfills and other special things that are done with module resolution, some of which presumably change over time.

While the solution I described in the first post is full of duct tape, it is not clear if this approach would require any less duct tape given the above. Plus, we then end up having to run both a “tsc --watch” on top of a “forge watch” process to make it all work together.

Regardless, if this configuration were supported by Atlassian (ie. you implement this in one of your sample Forge boilerplate apps and you keep it up-to-date), I suppose it could work.

From the outside, I assume the problem is complicated primarily because the bundling is done within the Docker container, and synchronizing the TS versions and build config between the host and the container is not entirely trivial.

Given your suggestion that we bundle our TS into a single .js file from the host, is there any reason why Atlassian cannot adopt that model and do its bundling on the host too? It seems like that would reduce the complexity of the problem.

Alternatively, what about a commitment to upgrading to the most recent TypeScript version (say) every quarter so that you are never tracking that far behind?

Scott

3 Likes

It looks like we’d have to replicate all of the logic in @forge/bundler

What I’m suggesting isn’t replacing the Forge bundler, but rather adding another layer on top, with your custom webpack and TS configurations which will end up bundling your code into something the Forge bundler can understand. This shouldn’t need to replicate the Forge bundler at all, although your custom bundler must bundle the code into something the Forge bundler understands (which as far as I’m aware is pretty standard and shouldn’t be an issue).

Plus, we then end up having to run both a “tsc --watch” on top of a “forge watch” process to make it all work together.

This is true. We will look into ways to make the existing bundler more flexible / allow you to specify your own config/versions. In fact, we already have an existing ticket for this. However, adding your own separate bundling on top should be an adequate workaround in the meantime.

From the outside, I assume the problem is complicated primarily because the bundling is done within the Docker container, and synchronizing the TS versions and build config between the host and the container is not entirely trivial. Given your suggestion that we bundle our TS into a single .js file from the host, is there any reason why Atlassian cannot adopt that model and do its bundling on the host too? It seems like that would reduce the complexity of the problem.

I’m not quite sure I understand. The bundler watches your code from your file system, so when your custom compilation/bundling process updates the JS file that the manifest file points to, the bundler (regardless of where its running) will see those changes, run its bundling process on the new code and then the tunnel will point to the new (twice bundled) code.

Hi @kchan,

Thanks for the feedback. Is this something that Atlassian would be willing to provide somewhere as a sample/supported pattern? Even if we don’t need to replicate all of the @forge/bundler logic, it looks like we still need to replicate some of it.

For example, @forge/bundler overrides 20+ module imports with custom components. To implement our own bundler, we’d need to: (a) keep track of this presumably-shifting list, and (b) for each import above, modify our webpack config to spit out an additional layer of corresponding, unresolved "import"s or "require"s that will then be processed by your webpack instead of ours. I did not investigate in depth, but perhaps there are additional dependencies we’d need to consider.

While all of this is possible, it is a bunch of legwork to work around the problem of “you don’t support newer TS versions”. I also do not have high confidence that the situation with sourcemaps would end well.

I guess, in short, while it is a solution, it seems only slightly less fragile than my tests (that of patching your typescript versions in various modules). It has downsides (presumably, the sourcemaps will be all wrong), and it too will break over time (as the module list shifts).

If Atlassian could provide a public repo with a template for this pattern, that you update whenever a required due to changes in the Forge bundler, and if you figure out how to rewrite the sourcemaps, that would be useful. But maybe it’s a harder problem than just making the TypeScript versions configurable?

From the outside, I assume the problem is complicated primarily because the bundling is done within the Docker container, and synchronizing the TS versions and build config between the host and the container is not entirely trivial. Given your suggestion that we bundle our TS into a single .js file from the host, is there any reason why Atlassian cannot adopt that model and do its bundling on the host too? It seems like that would reduce the complexity of the problem.

I’m not quite sure I understand. The bundler watches your code from your file system, so when your custom compilation/bundling process updates the JS file that the manifest file points to, the bundler (regardless of where its running) will see those changes, run its bundling process on the new code and then the tunnel will point to the new (twice bundled) code.

The gist of the question is: Atlassian is running the final bundler inside the Docker container. Is there any reason why you (or we) cannot do this final bundling on the host?

If out-of-the-box configurable TS versions are not possible, at least if you were to make a public repo somewhere containing the @forge/bundler sources (with some semblance of a documented API), we could invoke or patch it directly, while still being able to pull in your latest changes when needed. We could then just run this and dump a single forge.js into the output directory as you suggested, without having to worry about exactly what is or is not imported.

If this method is possible, it would appear to be more manageable long-term than having to worry about the knockout list of resolved imports and/or deal with sourcemap confusion.

Scott

2 Likes

Your points are very valid. My suggestions are definitely workarounds and not long term solutions. You’ve pointed out some valid flaws/complications. I’ve chatted with the team and we are planning to look into making webpack config and TS versions configurable, but I can’t provide any timelines yet.

The gist of the question is: Atlassian is running the final bundler inside the Docker container. Is there any reason why you (or we) cannot do this final bundling on the host?

I checked with the team, and we are also planning to move the bundling to outside the Docker container at some point. It is in our backlog.

I can try ask the team to prioritise these if this issue is blocking your app’s development?

4 Likes

Hi @kchan,

Yes, it would be great if this could be prioritized. Thanks!

Scott

1 Like

Not sure if it’s a good idea to revive this topic or not – but to work around this, I think the suggestion from the Atlassian team to just bundle your code beforehand is actually a pretty nice one.

I’m using Rollup to bundle my backend TypeScript code into CommonJS, which is then the “application code” that is deployed or tunneled to Forge. I really only care about TypeScript at compile time in my environment, I don’t care what Forge thinks, so I don’t see what the issue is with just sending them vanilla JavaScript.

Same idea is valid for any frontend code (Custom UI bundled with Vite, in my case). I have a monorepo project of sorts with a backend and frontend folder with my respective TypeScript projects, that spit out vanilla JavaScript builds to an app folder, which is where my manifest.yml lives.

Integrate Rollup’s watch functionality, and you have a pretty strong development environment setup if you run things with a package like concurrently. I think the idea of taking as much out of the Forge environment’s purview as possible is advantageous.

1 Like

Hi @BM_50,

Out of curiosity, using that approach, were you able to get sourcemaps working correctly, as well as the module import overrides? Although it seems like one could perhaps do without the former, the latter is presumably required (at least if your application uses any of those modules).

Scott

Hey @scott.dudley, are you referring to debugging with Tunnel for your backend functions/resolvers? Even when using Forge tunnel, I’m using my “bundled backend”, and it gets rebuilt whenever I make changes which of course the tunnel picks up as well. Perhaps I’m missing some of the nuance you elaborated earlier, so please let me know (I’m happy to learn).

If it helps, here’s my Rollup and tsconfig, respectively.

import typescript from '@rollup/plugin-typescript';
import { nodeResolve } from '@rollup/plugin-node-resolve';
import commonjs from '@rollup/plugin-commonjs';

export default {
  input: 'src/main.ts',
  output: {
    file: '../app/src/index.js',
    format: 'cjs'
  },
  plugins: [nodeResolve({exportConditions: ["node"]}), commonjs(),  typescript()]
};
{
    "compilerOptions": {
        "moduleResolution": "node",
        "noImplicitAny": true,
        "noPropertyAccessFromIndexSignature": false,
        "strictNullChecks": true,
    },
    "extends": "@tsconfig/node16-strictest-esm/tsconfig.json", //https://github.com/tsconfig/bases
    "include": [
        "src"
    ],
}

@kchan It’s been almost a year now - has there been any movement towards making it easier for users to use newer versions of typescript? Many npm packages are now using newer typescript features such as Labeled Tuple Elements and they are simply not usable in Forge using the built in version of typescript.

@scott.dudley are you still taking the same approach you mentioned above and do you have any further suggestions for people who take this route? I also saw @BM_50 's post from this spring and wonder which tools are most appropriate. To date I’ve just been letting Forge do all the bundling.

1 Like

@jeffryan Still taking the same approach, but it’s just a prototype that we’re not shipping anywhere yet (for this reason, among others). It requires significant patching effort any time you upgrade any of the @forge module versions and I wouldn’t want to depend on this for anything running in production.

@kchan Even if trying to use the existing built-in version of TS in Forge, the situation quickly gets untenable related to reasons that Jeff mentioned. For example, if you suddenly need to pull in a newer version of a npm package due to a security problem, but that newer version requires a TS version beyond what Forge is using, you end up between a rock and a hard place. The TS version really needs to be configurable.

2 Likes

Thanks for the update @scott.dudley

Ugh, I hadn’t even considered the security patch situation. Yes this could get ugly quickly and frankly it is scaring me away from using typescript packages. Unfortunately that is becoming the norm these days.

I’m currently trying to use a package that was written in typescript but which also includes already transpiled js code in the published package. Unfortunately Forge seems to want to re-transpile the package from the ts source and fails due to the version of typescript. I can’t seem to find a way to tell Forge to ignore the ts code and use the transpiled js. I’ve tried various include/exclude options in my tsconfig.json without success.

@kchan do you have any suggestions on how to tell forge to not transpile npm packages that already ship with js?

Furthermore, you can’t easily use your own webpack or Rollup or whatever bundler to produce code and feed that to the Forge bundler. Aside from the sourcemaps probably getting munged (admittedly just a guess on my part), the custom module thing looks like a real problem.

The @forge/bundler package automatically overwrites imports to a whole bunch of npm packages with Forge-compatible versions. If you use your own bundler, you’re not going to get the Forge-compatible version because it’s your bundler resolving the imports. I don’t know what the impact of that is, but I assume Atlassian is doing these overrides for a good reason.

Impacted modules include (at least as of the old bundler v1.0.17):

  • browserify-sign
  • randombytes
  • fs
  • http
  • https
  • path
  • querystring
  • stream
  • and a whole bunch of crypto-related stuff

Some of the overwritten module replacements are even hosted within the @forge/bundler module itself, so you’d have to figure out how to import these replacement into your own code. Also, this list of modules can presumably change at any time when you upgrade the bundler. All of this also fits into the same category of “things I wouldn’t want to use in production”.

2 Likes

The @forge/bundler package automatically overwrites imports to a whole bunch of npm packages with Forge-compatible versions. If you use your own bundler, you’re not going to get the Forge-compatible version because it’s your bundler resolving the imports. I don’t know what the impact of that is, but I assume Atlassian is doing these overrides for a good reason.

You make a good point. I have been using my self-written Nx plugin to scaffold and build Forge apps, which essentially uses webpack to compile and bundle TS code before handing it to the Forge platform. So far, this has worked without any issues, but I think you are right that this approach is not using the patched modules.

The solution to this may be to extract a webpack plugin from @forge/bundler so we can create our own Forge artifacts with our own tooling. @forge/bundler could use the same plugin to do the same it does today.

1 Like

FYI: Regarding use of already transpiled packages, I found I can avoid forges old typescript version choking on those packages by using them from a js module instead of a ts module. In that case it’ll use the transpiled js code instead of trying to use ts code. Then, my js module just re-exports functions as necessary so I can use them from other ts modules in my app. This isn’t great but it allowed me to get beyond the typescript errors.

1 Like

Upgraded Typescript - WooHoo! Forge CLI version 6.0.0 - 25th October 2022

1 Like

@jeffryan Were you able to figure this out? this is exactly my issue. I’m able to build the code just fine, but when I run forge deploy it fails on the ℹ Packaging app files step because forge deploy wants to look at the src code, even though I’m telling it via manifest.yml to only look at the build folder which only has .js files (so it really shouldn’t be looking at the src folder)

The only hack I can think of is literally deleting my src folder when I’m about to run forge deploy step and adding it back after forge deploy is done.

I described my workaround later in the thread: Using alternate version of TypeScript - #16 by jeffryan

Not great but it worked

1 Like