[URGENT] It seems NGROK no longer allows anonymous access - forge tunnel is broken

Forge
,
1

I have just tried to tunnel into one of my Forge apps when I was confronted with the following:

Output of forge tunnel --verbose:

t=2024-01-02T19:47:01+0000 lvl=info msg="open config file" path=/tunnel/node_modules/@forge/tunnel/config/ngrok.yml err=nil
t=2024-01-02T19:47:01+0000 lvl=dbug msg="ignoring manifest file with unhandled mimetype" name=assets/glyphicons-halflings-regular.ttf ext=.ttf
t=2024-01-02T19:47:01+0000 lvl=dbug msg="ignoring manifest file with unhandled mimetype" name=assets/glyphicons-halflings-regular.eot ext=.eot
t=2024-01-02T19:47:01+0000 lvl=dbug msg="starting component" obj=app name="Tunnel session"
t=2024-01-02T19:47:01+0000 lvl=dbug msg="starting component" obj=app name=updater
t=2024-01-02T19:47:01+0000 lvl=dbug msg="starting component" obj=app name=web
t=2024-01-02T19:47:01+0000 lvl=info msg="starting web service" obj=web addr=127.0.0.1:4040 allow_hosts=[]
t=2024-01-02T19:47:01+0000 lvl=dbug msg="starting component" obj=app name="signal handler"
t=2024-01-02T19:47:01+0000 lvl=dbug msg="checking for updates periodically" obj=updater interval=6h0m0s
t=2024-01-02T19:47:01+0000 lvl=dbug msg="check for update" obj=updater
t=2024-01-02T19:47:01+0000 lvl=dbug msg="connecting to" obj=tunnels.session addr=connect.us.ngrok-agent.com:443
t=2024-01-02T19:47:01+0000 lvl=dbug msg=dial obj=tunnels.session network=tcp address=connect.us.ngrok-agent.com:443 timeout=10s
t=2024-01-02T19:47:01+0000 lvl=info msg=start pg=/api/tunnels id=d39d07071394a5c0
t=2024-01-02T19:47:01+0000 lvl=warn msg="ngrok is not yet ready to start tunnels" pg=/api/tunnels id=d39d07071394a5c0 err="a successful ngrok tunnel session has not yet been established"
t=2024-01-02T19:47:01+0000 lvl=info msg=end pg=/api/tunnels id=d39d07071394a5c0 status=503 dur=6.752958ms
t=2024-01-02T19:47:01+0000 lvl=info msg=start pg=/api/tunnels id=e813875414073d71
t=2024-01-02T19:47:01+0000 lvl=warn msg="ngrok is not yet ready to start tunnels" pg=/api/tunnels id=e813875414073d71 err="a successful ngrok tunnel session has not yet been established"
t=2024-01-02T19:47:01+0000 lvl=info msg=end pg=/api/tunnels id=e813875414073d71 status=503 dur=1.071958ms
t=2024-01-02T19:47:01+0000 lvl=dbug msg="open stream" obj=tunnels.session obj=csess id=b8a77bf1e2f0 reqtype=0 err=nil
t=2024-01-02T19:47:01+0000 lvl=dbug msg="encode request" obj=tunnels.session obj=csess id=b8a77bf1e2f0 sid=5 req="&{Version:[3 2] ClientID: Extra:{OS:linux Arch:amd64 Authtoken:HIDDEN Version:3.5.0 Hostname: UserAgent:ngrok-agent-go/3.5.0 ({\"proxy_type\":\"none\",\"config_version\":\"2\"}) ngrok-go/1.7.0 Metadata: Cookie: HeartbeatInterval:10000000000 HeartbeatTolerance:15000000000 Fingerprint:<nil> UpdateUnsupportedError:0xc000582330 StopUnsupportedError:0xc000582330 RestartUnsupportedError:0xc000582330 ProxyType: MutualTLS:false ServiceRun:false ConfigVersion: CustomInterface:false CustomCAs:false ClientType:ngrok-agent-go}}" err=nil
t=2024-01-02T19:47:01+0000 lvl=dbug msg="decoded response" obj=tunnels.session obj=csess id=b8a77bf1e2f0 sid=5 resp="&{Version: ClientID: Error:Usage of ngrok requires a verified account and authtoken.\n\nSign up for an account: https://dashboard.ngrok.com/signup\nInstall your authtoken: https://dashboard.ngrok.com/get-started/your-authtoken\r\n\r\nERR_NGROK_4018\r\n Extra:{Version: Region: Cookie: AccountName: SessionDuration:0 PlanName: Banner: DeprecationWarning:<nil>}}" err=nil
t=2024-01-02T19:47:01+0000 lvl=eror msg="failed to reconnect session" obj=tunnels.session obj=csess id=672bb6d4d42e err="authentication failed: Usage of ngrok requires a verified account and authtoken.\n\nSign up for an account: https://dashboard.ngrok.com/signup\nInstall your authtoken: https://dashboard.ngrok.com/get-started/your-authtoken\r\n\r\nERR_NGROK_4018\r\n"
t=2024-01-02T19:47:01+0000 lvl=dbug msg=disconnect obj=tunnels.session sess=&{raw:0xc0005aa140} err="authentication failed: Usage of ngrok requires a verified account and authtoken.\n\nSign up for an account: https://dashboard.ngrok.com/signup\nInstall your authtoken: https://dashboard.ngrok.com/get-started/your-authtoken\r\n\r\nERR_NGROK_4018\r\n"
t=2024-01-02T19:47:01+0000 lvl=dbug msg="sleep before reconnect" obj=tunnels.session obj=csess id=672bb6d4d42e secs=0
t=2024-01-02T19:47:01+0000 lvl=eror msg="session closing" obj=tunnels.session err="authentication failed: Usage of ngrok requires a verified account and authtoken.\n\nSign up for an account: https://dashboard.ngrok.com/signup\nInstall your authtoken: https://dashboard.ngrok.com/get-started/your-authtoken\r\n\r\nERR_NGROK_4018\r\n"
t=2024-01-02T19:47:01+0000 lvl=dbug msg="component stopped" obj=app name="Tunnel session" err="authentication failed: Usage of ngrok requires a verified account and authtoken.\n\nSign up for an account: https://dashboard.ngrok.com/signup\nInstall your authtoken: https://dashboard.ngrok.com/get-started/your-authtoken\r\n\r\nERR_NGROK_4018\r\n"
t=2024-01-02T19:47:01+0000 lvl=info msg="received stop request" obj=app stopReq="{err:{Remote:true Inner:{Inner:0xc000520310}} restart:false}"
t=2024-01-02T19:47:01+0000 lvl=eror msg="terminating with error" obj=app err="authentication failed: Usage of ngrok requires a verified account and authtoken.\n\nSign up for an account: https://dashboard.ngrok.com/signup\nInstall your authtoken: https://dashboard.ngrok.com/get-started/your-authtoken\r\n\r\nERR_NGROK_4018\r\n"
t=2024-01-02T19:47:01+0000 lvl=info msg=start pg=/api/tunnels id=024f41ebbec73408
t=2024-01-02T19:47:01+0000 lvl=dbug msg="waiting for all components to stop" obj=app
t=2024-01-02T19:47:01+0000 lvl=dbug msg="waiting for components to stop" obj=app remaining=3
t=2024-01-02T19:47:01+0000 lvl=dbug msg="waiting for components to stop" obj=app remaining=2
t=2024-01-02T19:47:01+0000 lvl=dbug msg="waiting for components to stop" obj=app remaining=1
t=2024-01-02T19:47:01+0000 lvl=dbug msg="component stopped" obj=app name=updater err=nil
t=2024-01-02T19:47:01+0000 lvl=dbug msg="component stopped" obj=app name="signal handler" err=nil
t=2024-01-02T19:47:01+0000 lvl=dbug msg="all components stopped" obj=app
t=2024-01-02T19:47:01+0000 lvl=dbug msg="component stopped" obj=app name=web err=nil
t=2024-01-02T19:47:01+0000 lvl=crit msg="command failed" err="authentication failed: Usage of ngrok requires a verified account and authtoken.\n\nSign up for an account: https://dashboard.ngrok.com/signup\nInstall your authtoken: https://dashboard.ngrok.com/get-started/your-authtoken\r\n\r\nERR_NGROK_4018\r\n"
t=2024-01-02T19:47:01+0000 lvl=dbug msg=disconnect obj=tunnels.session sess=&{raw:0xc0005aa140} err="context canceled"
t=2024-01-02T19:47:01+0000 lvl=info msg="no more state changes" obj=tunnels.session
t=2024-01-02T19:47:01+0000 lvl=dbug msg=disconnect obj=tunnels.session sess=&{raw:0xc0005aa140} err=nil
t=2024-01-02T19:47:01+0000 lvl=warn msg="failed to check for update" obj=updater err="Post \"https://update.equinox.io/check\": context canceled"
t=2024-01-02T19:47:01+0000 lvl=dbug msg="tunnel session closing permanently" obj=tunnels.session err="context canceled"
t=2024-01-02T19:47:01+0000 lvl=dbug msg="open stream" obj=tunnels.session obj=csess id=b8a77bf1e2f0 reqtype=1 err="remote gone away"
NAME:
t=2024-01-02T19:47:01+0000 lvl=dbug msg="start tunnel listen" obj=tunnels.session name=f810eef1-7081-4454-93d1-0234fafcdfee proto=https opts="&{Domain: Hostname: Subdomain: Auth: HostHeaderRewrite:false LocalURLScheme:http ProxyProto:0 VerifyUpstreamTLS:false UpstreamCAPEM:[] Compression:<nil> CircuitBreaker:<nil> IPRestriction:<nil> BasicAuth:<nil> OAuth:<nil> OIDC:<nil> WebhookVerification:<nil> MutualTLSCA:<nil> RequestHeaders:<nil> ResponseHeaders:<nil> WebsocketTCPConverter:<nil> UserAgentFilter:<nil> Policies:<nil>}" labels=map[] err="failed to start tunnel: remote gone away"
t=2024-01-02T19:47:01+0000 lvl=warn msg="failed to start tunnel" pg=/api/tunnels id=024f41ebbec73408 err="failed to start tunnel: remote gone away"
t=2024-01-02T19:47:01+0000 lvl=info msg=end pg=/api/tunnels id=024f41ebbec73408 status=502 dur=18.68725ms
failed to start tunnel
▶️  GraphQL https://api.atlassian.com/graphql
Query: 
      mutation forge_cli_deleteApplicationTunnels($input: DeleteAppTunnelInput!) {
        deleteAppTunnels(input: $input) {
          success
          errors {
            message
            extensions {
              errorType
              statusCode
            }
          }
        }
      }
    
Variables: {
  "input": {
    "appId": "ari:cloud:ecosystem::app/__APPID__",
    "environmentKey": "default"
  }
}
start - start tunnels by name from the configuration file

USAGE:
  ngrok start [flags]

DESCRIPTION: 
  Starts tunnels by name from the configuration file. You may specify any
  number of tunnel names. You may start all tunnels in the configuration
  file with the --all switch.

TERMS OF SERVICE: https://ngrok.com/tos

EXAMPLES: 
  ngrok start dev        # start tunnel named 'dev' in the configuration file
  ngrok start web blog   # start tunnels named 'web' and 'blog'
  ngrok start --all      # start all tunnels defined in the config file

OPTIONS:
      --all                 start all tunnels in the configuration file
      --authtoken string    ngrok.com authtoken identifying a user
      --config strings      path to config files; they are merged if multiple
  -h, --help                help for start
      --log string          path to log file, 'stdout', 'stderr' or 'false' (default "false")
      --log-format string   log record format: 'term', 'logfmt', 'json' (default "term")
      --log-level string    logging level: 'debug', 'info', 'warn', 'error', 'crit' (default "info")
      --none                start running no tunnels
◀️  GraphQL
Request ID: a3bbea6574604a1b82c80f4312abfe9a
Result: {
  "deleteAppTunnels": {
    "success": true,
    "errors": null
  }
}
{"__tunnel_error__":true,"name":"NgrokError","attributes":{}}
Error: Failed to start tunnel, could not establish a connection.

Error: Failed to start tunnel, could not establish a connection.
    at NgrokCreateTunnelService.establishTunnel (/tunnel/node_modules/@forge/tunnel/out/services/create-tunnel-service.js:39:19)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
    at async StartTunnelCommand.startFaaSTunnelServer (/tunnel/node_modules/@forge/tunnel/out/command/start-tunnel-command.js:38:35)
    at async StartTunnelCommand.execute (/tunnel/node_modules/@forge/tunnel/out/command/start-tunnel-command.js:71:38)
    at async runTunnel (/tunnel/node_modules/@forge/sandbox-tunnel/out/index.js:34:24)

It seems to me that the basic design of forge tunnel using an anonymous NGROK tunnel no longer works, which will require a redesign of the Forge CLI to either prompt for an NGROK account or use another ingress provider. Or to make a special agreement with NGROK (they are Atlassian customers anyway :slight_smile: )

I will test whether I can make it work by providing custom NGROK configuration to the Forge CLI and post my updates here.

2

See https://developer.atlassian.com/platform/forge/changelog/#CHANGE-1352

2 Likes
3

Thank you! Since this is a breaking change and since it was already announced in December, I wonder why the latest version of Forge CLI still neither handles ngrok authentication errors in a user-friendly way, nor fails if no ngrok credentials are provided…

1 Like
4

Plus: it would be great to be able to save the NGROK configuration into the environment, similarly to how forge login does so with the access token, so that you could keep using forge tunnel globally. Of course, you can alias anything, but it would be great to keep the streamlined developer experience here.

5

Hi, PM from ngrok here. Unfortunately, we needed to remove anonymous access to our service as a mechanism to combat abuse of the platform. We’d love to work with the team at Forge to design the best customer experience going forward and to use the latest open source ngrok agent SDKs if they are not already doing so. Feel free to reach out to russ at ngrok dot com and I can connect you to the engineering team. You can also open any issues or questions you have in the appropriate agent sdk repo.

Also, please note that we plan to block older ngrok agents from connecting to our service beginning in the middle of January. If you are using an ngrok agent prior to 3.2, they will no longer connect to the ngrok service. This does not impact anyone using the native agent SDK libraries.

6 Likes
6

Thanks for trying to help @RussSavage .

The problem for many developers seems to be that a third party (ngrok) is needed for developing with Atlassian forge.
Ngrok has separate licensing, and also opens a port from your computer to the internet. This is sometimes a compliance problem and has security implications.

7

Thanks @marc! Sorry if I wasn’t clear, I work for ngrok (that third party you speak of :grinning:). Just wanted to offer any design help for improving this integration if needed.

Also, just to avoid confusion, ngrok doesn’t “open a port from your computer to the internet” in the traditional sense. The ngrok agent opens a secure, outbound persistent TLS connection to the ngrok service, which handles sending traffic from the public internet to your local service.

2 Likes
8

Hi all, I’m the Engineering Manager for the team owning the Forge CLI. Apologies for the issue this has caused you and your teams.

Thanks for your patience on this, current guidance is linked in the change log here.
We are working on some improvements that will trigger developers to setup and use an Ngrok account. This will prevent similar issues in the future. The plan is to release this next week.

Long term, we want to make the tunnelling experience seamless, without having to register and provide your credentials separately, which will remove the current point of friction you’re facing.
We are currently exploring options to make this a reality.

3 Likes
9

Hi @Benny,
These are great plans!
Can you consider allowing developers to specify their own https endpoint? I’m asking because we have an internet accessible proxy host, where we would like to route all tunnels through.
We do the same with connect, where we specify the proxy host in our descriptor. It would be great if that worked in Forge, too.

10

Hi @marc,

Thanks for providing that feedback, I have created a Forge ticket for that feature request and have passed it on to the Engineers exploring options to take it into consideration.

Ticket created: [FRGE-1335] - Ecosystem Jira

1 Like