Can't post multipart form data file in new NodeJS runtime

GyorgyVARGA · February 18, 2024, 11:01pm

Hello!

I’ve got a custom UI Jira plugin with the following manifest.yml app config:

app:
  runtime:
    name: nodejs18.x

I’ve got a Forge function which handles a form with a file too.

import Resolver from '@forge/resolver';
import api, { route } from "@forge/api";
import FormData from 'form-data';

const resolver = new Resolver();

resolver.define('postOfficeData', async (req) => {
    const formData = new FormData();

    Object.keys(req.payload).forEach(key => {
        formData.append(key, req.payload[key]);
    });

    const response = await api.fetch('https://jira-plugin-serverless.gyurmatag.workers.dev/add-office', {
        method: 'POST',
        body: formData
    });

    if (!response.ok) {
        throw new Error(`Error from Cloudflare Worker: ${response}`);
    }

    return { success: true, message: 'Office data saved successfully.' };
});

The office data that I am sending to the Forge function is like this:

{
    "officeName": "hrthrthrt",
    "officeRegion": "australia",
    "streetAddress": "cvbcvb",
    "city": "dfgdfg",
    "country": "bdf",
    "address": "cvbcvb",
    "state": "dfgdfg",
    "zipCode": "dfgd",
    "electricity": "dfg",
    "gas": "dfg",
    "water": "dfgd",
    "image": {
         "path": "Dunder M.png",
         "preview": "blob:https://1414a28f-21bf-42ca-8bf2-e4ffc6f7d168.cdn.prod.atlassian- 
                             dev.net/c6c07c00-4c2a-405d-b757-af4240489be1",
         "lastModified": 1708253638710,
         "lastModifiedDate": "Sun Feb 18 2024 11:53:58 GMT+0100 (közép-európai téli idő)",
         "name": "Dunder M.png",
         "size": 623999,
         "type": "image/png",
         "webkitRelativePath": ""
    }
}

The endpoint is working totally fine when calling from HTTPie. When I am calling the forge function from the React app it gets me this error:
Error: There was an error invoking the function - source.on is not a function
What is the problem here? I thought the new Nodejs runtime resolved it?
Can somebody please help me? Maybe @AdamMoore ?

dmorrow · February 19, 2024, 6:29am

Hi @GyorgyVARGA ,

I’m not sure I follow your explanation. Is your fetch call failing or is your resolver not being called or something else? Can you add some logs to your code and include the output to explain.

Regards,
Dugald

GyorgyVARGA · February 19, 2024, 7:22am

HI @dmorrow! Thanks for you quick reply. The endpoint in my Cloudflare worker doesn’t get called. The Forge function is failing here.
Here is more log:

Error: There was an error invoking the function - source.on is not a function
    at fe.error (https://forge.cdn.prod.atlassian-dev.net/global-bridge.js:2:153277)
    at Object.<anonymous> (https://forge.cdn.prod.atlassian-dev.net/global-bridge.js:2:161050)
    at JSON.parse (<anonymous>)
    at Te.o (https://forge.cdn.prod.atlassian-dev.net/global-bridge.js:2:160909)
    at Te (https://forge.cdn.prod.atlassian-dev.net/global-bridge.js:2:161062)
    at s.on (https://forge.cdn.prod.atlassian-dev.net/global-bridge.js:2:165722)
    at je (https://forge.cdn.prod.atlassian-dev.net/global-bridge.js:2:165859)
    at https://forge.cdn.prod.atlassian-dev.net/global-bridge.js:2:171845
    at e.try (https://forge.cdn.prod.atlassian-dev.net/global-bridge.js:2:144329)
    at https://forge.cdn.prod.atlassian-dev.net/global-bridge.js:2:171642

Can you please help me? It’s really blocking us.

JoshuaHwang · February 19, 2024, 10:22am

Where are you seeing this error? The actual logs shows in the developer console or are these logs from the browser?

If these are from the browser then we’d need to take a look at the browser code.

Building a minimal reproducible example would make debugging faster and ultimately easier to help you.

GyorgyVARGA · February 19, 2024, 11:42am

@JoshuaHwang These logs are from the browser. Here is the React code invoking the function


  const handleSave = () => {
    if (selectedOfficeIndex === null) {
      console.error('No office selected');
      return;
    }

    const currentOfficeData = offices[selectedOfficeIndex];

    if (!currentOfficeData) {
      console.error('Invalid office data');
      return;
    }

    setIsSubmitting(true);

    invoke('postOfficeData', { ...currentOfficeData })
      .then(response => {
        console.log('Forge function responded:', response.message);
        setIsSubmitting(false);
      })
      .catch(error => {
        console.error('Forge function responded:', error);
        setIsSubmitting(false);
      });
  };

The currentOfficeData is in the post text. Can you please help me with that, or should I let you access my Github repository?
Thank you very much!

GyorgyVARGA · February 20, 2024, 1:24pm

@JoshuaHwang can you please get back to me?

JoshuaHwang · February 21, 2024, 4:04am

Hi Gyorgy

I’m not involved with the frontend work. I’ll send this question to the right team and they can continue helping you from there.

GyorgyVARGA · February 21, 2024, 7:04am

Actually when I am calling my Cloudflare worker endpoint directly from the React app client, like this:

const handleSave = () => {
    if (selectedOfficeIndex === null) {
      console.error('No office selected');
      return;
    }

    const currentOfficeData = offices[selectedOfficeIndex];

    if (!currentOfficeData) {
      console.error('Invalid office data');
      return;
    }

    setIsSubmitting(true);

    const formData = new FormData();

    Object.keys(currentOfficeData).forEach(key => {
      formData.append(key, currentOfficeData[key]);
    });

    if (currentOfficeData.image) {
      formData.append('image', currentOfficeData.image, currentOfficeData.image.name);
    }

    formData.append('companyId', companyId);

    fetch('https://jira-plugin-serverless.gyurmatag.workers.dev/add-office', {
      method: 'POST',
      body: formData
    })
      .then(response => {
        if (!response.ok) {
          throw new Error(`Error from Cloudflare Worker: ${response.status}`);
        }
        return response.json();
      })
      .then(data => {
        console.log('Cloudflare worker responded:', data.message);
        setIsSubmitting(false);
      })
      .catch(error => {
        console.error('Cloudflare worker responded:', error);
        setIsSubmitting(false);
      });
  };

It is working fine. When I am invoking the Forge function that is when the error happens. So something is wrong with that.

AlexeyKotlyarov · February 22, 2024, 1:53am

I believe you are running into a problem with form-data package:

github.com/form-data/form-data

[error] can not append "object" as "Blob"

opened 04:28AM - 22 Feb 22 UTC

sylvanfrost

## environment node - 15.7.0 (first version has blob as experimental) form-dat…a - ^4.0.0 ## problem 1. I tried to append form data as Blob. 1. I need to append as real object not object as string. 1. If has other way to append form data as real object I will go that way but I don't know how. ## why My api develop by Hapi framework and validate by Joi. When api get a request Hapi will use Joi validate payload first (not my code), I need to validate some key in payload as my expected object joi exmaple ```javascript Joi.object({ "a": Joi.object({ "a": Joi.string(), "b": Joi.number().required() }); }); ``` ## remark JavaScript in html file can append blob ## code ```javascript const { Blob } = require("buffer"); const formData = require("form-data"); const objectHere = { "a": 1, "b": 2 } const objectAsBlob = new Blob([JSON.stringify(objectHere, null, 0)], { type: "application/json" }); formdata.append("data", objectAsBlob); // error at this line ``` ## error ```bash > node ./index.js Debugger attached. Waiting for the debugger to disconnect... C:\Users\username\poc\node_modules\delayed-stream\lib\delayed_stream.js:33 source.on('error', function() {}); ^ TypeError: source.on is not a function at Function.DelayedStream.create (C:\Users\username\poc\node_modules\delayed-stream\lib\delayed_stream.js:33:10) at FormData.CombinedStream.append (C:\Users\username\poc\node_modules\combined-stream\lib\combined_stream.js:45:37) at FormData.append (C:\Users\username\poc\node_modules\form-data\lib\form_data.js:75:3) at C:\Users\username\poc\index.js:48:24 at Array.forEach (<anonymous>) at GenerateServerInjectionOptions (C:\Users\username\poc\index.js:44:23) at Object.<anonymous> (C:\Users\username\poc\index.js:67:14) at Module._compile (node:internal/modules/cjs/loader:1108:14) at Object.Module._extensions..js (node:internal/modules/cjs/loader:1137:10) at Module.load (node:internal/modules/cjs/loader:973:32) npm ERR! code 1 npm ERR! path C:\Users\username\poc npm ERR! command failed npm ERR! command C:\Windows\system32\cmd.exe /d /s /c node ./index.js npm ERR! A complete log of this run can be found in: npm ERR! C:\Users\username\AppData\Local\npm-cache\_logs\2022-02-22T04_16_37_431Z-debug.log Waiting for the debugger to disconnect... ``` ## in html (work) ```html <script> const xhr = new XMLHttpRequest(); const formData = new FormData(); const objectHere = { "a": 1, "b": 2 } const objectAsBlob = new Blob([JSON.stringify(objectHere, null, 0)], { type: "application/json" }); formData.append("data", objectAsBlob); xhr.open("POST", "http://localhost:8080/poc"); xhr.send(formData); // can sent and receive </script> ```

Can you please verify whether the types in req.payload are what you expect them to be? I also note that you iterate every key in the object and then separately process .image which is a key itself.

One of the suggestions in the bug report is also to switch to formdata-node (although I haven’t tested it).

If you still can’t find anything wrong, please also run the code on Node.js itself (e.g. on your local machine), replacing api.fetch with node-fetch. Forge’s Node.js runtime doesn’t touch the request body so if there is a discrepancy it’d be a bug.

GyorgyVARGA · February 27, 2024, 10:25am

Hi @AlexeyKotlyarov !

I managed to make the POST request work with using node-fetch.

But I’ve got a problem.

Here is my resolver function:

import fetch from ‘node-fetch’;

resolver.define('postOfficeData', async (req) => {
    const formData = new FormData();

    console.log(req.payload.image)

    Object.keys(req.payload).forEach(key => {
        formData.append(key, req.payload[key]);
    });

    const response = await fetch('https://jira-plugin-serverless.gyurmatag.workers.dev/add-office', {
        method: 'POST',
        body: formData
    });

    if (!response.ok) {
        throw new Error(`Error from Cloudflare Worker: ${response}`);
    }

    return { success: true, message: 'Office data saved successfully.' };
});

The image console.log gets me this:

{
  path: 'Képernyőfotó 2024-02-27 - 0.04.39.png',
  preview: 'blob:http://localhost:8000/22f9349c-ddb5-4ec7-bc66-3a9bb68702c1'
}

Which is why I can recognize it as a file in my Cloudflare worker:

	const file = formData.get('image') as unknown as File;
	const fileBuffer = await file.arrayBuffer();

This fails with this:
"TypeError: file.arrayBuffer is not a function"

The image is not passed correctly to the forge function I think. Here is how I invoke the function:

console.log(currentOfficeData.image);

    invoke('postOfficeData', currentOfficeData)
      .then((response) => {
        if (!response.success) {
          throw new Error(`Error from Cloudflare Worker: ${response.message}`);
        }
        console.log('Cloudflare worker responded:', response.message);
        setIsSubmitting(false);
      })
      .catch((error) => {
        console.error('Cloudflare worker responded:', error);
        setIsSubmitting(false);
      });

The console.log gets me this in the browser which is a valid file object:

{
  path:  "Képernyőfotó 2024-02-27 - 0.04.39.png",
  preview: "blob:http://localhost:8000/6b215b0f-2eb5-4dda-8e3a-21c2dc4aa7b5",
  lastModified: 1708988684608,
  lastModifiedDate: Tue Feb 27 2024 00:04:44 GMT+0100 (közép-európai téli idő) {},
  name: "Képernyőfotó 2024-02-27 - 0.04.39.png",
  size: 168460,
  type: "image/png",
  webkitRelativePath: "",
}

Can you please help me what can be the problem here?

GyorgyVARGA · March 8, 2024, 7:35am

@AlexeyKotlyarov can you please get back to this? It is still not working.

AlexeyKotlyarov · March 12, 2024, 1:03am

Looks like the problem might be in sending the image from the browser to the Forge function. I’ve asked our UI team to take a look as well, but meanwhile:

Can you reproduce the problem with a dummy file you create in the Forge function (not coming from the UI)?
- If so, does the same code run in a normal Node.js environment?

AlvaroBezerra · March 12, 2024, 4:39am

Hey @GyorgyVARGA,

Looking at your examples, it seems like the problem is that you are trying to post binary data (a file) from the frontend to your Resolver and that will not work since only JSON payloads are supported for invoke calls.

I can see two options for you to solve this:

Recommended: Post the file to your remote endpoint directly from the frontend using the browser’s native Fetch API
If the above is not feasible (e.g: you want to call from the Resolver so that you don’t expose auth secrets to the client), you might need to parse your file as base64 strings before the invoke call, and parse it back to binary in your Resolver implementation. Note that this approach is only recommended if you are dealing with small files, otherwise, you are going start to hit payload size limits for larger files.

Hope this helps

GyorgyVARGA · March 12, 2024, 9:18am

Hello @AlvaroBezerra
The first option is that I am doing currently as a workaround. Is it safe to do so? Should I convert all of my requests that it directly call my remote API endpoint (which are on Cloduflare)? Currently I am calling the forge invoke from my plugin React static app and the forge function is calling my remote endpoint. What is considered best practice? From security standpoint I went with calling my remote endpoint indirectly with a forge function, but in this case where I post an image it is not working as discussed above.

Also in the future will you support calling invoke functions with multipart form data as well, not just JSON?

HeyJoe · March 27, 2024, 3:59am

I’ve added an example of working with the formdata-node package in a Forge app as a comment on FRGE-114: [FRGE-114] - Ecosystem Jira

In particular I noticed that the Content-type header for the multi-part body was not being set correctly by default as one would typically expect.

I found that using the form-data-encoder - npm package to set the content-type header explicitly seemed to fix the problem.

GyorgyVARGA · March 27, 2024, 8:14am

@HeyJoe Thanks. But what about calling a Forge function with multi-part form data from the UI?

EricStratton · September 24, 2024, 7:38pm

Edit: In case anyone else stumbles on this and is running into the same issue, I needed to import Readable from ‘node:stream’.

I stumbled on this thread after running into the same Error: There was an error invoking the function - source.on is not a function issue. I’m trying to generate a text file in my resolver function and then POST it as an attachment to a Jira issue. I was following the Jira REST API docs for adding attachments.

I then tried following the example provided by @HeyJoe, but encountered a problem where the Readable interface was undefined, causing a failure when attempting to call .from. Since I’m using the Node.js 20.x runtime, I expected the stream module to be available natively.

Has anyone else faced this problem? Any help would be greatly appreciated!

HeyJoe · September 27, 2024, 8:56am

Hey @EricStratton - When I tested this out, I found that I had to use the formdata-node - npm package. I couldn’t get it working using the native interfaces in Node. Did you try this approach? I vaguely recall that source.on is not a function was an error I worked around by using this alternative package.

(Disclaimer: I don’t code full-time and figuring out why the native Node interfaces weren’t working went a bit beyond my own personal JavaScript skills).