mw stack deploy does not support JSON arrays in .env file

[AdFisch]

Describe the bug A clear and concise description of what the bug is.

When using JSON arrays in an .env file, the Mittwald parser seems to remove all quotes and thus breaks them. Various workarounds like escaping the quotes (") or placing the array in a single quoted string ('"') did not work.

Detailed example:

When configurating n8n, NODES_EXCLUDE should be used to remove potentially harmful nodes that are not required in the instance. I use a docker-compose.yml for my n8n stack across instances and a prod.env file to customize each instance (e. g. different secrets). Thus, NODES_EXCLUDE must be set in prod.env and contain a valid JSON array with the nodes to be excluded:

docker-compose.yml:

` n8n-main:
image: n8nio/n8n:stable
restart: unless-stopped
depends_on:
postgresql

    environment:
        NODES_EXCLUDE: "${N8N_NODES_EXCLUDE}" # also tried without quotes here
    
    ...`

prod.env:

N8N_NODES_EXCLUDE=["n8n-nodes-base.executeCommand","n8n-nodes-base.localFileTrigger","n8n-nodes-base.readBinaryFile","n8n-nodes-base.readBinaryFiles","n8n-nodes-base.writeBinaryFile","n8n-nodes-base.readWriteFile","n8n-nodes-base.moveBinaryData","n8n-nodes-base.e2eTest"]
(also tried with escaped quotes and single-enquoted strings)

The Mittwald .env parser seems to remove all quotes resulting in the following ENV value in mStudio (which is no valid JSON and, thus, does not work:

NODES_EXCLUDE=[n8n-nodes-base.executeCommand,n8n-nodes-base.localFileTrigger,n8n-nodes-base.readBinaryFile,n8n-nodes-base.readBinaryFiles,n8n-nodes-base.writeBinaryFile,n8n-nodes-base.readWriteFile,n8n-nodes-base.moveBinaryData,n8n-nodes-base.e2eTest]

To Reproduce Steps to reproduce the behavior:

1. Use a JSON array in your .env file, e. g. the one above.
2. Reference the variable from the .env file in your docker-compose file, e. g. like shown above
3. Deploy the stack using "mw stack deploy --env-file=prod.env"
4. Check resulting .env value in mStudio.

Expected behavior A clear and concise description of what you expected to
happen.

There should be some way to use quotes in .env file (and thus making JSON arrays available), e. g. when escaping them etc.

Environment (please complete the following information):

• OS: WSL Ubuntu
• Shell: WSL bash in Windows PowerShell
• Version (output of mw --version): @mittwald/cli/1.12.0 wsl-x64 node-v20.19.6

[gandie]

Hey @AdFisch ! Thanks for the bug report! I am currently working on #1590 , which came up via #1574 .

The behavior you describe might be rooted in the same area.

Maybe i can spin up a testcase for this, which would help determine where exactly our quotes are stripped. Will keep you posted! 👍

[AdFisch]

Hi @gandie,

thanks for your reply. From my testing, the quotes are probably stripped during the .env parsing. I assume this because the quotes are not stripped when putting the JSON array directly in the docker-compose.yml like this:

NODES_EXCLUDE: "[\"n8n-nodes-base.executeCommand\",\"n8n-nodes-base.localFileTrigger\",\"n8n-nodes-base.readBinaryFile\",\"n8n-nodes-base.readBinaryFiles\",\"n8n-nodes-base.writeBinaryFile\",\"n8n-nodes-base.readWriteFile\",\"n8n-nodes-base.moveBinaryData\",\"n8n-nodes-base.e2eTest\"]"

As written in the initial post, the same string (even with escaped quotes) does not work in an .env file.

However, for my use case putting the JSON array inside the .env file would be much better because I intend to use the same docker-compose file across multiple projects and individualize them using .env files.

If you can find a solution it would be highly appreciated. Thank you.

[gandie]

[...]. From my testing, the quotes are probably stripped during the .env parsing. I assume this because the quotes are not stripped when putting the JSON array directly in the docker-compose.yml like this:

Well this is odd. In the meantime i've created a test to isolate the faulty behavior in environment processing. Strangely, i was not able to reproduce this. This could mean two things:

• Exactly the one place i've mocked in test is where the bug lives. There is a high chance this is what happened. I feel lucky today 😆
• The bug is gone already, fixed it during rework of environment handling for other issue

I can exclude the possibility of this being an API bug for now, as you've already tried with JSON list directly in compose file, which worked if i understood you correctly.

Will now dive deeper into the rabbit hole to find which one is true here. A separate test for my first finding will reveal what's going on i guess. With my current test implementation i was able to conserve proper JSON through the whole deploy process, even asserting equality on some JSON from input env, 1:1 found in outgoing stack request. Looks like:

   // simulate env file input, mimes "collectEnvironment" output
   // ...probably exactly where the bug lives ?!
    const env = {
      CUSTOM_EXCLUDES: JSON.stringify(["foo", "bar"])
    }

... which is later compared against the very same thing: JSON.stringify(["foo", "bar"]) !

So if i do the same thing just for simulated "collectEnvironment", it might reveal something interesting ... 🙏

However, for my use case putting the JSON array inside the .env file would be much better because I intend to use the same docker-compose file across multiple projects and individualize them using .env files.

Yeah, that's very understandable. cli tool should help you to achieve that, i mean that's what such env files are actually there for 😉

[AdFisch]

Just to make sure your test is aligned with my setup:

A) You have an your-file-name.env file which contains variable with a JSON array written in one of the following ways:

`N8N_NODES_EXCLUDE=["n8n-nodes-base.executeCommand"] # or:

N8N_NODES_EXCLUDE='["n8n-nodes-base.executeCommand"]' # or:

N8N_NODES_EXCLUDE="["n8n-nodes-base.executeCommand"]"`

B) You have docker-compose.yml file that references the JSON array from above in one of the following ways:

`environment:

NODES_EXCLUDE: "${N8N_NODES_EXCLUDE}" # or:

NODES_EXCLUDE: ${N8N_NODES_EXCLUDE}`

C) You called mw stack deploy --env-file "your-file-name.env".

Result:

• You could then see the value ["n8n-nodes-base.executeCommand"] (with quotes) for ENV NODES_EXCLUDE in mStudio?
• If yes, which combination did you use? Then I can give it another try as well.
• Of no (you also see string without quotes) I assume all quotes are removed while parsing .env somewhere in the CLI.

Thank you very much for your help!

[gandie]

Gotcha!

Definitively the .env file parsing at the beginning. After first isolation this one lit up directly.

My test.env file:

NODES_EXCLUDE=["node1","node2","node3"]

... running through "collectEnvironment" function in test yields:

 FAIL  src/lib/resources/stack/env.test.ts
  ● collectEnvironment › should preserve JSON array strings from env file

    expect(received).toEqual(expected) // deep equality

    - Expected  - 1
    + Received  + 1

      Object {
    -   "NODES_EXCLUDE": "[\"node1\",\"node2\",\"node3\"]",
    +   "NODES_EXCLUDE": "[node1,node2,node3]",
        "NODE_ENV": "production",
      }

      22 |
      23 |     // Assertions
    > 24 |     expect(result).toEqual({
         |                    ^
      25 |       NODE_ENV: "production",
      26 |       NODES_EXCLUDE: '["node1","node2","node3"]', // JSON array must remain a string
      27 |     });

      at Object.<anonymous> (src/lib/resources/stack/env.test.ts:24:20)

This is exactly our issue and fits perfectly into our observations. Now we have isolated the rascal, which is the most important step for fixing imo.

Now comes the funny part:

Digging deeper revealed that we actually use a node module for env file parsing! Digging even deeper into its source revealed this lovely comment:

// perhaps in the future we can use @bevry/json's toJSON and parseJSON and JSON.stringify to support more advanced types

... i spare you the whole parser code following up, but it includes one line that immediately caught my eye:

const value = match[2].trim().replace(/['"]+/g, '')

Tada! Quick dryrun make sure we're not crazy:

~ $ node
Welcome to Node.js v24.12.0.
Type ".help" for more information.
> const match = ['foo', 'bar', 'ham=["eggs", "salad"]']
undefined
> const value = match[2].trim().replace(/['"]+/g, '')
undefined
> value
'ham=[eggs, salad]'  // booya!

So, my nightmare has come true, the bug lives deep down in the env file parser library 😜

Props to your intuition, your guess was 100% correct! As i am quite new to the project, it might take me some time to actually fix this. Maybe this will even lead to more drastic architectural changes, e.g. switching env file parser library entirely. Maybe i have to fork it 😆

Thanks a lot for this bug report again, this was a fun riddle, i've learnt a lot so far. Will keep you posted regarding the fix!