Invalid PhysicalResourceId when using RawOutput.STRING

[MartinAltmayerTMH]

I'm still having issues with the new plaintext secrets.

I create a SOPS file using

sops edit --input-type binary sops_test.yaml

and enter the string abc (without any JSON/YAML structure). Now a

sops decrypt --output-type binary sops_test.yaml

correctly returns abc.

However, trying to deploy the secret fails with the error message

CREATE_FAILED        | Custom::SopsSync            | Secret/SopsSync/Resource/Default (SecretSopsSyncCA4EC563) Invalid PhysicalResourceId
❌  SopsTestStack failed: _ToolkitError: The stack named SopsTestStack failed to deploy: UPDATE_ROLLBACK_COMPLETE: Invalid PhysicalResourceId

Here is my CDK code:

  const encryptionKey = new kms.Key(this, "EncryptionKey", {
    alias: "sops-test",
  });

  const sopsProvider = new SopsSyncProvider(this, "SopsSyncProvider", {
    vpc,
    vpcSubnets: subnetSelection,
    securityGroups: [securityGroup],
  });

  const secret = new SopsSecret(this, "Secret", {
    secretName: "sops-test",
    sopsFilePath: "sops_test.yaml",
    description: "This is a test",
    sopsProvider: sopsProvider,
    rawOutput: RawOutput.STRING,
    sopsFileFormat: "binary",
    encryptionKey,
  });

I use version 2.0.3.

[markussiebert]

ok, was able to reproduce :-) thank you for your clear definition of the steps to reproduce!

sops decrypt --input-type binary --output-type binary test.yaml returns the same "error".

(Thats the same what this construct does in the lambda part)

I checked the secret created according to your instructions in more detail. It turns out that it is not purely binary, but a hybrid combination of YAML and binary. The input format appears to be binary, as indicated by the presence of a data key, while the output format is derived from the file extension and is actually YAML. If you create "real" binary secrets, they are stored as json
with "data" key.

So somehow sops cli seems to "handle" this ... but I don't know how.

So what we know:

sops decrypt test.yaml 

data: |
    abc

the same as

sops decrypt --input-type yaml test.yaml

data: |
    abc

but

sops decrypt --input-type binary test.yaml

Error unmarshalling input json: invalid character 'd' looking for beginning of value

So a workaround for you might be using output binary

It might be sufficient to use additionally --output-type binary when creating the secret.

sops edit --input-type binary --output-type binary test1.yaml # even if named .yaml it's actually json

with this it's getting wired:

sops decrypt --input-type yaml test1.yaml 
sops decrypt --input-type json test1.yaml
sops decrypt --input-type json test1.yaml

# both return
data: |
    abc

# adding --output-type binary strips the data: |
abc

# dotenv
data=abc\n

# json
{
        "data": "abc\n"
}

What I learned, sops already can convert formats - I was not aware of this. Don't know how to handle this in this construct. Will it be documentation of the "limits" and "what to take care of" or can we handle it in code. Tend towards documentation.

[MartinAltmayerTMH]

Thanks for looking into this! I was assuming that JSON and Yaml were equally well supported, but that does not seem to be the case.
Using your suggestion

sops edit --input-type binary --output-type binary test1.json

I am able to successfully upload plaintext secrets.

(I'll leave the issue open, in case you want to change some documentation. Please close it, when done.)