EJSON Merger

EJSON Merger is a simple bash script, which uses the tools spruce, jq and ejson. The main purpose of the script is, to merge multiple yaml/yml and json scripts together into one big map, which can be exported to a specific file or stdout. Since the merge process is done via spurce, you are able to use all spruce operators across all your files. See all spruce operators here. This tool gives you great advantages in your CI builds, since you are able to spread information into different files, but can join them with each other. But also, you can safely upload secrets to a repo.

Rules

• Each EJSON file has to contain it's public key. All data is given in the .data field. If this field isn't set, the EJSON file is not considered.
• JQ is required
• Spruce is required.
• EJSON is optional.
• Ruby YAML gem is required for YAML outputs.

Example

Create a new EJSON Key-Pair (More Information)

ejson keygen

Will give you something like this (Don't use these!!):

Public Key:
ff4bbf46acd0b467ee48f6e75041bc5b45442bb4b32f4bb0a2bfa928d2c21e44
Private Key:
b9f24a02dabd1f05c327c51a88f99390dab0835f0e56d4766885648cda2a51d6

First create the EJSON File, should look like:

{
  "_public_key": "ff4bbf46acd0b467ee48f6e75041bc5b45442bb4b32f4bb0a2bfa928d2c21e44",
  "data": {
     "some_password": "muchUnsecret",
     "database": {
       "user": "postgresql",
       "password": "postgresql"
     }
  }
}

Next, decrypt the created EJSON file:

$ ejson encrypt example/my-secrets.ejson
Wrote 562 bytes to example/my-secrets.ejson.

The content of example/my-secrets.ejson should look like this now:

{
  "_public_key": "ff4bbf46acd0b467ee48f6e75041bc5b45442bb4b32f4bb0a2bfa928d2c21e44",
  "data": {
     "some_password": "EJ[1:cwva0hMYQ0Si3CVnXPvFOehf9i5Le6IYQXkR8NIYlRc=:aMfnwm79BF02LbN/q9rP6JkjfNVb0RmX:E/aFMFo5YpPIitMqgQYl3DT/POUkhEcKqR2KYQ==]",
     "database": {
       "user": "EJ[1:cwva0hMYQ0Si3CVnXPvFOehf9i5Le6IYQXkR8NIYlRc=:m+rB13UUhxG6k51HuhrIrQsXLJ4g6zJF:/jyC+uV7210F1KzjGHZ8Ub/Eg/EyoF3facU=]",
       "password": "EJ[1:cwva0hMYQ0Si3CVnXPvFOehf9i5Le6IYQXkR8NIYlRc=:hYP7OqZlGkRQc2BjD9bXfUr+8F/otS75:00D6UzYcZKGLeyIBZGii/mNrFw3w7AzW6Ks=]"
     }
  }
}

As you can see your secrets are now encrypted and can only be decrypted by the matching private key. This allows you to store secrets in git repositories in a secure way, if that's what you are looking for. Next we want to create a Reference to these secrets. let's create example/configuration.yml:

...
  # Make Usage of Spruce Operators
  user: (( grab $.SECRETS.database.user ))
  host: postgres
  password: (( grab $.SECRETS.database.user ))
  database: (( concat $.PROJECT.name "-" $.PROJECT.version ))

Now you it's time to implement the decryption and merging part, which means running the script with correct parameters. It's very simple, please consider the Options section for all possibilities. For this example, the script needs the private key and the destination (where ejson and other configs are located [./example]). We want the Output to STDOUT in YAML format. The command would look like this:

bash ejson-merger.sh -p "b9f24a02dabd1f05c327c51a88f99390dab0835f0e56d4766885648cda2a51d6" -m ./example -s ./example -y

If we haven't done any Syntax Errors the output will look like this:

---
PROJECT:
  maintainer: oliverbaehler
  name: example
  version: 1.0.0
database:
  args:
    database: example-1.0.0
    host: postgres
    password: postgresql
    user: postgresql
  name: example
report_stats: false
server_name: localhost:8800
signing_key_path: "/src/.buildkite/test.signing.key"
suppress_key_server_warning: true
trusted_key_servers:
- server_name: matrix.org

As you can see our secrets were merged in clear text and other spruce operations were executed as well. Since giving this information to STDOUT makes the entire encryption part useless, I would suggest you using the -f option. This way the output will be generated into a file which then can be use by other Scripts or whatever. It's time, try it yourself? :)

You can find these files in the Example Folder

Options

To see all options available, call the -h option:

ejson-merger.sh -h

The Script currently supports the following options:

Usage: ejson-merger.sh [-h] [-p] "ejson_key" [-s] "directory" [-m] "merge_directory" [-k] "secret_key" [-f] "dir/filename" [-r]
    -p ejson_key       Add EJSON private key to decrypt ejson files
    -s directory       Source directory for ejson files (will be searched recursive) [Default "."]
    -m merge_dir       Directory where your json/yml files are located to merge with secrets [Default "."]
    -k secret_key      Top level key for secrets to be mapped to - (( grab $.secret_key.* )) [Default "SECRETS"]
    -f dir/filename    Merge all files with Secrets in one json file. Given parameter is the name of the generated JSON/YAML
    -r                 Remove Secrets from merged files [Default: True]
    -y                 Output in YAML format [Default: JSON]
    -h                 Show this context

Script logs events/errors to [./ejson-merger.sh.log]