hints for handling complex JSON?

[JeffMelton]

It's me again :)

In my day job, I help support almost 6000 Kubernetes clusters running on VMware on-prem in retail locations. Each of those has between 6 and 13 nodes, with between 200 and 500 pods (depending on how 🔥 the day is). When I'm on call, that scopes out to 10s of thousands of clusters, with astronomical numbers of nodes and pods. So the scale I work with is massive. On top of that, Kubernetes and VMware JSON objects are massive and complex, with deeply nested structures that very few tools handle gracefully. I say all that as context for why I always have my eyes open for tools that will make my job easier. Murex's built-in support for structured data and your commitment to stability and backward-compatibility are why I'm here. I've already noted a few ways I find Murex easier to work with than my current daily driver (Elvish), and while it isn't as immediately familiar as Nushell, it also doesn't seem likely to introduce breaking changes as frequently.

In a typical workflow, I'll grab JSON from a number of similar resources, and I'll parse those objects for further action or for human-readable display. The JSON response will often consist of an outer array with root-level key/value pairs, keys that have array values consisting of a mix of key/value pairs and other keys with array values. [I can't provide a full JSON response, I'm afraid — they're just too big for me to go through to make sure I'm not leaking info that would put me on infosec's naughty list.] A trivial example using jq:

❯ kubectl get no -o json -l gpu=t4 -> jq '.items[] | {hostname: .metadata.labels."kubernetes.io/hostname", allocatable: .status.allocatable."nvidia.com/gpu"}'
  "hostname": "node1",
  "allocatable": "1"
}
{
  "hostname": "node2",
  "allocatable": "1"
}
{
  "hostname": "node3",
  "allocatable": "1"
}

[jless shows the path to the first item's hostname above as: .input.items[0].metadata.labels["kubernetes.io/hostname"]]

The first unexpected outcome here is that I have to cast JSON output explicitly to JSON. Murex only sees a generic string. I can get at .items[] fairly easily: … -> :json: [items] Once I've done that, I have jsonl — an array of items. I can't just … -> :json: [items] -> [metadata] or … -> :json: [items] -> [status]. But I was able to get what I need with:

kubectl get no -o json -l gpu=t4 -> :json: [items] -> foreach item {out $item[metadata] -> :json: [labels] -> [kubernetes.io/hostname] -> set name; out $name}

The foreach makes sense since it's an array, though I wonder if there's a way to do that concurrently. [PowerShell, Elvish, and Nushell all have some kind of parallel each, in order of increasing speed.] Casting explicitly to JSON twice feels clunky, and it leads me to assume I'm doing something nonsensical.

Using the path approach (e.g., [[ /metadata/labels ]] gets me nowhere. I've seen a couple references to multi-dimensional JSON in the docs, but mostly in regard to creating it, not parsing it. I never create raw JSON, and really the only thing I'd do with the jq output above is convert it to something more human-readable by passing it into mlr or something.

Same with a JSON response from vSphere:

govc vm.info -json node3 -> :json: [virtualMachines] -> foreach vm { out $vm[summary] -> :json: [vm] -> [value] }
govc vm.info -json node3 -> jq -r '.virtualMachines[].summary.vm.value'

[the jless path looks like .input.virtualMachines[0].summary.vm.value]

As ever, I recognize I'm a n00b here, so I start with the assumption I'm just holding it wrong. :) Are there more idiomatic ways to approach parsing complex JSON like this?

[lmorg]

Sounds like a few issues.

Casting

Generic is the default output from any POSIX pipe because it's hard to assume what an output will be for tools that just pass raw byte streams.

You can create a function to override that default, eg

function vminfo {
    cast json
    exec govc vm.info -json @PARAMS
}

so now your command line looks like:

vminfo node3 -> [virtualMachines] ...

I've been hesitant to do any automatic detection of data types based on their stdout/stdin because it is error prone. But if you know of a way it can be implemented in a reliable and deterministic way, then it's definitely something I can consider.

Element vs Index

Index [ ... ] is useful because it's case insensitive and supports a bunch of other grouping and ranging functions. But if you need to access nested structures then you're better off with using elements: https://murex.rocks/parser/element.html#description

Elements take the first character as the separator. So if we take the following path

.input.items[0].metadata.labels["kubernetes.io/hostname"]

...that could be used in element:

-> [[,input,items,0,metadata,labels,kubernetes.io/hostname]] ->

Foreach

Your following code can be golfed quite a bit:

... -> foreach item {out $item[metadata] } -> ...

sharing this as a baseline ^

foreach also passes the object to stdin. So if you know you're just filtering operating on that object you can write your code like the follow:

... -> foreach { -> [metadata] } -> ...

However if you're just working in the command line, you'll want something that's a little more ergonomic on your fingers. So the lambda [{ ... }] syntax might work better for you here.

# before:

-> :json: [items] -> foreach item {out $item[metadata] -> :json: [labels] -> [kubernetes.io/hostname] -> set name; out $name}

# after:
-> :json: [lines] -> [{$.v.item.metadata -> [[,labels,kubernetes.io/hostname]]}]

https://murex.rocks/parser/lambda.html

It's not particularly readable though so I tend to discourage this for shell scripts. However working in the REPL is usually more about speed than readability

I'm also working on the assumption that .labels["kubernetes.io/hostname"] is a JSON-encoded string inside metadata rather than an object itself. ie something like:

{
    "items": [
        "{\"labels\":{\"kubernetes.io/hostname\":\"foobar\"}}",
        "{\"labels\":{\"kubernetes.io/hostname\":\"foobar\"}}"
    ]
}

which will by why you need to re-cast.

Parallelism

There isn't presently a way to iterate through a loop in parallel. I'd be interested to understand your use case because it's something I could easily add (Murex is already heavily multi-threaded).

Looking at Nushell, it has something called per-each, is that the kind of thing you had in mind?

Presently you could use bg to spawn each loop iteration in it's own thread. eg

... -> foreach item { bg { long-running-command $item } }

That comes with several risks:

1. everything will get kicked off at the same time. So if you're working with thousands (or more) of items in your array then you will crash Murex. Possibly your computer too.
2. if you need to do anything meaningful with your output, and I'd wager you would want to, you then might need to manually handle those streams with either murex named pipes or writing to disk

So having something managed is definitely preferable.

Maybe an additional flag could work here, eg -p n where n is the number of parallel worker groups. And output can be buffered so each command doesn't fight to write to stdout while another command is writing to it?

[JeffMelton]

This is all really helpful, thank you. For clarity, the input object roughly looks like (LLM-generated; the quoting is correct).

# actual
{
  "apiVersion": "v1",
  "items": [
    {
      "apiVersion": "v1",
      "kind": "Node",
      "metadata": {
        "name": "node1",
        "labels": {
          "gpu": "t4",
          "kubernetes.io/hostname": "node1-hostname"
        }
      },
      "status": {
        "capacity": {
          "cpu": "16",
          "memory": "64Gi",
          "nvidia.com/gpu": "4"
        },
        "allocatable": {
          "cpu": "15",
          "memory": "63Gi",
          "nvidia.com/gpu": "4"
        }
      }
    },
    {
      "apiVersion": "v1",
      "kind": "Node",
      "metadata": {
        "name": "node2",
        "labels": {
          "gpu": "t4",
          "kubernetes.io/hostname": "node2-hostname"
        }
      },
      "status": {
        "capacity": {
          "cpu": "32",
          "memory": "128Gi",
          "nvidia.com/gpu": "8"
        },
        "allocatable": {
          "cpu": "31",
          "memory": "127Gi",
          "nvidia.com/gpu": "8"
        }
      }
    }
  ],
  "kind": "List",
  "metadata": {
    "resourceVersion": "123456",
    "selfLink": "/api/v1/nodes"
  }
}

I figured casting multiple times was PEBKAC. 😆 I'll take another stab at it. I had looked at element (though I'd overlooked the bit about the first character being the separator) because it seemed like I should be able to use it. Since the JSON response is an array of items I'll want to iterate over, it could I then do something like this?

-> :json: [items] -> { 
    [[,metadata,labels,kubernetes.io/hostname]] -> set node_name
    [[,status,allocatable,nvidia.com/gpu]] -> set allocatable
}

As far as it concerns parallelism:

One of my non-interactive use cases is discovering unhealthy clusters (our monitoring/alert coverage is less than ideal). I start with a Splunk query that surfaces a number (usually between 15 and 40) of potential issues. I parse that query object for cluster IDs, then connect in parallel to each cluster (kubectl get --context <cluster ID>) to validate the Splunk query. In that same subprocess, I check each cluster's Prometheus Alertmanager instance to make sure we don't already have an open incident or active silence. That narrows the list of clusters to only those needing attention. Then in a single thread, I pretty-print a table with some information and background open a URL pointing to the site's incident queue. Using 9 threads on my M1 MacBook Pro, those can take a couple minutes to run (except in PowerShell which is significantly slower, despite its native parallelism).

My interactive REPL use cases are often larger in scale because I find it quicker to scan the entire estate than to pop out of the terminal to Splunk for initial filtering. We're looking at several thousand processes there.

Nushell has par-each (e.g., parallel-each), PowerShell has ForEach-Object, and for non-interactive use cases where I don't have access built-in parallelism, I have been using rust-parallel. Each of those have flags that let you set the number of forks, though I think they default to using the number of CPU cores available --- I often use n(cores) - 1. Elvish has peach, but the number of forks it will spawn is unbounded, which I discovered when I crashed my terminal emulator. 🫠 All of the above collect output from threads in a way that allows capturing into a single variable, pipe, or file handle. The approach taken by Nushell, PowerShell, and rust-parallel seems robust to me, with sane defaults as well as a flag the operator can use.

To be fair, the scale of operations I help support is an order of magnitude greater than most engineers will ever encounter. I've yet to quantify Murex's capabilities, but once I've gotten a few of these things written and tested, I'll do so. If the Murex code base is already highly concurrent and it's not a massive effort to add some parallelism, that's a big win in my eyes. If you'd have to follow Fish's example and rebuild the ship of Theseus in order to support it, the juice might not be worth the squeeze. YMMV, as they say.

Thanks again for the product and the interaction. I'm enjoying myself. 🤓

[lmorg]

-> :json: [items] -> { 
    [[,metadata,labels,kubernetes.io/hostname]] -> set node_name
    [[,status,allocatable,nvidia.com/gpu]] -> set allocatable
}

^ this isn't valid syntax.

1. missing foreach
2. [[,metadata,labels,kubernetes.io/hostname]] isn't being selected from anything (eg stdin nor var)
3. you couldn't read from stdin multiple times. So if you needed to read two properties from items then your first approach using variables would be the correct one
4. you're also going to overwrite node_name and allocatable on each iteration. so unless you're reading from those values within the foreach loop, you'll find those vars will just have the last item in the array once foreach has completed

If this was a shell script, you could probably write it to output a touple, eg

-> :json: [items] -> foreach item {
    %{
        name:  $item[[,metadata,labels,kubernetes.io/hostname]]
        alloc: $item[[,status,allocatable,nvidia.com/gpu]]
    }
}

which with your sample dataset will produce:

{
    "alloc": "4",
    "name": "node1-hostname"
}{
    "alloc": "8",
    "name": "node2-hostname"
}

of data type jsonl. If you were to -> format json then it would look like

[
    {
        "alloc": "4",
        "name": "node1-hostname"
    },
    {
        "alloc": "8",
        "name": "node2-hostname"
    }
]

but you shouldn't need to add that format json because jsonl is a first class citizen in Murex.

You then have a data file you can do with as you wish...or you could just run a command instead of using %{...} if you prefer

---

Playing around with your sample dataset, the lambda might look like this:

~ » $json -> :json: [items] -> [{$.v.metaData.labels.[kubernetes.io/hostname]}]
node1-hostname
node2-hostname

This would be str (which Murex would treat as an array of strings with any array-based commands)

---

Regarding par-each, I can easily add a flag into foreach for parallelism. You wouldn't get much value for the examples above because %[ ... ] shouldn't add much, if any, overhead. But if you were to execute an external command then there would definitely be a speed up.

[JeffMelton]

Here's what I came up with for one of those functions:

function check_gpu_allocatable {
    # function to show whether Kubernetes has marked a node's GPU as allocatable
    kubectl get no -l gpu=t4 -o json -> :json: [items] -> foreach item {
        -> [[,metadata,labels,kubernetes.io/hostname]] -> set node \
        && out $item -> :json: [[,status,allocatable,nvidia.com/gpu]] -> set bool allocatable \
        && %{ node: $node, allocatable: $allocatable }
    } -> mlr --j2p reorder -f node,allocatable
}

[JeffMelton]

-> :json: [items] -> { 
    [[,metadata,labels,kubernetes.io/hostname]] -> set node_name
    [[,status,allocatable,nvidia.com/gpu]] -> set allocatable
}

^ this isn't valid syntax.

1. missing foreach
2. [[,metadata,labels,kubernetes.io/hostname]] isn't being selected from anything (eg stdin nor var)
3. you couldn't read from stdin multiple times. So if you needed to read two properties from items then your first approach using variables would be the correct one
4. you're also going to overwrite node_name and allocatable on each iteration. so unless you're reading from those values within the foreach loop, you'll find those vars will just have the last item in the array once foreach has completed

If this was a shell script, you could probably write it to output a touple, eg

-> :json: [items] -> foreach item {
    %{
        name:  $item[[,metadata,labels,kubernetes.io/hostname]]
        alloc: $item[[,status,allocatable,nvidia.com/gpu]]
    }
}

which with your sample dataset will produce:

{
    "alloc": "4",
    "name": "node1-hostname"
}{
    "alloc": "8",
    "name": "node2-hostname"
}

of data type jsonl. If you were to -> format json then it would look like

[
    {
        "alloc": "4",
        "name": "node1-hostname"
    },
    {
        "alloc": "8",
        "name": "node2-hostname"
    }
]

but you shouldn't need to add that format json because jsonl is a first class citizen in Murex.

You then have a data file you can do with as you wish...or you could just run a command instead of using %{...} if you prefer

Playing around with your sample dataset, the lambda might look like this:

~ » $json -> :json: [items] -> [{$.v.metaData.labels.[kubernetes.io/hostname]}]
node1-hostname
node2-hostname

This would be str (which Murex would treat as an array of strings with any array-based commands)

Regarding par-each, I can easily add a flag into foreach for parallelism. You wouldn't get much value for the examples above because %[ ... ] shouldn't add much, if any, overhead. But if you were to execute an external command then there would definitely be a speed up.

Looks like I have some golfing to do. Thanks!

[lmorg]

out $item -> :json: [[,status,allocatable,nvidia.com/gpu]] -> set bool allocatable

$item is already json, so you can save yourself a cast if you drop out (bar variables print rather than execute, like they would in Bash).

eg

$item -> [[,status,allocatable,nvidia.com/gpu]] -> set bool allocatable

Looks like I have some golfing to do. Thanks!

Any time. Keep the questions coming 👍