Proposal: add decoder for nested optionals, handling Nils in the path

[jhillyerd]

Problem Summary

The new use decoder API lacks an easy way to cherry-pick a nested value when the path to that value may have missing or null elements. It is possible with the current API by mixing optional_field, optional, and at – but the method is not simple or easily discoverable.

Cherry-picking optional nested values is very helpful when the Gleam type you wish to decode into does not closely match the schema you are decoding from, or if you don’t have a schema and are iterating on the design of your type & decoder.

Proposed Solution Summary

Add an nullable_subfield function to the dynamic/decode module, similar to the existing subfield function, but allowing for a default value, signature:

pub fn nullable_subfield(
  field_path: List(name),
  default: t,
  field_decoder: Decoder(t),
  next: fn(t) -> Decoder(final),
) -> Decoder(final)

In addition, we can consider modifying the existing optionally_at function to handle “null” and not just missing values. Its current behavior might be considered a bug, but altering it might also be considered an API change.

Problem Detail

Writing a decoder to optionally extract a value for a.bc from the below JSON samples is more difficult than one would expect.

{"id": 1}
{"id": 1, "a": null}
{"id": 1, "a": {"b": null}}
{"id": 1, "a": {"b": {"c": null}}}
{"id": 1, "a": {"b": {"c": “target value”}}}

There are several difficulties:

1. There is no optional subfield function for the new use decoder API, so one must start with optional_field and then pass in regular Decoder to navigate deeper into the tree Louis pointed out that optional_at can be combined with then to make it a use decoder.
2. optionally_at does not handle null values along the path, only missing keys (see failure example below)
3. While one_of + at can be a concise way to decode a nested value, it does not allow the caller to differentiate a missing value from an invalid one, masking errors (see failure example below)
4. The solution I found, chaining optional and at results in a nested set of Options that need to be flattened

Problem Example Code

This example illustrates the existing way to decode this, along with some assertions you may use to test alternatives if you feel I’ve missed something. It should be usable in a fresh gleam project after running gleam add gleam_json.

The example below succeeds, but the combination of optional_field, optional, and at is difficult to read, non-obvious, and requires a matching chain of option.flatten calls to unwrap the result.

import gleam/dynamic/decode.{
  at, field, int, optional, optional_field, string, success,
}
import gleam/io
import gleam/json
import gleam/list
import gleam/option.{type Option, None, Some, flatten}

pub type Thing {
  Thing(id: Int, c: Option(String))
}

pub fn main() {
  let thing_decoder = {
        use id <- field("id", int)
        use str_val <- optional_field(
          "a",
          None,
          optional(at(["b"], optional(at(["c"], optional(string))))),
        )

        success(Thing(id:, c: str_val |> flatten |> flatten))
  }

  let none1 = "{\"id\": 1}"
  let none2 = "{\"id\": 1, \"a\": null}"
  let none3 = "{\"id\": 1, \"a\": {\"b\": null}}"
  let none4 = "{\"id\": 1, \"a\": {\"b\": {\"c\": null}}}"

  [none1, none2, none3, none4]
  |> list.each(fn(o) {
        io.println("input: " <> o)
        let assert Ok(Thing(1, None)) = json.parse(o, thing_decoder)
  })

  let some1 = "{\"id\": 1, \"a\": {\"b\": {\"c\": \"string\"}}}"
  io.println("input: " <> some1)
  let assert Ok(Thing(1, Some("string"))) = json.parse(some1, thing_decoder)

  let error1 = "{\"id\": 1, \"a\": {\"b\": {\"c\": 7}}}"
  io.println("input: " <> error1)
  let assert Error(json.UnableToDecode([
        decode.DecodeError("String", "Int", ["b", "c"]),
  ])) = json.parse(error1, thing_decoder)
}

Failing Examples

Trying to use optionally_at instead:

  let thing_decoder = {
    use id <- field("id", int)
    use str_val <- then(optionally_at(["a", "b", "c"], None, optional(string)))
    
    success(Thing(id:, c: str_val))
  }

Will fail to decode when a is null:

input: {"id": 1, "a": null}
runtime error: let assert

Pattern match failed, no pattern matched the value.

unmatched value:
  Error(UnableToDecode([DecodeError("Dict", "Atom", ["a"])]))

Trying to use one_of instead:

  let thing_decoder = {
    use id <- field("id", int)
    use str_val <- then(
      one_of(at(["a", "b", "c"], string) |> decode.map(Some), [success(None)]),
    )

    success(Thing(id:, c: str_val))
  }

Will decode a None, instead of failing when given a number instead of a string:

runtime error: let assert

Pattern match failed, no pattern matched the value.

unmatched value:
  Ok(Thing(1, None))

Solution Detail

My ideal solution would allow the decoder block above to be written as:

  let thing_decoder = {
    use id <- field("id", int)
    use str_val <- nullable_subfield([“a”, “b”, “c”], None, optional(string))

    success(Thing(id:, c: str_val))
  }

[lpil]

Hello! This would be written today like so:

use value <- decode.then(decode.optionally_at(
  ["a", "b", "c"],
  None,
  decode.optional(decode.string),
))

We could possibly add this new function, but it's best to wait some time to get an understanding of how the current API is going. The stdlib is very conservative so we delaying any additions is good.

[jhillyerd]

Thanks, I was looking for a function glue regular decoders to use, I didn't realize then was it -- I may send a docs PR to add an example of that. I've updated my failing examples above to use then so they are easier to read.

However, then + optionally_at doesn't actually work in the scenario I gave, it fails when it encounters a null: UnableToDecode([DecodeError("Dict", "Atom", ["a"])]). This feels like a bug/oversight to me, given we often can't control how the JSON given to us has been encoded, sometimes values may be null and other times they will not be present.

[lpil]

Could you give an example of when it would fail? I thought I tested with all your examples above, but perhaps I am misremembering.

[jhillyerd]

Yes, given a decoder containing:

use str_val <- decode.then(decode.optionally_at(
  ["a", "b", "c"],
  None,
  optional(string),
))

And the following 3 JSON inputs:

{"a": {}}
{"a": {"b": {}}}
{"a": {"b": null}}

The first two will decode to None as expected, but the third case will fail:

runtime error: let assert

Pattern match failed, no pattern matched the value.

unmatched value:
  Error(UnableToDecode([DecodeError("Dict", "Atom", ["a", "b"])]))

I am running these tests with Erlang, not sure if JS would behave any differently.

[jhillyerd]

I've created a proposed fix here: gleam-lang/stdlib#809

[jhillyerd]

My proposed fix in gleam-lang/stdlib#809 ended up raising a concern about preventing callers from intentionally failing on Nil when they desire that behavior. That's not likely to be a concern for JSON decoding, but certainly could be an issue for other uses of dynamic. Instead of changing optionally_at, it may be better to introduce a new function. A possible implementation is in gleam-lang/stdlib#811

I expect we will need more community interest to drive this forward, so please leave a comment if you have questions or a use case for this.

I've made a number of edits to my original proposal to correct mistakes and focus on the Nil issue.