JSON

[Daniel15]

Is your feature request related to a problem? Please describe.

Thank you for working on this useful project! I was wondering whether you'd consider adding support for JSON files?

Describe the solution you'd like

Initially it could be something basic, like just a list of paths to ignore.

Round-tripping JSON shouldn't be too difficult, and a lexer + parser that retains trivia should let you retain the same formatting on output.

Describe alternatives you've considered

N/A

Additional context

[VorpalBlade]

While I have not thought about json specifically, I did consider xml at one point. Json should be simpler than XML.

One thing that would be needed would be a format-preserving Json parser and serialiser (e.g. not change newlines, indentation, no reformatting numbers, etc). For INI files I wrote my own code for this, but I'd rather not for Json as it is a more complex format.

I also don't have a personal need for this, and this is a hobby project. So, contributions are welcome. If anyone is interested I'm willing to discuss the design (the first new format to support will be the hardest design-wise, as we need to make the code generic over the file format).

[VorpalBlade]

So I had a think about what would be needed:

JSON

• Find or write a library for round trip parsing of JSON. Preferably JSON5 (superset of JSON with support for comments and trailing commas) as that would be more general. I have not found any for Rust at least. Nor really for any other language. A stream SAX style parser would be preferable for the algorithms needed. There is no need to build a full DOM for the system state (and we can use the stream from the SAX parser to build a DOM ourselves for the source state).
  • Things to keep in mind is that number formatting can vary and we want to treat scalar values as opaque strings as far as possible (otherwise we might reformat 0.10 to 0.1 for example, along with various nasty issues around floating point precision etc).
  • If we have to write our own parser it would be preferable to use the winnow parsing library, as that is already in use for parsing the configuration script. Keeps the number of dependencies down (which help with compile time and binary size)
• Figure out a way to describe matches over the data structure:
  • JSONpath seems like the obvious choice, but most Rust implementations seem tied to a specific library for parsing the JSON data. There is one library to parse jsonpath syntax (rsonpath) but the associated engine for evaluating that seems not reusable for our purposes.
  • Perhaps a simplified version of JSONpath might be suitable instead.
• The current merge and filter algorithms are tightly coupled to how INI files are represented. I believe the general approach should work, but there are too many INI-specific quirks for a general implementation to be possible (for example, handling keys before the first section in INI files). That said, I do believe that JSON and some other potential future formats like YAML could share an implementation.
  • It isn't clear how to handle arrays, as the items don't have unique keys. Some thought is required here. Sometimes it might represent an unsorted collection, other times it is a sorted list, otherwise it might be conceptually keyed on a field in the objects stored in the array.
• Finally there would need to be a language directive in the chezmoi_modify_manager config that selected the processing pipeline to use, along with some tweaks to which directives are valid for which file formats.
  • ignore, remove, add:remove seems fairly obvious how to support for JSON
  • set and the currently existing transforms is fairly straight forward for scalars in JSON
  • Trying to set or add:hide an array or object in JSON would require some thought, can be skipped initially, but lets avoid painting ourself into a corner with the design at least.

Other formats

Side note about about XML (since I also considered this case): XML would be more complicated: you may not have unique keys, you can have text as siblings to nodes, you have attributes on nodes (not just contents) etc.

Side note about toml: toml_edit exists as a Rust library, but this is a DOM style library as far as I know. Not ideal since the merge and filter algorithms is written for a streaming approach.

[vasi]

It looks like there's a SAX-style XML parser out there: https://docs.rs/xmlparser/latest/xmlparser/. It still sounds pretty awkward to support in some ways. Eg: let's say I want to support KXMLGUI files. The parts of KDE that write those files auto-prettify them, so it's possible that if a line gets too long, it reflows! How are we supposed to diff that?

[VorpalBlade]

Xml would be even more complex than Json (see side note at the end of my comment above). I don't know how to handle it, nor do I currently have the energy to embark on a project like that.

In fact, both json and xml support would have to be primarily done by someone other than me.