[gh-476] Sanitise HCL variables before storing on job submission

[Juanadelacuesta]

Description

Currently Nomad only handles HCL variables with new lines, any other non alphanumeric character is left untouched and stored unescaped, which can cause errors while re starting a stopped job, particularly from the UI.

it fixes #476

Testing & Reproduction steps

Links

Contributor Checklist

• Changelog Entry If this PR changes user-facing behavior, please generate and add a
  changelog entry using the make cl command.
• Testing Please add tests to cover any new functionality or to demonstrate bug fixes and
  ensure regressions will be caught.
• Documentation If the change impacts user-facing functionality such as the CLI, API, UI,
  and job configuration, please update the Nomad website documentation to reflect this. Refer to
  the website README for docs guidelines. Please also consider whether the
  change requires notes within the upgrade guide.

Reviewer Checklist

• Backport Labels Please add the correct backport labels as described by the internal
  backporting document.
• Commit Type Ensure the correct merge method is selected which should be "squash and merge"
  in the majority of situations. The main exceptions are long-lived feature branches or merges where
  history should be preserved.
• Enterprise PRs If this is an enterprise only PR, please add any required changelog entry
  within the public repository.

[tgross]

LGTM

[tgross]

This fixes the Terraform provider and the CLI, because they both use this api package, but does the UI already have an equivalent to this fix? (cc @philrenaud )

Should we add a warning to the API documentation for the JobSubmission field that third-party callers of the HTTP API need to perform this operation as well?

[philrenaud]

The UI runs output here through this jsonToHcl() helper method

It adds \n between multiple VariableFlags / Variables entries, and the codemirror editor we use for rendering generally handles them correctly. It's not perfect, though: Your example in the test,

"test": `"foo": "bar"`

renders like this:

which isn't perfect but it's probably enough of an edge case that it's been okay since we've had HCL-in-UI.

============

I'm not sure if this is what url.QueryEscape() is doing, but if we're given output like what's in the test below (%22foo%22%3A+%22bar%22), we will show that verbatim

[jrasell]

LGTM!

[philrenaud]

Testing this out made me remember we have two different variable-related properties in Job submission data: VariableFlags, which this PR addresses and are passed via CLI props, and Variables, which are what get entered via -var-file, etc.:

nomad/api/jobs.go

Lines 978 to 984 in e206993

	// VariableFlags contains the CLI "-var" flag arguments as submitted with the
	// job (hcl2 only).
	VariableFlags map[string]string
	// Variables contains the opaque variables configuration as coming from
	// a var-file or the WebUI variables input (hcl2 only).
	Variables string

I think that VariableFlags are affected here, but Variables are not. I'm not sure if the bug in question could show up in that format, but wanted to mention it.

=========================

All that aside, I'm also noticing some perhaps erroneous character additions. Here are screenshots showing VariableFlags and output for a job where a cli-provided multi-line variable has a + character added:

on main:

on f-NET-11421-vars:

command to run was:

nomad job run -var meta_string="line1 \
\
\
\
line2" ../nomad-job-specs/unescaped_var_test.nomad.hcl

Note the + added between "line1" and "line2"