Proposal: an attribute to embed file data at compile time

[TanklesXL]

Following up on some discussions that were had in discord, this proposal entails the addition of the @embed attribute (name open for discussion, but for convenience here I will use the aforementioned attribute name), inspired by go's //go:embed directive that was added in go 1.16.

The goal

To add a way to include file data in gleam code at compile-time rather than depending on accessing the filesystem and hoping that certain files exist at runtime.

Scope

This proposal is limited to the inclusion of single files, i.e. 1-1 mappings of files to gleam constants, this is a subset of the functionality that go's embed directive provides. We aren't going to consider embedding filesystems or directories for the time being.

The syntax

The impacts on gleam's syntax here (aside from the addition of the @embed attribute) involves 1 potential change (examples 3 and 4 below), or none at all.

Given some file foo.txt existing on the build computer and a string constant bar, I would expect an embed to look like:

// embedding `foo.txt` when it exists in the same directory as the gleam file containing `bar`

@embed("foo.txt")
const bar: String = ""

// embedding `foo.txt` when it exists in a different directory than the gleam file containing `bar`

@embed("../inputs/foo.txt")
const bar: String = ""

// embedding `foo.txt` when it exists in the same directory as the gleam file containing `bar`, with a change to gleam's syntax to not require a value defined when `@embed` is present.
 
@embed("foo.txt")
const bar: String

// embedding `foo.txt` when it exists in a different directory than the gleam file containing `bar`

@embed("../inputs/foo.txt")
const bar: String

Considerations

1. What types should be allowed as embedded?

String seems an obvious choice, but what if a file is not guaranteed to be utf-8 encoded? We might want to allow BitString as an option. If a constant is declared with @embed as String and is not valid utf-8, should the compiler fail and report an error? If a constant is declared with @embed as BitString should compilation succeed so long as the file is found and can be read successfully?

// embedding `foo.txt` when it exists in the same directory as the gleam file containing `bar`

@embed("foo.txt")
const bar: BitString = <<>>

// embedding `foo.txt` when it exists in a different directory than the gleam file containing `bar`

@embed("../inputs/foo.txt")
const bar: BitString = <<>>

// embedding `foo.txt` when it exists in the same directory as the gleam file containing `bar`, with a change to gleam's syntax to not require a value defined when `@embed` is present.
 
@embed("foo.txt")
const bar: BitString

// embedding `foo.txt` when it exists in a different directory than the gleam file containing `bar`

@embed("../inputs/foo.txt")
const bar: BitString

2. Does a value need to be defined?

Having the value present in examples 1 & 2 is consistent with gleam's current const syntax, and might be helpful if we want to consider default values for embedded files, with a warning at compile-time if the file is not found. The problem with this is we now have different behaviours for an embed when it has a value other than "" or <<>>.

My personal recommendation is as follows:
I would personally prefer to not need to provide a value for constants that use the @embed attribute (as show in examples 3 & 4). However, the change of allowing a value to not be provided for a constant differs from gleam's current syntax and likely has further ramifications with regards to how parsing the language is done. If allowing a value to not be provided is deemed too complicated or not a worthwhile change, then I am in favour of requiring String values decorated with @embed to always be given a value of "" and for BitString values decorated with @embed to always be given a value of <<>>.

3. Relative paths

Should paths be relative to the gleam file that contains the const with @embed, or should they be relative to the root of the project (where gleam.toml is)?

Behaviour

• if a constant is declared with @embed and the file is not present at the specified path, compilation fails and an error is returned to the user explaining that the file was not found.
• if the file is found and can be successfully encoded as the declared type, it is included in the compiled output as the value of the const

[thelinuxlich]

This is intriguing, I wonder if it could be extended to accept a parser, think about the possibility of having views in HTML and mustache-tags replaced by actual gleam variables.

[TanklesXL]

Yeah being able to load test fixtures and JSON blobs is one of my main uses for embed in Go tests. It's so handy to not need massive multiline strings polluting your test files.

[TanklesXL]

No clue on the templates but I am intrigued by the idea

[sporto]

I think we will need to pass a parser.

Eg.

fn parseMustache(raw: String) -> Result(Mustache, MustacheParseError)

@embed("templates/layout.mustache", parseMustache)
const layout: Mustache

Which is starting to look like compile time code evaluation

[thelinuxlich]

Maybe parser should always be external fn?

[lpil]

If it's in Gleam we're a long way off that unfortunately.

[inoas]

I think = "" should be gone unless it is a fallback default because that is how it reads.

[brettkolodny]

I could imagine a scenario where you want to generate some files before compiling as part of a build step, like maybe some static site generation where you want to pull in up to date content every time you build.

The idea of having a default that causes an error is similar to how Erlang NIFs are set up though maybe not a comparable situation.

[inoas]

tbh I don't like the idea of a default. I think the file should just be present and readable at compile time. If you want a default just supply a default file manually. But IF we want to keep the syntax with = "" it should treat that as a default.

Edit: Also there should always be a way to not have a default so we need to change the syntax anyway?

[TanklesXL]

The panic idea seems a bit weird here in that it’s mixing compile time and runtime behaviours.

I would also prefer no defaults, my recommendation for keeping = “” was based off minimizing changes to the language

[brettkolodny]

I guess it's weird since it's a const, but personally I don't think it's really going any further than what it's already going to do. But I'd be on board with no defaults as well.

[inoas]

The panic idea seems a bit weird here in that it’s mixing compile time and runtime behaviours.

good point!

[bcpeinhardt]

Hmmm. So I think this would be a great feature and use of the new attribute syntax. I think I lean toward a very minimal version of this as opposed to some of the ideas above.

I like the idea of bitstring being the only really acceptable use. That way the only error case possible is that it can't find the file (feel free to point out the other error cases I'm missing). Maybe add an @embed_string as well.

I think the way to make this syntax make sense is to actually append the bitstrings, so

@embed("foo.txt")
const bar: BitString = <<"Hello, ":utf8>>

where the file foo.txt contains the text World would produce the bitstring <<"Hello, World":utf8>>. Then people can simply
provide a <<>> most of the time.

I think the only other question was the filepath relative location. I think it should be relative to gleam.toml. If the same asset is used multiple places in the project the path will be the same everywhere, so refactoring etc. is easier.

[hayleigh-dot-work]

Allowing arbitrary relative paths are an attack vector if we let the break package boundaries.

I'd bet good money on the most-common operation being an immediate bit_string.to_string if we make all embeddings BitString only.

[bcpeinhardt]

I agree with both of these points.

Paths should certainly be sanitized against breaking package boundaries, and I think should probably generate a compile time error if they do. I'd be curious to know how go's embed and rust's include handle this. I can't think of a use case for loading file contents outside the project.

Do attributes lex and parse the expressions/statements they modify? I wonder if having a single attribute modify String and BitString and behave differently depending on what they find would be easier or harder to implement than two attributes, say @embed and @embed_utf8.

[lpil]

I don't see any reason why it couldn't behave differently depending on the type annotation given to the constant.

[inoas]

I am now more uncertain if I would want this to be part of the @attribs-syntax.

Is this for embedding arbitrary binary data? To say include small PNGs in source code?

Otherwise could this be part of a heredoc/nowdoc syntax?

heredoc via file

let my_string = << "path/to/my/string/file.txt"

inlined heredoc:

let my_string = <<<
  Foo Bar Quux
  >>>

If we want to also include non string binaries:

let my_string = <<String "path/to/my/string/file.txt"

let my_bit_string = <<BitString "path/to/my/string/file.txt"

[chuckwondo]

I much prefer something along these lines, whether using a heredoc type of syntax or not.

I much prefer some sort of syntax that indicates the source of the value on the RHS, rather than some sort of @ "label" above with a somewhat non-obvious meaning of some placeholder/default value on the RHS, particularly when using the const keyword.