Compile to Erlang abstract format

[lpil]

Currently we compile to Erlang source. This means that when there is a runtime error the line numbers in the stack trace are incorrect.

Erlang abstract format is the data structure that Erlang parses to. https://www.erlang.org/doc/apps/erts/absform.html

If we targeted this then we would be able to annotate the generated code with line numbers, making stack traces accurate.

The Erlang compiler exposes and API for compiling abstract forms, we could adapt our existing Erlang compiler code to use this.

If we target a textual representation of the abstract forms then they we may choose to forego pretty printing and gain some performance improvement as a result.

If we target a binary representation we may get better performance still but we would need to consider how to snapshot test this code. Likely we would need to implement some printer for it.

Are there other benefits or considerations?

This would be a large amount of work. A full rewrite of the Erlang code generator.

[inoas]

I really love to see the intermediate Erlang, It helps me:

• learn Erlang
• adapt from that and write FFI code used from gleam world again
• understand runtime execution flow on the beam

I'd love to have correct line numbers for crashes but would prefer if the source maps would be added to via beam patches somehow if possible for the reasons above.

[inoas]

• Also trust is increased by gleam compiling to Erlang instead of abstract format because it is easy to verify what the compiler does and verify that it is stable and production ready and bug free.

[lpil]

Erlang abstract forms can be decompiled into Erlang source for introspection. We could make some tooling for that if that is convenient.

[inoas]

Maybe that could also be included in the erlang code gen test part of the gleam compiler and the erlang source snapshots could be kept?

[giacomocavalieri]

That sounds interesting, I have just a couple of questions since I'm not too familiar with the abstract format and how it works:

• would we loose the optimizations the Erlang compiler is applying by sidestepping the textual representation? Or those would still be applied down the compilation pipeline?
• by pretty printing the standard format how simple would it be to verify that the generated code is correct in a snapshot? Would we need to also have some way to run it to check it's behaving the way we want it to?

[lpil]

The compiler optimisations come after parsing rather than during parsing, so we would not lose any in targeting abstract forms, which is what the parser outputs.

We would continue to use snapshots in the same way, but we would be snapshotting some textual format of the abstract forms.

[giacomocavalieri]

Ok that sounds great then! I don't see many downsides besides the huge work that would be 😆

[lpil]

I'm seeing this as a good "we've got pretty much everything finished and now we want to optimise" type item of work

[giacomocavalieri]

Yeah totally!