The Veracruz build system is not fit for purpose

[dominic-mulligan-arm]

Based on various discussions in the last two Veracruz public meetings, it's become clear that there's a number of problems with the current Veracruz build system, not all of which are immediately apparent. We have:

• AWS CodeBuild integration with Github is not very smooth,
• Various Docker containers used in the Veracruz build and development process are too heavyweight, and can be engineered in a different way to reduce CI build times,
• Veracruz supports a lot of different platforms, but not all of these are currently being tested by the CI system (e.g. SGX, AWS Nitro),
• Veracruz is about to support additional platforms (Linux, seL4 via IceCap) making the build even more complicated,
• The build process is dragging in multiple versions of the same dependency and linking against them,
• There are problems doing parallel builds with Veracruz, perhaps related to our use of Xargo,
• We should be aiming to produce reproducible builds of the core Veracruz runtime (and maybe examples), but this invites discussion on what "reproducible builds" means, and how long these builds should remain reproducible,
• ...and so on.

The purpose of this discussion is to properly summarise all of the different problems with the current build system (I have probably missed many from the list above that have been discussed, previously). Moreover, once all problems are identified we should try to form some consensus on what needs to be done to fix them.

[nspin]

One aspect of the build system that I perceive as a problem is that Cargo reigns over most of the build. For example, ./veracruz-server-test/build.rs invokes make in ./test-collateral. I don't think Cargo can or should serve this role. Instead, I suggest that we demote our crates to subjects of the build system, and try to limit Cargo's responsibility to just building Rust code.

More generally, this fits into what I perceive as a wider theme, which is that in some instances code is mixed in with the build system in ways that complicates both the code and the build system.

[dreemkiller]

I agree that using build.rs to call external tools (such as make but also bash scripts or anything else) is not the way to go (even though I added a lot of those build.rs files). build.rs is just not flexible enough to do this correctly or consistently.

Limiting cargo to building rust code seems correct.

[geky]

Just a note that Cargo has some personality issues and gets a bit grumpy if it's not the build system in control:
https://github.com/ARM-software/bento-linker/blob/b6cbdb6b7b35ebd96fcac070e5045919acdb65bf/examples/rust/Makefile#L30-L39

Also note that for any crate that provides a user-facing Rust API, Cargo still needs to be in charge, otherwise the crate would not get fully built if added as a user's dependency. veracruz-server comes to mind.