Skip to content

Latest commit

 

History

History
178 lines (135 loc) · 13.9 KB

35-go.md

File metadata and controls

178 lines (135 loc) · 13.9 KB

What's the optimal experience for Go?

Assume we can do in-bramble incremental builds? With that I could do this:

def build():
    return go_build(files["./"])
def mod_tidy():
    return go_mod_tidy(files["./"])

and then you just bramble build :build and bramble run :mod_tidy.

This does create a bit of a duality though. What if someone wants to use the Go command in a way that's not available/supported.

You could just run go build and then you'd need to build everything again from scratch.

You could just run bramble run :go_build go build but then the go command is going to build stuff the way it likes to build things and it will ignore the artifacts available within that context. The cache would be cleared on every run. You also can't fix this in software because even a go command that is aware of the build cache won't be able to write new stuff to the store.

So if this is how it all plays out it seems like people would have an unsatisfying experience and try and work towards compromises, so what would be ideal?

Ideal is maybe that we provide a Go command that can leverage bramble while still being the interface for the build. We basically provide a go command and that go command runs bramble in order to work. We could do this with bramble run but then it would need the ability to write to the store, could provide an api for that.

So basically we want go commands that are an alias for bramble run something and then the go command is really a wrapper that is interacting with bramble how we want.

Ok, so key takeaway is: allow a process in the bramble run sandbox to write/build to the store via a socket (or whatever) to allows more dynamic use-cases.

If we start with a Go project that has a handful of dependencies. Typically you would go mod tidy, but go mod tidy downloads many dependencies before writing the full resolution state to go.sum.

Ok, so what if we:

  1. Scan project
  2. Take existing go.sum and download any of those dependencies with dynamic derivations.
  3. Run go mod tidy
  4. Return more dynamic derivation derivations with that data to redownload again.

Not sure how to get around the double download problem. Even if we could get the download logic into bramble we would still need to

How would we handle basic go building?

Let's write out all the pieces and how they would work. Core assumptions:

  1. The project exists and has a working go.mod and go.sum file.
  2. Builds must be handled incrementally with derivations, no build cache on disk in a derivation that is then wiped.

First, let's download dependencies for go.mod.

If we check out vgo2nix we can see in the example deps file that it's pulling dependencies from git. Going to assume that wherever those are downloaded, when they're loaded into the build they're passed as a list to a derivation. The derivation then either loads them into GOPATH or the module package store and conducts the build. Since we're providing the functional guarantees of modules, presumably it will be ok to put modules in GOPATH and build from there. We do run into issues with case-insensitive filesystems that are solved with the module file store.

Vgo2nix also pulls the full git repo to get the sha256, we can prob skip that step. Also worth noting that it may seem tempting to want to use the GOPROXY cache instead of git, but our build cache should provide an equivalent performance improvement.

To get more detailed:

We have a derivation like this:

def foo():
    build_go_mod(
        go_version="1.13",
        module_config=files(["go.mod", "go.sum"]),
        project_files=files(["./**/*.go"]),
        bramble_mod_file=file("./go_mod.bramble"),
    )

This would pull all the go files into the build along with go.mod and go.sum. We also pass a special file for the bramble_mod_file. This is the generated file that we're going to output.

To generate the bramble_mod_file we construct a derivation within build_go_mod that might look like this:

def build_go_mod(module_config=[], project_files=[], bramble_mod_file=None):
    # Uses the network, generates the file and then references the file instead
    # of using the network in future situations.
    drv = special_generate_built_in(
        sources=module_config,
        generated_file=bramble_mod_file,
        function="dependencies")

This is all magic at the moment, but the idea is that this derivation uses the network to calculate the dependencies whenever go.mod and go.sum change. From within that derivation we execute similar logic to vgo2nix and create a file that is written to disk at the generated filepath. The value of that file is then loaded dynamically into the build graph.

Once that is set up, then we need to build the binary. Need more detail here later, but the idea is that we use the output of go build -n to create a derivation that outputs deriavations. Each build step in go build -a -n is turned into a derivation.

So we have:

  1. All source files for this project
  2. All source files of all dependencies

When we compile a lib, we want to just look at the lib files and produce an .a file. So on a basic level, we need something that outputs the build graph. When we create derivations that reference the output of the derivation that downloaded the source.

A simple module build section for a specific module looks like this:

#
# github.com/pierrec/lz4/v3
#

mkdir -p $WORK/b305/
cat >$WORK/b305/go_asm.h << 'EOF' # internal
EOF
cd /home/maxm/go/pkg/mod/github.com/pierrec/lz4/v3@v3.3.2
/home/maxm/bramble/bramble_store_padding/bramble_/540ihg6hqyyzysnmxbs1smzb4l6s2dz5-go-1.16.9/share/go/pkg/tool/linux_amd64/asm -p github.com/pierrec/lz4/v3 -trimpath "$WORK/b305=>" -I $WORK/b305/ -I /home/maxm/bramble/bramble_store_padding/bramble_/540ihg6hqyyzysnmxbs1smzb4l6s2dz5-go-1.16.9/share/go/pkg/include -D GOOS_linux -D GOARCH_amd64 -gensymabis -o $WORK/b305/symabis ./decode_amd64.s
cat >$WORK/b305/importcfg << 'EOF' # internal
# import config
packagefile encoding/binary=$WORK/b058/_pkg_.a
packagefile errors=$WORK/b004/_pkg_.a
packagefile fmt=$WORK/b020/_pkg_.a
packagefile github.com/pierrec/lz4/v3/internal/xxh32=$WORK/b306/_pkg_.a
packagefile io=$WORK/b029/_pkg_.a
packagefile io/ioutil=$WORK/b044/_pkg_.a
packagefile math/bits=$WORK/b024/_pkg_.a
packagefile os=$WORK/b030/_pkg_.a
packagefile runtime=$WORK/b008/_pkg_.a
packagefile runtime/debug=$WORK/b160/_pkg_.a
packagefile strings=$WORK/b041/_pkg_.a
packagefile sync=$WORK/b014/_pkg_.a
EOF
cd /home/maxm/go/src/github.com/maxmcd/bramble
/home/maxm/bramble/bramble_store_padding/bramble_/540ihg6hqyyzysnmxbs1smzb4l6s2dz5-go-1.16.9/share/go/pkg/tool/linux_amd64/compile -o $WORK/b305/_pkg_.a -trimpath "$WORK/b305=>" -p github.com/pierrec/lz4/v3 -lang=go1.12 -buildid 5tJja6MFggLI9zzG4-z5/5tJja6MFggLI9zzG4-z5 -goversion go1.16.9 -symabis $WORK/b305/symabis -D "" -importcfg $WORK/b305/importcfg -pack -asmhdr $WORK/b305/go_asm.h -c=4 /home/maxm/go/pkg/mod/github.com/pierrec/lz4/v3@v3.3.2/block.go /home/maxm/go/pkg/mod/github.com/pierrec/lz4/v3@v3.3.2/debug_stub.go /home/maxm/go/pkg/mod/github.com/pierrec/lz4/v3@v3.3.2/decode_amd64.go /home/maxm/go/pkg/mod/github.com/pierrec/lz4/v3@v3.3.2/errors.go /home/maxm/go/pkg/mod/github.com/pierrec/lz4/v3@v3.3.2/lz4.go /home/maxm/go/pkg/mod/github.com/pierrec/lz4/v3@v3.3.2/lz4_go1.10.go /home/maxm/go/pkg/mod/github.com/pierrec/lz4/v3@v3.3.2/reader.go /home/maxm/go/pkg/mod/github.com/pierrec/lz4/v3@v3.3.2/writer.go
cd /home/maxm/go/pkg/mod/github.com/pierrec/lz4/v3@v3.3.2
/home/maxm/bramble/bramble_store_padding/bramble_/540ihg6hqyyzysnmxbs1smzb4l6s2dz5-go-1.16.9/share/go/pkg/tool/linux_amd64/asm -p github.com/pierrec/lz4/v3 -trimpath "$WORK/b305=>" -I $WORK/b305/ -I /home/maxm/bramble/bramble_store_padding/bramble_/540ihg6hqyyzysnmxbs1smzb4l6s2dz5-go-1.16.9/share/go/pkg/include -D GOOS_linux -D GOARCH_amd64 -o $WORK/b305/decode_amd64.o ./decode_amd64.s
/home/maxm/bramble/bramble_store_padding/bramble_/540ihg6hqyyzysnmxbs1smzb4l6s2dz5-go-1.16.9/share/go/pkg/tool/linux_amd64/pack r $WORK/b305/_pkg_.a $WORK/b305/decode_amd64.o # internal
/home/maxm/bramble/bramble_store_padding/bramble_/540ihg6hqyyzysnmxbs1smzb4l6s2dz5-go-1.16.9/share/go/pkg/tool/linux_amd64/buildid -w $WORK/b305/_pkg_.a # internal

A few notes:

  1. cd /home/maxm/go/src/github.com/maxmcd/bramble is very confusing. This means module compilation state depends on the source files of the end-module being compiled. This cd is passed to the compile command with the -D "" flag. This means: -D path: set relative path for local imports (from go tool compile). So maybe this is fine, I think local imports aren't allowed in module compilation. Ah, confirmed. So local imports are relative to the source directory, which in this case is maxmcd/bramble. Strange, seems safe to replace cd /home/maxm/go/src/github.com/maxmcd/bramble with cd . or similar.
  2. When set up in bramble, we'll be symlinking all of the module source locations into a synthetic directory that is pretending to be the module store or the GOPATH. So directories above like /home/maxm/go/pkg/mod/github.com/pierrec/lz4/ will be pointed to a temporary directory. We could readlink -f our way to converting those to their real location in the bramble store.
  3. gcc is sometimes called out to, would need to have gcc (and any other programs that are called) available in the build context.
  4. $WORK/b305/ is the temporary directory for this specific build. It will be referenced later by other builds.

So if we took the chunk above and did a lot of replacing it could end up like this:

#
# github.com/pierrec/lz4/v3
#

mkdir -p $OUTPUT_b305/
cat >$OUTPUT_b305/go_asm.h << 'EOF' # internal
EOF
cd /home/maxm/bramble/bramble_store_padding/bramble_/icpfggjznz3jxnctxtcky55g7zhbsk4u
/home/maxm/bramble/bramble_store_padding/bramble_/540ihg6hqyyzysnmxbs1smzb4l6s2dz5-go-1.16.9/share/go/pkg/tool/linux_amd64/asm -p github.com/pierrec/lz4/v3 -trimpath "$OUTPUT_b305=>" -I $OUTPUT_b305/ -I /home/maxm/bramble/bramble_store_padding/bramble_/540ihg6hqyyzysnmxbs1smzb4l6s2dz5-go-1.16.9/share/go/pkg/include -D GOOS_linux -D GOARCH_amd64 -gensymabis -o $OUTPUT_b305/symabis ./decode_amd64.s
cat >$OUTPUT_b305/importcfg << 'EOF' # internal
# import config
packagefile encoding/binary=$OUTPUT_b058/_pkg_.a
packagefile errors=$OUTPUT_b004/_pkg_.a
packagefile fmt=$OUTPUT_b020/_pkg_.a
packagefile github.com/pierrec/lz4/v3/internal/xxh32=$OUTPUT_b306/_pkg_.a
packagefile io=$OUTPUT_b029/_pkg_.a
packagefile io/ioutil=$OUTPUT_b044/_pkg_.a
packagefile math/bits=$OUTPUT_b024/_pkg_.a
packagefile os=$OUTPUT_b030/_pkg_.a
packagefile runtime=$OUTPUT_b008/_pkg_.a
packagefile runtime/debug=$OUTPUT_b160/_pkg_.a
packagefile strings=$OUTPUT_b041/_pkg_.a
packagefile sync=$OUTPUT_b014/_pkg_.a
EOF
# cd /home/maxm/go/src/github.com/maxmcd/bramble
/home/maxm/bramble/bramble_store_padding/bramble_/540ihg6hqyyzysnmxbs1smzb4l6s2dz5-go-1.16.9/share/go/pkg/tool/linux_amd64/compile -o $OUTPUT_b305/_pkg_.a -trimpath "$OUTPUT_b305=>" -p github.com/pierrec/lz4/v3 -lang=go1.12 -buildid 5tJja6MFggLI9zzG4-z5/5tJja6MFggLI9zzG4-z5 -goversion go1.16.9 -symabis $OUTPUT_b305/symabis -D "" -importcfg $OUTPUT_b305/importcfg -pack -asmhdr $OUTPUT_b305/go_asm.h -c=4 /home/maxm/bramble/bramble_store_padding/bramble_/icpfggjznz3jxnctxtcky55g7zhbsk4u/block.go /home/maxm/bramble/bramble_store_padding/bramble_/icpfggjznz3jxnctxtcky55g7zhbsk4u/debug_stub.go /home/maxm/bramble/bramble_store_padding/bramble_/icpfggjznz3jxnctxtcky55g7zhbsk4u/decode_amd64.go /home/maxm/bramble/bramble_store_padding/bramble_/icpfggjznz3jxnctxtcky55g7zhbsk4u/errors.go /home/maxm/bramble/bramble_store_padding/bramble_/icpfggjznz3jxnctxtcky55g7zhbsk4u/lz4.go /home/maxm/bramble/bramble_store_padding/bramble_/icpfggjznz3jxnctxtcky55g7zhbsk4u/lz4_go1.10.go /home/maxm/bramble/bramble_store_padding/bramble_/icpfggjznz3jxnctxtcky55g7zhbsk4u/reader.go /home/maxm/bramble/bramble_store_padding/bramble_/icpfggjznz3jxnctxtcky55g7zhbsk4u/writer.go
cd /home/maxm/bramble/bramble_store_padding/bramble_/icpfggjznz3jxnctxtcky55g7zhbsk4u
/home/maxm/bramble/bramble_store_padding/bramble_/540ihg6hqyyzysnmxbs1smzb4l6s2dz5-go-1.16.9/share/go/pkg/tool/linux_amd64/asm -p github.com/pierrec/lz4/v3 -trimpath "$OUTPUT_b305=>" -I $OUTPUT_b305/ -I /home/maxm/bramble/bramble_store_padding/bramble_/540ihg6hqyyzysnmxbs1smzb4l6s2dz5-go-1.16.9/share/go/pkg/include -D GOOS_linux -D GOARCH_amd64 -o $OUTPUT_b305/decode_amd64.o ./decode_amd64.s
/home/maxm/bramble/bramble_store_padding/bramble_/540ihg6hqyyzysnmxbs1smzb4l6s2dz5-go-1.16.9/share/go/pkg/tool/linux_amd64/pack r $OUTPUT_b305/_pkg_.a $OUTPUT_b305/decode_amd64.o # internal
/home/maxm/bramble/bramble_store_padding/bramble_/540ihg6hqyyzysnmxbs1smzb4l6s2dz5-go-1.16.9/share/go/pkg/tool/linux_amd64/buildid -w $OUTPUT_b305/_pkg_.a # internal

Changes:

  1. We've changed /home/maxm/go/pkg/mod/github.com/pierrec/lz4/v3@v3.3.2 to a bramble store path. Presumably those files would live in the bramble store for us.
  2. Commented out the cd to the project directory.
  3. Replaced $WORK/b014 with $OUTPUT_b014, if we decided on a general reference format like this we could pass this script as input to a bramble-post processing script that would turn this into a derivation graph. The output environment variable would be replaced with the correct path in the bramble store. This part is important because the work directory names are effectively random. They could change between different invocations of go build -a -n, so we need to normalize them before they get into the derivation body.
  4. We also need to swap out the filepath references with references to the source derivations.

This could be the whole enchilada. Seems the combination of file generation and graph patching work well.