WIO - Whole-In-One to Go

WIO (Whole-In-One to Go) allows you to develop subcommands in Git way still keeping them in a single binary by embedding executables in a binary

Introduction

You know Git, right? Git implements variety of subcommends in a simple and extensible way, in that you can implement your own git foo subcommand only by placing an executable named git-foo in your $PATH. One of the benefit of this way is that you can use any language you like to implement your own Git subcommands. And actually, some of Git commands are implemented in Shellscript or Perl.

On the other hand, one of the beauty of Go is a single binary, which is portable and easy to distribute, deploy and execute.

I dreamed the combination of the flexibility of Git and the simplicity of Go. That's where WIO was born.

Installation

Clone Git repository and run the build script.

For Linux and macOS:

$ git clone github.com/Maki-Daisuke/go-whole-in-one
$ cd go-whole-in-one/cmd/wio
$ ./make.sh install

For Windows:

$ git clone github.com/Maki-Daisuke/go-whole-in-one
$ cd go-whole-in-one\cmd\wio
$ .\make.ps1 install

How to Use

First of all, make your project directory:

$ mkdir mycmd
$ cd mycmd

Then, init your project:

$ wio init

This command creates the following three files:

$ ls
main.go  pack.go  packing-list

Build and run it:

$ go build
$ ./mycmd
usage: mycmd [--version] [--help] <subcommand> <args>

It works! But, nothing interesting happens, yet.

Now, implement your first subcommand in Shellscript:

$ cat <<EOS > mycmd-hello
#!/bin/sh
echo 'Hello, world!'
EOS
$ chmod +x mycmd-hello

Then, build it using wio build, and here, do not forget to add the path to your subcommand, that is the current working directory in this case, into PATH environment variable:

$ PATH=`pwd`:$PATH wio build
$ ./mycmd hello
Hello, world!

You made your own command having a subcommand. Hooray!

You can copy it to another machine:

$ scp ./mycmd another-machine:~
$ ssh another-machine
another-machine> ./mycmd hello
Hello, world!

It works! You don't need to copy mycmd-hello, because it's just a single binary executable.

You can easily add more subcommands. Let's add one more:

$ cat <<EOS > mycmd-jsonpp
#!/bin/sh

jq . $*
EOS
$ chmod +x mycmd-jsonpp

You made it! But wait! This shellscript does not work on machines without jq since it depends on it. Don't worry. You can embed jq by adding it in your packing-list:

$ echo jq >> packing-list
$ cat packing-list
# This file was generated by wio-init.
mycmd-*
jq

Let's build it and use it on the other machine:

$ PATH=`pwd`:$PATH wio build
$ scp ./mycmd another-machine:~
$ ssh another-machine
another-machine> ./mycmd jsonpp
jq: error while loading shared libraries: libonig.so.2: cannot open shared object file: No such file or directory

Oops! It doesn't work... Ok, you need Oniguruma to run jq. Let's add it into the packing-list:

$ echo /usr/lib/x86_64-linux-gnu/libonig.so.2 >> packing-list
$ PATH=`pwd`:$PATH wio build
$ scp ./mycmd another-machine:~
$ ssh another-machine
another-machine> echo '{"my":"cmd","hello":"world"}'  |./mycmd jsonpp
{
  "my": "cmd",
  "hello": "world"
}

It works, hooray!

As you see, you can embed any kind of file you want, and make self-contained binary executables.

More Examples

In fact, wio command itself is a WIO-application.

See its source code for an example.

Command-Line Interface

wio command has three subcommands as follows. The all subcommands are intended to be called in the your project directory, that is, where main package resides.

wio init

Initializes a new WIO application project with boilerplates. As a result, it creates tree files: main.go, pack.go and packing-list.

wio generate

Generates and/or updates pack.go, which all embedded files reside in, in regarding to paking-list.

wio build

Shortcut for wio-generate && go build.

Packing-list

You can embed any file by adding in packing-list. wio generate reads packing-list, compresses listed files, and then embeds them into your application code.

Syntax of packing-list

• Each line shows one item
  • That is, a file to embed
• # begins comment
  • Text from # to end of line is ignored
• Wrapping white characters are ignored
  • Thus, the line foo bar # comment is equivalent to foo bar
• Look up commands by default
  • The line foo means, wio generate looks up a command named foo in your PATH
• If a line contains / or platform dependent path-separator, it is regarded as file path instead of command
  • You need to prepend ./ when you embed a file in the current directory
  • E.g. ./not-command
• You can use file glob like *, ? and [abc]
  • That indicates to embed all matched files
  • Wildcards can be used for commands. That means, to look up all commands matching glob from PATH, and embed all of them.

How it works

Essentially, it has three phases: Generate -> Deploy -> Execute

Generate

Generate phase is processed by wio generate command. It lists up files specified by packing-list and compresses them and generate Go source code which contains compressed data embeded. You need to run wio generate in your project directory when files you want to embed are changed.

You can run go generate instead of wio generate. It essentially does the same thing, but go generate && go build is more common idiom and handy when you use other generate-compatible packages.

Deploy

This phase is kicked when a WIO application is executed first time. It decompresses and unpack embedded files into a cache directory under system's temporary directory (which is usually /tmp on Linux).

This process is executed only once per user. Each user has her/his own cache directory, which can be read only by the user who executed the command. In other words, users who cannot run the command cannot read/write files embedded in the command. That keeps your confidentaial data secure.

Execute

Finally, this phase executes your command. But, before executing the command, it prepares execution environment. More specific, it sets the following three environment variables:

• WIOPATH
  • This environment variable holds path to cache firectory in which WIO unpacks embedded files.
  • This allows subcommands to access embedded files during execution.
• PATH
  • WIO prepends path to cache directory at the head of PATH environement variable.
  • In other words, when you invoke a command in subcommands, it will search command executable in the cache directory at first.
• LD_LIBRARY_PATH / DYLD_LIBRARY_PATH
  • This allows subcommands dynamically link shared libraries embedded.
  • In cotrast to PATH, WIO appends cache directory at the tail of LD_LIBRARY_PATH. That means, libraries you specify take a priority.
  • On macOS (Dawrin), DYLD_LIBRARY_PATH is set instead of LD_LIBRARY_PATH.

After setting those environment variables, it looks up approapriate subcommand and executes it.

CAVEAT

Despite WIO sets DYLD_LIBRARY_PATH on macOS (Darwin), it does not work as expected in most environments, unfortunately. Because of security concerns, DYLD_LIBRARY_PATH is ignored by SIP-protected binaries including /bin/sh. And, SIP (System Integrity Protection) is enebled by default on El Capitan and newer. Thus, the example above to use jq does not work on macOS.

However, you can explicitly set DYLD_LIBRARY_PATH by yourself like this:

#!/bin/sh

export DYLD_LIBRARY_PATH=$WIOPATH
jq . $*

This works as you expect.

Built-in Subcommands

Sometimes, you want to implement subcommands in Go and integrate them in your main command, because it's less overhead. You can do it by implementing wio.Command interface and registering it as built-in subcommand with wio.Register.

For your convenience, you can use FuncCommand type. For example:

func main(){
  wio.Register("mycmd", wio.FuncCommand(func(subname string, args []string){
    fmt.Println("This is my command!")
  }))
  wio.Exec(os.Args[1:])
}

See API reference for details.

WIO registers the following predefined built-in subcommands by default:

• help, --help, -h
  • Shows simple help message
• version, --version, -v
  • Shows version number

Yes, --help and --version look like options, but they are actually implemented as subcommands.

Of course, you can customize your subcommands as you like by editing your main.go.

API

See Godoc.

License

The Simplified BSD License (2-clause). See LICENSE file also.

Author

Daisuke (yet another) Maki