Build instructions

[dimakuv]

Prerequisites

OS distro used: Ubuntu 22.04.

• Recompile Linux kernel with required configs:

# on some OS distros, the Linux config doesn't have FS and VSOCK by default
CONFIG_VIRTIO_FS=y
CONFIG_VSOCKETS=y

# choose the rest of the configs during `make`, including CONFIG_VHOST_VSOCK=y

• Install latest virtiofsd:

sudo apt install libcap-ng-dev libseccomp-dev

git clone https://gitlab.com/virtio-fs/virtiofsd.git
cargo build --release
sudo cp target/release/virtiofsd /usr/local/bin/

• Install latest socat (at least v1.7.4):

wget http://www.dest-unreach.org/socat/download/socat-1.8.0.0.tar.gz
tar xzf socat-1.8.0.0.tar.gz
cd socat-1.8.0.0/
./configure
make
sudo make install

• Allow vsock driver for everyone (hack, don't use in prod):

sudo modprobe vhost_vsock
sudo chmod 666 /dev/vhost-vsock

• Get TDX software stack.

• Install Rust nightly-2023-08-28, required for TD-Shim. Also install NASM, Clang and LLVM (Ubuntu 22.04 has sufficiently new ones.)

• Get latest TD-Shim and build it:

cargo xbuild -p td-shim --target x86_64-unknown-none --release --features=main,tdx --no-default-features

# the below builds the Example payload, just for testing
cargo xbuild -p td-payload --target x86_64-unknown-none --release --bin example --features=tdx,start,cet-shstk,stack-guard
cargo run -p td-shim-tools --bin td-shim-ld -- target/x86_64-unknown-none/release/ResetVector.bin target/x86_64-unknown-none/release/td-shim -t executable -p target/x86_64-unknown-none/release/example -o target/release/final-elf.bin

# test the Example payload
/usr/bin/qemu-system-x86_64 -no-reboot -name debug-threads=on -enable-kvm -smp 1,sockets=1 -object tdx-guest,id=tdx,debug=on -machine q35,accel=kvm,kernel_irqchip=split,confidential-guest-support=tdx -no-hpet -cpu host,host-phys-bits,+invtsc -m 2G -nographic -vga none -bios target/release/final-elf.bin

# on Canonical Ubuntu 23.10 TDX, the QEMU args are different
#   see https://github.com/canonical/tdx/blob/mantic-23.10/guest-tools/run_td.sh
#   and https://github.com/canonical/tdx/releases/tag/1.2
/usr/bin/qemu-system-x86_64 -no-reboot -accel kvm -smp 1 -object tdx-guest,id=tdx -machine q35,kernel_irqchip=split,hpet=off,memory-encryption=tdx,memory-backend=ram1 -object memory-backend-ram,id=ram1,size=2G,private=on -cpu host,host-phys-bits,+invtsc -m 2G -nographic -vga none -bios target/release/final-elf.bin

Building Gramine-VM/TDX

$ meson setup build-debug/ --werror --buildtype=debug -Dtests=enabled \
    -Dskeleton=enabled -Ddirect=enabled -Dsgx=enabled -Dvm=enabled -Dtdx=enabled \
    --prefix=$PWD/built-debug
$ ninja -C build-debug/
$ ninja -C build-debug/ install

Running Gramine-VM/TDX

Gramine-VM/TDX reuses completely the direct/sgx manifests, Makefiles and etc. So LibOS tests, Examples, CI-Examples should more or less run as is (one may need to adjust sgx.enclave_size since VMs require more space).

$ cd CI-Examples/helloworld
$ make SGX=1

$ gramine-vm helloworld
Hello world!

$ gramine-tdx helloworld
Hello world!

Note: currently for network-based applications, we must manually start socat daemon that will proxy VSOCK<->TCP. See examples like Redis and Memcached on how to run.

Random notes

• Under the hood, gramine-vm/gramine-tdx script starts this:

  virtiofsd --socket-path /tmp/vhostfs --shared-dir / --log-level error --sandbox none &
  

• Glibc must be built without Gramine-syscall patches; for this, my branch has the logic to skip the patches when Dvm=enabled or Dtdx=enabled is set (technically, the same should be done for Musl, GCC/libomp, and others)
• QEMU propagates PWD into Gramine's VM, so that Gramine can construct proper absolute filenames.

[dimakuv]

On default Ubuntu 24.04

Linux config on Ubuntu 24.04 already comes with virtio-vsock and virtio-fs, though as manually loadable modules, so:

sudo modprobe virtiofs
sudo modprobe vhost-vsock

[dimakuv]

On Canonical Ubuntu 23.10 TDX

This repo: https://github.com/canonical/tdx/blob/mantic-23.10/

Linux config on Ubuntu 23.10 already comes with virtio-vsock and virtio-fs, though as manually loadable modules, so:

sudo modprobe virtiofs
sudo modprobe vhost-vsock

We can use the latest release of TD-Shim (v0.8.0). Build as usual, but be aware -- I had to install the latest Rust nightly, otherwise some dependencies didn't build. Also, in case of build errors, it's simpler to just purge the whole TD-Shim repo and re-create it and re-build (I had some issues that looked like left-overs from previous failing builds).

This OS distro uses a patched QEMU v8.0.4 (to support TDX), and things changed a bit with respect to QEMU's TDX-related command line options. So we must modify gramine-vm.in template script. In particular:

diff --git a/tools/gramine-vm.in b/tools/gramine-vm.in
@@ -91,9 +91,9 @@ QEMU_CPU_NUM=${QEMU_CPU_NUM:-"1"}
 QEMU_PATH="qemu"
 QEMU_VM="-cpu host,host-phys-bits,-kvm-steal-time,pmu=off,+tsc-deadline,+invtsc \
     -m $QEMU_MEM_SIZE -smp $QEMU_CPU_NUM"
-QEMU_OPTS="-enable-kvm -vga none -display none -no-reboot -monitor none -machine hpet=off \
-    -object memory-backend-file,id=mem,size=$QEMU_MEM_SIZE,mem-path=/dev/shm,share=on \
-    -numa node,memdev=mem"
+QEMU_OPTS="-enable-kvm -vga none -display none -no-reboot -monitor none \
+    -object memory-backend-memfd,id=mem,size=$QEMU_MEM_SIZE,private=on \
+    -M memory-backend=mem,hpet=off"

 if [ "$TDSHIM_PAL_PATH" == "" ]; then
 QEMU_MACHINE="-M q35,kernel_irqchip=split"

Interestingly, virtiofsd seems to require -ojbect memory-backend-memfd,...,share=on, as told by their README. But this conflicts with QEMU TDX requirements which only work with ...,private=on, see e.g. this script as example. Anyway, using private=on seems to satisfy both virtiofsd and TDX VM.

Note on testing TD-Shim

The TD-Shim documentation on GitHub doesn't mention how to quickly test the example ELF payload on modern Ubuntu + QEMU. Here's the working command on Ubuntu 23.10 TDX and QEMU v8.0.4:

/usr/bin/qemu-system-x86_64 -no-reboot -accel kvm -smp 1 \
  -machine q35,kernel_irqchip=split,hpet=off,memory-encryption=tdx,memory-backend=ram1 \
  -object tdx-guest,id=tdx \
  -object memory-backend-ram,id=ram1,size=2G,private=on \
  -cpu host -m 2G -nographic -vga none \
  -bios target/release/final-elf.bin

[ya0guang]

Hi,

Thanks a lot for publishing this great project and this manual. I'm not sure if gramine-tdx can support Ubuntu 24.04, but I would like to try it on Ubuntu 24.04 as I can not change the OS version.

When I tried to compile gramine-tdx on Ubuntu 24.04, I experienced an error related to protobuf.

 ninja -C build-debug/
ninja: Entering directory `build-debug/'
[71/824] Generating common/proto/aesm.pb-c.[ch] with a custom command
FAILED: common/proto/aesm.pb-c.c 
/usr/bin/protoc-c --c_out=common/proto -I/home/user1/Artifacts/gramine-tdx/common/proto /home/user1/Artifacts/gramine-tdx/common/proto/aesm.proto
/usr/bin/protoc-c: symbol lookup error: /usr/bin/protoc-c: undefined symbol: _ZN6google8protobuf8internal16InternalMetadata15CheckedDestructEv
[183/824] Compiling C object pal/regression/libtest_pal.a.p/meson-genera....._subprojects_tomlc99-208203af46bdbdb29ba199660ed78d09c220b6c5_toml.c.o^C^C
ninja: build stopped: interrupted by user.

Protobuf versions:

❯ apt list --installed | grep protobuf

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

libprotobuf-c-dev/noble,now 1.4.1-1ubuntu4 amd64 [installed]
libprotobuf-c1/noble,now 1.4.1-1ubuntu4 amd64 [installed,automatic]
libprotobuf-dev/noble,now 3.21.12-8.2build1 amd64 [installed,automatic]
libprotobuf-lite32t64/noble,now 3.21.12-8.2build1 amd64 [installed,automatic]
libprotobuf32t64/noble,now 3.21.12-8.2build1 amd64 [installed,automatic]
protobuf-c-compiler/noble,now 1.4.1-1ubuntu4 amd64 [installed]
protobuf-compiler/noble,now 3.21.12-8.2build1 amd64 [installed]
python3-protobuf/noble,now 3.21.12-8.2build1 amd64 [installed]

Any insightful ideas are welcome! @StanPlatinum