ebpf-traffic-tap

This is a small example project of an eBPF-based traffic tap for Kubernetes environments.

Architecture Overview

Overall architecture of the eBPF traffic tapping solution:

Note that this repository is providing only the "tcptracker" box of that picture, which is composed by an USERSPACE portion and a KERNELSPACE portion (the eBPF program). Some eBPF architectural keypoints:

• Check Chapter 7 of "Learning eBPF" book on why syscall kprobes have been selected (opposed to other possibilities like "normal" kprobes, XDP, tracepoints, etc): syscall kprobes are very stable across different kernel versions. This makes this project as flexible as possible. Also note that the Fentry/Fexit techinque would be more efficient than syscall kprobes: the fexit hook has access to the input parameters to the function. However fentry/fexit are available only for very recent kernels.
• When data is moved from the tcptracker USERSPACE container/POD to the outside world, most likely it will need to be encrypted to comply with security policies.
• Correctly tapping an arbitrary network program requires knowledge of all possible syscalls it uses to send data out (or having the eBPF kernel program attaching by default to all interesting syscalls).
• Tapping only the desired traffic will probably require some strong filtering capability hardwired directly into the eBPF kernel program (TODO).

OS Requirements

• Linux with Kernel 4.18 or higher with BTF enabled (e.g. RH8 or OL8 or higher). BTF became enabled by default on most Linux distributions with kernel 5.14 or higher. You can check if your kernel has BTF enabled by verifying if /sys/kernel/btf/vmlinux exists on your system.
• eBPF enabled in the host
• Right privileges to install eBPF kernel programs are: CAP_PERFMON+CAP_BPF (for eBPF) and CAP_NET_ADMIN+CAP_BPF (for XDP)

Demo

To test locally the eBPF traffic tap, you can follow these steps:

git clone https://github.com/f18m/ebpf-tcptracker.git && cd ebpf-tcptracker

cd ebpf-userspace-program && goreleaser build --clean --snapshot && docker build . -t ebpf-tcptracker/userspace-agent:latest ; cd ..
cd demo/http-traffic-server/ && docker build . -t http-traffic-server:latest ; cd ../..
cd demo/http-traffic-client/ && docker build . -t http-traffic-client:latest ; cd ../..

docker compose -f demo/docker-compose.yaml up

This procedure requires you to have

• golang tools on your machine (golang >= 1.21)
• Goreleaser tool installed, see this page
• docker
• docker-compose
• Linux kernel new enough (see the section 'OS Requirements')

References

eBPF CO-RE documentation links

eBPF CO-RE are the most-recent "way" (as of Nov 2023) to write eBPF kernel programs. They have been introduced by Andrii Nakryiko at the end of 2021, see https://nakryiko.com/posts/bpf-core-reference-guide/. A good guide on eBPF CO-RE is also Chapter 5 of "Learning eBPF" book by Liz Rice.

Main open-source projects using eBPF CO-RE syntax

• https://github.com/parca-dev/parca-agent : Parca Agent (see 'bpf' folder) for Parca continuous profiler
• https://github.com/grafana/beyla : Beyla

Some eBPF resources that have not yet migrated to the CO-RE syntax

• https://ebpf.io/labs/ : Isovalent hands-on labs
• https://github.com/iovisor/bcc : BCC is using Python, see https://www.brendangregg.com/blog/2020-11-04/bpf-co-re-btf-libbpf.html

Notes for future implementation

• If we can accept as minimal requirement the kernel 5.8, the "BPF ring buffers" can be used as replacement for the "BPF perf buffers" (see Chapter 4 of "Learning eBPF")
• Consider switching to Rust and see https://www.deepfence.io/blog/aya-your-trusty-ebpf-companion : Aya library to write everything in Rust