Gthulhu Project

Overview

Gthulhu aims to provide an orchestrable, distributed scheduler solution for the cloud-native ecosystem to meet the dynamic and diverse needs of cloud-native applications, such as:

• Trading systems requiring low-latency processing capabilities
• Big data analytics requiring high-throughput computing resources
• Machine learning tasks requiring flexible resource allocation

The default Linux Kernel scheduler emphasizes fairness and cannot be optimized for the specific needs of different applications. Moreover, when these applications run in a distributed architecture, traditional schedulers often fail to effectively coordinate and allocate resources, leading to performance bottlenecks and resource waste.

┌─────────────────────────────────────────────────────────────────────────────────┐
│                              Gthulhu Architecture                               │
├─────────────────────────────────────────────────────────────────────────────────┤
│                                                                                 │
│   ┌─────────────┐         ┌─────────────────────┐         ┌─────────────────┐   │
│   │    User     │ ──────▶ │      Manager        │ ──────▶ │    MongoDB      │   │
│   │  (Web UI)   │         │ (Central Management)│         │  (Persistence)  │   │
│   └─────────────┘         └──────────┬──────────┘         └─────────────────┘   │
│                                      │                                          │
│                                      │ Query Pods via K8s API                   │
│                                      ▼                                          │
│                           ┌─────────────────────┐                               │
│                           │   Kubernetes API    │                               │
│                           │   (Pod Informer)    │                               │
│                           └─────────────────────┘                               │
│                                      │                                          │
│              ┌───────────────────────┼───────────────────────┐                  │
│              │                       │                       │                  │
│              ▼                       ▼                       ▼                  │
│   ┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐           │
│   │ Decision Maker  │     │ Decision Maker  │     │ Decision Maker  │           │
│   │   (Node 1)      │     │   (Node 2)      │     │   (Node N)      │           │
│   └────────┬────────┘     └────────┬────────┘     └────────┬────────┘           │
│            │                       │                       │                    │
│            ▼                       ▼                       ▼                    │
│   ┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐           │
│   │  sched_ext      │     │  sched_ext      │     │  sched_ext      │           │
│   │ (eBPF Scheduler)│     │ (eBPF Scheduler)│     │ (eBPF Scheduler)│           │
│   └─────────────────┘     └─────────────────┘     └─────────────────┘           │
│                                                                                 │
└─────────────────────────────────────────────────────────────────────────────────┘

Gthulhu allows users to configure scheduling strategies via REST APIs. The Gthulhu Control Plane (https://github.com/Gthulhu/api) then distributes these strategies to the Gthulhu Scheduler on each node, enabling cross-node coordinated scheduling and resource management.

Here is important information about Gthulhu:

• Gthulhu is a scheduler developed using eBPF and Golang, with core components defined in the qumun framework, which allows you to develop custom Linux schedulers in Go without modifying the kernel directly—leveraging the eBPF/sched_ext mechanism to implement customized scheduling policies. This is also the technical foundation that enables Gthulhu to provide an orchestrable, distributed scheduler.
• Gthulhu provides a helm chart for easy deployment on Kubernetes clusters, which includes the tech stack that Gthulhu depends on.
• For real-world use cases of Gthulhu, please refer to Improving Network Performance with Custom eBPF-based Schedulers.

Meaning of the Name

The name Gthulhu is inspired by Cthulhu, a mythical creature known for its many tentacles. Just as tentacles can grasp and steer, Gthulhu symbolizes the ability to take the helm and navigate the complex world of modern distributed systems — much like how Kubernetes uses a ship’s wheel as its emblem.

The prefix “G” comes from Golang, the language at the core of this project, highlighting both its technical foundation and its developer-friendly design.

Underneath, Gthulhu runs on the qumun framework (qumun means “heart” in the Bunun language, an Indigenous people of Taiwan), reflecting the role of a scheduler as the beating heart of the operating system. This not only emphasizes its central importance in orchestrating workloads but also shares a piece of Taiwan’s Indigenous culture with the global open-source community.

DEMO

Click the image below to see our DEMO on YouTube!

Prerequisites

To build and run Gthulhu, ensure you have the following prerequisites installed on your system

• Go 1.22+
• LLVM/Clang 17+
• libbpf
• Linux kernel 6.12+ with sched_ext support

See Installation guide for detailed installation instructions.

Usage

Setting Up Dependencies

First, clone the required dependencies:

make dep
git submodule init
git submodule sync
git submodule update
cd scx
cargo build --release -p scx_rustland

This will clone libbpf and the custom libbpfgo fork needed for the project.

Linting the Code

To ensure code quality, run the linter:

make lint

Building the Scheduler

Build the scheduler with:

make build

This compiles the BPF program, builds libbpf, generates the skeleton, and builds the Go application.

Cross-compilation for arm64 is supported by setting the ARCH variable:

make build ARCH=arm64

Testing the Scheduler

To test the scheduler in a virtual environment using kernel v6.12.2:

make test

This uses vng (virtual kernel playground) to run the scheduler with the appropriate kernel version.

You can also test with a specific kernel version:

make test KERNEL_VERSION=6.13

Portability Testing

Gthulhu is automatically tested for portability across multiple Linux kernel versions (6.12+) through a daily scheduled GitHub Actions workflow. This ensures that the released packages remain compatible with newer kernel versions. The portability tests run against kernel versions 6.12, 6.13, 6.14, 6.15, 6.16, and 6.17.

Launching Gthulhu by using schedkit

First, install schedctl from schedkit (created by @dottorblaster):

$ git clone https://github.com/schedkit/schedctl.git
$ cd schedctl
$ make install

Then, you can launch Gthulhu with:

$ sudo ./schedctl run gthulhu
Trying to pull ghcr.io/schedkit/gthulhu:latest...
Getting image source signatures
Copying blob sha256:a517a5a43837a7785dad62f579950a8abe4d1bd2ae5a096bda78150a1cc70c64
Copying blob sha256:2d35ebdb57d9971fea0cac1582aa78935adf8058b2cc32db163c98822e5dfa1b
Copying config sha256:85fb26cbfed25f79e321ef4e0a3e4e6fc7f01a957dbcee6aed815bdb93458136
Writing manifest to image destination
Storing signatures
Container ghcr.io/schedkit/gthulhu:latest started successfully
$ sudo podman ps -a
CONTAINER ID  IMAGE                            COMMAND     CREATED        STATUS            PORTS       NAMES
015d1b5fbe96  ghcr.io/schedkit/gthulhu:latest  /main       6 seconds ago  Up 6 seconds ago              gthulhu

Running with Docker

To run the scheduler in a Docker container, you can either build locally or use the pre-built image from GitHub Container Registry:

Using the pre-built image from GitHub Packages:

docker run --privileged=true --pid host --rm ghcr.io/gthulhu/gthulhu:latest /gthulhu/main

Building locally:

make image
docker run --privileged=true --pid host --rm  127.0.0.1:25000/gthulhu:latest /gthulhu/main

Debugging

If you need to inspect the BPF components, you can use:

sudo bpftool prog list            # List loaded BPF programs
sudo bpftool map list             # List BPF maps
sudo cat /sys/kernel/debug/tracing/trace_pipe # View BPF trace output

Stress Testing by using stress-ng

stress-ng -c 20 --timeout 20s --metrics-brief

Troubleshooting

See Installation guide.

License

This software is distributed under the terms of the Apache License 2.0.

Contributing

See Contributing guide.

Community Resources

• NotebookLM: includes all of materials used in the project, including the pptx, design documents, and more.
• GitHub Discussion: a place for community discussions, questions, and feature requests.

Star History

Special Thanks

• scx
• libbpfgo