add new topics (#132)

* add goroutine text

* update

* fix funclatency

* update doc

* update about nginx

* update nginx

* u[date co

* fix compile

.github/workflows/test-eunomia.yaml

@@ -79,3 +79,7 @@ jobs:
       run: |
         ./ecc src/25-signal/signal.bpf.c src/25-signal/signal.h
         sudo timeout -s 2 3 ./ecli run src/25-signal/package.json || if [ $? = 124 ]; then exit 0; else exit $?; fi
+    - name: test 31 goroutine
+      run: |
+          ./ecc src/31-goroutine/goroutine.bpf.c src/31-goroutine/goroutine.h
+          # todo

.github/workflows/test-libbpf.yml

@@ -61,6 +61,10 @@ jobs:
       run: |
         make -C src/30-sslsniff
         sudo timeout -s 2 3 src/30-sslsniff/sslsniff || if [ $? = 124 ]; then exit 0; else exit $?; fi
+
+    - name: test 33 funclatency
+      run: |
+          make -C src/33-funclatency
     - name: test 35-user-ringbuf
       run: |
         make -C src/35-user-ringbuf

src/31-goroutine/README.md

@@ -1,33 +1,112 @@
-# goroutine trace
-
-**UNFINISHED YET**: The offset of goid field is hardcoded. It was only tested on the bundled `go-server-http`. It MAY NOT WORK on other go programs.
-
-The bundled of program was compiled using go 1.17.0. The executable and source could be found at folder `go-server-http`.
-
-This example traces the state switch of goroutines, and prints the corresponding state, goid, pid and tgid.
-
-```console
-root@mnfe-pve:~/bpf-developer-tutorial/src/31-goroutine# ecc goroutine.bpf.c goroutine.h 
-INFO [ecc_rs::bpf_compiler] Compiling bpf object...
-INFO [ecc_rs::bpf_compiler] Generating export types...
-INFO [ecc_rs::bpf_compiler] Generating package json..
-INFO [ecc_rs::bpf_compiler] Packing ebpf object and config into package.json...
-root@mnfe-pve:~/bpf-developer-tutorial/src/31-goroutine# ecli-rs run package.json 
-INFO [faerie::elf] strtab: 0x6fb symtab 0x738 relocs 0x780 sh_offset 0x780
-INFO [bpf_loader_lib::skeleton::preload::section_loader] User didn't specify custom value for variable __eunomia_dummy_goroutine_execute_data_ptr, use the default one in ELF
-TIME     STATE  GOID   PID    TGID   
-INFO [bpf_loader_lib::skeleton] Running ebpf program...
-21:00:47  DEAD(6) 0    2542844 2542844
-21:00:47  RUNNABLE(1) 0 2542844 2542844
-21:00:47  DEAD(6) 0    2542844 2542844
-21:00:47  RUNNING(2) 1 2542844 2542844
-21:00:47  DEAD(6) 0    2542844 2542844
-21:00:47  RUNNABLE(1) 0 2542844 2542844
-21:00:47  RUNNABLE(1) 1 2542844 2542844
-21:00:47  RUNNING(2) 2 2542847 2542844
-21:00:47  WAITING(4) 2 2542847 2542844
-....
+# eBPF 实践教程：使用 eBPF 跟踪 Go 协程状态
+
+Go 是 Google 创建的一种广受欢迎的编程语言，以其强大的并发模型而著称。Go 语言的一个重要特点是协程（goroutine）的使用——这些协程是轻量级、由 Go 运行时管理的线程，使得编写并发程序变得非常简单。然而，在实时环境中理解和跟踪这些协程的执行状态，尤其是在调试复杂系统时，可能会面临很大的挑战。
+
+这时我们可以利用 eBPF（扩展伯克利包过滤器）技术。eBPF 最初设计用于网络数据包过滤，但随着时间的推移，eBPF 已经发展成为一个强大的工具，用于跟踪和监控系统行为。通过使用 eBPF，我们可以深入到内核，收集有关 Go 程序运行时行为的数据，包括协程的状态。本文将探讨如何使用 eBPF 跟踪 Go 程序中的协程状态转换。
+
+## 背景：协程与 eBPF
+
+### 协程
+
+协程是 Go 语言的核心特性之一，它提供了一种简单而高效的并发处理方式。与传统的线程不同，协程由 Go 运行时管理，而不是由操作系统管理，因此更加轻量化。协程可以在以下几种状态之间进行转换：
+
+- **RUNNABLE（可运行）**：协程已准备好运行。
+- **RUNNING（运行中）**：协程正在执行中。
+- **WAITING（等待）**：协程正在等待某个事件（如 I/O 或定时器）。
+- **DEAD（终止）**：协程执行完毕并已终止。
+
+理解这些状态以及协程之间的状态转换对于诊断性能问题、确保 Go 程序的高效运行至关重要。
+
+### eBPF
+
+eBPF 是一种强大的技术，它允许开发人员在不修改内核源代码或加载内核模块的情况下，在 Linux 内核中运行自定义程序。eBPF 最初用于数据包过滤，但现在已扩展为一种多功能工具，广泛应用于性能监控、安全和调试。
+
+通过编写 eBPF 程序，开发人员可以跟踪各种系统事件，包括系统调用、网络事件和进程执行。在本文中，我们将重点介绍如何使用 eBPF 跟踪 Go 程序中协程的状态转换。
+
+## eBPF 内核代码
+
+现在，让我们深入探讨实现该跟踪功能的 eBPF 内核代码。
+
+```c
+#include <vmlinux.h>
+#include "goroutine.h"
+#include <bpf/bpf_core_read.h>
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_tracing.h>
+
+#define GOID_OFFSET 0x98
+
+struct {
+  __uint(type, BPF_MAP_TYPE_RINGBUF);
+  __uint(max_entries, 256 * 1024);
+} rb SEC(".maps");
+
+SEC("uprobe/./go-server-http/main:runtime.casgstatus")
+int uprobe_runtime_casgstatus(struct pt_regs *ctx) {
+  int newval = ctx->cx;
+  void *gp = ctx->ax;
+  struct goroutine_execute_data *data;
+  u64 goid;
+  if (bpf_probe_read_user(&goid, sizeof(goid), gp + GOID_OFFSET) == 0) {
+    data = bpf_ringbuf_reserve(&rb, sizeof(*data), 0);
+    if (data) {
+      u64 pid_tgid = bpf_get_current_pid_tgid();
+      data->pid = pid_tgid;
+      data->tgid = pid_tgid >> 32;
+      data->goid = goid;
+      data->state = newval;
+      bpf_ringbuf_submit(data, 0);
+    }
+  }
+  return 0;
+}
+
+char LICENSE[] SEC("license") = "GPL";
 ```
 
+1. **头文件**：代码首先包含了必要的头文件，如 `vmlinux.h`（提供内核定义）和 `bpf_helpers.h`（提供 eBPF 程序的辅助函数）。
+2. **GOID_OFFSET**：`goid` 字段的偏移量被硬编码为 `0x98`，这是特定于所跟踪的 Go 版本和程序的。此偏移量在不同的 Go 版本或程序中可能有所不同。
+3. **环形缓冲区映射**：定义了一个 BPF 环形缓冲区映射，用于存储协程的执行数据。这个缓冲区允许内核高效地将信息传递到用户空间。
+4. **Uprobe**：该 eBPF 程序的核心是一个附加到 Go 程序中 `runtime.casgstatus` 函数的 uprobe（用户级探针）。该函数负责改变协程的状态，因此非常适合用来拦截和跟踪状态转换。
+5. **读取协程 ID**：`bpf_probe_read_user` 函数从用户空间内存中读取协程 ID（`goid`），使用的是预定义的偏移量。
+6. **提交数据**：如果成功读取了协程 ID，则数据会与进程 ID、线程组 ID 以及协程的新状态一起存储在环形缓冲区中。随后，这些数据会提交到用户空间以供分析。
+
+## 运行程序
+
+要运行此跟踪程序，请按照以下步骤操作：
+
+1. **编译 eBPF 代码**：使用类似 `ecc`（eBPF 编译集合）这样的编译器编译 eBPF 程序，并生成一个可以由 eBPF 加载器加载的包。
+
+    ```bash
+    ecc goroutine.bpf.c goroutine.h
+    ```
+
+2. **运行 eBPF 程序**：使用 eBPF 加载器运行编译后的 eBPF 程序。
+
+    ```bash
+    ecli-rs run package.json
+    ```
+
+3. **输出**：程序将输出协程的状态转换及其 `goid`、`pid` 和 `tgid`。以下是一个示例输出：
+
+    ```console
+    TIME     STATE       GOID   PID    TGID   
+    21:00:47 DEAD(6)     0      2542844 2542844
+    21:00:47 RUNNABLE(1) 0      2542844 2542844
+    21:00:47 RUNNING(2)  1      2542844 2542844
+    21:00:47 WAITING(4)  2      2542847 2542844
+    ```
+
+完整代码可以在 <https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/31-goroutine> 找到。
+
+如果你想了解更多关于 eBPF 的知识和实践，你可以访问我们的教程代码库 <https://github.com/eunomia-bpf/bpf-developer-tutorial> 或网站 <https://eunomia.dev/tutorials/> 获取更多示例和完整教程。
+
+内核模式 eBPF 运行时的 `Uprobe` 可能会带来较大的性能开销。在这种情况下，你也可以考虑使用用户模式的 eBPF 运行时，例如 [bpftime](https://github.com/eunomia-bpf/bpftime)。bpftime 是基于 LLVM JIT/AOT 的用户模式 eBPF 运行时，它可以在用户模式下运行 eBPF 程序，并且在处理 `uprobe` 时比内核模式 eBPF 更快。
+
+### 结论
+
+使用 eBPF 跟踪协程状态可以深入了解 Go 程序的执行情况，尤其是在传统调试工具可能无法胜任的生产环境中。通过利用 eBPF，开发人员可以监控和诊断性能问题，确保 Go 应用程序高效运行。
+
+请注意，本 eBPF 程序中使用的偏移量是特定于所跟踪的 Go 版本和程序的。随着 Go 的发展，这些偏移量可能会发生变化，需要对 eBPF 代码进行更新。
 
-This example is provided as GPL license
+在未来的探索中，我们可以将这种方法扩展到跟踪 Go 程序或其他语言的其他方面，展示 eBPF 在现代软件开发中的多功能性和强大作用。

src/31-goroutine/README_en.md

@@ -0,0 +1,112 @@
+# eBPF Practical Tutorial: Using eBPF to Trace Go Routine States
+
+Go, the popular programming language created by Google, is known for its powerful concurrency model. One of the key features that makes Go stand out is the use of goroutines—lightweight, managed threads that make it easy to write concurrent programs. However, understanding and tracing the execution states of these goroutines in real time can be challenging, especially when debugging complex systems.
+
+Enter eBPF (Extended Berkeley Packet Filter), a technology originally designed for network packet filtering, but which has since evolved into a powerful tool for tracing and monitoring system behavior. By leveraging eBPF, we can tap into the kernel and gather insights about the runtime behavior of Go programs, including the states of goroutines. This blog post explores how to use eBPF to trace the state transitions of goroutines in a Go program.
+
+## Background: Goroutines and eBPF
+
+### Goroutines
+
+Goroutines are a core feature of Go, providing a simple and efficient way to handle concurrency. Unlike traditional threads, goroutines are managed by the Go runtime rather than the operating system, making them much more lightweight. Goroutines can switch states, such as:
+
+- **RUNNABLE**: The goroutine is ready to run.
+- **RUNNING**: The goroutine is currently executing.
+- **WAITING**: The goroutine is waiting for some event (e.g., I/O, timers).
+- **DEAD**: The goroutine has finished executing and is terminated.
+
+Understanding these states and how goroutines transition between them is crucial for diagnosing performance issues and ensuring that your Go programs are running efficiently.
+
+### eBPF
+
+eBPF is a powerful technology that allows developers to run custom programs inside the Linux kernel without changing the kernel source code or loading kernel modules. Initially designed for packet filtering, eBPF has grown into a versatile tool used for performance monitoring, security, and debugging.
+
+By writing eBPF programs, developers can trace various system events, including system calls, network events, and process execution. In this blog, we'll focus on how eBPF can be used to trace the state transitions of goroutines in a Go program.
+
+## The eBPF Kernel Code
+
+Let's dive into the eBPF kernel code that makes this tracing possible.
+
+```c
+#include <vmlinux.h>
+#include "goroutine.h"
+#include <bpf/bpf_core_read.h>
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_tracing.h>
+
+#define GOID_OFFSET 0x98
+
+struct {
+  __uint(type, BPF_MAP_TYPE_RINGBUF);
+  __uint(max_entries, 256 * 1024);
+} rb SEC(".maps");
+
+SEC("uprobe/./go-server-http/main:runtime.casgstatus")
+int uprobe_runtime_casgstatus(struct pt_regs *ctx) {
+  int newval = ctx->cx;
+  void *gp = ctx->ax;
+  struct goroutine_execute_data *data;
+  u64 goid;
+  if (bpf_probe_read_user(&goid, sizeof(goid), gp + GOID_OFFSET) == 0) {
+    data = bpf_ringbuf_reserve(&rb, sizeof(*data), 0);
+    if (data) {
+      u64 pid_tgid = bpf_get_current_pid_tgid();
+      data->pid = pid_tgid;
+      data->tgid = pid_tgid >> 32;
+      data->goid = goid;
+      data->state = newval;
+      bpf_ringbuf_submit(data, 0);
+    }
+  }
+  return 0;
+}
+
+char LICENSE[] SEC("license") = "GPL";
+```
+
+1. **Header Files**: The code begins by including necessary header files, such as `vmlinux.h`, which provides kernel definitions, and `bpf_helpers.h`, which offers helper functions for eBPF programs.
+2. **GOID_OFFSET**: The offset of the `goid` field is hardcoded to `0x98`, which is specific to the Go version and the program being traced. This offset may vary between different Go versions or programs.
+3. **Ring Buffer Map**: A BPF ring buffer map is defined to store the goroutine execution data. This buffer allows the kernel to pass information to user space efficiently.
+4. **Uprobe**: The core of this eBPF program is an uprobes (user-level probe) attached to the `runtime.casgstatus` function in the Go program. This function is responsible for changing the state of a goroutine, making it an ideal place to intercept and trace state transitions.
+5. **Reading Goroutine ID**: The `bpf_probe_read_user` function reads the goroutine ID (`goid`) from the user space memory, using the predefined offset.
+6. **Submitting Data**: If the goroutine ID is successfully read, the data is stored in the ring buffer along with the process ID, thread group ID, and the new state of the goroutine. This data is then submitted to the user space for analysis.
+
+## Running the Program
+
+To run this tracing program, follow these steps:
+
+1. **Compile the eBPF Code**: Compile the eBPF program using a compiler like `ecc` (eBPF Compiler Collection) and generate a package that can be loaded by an eBPF loader.
+
+    ```bash
+    ecc goroutine.bpf.c goroutine.h
+    ```
+
+2. **Run the eBPF Program**: Use an eBPF loader to run the compiled eBPF program.
+
+    ```bash
+    ecli-rs run package.json
+    ```
+
+3. **Output**: The program will output the state transitions of goroutines along with their `goid`, `pid`, and `tgid`. Here’s an example of the output:
+
+    ```console
+    TIME     STATE       GOID   PID    TGID   
+    21:00:47 DEAD(6)     0      2542844 2542844
+    21:00:47 RUNNABLE(1) 0      2542844 2542844
+    21:00:47 RUNNING(2)  1      2542844 2542844
+    21:00:47 WAITING(4)  2      2542847 2542844
+    ```
+
+You can find the complete code in <https://github.com/eunomia-bpf/bpf-developer-tutorial/tree/main/src/31-goroutine>
+
+If you want to learn more about eBPF knowledge and practices, you can visit our tutorial code repository <https://github.com/eunomia-bpf/bpf-developer-tutorial> or website <https://eunomia.dev/tutorials/> to get more examples and complete tutorials.
+
+`Uprobe` in kernel mode eBPF runtime may also cause relatively large performance overhead. In this case, you can also consider using user mode eBPF runtime, such as [bpftime](https://github.com/eunomia-bpf/bpftime). bpftime is a user mode eBPF runtime based on LLVM JIT/AOT. It can run eBPF programs in user mode, compatible with kernel mode eBPF and can be faster for `uprobe`.
+
+### Conclusion
+
+Tracing goroutine states using eBPF provides deep insights into the execution of Go programs, especially in production environments where traditional debugging tools may fall short. By leveraging eBPF, developers can monitor and diagnose performance issues, ensuring their Go applications run efficiently.
+
+Keep in mind that the offsets used in this eBPF program are specific to the Go version and the program being traced. As Go evolves, these offsets may change, requiring updates to the eBPF code.
+
+In future explorations, we can extend this approach to trace other aspects of Go programs or even other languages, demonstrating the versatility and power of eBPF in modern software development.

src/31-goroutine/goroutine.bpf.c

@@ -28,8 +28,6 @@
 #include <bpf/bpf_helpers.h>
 #include <bpf/bpf_tracing.h>
 
-
-
 #define GOID_OFFSET 0x98
 
 struct {
@@ -40,7 +38,7 @@ struct {
 SEC("uprobe/./go-server-http/main:runtime.casgstatus")
 int uprobe_runtime_casgstatus(struct pt_regs *ctx) {
   int newval = ctx->cx;
-  void *gp = ctx->ax;
+  void *gp = (void*)ctx->ax;
   struct goroutine_execute_data *data;
   u64 goid;
   if (bpf_probe_read_user(&goid, sizeof(goid), gp + GOID_OFFSET) == 0) {