diff --git a/docs/docs/data-sources/containers.md b/docs/docs/data-sources/containers.md new file mode 100644 index 000000000000..638c7d25bcaf --- /dev/null +++ b/docs/docs/data-sources/containers.md @@ -0,0 +1,99 @@ +# Containers Data Source + +The [container enrichment](../integrating/container-engines.md) feature gives Tracee the ability to extract details about active containers and link this information to the events it captures. + +The [data source](./overview.md) feature makes the information gathered from active containers accessible to signatures. When an event is captured and triggers a signature, that signature can retrieve information about the container using its container ID, which is bundled with the event being analyzed by the signature. + +## Enabling the Feature + +The data source does not need to be enabled, but requires that the `container enrichment` feature is. To enable the enrichment feature, execute trace with `--containers`. For more information you can read [container enrichment](../integrating/container-engines.md) page. + +## Internal Data Organization + +From the [data-sources documentation](../data-sources/overview.md), you'll see that searches use keys. It's a bit like looking up information with a specific tag (or a key=value storage). + +The `containers data source` operates straightforwardly. Using `string` keys, which represent the container IDs, you can fetch `map[string]string` values as shown below: + +```go + schemaMap := map[string]string{ + "container_id": "string", + "container_name": "string", + "container_image": "string", + "k8s_pod_id": "string", + "k8s_pod_name": "string", + "k8s_pod_namespace": "string", + "k8s_pod_sandbox": "bool", + } +``` + +From the structure above, using the container ID lets you access details like the originating Kubernetes pod name or the image utilized by the container. + +## Using the Containers Data Source + +> Make sure to read [Golang Signatures](../events/custom/golang.md) first. + +### Signature Initialization + +During the signature initialization, get the containers data source instance: + +```go +type e2eContainersDataSource struct { + cb detect.SignatureHandler + containersData detect.DataSource +} + +func (sig *e2eContainersDataSource) Init(ctx detect.SignatureContext) error { + sig.cb = ctx.Callback + containersData, ok := ctx.GetDataSource("tracee", "containers") + if !ok { + return fmt.Errorf("containers data source not registered") + } + sig.containersData = containersData + return nil +} +``` + +Then, to each event being handled, you will `Get()`, from the data source, the information needed. + +### On Events + +Given the following example: + +```go +func (sig *e2eContainersDataSource) OnEvent(event protocol.Event) error { + eventObj, ok := event.Payload.(trace.Event) + if !ok { + return fmt.Errorf("failed to cast event's payload") + } + + switch eventObj.EventName { + case "sched_process_exec": + containerId := eventObj.Container.ID + if containerId == "" { + return fmt.Errorf("received non container event") + } + + container, err := sig.containersData.Get(containerId) + if !ok { + return fmt.Errorf("failed to find container in data source: %v", err) + } + + containerImage, ok := container["container_image"].(string) + if !ok { + return fmt.Errorf("failed to obtain the container image name") + } + + m, _ := sig.GetMetadata() + + sig.cb(detect.Finding{ + SigMetadata: m, + Event: event, + Data: map[string]interface{}{}, + }) + } + + return nil +} +``` + +You may see that, through the `event object container ID` information, you may query the data source and obtain the `container name` or any other information listed before. diff --git a/docs/docs/data-sources/overview.md b/docs/docs/data-sources/overview.md index 89f7004a2a40..eba716783bb7 100644 --- a/docs/docs/data-sources/overview.md +++ b/docs/docs/data-sources/overview.md @@ -1,38 +1,65 @@ # Data Sources (Experimental) -Data sources are a new feature, which will be the base of allowing access to dynamic data stores in signature writing (currently only available in golang). -Data sources are currently an experimental feature and in active development, and usage is opt-in. +Data sources are a new feature, which will be the base of allowing access to +dynamic data stores in signature writing (currently only available in golang). + +> Data sources are currently an experimental feature and in active development, +> and usage is opt-in. ## Why use data sources? -Data sources should be used when a signature requires access to data not available to it from the events it receives. -For example, a signature may need access to additional data about a container where an event was generated. Using tracee's builtin container data source it can do so without additionally tracking container lifecycle events. +Signatures should opt for data sources when they need access to data beyond what +is provided by the events they process. + +For instance, a signature may need access to data about the container where the +event being processed was generated. With Tracee's integrated container data +source, this can be achieved without the signature having to separately monitor +container lifecycle events. ## What data sources can I use -Currently, only builtin data sources from tracee are available. -Initially only a data source for containers will be available, but the list will be expanded as this and other features are further developed. +For now, only the built-in data sources from Tracee are at your disposal. +Looking ahead, there are plans to enable integration of data sources into Tracee +either as plugins or extensions. + +Currently, two primary data source exist: + +1. Containers: Provides metadata about containers given a container id. +1. Process Tree: Provides access to a tree of ever existing processes and threads. + +This list will be expanded as other features are developed. ## How to use data sources -In order to use a data source in a signature you must request access to it in the `Init` stage. This can be done through the `SignatureContext` passed at that stage as such: + +In order to use a data source in a signature you must request access to it in +the `Init` stage. This can be done through the `SignatureContext` passed at that +stage as such: + ```golang func (sig *mySig) Init(ctx detect.SignatureContext) error { ... containersData, ok := ctx.GetDataSource("tracee", "containers") - if !ok { - return fmt.Errorf("containers data source not registered") - } + if !ok { + return fmt.Errorf("containers data source not registered") + } if containersData.Version() > 1 { - return fmt.Errorf("containers data source version not supported, please update this signature") - } - sig.containersData = containersData + return fmt.Errorf("containers data source version not supported, please update this signature") + } + sig.containersData = containersData } ``` -As you can see we have requested access to the data source through two keys, a namespace, and a data source ID. Namespaces are used to avoid name conflicts in the future when custom data sources can be integrated. All of tracee's builtin data sources will be available under the "tracee" namespace. -After checking the data source is available, we suggest to add a version check against the data source. Doing so will let you avoid running a signature which was not updated to run with a new data source schema. +As you can see, access to the data source has been requested using two keys: a +namespace and a data source ID. Namespaces are employed to prevent name +conflicts in the future when integrating custom data sources. All built-in data +sources from Tracee will be available under the "tracee" namespace. + +After verifying the data source's availability, it's suggested to include a +version check against the data source. This approach ensures that outdated +signatures aren't run with a newer data source schema. + +Now, in the `OnEvent` function, you may use the data source like so: -Now, in the `OnEvent` function, you may use the data source like so: ```golang container, err := sig.containersData.Get(containerId) if !ok { @@ -40,5 +67,10 @@ if !ok { } containerName := container["container_name"].(string) -``` -Each Data source comes with one querying method `Get(key any) map[string]any`. In the above example, omitting the type validation when checking the key, which was safe to do by following the schema (given through the `Schema()` method), a json representation of the returned map, and initially checking the data source version. \ No newline at end of file +``` + +Each Data source provides a querying method `Get(key any) map[string]any`. In +the provided example, type validation is omitted during key verification. This +omission is safe when adhering to the schema (provided by the `Schema()` +method), considering the JSON representation of the returned map, and after an +initial check of the data source version. diff --git a/docs/docs/data-sources/process-tree.md b/docs/docs/data-sources/process-tree.md new file mode 100644 index 000000000000..276d39666a68 --- /dev/null +++ b/docs/docs/data-sources/process-tree.md @@ -0,0 +1,282 @@ +# Process Tree Data + +The `Process Tree` feature offers a structured view of processes and threads active in the system where Tracee is deployed. This setup facilitates quick access, updates, and tracking of processes, child processes, and related threads. All relationship and metadata data points for processes and threads are versioned, so you can pull data snapshots from a precise timestamp. + +## Enabling the Feature + +To switch on the `Process Tree` feature, run the command: + +```bash +sudo tracee --output option:sort-events --output json --output option:parse-arguments --proctree source=both --events +``` + +The underlying structure is populated using the core `sched_process_fork`, `sched_process_exec`, and `sched_process_exit` events and their args. There's also an option to bootstrap the process tree through a secondary route using internal signal events. + +> Introducing this secondary event source is strategic: it reduces interference with actively traced events, leading to more accurate and granular updates in the process tree. + +The number of processes retained in the tree hinges on cache size. We have two separate caches at play: one for processes and another for threads. Both default to a size of 32K, supporting tracking for up to 32,768 processes and the same number of threads. It's worth noting that these are LRU caches: once full, they'll evict the least recently accessed entries to accommodate fresh ones. + +## Command Line Option + +```bash +$ tracee --proctree help +Example: + --proctree source=[none|events|signals|both] + none | process tree is disabled (default). + events | process tree is built from events. + signals | process tree is built from signals. + both | process tree is built from both events and signals. + --proctree process-cache=8192 | will cache up to 8192 processes in the tree (LRU cache). + --proctree thread-cache=4096 | will cache up to 4096 threads in the tree (LRU cache). + +Use comma OR use the flag multiple times to choose multiple options: + --proctree source=A,process-cache=B,thread-cache=C + --proctree process-cache=X --proctree thread-cache=Y +``` + +## Internal Data Organization + +For those looking to develop signatures or simply understand the underpinnings of the `Process Tree` feature, a grasp on its internal data organization is invaluable. At its core, the system is structured for fast access, updating, and tracking. + +### Hash Indexing + +Every entity in the `Process Tree`, be it a process or thread, is indexed using a distinctive hash, formulated by combining a task's `start time` and `thread id`. Events in the system come attached with this hash in their context under the `EntityID` label. + +### Core Components + +1. **ProcessTree**: A macro view of all the processes and threads active in the system. + - **Processes**: Defined either as a single-threaded application or the lead thread in a multi-thread application where the PID and TID are identical. + - **Threads**: Also known as Light-Weight Processes by the kernel, they include both separate threads and the thread group leader. Threads under the same leader share a PID but possess distinct TIDs. + +2. **Process**: A representation of individual processes. It contains: + - The process metadata using the `TaskInfo` structure. + - Information on its executable and interpreter using the `FileInfo` structure. + - References to its parent, child processes, and sibling threads within the same thread group. + +3. **Thread**: A representation of system threads. It contains: + - The thread metadata using the `TaskInfo` structure. + - Links to its parent and the its thread group leader. + +4. **TaskInfo**: From task names, PIDs, TIDs, PPIDs, ownership details, to start and end timestamps, it's all cataloged here. As tasks evolve, certain properties might shift. These changes are recorded using changelogs. + +5. **TaskInfo**: Acts as the central repository for task-specific attributes, including task names, PIDs (Process IDs), TIDs (Thread IDs), PPIDs (Parent Process IDs), and ownership UID/GID specifications. As task states transition within the kernel space, certain properties are subject to modification; such alterations are persistently tracked using changelogs. + +6. **FileInfo**: This structure aggregates file metadata, capturing attributes like path, device, and inode details. In the realm of processes, `FileInfo` is responsible for maintaining records of binaries and interpreters, with alterations being tracked in changelogs. + +## Process Tree Artifacts + +In an upcoming update, the process tree will be enhanced with the addition of `artifacts`. Each process within the tree will be augmented with these "artifacts" to denote a task's various interactions and operations within the system. These artifacts, sourced from the tracing events provided by Tracee, offer a detailed depiction of a process's activities at the system level. Potential artifacts encompass: + +- **File Operations**: Opened files, read/write activities, file deletion, and attribute changes. +- **Network Activities**: Sockets created, inbound/outbound connections, transmitted/received data packets, and protocol-specific operations (like TCP handshakes or UDP transmissions). +- **System Calls**: Executed syscalls, their arguments, and return values. +- **Memory Activities**: Memory allocation, deallocation, and page faults. +- **Device Interactions**: I/O operations on devices, device mounting/unmounting. +- **Kernel Module Activities**: Module load and unload operations. +- **Security-Related Activities**: Capabilities changes, SELinux operations, and AppArmor profile transitions. + +This enhancement aims to offer developers and sysadmins a more detailed and granular view of task behaviors, paving the way for better system monitoring, diagnostics, and potential threat detection. + +## Using the Process Tree + +The process tree is only available internally, to tracee's components, but, through the [datasource](./overview.md) mechanism, signatures are able to query the tree data using the data source process tree API. + +### Accessing the Process Tree Data Source + +> Make sure to read [Golang Signatures](../events/custom/golang.md) first. + +During the signature initialization, get the process tree data source instance: + +```go +type e2eProcessTreeDataSource struct { + cb detect.SignatureHandler + processTreeDS detect.DataSource +} + +// Init is called once when the signature is loaded. +func (sig *e2eProcessTreeDataSource) Init(ctx detect.SignatureContext) error { + sig.cb = ctx.Callback + + processTreeDataSource, ok := ctx.GetDataSource("tracee", "process_tree") + if !ok { + return fmt.Errorf("data source tracee/process_tree is not registered") + } + + sig.processTreeDS = processTreeDataSource + + return nil +} +``` + +Then, to each event being handled, you will `Get()`, from the data source, the information needed. There are 3 types of information that can be requested: + +1. datasource.ProcKey: for process information retrieval. +2. datasource.ThreadKey: for thread information retrieval. +3. datasource.LineageKey: for process lineage information retrieval. + +Before explaining each request type and how to use them, consider the following signature `OnEvent()` handler example: + +```go +// OnEvent is called when a subscribed event occurs. +func (sig *e2eProcessTreeDataSource) OnEvent(event protocol.Event) error { + eventObj, ok := event.Payload.(trace.Event) + if !ok { + return fmt.Errorf("failed to cast event's payload") + } + + switch eventObj.EventName { + case "sched_process_exec": + err = sig.check(&eventObj) + if err != nil { + return err + } + } + + // If all checks passed, send a finding + m, _ := sig.GetMetadata() + + sig.cb(detect.Finding{ + SigMetadata: m, + Event: event, + Data: map[string]interface{}{}, + }) + + return nil +} +``` + +Where the `check()` method will either be: + +- checkProcess() +- checkThread() +- checkLineage() + +> You can check related data structures directly in the [source code](https://github.com/aquasecurity/tracee/blob/7b095a8d9a11cbd11ac61d3eec4b0a0f77f66dd9/pkg/proctree/datasource.go#L59) for more information. Below you will find easy to understand examples. + +### Processes Information Retrieval + +Utilize the data source instance object saved from the `Init()` method, and use the information from the current event to query the process tree for details about the process that triggered the event. + +```go +func (sig *e2eProcessTreeDataSource) checkProcess(eventObj *trace.Event) error { + // Pick the process info from the data source + procQueryAnswer, err := sig.processTreeDS.Get( + datasource.ProcKey{ + EntityId: eventObj.ProcessEntityId, + Time: time.Unix(0, int64(eventObj.Timestamp)), + }) + if err != nil { + return fmt.Errorf(debug("could not find process")) + } + processInfo, ok := procQueryAnswer["process_info"].(datasource.ProcessInfo) + if !ok { + return fmt.Errorf(debug("could not extract info")) + } + + // Compare PID, NS PID and PPID + if processInfo.Pid != eventObj.HostProcessID { + return fmt.Errorf(debug("no match for pid")) + } + if processInfo.NsPid != eventObj.ProcessID { + return fmt.Errorf(debug("no match for ns pid")) + } + if processInfo.Ppid != eventObj.HostParentProcessID { + return fmt.Errorf(debug("no match for ppid")) + } + + // Check if the process lists itself in the list of its threads + threadExist := false + for tid := range processInfo.ThreadsIds { + if tid == eventObj.HostThreadID { + threadExist = true + break + } + } + if !threadExist { + return fmt.Errorf(debug("process not listed as thread")) + } +``` + +From the [data-sources documentation](../data-sources/overview.md), you'll see that searches use keys. It's a bit like looking up information with a specific tag (or a key=value storage). + +In the provided example, the `eventObj.ProcessEntityId` key (which is the process hash accompanying the event being handled) is utilized alongside the `datasource.ProcKey{}` argument to search for a process in the process tree. The resulting process is the one associated with the event under consideration. + +> Keep in mind that users can specify a time to retrieve the information. By using the event timestamp, you obtain data available up to that specific moment. + +Within the retrieved process object, you can find essential information about the running process. This includes details such as the binary associated with the executing program, the interpreter used for that program (either ld.so for ELF files or the relevant interpreters responsible for execution). In the near future, you can expect to see additional data related to the process, such as open files and sockets, known hosts and resolved names, utilized protocols, and more. + +### Threads Information Retrieval + +```go +// checkThread checks if thread info in the data source matches the info from the event. +func (sig *e2eProcessTreeDataSource) checkThread(eventObj *trace.Event) error { + // Pick the thread info from the data source + threadQueryAnswer, err := sig.processTreeDS.Get( + datasource.ThreadKey{ + EntityId: eventObj.ThreadEntityId, + Time: time.Unix(0, int64(eventObj.Timestamp)), + }, + ) + if err != nil { + return fmt.Errorf(debug("could not find thread")) + } + threadInfo, ok := threadQueryAnswer["thread_info"].(datasource.ThreadInfo) + if !ok { + return fmt.Errorf(debug("could not extract info")) + } + + // Compare TID, NS TID and PID + if threadInfo.Tid != eventObj.HostThreadID { + return fmt.Errorf(debug("no match for tid")) + } + if threadInfo.NsTid != eventObj.ThreadID { + return fmt.Errorf(debug("no match for ns tid")) + } + if threadInfo.Pid != eventObj.HostProcessID { + return fmt.Errorf(debug("no match for pid")) + } + + return nil +} +``` + +In the example, the `eventObj.ThreadEntityId` key is used alongside the `datasource.ThreadKey{}` argument to search for a thread in the process tree. For applications that use only one thread, or the primary thread in multi-threaded applications, you'll find entries in both the processes and threads sections of the process tree. However, for simpler threads (commonly referred to as regular threads), they appear solely in the threads section. + +### Lineage Information Retrieval + +Using the `eventObj.ProcessEntityId` key (the process hash from the current event) in conjunction with the `datasource.LineageKey{}` argument allows retrieval of not just a singular process but multiple processes up the chain of ancestry: process, its parent, the parent's parent, and so forth. This capability is crucial for signatures that require analysis of process lineage and the associated artifacts of each process in that lineage. + +```go +func (sig *e2eProcessTreeDataSource) checkLineage(eventObj *trace.Event) error { + maxDepth := 5 // up to 5 ancestors + process itself + + // Pick the lineage info from the data source. + lineageQueryAnswer, err := sig.processTreeDS.Get( + datasource.LineageKey{ + EntityId: eventObj.ProcessEntityId, + Time: time.Unix(0, int64(eventObj.Timestamp)), + MaxDepth: maxDepth, + }, + ) + if err != nil { + return fmt.Errorf(debug("could not find lineage")) + } + lineageInfo, ok := lineageQueryAnswer["process_lineage"].(datasource.ProcessLineage) + if !ok { + return fmt.Errorf("failed to extract ProcessLineage from data") + } + + compareMaps := func(map1, map2 map[int]uint32) bool { + return true // (or false) + } + + // First ancestor is the process itself: lineageInfo[0] (ProcessInfo object) + + for _, ancestor := range lineageInfo[1:] { + // do something with "ancestor" ProcessInfo + } + + return nil +} +``` + diff --git a/mkdocs.yml b/mkdocs.yml index 998364333857..19d9df0e4c99 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -498,22 +498,24 @@ nav: - net_packet_http_response: docs/events/builtin/network/net_packet_http_response.md - Behavioral Signatures: docs/events/builtin/signatures.md - Extra Events: + - bpf_attach: docs/events/builtin/extra/bpf_attach.md - do_sigaction: docs/events/builtin/extra/do_sigaction.md - - kallsysm_lookup_name: docs/events/builtin/extra/kallsyms_lookup_name.md - - vfs_read: docs/events/builtin/extra/vfs_read.md - - vfs_readv: docs/events/builtin/extra/vfs_readv.md - file_modification: docs/events/builtin/extra/file_modification.md - - security_file_mprotect: docs/events/builtin/extra/security_file_mprotect.md - - security_socket_setsockopt: docs/events/builtin/extra/security_socket_setsockopt.md - - bpf_attach: docs/events/builtin/extra/bpf_attach.md + - ftrace_hook: docs/events/builtin/extra/ftrace_hook.md + - hidden_kernel_module: docs/events/builtin/extra/hidden_kernel_module.md + - hooked_syscalls: docs/events/builtin/extra/hooked_syscalls.md + - kallsysm_lookup_name: docs/events/builtin/extra/kallsyms_lookup_name.md - magic_write: docs/events/builtin/extra/magic_write.md - mem_prot_alert: docs/events/builtin/extra/mem_prot_alert.md - - symbols_loaded: docs/events/builtin/extra/symbols_loaded.md - - symbols_collision: docs/events/builtin/extra/symbols_collision.md - - hidden_kernel_module: docs/events/builtin/extra/hidden_kernel_module.md - process_execute_failed: docs/events/builtin/extra/process_execute_failed.md - security_bpf_prog: docs/events/builtin/extra/security_bpf_prog.md - security_bprm_check: docs/events/builtin/extra/security_bprm_check.md + - security_file_mprotect: docs/events/builtin/extra/security_file_mprotect.md + - security_socket_setsockopt: docs/events/builtin/extra/security_socket_setsockopt.md + - symbols_collision: docs/events/builtin/extra/symbols_collision.md + - symbols_loaded: docs/events/builtin/extra/symbols_loaded.md + - vfs_read: docs/events/builtin/extra/vfs_read.md + - vfs_readv: docs/events/builtin/extra/vfs_readv.md - Custom Events: - Overview: docs/events/custom/overview.md - Go: docs/events/custom/golang.md @@ -540,6 +542,8 @@ nav: - Healthz: docs/integrating/healthz.md - Data Sources: - Overview: docs/data-sources/overview.md + - Containers: docs/data-sources/containers.md + - Process Tree: docs/data-sources/process-tree.md - Deep Dive: - Caching Events: docs/deep-dive/caching-events.md - Ordering Events: docs/deep-dive/ordering-events.md