Architecture: Policies and Datasets

Policy Service (orb-policies)

This microservice is responsible for:

• Agent Policy management
  • /policies/agent User facing CRUD management of Agent Policies
  • produce system events during policy CRUD actions for consumption by other microservices, esp Fleet which communicates with the agents on policy events
    • Create: no agent comms action since policy is not in a dataset yet
    • Update: if policy is used in any datasets, send policy update RPC over group channel ID for each dataset
    • Delete: if policy is used in any datasets, set Dataset to Invalid and send dataset revoke over group channel ID for each dataset
  • expose gRPC API for policy retrieval by other services
  • consult Fleet service via gRPC to retrieve available agent backend capabilities (e.g. pktvisor taps, inputs, handlers) for policy verification
• Dataset management
  • /policies/dataset User facing CRUD management of Datasets
  • produce system events during dataset CRUD actions for consumption by other microservices, esp Fleet which communicates with the agents on dataset events
    • Create: validate. send Dataset RPC to agent group channel ID
    • Update: only tags and sinks can change so no agent comms
    • Delete: send dataset revoke over group channel ID
  • consult Sink service via gRPC to verify Sink ID
  • consult Fleet service via gRPC to verify Agent Group ID
  • verify agent policy ID
  • expose gRPC API for dataset ID to sink ID mapping for sink services. This is so that sink consumers (like Prometheus) know which in configuration to use for which dataset

Agent Policy Data Model

DB Field	JSON Field	Public	ReadOnly	Description
id	id	X	X	UUIDv4
mf_owner_id				UUIDv4 tenant owner ID
name	name	X		A name label field
description	description	X		Description
backend	backend	X		Agent backend this policy is for (e.g. pktvisor). Cannot change after creation.
schema_version	schema_version	X		Schema version of the Agent Backend this policy schema, e.g. "1.0"
version	version	X	X	A monotonically increasing counter starting at 0 on creation and increasing with each policy update
policy	policy	X		JSON: Agent backend specific policy data. This is the canonical format, possibly converted to from policy_data
policy_data	policy_data	X		Optional Agent backend specific policy data in some non-JSON format (e.g. YAML or INI)
format	format	X		Format specifier for the data specified in "policy_data" field (e.g. "yaml"). Required if policy_data is not empty.
tags	tags	X		Orb tags field: defined through the API or UI
ts_created	ts_created	X	X	A timestamp of creation

Dataset Data Model

DB Field	JSON Field	Public	ReadOnly	Description
id	id	X	X	UUIDv4
mf_owner_id				UUIDv4 tenant owner ID
name	name	X		A name label field
agent_group_id	agent_group_id	X		UUIDv4 agent group ID. cannot be updated once created.
agent_policy_id	agent_policy_id	X		UUIDv4 agent policy ID. cannot be updated once created.
sink_ids	sink_ids	X		array of UUIDv4 sink ID
valid	valid	X	X	Boolean field indicating whether the dataset is valid (based on the validity of its components)
tags	tags	X		Orb tags field: defined through the API or UI
ts_created	ts_created	X	X	A timestamp of creation