Skip to content

Commit 05fd06c

Browse files
mergify[bot]MichaelKatsoulis
mergify[bot]
and
MichaelKatsoulis
authored
[8.x](backport #41889) Add support for podman metrics in docker module (#41967)
* Add support for podman metrics in docker module (#41889) --------- Co-authored-by: Michael Katsoulis <michaelkatsoulis88@gmail.com>
1 parent c43fb55 commit 05fd06c

File tree

15 files changed

+77
-16
lines changed

15 files changed

+77
-16
lines changed

‎CHANGELOG.next.asciidoc

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -402,6 +402,7 @@ https://github.com/elastic/beats/compare/v8.8.1\...main[Check the HEAD diff]
402402
- Only watch metadata for ReplicaSets in metricbeat k8s module {pull}41289[41289]
403403
- Add support for region/zone for Vertex AI service in GCP module {pull}41551[41551]
404404
- Add support for location label as an optional configuration parameter in GCP metrics metricset. {issue}41550[41550] {pull}41626[41626]
405+
- Add support for podman metrics in docker module. {pull}41889[41889]
405406

406407
*Metricbeat*
407408
- Add benchmark module {pull}41801[41801]

‎metricbeat/docs/modules/docker.asciidoc

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,9 @@ The Docker module is currently tested on Linux and Mac with the community
2222
edition engine, versions 1.11 and 17.09.0-ce. It is not tested on Windows,
2323
but it should also work there.
2424

25+
The Docker module supports collection of metrics from Podman's Docker-compatible API.
26+
It has been tested on Linux and Mac with Podman Rest API v2.0.0 and above.
27+
2528
[float]
2629
=== Module-specific configuration notes
2730

@@ -30,6 +33,9 @@ It is strongly recommended that you run Docker metricsets with a
3033
Docker API already takes up to 2 seconds. Specifying less than 3 seconds will
3134
result in requests that timeout, and no data will be reported for those
3235
requests.
36+
In the case of Podman, the configuration parameter `podman` should be set to `true`.
37+
This enables streaming of container stats output, which allows for more accurate
38+
CPU percentage calculations when using Podman.
3339

3440

3541
:edit_url:
@@ -62,6 +68,9 @@ metricbeat.modules:
6268
# If set to true, replace dots in labels with `_`.
6369
#labels.dedot: false
6470
71+
# Docker module supports metrics collection from podman's docker compatible API. In case of podman set to true.
72+
# podman: false
73+
6574
# Skip metrics for certain device major numbers in docker/diskio.
6675
# Necessary on systems with software RAID, device mappers,
6776
# or other configurations where virtual disks will sum metrics from other disks.

‎metricbeat/metricbeat.reference.yml

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -268,6 +268,9 @@ metricbeat.modules:
268268
# If set to true, replace dots in labels with `_`.
269269
#labels.dedot: false
270270

271+
# Docker module supports metrics collection from podman's docker compatible API. In case of podman set to true.
272+
# podman: false
273+
271274
# Skip metrics for certain device major numbers in docker/diskio.
272275
# Necessary on systems with software RAID, device mappers,
273276
# or other configurations where virtual disks will sum metrics from other disks.

‎metricbeat/module/docker/_meta/config.reference.yml

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,9 @@
1717
# If set to true, replace dots in labels with `_`.
1818
#labels.dedot: false
1919

20+
# Docker module supports metrics collection from podman's docker compatible API. In case of podman set to true.
21+
# podman: false
22+
2023
# Skip metrics for certain device major numbers in docker/diskio.
2124
# Necessary on systems with software RAID, device mappers,
2225
# or other configurations where virtual disks will sum metrics from other disks.

‎metricbeat/module/docker/_meta/config.yml

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,9 @@
1515
# If set to true, replace dots in labels with `_`.
1616
#labels.dedot: false
1717

18+
# Docker module supports metrics collection from podman's Docker-compatible API. In case of podman set to true.
19+
# podman: false
20+
1821
# Skip metrics for certain device major numbers in docker/diskio.
1922
# Necessary on systems with software RAID, device mappers,
2023
# or other configurations where virtual disks will sum metrics from other disks.

‎metricbeat/module/docker/_meta/docs.asciidoc

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,9 @@ The Docker module is currently tested on Linux and Mac with the community
1111
edition engine, versions 1.11 and 17.09.0-ce. It is not tested on Windows,
1212
but it should also work there.
1313

14+
The Docker module supports collection of metrics from Podman's Docker-compatible API.
15+
It has been tested on Linux and Mac with Podman Rest API v2.0.0 and above.
16+
1417
[float]
1518
=== Module-specific configuration notes
1619

@@ -19,3 +22,6 @@ It is strongly recommended that you run Docker metricsets with a
1922
Docker API already takes up to 2 seconds. Specifying less than 3 seconds will
2023
result in requests that timeout, and no data will be reported for those
2124
requests.
25+
In the case of Podman, the configuration parameter `podman` should be set to `true`.
26+
This enables streaming of container stats output, which allows for more accurate
27+
CPU percentage calculations when using Podman.

‎metricbeat/module/docker/config.go

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -19,14 +19,16 @@ package docker
1919

2020
// Config contains the config needed for the docker
2121
type Config struct {
22-
TLS *TLSConfig `config:"ssl"`
23-
DeDot bool `config:"labels.dedot"`
22+
TLS *TLSConfig `config:"ssl"`
23+
DeDot bool `config:"labels.dedot"`
24+
Podman bool `config:"podman"`
2425
}
2526

2627
// DefaultConfig returns default module config
2728
func DefaultConfig() Config {
2829
return Config{
29-
DeDot: true,
30+
DeDot: true,
31+
Podman: false,
3032
}
3133
}
3234

‎metricbeat/module/docker/cpu/cpu.go

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,7 @@ type MetricSet struct {
4040
cpuService *CPUService
4141
dockerClient *client.Client
4242
dedot bool
43+
podman bool
4344
}
4445

4546
// New creates a new instance of the docker cpu MetricSet.
@@ -68,12 +69,13 @@ func New(base mb.BaseMetricSet) (mb.MetricSet, error) {
6869
dockerClient: client,
6970
cpuService: &CPUService{Cores: cpuConfig.Cores},
7071
dedot: config.DeDot,
72+
podman: config.Podman,
7173
}, nil
7274
}
7375

7476
// Fetch returns a list of docker CPU stats.
7577
func (m *MetricSet) Fetch(r mb.ReporterV2) error {
76-
stats, err := docker.FetchStats(m.dockerClient, m.Module().Config().Timeout)
78+
stats, err := docker.FetchStats(m.dockerClient, m.Module().Config().Timeout, m.podman, m.Logger())
7779
if err != nil {
7880
return fmt.Errorf("failed to get docker stats: %w", err)
7981
}

‎metricbeat/module/docker/diskio/diskio.go

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -89,7 +89,7 @@ func New(base mb.BaseMetricSet) (mb.MetricSet, error) {
8989

9090
// Fetch creates list of events with diskio stats for all containers.
9191
func (m *MetricSet) Fetch(r mb.ReporterV2) error {
92-
stats, err := docker.FetchStats(m.dockerClient, m.Module().Config().Timeout)
92+
stats, err := docker.FetchStats(m.dockerClient, m.Module().Config().Timeout, false, m.Logger())
9393
if err != nil {
9494
return fmt.Errorf("failed to get docker stats: %w", err)
9595
}

‎metricbeat/module/docker/docker.go

Lines changed: 32 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,7 @@ import (
3434
"github.com/elastic/beats/v7/metricbeat/mb"
3535
"github.com/elastic/beats/v7/metricbeat/mb/parse"
3636
"github.com/elastic/elastic-agent-autodiscover/docker"
37+
"github.com/elastic/elastic-agent-libs/logp"
3738
)
3839

3940
// HostParser is a TCP host parser function for docker tcp host addresses
@@ -91,7 +92,7 @@ func NewDockerClient(endpoint string, config Config) (*client.Client, error) {
9192
}
9293

9394
// FetchStats returns a list of running containers with all related stats inside
94-
func FetchStats(client *client.Client, timeout time.Duration) ([]Stat, error) {
95+
func FetchStats(client *client.Client, timeout time.Duration, stream bool, logger *logp.Logger) ([]Stat, error) {
9596
ctx, cancel := context.WithTimeout(context.Background(), timeout)
9697
defer cancel()
9798
containers, err := client.ContainerList(ctx, container.ListOptions{})
@@ -108,7 +109,7 @@ func FetchStats(client *client.Client, timeout time.Duration) ([]Stat, error) {
108109
for _, container := range containers {
109110
go func(container types.Container) {
110111
defer wg.Done()
111-
statsQueue <- exportContainerStats(ctx, client, &container)
112+
statsQueue <- exportContainerStats(ctx, client, &container, stream, logger)
112113
}(container)
113114
}
114115

@@ -133,18 +134,41 @@ func FetchStats(client *client.Client, timeout time.Duration) ([]Stat, error) {
133134
// This is currently very inefficient as docker calculates the average for each request,
134135
// means each request will take at least 2s: https://github.com/docker/docker/blob/master/cli/command/container/stats_helpers.go#L148
135136
// Getting all stats at once is implemented here: https://github.com/docker/docker/pull/25361
136-
func exportContainerStats(ctx context.Context, client *client.Client, container *types.Container) Stat {
137+
// In case stream is true, we use get a stream of results for container stats. From the stream we keep the second result.
138+
// This is needed for podman use case where in case stream is false, no precpu stats are returned. The precpu stats
139+
// are required for the cpu percentage calculation. We keep the second result as in the first result, the stats are not correct.
140+
func exportContainerStats(ctx context.Context, client *client.Client, container *types.Container, stream bool, logger *logp.Logger) Stat {
137141
var event Stat
138142
event.Container = container
139-
140-
containerStats, err := client.ContainerStats(ctx, container.ID, false)
143+
containerStats, err := client.ContainerStats(ctx, container.ID, stream)
141144
if err != nil {
145+
logger.Debugf("Failed fetching container stats: %v", err)
142146
return event
143147
}
144-
145148
defer containerStats.Body.Close()
146-
decoder := json.NewDecoder(containerStats.Body)
147-
decoder.Decode(&event.Stats)
148149

150+
// JSON decoder
151+
decoder := json.NewDecoder(containerStats.Body)
152+
if !stream {
153+
if err := decoder.Decode(&event.Stats); err != nil {
154+
logger.Debugf("Failed decoding event: %v", err)
155+
return event
156+
}
157+
} else {
158+
// handle stream. Take the second result.
159+
count := 0
160+
for decoder.More() {
161+
if err := decoder.Decode(&event.Stats); err != nil {
162+
logger.Debugf("Failed decoding event: %v", err)
163+
return event
164+
}
165+
166+
count++
167+
// Exit after the second result
168+
if count == 2 {
169+
break
170+
}
171+
}
172+
}
149173
return event
150174
}

‎metricbeat/module/docker/memory/memory.go

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,7 @@ type MetricSet struct {
4343
memoryService *MemoryService
4444
dockerClient *client.Client
4545
dedot bool
46+
podman bool
4647
logger *logp.Logger
4748
}
4849

@@ -64,13 +65,14 @@ func New(base mb.BaseMetricSet) (mb.MetricSet, error) {
6465
memoryService: &MemoryService{},
6566
dockerClient: dockerClient,
6667
dedot: config.DeDot,
68+
podman: config.Podman,
6769
logger: logger,
6870
}, nil
6971
}
7072

7173
// Fetch creates a list of memory events for each container.
7274
func (m *MetricSet) Fetch(r mb.ReporterV2) error {
73-
stats, err := docker.FetchStats(m.dockerClient, m.Module().Config().Timeout)
75+
stats, err := docker.FetchStats(m.dockerClient, m.Module().Config().Timeout, m.podman, m.Logger())
7476
if err != nil {
7577
return fmt.Errorf("failed to get docker stats: %w", err)
7678
}

‎metricbeat/module/docker/network/network.go

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -66,7 +66,7 @@ func New(base mb.BaseMetricSet) (mb.MetricSet, error) {
6666

6767
// Fetch methods creates a list of network events for each container.
6868
func (m *MetricSet) Fetch(r mb.ReporterV2) error {
69-
stats, err := docker.FetchStats(m.dockerClient, m.Module().Config().Timeout)
69+
stats, err := docker.FetchStats(m.dockerClient, m.Module().Config().Timeout, false, m.Logger())
7070
if err != nil {
7171
return fmt.Errorf("failed to get docker stats: %w", err)
7272
}

‎metricbeat/module/docker/network_summary/network_summary.go

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -84,7 +84,7 @@ func New(base mb.BaseMetricSet) (mb.MetricSet, error) {
8484
// of an error set the Error field of mb.Event or simply call report.Error().
8585
func (m *MetricSet) Fetch(ctx context.Context, report mb.ReporterV2) error {
8686

87-
stats, err := docker.FetchStats(m.dockerClient, m.Module().Config().Timeout)
87+
stats, err := docker.FetchStats(m.dockerClient, m.Module().Config().Timeout, false, m.Logger())
8888
if err != nil {
8989
return fmt.Errorf("failed to get docker stats: %w", err)
9090
}

‎metricbeat/modules.d/docker.yml.disabled

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,9 @@
1818
# If set to true, replace dots in labels with `_`.
1919
#labels.dedot: false
2020

21+
# Docker module supports metrics collection from podman's Docker-compatible API. In case of podman set to true.
22+
# podman: false
23+
2124
# Skip metrics for certain device major numbers in docker/diskio.
2225
# Necessary on systems with software RAID, device mappers,
2326
# or other configurations where virtual disks will sum metrics from other disks.

‎x-pack/metricbeat/metricbeat.reference.yml

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -521,6 +521,9 @@ metricbeat.modules:
521521
# If set to true, replace dots in labels with `_`.
522522
#labels.dedot: false
523523

524+
# Docker module supports metrics collection from podman's docker compatible API. In case of podman set to true.
525+
# podman: false
526+
524527
# Skip metrics for certain device major numbers in docker/diskio.
525528
# Necessary on systems with software RAID, device mappers,
526529
# or other configurations where virtual disks will sum metrics from other disks.

0 commit comments

Comments
 (0)