Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

correct the usage of ephemeral storage volumes in the eviction message #130439

Open
Chaunceyctx opened this issue Feb 26, 2025 · 1 comment · May be fixed by #130446
Open

correct the usage of ephemeral storage volumes in the eviction message #130439

Chaunceyctx opened this issue Feb 26, 2025 · 1 comment · May be fixed by #130446
Labels
kind/bug Categorizes issue or PR as related to a bug. priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. sig/node Categorizes an issue or PR as relevant to SIG Node. triage/accepted Indicates an issue or PR is ready to be actively worked on.

Comments

@Chaunceyctx
Copy link
Contributor

What happened?

In k8s 1.32 cluster, I created a pod with emptydir volume and continuously wrote data into the emptydir volume to trigger node eviction. However, the event generated during the eviction is as follows:

115s        Warning   Evicted                       pod/test-xxxxxsas-b966b9bd4-pzbbc                        The node was low on resource: ephemeral-storage. Threshold quantity: 94576558032, available: 92034400Ki. Container container-1 was using 122Ki, request is 0, has larger consumption of ephemeral-storage.

The reported 122Ki value only includes the usage of rootfs and logs, without accounting for the usage of volumes such as emptydir.

usage = resource.NewQuantity(int64(*containerStats.Rootfs.UsedBytes+*containerStats.Logs.UsedBytes), resource.BinarySI)

This may lead to user confusion: why is the pod being evicted when it appears to be using so little ephemeral storage?
Therefore, it is necessary to include the consideration of emptydir in the eviction message.

/sig node
friendly ping @pacoxu @kannon92

What did you expect to happen?

the eviction message contains the correct usage of ephemeral storage volumes

How can we reproduce it (as minimally and precisely as possible)?

  1. create a pod with emptydir volume
  2. continuously write data into this volume directory to trigger node eviction

Anything else we need to know?

No response

Kubernetes version

$ kubectl version
# paste output here

Cloud provider

OS version

# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here

Install tools

Container runtime (CRI) and version (if applicable)

Related plugins (CNI, CSI, ...) and versions (if applicable)

@Chaunceyctx Chaunceyctx added the kind/bug Categorizes issue or PR as related to a bug. label Feb 26, 2025
@k8s-ci-robot k8s-ci-robot added the sig/node Categorizes an issue or PR as relevant to SIG Node. label Feb 26, 2025
@k8s-ci-robot k8s-ci-robot added the needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. label Feb 26, 2025
@haircommander haircommander moved this from Triage to Triaged in SIG Node Bugs Feb 26, 2025
@haircommander
Copy link
Contributor

/triage accepted
/priority important-longterm

@k8s-ci-robot k8s-ci-robot added triage/accepted Indicates an issue or PR is ready to be actively worked on. priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Feb 26, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Categorizes issue or PR as related to a bug. priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. sig/node Categorizes an issue or PR as relevant to SIG Node. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
Status: Triaged
Development

Successfully merging a pull request may close this issue.

3 participants