Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement a 120s timeout in case lctl process is taking too long #77

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

RDruon
Copy link
Contributor

@RDruon RDruon commented Sep 6, 2024

Demo below is using a timeout of 5s (instead of 120s from this PR):

[root@node ~]# RUST_LOG=debug ./lustrefs-exporter --port 34221
2024-09-06T15:49:34.617893Z  INFO lustrefs_exporter: Listening on http://0.0.0.0:34221/metrics
2024-09-06T15:49:41.238172Z DEBUG lustrefs_exporter: lctl jobstats timed out

[root@es18kxe-co-vm07 ~]# time curl http://localhost:34221/metrics | tail
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 75.1M    0 75.1M    0     0  15.0M      0 --:--:--  0:00:05 --:--:-- 16.7M
lustre_job_stats_total{operation="getattr",component="ost",target="testfs-OST0003",jobid="SLURM_JOB199002:0:client"} 0
lustre_job_stats_total{operation="setattr",component="ost",target="testfs-OST0003",jobid="SLURM_JOB199002:0:client"} 0
lustre_job_stats_total{operation="punch",component="ost",target="testfs-OST0003",jobid="SLURM_JOB199002:0:client"} 0
lustre_job_stats_total{operation="sync",component="ost",target="testfs-OST0003",jobid="SLURM_JOB199002:0:client"} 0
lustre_job_stats_total{operation="destroy",component="ost",target="testfs-OST0003",jobid="SLURM_JOB199002:0:client"} 0
lustre_job_stats_total{operation="create",component="ost",target="testfs-OST0003",jobid="SLURM_JOB199002:0:client"} 0
lustre_job_stats_total{operation="statfs",component="ost",target="testfs-OST0003",jobid="SLURM_JOB199002:0:client"} 0
lustre_job_stats_total{operation="get_info",component="ost",target="testfs-OST0003",jobid="SLURM_JOB199002:0:client"} 0
lustre_job_stats_total{operation="set_info",component="ost",target="testfs-OST0003",jobid="SLURM_JOB199002:0:client"} 0
lustre_job_stats_total{operation="quotactl",component="ost",target="testfs-OST0003",jobid="SLURM_JOB199002:0:client"} 0

real    0m5.014s
user    0m0.883s
sys     0m1.449s

@RDruon RDruon added the bug Something isn't working label Sep 6, 2024
@RDruon RDruon self-assigned this Sep 6, 2024
@RDruon RDruon requested a review from jgrund as a code owner September 6, 2024 16:04
Copy link

codecov bot commented Sep 6, 2024

Codecov Report

Attention: Patch coverage is 0% with 12 lines in your changes missing coverage. Please review.

Project coverage is 91.53%. Comparing base (fbb9f54) to head (181d1fd).

Files with missing lines Patch % Lines
lustrefs-exporter/src/main.rs 0.00% 12 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main      #77      +/-   ##
==========================================
- Coverage   91.73%   91.53%   -0.20%     
==========================================
  Files          41       41              
  Lines        5228     5236       +8     
  Branches     5228     5236       +8     
==========================================
- Hits         4796     4793       -3     
- Misses        397      405       +8     
- Partials       35       38       +3     
Flag Coverage Δ
2_14_0_ddn133 35.26% <ø> (ø)
2_14_0_ddn145 36.01% <ø> (ø)
all-tests 91.53% <0.00%> (-0.20%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link

github-actions bot commented Sep 6, 2024

Benchmark for 00d91d7

Click to view benchmark
Test Base PR %
jobstats 100 4.4±0.79ms 4.7±0.96ms +6.82%
jobstats 1000 42.2±6.69ms 46.0±7.90ms +9.00%

Copy link

github-actions bot commented Sep 6, 2024

Benchmark for 1933e6a

Click to view benchmark
Test Base PR %
jobstats 100 4.4±0.82ms 4.4±0.84ms 0.00%
jobstats 1000 44.9±8.17ms 43.3±7.94ms -3.56%

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant