-
Notifications
You must be signed in to change notification settings - Fork 375
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Reduce test flakiness for telemetry request spec #3962
Conversation
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #3962 +/- ##
=======================================
Coverage 97.85% 97.86%
=======================================
Files 1305 1305
Lines 78224 78222 -2
Branches 3887 3887
=======================================
+ Hits 76549 76553 +4
+ Misses 1675 1669 -6
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
before do | ||
# The tests here assert on time equality after rounding to a second; | ||
# these assertions will fail if the code executes just before | ||
# a second boundary and the assertions run just after the second boundary. | ||
# Align the runs closer to the beginning of a second to reduce flakiness. | ||
sleep 0 until Time.now.usec < 750000 | ||
end |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In the profiler tests I have a few similar situations where I don't know exactly the expected timestamp.
My suggestion here, rather than the sleep/loop (which doesn't exactly guarantee the timings we need -- just makes it more likely to happen) would be to do something like:
let!(:before_time) { Time.now.to_i }
# ...
it do
is_expected.to eq(
api_version: api_version,
application: application,
debug: debug,
host: host,
payload: payload,
request_type: request_type,
runtime_id: runtime_id,
seq_id: seq_id,
tracer_time: be_between(before_time, Time.now.to_i),
)
end
This way we validate the timestamp is correct without the loop or any flakiness.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I implemented this in #3966.
Use range match for tracer time instead of rounding and requiring exact match to rounded time, which does not always work (see #3962).
Use range match for tracer time instead of rounding and requiring exact match to rounded time, which does not always work (see #3962).
Use range match for tracer time instead of rounding and requiring exact match to rounded time, which does not always work (see #3962).
Use range match for tracer time instead of rounding and requiring exact match to rounded time, which does not always work (see #3962).
* Remove flakiness from telemetry request spec Use range match for tracer time instead of rounding and requiring exact match to rounded time, which does not always work (see #3962). * use match instead of eq --------- Co-authored-by: Oleg Pudeyev <code@olegp.name>
What does this PR do?
This PR inserts a brief wait (up to 0.25 seconds) for telemetry request spec to avoid the code under test and the test code happening in different whole seconds, which would then fail a test assertion.
Motivation:
Repair the following observed CI failure:
Additional Notes:
How to test the change?
On my machine the failure can be reliably reproduced by changing the sleep into
The required time is the approximate time of the second test's execution subtracted from 1,000,000.
Unsure? Have a question? Request a review!