Modify test and utils.py to fix flaky test #203
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The test
test_get_ip()
under theSanityTest
class insidetest_noipy.py
was found to be flaky, i.e. a test that both passes and fail despite no changes to the code or the tests itself.Using the pytest-flakefinder plugin, when the test was re-run more than once, it was failing. Upon looking at the code, the reason was because the the value of
HTTPBIN_URL
insideutils.py
was not being set back to its default value ofhttps://httpbin.org/ip
To fix it, I just changed the
test_get_ip()
test and have it be provided with the HTTPBIN_URL value instead of the value being read from the global variable.To reproduce flakiness:
pytest --flake-finder test/test_noipy.py::SanityTest::test_get_ip
After implementing the fix, all tests pass successfully, both with and without the pytest-flakefinder plugin being used.
In general, flaky tests are a pain to fix with regards to locating the root of the actual problem, so it's a good idea to fix them when you detect them. I'm aware that the tests might not be re-run at all, but this change makes the entire module more robust and future-proof so it's just a small fix for you to consider.