-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Initial benchmark API #42
Open
zyga
wants to merge
12
commits into
main
Choose a base branch
from
feature/benchmark
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This patch lays the groundwork for supporting micro-benchmarks inside libzt. Test suites can now visit benchmarks, in addition to test cases and other test suites. A benchmark is a function taking one argument of type zt_b, similar to zt_t for test cases. The typedef zt_b is a pointer to struct zt_benchmark, holding one parameter, a 64 bit counter, n, of desired number of iterations to execute. Internally libzt executes all benchmarks at least once, to ensure they do not crash. In verbose mode, when invoked with -v command line option, precise measurements are taken to compute the number of nanoseconds required to execute a single loop iteration. Timing is based on microsecond-accurate, portable, clock_t clock() function. There are several warm-up phases where the loop is executed enough times to take roughly ten milliseconds. In my crude measurements this stabilizes the result well enough to estimate the cost of a single iteration. Following that, benchmark.n is set to a value that should give about one second of execution. This is when final measurements are taken. I've experimented with several different ideas, and found significant noise in the early estimation phase, when the effective runtime was lower than 10ms, at one ms results were several orders of magnitude off the duration measured over 10ms. The duration of the complete test is currently over-exaggerated. I found no difference between desired runtime length of 1000ms and 100ms, suggesting there is some more room for improvement. There's a chance to improve accuracy by switching to non-portable, nanosecond-resolution APIs that internally fuel clock(), but this was not attempted yet. The code is not tested yet, manual pages are not complete but there is a small example of the new functionality. Signed-off-by: Zygmunt Krynicki <me@zygoon.pl>
Signed-off-by: Zygmunt Krynicki <me@zygoon.pl>
Signed-off-by: Zygmunt Krynicki <me@zygoon.pl>
This is not portable, a more portable fallback will follow. Signed-off-by: Zygmunt Krynicki <me@zygoon.pl>
Those are identical to the default case but defining them silences warning emitted by mscv by default.
The benchmark logic is implemented with QueryPerformanceFrequency and QueryPerformanceCounter
There is no guarantee that the loop will have at least one iteration.
Signed-off-by: Zygmunt Krynicki <me@zygoon.pl>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This patch lays the groundwork for supporting micro-benchmarks
inside libzt. Test suites can now visit benchmarks, in addition
to test cases and other test suites. A benchmark is a function
taking one argument of type zt_b, similar to zt_t for test cases.
The typedef zt_b is a pointer to struct zt_benchmark, holding one
parameter, a 64 bit counter, n, of desired number of iterations
to execute.
Internally libzt executes all benchmarks at least once, to ensure
they do not crash. In verbose mode, when invoked with -v command line
option, precise measurements are taken to compute the number of
nanoseconds required to execute a single loop iteration.
Timing is based on microsecond-accurate, portable, clock_t clock()
function. There are several warm-up phases where the loop is executed
enough times to take roughly ten milliseconds. In my crude measurements
this stabilizes the result well enough to estimate the cost of a single
iteration.
Following that, benchmark.n is set to a value that should give about
one second of execution. This is when final measurements are taken.
I've experimented with several different ideas, and found significant
noise in the early estimation phase, when the effective runtime was
lower than 10ms, at one ms results were several orders of magnitude
off the duration measured over 10ms.
The duration of the complete test is currently over-exaggerated.
I found no difference between desired runtime length of 1000ms
and 100ms, suggesting there is some more room for improvement.
There's a chance to improve accuracy by switching to non-portable,
nanosecond-resolution APIs that internally fuel clock(), but this
was not attempted yet.
The code is not tested yet, manual pages are not complete but there
is a small example of the new functionality.
Signed-off-by: Zygmunt Krynicki me@zygoon.pl