Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci: add extensive benchmark testing #927

Draft
wants to merge 14 commits into
base: main
Choose a base branch
from
Draft

ci: add extensive benchmark testing #927

wants to merge 14 commits into from

Conversation

wellwelwel
Copy link
Owner

@wellwelwel wellwelwel commented Jan 24, 2025

Closes #584. Closes #740. Closes #926.


🚧 WIP.

Check the #927 (comment) to see what's going on 💡

This comment was marked as off-topic.

Copy link
Contributor

github-actions bot commented Jan 24, 2025

🎖️ Benchmarks

🏃🏻‍♀️ 1/4 Execution Tests

Note

Focuses solely in execution, using a simple assert(true) or assert(false) from Node.js and searches for files in four levels of depth.

  • success: a suite of 5 tests that will pass.
  • failure: a suite of 5 tests that will fail.
  • balanced: a suite of 10 tests where 5 tests will fail and 5 tests will pass.
See Results

🃏 Jest

  • success
  🐷 Poku (e0ff757) ran
    5.16 ± 0.22 times faster than jest
See commands
source
node --experimental-vm-modules ./node_modules/jest/bin/jest.js "./test/execution/success/jest"
poku
node ./node_modules/poku/lib/bin/index.js "./test/execution/success/poku"
  • failure
  🐷 Poku (e0ff757) ran
    5.34 ± 0.24 times faster than jest
See commands
source
node --experimental-vm-modules ./node_modules/jest/bin/jest.js "./test/execution/failure/jest"
poku
node ./node_modules/poku/lib/bin/index.js "./test/execution/failure/poku"
  • balanced
  🐷 Poku (e0ff757) ran
    4.42 ± 0.07 times faster than jest
See commands
source
node --experimental-vm-modules ./node_modules/jest/bin/jest.js "./test/execution/balanced/jest"
poku
node ./node_modules/poku/lib/bin/index.js "./test/execution/balanced/poku"

⚡️ Vitest

  • success
  🐷 Poku (e0ff757) ran
    4.58 ± 0.06 times faster than vitest
See commands
source
node ./node_modules/vitest/vitest.mjs run "./test/execution/success/vitest"
poku
node ./node_modules/poku/lib/bin/index.js "./test/execution/success/poku"
  • failure
  🐷 Poku (e0ff757) ran
    4.62 ± 0.15 times faster than vitest
See commands
source
node ./node_modules/vitest/vitest.mjs run "./test/execution/failure/vitest"
poku
node ./node_modules/poku/lib/bin/index.js "./test/execution/failure/poku"
  • balanced
  🐷 Poku (e0ff757) ran
    4.31 ± 0.11 times faster than vitest
See commands
source
node ./node_modules/vitest/vitest.mjs run "./test/execution/balanced/vitest"
poku
node ./node_modules/poku/lib/bin/index.js "./test/execution/balanced/poku"

☕️ Mocha

  • success
  🐷 Poku (e0ff757) ran
    1.82 ± 0.04 times faster than mocha
See commands
source
node ./node_modules/mocha/bin/mocha.js --parallel "./test/execution/success/mocha/**"
poku
node ./node_modules/poku/lib/bin/index.js "./test/execution/success/poku"
  • failure
  🐷 Poku (e0ff757) ran
    1.81 ± 0.06 times faster than mocha
See commands
source
node ./node_modules/mocha/bin/mocha.js --parallel "./test/execution/failure/mocha/**"
poku
node ./node_modules/poku/lib/bin/index.js "./test/execution/failure/poku"
  • balanced
  🐷 Poku (e0ff757) ran
    1.12 ± 0.03 times faster than mocha
See commands
source
node ./node_modules/mocha/bin/mocha.js --parallel "./test/execution/balanced/mocha/**"
poku
node ./node_modules/poku/lib/bin/index.js "./test/execution/balanced/poku"

🐢 Node.js (built-in)

  • success
  node ran
    1.16 ± 0.04 times faster than 🐷 Poku (e0ff757)
See commands
source
node --test "./test/execution/success/node/**/**.spec.js"
poku
node ./node_modules/poku/lib/bin/index.js "./test/execution/success/poku"
  • failure
  node ran
    1.05 ± 0.05 times faster than 🐷 Poku (e0ff757)
See commands
source
node --test "./test/execution/failure/node/**/**.spec.js"
poku
node ./node_modules/poku/lib/bin/index.js "./test/execution/failure/poku"
  • balanced
  node ran
    1.11 ± 0.04 times faster than 🐷 Poku (e0ff757)
See commands
source
node --test "./test/execution/balanced/node/**/**.spec.js"
poku
node ./node_modules/poku/lib/bin/index.js "./test/execution/balanced/poku"

Important

Please take into consideration that benchmarks do not indicate the competitiveness of one over the other; rather, they serve as a metric to monitor and objectively assess the current performance state of the project.

Repository owner deleted a comment from github-actions bot Jan 24, 2025
@wellwelwel wellwelwel changed the title ci: prepare environment for extensive benchmark testing ci: add extensive benchmark testing Jan 24, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Improve benchmarks
1 participant