Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

baseline http vs conductor overhead benchmark #622

Open
wants to merge 13 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
104 changes: 104 additions & 0 deletions .github/workflows/baseline_http_bench.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,104 @@
name: Benchmark

on:
pull_request:
paths-ignore:
- "docs/**"
- "website/**"

jobs:
benchmark:
name: Benchmark
env:
K6_VERSION: 0.48.0
runs-on: ubuntu-22.04
steps:
- name: Checkout Code
uses: actions/checkout@v4

- name: Setup Rust
uses: actions-rs/toolchain@v1
with:
toolchain: stable
override: true

- name: Setup K6
run: |
sudo gpg -k
sudo gpg --no-default-keyring --keyring /usr/share/keyrings/k6-archive-keyring.gpg --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys C5AD17C747E3415A3642D57D77C6C491D6AC1D69
echo "deb [signed-by=/usr/share/keyrings/k6-archive-keyring.gpg] https://dl.k6.io/deb stable main" | sudo tee /etc/apt/sources.list.d/k6.list
sudo apt-get update
sudo apt-get install k6=${{ env.K6_VERSION }}

- name: Run Actix Web Server
uses: JarvusInnovations/background-action@v1
with:
run: cargo run --release
working-directory: ./baseline_http_bench/baseline_server
wait-on: http-get://127.0.0.1:4000/baseline
tail: true
wait-for: 5m
log-output-if: failure
log-output: true

# Benchmark Actix Web with k6
- name: Benchmark Actix Web
run: |
k6 run --summary-export actix_results.json baseline_http_bench/baseline_http.js

# Stop Actix Web server
- name: Stop Actix Web Server
run: pkill -f actix_web_server || true


# Build and run Conductor server in the background
- name: Run Conductor Server
uses: JarvusInnovations/background-action@v1
with:
run: cargo run --release --bin conductor ./test_config/mocked_config.yaml
wait-on: http-get://127.0.0.1:8000/baseline
tail: true
wait-for: 8m
log-output-if: failure
log-output: true

- name: Cooldown Period
run: |
echo "Cooling down for 30 seconds..."
sleep 30

# Benchmark Conductor with k6
- name: Benchmark Conductor
run: |
k6 run --summary-export conductor_results.json baseline_http_bench/conductor.js

# Generate Markdown report
- name: Generate Markdown Report
id: generate-report
run: |
# Extract values from JSON
ACTIX_RPS=$(jq '.metrics.http_reqs.rate' actix_results.json)
ACTIX_P95=$(jq '.metrics.http_req_duration["p(95)"]' actix_results.json)
CONDUCTOR_RPS=$(jq '.metrics.http_reqs.rate' conductor_results.json)
CONDUCTOR_P95=$(jq '.metrics.http_req_duration["p(95)"]' conductor_results.json)

# Handle potential nulls or missing values
ACTIX_RPS_ROUNDED=$(printf "%.0f" ${ACTIX_RPS:-0})
ACTIX_P95_ROUNDED=$(printf "%.0f" ${ACTIX_P95:-0})
CONDUCTOR_RPS_ROUNDED=$(printf "%.0f" ${CONDUCTOR_RPS:-0})
CONDUCTOR_P95_ROUNDED=$(printf "%.0f" ${CONDUCTOR_P95:-0})

# Generate Markdown report
echo "## Pure HTTP vs Conductor Overhead Benchmark Results" > benchmark_results.md
echo "| Implementation | Requests/sec | P95 Latency (ms) |" >> benchmark_results.md
echo "|----------------|--------------|------------------|" >> benchmark_results.md
echo "| Pure HTTP (Actix Web) | $ACTIX_RPS_ROUNDED | $ACTIX_P95_ROUNDED |" >> benchmark_results.md
echo "| Conductor | $CONDUCTOR_RPS_ROUNDED | $CONDUCTOR_P95_ROUNDED |" >> benchmark_results.md

# Post Comment on PR
- name: Comment on Pull Request
uses: thollander/actions-comment-pull-request@v2
if: ${{ github.event_name == 'pull_request' }}
with:
filePath: ./benchmark_results.md
comment_tag: benchmark-results
2 changes: 1 addition & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
[workspace]
resolver = "2"
members = ["bin/*", "libs/*", "plugins/*"]
exclude = ["bin/npm", "tests/test-server"]
exclude = ["bin/npm", "tests/test-server", "baseline_http_bench/baseline_server"]

[workspace.dependencies]
tokio = "1.41.0"
Expand Down
25 changes: 25 additions & 0 deletions baseline_http_bench/baseline_http.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
import http from 'k6/http'
import { check, sleep } from 'k6'

export let options = {
thresholds: {
http_req_duration: ['p(95)<500'],
http_req_failed: ['rate<0.01'],
},
stages: [
{ duration: '20s', target: 1000 },
{ duration: '1m', target: 1000 },
{ duration: '20s', target: 5000 },
{ duration: '2m', target: 5000 },
{ duration: '20s', target: 0 },
],
}

export default function () {
const url = 'http://127.0.0.1:4000/baseline'
const res = http.get(url)

check(res, { 'status is 200': (r) => r.status === 200 })

sleep(0.1)
}
Loading
Loading