Seed experiment #2025

tokatoka · 2024-08-13T13:32:14Z

This is a experiment to see how five fuzzers (afl, aflpp, libfuzzer, honggfuzz, libafl) behaves when supplied with a empty seeds and saturated seed corpus.

First I want do an empty seed experiment.
Later i will change it to use ossfuzz corpus

tokatoka · 2024-08-13T13:36:27Z

The command is

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-14-empty_seeds-1 --fuzzers aflplusplus_empty_seeds libafl_empty_seeds honggfuzz_empty_seeds libfuzzer_empty_seeds afl_empty_seeds

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-14-empty_seeds-2 --fuzzers aflplusplus libafl honggfuzz libfuzzer afl

@DonggeLiu
Could you run the command?

DonggeLiu · 2024-08-13T14:27:32Z

gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-empty_seeds --fuzzers aflplusplus libafl honggfuzz libfuzzer afl aflplusplus_empty_seeds libafl_empty_seeds honggfuzz_empty_seeds libfuzzer_empty_seeds afl_empty_seeds

Sure, but there are 2 issues:

Could you please follow the same date format in experiment names? 2024-08-14. This help us keep the cloud storage more organized.
Like last time, this experiment requests too many fuzzers. Could you please split them into 2 experiments. Let's keep #fuzzers <= 5 in each exp.

tokatoka · 2024-08-13T14:34:45Z

ok i edited and fixed the issues
#2025 (comment)

tokatoka · 2024-08-13T15:38:58Z

this is dependent on #2023 #2021

DonggeLiu · 2024-08-15T00:14:57Z

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-15-empty_seeds-1 --fuzzers aflplusplus_empty_seeds libafl_empty_seeds honggfuzz_empty_seeds libfuzzer_empty_seeds afl_empty_seeds

DonggeLiu · 2024-08-15T00:19:23Z

Ops, experiment name cannot contain _.

DonggeLiu · 2024-08-15T00:19:34Z

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-15-empty-seeds-1 --fuzzers aflplusplus_empty_seeds libafl_empty_seeds honggfuzz_empty_seeds libfuzzer_empty_seeds afl_empty_seeds

DonggeLiu · 2024-08-15T00:23:58Z

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-14-empty-seeds-2 --fuzzers aflplusplus libafl honggfuzz libfuzzer afl

tokatoka · 2024-08-16T10:59:22Z

it looks like it didn't run
can you check what is the reason?

DonggeLiu · 2024-08-17T01:10:08Z

2024-08-14-empty-seeds-2

it looks like it didn't run
can you check what is the reason?

I think it is because of the same issue I fixed in #2023
Now that you have updated your branch, let me re-run the exp.

DonggeLiu · 2024-08-17T01:10:36Z

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-17-empty-seeds-1 --fuzzers aflplusplus libafl honggfuzz libfuzzer afl

DonggeLiu · 2024-08-17T01:15:45Z

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-17-empty-seeds-2 --fuzzers aflplusplus_empty_seeds libafl_empty_seeds honggfuzz_empty_seeds libfuzzer_empty_seeds afl_empty_seeds

DonggeLiu · 2024-08-17T01:24:21Z

Experiment 2024-08-17-empty-seeds-1 data and results will be available later at:
The experiment data.
The experiment report.
The experiment report(experimental).

Experiment 2024-08-17-empty-seeds-2 data and results will be available later at:
The experiment data.
The experiment report.
The experiment report(experimental).

DonggeLiu · 2024-08-17T01:28:36Z

Hi @tokatoka, I started two test exp above.
If they work well, could you please rebase to adopt this change?
I forgot to revert temp changes in a previous PR.

Thanks!

tokatoka · 2024-08-19T10:45:17Z

hi, i merged the master branch to revert it.
now it's ready

tokatoka · 2024-08-20T14:32:00Z

now it's truly okay (with the green ci)
could you run the same command? @DonggeLiu

DonggeLiu · 2024-08-20T23:48:15Z

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-21-empty-seeds-1 --fuzzers aflplusplus libafl honggfuzz libfuzzer afl

DonggeLiu · 2024-08-20T23:57:17Z

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-21-empty-seeds-2 --fuzzers aflplusplus_empty_seeds libafl_empty_seeds honggfuzz_empty_seeds libfuzzer_empty_seeds afl_empty_seeds

tokatoka · 2024-08-21T06:43:43Z

hmm the experiment-data has data in its folder but the report is not generated
do you know the reason? @DonggeLiu

DonggeLiu · 2024-08-22T00:39:02Z

Let's debug this on the other PR #2028
This experiment also had many runtime crashes, for example.

tokatoka · 2024-08-28T01:50:57Z

resolved it

tokatoka · 2024-11-21T13:22:05Z

@DonggeLiu
Hello, Dongge.
I would like to continue this experiment.
From what i observed during last several months, I think the experiment will likely fail if i have multiple fuzzers run together, but it is fine if they are run separately.

Therefore can you run the previous command separately in 5 trials?
Same experiment, same workload, but we just don't run them in parallel.
1:

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-27-empty-seeds-1 --fuzzers aflplusplus_empty_seeds

2:

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-27-empty-seeds-1 --fuzzers libafl_empty_seeds

3:

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-27-empty-seeds-1 --fuzzers honggfuzz_empty_seeds

4:

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-27-empty-seeds-1 --fuzzers libfuzzer_empty_seeds

5:

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-27-empty-seeds-1 --fuzzers afl_empty_seeds

DonggeLiu · 2024-11-22T00:40:55Z

Same experiment, same workload, but we just don't run them in parallel.

Yep sure, you mean run benchmarks in parallel experiments but not in one experiment right?

DonggeLiu · 2024-11-22T00:41:08Z

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-11-22-empty-seeds-1 --fuzzers aflplusplus_empty_seeds

DonggeLiu · 2024-11-22T00:41:46Z

ops you need to add the dummy comment to trigger cloud exps : )

tokatoka · 2024-11-22T00:55:22Z

Yep sure, you mean run benchmarks in parallel experiments but not in one experiment right?

yes. so you'll comment the command 5 times. but with different fuzzers each time instead of 5 fuzzers at once.

ops you need to add the dummy comment to trigger cloud exps : )

done

DonggeLiu · 2024-11-22T01:59:07Z

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-11-22-empty-seeds-1 --fuzzers aflplusplus_empty_seeds

DonggeLiu · 2024-11-22T02:10:45Z

service/experiment-config.yaml

-preemptible_runners: true
+#cloud_compute_zone: us-central1-c
+experiment_filestore: /tmp/experiment-data
+report_filestore: /tmp/report-data


Did you forget to undo these changes?

yes, sorry. i was doing local exp. but i forgot to revert it.
now i reverted it to origin/main

DonggeLiu · 2024-11-22T02:11:28Z

Noting the exp commands for myself:

gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-11-22-empty-seeds-1 --fuzzers aflplusplus_empty_seeds

gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-11-22-empty-seeds-2 --fuzzers libafl_empty_seeds

gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-11-22-empty-seeds-3 --fuzzers honggfuzz_empty_seeds

gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-11-22-empty-seeds-4 --fuzzers libfuzzer_empty_seeds

gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-11-22-empty-seeds-5 --fuzzers afl_empty_seeds

DonggeLiu · 2024-11-22T02:49:03Z

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-11-22-empty-seeds-1 --fuzzers aflplusplus_empty_seeds

tokatoka · 2024-11-22T13:08:10Z

ok it seems it is running.
could you run the remaining 4 too?

and could you run all these 4 on the same day?

DonggeLiu · 2024-11-23T22:36:35Z

ok it seems it is running. could you run the remaining 4 too?

and could you run all these 4 on the same day?

Yep they can, I was distracted by other tasks.

DonggeLiu · 2024-11-23T22:37:16Z

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-11-24-empty-seeds-2 --fuzzers libafl_empty_seeds

DonggeLiu · 2024-11-23T22:47:59Z

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-11-24-empty-seeds-3 --fuzzers honggfuzz_empty_seeds

DonggeLiu · 2024-11-23T22:53:26Z

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-11-24-empty-seeds-4 --fuzzers libfuzzer_empty_seeds

DonggeLiu · 2024-11-24T05:03:33Z

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-11-24-empty-seeds-5 --fuzzers afl_empty_seeds

empty

ad7daed

dummy

caca3f0

libfuzzer empty seeds

546aae8

shutil

1d29183

tokatoka changed the title ~~Seed experiment.~~ Seed experiment Aug 13, 2024

Merge branch 'google_master' into seed_exp

fb3d116

tokatoka added 2 commits August 19, 2024 12:43

Merge branch 'master' into seed_exp

d2d850e

Merge branch 'seed_exp' of github.com:tokatoka/fuzzbench into seed_exp

1d8f1a3

tokatoka added 2 commits August 19, 2024 13:36

change to latest commit for the empty_seed fuzzers too

616cfc2

update to latest

71823ee

Merge branch 'master' into seed_exp

0f7640b

commit hash

0ef9a84

DonggeLiu mentioned this pull request Aug 28, 2024

Debug 2025 #2040

Open

tokatoka added 6 commits August 28, 2024 13:24

shutil

5a717af

push

b8eecbe

Merge remote-tracking branch 'origin/google_master' into seed_exp

a0dcaef

run.sh

a6b86d2

sk3

103f879

sk4

e20a0ab

tokatoka closed this Nov 20, 2024

tokatoka reopened this Nov 21, 2024

Update gcbrun_experiment.py

6292a62

DonggeLiu reviewed Nov 22, 2024

View reviewed changes

Update experiment-config.yaml

bf66bec

tokatoka closed this Jan 16, 2025

Seed experiment #2025

Seed experiment #2025

Conversation

tokatoka commented Aug 13, 2024 • edited Loading

tokatoka commented Aug 13, 2024 • edited Loading

DonggeLiu commented Aug 13, 2024

tokatoka commented Aug 13, 2024 • edited Loading

tokatoka commented Aug 13, 2024

DonggeLiu commented Aug 15, 2024

DonggeLiu commented Aug 15, 2024

DonggeLiu commented Aug 15, 2024

DonggeLiu commented Aug 15, 2024

tokatoka commented Aug 16, 2024 • edited Loading

DonggeLiu commented Aug 17, 2024

DonggeLiu commented Aug 17, 2024

DonggeLiu commented Aug 17, 2024

DonggeLiu commented Aug 17, 2024

DonggeLiu commented Aug 17, 2024

tokatoka commented Aug 19, 2024

tokatoka commented Aug 20, 2024

DonggeLiu commented Aug 20, 2024

DonggeLiu commented Aug 20, 2024

tokatoka commented Aug 21, 2024

DonggeLiu commented Aug 22, 2024 • edited Loading

tokatoka commented Aug 28, 2024

tokatoka commented Nov 21, 2024 • edited Loading

DonggeLiu commented Nov 22, 2024

DonggeLiu commented Nov 22, 2024

DonggeLiu commented Nov 22, 2024

tokatoka commented Nov 22, 2024 • edited Loading

DonggeLiu commented Nov 22, 2024

DonggeLiu Nov 22, 2024

Choose a reason for hiding this comment

tokatoka Nov 22, 2024

Choose a reason for hiding this comment

DonggeLiu commented Nov 22, 2024

DonggeLiu commented Nov 22, 2024

tokatoka commented Nov 22, 2024

DonggeLiu commented Nov 23, 2024

DonggeLiu commented Nov 23, 2024

DonggeLiu commented Nov 23, 2024

DonggeLiu commented Nov 23, 2024

DonggeLiu commented Nov 24, 2024

tokatoka commented Aug 13, 2024 •

edited

Loading

tokatoka commented Aug 13, 2024 •

edited

Loading

tokatoka commented Aug 13, 2024 •

edited

Loading

tokatoka commented Aug 16, 2024 •

edited

Loading

DonggeLiu commented Aug 22, 2024 •

edited

Loading

tokatoka commented Nov 21, 2024 •

edited

Loading

tokatoka commented Nov 22, 2024 •

edited

Loading