Fuzzing.

[collinsonindo]

It would be nice if we could test the library for resilience against arbitrary input.

I've tried to eyeball the code that does parsing looking for something that may be a potential risk but so far found none.

But would be better if that was automated.

I propose something like libfuzzer and calling the cli endpoints with maliciously crafted input files to see if it triggers any issues

Would a CI running libfuzzer be a welcome pull request?

[jesperpedersen]

Yes, type it up and lets see it - start small and we can build on it.

I have been using https://github.com/jesperpedersen/libfaults during early development for something similar.

[collinsonindo]

Hi, this seems a bit harder than it looks,

Specifically I wanted to test if the configuration parsers are safe, the problem is the api doesn't expose any way for raw parsing of characters.

Using libfuzzer, we specifically have code like

#include <stdint.h>
#include <stddef.h>

extern "C" int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size){

}

Where data is randomly generated bytes. This one is already compiling and running(although needs clang for compiling)

But then we have coverage testing where I provide it with a seed corpus, which for our case is contents of a valid file which then the fuzzer modifies it randomly and passes it to the reader.
Now the problem as mentioned is the library does not expose an api for parsing raw bytes in memory, only for files. So I'm a bit stuck on the way forward.

A tutorial can be found here

[jesperpedersen]

Nobody said it was going to be easy :)

You are running into the classic problem; the testing library assumes requirements of the application itself - which aren't true. We are not going to expose functions in the API just to have them tested, so we need to find another way.

F.ex. having a function that reads from memory, and a function that reads from a file - then a common function that does the work (but that function is static for sure). So, you need to figure out how to transform that function at run-time into something you can test - maybe look into the LLVM byte code API...

Idea is good, implementation may be very hard.

[collinsonindo]

Yea, it doesn't make sense to expose a new api for testing.

Was thinking of writing contents to file, and then calling pgagroal_read_configuration on that file for testing, the function is exposed as a library symbol, so it would hopefully work

[collinsonindo]

Okay managed to get it to work,

https://github.com/collinsonindo/pgagroal/tree/fuzzer

Fuzzer code is at https://github.com/collinsonindo/pgagroal/blob/fuzzer/fuzz/target.c

Needs a clang compiler to work, gcc doesn't support -fsanitize=fuzzer

to compile cd into fuzz directory and then, use

clang -g -O1 -fsanitize=fuzzer -std=c17 ./target.c -lpgagroal -o ./pgagroal-fuzzer

And then ./pgagroal-fuzzer ./seeds

which will use some seeds as a starting point.

Output should be a lot of

pgagroal-fuzzer: No main configuration section [pgagroal] found in file <./pgagoral-fuzzer.conf>
pgagroal-fuzzer: No main configuration section [pgagroal] found in file <./pgagoral-fuzzer.conf>
pgagroal-fuzzer: No main configuration section [pgagroal] found in file <./pgagoral-fuzzer.conf>
pgagroal-fuzzer: Key <> with value <l]roroal]> out of any section (line 1 of file <./pgagoral-fuzzer.conf>)
pgagroal-fuzzer: No main configuration section [pgagroal] found in file <./pgagoral-fuzzer.conf>
pgagroal-fuzzer: Key <> with value <l]roroapg�groall]> out of any section (line 1 of file <./pgagoral-fuzzer.conf>)
pgagroal-fuzzer: No main configuration section [pgagroal] found in file <./pgagoral-fuzzer.conf>
pgagroal-fuzzer: No main configuration section [pgagroal] found in file <./pgagoral-fuzzer.conf>
pgagroal-fuzzer: No main configuration section [pgagroal] found in file <./pgagoral-fuzzer.conf>
pgagroal-fuzzer: No main configuration section [pgagroal] found in file <./pgagoral-fuzzer.conf>

But it should run.

Is there a reason we print to stdout on no configuration instead of logging?

[jesperpedersen]

We should do this in src/test/ and have cmake integration that only compiles if clang is used. Then a CI job can execute it when using clang. Lets start with those changes.

Logging has to be initialized before we can use it - maybe there are call stacks where we can use logging instead. Create separate pull requests for those