You've been handed the unpleasant task of performing an audit of code written by your company's least-favorite contractor, Shoddycorp's Cut-Rate Contracting. The program is designed to read in gift card files in a legacy binary file format, and then display them in either a simple text output format, or in JSON format. Unfortunately, Shoddycorp isn't returning your calls, so you'll have to make do with the comments in the file and your own testing.
Justin Cappos (JAC) and Brendan Dolan-Gavitt (BDG) have read through the code already. It's a mess. We've tried to annotate a few places that look suspicious, but there's no doubt that there are many bugs lurking in this code. Make no mistake, this is not an example of C code that you want to imitate.
In order to complete this assignment, you are required to use the git VCS. Before beginning to write your code, you should first install git and set up a git repository to store your code in. The git binaries can be installed by your local package manager or at https://git-scm.com/downloads. For a cheat-sheet of git commands, please see https://github.com/nyutandononline/CLI-Cheat-Sheet/blob/master/git-commands.md. Although we will not be checking your commit messages or git log, it is recommended that you write descriptive commit messages so that the evolution of your repository is easy to understand. For a guide to writing good commit messages, please read https://chris.beams.io/posts/git-commit/ and https://git.kernel.org/pub/scm/git/git.git/tree/Documentation/SubmittingPatches#n104 (lines 104 – 160).
After git is installed, you will want to configure your git user to sign your commits. This can be done by following the guide at https://git-scm.com/book/en/v2/Git-Tools-Signing-Your-Work.
After installing and configuring git, you should then create a GitHub account at https://github.com, and add an ssh key using the guide found at https://help.github.com/en/articles/adding-a-new-ssh-key-to-your-github-account.
After creating a GitHub account, you are ready to create a git repository for the project you will be working on. The recommended way to do this is to create a repository using the GitHub Web interface. This is recommended because it can automatically generate important files for you, such as the license for your project and the gitignore file to avoid unintentionally tracking unnecessary files. After you create the repository on GitHub, you can clone it to your local machine using the following command.
$ git clone git@github.com:<your_username>/<your_repository_name>.git
Be sure to make the repository private.
The next step is to set up Travis CI. To do this, go to https://travis-ci.org and authorize Travis to link with your GitHub account. After this, you should create a .travis.yml file in the root of your repository. For information on how to set up a .travis.yml file, you can read the Travis tutorial (https://docs.travis-ci.com/user/tutorial/) and the guide to using Travis with C and C++ (https://docs.travis-ci.com/user/languages/c/). Finally, you are required to set up Travis CI. This will take the form of creating a .travis.yml file which will contain the settings for Travis. More information about the .travis.yaml file for C projects can be found at https://docs.travis-ci.com/user/languages/c/. In Part 2 of this project you will be expanding the functionality of Travis by including tests.
Start off by copying the files from this repository into your own, and add them to git. The files you need are:
examplefile.gft giftcardexamplewriter.c giftcard.h giftcardreader.c Makefile
Read through the giftcardreader.c program (and its accompanying
header, giftcard.h) to get a feel for it. You should also try building
and running it with the included examplefile.gft file, to see what its
normal output is. You may find it helpful to use a debugger like gdb
to step through the program as it executes, to get an understanding of
its normal flow of control.
For this part, your job will be to find some flaws in the program, and then create test cases (i.e., binary gift cards) that expose flaws in the program. You should write:
- Two test cases,
crash1.gftandcrash2.gft, that cause the program to crash; each crash should have a different root cause. - One test case,
hang.gft, that causes the program to loop infinitely. (Hint: you may want to examine the "program" record type to find a bug that causes the program to loop infinitely.) - A text file,
bugs.txtexplaining the bug triggered by each of your three test cases.
To create your own test files, you may want to look at
giftcardexamplewriter.c. Although it is no better written than
giftcardreader.c, it should help you understand the basics of the file
format and how to produce gift card files.
Finally, fix the bugs that are triggered by your test cases, and verify that the program no longer crashes / hangs on your test cases. To make sure that these bugs don't come up again as the code evolves, have Travis automatically build and run the program on your test suite.
As discussed in class, an important part of understanding how well your
test suite exercises your program's behaviors is coverage. To start
off, measure the coverage that your program achieves with the test cases
you created in Part 2. To do this, you should build giftcardreader
with the --coverage option to gcc, run your test suite, and then
produce a coverage report using lcov (details on how to do this can be
found in the lecture slides).
You should notice that there are portions of the program that are uncovered (i.e., the code was not executed while processing your test suite). Pick two lines of code from the program that are currently not covered and create test cases that cover them.
An easy and effective way of finding crashes and getting higher coverage
in a program is to fuzz it with a fuzzer like AFL. Fuzz the program
using AFL, following the quick-start
instructions. To
make the fuzzing more effective, you should provide AFL
with all of the test files you have created in its input directory. Let
the fuzzer run for at least two hours, and then examine the test cases
(in the queue directory) and crashes/hangs (in the crashes and
hangs directories).
Add the generated test cases (but not the crashes or hangs) to your test suite, and produce a new coverage report. You should see that the tests generated by the fuzzer reach more parts of the gift card program.
Finally, pick two crashes/hangs and fix the bugs in the program that cause them. Then, re-run the program on all the crashes and hangs found by AFL. You should find that not all of the crashes found by AFL originally crash the program now---although AFL tries its best to figure out which crashes are caused by unique bugs, if often overcounts.
Add the generated tests to your repository and have Travis run them.
Note that depending on how long you ran the fuzzer and how fast your
machine is, there may be a lot of redundant test cases! To keep only the
ones that exercise new behavior in your program, you can use the
afl-tmin tool.
To complete the assignment, commit your updated code, your handwritten tests, the fuzzer-generated tests, and a brief writeup explaining the bugs you found and fixed in this part.
Total points: 100
Part 1 is worth 20 points:
- 10 points for signed commits
- 10 points for Travis configuration
Part 2 is worth 40 points:
- 15 points for your test cases and fixes
- 15 points for the bug writeup
- 10 points for Travis regression testing
Part 3 is worth 40 points:
- 10 points for handwritten tests
- 10 points for fuzzer-generated tests
- 10 points for your code fixes
- 10 points for writeup
On NYU Classes, submit a link to your GitHub repository. The repository should be private, and you should add both the instructor and TA's GitHub account as a contributor to give them access for grading.
For this section, your instructor is: moyix
Your TA is: Manan-bit
The repository should contain:
- Part 1
- Your .travis.yml
- At least one signed commit
- Part 2
- A directory named
part2that containscrash1.gft,crash2.gft,hang.gft, andwriteup.txt - An updated .travis.yml that runs your tests
- A commit with the fixed version of the code (if you like, this commit can also contain the files mentioned above)
- A directory named
- Part 3
- A directory named
part3that containscov1.gft,cov2.gft,fuzzer1.gft,fuzzer2.gft, andwriteup.txt - An updated .travis.yml that runs the new tests
- A commit with the fixed version of the code (if you like, this commit can also contain the files mentioned above)
- A directory named
Despite the fixes you've made, there are almost certainly still many bugs lurking in the program. Although it is possible to get to a secure program by repeatedly finding and fixing bugs (at least when the program is this small), it's a lot of work, and just because a fuzzer stops finding bugs doesn't mean that the program is bug-free!
Realistically, this program is probably unsalvageable in its current state. It would be better in this case to rewrite it from scratch, either in C using a very defensive programming style, or in a safer language like Python or Rust. Once this assignment is complete, we will release a cleanly-written version of the program (written in C) that should be relatively bug-free [1]. You'll notice that it's a lot more verbose, and checks for many more errors than the buggy version---writing safe C code is difficult!
[1] Although you are encouraged to try to prove us wrong by finding bugs in it!