diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..10b6cc1 --- /dev/null +++ b/LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2024 Software Mansion + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/README.md b/README.md index a754831..f752ec5 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,20 @@ # cairo-coverage +`cairo-coverage` is a utility designed to generate coverage reports for code written in the Cairo programming language. + +> ⚠️ **IMPORTANT**: +> Please note that this repository is actively being developed and is currently in an alpha release stage. +> If you encounter any issues, please report them to us via the issues +> tab on our GitHub [repository](https://github.com/software-mansion/cairo-coverage). +> +> We currently don't support: +> - Branch coverage +> - Contracts +> +> Things that might not work as expected: +> - Macros coverage +> - Counters for how many times line was executed + ## Installation To install the latest stable version of `cairo-coverage`, run: @@ -14,149 +29,75 @@ If you want to install a specific version, run the following command with the re curl -L https://raw.githubusercontent.com/software-mansion/cairo-coverage/main/scripts/install.sh | sh -s -- v0.1.0 ``` -## Example - -Let's say you have a following program +## Integrated tools -```rust -pub enum Operation { - Add, - Multiply, -} +- [x] [Starknet Foundry](https://github.com/foundry-rs/starknet-foundry) - check how to use it + with `cairo-coverage` [here](https://foundry-rs.github.io/starknet-foundry/testing/coverage.html) +- [ ] Cairo Test +## Usage -fn add(a: i32, b: i32) -> i32 { - a + b -} +### Generate coverage report -fn multiply(a: i32, b: i32) -> i32 { - a * b -} +To generate the report, run `cairo-coverage` with one or more `` arguments, which specify the paths +to the json files containing the trace data to be used for generating the coverage report. +Optionally, you can provide the path to the output file using `--output-path `. If not specified, the +output file will default to being saved as `coverage.lcov`. -pub fn calculator(a: i32, b: i32, operation: Operation) -> i32 { - match operation { - Operation::Add => add(a, b), - Operation::Multiply => multiply(a, b), - } -} +### Output format -``` +The generated output file is in the `lcov` format. For your convenience, you can find an explanation along with a simple +example of the `lcov` format [here](./lcov.md). -and you cover it with tests +#### Example -```rust -#[test] -fn calculator_add() { - assert_eq!(calculator(2, 3, Operation::Add), 5); - assert_eq!(calculator(-1, 1, Operation::Add), 0); -} +```shell +cairo-coverage path/to/trace/1.json path/to/trace/2.json path/to/trace/3.json ``` -When running with `cairo-coverage` you will get a coverage report in `.lcov` format: - -```lcov -TN: -SF:/path/to/your/project/src/lib.cairo -FN:7,8,add -FN:11,12,multiply -FN:15,20,calculator -FNDA:2,add -FNDA:0,multiply -FNDA:2,calculator -FNF:3 -FNH:2 -DA:7,2 -DA:8,2 -DA:11,0 -DA:12,0 -DA:15,2 -DA:16,2 -DA:17,2 -DA:18,0 -DA:19,2 -LF:9 -LH:6 -end_of_record +### Coverage statistics + +Various tools exist that can produce coverage statistics based on an lcov file. One of the most well-known among these +is [genhtml](https://github.com/linux-test-project/lcov/blob/master/bin/genhtml), +a tool that generates an html summary report using the coverage data in an lcov file, `genhtml` is part of +the [lcov package](https://github.com/linux-test-project/lcov/tree/master). + +## Usage in GitHub actions + +A variety of GitHub actions are available for analyzing coverage data for continuous integration purposes, which can +accept input in the form of an lcov file. +Examples of such actions include [CodeCov](https://github.com/codecov/codecov-action) +and [Coveralls](https://github.com/coverallsapp/github-action). + +The example workflow below illustrates how to use coverage report generated by `snforge` in conjunction with `CodeCov`: + +```yaml +name: Example cairo coverage workflow +on: + pull_request: +jobs: + check: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: foundry-rs/setup-snfoundry@v3 + + - name: Run tests and generate report + run: snforge test --coverage + + - name: Upload coverage to Codecov + uses: codecov/codecov-action@v4 + with: + file: ./coverage.lcov + token: ${{ secrets.CODECOV_TOKEN }} ``` -Let's break it down - -### Explanation - -1. **General Information** - - **TN**: Test Name (optional, left empty) - - **SF**: Source File path `/path/to/your/project/src/lib.rs` - -2. **Function Details** - - **FN:7,8,add**: The `add` function starts at line 7 and ends at line 8. - - **FN:11,12,multiply**: The `multiply` function starts at line 11 and ends at line 12. - - **FN:15,20,calculator**: The `calculator` function starts at line 15 and ends at line 20. - -3. **Function Hit Details** - - **FNDA:2,add**: The `add` function was executed 2 times. - - **FNDA:0,multiply**: The `multiply` function was not executed in the tests. - - **FNDA:2,calculator**: The `calculator` function was executed 2 times. - -4. **Function Summary** - - **FNF:3**: The number of functions found in the source file. There are 3 functions: `add`, `multiply`, - and `calculator`. - - **FNH:2**: The number of functions that were hit. 2 out of the 3 functions were - executed: `add` and `calculator`. - -5. **Line Execution Details** - - **DA:\,\**: Indicates whether each line was executed and how many times. - - Here's the details of line coverage: - - | Line | Hits | Explanation | - |------|------|--------------------------------------------------| - | 7 | 2 | Line 7 (definition of `add`) hit 2 times | - | 8 | 2 | Line 8 (body of `add`) hit 2 times | - | 11 | 0 | Line 11 (definition of `multiply`) not hit | - | 12 | 0 | Line 12 (body of `multiply`) not hit | - | 15 | 2 | Line 15 (definition of `calculator`) hit 2 times | - | 16 | 2 | Line 16 (start of `match`) hit 2 times | - | 17 | 2 | Line 17 (call to `add` in `match`) hit 2 times | - | 18 | 0 | Line 18 (call to `multiply` in `match`) not hit | - | 19 | 2 | Line 19 (end of `match`) hit 2 times | - -6. **Line Coverage Summary** - - **LF:9**: The total number of lines in the file is 9. - - **LH:6**: The total number of lines that were hit (executed at least once) is 6. - -7. **End of Record** - - **end_of_record**: Indicates the end of this coverage record. - -### Summary - -1. **Functions Coverage**: - - Three functions are defined. - - Two functions are executed at least once: `add` and `calculator`. - - One function (`multiply`) was not executed. - -2. **Line Coverage**: - - 9 lines of code in total. - - 6 lines were executed during testing. - - Lines 11 and 12 (`multiply` function) and line 19 (match arm for `Multiply` operation) were not executed. - -> 📝 **Note** -> -> This format is for a single file. If there are multiple files, each file's report will be concatenated together -> with `end_of_record` separating them, like this: -> ```lcov -> TN: -> SF:/path/to/your/project/src/operations.cairo -> FN:7,8,add -> FN:11,12,multiply -> FNDA:2,add -> FNDA:0,multiply -> ... other metrics ... -> end_of_record -> TN: -> SF:/path/to/your/project/src/lib.cairo -> FN:3,8,calculator -> FNDA:2,calculator -> ... other metrics ... -> LH:10 -> end_of_record -> ``` +## External tools integration + +`cairo-coverage` is tool-agnostic which means that it accepts input from any tool. However, these tools need to generate +trace data in a specific expected format - +the same format which is accepted by the [cairo-profiler](https://github.com/software-mansion/cairo-profiler/tree/main). +For the exact code implementation of this format, please refer +to [this page](https://github.com/software-mansion/cairo-profiler/blob/main/crates/trace-data/src/lib.rs). +For a practical example of how this format appears in a file, consider +examining [this json file](./crates/cairo-coverage/tests/data/simple/snfoundry_trace/simple_tests::test_call::my_test.json). diff --git a/lcov.md b/lcov.md new file mode 100644 index 0000000..12535db --- /dev/null +++ b/lcov.md @@ -0,0 +1,140 @@ +## Example + +Let's say you have a following program + +```rust +pub enum Operation { + Add, + Multiply, +} + + +fn add(a: i32, b: i32) -> i32 { + a + b +} + +fn multiply(a: i32, b: i32) -> i32 { + a * b +} + +pub fn calculator(a: i32, b: i32, operation: Operation) -> i32 { + match operation { + Operation::Add => add(a, b), + Operation::Multiply => multiply(a, b), + } +} + +``` + +and you cover it with tests + +```rust +#[test] +fn calculator_add() { + assert(calculator(2, 3, Operation::Add) == 5, ''); + assert(calculator(-1, 1, Operation::Add) == 0, ''); +} +``` + +When running with `cairo-coverage` you will get a coverage report in `.lcov` format: + +```lcov +TN: +SF:/Users/karol/Starkware/temp_pr/cairo-coverage/crates/cairo-coverage/tests/data/readme_example/src/lib.cairo +FN:8,8,readme_example::add +FNDA:4,readme_example::add +FN:16,18,readme_example::calculator +FNDA:4,readme_example::calculator +FN:12,12,readme_example::multiply +FNDA:0,readme_example::multiply +FNF:3 +FNH:2 +DA:8,4 +DA:12,0 +DA:16,2 +DA:17,4 +DA:18,0 +LF:5 +LH:3 +end_of_record +``` + +Let's break it down + +### Explanation + +1. **General Information** + - **TN**: Test Name (optional, left empty) + - **SF**: Source File path `/path/to/your/project/src/lib.rs` + +2. **Function Details** + - **FN:8,8,readme_example::add**: The `add` function starts at line 8 and ends at line 8. + - **FN:12,12,readme_example::multiply**: The `multiply` function starts at line 12 and ends at line 12. + - **FN:16,18,readme_example::calculator**: The `calculator` function starts at line 16 and ends at line 18. + +3. **Function Hit Details** + - **FNDA:4,readme_example::add**: The `add` function was executed 4 times (Currently not accurate as expected is to + be 2). + - **FNDA:0,readme_example::multiply**: The `multiply` function was not executed in the tests. + - **FNDA:4,readme_example::calculator**: The `calculator` function was executed 4 times (Currently not accurate as + expected is to be 2). + +4. **Function Summary** + - **FNF:3**: The number of functions found in the source file. There are 3 functions: `add`, `multiply`, + and `calculator`. + - **FNH:2**: The number of functions that were hit. 2 out of the 3 functions were + executed: `add` and `calculator`. + +5. **Line Execution Details** + - **DA:\,\**: Indicates whether each line was executed and how many times. + + Here's the details of line coverage: + + | Line | Hits | Explanation | + |------|------|-------------------------------------------------| + | 8 | 4 | Line 8 (body of `add`) hit 4 times | + | 12 | 0 | Line 12 (body of `multiply`) not hit | + | 16 | 2 | Line 16 (start of `match`) hit 2 times | + | 17 | 4 | Line 17 (call to `add` in `match`) hit 4 times | + | 18 | 0 | Line 18 (call to `multiply` in `match`) not hit | + +6. **Line Coverage Summary** + - **LF:9**: The total number of lines in the file is 9. + - **LH:6**: The total number of lines that were hit (executed at least once) is 6. + +7. **End of Record** + - **end_of_record**: Indicates the end of this coverage record. + +### Summary + +1. **Functions Coverage**: + - Three functions are defined. + - Two functions are executed at least once: `add` and `calculator`. + - One function (`multiply`) was not executed. + +2. **Line Coverage**: + - 5 lines of code in total. + - 3 lines were executed during testing. + - Lines 11 and 12 (`multiply` function) and line 19 (match arm for `Multiply` operation) were not executed. + +> 📝 **Note** +> +> This format is for a single file. If there are multiple files, each file's report will be concatenated together +> with `end_of_record` separating them, like this: +> ```lcov +> TN: +> SF:/path/to/your/project/src/operations.cairo +> FN:8,8,readme_example::add +> FNDA:4,readme_example::add +> FN:12,12,readme_example::multiply +> FNDA:0,readme_example::multiply +> ... other metrics ... +> end_of_record +> TN: +> SF:/path/to/your/project/src/lib.cairo +> FN:16,18,readme_example::calculator +> FNDA:4,readme_example::calculator +> ... other metrics ... +> LH:10 +> end_of_record +> ```