Skip to content

Commit

Permalink
Update README and add LICENSE (#32)
Browse files Browse the repository at this point in the history
- Added missing `LICENSE` file
- Moved lcov example to a separate file
- Updated `README`

---------

Co-authored-by: Karol Sewilo <karol.sewilo@swmansion.com>
Co-authored-by: Karol Sewiło <95349104+ksew1@users.noreply.github.com>
Co-authored-by: Tomasz Rejowski <34059895+Arcticae@users.noreply.github.com>
  • Loading branch information
4 people authored Sep 3, 2024
1 parent 8ad49bf commit 7e17833
Show file tree
Hide file tree
Showing 3 changed files with 236 additions and 134 deletions.
21 changes: 21 additions & 0 deletions LICENSE
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
MIT License

Copyright (c) 2024 Software Mansion

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
209 changes: 75 additions & 134 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,20 @@
# cairo-coverage

`cairo-coverage` is a utility designed to generate coverage reports for code written in the Cairo programming language.

> ⚠️ **IMPORTANT**:
> Please note that this repository is actively being developed and is currently in an alpha release stage.
> If you encounter any issues, please report them to us via the issues
> tab on our GitHub [repository](https://github.com/software-mansion/cairo-coverage).
>
> We currently don't support:
> - Branch coverage
> - Contracts
>
> Things that might not work as expected:
> - Macros coverage
> - Counters for how many times line was executed
## Installation

To install the latest stable version of `cairo-coverage`, run:
Expand All @@ -14,149 +29,75 @@ If you want to install a specific version, run the following command with the re
curl -L https://raw.githubusercontent.com/software-mansion/cairo-coverage/main/scripts/install.sh | sh -s -- v0.1.0
```

## Example

Let's say you have a following program
## Integrated tools

```rust
pub enum Operation {
Add,
Multiply,
}
- [x] [Starknet Foundry](https://github.com/foundry-rs/starknet-foundry) - check how to use it
with `cairo-coverage` [here](https://foundry-rs.github.io/starknet-foundry/testing/coverage.html)
- [ ] Cairo Test

## Usage

fn add(a: i32, b: i32) -> i32 {
a + b
}
### Generate coverage report

fn multiply(a: i32, b: i32) -> i32 {
a * b
}
To generate the report, run `cairo-coverage` with one or more `<PATH_TO_TRACE_DATA>` arguments, which specify the paths
to the json files containing the trace data to be used for generating the coverage report.
Optionally, you can provide the path to the output file using `--output-path <OUTPUT_PATH>`. If not specified, the
output file will default to being saved as `coverage.lcov`.

pub fn calculator(a: i32, b: i32, operation: Operation) -> i32 {
match operation {
Operation::Add => add(a, b),
Operation::Multiply => multiply(a, b),
}
}
### Output format

```
The generated output file is in the `lcov` format. For your convenience, you can find an explanation along with a simple
example of the `lcov` format [here](./lcov.md).

and you cover it with tests
#### Example

```rust
#[test]
fn calculator_add() {
assert_eq!(calculator(2, 3, Operation::Add), 5);
assert_eq!(calculator(-1, 1, Operation::Add), 0);
}
```shell
cairo-coverage path/to/trace/1.json path/to/trace/2.json path/to/trace/3.json
```

When running with `cairo-coverage` you will get a coverage report in `.lcov` format:

```lcov
TN:
SF:/path/to/your/project/src/lib.cairo
FN:7,8,add
FN:11,12,multiply
FN:15,20,calculator
FNDA:2,add
FNDA:0,multiply
FNDA:2,calculator
FNF:3
FNH:2
DA:7,2
DA:8,2
DA:11,0
DA:12,0
DA:15,2
DA:16,2
DA:17,2
DA:18,0
DA:19,2
LF:9
LH:6
end_of_record
### Coverage statistics

Various tools exist that can produce coverage statistics based on an lcov file. One of the most well-known among these
is [genhtml](https://github.com/linux-test-project/lcov/blob/master/bin/genhtml),
a tool that generates an html summary report using the coverage data in an lcov file, `genhtml` is part of
the [lcov package](https://github.com/linux-test-project/lcov/tree/master).

## Usage in GitHub actions

A variety of GitHub actions are available for analyzing coverage data for continuous integration purposes, which can
accept input in the form of an lcov file.
Examples of such actions include [CodeCov](https://github.com/codecov/codecov-action)
and [Coveralls](https://github.com/coverallsapp/github-action).

The example workflow below illustrates how to use coverage report generated by `snforge` in conjunction with `CodeCov`:

```yaml
name: Example cairo coverage workflow
on:
pull_request:
jobs:
check:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: foundry-rs/setup-snfoundry@v3

- name: Run tests and generate report
run: snforge test --coverage

- name: Upload coverage to Codecov
uses: codecov/codecov-action@v4
with:
file: ./coverage.lcov
token: ${{ secrets.CODECOV_TOKEN }}
```
Let's break it down

### Explanation

1. **General Information**
- **TN**: Test Name (optional, left empty)
- **SF**: Source File path `/path/to/your/project/src/lib.rs`

2. **Function Details**
- **FN:7,8,add**: The `add` function starts at line 7 and ends at line 8.
- **FN:11,12,multiply**: The `multiply` function starts at line 11 and ends at line 12.
- **FN:15,20,calculator**: The `calculator` function starts at line 15 and ends at line 20.

3. **Function Hit Details**
- **FNDA:2,add**: The `add` function was executed 2 times.
- **FNDA:0,multiply**: The `multiply` function was not executed in the tests.
- **FNDA:2,calculator**: The `calculator` function was executed 2 times.

4. **Function Summary**
- **FNF:3**: The number of functions found in the source file. There are 3 functions: `add`, `multiply`,
and `calculator`.
- **FNH:2**: The number of functions that were hit. 2 out of the 3 functions were
executed: `add` and `calculator`.

5. **Line Execution Details**
- **DA:\<line number\>,\<hit count\>**: Indicates whether each line was executed and how many times.

Here's the details of line coverage:

| Line | Hits | Explanation |
|------|------|--------------------------------------------------|
| 7 | 2 | Line 7 (definition of `add`) hit 2 times |
| 8 | 2 | Line 8 (body of `add`) hit 2 times |
| 11 | 0 | Line 11 (definition of `multiply`) not hit |
| 12 | 0 | Line 12 (body of `multiply`) not hit |
| 15 | 2 | Line 15 (definition of `calculator`) hit 2 times |
| 16 | 2 | Line 16 (start of `match`) hit 2 times |
| 17 | 2 | Line 17 (call to `add` in `match`) hit 2 times |
| 18 | 0 | Line 18 (call to `multiply` in `match`) not hit |
| 19 | 2 | Line 19 (end of `match`) hit 2 times |

6. **Line Coverage Summary**
- **LF:9**: The total number of lines in the file is 9.
- **LH:6**: The total number of lines that were hit (executed at least once) is 6.

7. **End of Record**
- **end_of_record**: Indicates the end of this coverage record.

### Summary

1. **Functions Coverage**:
- Three functions are defined.
- Two functions are executed at least once: `add` and `calculator`.
- One function (`multiply`) was not executed.

2. **Line Coverage**:
- 9 lines of code in total.
- 6 lines were executed during testing.
- Lines 11 and 12 (`multiply` function) and line 19 (match arm for `Multiply` operation) were not executed.

> 📝 **Note**
>
> This format is for a single file. If there are multiple files, each file's report will be concatenated together
> with `end_of_record` separating them, like this:
> ```lcov
> TN:
> SF:/path/to/your/project/src/operations.cairo
> FN:7,8,add
> FN:11,12,multiply
> FNDA:2,add
> FNDA:0,multiply
> ... other metrics ...
> end_of_record
> TN:
> SF:/path/to/your/project/src/lib.cairo
> FN:3,8,calculator
> FNDA:2,calculator
> ... other metrics ...
> LH:10
> end_of_record
> ```
## External tools integration
`cairo-coverage` is tool-agnostic which means that it accepts input from any tool. However, these tools need to generate
trace data in a specific expected format -
the same format which is accepted by the [cairo-profiler](https://github.com/software-mansion/cairo-profiler/tree/main).
For the exact code implementation of this format, please refer
to [this page](https://github.com/software-mansion/cairo-profiler/blob/main/crates/trace-data/src/lib.rs).
For a practical example of how this format appears in a file, consider
examining [this json file](./crates/cairo-coverage/tests/data/simple/snfoundry_trace/simple_tests::test_call::my_test.json).
140 changes: 140 additions & 0 deletions lcov.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,140 @@
## Example

Let's say you have a following program

```rust
pub enum Operation {
Add,
Multiply,
}


fn add(a: i32, b: i32) -> i32 {
a + b
}

fn multiply(a: i32, b: i32) -> i32 {
a * b
}

pub fn calculator(a: i32, b: i32, operation: Operation) -> i32 {
match operation {
Operation::Add => add(a, b),
Operation::Multiply => multiply(a, b),
}
}

```

and you cover it with tests

```rust
#[test]
fn calculator_add() {
assert(calculator(2, 3, Operation::Add) == 5, '');
assert(calculator(-1, 1, Operation::Add) == 0, '');
}
```

When running with `cairo-coverage` you will get a coverage report in `.lcov` format:

```lcov
TN:
SF:/Users/karol/Starkware/temp_pr/cairo-coverage/crates/cairo-coverage/tests/data/readme_example/src/lib.cairo
FN:8,8,readme_example::add
FNDA:4,readme_example::add
FN:16,18,readme_example::calculator
FNDA:4,readme_example::calculator
FN:12,12,readme_example::multiply
FNDA:0,readme_example::multiply
FNF:3
FNH:2
DA:8,4
DA:12,0
DA:16,2
DA:17,4
DA:18,0
LF:5
LH:3
end_of_record
```

Let's break it down

### Explanation

1. **General Information**
- **TN**: Test Name (optional, left empty)
- **SF**: Source File path `/path/to/your/project/src/lib.rs`

2. **Function Details**
- **FN:8,8,readme_example::add**: The `add` function starts at line 8 and ends at line 8.
- **FN:12,12,readme_example::multiply**: The `multiply` function starts at line 12 and ends at line 12.
- **FN:16,18,readme_example::calculator**: The `calculator` function starts at line 16 and ends at line 18.

3. **Function Hit Details**
- **FNDA:4,readme_example::add**: The `add` function was executed 4 times (Currently not accurate as expected is to
be 2).
- **FNDA:0,readme_example::multiply**: The `multiply` function was not executed in the tests.
- **FNDA:4,readme_example::calculator**: The `calculator` function was executed 4 times (Currently not accurate as
expected is to be 2).

4. **Function Summary**
- **FNF:3**: The number of functions found in the source file. There are 3 functions: `add`, `multiply`,
and `calculator`.
- **FNH:2**: The number of functions that were hit. 2 out of the 3 functions were
executed: `add` and `calculator`.

5. **Line Execution Details**
- **DA:\<line number\>,\<hit count\>**: Indicates whether each line was executed and how many times.

Here's the details of line coverage:

| Line | Hits | Explanation |
|------|------|-------------------------------------------------|
| 8 | 4 | Line 8 (body of `add`) hit 4 times |
| 12 | 0 | Line 12 (body of `multiply`) not hit |
| 16 | 2 | Line 16 (start of `match`) hit 2 times |
| 17 | 4 | Line 17 (call to `add` in `match`) hit 4 times |
| 18 | 0 | Line 18 (call to `multiply` in `match`) not hit |

6. **Line Coverage Summary**
- **LF:9**: The total number of lines in the file is 9.
- **LH:6**: The total number of lines that were hit (executed at least once) is 6.

7. **End of Record**
- **end_of_record**: Indicates the end of this coverage record.

### Summary

1. **Functions Coverage**:
- Three functions are defined.
- Two functions are executed at least once: `add` and `calculator`.
- One function (`multiply`) was not executed.

2. **Line Coverage**:
- 5 lines of code in total.
- 3 lines were executed during testing.
- Lines 11 and 12 (`multiply` function) and line 19 (match arm for `Multiply` operation) were not executed.

> 📝 **Note**
>
> This format is for a single file. If there are multiple files, each file's report will be concatenated together
> with `end_of_record` separating them, like this:
> ```lcov
> TN:
> SF:/path/to/your/project/src/operations.cairo
> FN:8,8,readme_example::add
> FNDA:4,readme_example::add
> FN:12,12,readme_example::multiply
> FNDA:0,readme_example::multiply
> ... other metrics ...
> end_of_record
> TN:
> SF:/path/to/your/project/src/lib.cairo
> FN:16,18,readme_example::calculator
> FNDA:4,readme_example::calculator
> ... other metrics ...
> LH:10
> end_of_record
> ```

0 comments on commit 7e17833

Please sign in to comment.