-
Notifications
You must be signed in to change notification settings - Fork 4
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Code coverage tooling #12
Merged
Merged
Changes from all commits
Commits
Show all changes
5 commits
Select commit
Hold shift + click to select a range
9e5d4c6
basic code coverage support with simplecov
anmarchenko 06fa574
enable codecov for circle ci builds
anmarchenko 9c89128
CircleCI: set parallel executors count to 8 because we don't have so …
anmarchenko 01ab3ae
try to make coverage pass
anmarchenko bd8cb35
do not generate jruby coverage report
anmarchenko File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,25 @@ | ||
| SimpleCov.enable_coverage :branch | ||
|
|
||
| SimpleCov.add_filter %r{/vendor/} | ||
| SimpleCov.add_filter %r{/spec/support/} | ||
|
|
||
| SimpleCov.coverage_dir ENV.fetch("COVERAGE_DIR", "coverage") | ||
|
|
||
| # Each test run requires its own unique command_name. | ||
| # When running `rake spec:test_name`, the test process doesn"t have access to the | ||
| # rake task process, so we have come up with unique values ourselves. | ||
| # | ||
| # The current approach is to combine the ruby engine (ruby-2.7,jruby-9.2), | ||
| # program name (rspec/test), command line arguments (--pattern spec/**/*_spec.rb), | ||
| # and the loaded gemset. | ||
| # | ||
| # This should allow us to distinguish between runs with the same tests, but different gemsets: | ||
| # * appraisal ruby-3.2.0-rspec-3 rake spec:rspec | ||
| # * appraisal ruby-3.2.0-minitest-5 rake spec:minitest | ||
| # | ||
| # Subsequent runs of the same exact test suite should have the same command_name. | ||
| command_line_arguments = ARGV.join(" ") | ||
| gemset_hash = Digest::MD5.hexdigest Gem.loaded_specs.values.map { |x| "#{x.name}#{x.version}" }.sort.join | ||
| ruby_engine = "#{RUBY_ENGINE}-#{RUBY_ENGINE_VERSION}" | ||
|
|
||
| SimpleCov.command_name "#{ruby_engine}:#{gemset_hash}:#{$PROGRAM_NAME} #{command_line_arguments}" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,125 @@ | ||
| # Developing | ||
|
|
||
| This guide covers some of the common how-tos and technical reference material for developing changes within the CI visibility library. | ||
|
|
||
| ## Table of Contents | ||
|
|
||
| - [Setting up](#setting-up) | ||
| - [Testing](#testing) | ||
| - [Writing tests](#writing-tests) | ||
| - [Running tests](#running-tests) | ||
| - [Checking test coverage](#checking-test-coverage) | ||
| - [Checking code quality](#checking-code-quality) | ||
|
|
||
| ## Setting up | ||
|
|
||
| *NOTE: To test locally, you must have `Docker` and `Docker Compose` installed. See the [Docker documentation](https://docs.docker.com/compose/install/) for details.* | ||
|
|
||
| The CI visibility library uses Docker Compose to create a Ruby environment to develop and test within, as well | ||
| as containers for any dependencies that might be necessary for certain kinds of tests. | ||
|
|
||
| To start a development environment, choose a target Ruby version then run the following: | ||
|
|
||
| ```bash | ||
| # In the root directory of the project... | ||
| cd ~/datadog-ci-rb | ||
|
|
||
| # Create and start a Ruby 3.2 test environment with its dependencies | ||
| docker-compose run --rm datadog-ci-3.2 /bin/bash | ||
|
|
||
| # Then inside the container (e.g. `root@2a73c6d8673e:/app`)... | ||
| # Install the library dependencies | ||
| bundle install | ||
|
|
||
| # Install build targets | ||
| bundle exec appraisal install | ||
| ``` | ||
|
|
||
| Then within this container you can [run tests](#running-tests), [check code quality](#checking-code-quality), or | ||
| [run static typing checks](/docs/StaticTypingGuide.md). | ||
|
|
||
| ## Testing | ||
|
|
||
| The test suite uses [RSpec](https://rspec.info/) tests to verify the correctness of both the core trace library and its integrations. | ||
|
|
||
| ### Writing tests | ||
|
|
||
| New tests should be written as RSpec tests in the `spec/ddtrace` folder. Test files should generally mirror the structure of `lib`. | ||
|
|
||
| All changes should be covered by a corresponding RSpec tests. Unit tests are preferred, and integration tests are accepted where appropriate (e.g. acceptance tests, verifying compatibility with datastores, etc) but should be kept to a minimum. | ||
|
|
||
| #### Considerations for CI | ||
|
|
||
| All tests should run in CI. When adding new `spec.rb` files, you may need to add a test task to ensure your test file is run in CI. | ||
|
|
||
| - Ensure that there is a corresponding Rake task defined in `Rakefile` under the the `spec` namespace, whose pattern matches your test file. | ||
| - Verify the Rake task is configured to run for the appropriate Ruby runtimes in the `ci` Rake task. | ||
|
|
||
| ### Running tests | ||
|
|
||
| Simplest way to run tests is to run `bundle exec rake ci`, which will run the entire test suite, just as CI does. | ||
|
|
||
| #### For the core library | ||
|
|
||
| Run the tests for the core library with: | ||
|
|
||
| ```bash | ||
| bundle exec rake spec:main | ||
| ``` | ||
|
|
||
| #### For integrations | ||
|
|
||
| Integrations which interact with dependencies not listed in the `datadog-ci` gemspec will need to load these dependencies to run their tests. | ||
|
|
||
| To do so, load the dependencies using [Appraisal](https://github.com/thoughtbot/appraisal). You can see a list of available appraisals with `bundle exec appraisal list`, or examine the `Appraisals` file. | ||
|
|
||
| Then to run tests, prefix the test commain with the appraisal. | ||
|
|
||
| `bundle exec appraisal <appraisal_name> rake <test_comand>` | ||
|
|
||
| For example: | ||
|
|
||
| ```bash | ||
| # Runs tests for rspec-3 | ||
| $ bundle exec appraisal ruby-3.2.0-rspec-3 rake spec:rspec | ||
| # Runs tests for minitest-5 | ||
| $ bundle exec appraisal ruby-3.2.0-minitest-5 rake spec:minitest | ||
| ``` | ||
|
|
||
| #### Passing arguments to tests | ||
|
|
||
| When running tests, you may pass additional args as parameters to the Rake task. For example: | ||
|
|
||
| ```bash | ||
| # Runs minitest integration tests with seed 1234 | ||
| $ bundle exec appraisal ruby-3.2.0-minitest-5 rake spec:minitest'[--seed,1234]' | ||
| ``` | ||
|
|
||
| This can be useful for replicating conditions from CI or isolating certain tests. | ||
|
|
||
| #### Checking test coverage | ||
|
|
||
| You can check test code coverage by creating a report *after* running a test suite: | ||
|
|
||
| ```bash | ||
| # Run the desired test suite | ||
| $ bundle exec appraisal ruby-3.2.0-rspec-3 rake spec:rspec | ||
| # Generate report for the suite executed | ||
| $ bundle exec rake coverage:report | ||
| ``` | ||
|
|
||
| A webpage will be generated at `coverage/report/index.html` with the resulting report. | ||
|
|
||
| Because you are likely not running all tests locally, your report will contain partial coverage results. | ||
| You *must* check the CI step `coverage` for the complete test coverage report, ensuring coverage is not | ||
| decreased. | ||
|
|
||
| ## Checking code quality | ||
|
|
||
| This library uses [standardrb](https://github.com/standardrb/standard) to enforce code style and quality. | ||
|
|
||
| To check, run: | ||
|
|
||
| ```bash | ||
| bundle exec standardrb | ||
| ``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great! Do need to use our fork for simplecov. 👍
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Instead of using our fork I disabled jruby coverage reporting as jruby support is questionable for this gem