basic code coverage support with simplecov

DataDog · Aug 30, 2023 · 9e5d4c6 · 9e5d4c6
1 parent 038b95c
commit 9e5d4c6
Show file tree

Hide file tree

Showing 102 changed files with 869 additions and 3 deletions.
diff --git a/.simplecov b/.simplecov
@@ -0,0 +1,26 @@
+SimpleCov.enable_coverage :branch
+
+# Exclude code not maintained by this project
+SimpleCov.add_filter %r{/vendor/}
+SimpleCov.add_filter %r{/spec/support/}
+
+SimpleCov.coverage_dir ENV.fetch("COVERAGE_DIR", "coverage")
+
+# Each test run requires its own unique command_name.
+# When running `rake spec:test_name`, the test process doesn"t have access to the
+# rake task process, so we have come up with unique values ourselves.
+#
+# The current approach is to combine the ruby engine (ruby-2.7,jruby-9.2),
+# program name (rspec/test), command line arguments (--pattern spec/**/*_spec.rb),
+# and the loaded gemset.
+#
+# This should allow us to distinguish between runs with the same tests, but different gemsets:
+#   * appraisal ruby-3.2.0-rspec-3 rake spec:rspec
+#   * appraisal ruby-3.2.0-minitest-5 rake spec:minitest
+#
+# Subsequent runs of the same exact test suite should have the same command_name.
+command_line_arguments = ARGV.join(" ")
+gemset_hash = Digest::MD5.hexdigest Gem.loaded_specs.values.map { |x| "#{x.name}#{x.version}" }.sort.join
+ruby_engine = "#{RUBY_ENGINE}-#{RUBY_ENGINE_VERSION}"
+
+SimpleCov.command_name "#{ruby_engine}:#{gemset_hash}:#{$PROGRAM_NAME} #{command_line_arguments}"
diff --git a/Gemfile b/Gemfile
@@ -20,6 +20,9 @@ gem "standard", "~> 1.31.0"
 gem "yard"
 gem "webrick"
 
+gem "simplecov"
+gem "simplecov-cobertura", "~> 2.1.0"
+
 # type checking
 group :check do
   if RUBY_PLATFORM != "java"

diff --git a/README.md b/README.md
@@ -112,9 +112,7 @@ end
 
 ## Development
 
-After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
-
-To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and the created tag, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+See [development guide](/docs/DevelopmentGuide.md).
 
 ### Static typing
 

diff --git a/docs/DevelopmentGuide.md b/docs/DevelopmentGuide.md
@@ -0,0 +1,125 @@
+# Developing
+
+This guide covers some of the common how-tos and technical reference material for developing changes within the CI visibility library.
+
+## Table of Contents
+
+- [Setting up](#setting-up)
+- [Testing](#testing)
+  - [Writing tests](#writing-tests)
+  - [Running tests](#running-tests)
+  - [Checking test coverage](#checking-test-coverage)
+- [Checking code quality](#checking-code-quality)
+
+## Setting up
+
+*NOTE: To test locally, you must have `Docker` and `Docker Compose` installed. See the [Docker documentation](https://docs.docker.com/compose/install/) for details.*
+
+The CI visibility library uses Docker Compose to create a Ruby environment to develop and test within, as well
+as containers for any dependencies that might be necessary for certain kinds of tests.
+
+To start a development environment, choose a target Ruby version then run the following:
+
+```bash
+# In the root directory of the project...
+cd ~/datadog-ci-rb
+
+# Create and start a Ruby 3.2 test environment with its dependencies
+docker-compose run --rm datadog-ci-3.2 /bin/bash
+
+# Then inside the container (e.g. `root@2a73c6d8673e:/app`)...
+# Install the library dependencies
+bundle install
+
+# Install build targets
+bundle exec appraisal install
+```
+
+Then within this container you can [run tests](#running-tests), [check code quality](#checking-code-quality), or
+[run static typing checks](/docs/StaticTypingGuide.md).
+
+## Testing
+
+The test suite uses [RSpec](https://rspec.info/) tests to verify the correctness of both the core trace library and its integrations.
+
+### Writing tests
+
+New tests should be written as RSpec tests in the `spec/ddtrace` folder. Test files should generally mirror the structure of `lib`.
+
+All changes should be covered by a corresponding RSpec tests. Unit tests are preferred, and integration tests are accepted where appropriate (e.g. acceptance tests, verifying compatibility with datastores, etc) but should be kept to a minimum.
+
+#### Considerations for CI
+
+All tests should run in CI. When adding new `spec.rb` files, you may need to add a test task to ensure your test file is run in CI.
+
+- Ensure that there is a corresponding Rake task defined in `Rakefile` under the the `spec` namespace, whose pattern matches your test file.
+- Verify the Rake task is configured to run for the appropriate Ruby runtimes in the `ci` Rake task.
+
+### Running tests
+
+Simplest way to run tests is to run `bundle exec rake ci`, which will run the entire test suite, just as CI does.
+
+#### For the core library
+
+Run the tests for the core library with:
+
+```bash
+bundle exec rake spec:main
+```
+
+#### For integrations
+
+Integrations which interact with dependencies not listed in the `datadog-ci` gemspec will need to load these dependencies to run their tests.
+
+To do so, load the dependencies using [Appraisal](https://github.com/thoughtbot/appraisal). You can see a list of available appraisals with `bundle exec appraisal list`, or examine the `Appraisals` file.
+
+Then to run tests, prefix the test commain with the appraisal.
+
+`bundle exec appraisal <appraisal_name> rake <test_comand>`
+
+For example:
+
+```bash
+# Runs tests for rspec-3
+$ bundle exec appraisal ruby-3.2.0-rspec-3 rake spec:rspec
+# Runs tests for minitest-5
+$ bundle exec appraisal ruby-3.2.0-minitest-5 rake spec:minitest
+```
+
+#### Passing arguments to tests
+
+When running tests, you may pass additional args as parameters to the Rake task. For example:
+
+```bash
+# Runs minitest integration tests with seed 1234
+$ bundle exec appraisal ruby-3.2.0-minitest-5 rake spec:minitest'[--seed,1234]'
+```
+
+This can be useful for replicating conditions from CI or isolating certain tests.
+
+#### Checking test coverage
+
+You can check test code coverage by creating a report *after* running a test suite:
+
+```bash
+# Run the desired test suite
+$ bundle exec appraisal ruby-3.2.0-rspec-3 rake spec:rspec
+# Generate report for the suite executed
+$ bundle exec rake coverage:report
+```
+
+A webpage will be generated at `coverage/report/index.html` with the resulting report.
+
+Because you are likely not running all tests locally, your report will contain partial coverage results.
+You *must* check the CI step `coverage` for the complete test coverage report, ensuring coverage is not
+decreased.
+
+## Checking code quality
+
+This library uses [standardrb](https://github.com/standardrb/standard) to enforce code style and quality.
+
+To check, run:
+
+```bash
+bundle exec standardrb
+```
diff --git a/gemfiles/jruby_9.4.0.0_cucumber_3.gemfile b/gemfiles/jruby_9.4.0.0_cucumber_3.gemfile
@@ -13,6 +13,8 @@ gem "appraisal"
 gem "standard", "~> 1.31.0"
 gem "yard"
 gem "webrick"
+gem "simplecov"
+gem "simplecov-cobertura", "~> 2.1.0"
 gem "cucumber", "~> 3"
 
 group :check do

diff --git a/gemfiles/jruby_9.4.0.0_cucumber_3.gemfile.lock b/gemfiles/jruby_9.4.0.0_cucumber_3.gemfile.lock
@@ -40,6 +40,7 @@ GEM
       msgpack
     debase-ruby_core_source (3.2.1)
     diff-lcs (1.5.0)
+    docile (1.4.0)
     ffi (1.15.5-java)
     gherkin (5.1.0)
     json (2.6.3-java)
@@ -101,6 +102,15 @@ GEM
       rubocop (>= 1.7.0, < 2.0)
       rubocop-ast (>= 0.4.0)
     ruby-progressbar (1.13.0)
+    simplecov (0.22.0)
+      docile (~> 1.1)
+      simplecov-html (~> 0.11)
+      simplecov_json_formatter (~> 0.1)
+    simplecov-cobertura (2.1.0)
+      rexml
+      simplecov (~> 0.19)
+    simplecov-html (0.12.3)
+    simplecov_json_formatter (0.1.4)
     spoon (0.0.6)
       ffi
     standard (1.31.0)
@@ -134,6 +144,8 @@ DEPENDENCIES
   rspec
   rspec-collection_matchers
   rspec_junit_formatter
+  simplecov
+  simplecov-cobertura (~> 2.1.0)
   standard (~> 1.31.0)
   webrick
   yard

diff --git a/gemfiles/jruby_9.4.0.0_cucumber_4.gemfile b/gemfiles/jruby_9.4.0.0_cucumber_4.gemfile
@@ -13,6 +13,8 @@ gem "appraisal"
 gem "standard", "~> 1.31.0"
 gem "yard"
 gem "webrick"
+gem "simplecov"
+gem "simplecov-cobertura", "~> 2.1.0"
 gem "cucumber", "~> 4"
 
 group :check do

diff --git a/gemfiles/jruby_9.4.0.0_cucumber_4.gemfile.lock b/gemfiles/jruby_9.4.0.0_cucumber_4.gemfile.lock
@@ -60,6 +60,7 @@ GEM
       msgpack
     debase-ruby_core_source (3.2.1)
     diff-lcs (1.3)
+    docile (1.4.0)
     ffi (1.15.5-java)
     i18n (1.14.1)
       concurrent-ruby (~> 1.0)
@@ -128,6 +129,15 @@ GEM
       rubocop (>= 1.7.0, < 2.0)
       rubocop-ast (>= 0.4.0)
     ruby-progressbar (1.13.0)
+    simplecov (0.22.0)
+      docile (~> 1.1)
+      simplecov-html (~> 0.11)
+      simplecov_json_formatter (~> 0.1)
+    simplecov-cobertura (2.1.0)
+      rexml
+      simplecov (~> 0.19)
+    simplecov-html (0.12.3)
+    simplecov_json_formatter (0.1.4)
     spoon (0.0.6)
       ffi
     standard (1.31.0)
@@ -166,6 +176,8 @@ DEPENDENCIES
   rspec
   rspec-collection_matchers
   rspec_junit_formatter
+  simplecov
+  simplecov-cobertura (~> 2.1.0)
   standard (~> 1.31.0)
   webrick
   yard

diff --git a/gemfiles/jruby_9.4.0.0_cucumber_5.gemfile b/gemfiles/jruby_9.4.0.0_cucumber_5.gemfile
@@ -13,6 +13,8 @@ gem "appraisal"
 gem "standard", "~> 1.31.0"
 gem "yard"
 gem "webrick"
+gem "simplecov"
+gem "simplecov-cobertura", "~> 2.1.0"
 gem "cucumber", "~> 5"
 
 group :check do

diff --git a/gemfiles/jruby_9.4.0.0_cucumber_5.gemfile.lock b/gemfiles/jruby_9.4.0.0_cucumber_5.gemfile.lock
@@ -60,6 +60,7 @@ GEM
       msgpack
     debase-ruby_core_source (3.2.1)
     diff-lcs (1.5.0)
+    docile (1.4.0)
     ffi (1.15.5-java)
     i18n (1.14.1)
       concurrent-ruby (~> 1.0)
@@ -128,6 +129,15 @@ GEM
       rubocop (>= 1.7.0, < 2.0)
       rubocop-ast (>= 0.4.0)
     ruby-progressbar (1.13.0)
+    simplecov (0.22.0)
+      docile (~> 1.1)
+      simplecov-html (~> 0.11)
+      simplecov_json_formatter (~> 0.1)
+    simplecov-cobertura (2.1.0)
+      rexml
+      simplecov (~> 0.19)
+    simplecov-html (0.12.3)
+    simplecov_json_formatter (0.1.4)
     spoon (0.0.6)
       ffi
     standard (1.31.0)
@@ -166,6 +176,8 @@ DEPENDENCIES
   rspec
   rspec-collection_matchers
   rspec_junit_formatter
+  simplecov
+  simplecov-cobertura (~> 2.1.0)
   standard (~> 1.31.0)
   webrick
   yard

diff --git a/gemfiles/jruby_9.4.0.0_cucumber_6.gemfile b/gemfiles/jruby_9.4.0.0_cucumber_6.gemfile
@@ -13,6 +13,8 @@ gem "appraisal"
 gem "standard", "~> 1.31.0"
 gem "yard"
 gem "webrick"
+gem "simplecov"
+gem "simplecov-cobertura", "~> 2.1.0"
 gem "cucumber", "~> 6"
 
 group :check do

diff --git a/gemfiles/jruby_9.4.0.0_cucumber_6.gemfile.lock b/gemfiles/jruby_9.4.0.0_cucumber_6.gemfile.lock
@@ -61,6 +61,7 @@ GEM
       msgpack
     debase-ruby_core_source (3.2.1)
     diff-lcs (1.5.0)
+    docile (1.4.0)
     ffi (1.15.5-java)
     i18n (1.14.1)
       concurrent-ruby (~> 1.0)
@@ -132,6 +133,15 @@ GEM
       rubocop (>= 1.7.0, < 2.0)
       rubocop-ast (>= 0.4.0)
     ruby-progressbar (1.13.0)
+    simplecov (0.22.0)
+      docile (~> 1.1)
+      simplecov-html (~> 0.11)
+      simplecov_json_formatter (~> 0.1)
+    simplecov-cobertura (2.1.0)
+      rexml
+      simplecov (~> 0.19)
+    simplecov-html (0.12.3)
+    simplecov_json_formatter (0.1.4)
     spoon (0.0.6)
       ffi
     standard (1.31.0)
@@ -170,6 +180,8 @@ DEPENDENCIES
   rspec
   rspec-collection_matchers
   rspec_junit_formatter
+  simplecov
+  simplecov-cobertura (~> 2.1.0)
   standard (~> 1.31.0)
   webrick
   yard

diff --git a/gemfiles/jruby_9.4.0.0_cucumber_7.gemfile b/gemfiles/jruby_9.4.0.0_cucumber_7.gemfile
@@ -13,6 +13,8 @@ gem "appraisal"
 gem "standard", "~> 1.31.0"
 gem "yard"
 gem "webrick"
+gem "simplecov"
+gem "simplecov-cobertura", "~> 2.1.0"
 gem "cucumber", "~> 7"
 
 group :check do

diff --git a/gemfiles/jruby_9.4.0.0_cucumber_7.gemfile.lock b/gemfiles/jruby_9.4.0.0_cucumber_7.gemfile.lock
@@ -53,6 +53,7 @@ GEM
       msgpack
     debase-ruby_core_source (3.2.1)
     diff-lcs (1.5.0)
+    docile (1.4.0)
     ffi (1.15.5-java)
     json (2.6.3-java)
     language_server-protocol (3.17.0.3)
@@ -115,6 +116,15 @@ GEM
       rubocop (>= 1.7.0, < 2.0)
       rubocop-ast (>= 0.4.0)
     ruby-progressbar (1.13.0)
+    simplecov (0.22.0)
+      docile (~> 1.1)
+      simplecov-html (~> 0.11)
+      simplecov_json_formatter (~> 0.1)
+    simplecov-cobertura (2.1.0)
+      rexml
+      simplecov (~> 0.19)
+    simplecov-html (0.12.3)
+    simplecov_json_formatter (0.1.4)
     spoon (0.0.6)
       ffi
     standard (1.31.0)
@@ -150,6 +160,8 @@ DEPENDENCIES
   rspec
   rspec-collection_matchers
   rspec_junit_formatter
+  simplecov
+  simplecov-cobertura (~> 2.1.0)
   standard (~> 1.31.0)
   webrick
   yard

diff --git a/gemfiles/jruby_9.4.0.0_cucumber_8.gemfile b/gemfiles/jruby_9.4.0.0_cucumber_8.gemfile
@@ -13,6 +13,8 @@ gem "appraisal"
 gem "standard", "~> 1.31.0"
 gem "yard"
 gem "webrick"
+gem "simplecov"
+gem "simplecov-cobertura", "~> 2.1.0"
 gem "cucumber", "~> 8"
 
 group :check do