Merge pull request #33 from DataDog/anmarchenko/agentless

[CIAPP-2959] CI visibility protocol and agentless mode support
DataDog · Oct 5, 2023 · d569b6c · d569b6c
2 parents f3be2c5 + db394f0
commit d569b6c
Show file tree

Hide file tree

Showing 187 changed files with 3,130 additions and 574 deletions.
diff --git a/Gemfile b/Gemfile
@@ -10,14 +10,16 @@ gem "ddtrace"
 
 gem "pry"
 gem "rake"
-gem "rspec"
 gem "os"
 
 gem "climate_control"
 
+gem "rspec"
 gem "rspec-collection_matchers"
 gem "rspec_junit_formatter"
 gem "appraisal"
+gem "timecop"
+
 gem "standard", "~> 1.31.0"
 
 gem "yard"

diff --git a/LICENSE-3rdparty.csv b/LICENSE-3rdparty.csv
@@ -1,2 +1,3 @@
 Component,Origin,License,Copyright
 dd-trace-rb,https://github.com/DataDog/dd-trace-rb,Apache 2.0,"Copyright 2016-Present Datadog, Inc."
+msgpack,https://rubygems.org/gems/msgpack,Apache-2.0,"Copyright (c) 2008-2015 Sadayuki Furuhashi"
diff --git a/README.md b/README.md
@@ -117,6 +117,70 @@ end
 | `service_name` | Service name used for `cucumber` instrumentation. | `'cucumber'` |
 | `operation_name` | Operation name used for `cucumber` instrumentation. Useful if you want rename automatic trace metrics e.g. `trace.#{operation_name}.errors`. | `'cucumber.test'` |
 
+## Agentless mode
+
+If you are using a cloud CI provider without access to the underlying worker nodes, such as GitHub Actions or CircleCI, configure the library to use the Agentless mode. For this, set the following environment variables:
+`DD_CIVISIBILITY_AGENTLESS_ENABLED=true (Required)` and `DD_API_KEY=your_secret_api_key (Required)`.
+
+Additionally, configure which [Datadog site](https://docs.datadoghq.com/getting_started/site/) you want to send data to:
+`DD_SITE=your.datadoghq.com` (datadoghq.com by default).
+
+Agentless mode can also be enabled via `Datadog.configure` (but don't forget to set DD_API_KEY environment variable):
+
+```ruby
+Datadog.configure { |c| c.ci.agentless_mode_enabled = true }
+```
+
+## Additional configuration
+
+### Add tracing instrumentations
+
+It can be useful to have rich tracing information about your tests that includes time spent performing database operations
+or other external calls like here:
+
+![Test trace with redis instrumented](./docs/screenshots/test-trace-with-redis.png)
+
+In order to achieve this you can configure ddtrace instrumentations in your configure block:
+
+```ruby
+Datadog.configure do |c|
+  #  ... ci configs and instrumentation here ...
+  c.instrument :redis
+  c.instrument :pg
+end
+```
+
+...or enable auto instrumentation in your test_helper/spec_helper:
+
+```ruby
+require "ddtrace/auto_instrument"
+```
+
+Note: in CI mode these traces are going to be submitted to CI Visibility,
+they will **not** show up in Datadog APM.
+
+For the full list of available instrumentations see [ddtrace documentation](https://github.com/DataDog/dd-trace-rb/blob/master/docs/GettingStarted.md)
+
+### Disabling startup logs
+
+Startup logs produce a report of tracing state when the application is initially configured.
+These logs are activated by default in test mode, if you don't want them you can disable this
+via `diagnostics.startup_logs.enabled = false` or `DD_TRACE_STARTUP_LOGS=0`.
+
+```ruby
+Datadog.configure { |c| c.diagnostics.startup_logs.enabled = false }
+```
+
+### Enabling debug mode
+
+Switching the library into debug mode will produce verbose, detailed logs about tracing activity, including any suppressed errors. This output can be helpful in identifying errors, confirming trace output, or catching HTTP transport issues.
+
+You can enable this via `diagnostics.debug = true` or `DD_TRACE_DEBUG=1`.
+
+```ruby
+Datadog.configure { |c| c.diagnostics.debug = true }
+```
+
 ## Contributing
 
 See [development guide](/docs/DevelopmentGuide.md), [static typing guide](docs/StaticTypingGuide.md) and [contributing guidelines](/CONTRIBUTING.md).

diff --git a/Steepfile b/Steepfile
@@ -4,6 +4,7 @@ target :lib do
   check "lib"
 
   ignore "lib/datadog/ci/configuration/settings.rb"
+  ignore "lib/datadog/ci/transport/gzip.rb"
 
   library "pathname"
   library "json"
@@ -18,4 +19,5 @@ target :lib do
   library "open3"
   library "rspec"
   library "cucumber"
+  library "msgpack"
 end
diff --git a/datadog-ci.gemspec b/datadog-ci.gemspec
@@ -39,4 +39,6 @@ Gem::Specification.new do |spec|
     ]].select { |fn| File.file?(fn) } # We don't want directories, only files
 
   spec.require_paths = ["lib"]
+
+  spec.add_dependency "msgpack"
 end
diff --git a/docs/screenshots/test-trace-with-redis.png b/docs/screenshots/test-trace-with-redis.png
diff --git a/gemfiles/jruby_9.4.0.0_cucumber_3.gemfile b/gemfiles/jruby_9.4.0.0_cucumber_3.gemfile
@@ -5,12 +5,13 @@ source "https://rubygems.org"
 gem "ddtrace"
 gem "pry"
 gem "rake"
-gem "rspec"
 gem "os"
 gem "climate_control"
+gem "rspec"
 gem "rspec-collection_matchers"
 gem "rspec_junit_formatter"
 gem "appraisal"
+gem "timecop"
 gem "standard", "~> 1.31.0"
 gem "yard"
 gem "webrick"

diff --git a/gemfiles/jruby_9.4.0.0_cucumber_3.gemfile.lock b/gemfiles/jruby_9.4.0.0_cucumber_3.gemfile.lock
@@ -2,6 +2,7 @@ PATH
   remote: ..
   specs:
     datadog-ci (0.1.1)
+      msgpack
 
 GEM
   remote: https://rubygems.org/
@@ -126,6 +127,7 @@ GEM
       lint_roller (~> 1.1)
       rubocop-performance (~> 1.19.0)
     thor (1.2.2)
+    timecop (0.9.8)
     unicode-display_width (2.4.2)
     webrick (1.8.1)
     yard (0.9.34)
@@ -149,6 +151,7 @@ DEPENDENCIES
   simplecov
   simplecov-cobertura (~> 2.1.0)
   standard (~> 1.31.0)
+  timecop
   webrick
   yard
 

diff --git a/gemfiles/jruby_9.4.0.0_cucumber_4.gemfile b/gemfiles/jruby_9.4.0.0_cucumber_4.gemfile
@@ -5,12 +5,13 @@ source "https://rubygems.org"
 gem "ddtrace"
 gem "pry"
 gem "rake"
-gem "rspec"
 gem "os"
 gem "climate_control"
+gem "rspec"
 gem "rspec-collection_matchers"
 gem "rspec_junit_formatter"
 gem "appraisal"
+gem "timecop"
 gem "standard", "~> 1.31.0"
 gem "yard"
 gem "webrick"

diff --git a/gemfiles/jruby_9.4.0.0_cucumber_4.gemfile.lock b/gemfiles/jruby_9.4.0.0_cucumber_4.gemfile.lock
@@ -2,6 +2,7 @@ PATH
   remote: ..
   specs:
     datadog-ci (0.1.1)
+      msgpack
 
 GEM
   remote: https://rubygems.org/
@@ -156,6 +157,7 @@ GEM
       ffi (~> 1.1)
     thor (1.2.2)
     thread_safe (0.3.6-java)
+    timecop (0.9.8)
     tzinfo (2.0.6)
       concurrent-ruby (~> 1.0)
     unicode-display_width (2.4.2)
@@ -181,6 +183,7 @@ DEPENDENCIES
   simplecov
   simplecov-cobertura (~> 2.1.0)
   standard (~> 1.31.0)
+  timecop
   webrick
   yard
 

diff --git a/gemfiles/jruby_9.4.0.0_cucumber_5.gemfile b/gemfiles/jruby_9.4.0.0_cucumber_5.gemfile
@@ -5,12 +5,13 @@ source "https://rubygems.org"
 gem "ddtrace"
 gem "pry"
 gem "rake"
-gem "rspec"
 gem "os"
 gem "climate_control"
+gem "rspec"
 gem "rspec-collection_matchers"
 gem "rspec_junit_formatter"
 gem "appraisal"
+gem "timecop"
 gem "standard", "~> 1.31.0"
 gem "yard"
 gem "webrick"

diff --git a/gemfiles/jruby_9.4.0.0_cucumber_5.gemfile.lock b/gemfiles/jruby_9.4.0.0_cucumber_5.gemfile.lock
@@ -2,6 +2,7 @@ PATH
   remote: ..
   specs:
     datadog-ci (0.1.1)
+      msgpack
 
 GEM
   remote: https://rubygems.org/
@@ -156,6 +157,7 @@ GEM
       ffi (~> 1.1)
     thor (1.2.2)
     thread_safe (0.3.6-java)
+    timecop (0.9.8)
     tzinfo (2.0.6)
       concurrent-ruby (~> 1.0)
     unicode-display_width (2.4.2)
@@ -181,6 +183,7 @@ DEPENDENCIES
   simplecov
   simplecov-cobertura (~> 2.1.0)
   standard (~> 1.31.0)
+  timecop
   webrick
   yard
 

diff --git a/gemfiles/jruby_9.4.0.0_cucumber_6.gemfile b/gemfiles/jruby_9.4.0.0_cucumber_6.gemfile
@@ -5,12 +5,13 @@ source "https://rubygems.org"
 gem "ddtrace"
 gem "pry"
 gem "rake"
-gem "rspec"
 gem "os"
 gem "climate_control"
+gem "rspec"
 gem "rspec-collection_matchers"
 gem "rspec_junit_formatter"
 gem "appraisal"
+gem "timecop"
 gem "standard", "~> 1.31.0"
 gem "yard"
 gem "webrick"

diff --git a/gemfiles/jruby_9.4.0.0_cucumber_6.gemfile.lock b/gemfiles/jruby_9.4.0.0_cucumber_6.gemfile.lock
@@ -2,6 +2,7 @@ PATH
   remote: ..
   specs:
     datadog-ci (0.1.1)
+      msgpack
 
 GEM
   remote: https://rubygems.org/
@@ -160,6 +161,7 @@ GEM
       ffi (~> 1.1)
     thor (1.2.2)
     thread_safe (0.3.6-java)
+    timecop (0.9.8)
     tzinfo (2.0.6)
       concurrent-ruby (~> 1.0)
     unicode-display_width (2.4.2)
@@ -185,6 +187,7 @@ DEPENDENCIES
   simplecov
   simplecov-cobertura (~> 2.1.0)
   standard (~> 1.31.0)
+  timecop
   webrick
   yard
 

diff --git a/gemfiles/jruby_9.4.0.0_cucumber_7.gemfile b/gemfiles/jruby_9.4.0.0_cucumber_7.gemfile
@@ -5,12 +5,13 @@ source "https://rubygems.org"
 gem "ddtrace"
 gem "pry"
 gem "rake"
-gem "rspec"
 gem "os"
 gem "climate_control"
+gem "rspec"
 gem "rspec-collection_matchers"
 gem "rspec_junit_formatter"
 gem "appraisal"
+gem "timecop"
 gem "standard", "~> 1.31.0"
 gem "yard"
 gem "webrick"

diff --git a/gemfiles/jruby_9.4.0.0_cucumber_7.gemfile.lock b/gemfiles/jruby_9.4.0.0_cucumber_7.gemfile.lock
@@ -2,6 +2,7 @@ PATH
   remote: ..
   specs:
     datadog-ci (0.1.1)
+      msgpack
 
 GEM
   remote: https://rubygems.org/
@@ -142,6 +143,7 @@ GEM
     sys-uname (1.2.3)
       ffi (~> 1.1)
     thor (1.2.2)
+    timecop (0.9.8)
     unicode-display_width (2.4.2)
     webrick (1.8.1)
     yard (0.9.34)
@@ -165,6 +167,7 @@ DEPENDENCIES
   simplecov
   simplecov-cobertura (~> 2.1.0)
   standard (~> 1.31.0)
+  timecop
   webrick
   yard
 

diff --git a/gemfiles/jruby_9.4.0.0_cucumber_8.gemfile b/gemfiles/jruby_9.4.0.0_cucumber_8.gemfile
@@ -5,12 +5,13 @@ source "https://rubygems.org"
 gem "ddtrace"
 gem "pry"
 gem "rake"
-gem "rspec"
 gem "os"
 gem "climate_control"
+gem "rspec"
 gem "rspec-collection_matchers"
 gem "rspec_junit_formatter"
 gem "appraisal"
+gem "timecop"
 gem "standard", "~> 1.31.0"
 gem "yard"
 gem "webrick"

diff --git a/gemfiles/jruby_9.4.0.0_cucumber_8.gemfile.lock b/gemfiles/jruby_9.4.0.0_cucumber_8.gemfile.lock
@@ -2,6 +2,7 @@ PATH
   remote: ..
   specs:
     datadog-ci (0.1.1)
+      msgpack
 
 GEM
   remote: https://rubygems.org/
@@ -136,6 +137,7 @@ GEM
     sys-uname (1.2.3)
       ffi (~> 1.1)
     thor (1.2.2)
+    timecop (0.9.8)
     unicode-display_width (2.4.2)
     webrick (1.8.1)
     yard (0.9.34)
@@ -159,6 +161,7 @@ DEPENDENCIES
   simplecov
   simplecov-cobertura (~> 2.1.0)
   standard (~> 1.31.0)
+  timecop
   webrick
   yard
 

diff --git a/gemfiles/jruby_9.4.0.0_minitest_5.gemfile b/gemfiles/jruby_9.4.0.0_minitest_5.gemfile
@@ -5,12 +5,13 @@ source "https://rubygems.org"
 gem "ddtrace"
 gem "pry"
 gem "rake"
-gem "rspec"
 gem "os"
 gem "climate_control"
+gem "rspec"
 gem "rspec-collection_matchers"
 gem "rspec_junit_formatter"
 gem "appraisal"
+gem "timecop"
 gem "standard", "~> 1.31.0"
 gem "yard"
 gem "webrick"

diff --git a/gemfiles/jruby_9.4.0.0_minitest_5.gemfile.lock b/gemfiles/jruby_9.4.0.0_minitest_5.gemfile.lock
@@ -2,6 +2,7 @@ PATH
   remote: ..
   specs:
     datadog-ci (0.1.1)
+      msgpack
 
 GEM
   remote: https://rubygems.org/
@@ -106,6 +107,7 @@ GEM
       lint_roller (~> 1.1)
       rubocop-performance (~> 1.19.0)
     thor (1.2.2)
+    timecop (0.9.8)
     unicode-display_width (2.4.2)
     webrick (1.8.1)
     yard (0.9.34)
@@ -129,6 +131,7 @@ DEPENDENCIES
   simplecov
   simplecov-cobertura (~> 2.1.0)
   standard (~> 1.31.0)
+  timecop
   webrick
   yard