v0.14.7 Testing checklist

[jorgeepc]

Testing Tracetest

This is the checklist on what we should do to assert that Tracetest is working fine on each release version. On each version release, we can copy the contents of this template checklist and open a Github Discussion to start each test.

On Tracetest, we work in two ways to test the entire system and guarantee that everything is working fine. One is through automatic tests, where we run unit, integration, and end-to-end tests on Tracetest CLI, Web UI, and Server.

Another source of tests is the manual tests that we execute on each release, simulating users and checking if everything is ok on these tests.

Manual Tests

Our manual tests aim to do some sanity checks to see if the main features are working as expected. Usually, we run it on each Tracetest release.

Tests to validate on Release Candidate

• Test server installation via CLI

  • Docker Compose and no demo API
  • Docker Compose and demo API
  • Kubernetes and no demo API
  • Kubernetes and demo API
  • Verify that Tracetest is not sending analytics if you have the TRACETEST_DEV env for docker install
  • Verify that Tracetest is not sending analytics if you have the TRACETEST_DEV env for k8s install

• Test DataStore provisioning and basic test

  • Open WebUI and go to /settings page. The provisioned Data Store should be selected
  • Create a test on WebUI that calls a demo API (like Pokeshop or Open Telemetry Store). This test should fetch traces correctly and run without errors

• Test Tracetest examples

  • Jaeger
  • Tempo
  • New Relic
  • Lightstep
  • OpenSearch
  • Elastic APM
  • SignalFX
  • Datadog
  • Amazon X-Ray
  • Honeycomb
  • Azure Application Insights
  • Signoz
  • Dynatrace

• Test CLI commands (detailed list here)

  • Resources: config
  • Resources: datastore
  • Resources: demo
  • Resources: variableset
  • Resources: pollingprofile
  • Resources: analyzer
  • Resources: testrunner
  • Resources: test
  • Resources: testsuite
  • Run tests and testsuites

• Testing benchmark

  • Run OpenTelemetry Demo tests with the current version and guarantee that the CLI and Server responded correctly

• Test specific features added/changed on this release:

  • Validate backwards compatibility with: fix(backend): Adding Composed Primary Keys fix(backend): Adding Composed Primary Keys #3345
  • fix(cli): consider overriden url on cli formatting fix: consider overriden url on cli formatting #3356
  • fix(frontend): Deeplink Base Url fix: Deeplink Base Url #3375
  • fix(frontend): Default Sort By fix: Default Sort By #3377
  • fix(frontend): Run Test Mode fix: Run Test Mode #3376
  • Validate start command: chore: Migrate start actions from CLI to agent chore: Migrate start actions from CLI to agent #3378
  • Validate collector config in DataStore settings: feat: Dynamic Collector Config feat: Dynamic Collector Config #3372
  • Test creation flow: create and edit functionality for http test
  • Test creation flow: create and edit functionality for grpc test
  • Test creation flow: create and edit functionality for kafka test
  • Test creation flow: create and edit functionality for traceid test
  • Test creation flow: test creation from demo examples
  • Test creation flow: test creation from cURL import
  • Test creation flow: test creation from Postman import
  • Validate test creation with the skip trace collection flag enabled
  • Validate skip trace collection for a running test

Tests to validate on Final Release

• Pipeline

  • Check if our release pipeline on Release Tracetest workflow on Github Actions worked correctly

• Test CLI updates

  • MacOS via homebrew
  • MacOS via curl script
  • Windows via chocolatey
  • Windows via manual download

• Docs

  • Double check Detailed installation docs and see if everything is documented correctly

• Add Major Changes

  • Add a summary with major changes to the release page

• Test specific features added/changed on this release:

  • Add a list of features here