[components] Components setup integration test

[benpankow]

Summary

Builds a little docs integration testing framework which allows us to define a series of steps that either:

1. Run a command

   run_command_and_snippet_output(
       cmd="dg code-location scaffold jaffle-platform",
       snippet_path=COMPONENTS_SNIPPETS_DIR / f"{next_snip_no()}-scaffold.txt",
       update_snippets=update_snippets,
       snippet_replace_regex=[
           # Mask tempdir path of `/whatever/folder/jaffle-platform` with `/.../jaffle-platform`
           (r" \/.*?\/jaffle-platform", " /.../jaffle-platform")
       ],
   )

• Check the contents of a file

  check_file(
      "pyproject.toml",
      COMPONENTS_SNIPPETS_DIR / f"{next_snip_no()}-pyproject.toml",
      update_snippets=update_snippets,
  )

• Update the contents of a file

  create_file(
      file_path=Path("jaffle_platform") / "components" / "ingest_files" / "replication.yaml",
      snippet_path=COMPONENTS_SNIPPETS_DIR / f"{next_snip_no()}-replication.yaml",
      contents="""
  ### foo bar
  """,
  )

Each step corresponds to a docs snippet file. By default, (tox -e docs_snapshot_test, run as part of BK CI) these tests ensure that output of each of these steps is identical to the snippet file. The tests will fail if the docs guide breaks or the output of any of the commands changes.

The test suite can also be run in a snippet update mode (tox -e docs_snapshot_update), which will update each of the snippets to match the actual output of each step, which we can use whenever we update the docs steps or the code it's using.

Also has some utilities for masking or removing parts of files/output we don't care about, or which might vary (e.g. timestamps, filepaths etc)

See https://github.com/dagster-io/dagster/pull/27375/files#diff-22c4df2c53a911cdff63776be42c290bca0a8f3d2c6134c27f48a40b0aba4692 for the example

Test Plan

New test suite.

[benpankow]

• [components docs] Add dg components check to docs #27410
• [components] Components setup integration test #27375 👈 (View in Graphite)
• master

This stack of pull requests is managed by Graphite. Learn more about stacking.

[schrockn]

This is cool.

Is the only way to refresh the generate snippets is to run the tests?

[benpankow]

Is the only way to refresh the generate snippets is to run the tests?

Right now yes, main reason being that tox gives us a nice self-contained Python environment, but no reason you couldn't invoke the Python fn directly outside of a pytest context as long as your env was set up prooperly.

[schrockn]

It would be very helpful to have a makefile somewhere that has an entry like regenerate_snippets even if it just invokes the test. Pretty unintuitive as is and you have to just magically know. Makefile is a good way to document this tribal knowledge

[benpankow]

It would be very helpful to have a makefile somewhere that has an entry like regenerate_snippets even if it just invokes the test. Pretty unintuitive as is and you have to just magically know. Makefile is a good way to document this tribal knowledge

Agreed, added a couple Makefile commands in the docs folder.

[github-actions]

Deploy preview for dagster-docs ready!

Preview available at https://dagster-docs-c9cm9j2l8-elementl.vercel.app

Direct link to changed pages:

• https://dagster-docs-c9cm9j2l8-elementl.vercel.app/guides/build/components/existing-code-location
• https://dagster-docs-c9cm9j2l8-elementl.vercel.app/guides/build/components/index
• https://dagster-docs-c9cm9j2l8-elementl.vercel.app/guides/build/components/setting-up-a-deployment