Skip to content

Commit 9b7bbd9

Browse files
benpankowbenpankow
committed
[components] Components setup integration test
1 parent 0a1514b commit 9b7bbd9

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

47 files changed

+916
-248
lines changed

‎.buildkite/dagster-buildkite/dagster_buildkite/steps/packages.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -316,7 +316,7 @@ def k8s_extra_cmds(version: AvailablePythonVersion, _) -> List[str]:
316316
# snippets in all python versions since we are testing the core code exercised by the
317317
# snippets against all supported python versions.
318318
unsupported_python_versions=AvailablePythonVersion.get_all_except_default(),
319-
pytest_tox_factors=["all", "integrations"],
319+
pytest_tox_factors=["all", "integrations", "docs_snapshot_test"],
320320
),
321321
PackageSpec(
322322
"examples/project_fully_featured",

‎docs/docs-beta/docs/guides/build/components/index.md

Lines changed: 35 additions & 246 deletions
Large diffs are not rendered by default.

‎docs/docs-beta/src/code-examples-content.js

Lines changed: 23 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎examples/docs_beta_snippets/docs_beta_snippets/guides/components/index/1-help.txt

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
$ dg --help
2+
3+
Usage: dg [OPTIONS] COMMAND [ARGS]...
4+
5+
CLI for working with Dagster components.
6+
7+
Commands:
8+
code-location Commands for operating code location directories.
9+
component Commands for operating on components.
10+
component-type Commands for operating on components types.
11+
deployment Commands for operating on deployment directories.
12+
13+
Options:
14+
--clear-cache Clear the cache.
15+
--rebuild-component-registry Recompute and cache the set of available
16+
component types for the current environment.
17+
Note that this also happens automatically
18+
whenever the cache is detected to be stale.
19+
-v, --version Show the version and exit.
20+
-h, --help Show this message and exit.
21+
22+
Global options:
23+
--use-dg-managed-environment / --no-use-dg-managed-environment
24+
Enable management of the virtual environment
25+
with uv.
26+
--builtin-component-lib TEXT Specify a builitin component library to use.
27+
--verbose Enable verbose output for debugging.
28+
--disable-cache Disable the cache..
29+
--cache-dir PATH Specify a directory to use for the cache.

‎examples/docs_beta_snippets/docs_beta_snippets/guides/components/index/10-tree-jaffle-platform.txt

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
$ tree jaffle_platform
2+
3+
jaffle_platform
4+
├── __init__.py
5+
├── components
6+
│   └── ingest_files
7+
│   └── component.yaml
8+
├── definitions.py
9+
└── lib
10+
└── __init__.py
11+
12+
4 directories, 4 files

‎examples/docs_beta_snippets/docs_beta_snippets/guides/components/index/11-component.yaml

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
type: dagster_components.sling_replication_collection
2+
3+
params:
4+
replications:
5+
- path: replication.yaml

‎examples/docs_beta_snippets/docs_beta_snippets/guides/components/index/12-sling-setup-duckdb.txt

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
$ uv run sling conns set DUCKDB type=duckdb instance=/tmp/jaffle_platform.duckdb
2+
3+
9:00AM INF connection `DUCKDB` has been set in /.../.sling/env.yaml. Please test with `sling conns test DUCKDB`

‎examples/docs_beta_snippets/docs_beta_snippets/guides/components/index/13-sling-test-duckdb.txt

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
$ uv run sling conns test DUCKDB
2+
3+
9:00AM INF success!

‎examples/docs_beta_snippets/docs_beta_snippets/guides/components/index/14-curl.txt

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
$ curl -O https://raw.githubusercontent.com/dbt-labs/jaffle-shop-classic/refs/heads/main/seeds/raw_customers.csv &&
2+
curl -O https://raw.githubusercontent.com/dbt-labs/jaffle-shop-classic/refs/heads/main/seeds/raw_orders.csv &&
3+
curl -O https://raw.githubusercontent.com/dbt-labs/jaffle-shop-classic/refs/heads/main/seeds/raw_payments.csv

‎examples/docs_beta_snippets/docs_beta_snippets/guides/components/index/15-replication.yaml

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
2+
### jaffle_platform/components/ingest_files/replication.yaml
3+
4+
source: LOCAL
5+
target: DUCKDB
6+
7+
defaults:
8+
mode: full-refresh
9+
object: "{stream_table}"
10+
11+
streams:
12+
file://raw_customers.csv:
13+
object: "main.raw_customers"
14+
file://raw_orders.csv:
15+
object: "main.raw_orders"
16+
file://raw_payments.csv:
17+
object: "main.raw_payments"

‎examples/docs_beta_snippets/docs_beta_snippets/guides/components/index/16-duckdb-select.txt

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
$ duckdb /tmp/jaffle_platform.duckdb -c "SELECT * FROM raw_customers LIMIT 5;"
2+
3+
┌───────┬────────────┬───────────┬──────────────────┐
4+
│ id │ first_name │ last_name │ _sling_loaded_at │
5+
│ int32 │ varchar │ varchar │ int64 │
6+
├───────┼────────────┼───────────┼──────────────────┤
7+
│ 1 │ Michael │ P. │ ... |
8+
│ 2 │ Shawn │ M. │ ... |
9+
│ 3 │ Kathleen │ P. │ ... |
10+
│ 4 │ Jimmy │ C. │ ... |
11+
│ 5 │ Katherine │ R. │ ... |
12+
└───────┴────────────┴───────────┴──────────────────┘

‎examples/docs_beta_snippets/docs_beta_snippets/guides/components/index/17-jaffle-clone.txt

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
$ git clone --depth=1 https://github.com/dagster-io/jaffle-platform.git dbt && rm -rf dbt/.git

‎examples/docs_beta_snippets/docs_beta_snippets/guides/components/index/18-dg-list-component-types.txt

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
$ dg component-type list
2+
3+
dagster_components.dbt_project
4+
dagster_components.definitions
5+
dagster_components.pipes_subprocess_script_collection
6+
Assets that wrap Python scripts executed with Dagster's PipesSubprocessClient.
7+
dagster_components.sling_replication_collection

‎examples/docs_beta_snippets/docs_beta_snippets/guides/components/index/19-dg-component-type-info.txt

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
$ dg component-type info dagster_components.dbt_project
2+
3+
dagster_components.dbt_project
4+
5+
Scaffold params schema:
6+
7+
{
8+
"properties": {
9+
"init": {
10+
"default": false,
11+
"title": "Init",
12+
"type": "boolean"
13+
},
14+
"project_path": {
15+
"anyOf": [
16+
{
17+
"type": "string"
18+
},
19+
{
20+
"type": "null"
21+
}
22+
],
23+
"default": null,
24+
"title": "Project Path"
25+
}
26+
},
27+
"title": "DbtScaffoldParams",
28+
"type": "object"
29+
}
30+
31+
Component params schema:
32+
...

‎examples/docs_beta_snippets/docs_beta_snippets/guides/components/index/2-scaffold.txt

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
$ dg code-location scaffold jaffle-platform
2+
3+
Creating a Dagster code location at /.../jaffle-platform.
4+
Scaffolded files for Dagster project in /.../jaffle-platform.
5+
...

‎examples/docs_beta_snippets/docs_beta_snippets/guides/components/index/20-dg-scaffold-jdbt.txt

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
$ dg component scaffold dagster_components.dbt_project jdbt --project-path dbt/jdbt
2+
3+
Creating a Dagster component instance folder at /.../jaffle-platform/jaffle_platform/components/jdbt.

‎examples/docs_beta_snippets/docs_beta_snippets/guides/components/index/21-component-jdbt.yaml

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
type: dagster_components.dbt_project
2+
3+
params:
4+
dbt:
5+
project_dir: ../../../dbt/jdbt

‎examples/docs_beta_snippets/docs_beta_snippets/guides/components/index/22-project-jdbt.yaml

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
type: dagster_components.dbt_project
2+
3+
params:
4+
dbt:
5+
project_dir: ../../../dbt/jdbt
6+
asset_attributes:
7+
key: "target/main/{{ node.name }}"

‎examples/docs_beta_snippets/docs_beta_snippets/guides/components/index/23-duckdb-select-orders.txt

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
$ duckdb /tmp/jaffle_platform.duckdb -c "SELECT * FROM orders LIMIT 5;"
2+
3+
┌──────────┬─────────────┬────────────┬───┬───────────────┬──────────────────────┬──────────────────┬────────┐
4+
│ order_id │ customer_id │ order_date │ … │ coupon_amount │ bank_transfer_amount │ gift_card_amount │ amount │
5+
│ int32 │ int32 │ date │ │ double │ double │ double │ double │
6+
├──────────┼─────────────┼────────────┼───┼───────────────┼──────────────────────┼──────────────────┼────────┤
7+
│ 1 │ 1 │ 2018-01-01 │ … │ 0.0 │ 0.0 │ 0.0 │ 10.0 │
8+
│ 2 │ 3 │ 2018-01-02 │ … │ 0.0 │ 0.0 │ 0.0 │ 20.0 │
9+
│ 3 │ 94 │ 2018-01-04 │ … │ 1.0 │ 0.0 │ 0.0 │ 1.0 │
10+
│ 4 │ 50 │ 2018-01-05 │ … │ 25.0 │ 0.0 │ 0.0 │ 25.0 │
11+
│ 5 │ 64 │ 2018-01-05 │ … │ 0.0 │ 17.0 │ 0.0 │ 17.0 │
12+
├──────────┴─────────────┴────────────┴───┴───────────────┴──────────────────────┴──────────────────┴────────┤
13+
│ 5 rows 9 columns (7 shown) │
14+
└────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

‎examples/docs_beta_snippets/docs_beta_snippets/guides/components/index/3-tree.txt

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
$ cd jaffle-platform && tree
2+
3+
.
4+
├── jaffle_platform
5+
│   ├── __init__.py
6+
│   ├── components
7+
│   ├── definitions.py
8+
│   └── lib
9+
│   └── __init__.py
10+
├── jaffle_platform.egg-info
11+
│   ├── PKG-INFO
12+
│   ├── SOURCES.txt
13+
│   ├── dependency_links.txt
14+
│   ├── entry_points.txt
15+
│   ├── requires.txt
16+
│   └── top_level.txt
17+
├── jaffle_platform_tests
18+
│   └── __init__.py
19+
├── pyproject.toml
20+
└── uv.lock
21+
22+
6 directories, 12 files

‎examples/docs_beta_snippets/docs_beta_snippets/guides/components/index/4-pyproject.toml

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
...
2+
[tool.dagster]
3+
module_name = "jaffle_platform.definitions"
4+
project_name = "jaffle_platform"
5+
6+
[tool.dg]
7+
is_code_location = true
8+
is_component_lib = true
9+
...

‎examples/docs_beta_snippets/docs_beta_snippets/guides/components/index/5-definitions.py

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
from pathlib import Path
2+
3+
from dagster_components import build_component_defs
4+
5+
defs = build_component_defs(code_location_root=Path(__file__).parent.parent)

‎examples/docs_beta_snippets/docs_beta_snippets/guides/components/index/6-pyproject.toml

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
...
2+
[project.entry-points]
3+
"dagster.components" = { jaffle_platform = "jaffle_platform.lib"}
4+
...

‎examples/docs_beta_snippets/docs_beta_snippets/guides/components/index/7-dg-list-component-types.txt

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
$ dg component-type list
2+
3+
dagster_components.definitions
4+
dagster_components.pipes_subprocess_script_collection
5+
Assets that wrap Python scripts executed with Dagster's PipesSubprocessClient.

‎examples/docs_beta_snippets/docs_beta_snippets/guides/components/index/8-dg-list-component-types.txt

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
$ dg component-type list
2+
3+
dagster_components.definitions
4+
dagster_components.pipes_subprocess_script_collection
5+
Assets that wrap Python scripts executed with Dagster's PipesSubprocessClient.
6+
dagster_components.sling_replication_collection

‎examples/docs_beta_snippets/docs_beta_snippets/guides/components/index/9-dg-scaffold-sling-replication.txt

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
$ dg component scaffold dagster_components.sling_replication_collection ingest_files
2+
3+
Creating a Dagster component instance folder at /.../jaffle-platform/jaffle_platform/components/ingest_files.

‎examples/docs_beta_snippets/docs_beta_snippets_tests/docs_beta_snippets/guides/components/index/1-help.txt

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
$ dg --help
2+
3+
Usage: dg [OPTIONS] COMMAND [ARGS]...
4+
5+
CLI for working with Dagster components.
6+
7+
Commands:
8+
code-location Commands for operating code location directories.
9+
component Commands for operating on components.
10+
component-type Commands for operating on components types.
11+
deployment Commands for operating on deployment directories.
12+
13+
Options:
14+
--clear-cache Clear the cache.
15+
--rebuild-component-registry Recompute and cache the set of available
16+
component types for the current environment.
17+
Note that this also happens automatically
18+
whenever the cache is detected to be stale.
19+
-v, --version Show the version and exit.
20+
-h, --help Show this message and exit.
21+
22+
Global options:
23+
--use-dg-managed-environment / --no-use-dg-managed-environment
24+
Enable management of the virtual environment
25+
with uv.
26+
--builtin-component-lib TEXT Specify a builitin component library to use.
27+
--verbose Enable verbose output for debugging.
28+
--disable-cache Disable the cache..
29+
--cache-dir PATH Specify a directory to use for the cache.

‎examples/docs_beta_snippets/docs_beta_snippets_tests/docs_beta_snippets/guides/components/index/2-scaffold.txt

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
$ dg code-location scaffold jaffle-platform
2+
3+
Creating a Dagster code location at /.../jaffle-platform.
4+
Scaffolded files for Dagster project in /.../jaffle-platform.
5+
...

‎examples/docs_beta_snippets/docs_beta_snippets_tests/docs_beta_snippets/guides/components/index/3-tree.txt

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
$ cd jaffle-platform && tree
2+
3+
.
4+
├── jaffle_platform
5+
│   ├── __init__.py
6+
│   ├── components
7+
│   ├── definitions.py
8+
│   └── lib
9+
│   └── __init__.py
10+
├── jaffle_platform.egg-info
11+
│   ├── PKG-INFO
12+
│   ├── SOURCES.txt
13+
│   ├── dependency_links.txt
14+
│   ├── entry_points.txt
15+
│   ├── requires.txt
16+
│   └── top_level.txt
17+
├── jaffle_platform_tests
18+
│   └── __init__.py
19+
├── pyproject.toml
20+
└── uv.lock
21+
22+
6 directories, 12 files

‎examples/docs_beta_snippets/docs_beta_snippets_tests/docs_beta_snippets/guides/components/index/4-pyproject.toml

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
...
2+
[tool.dagster]
3+
module_name = "jaffle_platform.definitions"
4+
project_name = "jaffle_platform"
5+
6+
[tool.dg]
7+
is_code_location = true
8+
is_component_lib = true
9+
...

‎examples/docs_beta_snippets/docs_beta_snippets_tests/docs_beta_snippets/guides/components/index/5-definitions.py

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
from pathlib import Path
2+
3+
from dagster_components import build_component_defs
4+
5+
defs = build_component_defs(code_location_root=Path(__file__).parent.parent)

‎examples/docs_beta_snippets/docs_beta_snippets_tests/docs_beta_snippets/guides/components/index/6-pyproject.toml

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
...
2+
[project.entry-points]
3+
"dagster.components" = { jaffle_platform = "jaffle_platform.lib"}
4+
...

‎examples/docs_beta_snippets/docs_beta_snippets_tests/docs_beta_snippets/guides/components/index/7-dg-list-component-types.txt

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
$ dg component-type list
2+
3+
dagster_components.definitions
4+
dagster_components.pipes_subprocess_script_collection
5+
Assets that wrap Python scripts executed with Dagster's PipesSubprocessClient.

‎examples/docs_beta_snippets/docs_beta_snippets_tests/docs_beta_snippets/guides/components/index/8-dg-scaffold-sling-replication.txt

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
$ dg component scaffold dagster_components.sling_replication_collection ingest_files
2+
3+
Usage: dg component scaffold [OPTIONS] COMMAND [ARGS]...
4+
5+
Error: No such command 'dagster_components.sling_replication_collection'.

‎examples/docs_beta_snippets/docs_beta_snippets_tests/docs_beta_snippets/guides/components/index/9-tree-jaffle-platform.txt

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
$ tree jaffle_platform
2+
3+
jaffle_platform
4+
├── __init__.py
5+
├── components
6+
├── definitions.py
7+
└── lib
8+
└── __init__.py
9+
10+
3 directories, 3 files

‎examples/docs_beta_snippets/docs_beta_snippets_tests/snippet_checks/__init__.py

Whitespace-only changes.

0 commit comments

Comments
 (0)