Skip to content

Commit

Permalink
[docs] Add CliInvocationExample component
Browse files Browse the repository at this point in the history
  • Loading branch information
benpankow committed Feb 1, 2025
1 parent bcb7bfb commit 28c76ef
Show file tree
Hide file tree
Showing 7 changed files with 115 additions and 21 deletions.
32 changes: 16 additions & 16 deletions docs/docs-beta/docs/guides/build/components/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,11 +39,11 @@ If you have a local clone of the `dagster` repo, you can install a local version

Let's take a look at the help message for `dg`:

<CodeExample path="docs_beta_snippets/docs_beta_snippets/guides/components/index/1-help.txt" language="Bash" />
<CliInvocationExample path="docs_beta_snippets/docs_beta_snippets/guides/components/index/1-help.txt" language="bash" />

We want to scaffold a new code location:

<CodeExample path="docs_beta_snippets/docs_beta_snippets/guides/components/index/2-scaffold.txt" language="Bash" />
<CliInvocationExample path="docs_beta_snippets/docs_beta_snippets/guides/components/index/2-scaffold.txt" language="bash" />


This built a code location at `jaffle-platform` and initialized a new Python
Expand All @@ -52,7 +52,7 @@ management behavior, you won't need to worry about activating this virtual envir

Let's have a look at the scaffolded files:

<CodeExample path="docs_beta_snippets/docs_beta_snippets/guides/components/index/3-tree.txt" language="Bash" />
<CliInvocationExample path="docs_beta_snippets/docs_beta_snippets/guides/components/index/3-tree.txt" language="bash" />


You can see that we have a fairly standard Python project structure. There is a
Expand Down Expand Up @@ -100,7 +100,7 @@ Now that we've got a basic scaffold, we're ready to start building components. W

First let's set up Sling. If we query the available component types in our environment, we don't see anything Sling-related:

<CodeExample path="docs_beta_snippets/docs_beta_snippets/guides/components/index/7-dg-list-component-types.txt" language="Bash" />
<CliInvocationExample path="docs_beta_snippets/docs_beta_snippets/guides/components/index/7-dg-list-component-types.txt" language="bash" />

This is because the basic `dagster-components` package (which was installed when we scaffolded our code location) is lightweight and doesn't include components for specific integrations (like Sling). We can get access to a Sling component by installing the `sling` extra of `dagster-components`:

Expand All @@ -114,17 +114,17 @@ $ uv add 'dagster-components[sling]'

Now let's see what component types are available:

<CodeExample path="docs_beta_snippets/docs_beta_snippets/guides/components/index/8-dg-list-component-types.txt" language="Bash" />
<CliInvocationExample path="docs_beta_snippets/docs_beta_snippets/guides/components/index/8-dg-list-component-types.txt" language="bash" />


Great-- now we can see the `dagster_components.sling_replication` component type. Let's create a new instance of this component:

<CodeExample path="docs_beta_snippets/docs_beta_snippets/guides/components/index/9-dg-scaffold-sling-replication.txt" language="Bash" />
<CliInvocationExample path="docs_beta_snippets/docs_beta_snippets/guides/components/index/9-dg-scaffold-sling-replication.txt" language="bash" />


This adds a component instance to the project at `jaffle_platform/components/ingest_files`:

<CodeExample path="docs_beta_snippets/docs_beta_snippets/guides/components/index/10-tree-jaffle-platform.txt" language="Bash" />
<CliInvocationExample path="docs_beta_snippets/docs_beta_snippets/guides/components/index/10-tree-jaffle-platform.txt" language="bash" />


A single file, `component.yaml`, was created in the component folder. The `component.yaml` file is common to all Dagster components, and specifies the component type and any parameters used to scaffold definitions from the component at runtime.
Expand All @@ -140,15 +140,15 @@ The `path` parameter for a replication is relative to the same folder containing

But first, let's set up DuckDB:

<CodeExample path="docs_beta_snippets/docs_beta_snippets/guides/components/index/12-sling-setup-duckdb.txt" language="Bash" />
<CliInvocationExample path="docs_beta_snippets/docs_beta_snippets/guides/components/index/12-sling-setup-duckdb.txt" language="bash" />


<CodeExample path="docs_beta_snippets/docs_beta_snippets/guides/components/index/13-sling-test-duckdb.txt" language="Bash" />
<CliInvocationExample path="docs_beta_snippets/docs_beta_snippets/guides/components/index/13-sling-test-duckdb.txt" language="bash" />


Now let's download some files locally to use our Sling source (Sling doesn't support reading from the public internet):

<CodeExample path="docs_beta_snippets/docs_beta_snippets/guides/components/index/14-curl.txt" language="Bash" />
<CliInvocationExample path="docs_beta_snippets/docs_beta_snippets/guides/components/index/14-curl.txt" language="bash" />

And finally create `replication.yaml` referencing the downloaded files:

Expand All @@ -164,34 +164,34 @@ uv run dagster dev # will be dg dev in the future

Click "Materialize All", and we should now have tables in the DuckDB instance. Let's verify on the command line:

<CodeExample path="docs_beta_snippets/docs_beta_snippets/guides/components/index/16-duckdb-select.txt" language="Bash" />
<CliInvocationExample path="docs_beta_snippets/docs_beta_snippets/guides/components/index/16-duckdb-select.txt" language="bash" />


## Transform

We'll now download a pre-existing sample DBT project from github. We're going to use the data we are ingesting with Sling as an input for the DBT project. Clone the project (and delete the embedded git repo):

<CodeExample path="docs_beta_snippets/docs_beta_snippets/guides/components/index/17-jaffle-clone.txt" language="Bash" />
<CliInvocationExample path="docs_beta_snippets/docs_beta_snippets/guides/components/index/17-jaffle-clone.txt" language="bash" />

We'll need to create a Dagster DBT project component to interface with the dbt project. We can access the DBT project component by installing `dagster-components[dbt]` and `dbt-duckdb`:

```bash
$ uv add "dagster-components[dbt]" dbt-duckdb
```

<CodeExample path="docs_beta_snippets/docs_beta_snippets/guides/components/index/18-dg-list-component-types.txt" language="Bash" />
<CliInvocationExample path="docs_beta_snippets/docs_beta_snippets/guides/components/index/18-dg-list-component-types.txt" language="bash" />


There it is: `dagster_components.dbt_project`. We can access detailed info about a component type using the `dg component-type info` command. Let's have a look at the `dagster_components.dbt_project` component type:

<CodeExample path="docs_beta_snippets/docs_beta_snippets/guides/components/index/19-dg-component-type-info.txt" language="Bash" />
<CliInvocationExample path="docs_beta_snippets/docs_beta_snippets/guides/components/index/19-dg-component-type-info.txt" language="bash" />


The output of the above command shows the parameters (in JSON schema format) for both component generation and runtime loading of the component (the runtime parameters have been truncated here due to length).

Let's scaffold a new instance of the `dagster_components.dbt_project` component, providing the path to the dbt project we cloned earlier as the `project_path` scaffold paramater. We can pass this on the command line:

<CodeExample path="docs_beta_snippets/docs_beta_snippets/guides/components/index/20-dg-scaffold-jdbt.txt" language="Bash" />
<CliInvocationExample path="docs_beta_snippets/docs_beta_snippets/guides/components/index/20-dg-scaffold-jdbt.txt" language="bash" />

This creates a new component instance in the project at `jaffle_platform/components/jdbt`. Open `component.yaml` and you'll see:

Expand Down Expand Up @@ -219,7 +219,7 @@ asset graph is correct. Click "Materialize All" to materialize the new assets
defined via the DBT project component. We can verify that this worked by
viewing a sample of the newly materialized assets from the command line:

<CodeExample path="docs_beta_snippets/docs_beta_snippets/guides/components/index/23-duckdb-select-orders.txt" language="Bash" />
<CliInvocationExample path="docs_beta_snippets/docs_beta_snippets/guides/components/index/23-duckdb-select-orders.txt" language="bash" />


## Automation
Expand Down
2 changes: 1 addition & 1 deletion docs/docs-beta/scripts/generate-code-imports.js
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ const path = require('path');

const DOCUMENTATION_DIRECTORY = 'docs';
const VALID_DOCUMENT_EXTENSIONS = ['.md', '.mdx'];
const CODE_EXAMPLE_PATH_REGEX = /<CodeExample\s+[^>]*path=["']([^"']+)["'][^>]*>/g;
const CODE_EXAMPLE_PATH_REGEX = /<(?:(?:CodeExample)|(?:CliInvocationExample))\s+[^>]*path=["']([^"']+)["'][^>]*>/g;

/**
* Returns a list of file paths for a given `dir`.
Expand Down
12 changes: 12 additions & 0 deletions docs/docs-beta/src/code-examples-content.js

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

55 changes: 55 additions & 0 deletions docs/docs-beta/src/components/CliInvocationExample.tsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
import CodeBlock from '@theme/CodeBlock';
import React, {Suspense, useMemo} from 'react';
import {useLoadModule} from './CodeExample';

interface CliInvocationExampleProps {
path: string;
lineStart?: number;
lineEnd?: number;
startAfter?: string; // marker that indicates beginning of code snippet
endBefore?: string; // marker that indicates ending of code snippet
}

const CliInvocationExample: React.FC<CliInvocationExampleProps> = ({...props}) => {
return (
<Suspense>
<CliInvocationExampleInner {...props} />
</Suspense>
);
};

const CliInvocationExampleInner: React.FC<CliInvocationExampleProps> = (props) => {
const {path, lineStart, lineEnd, startAfter, endBefore, ...extraProps} = props;
const language = 'shell';

const cacheKey = JSON.stringify(props);
const {content, error} = useLoadModule(cacheKey, path, lineStart, lineEnd, startAfter, endBefore);

const [command, result] = useMemo(() => {
const [command, ...rest] = content.split('\n\n');
return [command, rest.join('\n\n')];
}, [content]);

if (error) {
return <div style={{color: 'red', padding: '1rem', border: '1px solid red'}}>{error}</div>;
}

return (
<>
<div className="cli-invocation-example-command">
<CodeBlock language={language} {...extraProps}>
{command || 'Loading...'}
</CodeBlock>
</div>
{command && result && (
<div className="cli-invocation-example-result">
<CodeBlock language={language} {...extraProps}>
{result || 'Loading...'}
</CodeBlock>
</div>
)}
</>
);
};

export default CliInvocationExample;
2 changes: 1 addition & 1 deletion docs/docs-beta/src/components/CodeExample.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ function processModule({
contentCache[cacheKey] = {content: lines.join('\n')};
}

function useLoadModule(
export function useLoadModule(
cacheKey: string,
path: string,
lineStart: number,
Expand Down
31 changes: 28 additions & 3 deletions docs/docs-beta/src/styles/custom.scss
Original file line number Diff line number Diff line change
Expand Up @@ -698,18 +698,43 @@ div[class^='announcementBar_'] {
background-color: var(--theme-color-background-default);
border: 1px solid var(--theme-color-keyline) !important;
transition: all .3s ease;

h2: {
font-size: 1.5rem;
margin-top: 0;
margin-bottom: 0;
font-weight: 600;
display: none;
}

&:hover {
background-color: var(--theme-color-background-light);
border: 1px solid var(--theme-color-border-default) !important;
transform: translateY(0px);
}
}
}

.cli-invocation-example-command{
code::before {
content: '$ ';
color: var(--theme-color-text-cyan);
font-weight: 600;
}
code {
border-bottom: 1px dashed var(--theme-color-border-default);
}
code, div, pre {
border-bottom-left-radius: 0 !important;
border-bottom-right-radius: 0 !important;
}
}

.cli-invocation-example-result {
// Negate the margin from the above code block
margin-top: calc(-1 * var(--ifm-leading) + 1px);

code, div, pre {
border-top-left-radius: 0 !important;
border-top-right-radius: 0 !important;
}
}
2 changes: 2 additions & 0 deletions docs/docs-beta/src/theme/MDXComponents.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
import TOCInline from '@theme/TOCInline';
import Link from '@docusaurus/Link';
import CliInvocationExample from '../components/CliInvocationExample';

export default {
// Re-use the default mapping
Expand All @@ -14,6 +15,7 @@ export default {
Tabs,
TabItem,
CodeExample,
CliInvocationExample,
TOCInline,
Link,
};

0 comments on commit 28c76ef

Please sign in to comment.