Skip to content

Commit 1054bde

Browse files
[daggy-u] update usage of context.add_output_metadata with MaterializeResult (#19575)
## Summary & Motivation With Dagster University being upgraded to v1.6, we should update our usage of `context.add_output_metadata` to returning a `MaterializeResult` with the `metadata` defined as a parameter. The corresponding changes to `project-dagster-university` can be found here: dagster-io/project-dagster-university#8 ## How I Tested These Changes --------- Co-authored-by: Erin Cochran <erin.k.cochran@gmail.com>
1 parent 22a4a84 commit 1054bde

File tree

3 files changed

+70
-50
lines changed

3 files changed

+70
-50
lines changed

docs/dagster-university/pages/dagster-essentials/extra-credit/asset-metadata-as-markdown.md

Lines changed: 22 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -89,35 +89,35 @@ In Lesson 9, you created the `adhoc_request` asset. During materialization, the
8989
pio.write_image(fig, file_path)
9090
```
9191

92-
3. Add the following to the import to the top of the file:
92+
3. Add the `base64` and `MaterializeResult` imports to the top of the file:
9393

9494
```python
9595
import base64
96+
from dagster import MaterializeResult
9697
```
9798

98-
4. Because you need to use the `context` object, you’ll need to add it to the asset function’s argument as the first argument:
99-
100-
```python
101-
@asset
102-
def adhoc_request(context, config: AdhocRequestConfig, taxi_zones, taxi_trips, database: DuckDBResource):
103-
```
104-
105-
5. After the last line in the asset, add the following code:
99+
4. After the last line in the asset, add the following code:
106100

107101
```python
108102
with open(file_path, 'rb') as file:
109103
image_data = file.read()
110104
```
111105

112-
6. Next, we’ll use base64 encoding to convert the chart to Markdown. After the `image_data` line, add the following code:
106+
5. Next, we’ll use base64 encoding to convert the chart to Markdown. After the `image_data` line, add the following code:
113107

114108
```python
115109
base64_data = base64.b64encode(image_data).decode('utf-8')
116-
md_content = f"![Image](data:image/jpeg;base64,{base64_data})"
110+
md_content = f"![Image](data:image/jpeg;base64,{base64_data})"
111+
```
112+
113+
6. Finally, we'll return a `MaterializeResult` object with the metadata specified as a parameter:
117114

118-
context.add_output_metadata({
119-
"preview": MetadataValue.md(md_content)
120-
})
115+
```python
116+
return MaterializeResult(
117+
metadata={
118+
"preview": MetadataValue.md(md_content)
119+
}
120+
)
121121
```
122122

123123
Let’s break down what’s happening here:
@@ -126,13 +126,13 @@ In Lesson 9, you created the `adhoc_request` asset. During materialization, the
126126
2. `base64.b64encode` encodes the image’s binary data (`image_data`) into base64 format.
127127
3. Next, the encoded image data is converted to a UTF-8 encoded string using the `decode` function.
128128
4. Next, a variable named `md_content` is created. The value of this variable is a Markdown-formatted string containing a JPEG image, where the base64 representation of the image is inserted.
129-
5. Using `context.add_output_metadata`, the image is passed in as metadata. The metadata will have a `preview` label in the Dagster UI.
129+
5. To include the metadata on the asset, we returned a `MaterializeResult` instance with the image passed in as metadata. The metadata will have a `preview` label in the Dagster UI.
130130
6. Using `MetadataValue.md`, the `md_content` is typed as Markdown. This ensures Dagster will correctly render the chart.
131131

132132
At this point, the code for the `adhoc_request` asset should look like this:
133133

134134
```python
135-
from dagster import Config, asset, MetadataValue, get_dagster_logger
135+
from dagster import Config, asset, MaterializeResult, MetadataValue, get_dagster_logger
136136
from dagster_duckdb import DuckDBResource
137137

138138
import plotly.express as px
@@ -148,7 +148,7 @@ class AdhocRequestConfig(Config):
148148
end_date: str
149149

150150
@asset
151-
def adhoc_request(context**,** config: AdhocRequestConfig, taxi_zones, taxi_trips, database: DuckDBResource):
151+
def adhoc_request(config: AdhocRequestConfig, taxi_zones, taxi_trips, database: DuckDBResource):
152152
"""
153153
The response to an request made in the `requests` directory.
154154
See `requests/README.md` for more information.
@@ -210,9 +210,11 @@ def adhoc_request(context**,** config: AdhocRequestConfig, taxi_zones, taxi_trip
210210
base64_data = base64.b64encode(image_data).decode('utf-8')
211211
md_content = f"![Image](data:image/jpeg;base64,{base64_data})"
212212

213-
context.add_output_metadata({
214-
"preview": MetadataValue.md(md_content)
215-
})
213+
return MaterializeResult(
214+
metadata={
215+
"preview": MetadataValue.md(md_content)
216+
}
217+
)
216218
```
217219

218220
---

docs/dagster-university/pages/dagster-essentials/extra-credit/coding-practice-metadata-taxi-zones-file.md

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -15,10 +15,13 @@ To practice what you’ve learned, add the record counts to the metadata for `ta
1515
The metadata you built should look similar to the code contained in the **View answer** toggle. Click to open it.
1616

1717
```python {% obfuscated="true" %}
18+
from dagster import MaterializeResult
19+
20+
1821
@asset(
1922
group_name="raw_files",
2023
)
21-
def taxi_zones_file(context):
24+
def taxi_zones_file():
2225
"""
2326
The raw CSV file for the taxi zones dataset. Sourced from the NYC Open Data portal.
2427
"""
@@ -29,5 +32,10 @@ def taxi_zones_file(context):
2932
with open(constants.TAXI_ZONES_FILE_PATH, "wb") as output_file:
3033
output_file.write(raw_taxi_zones.content)
3134
num_rows = MetadataValue.int(len(pd.read_csv(constants.TAXI_ZONES_FILE_PATH)))
32-
context.add_output_metadata({'Number of records': num_rows})
35+
36+
return MaterializeResult(
37+
metadata={
38+
'Number of records': MetadataValue.int(num_rows)
39+
}
40+
)
3341
```

docs/dagster-university/pages/dagster-essentials/extra-credit/materialization-metadata.md

Lines changed: 38 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ Now that we’ve covered definition metadata, let’s dive into the other type o
1414

1515
To add metadata to an asset, you need to do two things:
1616

17-
- Use the `context.add_output_metadata` function to pass in the data
17+
- Return a `MaterializeResult` object with the `metadata` parameter from your asset
1818
- Use the `MetadataValue` utility class to wrap the data, ensuring it displays correctly in the UI
1919

2020
Let’s add metadata to the `taxi_trips_file` asset to demonstrate further. This will add the count of records to the asset’s materialization metadata.
@@ -34,19 +34,19 @@ Let’s add metadata to the `taxi_trips_file` asset to demonstrate further. This
3434
group_name="raw_files",
3535
)
3636
def taxi_trips_file(context):
37-
"""
38-
The raw parquet files for the taxi trips dataset. Sourced from the NYC Open Data portal.
39-
"""
37+
"""
38+
The raw parquet files for the taxi trips dataset. Sourced from the NYC Open Data portal.
39+
"""
4040

41-
partition_date_str = context.asset_partition_key_for_output()
42-
month_to_fetch = partition_date_str[:-3]
41+
partition_date_str = context.asset_partition_key_for_output()
42+
month_to_fetch = partition_date_str[:-3]
4343

44-
raw_trips = requests.get(
45-
f"https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_{month_to_fetch}.parquet"
46-
)
44+
raw_trips = requests.get(
45+
f"https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_{month_to_fetch}.parquet"
46+
)
4747

48-
with open(constants.TAXI_TRIPS_TEMPLATE_FILE_PATH.format(month_to_fetch), "wb") as output_file:
49-
output_file.write(raw_trips.content)
48+
with open(constants.TAXI_TRIPS_TEMPLATE_FILE_PATH.format(month_to_fetch), "wb") as output_file:
49+
output_file.write(raw_trips.content)
5050
```
5151

5252
3. First, we need to calculate the number of records contained in the file. Copy and paste the following after the last line in the asset:
@@ -55,15 +55,19 @@ Let’s add metadata to the `taxi_trips_file` asset to demonstrate further. This
5555
num_rows = len(pd.read_parquet(constants.TAXI_TRIPS_TEMPLATE_FILE_PATH.format(month_to_fetch)))
5656
```
5757

58-
4. Next, we’ll pass in and type the data:
58+
4. Next, we’ll add the metadata with the specified type:
5959

6060
```python
61-
context.add_output_metadata({'Number of records':MetadataValue.int(num_rows)})
61+
return MaterializeResult(
62+
metadata={
63+
'Number of records': MetadataValue.int(num_rows)
64+
}
65+
)
6266
```
6367

6468
Let’s break down what’s happening here:
6569

66-
- `context.add_output_metadata` accepts a `dict`, where the key is the label or name of the metadata being passed and the value is the data itself. In this case, the key is `Number of records`. The value in this example is everything after `Number of records`.
70+
- The `metadata` parameter accepts a `dict`, where the key is the label or name of the metadata and the value is the data itself. In this case, the key is `Number of records`. The value in this example is everything after `Number of records`.
6771
- Using `MetadataValue.int`, the value of the `num_rows` variable is typed as an integer. This tells Dagster to render the data as an integer.
6872

6973
At this point, the asset should look like this:
@@ -73,25 +77,31 @@ Let’s add metadata to the `taxi_trips_file` asset to demonstrate further. This
7377
from dagster import asset, MetadataValue
7478

7579
@asset(
76-
partitions_def=monthly_partition,
77-
group_name="raw_files",
80+
partitions_def=monthly_partition,
81+
group_name="raw_files",
7882
)
7983
def taxi_trips_file(context):
80-
"""
81-
The raw parquet files for the taxi trips dataset. Sourced from the NYC Open Data portal.
82-
"""
84+
"""
85+
The raw parquet files for the taxi trips dataset. Sourced from the NYC Open Data portal.
86+
"""
87+
88+
partition_date_str = context.asset_partition_key_for_output()
89+
month_to_fetch = partition_date_str[:-3]
90+
91+
raw_trips = requests.get(
92+
f"https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_{month_to_fetch}.parquet"
93+
)
8394

84-
partition_date_str = context.asset_partition_key_for_output()
85-
month_to_fetch = partition_date_str[:-3]
95+
with open(constants.TAXI_TRIPS_TEMPLATE_FILE_PATH.format(month_to_fetch), "wb") as output_file:
96+
output_file.write(raw_trips.content)
8697

87-
raw_trips = requests.get(
88-
f"https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_{month_to_fetch}.parquet"
89-
)
98+
num_rows = len(pd.read_parquet(constants.TAXI_TRIPS_TEMPLATE_FILE_PATH.format(month_to_fetch)))
9099

91-
with open(constants.TAXI_TRIPS_TEMPLATE_FILE_PATH.format(month_to_fetch), "wb") as output_file:
92-
output_file.write(raw_trips.content)
93-
num_rows = len(pd.read_parquet(constants.TAXI_TRIPS_TEMPLATE_FILE_PATH.format(month_to_fetch)))
94-
context.add_output_metadata({'Number of records':MetadataValue.int(num_rows)})
100+
return MaterializeResult(
101+
metadata={
102+
'Number of records': MetadataValue.int(num_rows)
103+
}
104+
)
95105
```
96106

97107
---

0 commit comments

Comments
 (0)