-
Notifications
You must be signed in to change notification settings - Fork 124
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Refactors caching examples to be in a single place
Updates links and adds README.
- Loading branch information
Showing
14 changed files
with
62 additions
and
20 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,18 +1,8 @@ | ||
# Caching Graph Adapter | ||
Here you'll find two adapters that allow you to cache the results of your functions. | ||
|
||
You can use `CachingGraphAdapter` to cache certain nodes. | ||
The first one is the `DiskCacheAdapter`, which uses the `diskcache` library to store the results on disk. | ||
|
||
This is great for: | ||
The second one is the `CachingGraphAdapter`, which requires you to tag functions to cache along with the | ||
serialization format. | ||
|
||
1. Iterating during development, where you don't want to recompute certain expensive function calls. | ||
2. Providing some lightweight means to control recomputation in production, by controlling whether a "cached file" exists or not. | ||
|
||
For iterating during development, the general process would be: | ||
|
||
1. Write your functions. | ||
2. Mark them with `tag(cache="SERIALIZATION_FORMAT")` | ||
3. Use the CachingGraphAdapter and pass that to the Driver to turn on caching for these functions. | ||
a. If at any point in your development you need to re-run a cached node, you can pass | ||
its name to the adapter in the `force_compute` argument. Then, this node and its downstream | ||
nodes will be computed instead of loaded from cache. | ||
4. When no longer required, you can just skip (3) and any caching behavior will be skipped. | ||
Both have their sweet spots and trade-offs. We invite you play with them and provide feedback on which one you prefer. |
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,18 @@ | ||
# Caching Graph Adapter | ||
|
||
You can use `CachingGraphAdapter` to cache certain nodes. | ||
|
||
This is great for: | ||
|
||
1. Iterating during development, where you don't want to recompute certain expensive function calls. | ||
2. Providing some lightweight means to control recomputation in production, by controlling whether a "cached file" exists or not. | ||
|
||
For iterating during development, the general process would be: | ||
|
||
1. Write your functions. | ||
2. Mark them with `tag(cache="SERIALIZATION_FORMAT")` | ||
3. Use the CachingGraphAdapter and pass that to the Driver to turn on caching for these functions. | ||
a. If at any point in your development you need to re-run a cached node, you can pass | ||
its name to the adapter in the `force_compute` argument. Then, this node and its downstream | ||
nodes will be computed instead of loaded from cache. | ||
4. When no longer required, you can just skip (3) and any caching behavior will be skipped. |
35 changes: 35 additions & 0 deletions
35
examples/caching_nodes/caching_graph_adapter/business_logic.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,35 @@ | ||
import pandas as pd | ||
|
||
""" | ||
Copied from the hello world example. | ||
""" | ||
|
||
|
||
def avg_3wk_spend(spend: pd.Series) -> pd.Series: | ||
"""Rolling 3 week average spend.""" | ||
return spend.rolling(3).mean() | ||
|
||
|
||
def spend_per_signup(spend: pd.Series, signups: pd.Series) -> pd.Series: | ||
"""The cost per signup in relation to spend.""" | ||
return spend / signups | ||
|
||
|
||
def spend_mean(spend: pd.Series) -> float: | ||
"""Shows function creating a scalar. In this case it computes the mean of the entire column.""" | ||
return spend.mean() | ||
|
||
|
||
def spend_zero_mean(spend: pd.Series, spend_mean: float) -> pd.Series: | ||
"""Shows function that takes a scalar. In this case to zero mean spend.""" | ||
return spend - spend_mean | ||
|
||
|
||
def spend_std_dev(spend: pd.Series) -> float: | ||
"""Function that computes the standard deviation of the spend column.""" | ||
return spend.std() | ||
|
||
|
||
def spend_zero_mean_unit_variance(spend_zero_mean: pd.Series, spend_std_dev: float) -> pd.Series: | ||
"""Function showing one way to make spend have zero mean and unit variance.""" | ||
return spend_zero_mean / spend_std_dev |
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.