Lazy Patches #124
Closed
d-chambers
started this conversation in
Ideas
Replies: 2 comments
-
We might also consider a |
Beta Was this translation helpful? Give feedback.
0 replies
-
I am thinking we wont do this; just having an option to split spools would effectively be the same things while keeping the code more simple. |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
We need 'em.
Essentially, each patch that doesn't already have the
data
numpy array loaded into memory would lazily load the data when the data attribute is first accessed. This will probably be accomplished with fsspec and will make it easy to apply map-type parallelism to spools using Executors/Clients from various packages.Beta Was this translation helpful? Give feedback.
All reactions