Skip to content

Commit

Permalink
added flush
Browse files Browse the repository at this point in the history
  • Loading branch information
norlandrhagen committed Aug 10, 2023
1 parent eab555a commit 59927e1
Showing 1 changed file with 6 additions and 5 deletions.
11 changes: 6 additions & 5 deletions docs/source/advanced.rst
Original file line number Diff line number Diff line change
Expand Up @@ -165,9 +165,9 @@ one go and may be faster, if you have a Dask cluster available.
# Create LazyReferenceMapper to pass to MultiZarrToZarr
fs = fsspec.filesystem("file")
td = TemporaryDirectory()
tmpdir = str(td.name)
out = LazyReferenceMapper.create(10, tmpdir, fs)
os.makedirs("combined.parq")
out = LazyReferenceMapper.create(1000, "combined.parq", fs)
# Create references from input files
single_ref_sets = [hdf.SingleHdf5ToZarr(_).translate() for _ in files]
Expand All @@ -178,11 +178,12 @@ one go and may be faster, if you have a Dask cluster available.
concat_dims=["time"],
out=out).translate()
os.mkdir("combined.parq")
out.flush()
df.refs_to_dataframe(out_dict, "combined.parq")
fs = fsspec.implementations.reference.ReferenceFileSystem(
"combined.parq", lazy=True)
"combined.parq", remote_protocol="s3", target_protocol="file", lazy=True)
ds = xr.open_dataset(
fs.get_mapper(), engine="zarr",
backend_kwargs={"consolidated": False}
Expand Down

0 comments on commit 59927e1

Please sign in to comment.