Skip to content

Python wrapper of BitInformation.jl to easily compress xarray datasets based on their information content

License

Notifications You must be signed in to change notification settings

mattphysics/xbitinfo

 
 

Repository files navigation

xbitinfo: Retrieve information content and compress accordingly

Binder CI pre-commit.ci status Documentation Status pypi

This is an xarray-wrapper around BitInformation.jl to retrieve and apply bitrounding from within python. The package intends to present an easy pipeline to compress (climate) datasets based on the real information content.

How the science works

Paper

Klöwer, M., Razinger, M., Dominguez, J. J., Düben, P. D., & Palmer, T. N. (2021). Compressing atmospheric data into its real information content. Nature Computational Science, 1(11), 713–724. doi: 10/gnm4jj

Video

Video

Julia Repository

BitInformation.jl

How to install

pip install git+https://github.com/observingClouds/xbitinfo.git

pip install git+https://github.com/observingClouds/xbitinfo.git#egg=xbitinfo[complete]

How to use

import xarray as xr
import xbitinfo as xb
ds = xr.tutorial.load_dataset(inpath)
bitinfo = xb.get_bitinformation(ds, dim="lon")  # calling bitinformation.jl.bitinformation
keepbits = xb.get_keepbits(bitinfo, inflevel=0.99)  # get number of mantissa bits to keep for 99% real information
ds_bitrounded = xb.xr_bitround(ds, keepbits)  # bitrounding keeping only keepbits mantissa bits
ds_bitrounded.to_compressed_netcdf(outpath)  # save to netcdf with compression

Credits

About

Python wrapper of BitInformation.jl to easily compress xarray datasets based on their information content

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages

  • Jupyter Notebook 66.4%
  • Python 32.9%
  • Julia 0.7%