misleading progressbar in xb.get_bitinformation(dim=None)

[aaronspring]

• xbitinfo version: 0.0.1

Description

I would hope that the default dim=None works out of the box. Maybe that's too ambitious. This examples fails in the variable z (why is z the second?, its the first in ds.data_vars) for dimension month (which has two elements only).

I observed that 2 element dimensions often do not work. (However, it seems to have worked for v). Maybe we could raise more informative error messages in xbitinfo?

What I Did

ds = xr.tutorial.load_dataset("eraint_uvz")
info_per_bit = xb.get_bitinformation(ds, dim=None)
Processing v: 100%
3/3 [00:05<00:00, 1.33s/it]
Processing v: 100%
3/3 [00:01<00:00, 2.29it/s]
Processing v: 100%
3/3 [00:01<00:00, 2.67it/s]
Processing z: 0%
0/3 [00:01<?, ?it/s]
---------------------------------------------------------------------------
JuliaError                                Traceback (most recent call last)
Input In [3], in <cell line: 1>()
----> 1 info_per_bit = xb.get_bitinformation(ds)
      3 info_per_bit

File ~/xbitinfo/xbitinfo.py:153, in get_bitinformation(ds, dim, axis, label, overwrite, **kwargs)
     87 """Wrap `BitInformation.jl.bitinformation() <[https://github.com/milankl/BitInformation.jl/blob/main/src/mutual_information.jl>`__](https://github.com/milankl/BitInformation.jl/blob/main/src/mutual_information.jl%3E%60__).
     88 
     89 Parameters
   (...)
    149     BitInformation.jl_version:  ...
    150 """
    151 if dim is None and axis is None:
    152     # gather bitinformation on all axis
--> 153     return _get_bitinformation_along_all_dims(
    154         ds, label=label, overwrite=overwrite, **kwargs
    155     )
    156 else:
    157     # gather bitinformation along one axis
    158     if overwrite is False and label is not None:

File ~/xbitinfo/xbitinfo.py:229, in _get_bitinformation_along_all_dims(ds, label, overwrite, **kwargs)
    227     if label is not None:
    228         label = "_".join([label, d])
--> 229     info_per_bit_per_dim[d] = get_bitinformation(
    230         ds, dim=d, axis=None, label=label, overwrite=overwrite, **kwargs
    231     ).expand_dims("dim", axis=0)
    232 info_per_bit = xr.merge(info_per_bit_per_dim.values()).squeeze()
    233 return info_per_bit

File ~/xbitinfo/xbitinfo.py:205, in get_bitinformation(ds, dim, axis, label, overwrite, **kwargs)
    203 logging.debug(f"get_bitinformation(X, dim={dim}, {kwargs_str})")
    204 info_per_bit[var] = {}
--> 205 info_per_bit[var]["bitinfo"] = jl.eval(
    206     f"get_bitinformation(X, dim={axis_jl}, {kwargs_str})"
    207 )
    208 info_per_bit[var]["dim"] = dim
    209 info_per_bit[var]["axis"] = axis_jl - 1

File /srv/conda/envs/notebook/lib/python3.10/site-packages/julia/core.py:621, in Julia.eval(self, src)
    619 if src is None:
    620     return None
--> 621 ans = self._call(src)
    622 if not ans:
    623     return None

File /srv/conda/envs/notebook/lib/python3.10/site-packages/julia/core.py:549, in Julia._call(self, src)
    547 # logger.debug("_call(%s)", src)
    548 ans = self.api.jl_eval_string(src.encode('utf-8'))
--> 549 self.check_exception(src)
    551 return ans

File /srv/conda/envs/notebook/lib/python3.10/site-packages/julia/core.py:603, in Julia.check_exception(self, src)
    601 else:
    602     exception = sprint(showerror, self._as_pyobj(res))
--> 603 raise JuliaError(u'Exception \'{}\' occurred while calling julia code:\n{}'
    604                  .format(exception, src))

JuliaError: Exception 'AssertionError: Mask has 347040 unmasked values, 0 entries are adjacent.' occurred while calling julia code:
get_bitinformation(X, dim=1, masked_value=convert(Float32,NaN), set_zero_insignificant=true)

[observingClouds]

Hi @aaronspring,
the progress bar is a bit misleading here. What actually happens here, is that the bitinformation is retrieved first along dim longitude, latitude, level and finally month. following

print(ds.dims)
#Frozen({'longitude': 480, 'latitude': 241, 'level': 3, 'month': 2})

as expected.

However, because the bitinformation is retrieved very quickly, you only see:

Processing v: 100%
3/3 [...]

but intermediate steps would be

Processing z: 33%
1/3 [...]
Processing u: 66%
2/3 [...]

The exception is raised only when retrieving the bitinformation along the 2 element long dimension month, which is not possible. I think, the best way would be to raise a specific exception in BitInformation.jl when the dimensions are too short. This way, we could catch that exception and handle it appropriately. Handling it in the current, general state could be too broad.