Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

misleading progressbar in xb.get_bitinformation(dim=None) #97

Open
aaronspring opened this issue May 6, 2022 · 1 comment
Open

misleading progressbar in xb.get_bitinformation(dim=None) #97

aaronspring opened this issue May 6, 2022 · 1 comment
Labels
bug Something isn't working

Comments

@aaronspring
Copy link
Collaborator

  • xbitinfo version: 0.0.1

Description

I would hope that the default dim=None works out of the box. Maybe that's too ambitious. This examples fails in the variable z (why is z the second?, its the first in ds.data_vars) for dimension month (which has two elements only).

I observed that 2 element dimensions often do not work. (However, it seems to have worked for v). Maybe we could raise more informative error messages in xbitinfo?

What I Did

ds = xr.tutorial.load_dataset("eraint_uvz")
info_per_bit = xb.get_bitinformation(ds, dim=None)
Processing v: 100%
3/3 [00:05<00:00, 1.33s/it]
Processing v: 100%
3/3 [00:01<00:00, 2.29it/s]
Processing v: 100%
3/3 [00:01<00:00, 2.67it/s]
Processing z: 0%
0/3 [00:01<?, ?it/s]
---------------------------------------------------------------------------
JuliaError                                Traceback (most recent call last)
Input In [3], in <cell line: 1>()
----> 1 info_per_bit = xb.get_bitinformation(ds)
      3 info_per_bit

File ~/xbitinfo/xbitinfo.py:153, in get_bitinformation(ds, dim, axis, label, overwrite, **kwargs)
     87 """Wrap `BitInformation.jl.bitinformation() <[https://github.com/milankl/BitInformation.jl/blob/main/src/mutual_information.jl>`__](https://github.com/milankl/BitInformation.jl/blob/main/src/mutual_information.jl%3E%60__).
     88 
     89 Parameters
   (...)
    149     BitInformation.jl_version:  ...
    150 """
    151 if dim is None and axis is None:
    152     # gather bitinformation on all axis
--> 153     return _get_bitinformation_along_all_dims(
    154         ds, label=label, overwrite=overwrite, **kwargs
    155     )
    156 else:
    157     # gather bitinformation along one axis
    158     if overwrite is False and label is not None:

File ~/xbitinfo/xbitinfo.py:229, in _get_bitinformation_along_all_dims(ds, label, overwrite, **kwargs)
    227     if label is not None:
    228         label = "_".join([label, d])
--> 229     info_per_bit_per_dim[d] = get_bitinformation(
    230         ds, dim=d, axis=None, label=label, overwrite=overwrite, **kwargs
    231     ).expand_dims("dim", axis=0)
    232 info_per_bit = xr.merge(info_per_bit_per_dim.values()).squeeze()
    233 return info_per_bit

File ~/xbitinfo/xbitinfo.py:205, in get_bitinformation(ds, dim, axis, label, overwrite, **kwargs)
    203 logging.debug(f"get_bitinformation(X, dim={dim}, {kwargs_str})")
    204 info_per_bit[var] = {}
--> 205 info_per_bit[var]["bitinfo"] = jl.eval(
    206     f"get_bitinformation(X, dim={axis_jl}, {kwargs_str})"
    207 )
    208 info_per_bit[var]["dim"] = dim
    209 info_per_bit[var]["axis"] = axis_jl - 1

File /srv/conda/envs/notebook/lib/python3.10/site-packages/julia/core.py:621, in Julia.eval(self, src)
    619 if src is None:
    620     return None
--> 621 ans = self._call(src)
    622 if not ans:
    623     return None

File /srv/conda/envs/notebook/lib/python3.10/site-packages/julia/core.py:549, in Julia._call(self, src)
    547 # logger.debug("_call(%s)", src)
    548 ans = self.api.jl_eval_string(src.encode('utf-8'))
--> 549 self.check_exception(src)
    551 return ans

File /srv/conda/envs/notebook/lib/python3.10/site-packages/julia/core.py:603, in Julia.check_exception(self, src)
    601 else:
    602     exception = sprint(showerror, self._as_pyobj(res))
--> 603 raise JuliaError(u'Exception \'{}\' occurred while calling julia code:\n{}'
    604                  .format(exception, src))

JuliaError: Exception 'AssertionError: Mask has 347040 unmasked values, 0 entries are adjacent.' occurred while calling julia code:
get_bitinformation(X, dim=1, masked_value=convert(Float32,NaN), set_zero_insignificant=true)
@aaronspring aaronspring added the bug Something isn't working label May 6, 2022
@observingClouds
Copy link
Owner

Hi @aaronspring,
the progress bar is a bit misleading here. What actually happens here, is that the bitinformation is retrieved first along dim longitude, latitude, level and finally month. following

print(ds.dims)
#Frozen({'longitude': 480, 'latitude': 241, 'level': 3, 'month': 2})

as expected.

However, because the bitinformation is retrieved very quickly, you only see:

Processing v: 100%
3/3 [...]

but intermediate steps would be

Processing z: 33%
1/3 [...]
Processing u: 66%
2/3 [...]

The exception is raised only when retrieving the bitinformation along the 2 element long dimension month, which is not possible. I think, the best way would be to raise a specific exception in BitInformation.jl when the dimensions are too short. This way, we could catch that exception and handle it appropriately. Handling it in the current, general state could be too broad.

@aaronspring aaronspring changed the title eraint_uvz examples fails for dim=None misleading errorbar in xb.get_bitinformation(dim=None) May 6, 2022
@observingClouds observingClouds changed the title misleading errorbar in xb.get_bitinformation(dim=None) misleading progressbar in xb.get_bitinformation(dim=None) Nov 11, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants