chore/hollow out zarr3 #780

d-v-b · 2025-08-13T20:37:00Z

this PR removes the classes defined in numcodecs.zarr3, and instead re-exports those classes from zarr.codecs._numcodecs. This PR is a draft until zarr-developers/zarr-python#3376 is merged in zarr-python.

closes #773, #768, #778

codecov · 2025-09-23T16:42:18Z

Codecov Report

❌ Patch coverage is 34.78261% with 15 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.12%. Comparing base (dbcf37b) to head (f254a21).

Files with missing lines	Patch %	Lines
numcodecs/zarr3.py	40.00%	9 Missing ⚠️
numcodecs/tests/test_zarr3.py	14.28%	6 Missing ⚠️

❌ Your project check has failed because the head coverage (89.12%) is below the target coverage (100.00%). You can increase the head coverage or adjust the target coverage.

Additional details and impacted files

@@             Coverage Diff             @@
##             main     #780       +/-   ##
===========================================
- Coverage   99.82%   89.12%   -10.70%     
===========================================
  Files          64       64               
  Lines        2804     2547      -257     
===========================================
- Hits         2799     2270      -529     
- Misses          5      277      +272

Files with missing lines	Coverage Δ
numcodecs/tests/test_zarr3_import.py	`100.00% <100.00%> (ø)`
numcodecs/tests/test_zarr3.py	`40.00% <14.28%> (-60.00%)`	⬇️
numcodecs/zarr3.py	`47.36% <40.00%> (-52.64%)`	⬇️

... and 10 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

d-v-b · 2025-09-23T17:20:54Z

@zarr-developers/python-core-devs please take a look at this. I'm removing the zarr-python-dependent codecs from numcodecs and re-exporting codecs from zarr-python instead.

When zarr python 3.1.3 or greater is installed, importing a codec from numcodecs.zarr3 will raise a warning, with a suggestion to import that same codec from zarr.codecs.numcodecs instead. Otherwise, numcodecs.zarr3 is inaccessible.

This solves our long-standing circular dependency issue, but I want to make sure we get this right.

d-v-b · 2025-09-24T09:38:48Z

A summary of this PR:

removes zarr-python-dependent codec classes, since they are now defined in zarr.codecs.numcodecs. This change is not breaking because we still export these codecs from numcodecs.zarr3, but importing from numcodecs.zarr3 will generate a deprecation warning:

>>> from numcodecs.zarr3 import Blosc
<stdin>-1:1: DeprecationWarning: The numcodecs.zarr3 module is deprecated and will be removed in a future release of numcodecs. Import Blosc via zarr.codecs.numcodecs.Blosc instead. This requires Zarr Python >= 3.1.3.

removes tests for the codecs that are now defined in zarr.codecs.numcodecs, because these tests have been ported to Zarr Python/
removes zarr installation from our main CI workflow
removes hatch from pyproject.toml, uses uv instead. This was necessary to get around an obscure, hard-to-debug test failure. Instead of using hatch's dependency matrix support, we declare two separate, conflicting dependency groups and use UV to run tests in each one. The actual matrix is defined in a github action.
CI tests against zarr python 3.1.2 (to ensure that importing from numcodecs.zarr3 fails completely) and zarr python 3.1.3 (to ensure that import from numcodecs.zarr3 succeeds, but with a deprecation warning)

I would like to merge this today after the Zarr Python dev meeting, unless someone objects to these changes.

In the release after the next one, we can remove numcodecs.zarr3 completely.

maxrjones · 2025-09-24T12:45:42Z

Thank you for your efforts in resolving the circular dependency issue! I'd rather someone with more numcodecs experience give a formal review.

Should Zarr now be an optional dependency for numcodecs? It's currently only listed in dependency-groups as far as I see, which means it's not included as an optional dependency on release distributions

d-v-b · 2025-09-24T12:48:22Z

Should Zarr now be an optional dependency for numcodecs? It's currently only listed in dependency-groups as far as I see, which means it's not included as an optional dependency on release distributions

My feeling is that Zarr should not be listed as an optional dependency, because numcodecs doesn't actually use Zarr for anything. But as we are in a bit of a weird situation, maybe there is some reason to declare Zarr as an optional dep here? Open to ideas.

normanrz · 2025-09-24T16:08:31Z

Wouldn't an optional dep be the circular dependency we want to get rid of?

Thanks @d-v-b for working on this!

maxrjones · 2025-10-03T22:09:41Z

When I add this PR as a dependency in VirtualiZarr, I get this error:

ImportError while loading conftest '/Users/max/Documents/Code/zarr-developers/VirtualiZarr/conftest.py'.
conftest.py:17: in <module>
    from virtualizarr.manifests import ChunkManifest, ManifestArray
virtualizarr/__init__.py:3: in <module>
    from virtualizarr.accessor import (
virtualizarr/accessor.py:17: in <module>
    from virtualizarr.manifests import ManifestArray
virtualizarr/manifests/__init__.py:4: in <module>
    from virtualizarr.manifests.array import ManifestArray  # type: ignore # noqa
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
virtualizarr/manifests/array.py:6: in <module>
    from zarr.core.metadata.v3 import ArrayV3Metadata, RegularChunkGrid
.pixi/envs/upstream/lib/python3.13/site-packages/zarr/__init__.py:2: in <module>
    from zarr.api.synchronous import (
.pixi/envs/upstream/lib/python3.13/site-packages/zarr/api/synchronous.py:7: in <module>
    import zarr.api.asynchronous as async_api
.pixi/envs/upstream/lib/python3.13/site-packages/zarr/api/asynchronous.py:13: in <module>
    from zarr.core.array import (
.pixi/envs/upstream/lib/python3.13/site-packages/zarr/core/array.py:30: in <module>
    from zarr.codecs._v2 import V2Codec
.pixi/envs/upstream/lib/python3.13/site-packages/zarr/codecs/__init__.py:5: in <module>
    from zarr.codecs.crc32c_ import Crc32cCodec
.pixi/envs/upstream/lib/python3.13/site-packages/zarr/codecs/crc32c_.py:8: in <module>
    from crc32c import crc32c
E   ModuleNotFoundError: No module named 'crc32c'

maxrjones · 2025-10-03T22:10:27Z

Should Zarr now be an optional dependency for numcodecs? It's currently only listed in dependency-groups as far as I see, which means it's not included as an optional dependency on release distributions

My feeling is that Zarr should not be listed as an optional dependency, because numcodecs doesn't actually use Zarr for anything. But as we are in a bit of a weird situation, maybe there is some reason to declare Zarr as an optional dep here? Open to ideas.
Wouldn't an optional dep be the circular dependency we want to get rid of?

makes sense, thanks for explaining

d-v-b added 4 commits July 3, 2025 15:41

Update pyproject.toml

998579a

Merge branch 'main' of https://github.com/zarr-developers/numcodecs

a31ecb0

Merge branch 'main' of https://github.com/zarr-developers/numcodecs

e994840

test against zarr-with-numcodecs branch

8cc1fda

d-v-b mentioned this pull request Aug 13, 2025

remove the zarr dependency #778

Open

d-v-b changed the title ~~chore/remove zarr3~~ chore/hollow out zarr3 Aug 13, 2025

d-v-b added 3 commits September 15, 2025 21:21

Merge branch 'main' into chore/remove-zarr3

0b0b71a

Merge branch 'main' into chore/remove-zarr3

343f01c

weaken zarr dependency

e382ec1

warn when importing from numcodecs.zarr3

6de8605

d-v-b marked this pull request as ready for review September 23, 2025 17:17

update pre-commit

2ab0e09

d-v-b mentioned this pull request Sep 23, 2025

Allow Zstandard to decompress multiple concatenated frames #757

Open

7 tasks

d-v-b added 13 commits September 23, 2025 22:54

update importerror test

4622b8b

adjust exception text test

3b83ba4

remove print statement

a740d87

fix numpy dep, remove direct references in pyproject.toml

9ce658f

revert numpy dep changes

a85745c

revert revert of dep changes for test env

48443f8

drop hatch in favor of uv + parametrization in ci

618afbf

fix substitution syntax

70623ae

typo

af5f9c3

zarr.codecs entrypoints

3d3f8f3

install crc32c with zarr

d7bfd6c

remove zarr installation from main ci

a6dddca

remove zarr from i386 workload

f254a21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

chore/hollow out zarr3 #780

chore/hollow out zarr3 #780

Uh oh!

d-v-b commented Aug 13, 2025 •

edited

Loading

Uh oh!

codecov bot commented Sep 23, 2025 •

edited

Loading

Uh oh!

d-v-b commented Sep 23, 2025

Uh oh!

d-v-b commented Sep 24, 2025 •

edited

Loading

Uh oh!

maxrjones commented Sep 24, 2025

Uh oh!

d-v-b commented Sep 24, 2025

Uh oh!

normanrz commented Sep 24, 2025

Uh oh!

maxrjones commented Oct 3, 2025 •

edited

Loading

Uh oh!

maxrjones commented Oct 3, 2025

Uh oh!

Uh oh!

chore/hollow out zarr3 #780

Are you sure you want to change the base?

chore/hollow out zarr3 #780

Uh oh!

Conversation

d-v-b commented Aug 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Sep 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

d-v-b commented Sep 23, 2025

Uh oh!

d-v-b commented Sep 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

maxrjones commented Sep 24, 2025

Uh oh!

d-v-b commented Sep 24, 2025

Uh oh!

normanrz commented Sep 24, 2025

Uh oh!

maxrjones commented Oct 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

maxrjones commented Oct 3, 2025

Uh oh!

Uh oh!

d-v-b commented Aug 13, 2025 •

edited

Loading

codecov bot commented Sep 23, 2025 •

edited

Loading

d-v-b commented Sep 24, 2025 •

edited

Loading

maxrjones commented Oct 3, 2025 •

edited

Loading