Support newer versions of SpatialData with zarr v3 stores

.cursor/settings.json

@@ -0,0 +1,15 @@
+{
+  // Note: Python environments are version-specific. Choose one:
+  // For spatialdata 0.5.0: python/v0.5.0/.venv/bin/python3
+  // For spatialdata 0.6.1: python/v0.6.1/.venv/bin/python3
+  "python.defaultInterpreterPath": "${workspaceFolder}/python/v0.6.1/.venv/bin/python3",
+  "python.terminal.activateEnvironment": true,
+  "[python]": {
+    "editor.defaultFormatter": "ms-python.black-formatter",
+    "editor.formatOnSave": true
+  },
+  "python.analysis.extraPaths": [
+    "${workspaceFolder}/python"
+  ]
+}
+

.github/workflows/test.yml

@@ -0,0 +1,92 @@
+name: Test
+
+on:
+  push:
+    branches:
+      - main
+  pull_request:
+    branches:
+      - main
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Setup pnpm
+        uses: pnpm/action-setup@v4
+        with:
+          version: 10
+          run_install: false
+
+      - uses: actions/setup-node@v4
+        with:
+          node-version: '22'
+          cache: 'pnpm'
+
+      - uses: astral-sh/setup-uv@v4
+        with:
+          version: "latest"
+
+      - name: Install dependencies
+        run: pnpm install --frozen-lockfile
+
+      - name: Build packages
+        run: pnpm build
+
+      - name: Cache uv environments
+        uses: actions/cache@v4
+        with:
+          path: ~/.cache/uv
+          # could be overkill always running all versions in CI, may review in future.
+          key: ${{ runner.os }}-uv-${{ hashFiles('python/v0.5.0/pyproject.toml', 'python/v0.6.1/pyproject.toml', 'python/v0.7.0/pyproject.toml') }}
+
+      # Note: Version-specific environments are set up automatically by the fixture generation script
+      # No need to manually sync here unless you want to pre-warm the cache
+
+      - name: Generate test fixtures (Python spatialdata 0.5.0, 0.6.1, and 0.7.0)
+        run: pnpm test:fixtures:generate
+
+      - name: Run unit tests
+        run: pnpm test:unit
+
+      - name: Run integration tests with local server
+        shell: bash
+        run: |
+          # Verify fixtures exist before starting server
+          if [ ! -d "test-fixtures/v0.5.0/blobs.zarr" ]; then
+            echo "Error: Fixtures not found at test-fixtures/v0.5.0/blobs.zarr"
+            echo "Listing test-fixtures directory:"
+            ls -la test-fixtures/ || echo "test-fixtures directory does not exist"
+            exit 1
+          fi
+
+          # Start test server in background
+          pnpm test:server &
+          SERVER_PID=$!
+
+          # Wait for server to be ready (up to ~30s)
+          # Check both root and a fixture path to ensure server is fully ready
+          for i in {1..30}; do
+            if curl -sSf http://localhost:8080/ >/dev/null && \
+               curl -sSf http://localhost:8080/v0.5.0/blobs.zarr/zmetadata >/dev/null; then
+              echo "Test server is up and serving fixtures"
+              break
+            fi
+            echo "Waiting for test server on http://localhost:8080/ ..."
+            sleep 1
+          done
+
+          # Small additional delay to ensure server is fully ready
+          sleep 2
+
+          # Run integration tests (will hit http://localhost:8080/…)
+          pnpm test:integration
+
+          # Clean up server
+          kill "$SERVER_PID" || true

.gitignore

@@ -9,13 +9,23 @@ build/
 
 # Testing
 coverage/
+test-fixtures/
+validation-results/
+
+# Python virtual environments (version-specific, completely separate)
+python/v0.5.0/.venv/
+python/v0.6.1/.venv/
+python/v0.5.0/uv.lock
+python/v0.6.1/uv.lock
 
 # Environment
 .env
 .env.local
 
 # IDE
-.vscode/
+# Note: .vscode/settings.json and .cursor/settings.json are committed for Python interpreter setup
+.vscode/launch.json
+.vscode/tasks.json
 .idea/
 *.swp
 *.swo

.vscode/settings.json

@@ -0,0 +1,15 @@
+{
+  // Note: Python environments are version-specific. Choose one:
+  // For spatialdata 0.5.0: python/v0.5.0/.venv/bin/python3
+  // For spatialdata 0.6.1: python/v0.6.1/.venv/bin/python3
+  "python.defaultInterpreterPath": "${workspaceFolder}/python/v0.6.1/.venv/bin/python3",
+  "python.terminal.activateEnvironment": true,
+  "[python]": {
+    "editor.defaultFormatter": "ms-python.black-formatter",
+    "editor.formatOnSave": true
+  },
+  "python.analysis.extraPaths": [
+    "${workspaceFolder}/python"
+  ]
+}
+

README.md

@@ -18,6 +18,8 @@ This monorepo contains:
 
 - Node.js >= 20
 - pnpm >= 10
+- Python >= 3.12 (for generating test fixtures)
+- [uv](https://github.com/astral-sh/uv) (Python package manager)
 
 ### Installation
 
@@ -48,6 +50,231 @@ pnpm format
 pnpm docs:dev
 ```
 
+## Testing
+
+### Prerequisites for Testing
+
+To run the full test suite, you'll need:
+
+- Python >= 3.12
+- [uv](https://github.com/astral-sh/uv) - Python package manager
+
+#### Setting Up the Python Environment (not required for JS stuff, mostly generating test fixtures)
+
+The Python environment is managed by `uv` and defined in `python/pyproject.toml`. To set it up:
+
+**Installing uv:**
+```bash
+# Install uv (Unix/macOS)
+curl -LsSf https://astral.sh/uv/install.sh | sh
+
+# Or using pip/pipx
+pip install uv
+# or
+pipx install uv
+```
+
+
+**Setting up the environments:**
+
+The project uses completely separate Python environments for each spatialdata version. Each version has its own directory with its own `pyproject.toml` and virtual environment:
+
+```bash
+# Set up environment for spatialdata 0.5.0
+uv sync --directory python/v0.5.0
+
+# Set up environment for spatialdata 0.6.1
+uv sync --directory python/v0.6.1
+
+# Set up environment for spatialdata 0.7.0
+uv sync --directory python/v0.7.0
+
+# Or set up all at once (the fixture generation script will do this automatically)
+```
+
+Each environment:
+- Is completely independent with its own virtual environment at `python/v{version}/.venv/`
+- Has its own `pyproject.toml` that pins a specific spatialdata version
+- Is isolated from other versions (no shared base environment)
+
+**Note:** The fixture generation script automatically sets up these environments when needed. You only need to manually sync if you want to activate an environment directly or use it in your editor.
+
+**Version Mapping:**
+- **spatialdata 0.5.0** uses **OME-NGFF 0.4** format (multiscales at top level) in **zarr v2** stores (consolidated `zmetadata`)
+- **spatialdata 0.6.0+** uses **OME-NGFF 0.5** format (multiscales nested under `ome` key) in **zarr v3** stores (consolidated `zarr.json`)
+
+#### Editor Setup
+
+The project includes editor configuration files (`.vscode/settings.json` and `.cursor/settings.json`) that configure the Python interpreter to use one of the version-specific environments (defaults to v0.6.1).
+
+**VS Code / Cursor:**
+- The Python interpreter is automatically configured when you open the workspace
+- To switch versions: `Cmd/Ctrl+Shift+P` → "Python: Select Interpreter" → Choose:
+  - `./python/v0.5.0/.venv/bin/python3` for spatialdata 0.5.0
+  - `./python/v0.6.1/.venv/bin/python3` for spatialdata 0.6.1
+  - `./python/v0.7.0/.venv/bin/python3` for spatialdata 0.7.0
+
+**Other editors:**
+- Choose the appropriate virtual environment:
+  - `python/v0.5.0/.venv/bin/python3` for spatialdata 0.5.0
+  - `python/v0.6.1/.venv/bin/python3` for spatialdata 0.6.1
+  - `python/v0.7.0/.venv/bin/python3` for spatialdata 0.7.0
+
+
+### Running Tests
+
+```bash
+# Run all tests (unit + integration)
+pnpm test:all
+
+# Run only unit tests (fast, no fixtures needed)
+pnpm test:unit
+
+# Run only integration tests (requires fixtures)
+pnpm test:integration
+
+# Run tests from individual packages
+pnpm test
+```
+
+### Generating Test Fixtures
+
+Test fixtures are generated on-demand using the Python `spatialdata` library. Each version uses a separate Python environment with the specific spatialdata version pinned to ensure accurate fixture generation.
+
+Fixtures are stored in `test-fixtures/` (excluded from git).
+
+```bash
+# Generate fixtures for all spatialdata versions (0.5.0, 0.6.1, and 0.7.0)
+# This will automatically set up the version-specific environments if needed
+pnpm test:fixtures:generate
+
+# Generate fixtures for a specific version
+pnpm test:fixtures:generate:0.5.0
+pnpm test:fixtures:generate:0.6.1
+pnpm test:fixtures:generate:0.7.0
+```
+
+**How it works:**
+- The script uses separate environments: `python/v0.5.0/`, `python/v0.6.1/`, and `python/v0.7.0/`
+- Each environment has its own `pyproject.toml` with the spatialdata version pinned
+- The script automatically runs `uv sync` for each environment before generating fixtures
+- This ensures fixtures are generated with the exact spatialdata version being tested
+
+**Note:** Integration tests will automatically generate fixtures if they're missing, but you can pre-generate them for faster test runs.
+
+### Test Servers
+
+#### Test Fixture Server
+
+The test fixture server serves generated fixtures over HTTP for testing with `FetchStore`:
+
+```bash
+# Start the test fixture server (runs on http://localhost:8080)
+pnpm test:server
+```
+
+Once running, fixtures are accessible at:
+- `http://localhost:8080/test-fixtures/v0.5.0/blobs.zarr`
+- `http://localhost:8080/test-fixtures/v0.6.1/blobs.zarr`
+- `http://localhost:8080/test-fixtures/v0.7.0/blobs.zarr`
+
+The server provides directory listings and serves all zarr metadata files with appropriate CORS headers.
+
+#### CORS Proxy Server
+
+The CORS proxy server allows accessing spatialdata stores that don't have CORS headers enabled. This is useful for local development when testing against remote stores.
+
+**Standalone Usage:**
+
+```bash
+# Start the CORS proxy server (runs on http://localhost:8081)
+pnpm test:proxy
+```
+
+**Usage:**
+
+Proxy a remote URL using query parameter:
+```
+http://localhost:8081/?url=https://example.com/data.zarr/.zattrs
+```
+
+**Example:**
+
+If you have a spatialdata store at `https://example.com/mydata.zarr` that doesn't have CORS headers, you can access it through the proxy:
+
+```typescript
+import { readZarr } from '@spatialdata/core';
+
+// Instead of:
+// const sdata = await readZarr('https://example.com/mydata.zarr');
+
+// Use the proxy (query parameter form):
+const sdata = await readZarr('http://localhost:8081/?url=https://example.com/mydata.zarr');
+```
+
+**Automatic Proxy Management:**
+
+The validation script (`pnpm validate:datasets:js`) automatically manages its own proxy server. The proxy is started at the beginning of validation and stopped when complete, so you don't need to run it separately.
+
+**⚠️ Warning:** The CORS proxy is for local development only. It has no security restrictions and should never be exposed to the internet.
+
+### Dataset Validation
+
+The project includes a script to validate dataset compatibility with the JavaScript implementation using publicly available spatialdata datasets.
+
+#### Validating with JavaScript
+
+Test datasets with the JavaScript implementation in Node.js (outside browser):
+
+```bash
+# First, build the packages
+pnpm build
+
+# Validate all datasets with JS implementation
+pnpm validate:datasets:js
+
+# Validate a specific dataset
+pnpm validate:datasets:js -- --dataset "Xenium"
+
+# Output to a file
+pnpm validate:datasets:js -- --output-file validation-results-js.md
+```
+
+**Note:** The validation script automatically starts its own CORS proxy server, uses it for all requests, and shuts it down when validation completes. You don't need to run `pnpm test:proxy` separately.
+
+#### Understanding the Results
+
+The validation script generates a table showing which datasets work with the JavaScript implementation:
+
+- ✅ Success: Dataset loaded successfully
+- ❌ Failed: Dataset could not be loaded
+- ⏭️ Not tested: Dataset was skipped
+
+The detailed results include:
+- Element types present (images, labels, points, shapes, tables)
+- Coordinate systems
+- Error messages for failures
+
+This is useful for:
+- Testing compatibility of the JavaScript implementation with real-world datasets
+- Identifying issues with specific datasets
+- Tracking which datasets are known to work or fail
+
+#### Output Formats
+
+The validation script supports multiple output formats:
+
+```bash
+# Markdown (default) - Human-readable report
+pnpm validate:datasets:js -- --output-format markdown --output-file results.md
+
+# JSON - Machine-readable results
+pnpm validate:datasets:js -- --output-format json --output-file results.json
+
+# CSV - Spreadsheet-friendly format
+pnpm validate:datasets:js -- --output-format csv --output-file results.csv
+```
+
 
 ## 📝 License