diff --git a/README.md b/README.md index adc9704..daa5158 100644 --- a/README.md +++ b/README.md @@ -1,115 +1,129 @@ # Internet Quality Barometer (IQB) -[![Build Status](https://github.com/m-lab/iqb/actions/workflows/ci.yml/badge.svg)](https://github.com/m-lab/iqb/actions) [![codecov](https://codecov.io/gh/m-lab/iqb/branch/main/graph/badge.svg)](https://codecov.io/gh/m-lab/iqb) [![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/m-lab/iqb) +[![Build Status](https://github.com/m-lab/iqb/actions/workflows/ci.yml/badge.svg)](https://github.com/m-lab/iqb/actions) +[![codecov](https://codecov.io/gh/m-lab/iqb/branch/main/graph/badge.svg)](https://codecov.io/gh/m-lab/iqb) +[![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/m-lab/iqb) -This repository contains the source for code the Internet Quality Barometer (IQB) -library, and related applications and notebooks. +IQB is an open-source framework developed by +[Measurement Lab (M-Lab)](https://www.measurementlab.net/) that computes a +composite Internet quality score across real-world use cases: web browsing, +video streaming, video conferencing, audio streaming, online backup, and +gaming. Unlike single-metric speed tests, IQB evaluates whether a connection +meets the network requirements of each use case and aggregates the results +into a single score between 0 and 1. -## About IQB +Read the conceptual background: -IQB is an open-source project initiated by -[Measurement Lab (M-Lab)](https://www.measurementlab.net/). +- [M-Lab blog post](https://www.measurementlab.net/blog/iqb/) +- [Detailed framework report (PDF)](https://www.measurementlab.net/publications/IQB_report_2025.pdf) +- [Executive summary (PDF)](https://www.measurementlab.net/publications/IQB_executive_summary_2025.pdf) +- [ACM IMC 2025 poster](https://arxiv.org/pdf/2509.19034) -IQB is motivated by the need to redefine how we measure and understand Internet -performance to keep pace with evolving technological demands and user -expectations. IQB is a comprehensive framework for collecting data and -calculating a composite score, the “IQB Score”, which reflects -the quality of Internet experience. IQB takes a more holistic approach -than “speed tests” and evaluates Internet performance across various -use cases (web browsing, video streaming, online gaming, etc.), -each with its own specific network requirements (latency, throughput, etc.). +Live staging dashboard: [iqb.mlab-staging.measurementlab.net](https://iqb.mlab-staging.measurementlab.net/) -Read more about the IQB framework in: +--- -- M-Lab's [blog post](https://www.measurementlab.net/blog/iqb/). +## Repository Structure -- The IQB framework [detailed report]( -https://www.measurementlab.net/publications/IQB_report_2025.pdf) and -[executive summary]( -https://www.measurementlab.net/publications/IQB_executive_summary_2025.pdf). +| Directory | Description | +|-----------|-------------| +| `library/` | `mlab-iqb` Python package — scoring logic, cache API, data pipeline, CLI | +| `prototype/` | Streamlit web dashboard (Phase 1 prototype) | +| `analysis/` | Jupyter notebooks for research and experimentation | +| `data/` | Pipeline configuration and local Parquet cache | +| `docs/` | Documentation, design decision records, internals guide | -- The IQB [poster](https://arxiv.org/pdf/2509.19034) at ACM IMC 2025. +--- -## Repository Architecture - -### **`docs/`** - -Documentation, tutorials, design documents, and presentations. +## Quick Start -See [docs/README.md](docs/README.md) for details. +### Requirements -### **`library/`** +- Python 3.13 (see `.python-version`) +- [uv](https://astral.sh/uv) — install with `brew install uv` on macOS or + `sudo snap install astral-uv --classic` on Ubuntu -The IQB library containing methods for calculating the IQB score and data collection. +### Setup and Run -See [library/README.md](library/README.md) for details. +```bash +# Clone the repository +git clone git@github.com:m-lab/iqb.git +cd iqb -### **`prototype/`** +# Install all workspace dependencies +uv sync --dev -A Streamlit web application for applying and parametrizing the IQB framework -in different use cases. +# Run the Streamlit prototype +cd prototype +uv run streamlit run Home.py +``` -See [prototype/README.md](prototype/README.md) for how to run it locally. +The dashboard will be available at `http://localhost:8501`. -### **`analysis/`** +### Using the Library -Jupyter notebooks for exploratory data analysis, experimentation, and research. +```python +from iqb import IQBCache, IQBCalculator, IQBDatasetGranularity, IQBRemoteCache -See [analysis/README.md](analysis/README.md) for more information. +# Pull pre-computed data from GCS (requires gcloud auth) +cache = IQBCache(remote_cache=IQBRemoteCache()) -### **`data/`** +# Load monthly country-level data +entry = cache.get_cache_entry( + start_date="2025-10-01", + end_date="2025-11-01", + granularity=IQBDatasetGranularity.COUNTRY, +) -Workspace containing the default pipeline configuration, the default cache directory, -and instructions for generating new data using the pipeline. +# Filter to a specific country and extract the median percentile +p50 = entry.mlab.read_data_frame_pair(country_code="US").to_iqb_data(percentile=50) -See [data/README.md](data/README.md) for details. +# Calculate the IQB score +score = IQBCalculator().calculate_iqb_score(data={"m-lab": p50.to_dict()}) +print(f"IQB score: {score:.3f}") +``` -### **`.iqb`** +See [`analysis/00-template.ipynb`](analysis/00-template.ipynb) for a complete +walkthrough. -Symbolic link to [data](data) that simplifies running the pipeline on Unix. +### CLI -## Data Flow +```bash +# Check which cache entries are available locally and remotely +uv run iqb cache status -The components above connect as follows: +# Pull pre-computed data from GCS to the local cache +uv run iqb cache pull -d data/ -``` -BigQuery → [iqb pipeline run] → local cache/ → [IQBCache] → [IQBCalculator] → scores - ↕ - [iqb cache pull/push] ↔ GCS +# Run the pipeline to generate new data from BigQuery +uv run iqb pipeline run -d data/ ``` -The **pipeline** queries BigQuery for M-Lab NDT measurements and stores -percentile summaries as Parquet files in the local cache. To avoid expensive -re-queries, **`iqb cache pull`** can download pre-computed results from GCS -instead. The **`IQBCache`** API reads cached data, and **`IQBCalculator`** -applies quality thresholds and weights to produce IQB scores. The -**prototype** and **analysis notebooks** both consume scores through -these library APIs. +--- -## Understanding the Codebase +## Documentation -- To learn **how the data pipeline works**, read the -[internals guide](docs/internals/README.md) — it walks through queries, -the pipeline, the remote cache, and the researcher API in sequence. +| Document | Description | +|----------|-------------| +| [docs/architecture.md](docs/architecture.md) | System overview, data flow, component responsibilities, extensibility | +| [docs/developer_guide.md](docs/developer_guide.md) | Local setup, adding metrics and pages, testing, contribution workflow | +| [docs/user_guide.md](docs/user_guide.md) | IQB for consumers, policymakers, researchers, and ISPs | +| [library/README.md](library/README.md) | Library API, testing, linting, type checking | +| [prototype/README.md](prototype/README.md) | Running locally, Docker, Cloud Run deployment | +| [data/README.md](data/README.md) | Pipeline commands, cache format, GCS configuration | +| [analysis/README.md](analysis/README.md) | Notebook usage and conventions | +| [docs/internals/](docs/internals/README.md) | Sequential guide to how the data pipeline works | +| [docs/design/](docs/design/README.md) | Architecture decision records | +| [CONTRIBUTING.md](CONTRIBUTING.md) | Development environment, VSCode setup, component workflows | -- To understand **why specific technical decisions were made**, see the -[design documents](docs/design/README.md) — architecture decision records -covering cache design, data distribution, and more. +--- -## Quick Start +## Contributing -```bash -# Clone the repository -git clone git@github.com:m-lab/iqb.git -cd iqb +Contributions are welcome. Please read [CONTRIBUTING.md](CONTRIBUTING.md) for +development environment setup and [docs/developer_guide.md](docs/developer_guide.md) +for guidance on adding metrics, use cases, and dashboard pages. All changes +require passing tests, Ruff linting, and Pyright type checks before merge. -# Sync all dependencies (creates .venv automatically) -uv sync --dev - -# Run the Streamlit prototype -cd prototype -uv run streamlit run Home.py -``` +--- -See [CONTRIBUTING.md](CONTRIBUTING.md) for full development environment -setup, VSCode configuration, and component-specific workflows. diff --git a/docs/architecture.md b/docs/architecture.md new file mode 100644 index 0000000..f615408 --- /dev/null +++ b/docs/architecture.md @@ -0,0 +1,183 @@ +# IQB Architecture + +## Overview + +The Internet Quality Barometer (IQB) is a framework for computing a composite +score that reflects the quality of an Internet connection across a set of +real-world use cases. The system is structured as a monorepo containing four +distinct components: a scoring library, a Streamlit-based dashboard prototype, +Jupyter notebooks for exploratory analysis, and a data workspace managing the +pipeline that acquires and caches measurement data. + +The design enforces a clear separation between: + +- **Data acquisition** — querying BigQuery for M-Lab NDT measurements +- **Scoring logic** — computing IQB scores from aggregated measurement data +- **Visualization** — presenting scores and trends in an interactive dashboard + +This separation allows each layer to evolve independently. Researchers can work +with the library API and notebooks without touching the dashboard; dashboard +developers can consume pre-computed cached data without running the pipeline. + +--- + +## Repository Structure + +``` +iqb/ +├── library/ # mlab-iqb Python package (scoring logic, cache API, pipeline) +│ └── src/iqb/ +│ ├── calculator.py # IQBCalculator: weighted use-case scoring +│ ├── config.py # IQB_CONFIG: default thresholds and weights +│ ├── cache/ # IQBCache: read Parquet data from local cache +│ ├── pipeline/ # IQBPipeline: query BigQuery, write Parquet +│ ├── ghremote/ # IQBRemoteCache: sync with GCS remote cache +│ ├── queries/ # SQL query templates +│ └── cli/ # iqb command-line interface +├── prototype/ # Streamlit dashboard (Phase 1 prototype) +│ ├── Home.py # Application entry point +│ ├── pages/ # Additional Streamlit pages (e.g., IQB_Map.py) +│ ├── utils/ # Calculation helpers, constants, data loaders +│ ├── visualizations/ # Chart components (sunburst, maps) +│ ├── cache/ # Static JSON cache files per country/ASN +│ └── Dockerfile # Container image for Cloud Run deployment +├── analysis/ # Jupyter notebooks for research and experimentation +│ ├── 00-template.ipynb # Worked example using IQBCache + IQBCalculator +│ └── .iqb/ # Symlink to data workspace inside analysis env +├── data/ # Pipeline configuration and local Parquet cache +│ ├── pipeline.yaml # Query matrix (date ranges × granularities) +│ ├── cache/ # Parquet files written by iqb pipeline run +│ └── state/ # Remote cache manifest (ghremote) +└── docs/ # Documentation, design docs, internals guides +``` + +--- + +## Data Flow Pipeline + +```mermaid +flowchart LR + BQ["BigQuery\n(M-Lab NDT measurements)"] + pipeline["iqb pipeline run\n(IQBPipeline)"] + parquet["Local Parquet Cache\ncache/v1/{start}/{end}/{granularity}/"] + gcs["GCS Remote Cache\n(mlab-sandbox-iqb-us-central1)"] + cache_api["IQBCache\n(read + filter data)"] + calc["IQBCalculator\n(binary scoring → weighted aggregation)"] + dashboard["Streamlit Prototype\n(prototype/Home.py)"] + notebooks["Jupyter Notebooks\n(analysis/)"] + + BQ -->|"SQL queries via\npipeline.yaml matrix"| pipeline + pipeline -->|"Parquet + stats.json"| parquet + parquet <-->|"iqb cache push / pull"| gcs + parquet --> cache_api + cache_api -->|"IQBDataFramePair"| calc + calc -->|"IQB score (0–1)"| dashboard + calc -->|"IQB score (0–1)"| notebooks + gcs -->|"iqb cache pull"| parquet +``` + +### Pipeline Stages + +1. **Query** — `IQBPipeline` reads `pipeline.yaml` to determine the date + ranges and granularities (country, country_asn, subdivision1, + subdivision1_asn, city, city_asn) to query. It runs parameterised SQL + against BigQuery and writes `data.parquet` and `stats.json` per entry. + +2. **Cache** — Results are stored as Parquet files under + `cache/v1/{start_date}/{end_date}/{query_type}/`. The format supports + streaming reads and chunked row groups for memory efficiency. + +3. **Remote sync** — `IQBRemoteCache` (and its GitHub-backed variant + `IQBGitHubRemoteCache`) can push local results to GCS and pull + pre-computed files to avoid re-running expensive BigQuery queries. + +4. **Scoring** — `IQBCache` reads and filters Parquet data into an + `IQBDataFramePair`. The caller extracts a percentile (e.g., the 50th), + converts it to a flat measurement dict, and passes it to + `IQBCalculator.calculate_iqb_score()`. + +5. **Presentation** — The Streamlit prototype loads pre-computed JSON files + from `prototype/cache/` and calls the library to compute scores + interactively. + +--- + +## Scoring Logic and Visualization Separation + +The scoring logic is entirely contained in `library/src/iqb/`: + +- `IQB_CONFIG` in `config.py` encodes use cases, network requirements, + per-requirement weights, binary thresholds, and dataset weights. +- `IQBCalculator.calculate_iqb_score()` applies a two-level weighted + aggregation: first across datasets for each requirement (requirement + agreement score), then across requirements for each use case (use case + score), and finally across use cases (IQB score). +- The result is a single float in [0, 1]. + +The dashboard in `prototype/` depends on the library as a workspace member +(`mlab-iqb`) but contains no scoring logic itself. All calculation helpers +in `prototype/utils/calculation_utils.py` delegate to `IQBCalculator` and +manipulate `IQB_CONFIG` purely for session-specific overrides (custom +thresholds, user-adjusted weights). + +This boundary ensures that changes to the scoring methodology require updates +only in the library and are automatically reflected in both the prototype and +analysis notebooks without any additional code changes. + +--- + +## Extensibility + +**Adding a new use case** — Extend `IQB_CONFIG` in `config.py` with a new +key under `"use cases"`, defining its weight, network requirements, thresholds, +and dataset weights. `IQBCalculator` picks up the new use case automatically +without code changes. + +**Adding a new dataset** — Add the dataset name to the `"datasets"` sub-dict +for each network requirement that the dataset covers. Set the weight (`"w"`) +to 0 to include the dataset structurally while keeping it inactive; set it to +a positive value to activate it. Ensure data for the dataset is available in +`IQBCache` or supplied in the `data` dict passed to `calculate_iqb_score()`. + +**Adding a new metric (network requirement)** — Add the requirement key to +each use case in `IQB_CONFIG` and implement the corresponding binary scoring +branch in `IQBCalculator.calculate_binary_requirement_score()`. + +**Adding a new dashboard page** — Create a new `.py` file in +`prototype/pages/`. Streamlit automatically discovers and adds it to the +sidebar. Consume library APIs and `prototype/utils/` helpers rather than +reimplementing scoring logic. + +--- + +## Scalability Considerations + +- **Granularity matrix** — The `pipeline.yaml` matrix defines the full + Cartesian product of date ranges and granularities. Adding new time + periods or geographic granularities requires only a YAML change; no + code changes are needed. +- **Parquet format** — Parquet supports columnar reads and predicate pushdown. + Filtering by `country_code` or `asn` before loading data avoids reading + full datasets into memory. +- **GCS remote cache** — Pre-computed results can be shared across teams via + GCS without requiring anyone to run BigQuery queries. The manifest tracks + available entries, allowing incremental pulls. + +--- + +## Performance Considerations + +- **Caching at the pipeline level** — `IQBPipeline` checks for existing + Parquet files before executing BigQuery queries to avoid redundant cloud + spend. `stats.json` records bytes billed and duration for auditability. +- **Static JSON cache in prototype** — The dashboard reads pre-computed + JSON files from `prototype/cache/` rather than querying BigQuery at + runtime. This eliminates network latency and authentication requirements + for end users. +- **Percentile aggregation** — Raw measurement data is pre-aggregated to + percentile summaries (e.g., p50, p90) before being stored. This reduces + data volume and allows the dashboard to operate without streaming large + raw datasets. +- **Streamlit session state** — `session_state.py` caches computation + results within a user session to avoid recalculating scores on every + widget interaction. diff --git a/docs/developer_guide.md b/docs/developer_guide.md new file mode 100644 index 0000000..963bf65 --- /dev/null +++ b/docs/developer_guide.md @@ -0,0 +1,347 @@ +# IQB Developer Guide + +## Prerequisites + +| Tool | Version | Notes | +|------|---------|-------| +| Python | 3.13 | Specified in `.python-version` at repo root | +| uv | latest | Package and workspace manager | +| gcloud | any | Required only for BigQuery queries and GCS sync | + +### Installing uv + +On macOS: + +```bash +brew install uv +``` + +On Ubuntu: + +```bash +sudo snap install astral-uv --classic +``` + +For other platforms, see the [uv installation guide](https://docs.astral.sh/uv/getting-started/installation/). + +--- + +## Local Setup + +```bash +# Clone the repository +git clone git@github.com:m-lab/iqb.git +cd iqb + +# Install all workspace dependencies, including dev tools +uv sync --dev +``` + +`uv sync --dev` creates a `.venv` directory at the repo root and installs all +dependencies defined across workspace members (`library/`, `prototype/`, +`analysis/`, `docs/internals/`). The `mlab-iqb` library package is installed +in editable mode from `library/`. + +### VSCode Setup + +Open the repo root in VSCode after running `uv sync --dev`. VSCode will +prompt you to install the recommended extensions: + +- `ms-python.python` +- `ms-python.vscode-pylance` +- `charliermarsh.ruff` + +If you open VSCode before running `uv sync --dev`, Ruff will report a binary +not found error. Fix it by running the **IQB: Setup Development Environment** +task from the Command Palette (`Cmd+Shift+P` → "Tasks: Run Task"). + +After setup, reload the window (`Cmd+Shift+P` → "Developer: Reload Window"). + +--- + +## Running the Prototype + +```bash +cd prototype +uv run streamlit run Home.py +``` + +The app is available at `http://localhost:8501`. Streamlit reloads on file +save. To run with automatic reloading disabled (e.g., in a headless +environment): + +```bash +uv run streamlit run Home.py --server.headless true +``` + +### Running with Docker + +```bash +# Build from the repo root +docker build -f prototype/Dockerfile -t iqb-prototype:local . + +# Run and access at http://localhost:8501 +docker run --rm -p 8501:8501 iqb-prototype:local +``` + +--- + +## Code Structure Conventions + +### Library (`library/src/iqb/`) + +- Public constructors and top-level functions use **keyword-only arguments**. + This makes call sites explicit and allows adding parameters without + breaking callers. Prefer `IQBCache(remote_cache=...)` over positional args. +- Module-level constants (e.g., `IQB_CONFIG`) are the single source of truth + for configuration. Avoid duplicating threshold values in multiple places. +- Type annotations are enforced by Pyright. All new public functions must be + fully annotated. + +### Prototype (`prototype/`) + +- Scoring logic belongs in the library. The prototype only calls library APIs. +- `utils/calculation_utils.py` — functions that translate Streamlit session + state into library-compatible data structures. +- `utils/data_utils.py` — data loading and filtering helpers. +- `utils/constants.py` — UI constants (colors, input ranges, weight bounds). + Do not hardcode these values in page files. +- `visualizations/` — self-contained chart components. Pages import from + here; they do not construct Plotly figures directly. +- `session_state.py` — defines and initialises all Streamlit session state + keys. Add new state keys here, not inline in page files. +- `app_state.py` — the `IQBAppState` dataclass passed to utility functions. + +--- + +## Adding a New Dashboard Page + +Streamlit automatically discovers Python files in `prototype/pages/`. + +1. Create `prototype/pages/My_Page.py`. The filename becomes the sidebar + label (underscores are converted to spaces). +2. Import shared utilities: + +```python +import streamlit as st +from app_state import IQBAppState +from session_state import get_app_state +from utils.data_utils import load_cache_data +``` + +3. Retrieve session state and render: + +```python +state: IQBAppState = get_app_state() +st.title("My Page") +# Use state.manual_entry, state.thresholds, etc. +``` + +4. For charts, add a function in `visualizations/` and call it from the page. + +--- + +## Adding a New Metric (Network Requirement) + +Network requirements are the measurable properties evaluated per use case: +`download_throughput_mbps`, `upload_throughput_mbps`, `latency_ms`, +`packet_loss`. + +To add a new metric, e.g. `jitter_ms`: + +1. **Extend `IQB_CONFIG`** in `library/src/iqb/config.py`: + +```python +"jitter_ms": { + "w": 3, + "threshold min": 30, + "datasets": { + "m-lab": {"w": 1}, + "cloudflare": {"w": 0}, + "ookla": {"w": 0}, + }, +}, +``` + +Add this block under each use case's `"network requirements"` dict where +the metric is relevant. + +2. **Implement binary scoring** in `library/src/iqb/calculator.py`: + +```python +elif network_requirement == "jitter_ms": + return 1 if value < threshold else 0 +``` + +3. **Update constants** in `prototype/utils/constants.py` if the metric + needs UI input controls (input ranges, step values, display colours). + +4. **Update `session_state.py`** to initialise a default value for the + new metric in `manual_entry` and in any threshold override structures. + +5. **Add tests** in `library/tests/` covering the new scoring branch and + any edge cases (zero, boundary, above/below threshold). + +--- + +## Adding a New Use Case + +1. Add a new key to the `"use cases"` dict in `library/src/iqb/config.py`: + +```python +"cloud gaming": { + "w": 1, + "network requirements": { + "download_throughput_mbps": {"w": 4, "threshold min": 50, "datasets": {...}}, + "upload_throughput_mbps": {"w": 4, "threshold min": 50, "datasets": {...}}, + "latency_ms": {"w": 5, "threshold min": 10, "datasets": {...}}, + "packet_loss": {"w": 4, "threshold min": 0.005, "datasets": {...}}, + }, +}, +``` + +2. `IQBCalculator` automatically includes the new use case in score + computation without further code changes. + +3. Update `prototype/utils/constants.py` to assign a display colour for + the new use case in `USE_CASE_COLORS` if it will appear in charts. + +--- + +## Extending IQB Scoring Logic Safely + +- `IQB_CONFIG` is the only place weights and thresholds are defined. + Do not hard-code numeric thresholds elsewhere. +- `IQBCalculator.calculate_iqb_score()` accepts a `data` dict keyed by + dataset name, then by requirement name. Use this signature for all + new consumers; do not modify the function signature. +- Use `copy.deepcopy(IQB_CONFIG)` when creating per-session overrides + (as done in `prototype/utils/calculation_utils.py`). Never mutate the + module-level `IQB_CONFIG` at runtime. +- The `config` parameter of `IQBCalculator` currently only supports `None` + (uses the default). File-based configuration loading is stubbed and + raises `NotImplementedError`. Extend this path when adding YAML/JSON + configuration support. + +--- + +## Where Caching Should Be Applied + +| Layer | Mechanism | Location | +|-------|-----------|----------| +| BigQuery results | Parquet files | `data/cache/v1/` | +| Remote sharing | GCS via `iqb cache push/pull` | `gs://mlab-sandbox-iqb-us-central1` | +| Dashboard runtime | Static JSON per country/ASN | `prototype/cache/` | +| Streamlit session | `st.cache_data` / session state | `prototype/session_state.py` | + +Use `@st.cache_data` for functions that load or transform data files. Avoid +caching at the page level; cache at the data-loading function level to allow +pages to share results across re-renders. + +Use `IQBCache` (not direct Parquet reads) as the access layer in notebooks +and scripts to benefit from its filtering and error handling. + +--- + +## Testing Strategy + +Tests live in `library/tests/` and follow the `*_test.py` naming convention. + +```bash +# Run all tests from the repo root +uv sync --dev +cd library +uv run pytest + +# Run with coverage +uv run pytest --cov=src/iqb + +# Run a specific test file +uv run pytest tests/iqb_score_test.py + +# Run a specific test class or function +uv run pytest tests/iqb_score_test.py::TestIQBInitialization::test_init_with_name +``` + +### What to Test + +- **Scoring correctness** — verify `calculate_iqb_score()` returns known + values for known inputs. Cover boundary conditions (value exactly at + threshold, zero weight, all-zero data). +- **Config integrity** — assert that all use cases and requirements in + `IQB_CONFIG` have required keys (`w`, `threshold min`, `datasets`). +- **Cache reads** — use fixture Parquet files in `library/tests/fixtures/` + to test `IQBCache` without a live GCS connection. +- **Pipeline queries** — unit-test SQL template rendering independently of + BigQuery execution. + +### Test File Structure + +```python +"""tests/my_feature_test.py""" +from iqb import IQBCalculator + +class TestMyFeature: + def test_something(self): + calculator = IQBCalculator() + result = calculator.calculate_iqb_score(data={ + "m-lab": { + "download_throughput_mbps": 50, + "upload_throughput_mbps": 20, + "latency_ms": 30, + "packet_loss": 0.001, + } + }) + assert 0.0 <= result <= 1.0 +``` + +--- + +## Code Quality + +```bash +# Lint and auto-fix (from library/) +cd library +uv run ruff check --fix . +uv run ruff format . + +# Type checking +uv run pyright +``` + +Ruff and Pyright configurations are in `library/pyproject.toml`. CI runs +both checks on all pushes and pull requests to `main`. + +--- + +## Contribution Workflow + +1. **Fork and branch** — create a feature branch from `main`: + `git checkout -b feature/my-change` + +2. **Sync dependencies** — run `uv sync --dev` after pulling changes to + keep the lockfile consistent. + +3. **Write tests first** for library changes. Aim for full coverage of new + code paths in `library/tests/`. + +4. **Run quality checks locally** before pushing: + + ```bash + cd library + uv run pytest + uv run ruff check . + uv run pyright + ``` + +5. **Commit style** — write concise commit messages in the imperative mood + (`Add jitter metric to config`, not `Added jitter metric`). + +6. **Pull Request** — open a PR against `main`. CI will run tests, linting, + and type checks. Address all failures before requesting review. + +7. **Review and merge** — at least one maintainer review is required. + Squash merge is preferred to keep the commit history linear. + +See [CONTRIBUTING.md](../CONTRIBUTING.md) for component-specific workflows +and VSCode task configurations. diff --git a/docs/user_guide.md b/docs/user_guide.md new file mode 100644 index 0000000..4658dff --- /dev/null +++ b/docs/user_guide.md @@ -0,0 +1,207 @@ +# IQB User Guide + +## Intended Audiences + +This guide is written for four primary audiences: + +- **Consumers** — individuals who want to understand how well their Internet + connection performs for everyday tasks. +- **Policymakers** — government officials, regulators, and public interest + groups evaluating broadband quality at a regional or national level. +- **Researchers** — academics and data scientists studying Internet + performance trends and methodologies. +- **Internet Service Providers (ISPs)** — operators who want to understand + how their network performance is measured and contextualized. + +--- + +## What Is IQB? + +The Internet Quality Barometer (IQB) is an open-source framework developed +by [Measurement Lab (M-Lab)](https://www.measurementlab.net/) that produces +a composite quality score for Internet connections. + +Traditional "speed tests" measure a single number — how fast data moves — but +that number alone does not tell you whether a connection is suitable for video +conferencing, online gaming, or cloud backup. Different applications have +different network requirements. + +IQB addresses this by evaluating a connection across six use cases: + +| Use Case | Primary Concern | +|----------|----------------| +| Web browsing | Download speed, low latency | +| Video streaming | Sustained download speed | +| Audio streaming | Consistent download speed, low packet loss | +| Video conferencing | Symmetric speed, very low latency and packet loss | +| Online backup | Upload speed | +| Gaming | Extremely low latency, very low packet loss | + +For each use case, IQB tests whether the connection meets a minimum threshold +for each relevant network requirement. The results are combined into a single +score between 0 and 1. + +--- + +## How the IQB Score Is Calculated + +The IQB score is calculated in three steps. + +**Step 1 — Binary requirement check.** For each use case, IQB checks whether +a measured value meets the minimum required threshold: + +- For download and upload throughput, the connection must exceed the + threshold (more is better). +- For latency and packet loss, the connection must be below the threshold + (less is better). + +A requirement is scored 1 (pass) or 0 (fail) per dataset. + +**Step 2 — Requirement agreement score.** When multiple measurement datasets +are active, their pass/fail results are averaged. This reduces the influence +of any single dataset's measurement conditions. + +**Step 3 — Weighted aggregation.** Requirement scores within a use case are +combined using requirement-specific weights. Use case scores are then combined +using use-case weights. The final IQB score is a weighted average across all +active use cases. + +A score of 1.0 means the connection meets all thresholds for all use cases. +A score of 0.0 means it fails every threshold. In practice most connections +score somewhere in between. + +--- + +## Understanding Network Metrics + +### Latency (Round-Trip Time) + +Latency is the time it takes for a data packet to travel from your device to +a server and back, measured in milliseconds (ms). Lower is better. + +- Under 50 ms: excellent for all use cases including gaming and video calls +- 50–100 ms: acceptable for most tasks; may cause noticeable delay in gaming +- Over 100 ms: may cause interruptions in real-time applications + +Latency is determined by the physical distance to the server, congestion in +the network, and the quality of routing. + +### Throughput (Download and Upload Speed) + +Throughput is the rate at which data is transferred, measured in megabits per +second (Mbps). Higher is better. + +- **Download** — data received by your device (web pages, video streams) +- **Upload** — data sent from your device (video conference streams, backups) + +The minimum threshold varies by use case. Video conferencing requires +symmetric speeds (25 Mbps in each direction), while web browsing is +satisfied by lower thresholds (10 Mbps download). + +### Packet Loss + +Packet loss is the percentage of data packets that are sent but never +received. Lower is better; ideally 0%. + +Even small amounts of packet loss (0.5–1%) can cause visible degradation in +video calls and online gaming. Audio and video streaming may buffer or stall. + +--- + +## Interpreting Percentile Charts + +IQB scores and measurements are typically presented as percentile +distributions, not single points. + +A percentile chart shows what fraction of measured connections achieve a +given speed or score. For example: + +- **50th percentile (p50)** — the median. Half of connections perform above + this value, half below. +- **10th percentile (p10)** — 10% of connections perform at or below this + level; represents the experience of the worst-served users. +- **90th percentile (p90)** — 90% of connections perform at or below this + level; represents the best-served majority. + +When evaluating a region or ISP, the 10th percentile is often the most +policy-relevant number: it characterises the experience of users who are +most poorly served, not the average or the best case. + +--- + +## Comparing ISPs Responsibly + +IQB measurements do not rank ISPs as simply better or worse. Several +factors affect interpretation: + +**Coverage and sample size.** ISPs serving dense urban areas accumulate more +test samples than those serving rural areas. Smaller sample sizes produce +less stable estimates. Always check the sample count before drawing +conclusions from comparisons. + +**Geography.** Latency depends heavily on where a server is located relative +to users. An ISP that routes traffic locally may show lower latency than one +that backhaults traffic over long distances, regardless of underlying network +quality. + +**Test methodology.** M-Lab NDT measurements are conducted under specific +conditions. They reflect the performance experienced by users who run the +test, which may not represent all traffic or all times of day. + +**Time periods.** Compare measurements taken within the same time window. +Infrastructure upgrades or network events can cause significant changes +between periods. + +--- + +## Why Sample Size Matters + +A single measurement taken at one moment gives limited information. IQB +aggregates measurements from many users over a calendar month to produce +stable estimates. However: + +- Areas with fewer than a few hundred measurements in a period should be + interpreted with caution. +- A jurisdiction with 100,000 tests per month provides far more reliable + estimates than one with 200. +- The dashboard indicates sample counts where available. Treat low-sample + results as indicative, not conclusive. + +--- + +## Limitations of Internet Performance Measurements + +IQB is designed to be transparent about what it can and cannot measure. + +**Measurement conditions are not controlled.** Tests measure performance at +the moment a user runs them. Results can vary by time of day, congestion on +the local loop, the user's device, and the location of the test server. + +**IQB reflects a specific set of use cases.** The current framework covers +six use cases. A connection might perform well for web browsing but poorly +for gaming. The composite score is useful for general comparisons but does +not capture every dimension of quality. + +**Thresholds reflect current standards, not absolutes.** The thresholds used +to determine pass/fail for each requirement are based on current application +requirements. As applications evolve — for example, 4K video streaming +requires higher download speeds than 1080p — thresholds may need revision. + +**IQB does not measure network neutrality or congestion throttling.** The +score reflects observed performance, not the underlying causes. Poor scores +may result from infrastructure limitations, congestion, or other factors +unrelated to deliberate network management. + +**Data availability varies by region.** Countries with lower M-Lab test +deployment or lower user participation will have less comprehensive +measurement coverage. + +--- + +## Further Reading + +- M-Lab blog post: [Introducing IQB](https://www.measurementlab.net/blog/iqb/) +- IQB framework report: [Detailed Report (PDF)](https://www.measurementlab.net/publications/IQB_report_2025.pdf) +- IQB executive summary: [Executive Summary (PDF)](https://www.measurementlab.net/publications/IQB_executive_summary_2025.pdf) +- IQB poster (ACM IMC 2025): [arXiv](https://arxiv.org/pdf/2509.19034) +- Live prototype: [iqb.mlab-staging.measurementlab.net](https://iqb.mlab-staging.measurementlab.net/)