Add --heatmap flag for time-series heatmap visualization of I/O patterns

README.md

@@ -137,6 +137,48 @@ When enabled, two histogram charts are displayed alongside the results table:
 
 Both charts display separate lines for reads, writes, and all operations combined, making it easy to identify patterns in your code's I/O behavior.
 
+### Heatmap Visualization
+
+Use the `--heatmap` flag to visualize I/O operations as a time-series heatmap (available for `strace` and `fs_usage` measurement modes):
+
+**Example - Visualizing I/O patterns over time:**
+```python
+%%iops --heatmap
+import tempfile
+import os
+import time
+
+test_dir = tempfile.mkdtemp()
+
+try:
+    # Phase 1: Small writes over time
+    for i in range(5):
+        with open(os.path.join(test_dir, f'small_{i}.txt'), 'w') as f:
+            f.write('x' * 1024)  # 1 KB
+        time.sleep(0.05)
+
+    # Phase 2: Large writes over time
+    for i in range(5):
+        with open(os.path.join(test_dir, f'large_{i}.txt'), 'w') as f:
+            f.write('z' * (100 * 1024))  # 100 KB
+        time.sleep(0.05)
+
+finally:
+    import shutil
+    if os.path.exists(test_dir):
+        shutil.rmtree(test_dir)
+```
+
+When enabled, two heatmap heatmaps are displayed alongside the results table:
+1. **Operation Count Over Time**: Shows when I/O operations of different sizes occurred (X-axis: time, Y-axis: operation size in log scale, Color: operation count)
+2. **Total Bytes Over Time**: Shows data transfer patterns over time (X-axis: time, Y-axis: operation size in log scale, Color: total bytes)
+
+The heatmap visualization helps identify:
+- Temporal patterns in I/O behavior
+- When different I/O sizes occur during execution
+- Bursts or gaps in I/O activity
+- Correlation between application phases and I/O patterns
+
 ## Platform Support
 
 - **Linux/Windows**: Uses `psutil` for per-process I/O tracking
@@ -147,8 +189,8 @@ Both charts display separate lines for reads, writes, and all operations combine
 - Python 3.10+
 - IPython/Jupyter
 - psutil
-- matplotlib (for histogram visualization)
-- numpy (for histogram visualization)
+- matplotlib (for histogram and heatmap visualization)
+- numpy (for histogram and heatmap visualization)
 
 ## Dev Guide - Getting Started
 

docs/conf.py

@@ -24,7 +24,12 @@
 # -- General configuration ---------------------------------------------------
 # https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration
 
-extensions = ["sphinx.ext.mathjax", "sphinx.ext.napoleon", "sphinx.ext.viewcode"]
+extensions = [
+    "sphinx.ext.mathjax",
+    "sphinx.ext.napoleon",
+    "sphinx.ext.viewcode",
+    "nbsphinx",  # For executing and rendering Jupyter notebooks
+]
 
 extensions.append("autoapi.extension")
 extensions.append("nbsphinx")
@@ -61,4 +66,12 @@
 autoapi_add_toc_tree_entry = False
 autoapi_member_order = "bysource"
 
+# -- nbsphinx configuration -------------------------------------------------
+# Execute notebooks before conversion
+nbsphinx_execute = "always"
+# Allow errors in notebook execution (for documentation purposes)
+nbsphinx_allow_errors = True
+# Timeout for notebook execution (in seconds)
+nbsphinx_timeout = 600
+
 html_theme = "sphinx_rtd_theme"

docs/index.rst

@@ -85,4 +85,3 @@ Notes:
    into documentation for ReadTheDocs works as expected. For more information, see
    the Python Project Template documentation on
    `Sphinx and Python Notebooks <https://lincc-ppt.readthedocs.io/en/latest/practices/sphinx.html#python-notebooks>`_.
-

docs/notebooks.rst

@@ -9,6 +9,7 @@ These Jupyter notebooks demonstrate the key features of iops-profiler with pract
 
    notebooks/basic_usage
    notebooks/histogram_visualization
+   notebooks/heatmap_visualization
 
 Running the Notebooks
 ---------------------
@@ -72,6 +73,24 @@ Explore the histogram feature for visualizing I/O patterns:
 
 **Note:** Histogram mode is available on Linux and macOS, but not Windows.
 
+Time-Series Heatmap Visualization
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+:doc:`notebooks/heatmap_visualization`
+
+Explore the heatmap feature for visualizing I/O patterns over time:
+
+- Enabling heatmap mode with ``--heatmap``
+- Understanding when operations of different sizes occur
+- Analyzing temporal I/O patterns and bursts
+- Identifying phases of different I/O behavior
+- Comparing heatmap with histogram views
+- Real-world temporal analysis examples
+
+**Recommended for:** Users who want to understand how I/O behavior changes during program execution.
+
+**Note:** Heatmap mode is available on Linux (with strace) and macOS (with fs_usage), but not Windows.
+
 Additional Resources
 --------------------
 

docs/notebooks/heatmap_visualization.ipynb

@@ -0,0 +1,239 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Heatmap Visualization Example\n",
+    "\n",
+    "This notebook demonstrates the heatmap feature of iops-profiler, which provides a time-series heatmap view of I/O operations.\n",
+    "\n",
+    "The heatmap shows:\n",
+    "- **X-axis**: Time (runtime in seconds)\n",
+    "- **Y-axis**: Operation size (bytes, log scale)\n",
+    "- **Color**: Either the number of operations or total bytes transferred\n",
+    "\n",
+    "This visualization helps identify I/O patterns over time, showing when different sizes of operations occur during program execution."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Load the iops-profiler extension\n",
+    "%load_ext iops_profiler"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Example 1: Simple I/O with Varying Sizes\n",
+    "\n",
+    "Let's create a workload that performs I/O operations of different sizes over time:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%%iops --heatmap\n",
+    "import tempfile\n",
+    "import os\n",
+    "import time\n",
+    "\n",
+    "# Create a temporary directory for our test files\n",
+    "test_dir = tempfile.mkdtemp()\n",
+    "\n",
+    "try:\n",
+    "    # Phase 1: Small writes (1 KB each)\n",
+    "    for i in range(5):\n",
+    "        with open(os.path.join(test_dir, f'small_{i}.txt'), 'w') as f:\n",
+    "            f.write('x' * 1024)  # 1 KB\n",
+    "        time.sleep(0.05)  # Small delay to spread operations over time\n",
+    "    \n",
+    "    # Phase 2: Medium writes (10 KB each)\n",
+    "    for i in range(5):\n",
+    "        with open(os.path.join(test_dir, f'medium_{i}.txt'), 'w') as f:\n",
+    "            f.write('y' * (10 * 1024))  # 10 KB\n",
+    "        time.sleep(0.05)\n",
+    "    \n",
+    "    # Phase 3: Large writes (100 KB each)\n",
+    "    for i in range(5):\n",
+    "        with open(os.path.join(test_dir, f'large_{i}.txt'), 'w') as f:\n",
+    "            f.write('z' * (100 * 1024))  # 100 KB\n",
+    "        time.sleep(0.05)\n",
+    "    \n",
+    "    # Phase 4: Mixed reads - read back all files in order\n",
+    "    for size_category in ['small', 'medium', 'large']:\n",
+    "        for i in range(5):\n",
+    "            filepath = os.path.join(test_dir, f'{size_category}_{i}.txt')\n",
+    "            if os.path.exists(filepath):\n",
+    "                with open(filepath, 'r') as f:\n",
+    "                    _ = f.read()\n",
+    "                time.sleep(0.03)\n",
+    "\n",
+    "finally:\n",
+    "    # Cleanup\n",
+    "    import shutil\n",
+    "    if os.path.exists(test_dir):\n",
+    "        shutil.rmtree(test_dir)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The heatmap above shows:\n",
+    "\n",
+    "1. **Left plot (Operation Count)**: How many I/O operations occurred in each time/size bin\n",
+    "2. **Right plot (Total Bytes)**: How much data was transferred in each time/size bin\n",
+    "\n",
+    "You can see distinct phases:\n",
+    "- Early time: Small operations (1 KB writes)\n",
+    "- Middle time: Medium operations (10 KB writes)\n",
+    "- Later time: Large operations (100 KB writes)\n",
+    "- Final time: Mixed reads of all sizes"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Example 2: Bursty I/O Pattern\n",
+    "\n",
+    "Let's create a workload with bursts of activity at different scales:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%%iops --heatmap\n",
+    "import tempfile\n",
+    "import os\n",
+    "import time\n",
+    "\n",
+    "test_dir = tempfile.mkdtemp()\n",
+    "\n",
+    "try:\n",
+    "    # Burst 1: Many small operations\n",
+    "    for i in range(20):\n",
+    "        with open(os.path.join(test_dir, f'burst1_{i}.txt'), 'w') as f:\n",
+    "            f.write('a' * 512)  # 512 bytes\n",
+    "    \n",
+    "    # Pause\n",
+    "    time.sleep(0.2)\n",
+    "    \n",
+    "    # Burst 2: Few large operations\n",
+    "    for i in range(3):\n",
+    "        with open(os.path.join(test_dir, f'burst2_{i}.txt'), 'w') as f:\n",
+    "            f.write('b' * (200 * 1024))  # 200 KB\n",
+    "    \n",
+    "    # Pause\n",
+    "    time.sleep(0.2)\n",
+    "    \n",
+    "    # Burst 3: Medium-sized operations\n",
+    "    for i in range(10):\n",
+    "        with open(os.path.join(test_dir, f'burst3_{i}.txt'), 'w') as f:\n",
+    "            f.write('c' * (5 * 1024))  # 5 KB\n",
+    "\n",
+    "finally:\n",
+    "    import shutil\n",
+    "    if os.path.exists(test_dir):\n",
+    "        shutil.rmtree(test_dir)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "This heatmap shows a bursty pattern with clear gaps between activity phases. The color intensity helps identify which phases had the most activity or transferred the most data."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Comparing with Histogram View\n",
+    "\n",
+    "For comparison, here's the same workload with the histogram view:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%%iops --histogram\n",
+    "import tempfile\n",
+    "import os\n",
+    "import time\n",
+    "\n",
+    "test_dir = tempfile.mkdtemp()\n",
+    "\n",
+    "try:\n",
+    "    # Same workload as Example 1\n",
+    "    for i in range(5):\n",
+    "        with open(os.path.join(test_dir, f'small_{i}.txt'), 'w') as f:\n",
+    "            f.write('x' * 1024)\n",
+    "        time.sleep(0.05)\n",
+    "    \n",
+    "    for i in range(5):\n",
+    "        with open(os.path.join(test_dir, f'medium_{i}.txt'), 'w') as f:\n",
+    "            f.write('y' * (10 * 1024))\n",
+    "        time.sleep(0.05)\n",
+    "    \n",
+    "    for i in range(5):\n",
+    "        with open(os.path.join(test_dir, f'large_{i}.txt'), 'w') as f:\n",
+    "            f.write('z' * (100 * 1024))\n",
+    "        time.sleep(0.05)\n",
+    "\n",
+    "finally:\n",
+    "    import shutil\n",
+    "    if os.path.exists(test_dir):\n",
+    "        shutil.rmtree(test_dir)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The histogram shows the distribution of operation sizes but doesn't reveal when they occurred. The heatmap adds the temporal dimension, making it easier to:\n",
+    "\n",
+    "- Identify when different I/O patterns occur during execution\n",
+    "- Spot bursts or gaps in I/O activity\n",
+    "- Correlate I/O behavior with application phases\n",
+    "- Debug performance issues related to I/O timing"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.0"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}