Releases: frgfm/torch-cam
v0.4.0: Evaluation metrics and PyTorch 2.0
This minor release adds evaluation metrics to the package and bumps PyTorch to version 2.0
Note: TorchCAM 0.4.0 requires PyTorch 2.0.0 or higher.
Highlights
Evaluation metrics
This release comes with a standard way to evaluate interpretability methods. This allows users to better evaluate models' robustness:
from functools import partial
from torchcam.metrics import ClassificationMetric
metric = ClassificationMetric(cam_extractor, partial(torch.softmax, dim=-1))
metric.update(input_tensor)
metric.summary()
What's Changed
New Features 🚀
- feat: Added CAM evaluation metric by @frgfm in #172
- ci: Added FUNDING button by @frgfm in #182
- feat: Added new TV models to demo by @frgfm in #184
- ci: Added precommits, bandit & autoflake by @frgfm in #192
- feat: Removes model hooks when the context manager exits by @frgfm in #198
Bug Fixes 🐛
- chore: Applied post-release modifications by @frgfm in #180
- fix: Fixed division by zero during normalization by @frgfm in #185
- fix: Fixed zero division for weight computation in gradient based methods by @frgfm in #187
- docs: Fixes README badges by @frgfm in #194
- ci: Fixes issue templates by @frgfm in #196
- fix: Fixes SmoothGradCAMpp by @frgfm in #204
Improvements
- docs: Improved documentation build script by @frgfm in #183
- docs: Updates README & CONTRIBUTING by @frgfm in #193
- feat: Removed param grad computation in scripts by @frgfm in #201
- style: Updates precommit hooks and mypy config by @frgfm in #203
- refactor: Replaces flake8 by ruff and updates python version by @frgfm in #211
- style: Bumps ruff & black, removes isort & pydocstyle by @frgfm in #216
- style: Bumps ruff and updates torch & torchvision version specifiers by @frgfm in #219
- docs: Updates CONTRIBUTING & README by @frgfm in #220
- test: Speeds up test suite using plugins by @frgfm in #222
- ci: Adds multiple build CI jobs by @frgfm in #223
- feat: Removes warnings for torchvision and matplotlib by @frgfm in #224
Full Changelog: v0.3.2...v0.4.0
v0.3.2: Some CAM fixes and support of batch processing
This patch release fixes the Score-CAM methods and improves the base API for CAM computation.
Note: TorchCAM 0.3.2 requires PyTorch 1.7.0 or higher.
Highlights
😯 Batch processing
CAM computation now supports batch sizes larger than 1 (#143) ! Practically, this means that you can compute CAMs for multiple samples at the same time, which will let you make the most of your GPU as well ⚡
The following snippet:
import torch
from torchcam.methods import LayerCAM
from torchvision.models import resnet18
# A preprocessed (resized & normalized) tensor
img_tensor = torch.rand((2, 3, 224, 224))
model = resnet18(pretrained=True).eval()
# Hook your model before inference
cam_extractor = LayerCAM(model)
out = model(img_tensor)
# Compute the CAM
activation_map = cam_extractor(out[0].argmax().item(), out)
print(activation_map[0].ndim)
will yield 3
as the batch dimension is now also used.
🖌️ Documentation theme
New year, new documentation theme!
For clarity and improved interface, the documentation theme was changed from Read the Docs to Furo (#162)
This comes with nice features like dark mode and edit button!
🖱️ Contribution process
Contributions are important to OSS projects, and for this reason, a few improvements were made to the contribution process:
- added a Makefile for easier development (#109)
- added a dedicated README for the documentation (#109)
- updated CONTRIBUTING (#109, #166)
Breaking changes
CAM signature
CAM extractors now outputs a list of tensors. The size of the list is equal to the number of target layers and ordered the same way.
Each of these elements used to be a 2D spatial tensor, and is now a 3D tensor to include the batch dimension:
# Model was hooked and a tensor of shape (2, 3, 224, 224) was forwarded to it
amaps = cam_extractor(0, out)
for elt in amaps: print(elt.shape)
will, from now on, yield
torch.Size([2, 7, 7])
What's Changed
Breaking Changes 🛠
New Features 🚀
- ci: Added release note template and a job to check PR labels by @frgfm in #138
- docs: Added CITATION file by @frgfm in #144
Bug Fixes 🐛
- fix: Updated headers and added pydocstyle by @frgfm in #137
- chore: Updated PyTorch version specifier by @frgfm in #149
- docs: Fixed deprecated method call by @frgfm in #158
- chore: Fixed jinja2 deps (subdep of sphinx) by @frgfm in #159
- docs: Fixed docstring of ISCAM by @frgfm in #160
- docs: Fixed multi-version build by @frgfm in #163
- docs: Fixed codacy badge by @frgfm in #164
- docs: Fixed typo in CONTRIBUTING by @frgfm in #166
- docs: Fixed author entry in pyproject by @frgfm in #168
- style: Fixed import order by @frgfm in #175
Improvements
- docs: Added PR template and tools for contributing by @frgfm in #109
- refactor: Removed unused import by @frgfm in #110
- feat: Added text strip for multiple target selection in demo by @frgfm in #111
- refactor: Updated environment collection script by @frgfm in #112
- style: Updated flake8 config by @frgfm in #115
- ci: Updated isort config and related CI job by @frgfm in #118
- ci: Speeded up the example script CI check by @frgfm in #130
- refactor: Updated the timing function for latency eval by @frgfm in #129
- docs: Updated TOC of documentation by @frgfm in #161
- refactor: Updated build config and documentation theme by @frgfm in #162
- style: Updated mypy and isort configs by @frgfm in #167
- chore: Improved version specifiers and fixed conda recipe by @frgfm in #169
- docs: Fixed README badge and updated documentation by @frgfm in #170
- ci: Updated release job by @frgfm in #173
- refactor: Improved target resolution by @frgfm in #174
- ci: Updated the trigger for the release job by @frgfm in #176
- docs: Updated landing page screenshot by @frgfm in #179
Full Changelog: v0.3.1...v0.3.2
v0.3.1: Improved demo & reorganized package
This patch release adds new features to the demo and reorganizes the package for a clearer hierarchy.
Note: TorchCAM 0.3.1 requires PyTorch 1.5.1 or higher.
Highlights
CAM fusion is coming to the demo 🚀
With release 0.3.0, the support of multiple target layers was added as well as CAM fusion. The demo was updated to automatically fuse CAMs when you hooked multiple layers (add a "+" separator between each layer name):
Breaking changes
Submodule renaming
To anticipate further developments of the library, modules were renamed:
torchcam.cams
was renamed intotorchcam.methods
torchcam.cams.utils
was renamed and made private (torchcam.methods._utils
) since it's API may evolve quickly- activation-based CAM methods are now implemented in
torchcam.methods.activation
rather thantorchcam.cams.cam
- gradient-based CAM methods are now implemented in
torchcam.methods.gradient
rather thantorchcam.cams.gradcam
0.3.0 | 0.3.1 |
---|---|
>>> from torchcam.cams import LayerCAM |
>>> from torchcam.methods import LayerCAM |
What's Changed
- chore: Made post release modifications by @frgfm in #103
- docs: Updated changelog by @frgfm in #104
- feat: Added possibility to retrieve multiple CAMs in demo by @frgfm in #105
- refactor: Reorganized package hierarchy by @frgfm in #106
- docs: Fixed LaTeX syntax in docstrings by @frgfm in #107
Full Changelog: v0.3.0...v0.3.1
v0.3.0: Support of Layer-CAM & multi-layer CAM computation
This release extends CAM methods with Layer-CAM, greatly improves the core features (CAM computation for multiple layers at once, CAM fusion, support of torch.nn.Module
), while improving accessibility for entry users.
Note: TorchCAM 0.3.0 requires PyTorch 1.5.1 or higher.
Highlights
Enters Layer-CAM
The previous release saw the introduction of Score-CAM variants, and this one introduces you to Layer-CAM, which is meant to be considerably faster, while offering very competitive localization cues!
Just like any other CAM methods, you can now use it as follows:
from torchcam.cams import LayerCAM
# model = ....
# Hook the model
cam_extractor = LayerCAM(model)
Consequently, the illustration of visual outputs for all CAM methods has been updated so that you can better choose the option that suits you:
Computing CAMs for multiple layers & CAM fusion
A class activation map is specific to a given layer in a model. To fully capture the influence of visual traits on your classification output, you might want to explore the CAMs for multiple layers.
For instance, here are the CAMs on the layers "layer2", "layer3" and "layer4" of a resnet18
:
from torchvision.io.image import read_image
from torchvision.models import resnet18
from torchvision.transforms.functional import normalize, resize, to_pil_image
import matplotlib.pyplot as plt
from torchcam.cams import LayerCAM
from torchcam.utils import overlay_mask
# Download an image
!wget https://www.woopets.fr/assets/races/000/066/big-portrait/border-collie.jpg
# Set this to your image path if you wish to run it on your own data
img_path = "border-collie.jpg"
# Get your input
img = read_image(img_path)
# Preprocess it for your chosen model
input_tensor = normalize(resize(img, (224, 224)) / 255., [0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
# Get your model
model = resnet18(pretrained=True).eval()
# Hook the model
cam_extractor = LayerCAM(model, ["layer2", "layer3", "layer4"])
out = model(input_tensor.unsqueeze(0))
cams = cam_extractor(out.squeeze(0).argmax().item(), out)
# Plot the CAMs
_, axes = plt.subplots(1, len(cam_extractor.target_names))
for idx, name, cam in zip(range(len(cam_extractor.target_names)), cam_extractor.target_names, cams):
axes[idx].imshow(cam.numpy()); axes[idx].axis('off'); axes[idx].set_title(name);
plt.show()
Now, the way you would combine those together is up to you. By default, most approaches use an element-wise maximum. But, LayerCAM has its own fusion method:
# Let's fuse them
fused_cam = cam_extractor.fuse_cams(cams)
# Plot the raw version
plt.imshow(fused_cam.numpy()); plt.axis('off'); plt.title(" + ".join(cam_extractor.target_names)); plt.show()
# Overlay it on the image
result = overlay_mask(to_pil_image(img), to_pil_image(fused_cam, mode='F'), alpha=0.5)
# Plot the result
plt.imshow(result); plt.axis('off'); plt.title(" + ".join(cam_extractor.target_names)); plt.show()
Support of torch.nn.Module
as target_layer
While making the API more robust, CAM constructors now also accept torch.nn.Module
as target_layer
. Previously, you had to pass the name of the layer as string, but you can now pass the object reference directly if you prefer:
from torchcam.cams import LayerCAM
# model = ....
# Hook the model
cam_extractor = LayerCAM(model, model.layer4)
⚡ Latency benchmark ⚡
Since CAMs can be used from localization or production pipelines, it is important to consider latency along with pure visual output quality. For this reason, a latency evaluation script has been included in this release along with a full benchmark table.
Should you wish to have latency metrics on your dedicated hardware, you can run the script on your own:
python scripts/eval_latency.py SmoothGradCAMpp --size 224
Notebooks ⏯️
Do you prefer to only run code rather than write it? Perhaps you only want to tweak a few things?
Then enjoy the brand new Jupyter notebooks than you can either run locally or on Google Colab!
🤗 Live demo 🤗
The ML community was recently blessed by HuggingFace with their beta of Spaces, which let you host free-of-charge your ML demos!
Previously, you were able to run the demo locally on deploy it on your own, but now, you can enjoy the live demo of TorchCAM 🎨
Breaking changes
Multiple CAM output
Since CAM extractor can now compute the resulting maps for multiple layer at a time, the return type of all CAMs has been changed from torch.Tensor
to List[torch.Tensor]
with N elements, where N is the number of target layers.
0.2.0 | 0.3.0 |
---|---|
>>> from torchcam.cams import SmoothGradCAMpp >>> extractor = SmoothGradCAMpp(model) >>> out = model(input_tensor.unsqueeze(0)) >>> print(type(cam_extractor(out.squeeze(0).argmax().item(), out))) <class 'torch.Tensor'> |
>>> from torchcam.cams import SmoothGradCAMpp >>> extractor = SmoothGradCAMpp(model) >>> out = model(input_tensor.unsqueeze(0)) >>> print(type(cam_extractor(out.squeeze(0).argmax().item(), out))) <class 'list'> |
New features
CAMs
Implementations of CAM method
- Added support of conv1x1 as FC candidate in base CAM #69 (@frgfm)
- Added support of LayerCAM #77 (@frgfm)
- Added support of
torch.nn.Module
astarget_layer
orfc_layer
#83 (@frgfm) - Added support of multiple target layers for all CAM methods #89 #92 (@frgfm)
- Added layer-specific CAM fusion method #93 (@frgfm)
Scripts
Side scripts to make the most out of TorchCAM
Test
Verifications of the package well-being before release
- Added unittests to verify that conv1x1 can be used as FC in base CAM #69 (@frgfm)
- Added unittest for LayerCAM #77 (@frgfm)
- Added unittest for gradient-based CAM method for models with in-place ops #80 (@frgfm)
- Added unittest to check support of
torch.nn.Module
astarget_layer
in CAM constructor #83 #88 (@frgfm) - Added unittest for CAM fusion #93 (@frgfm)
Documentation
Online resources for potential users
- Added LayerCAM ref in the README and in the documentation #77 (@frgfm)
- Added CODE_OF_CONDUCT #86 (@frgfm)
- Added changelog to the documentation #91 (@frgfm)
- Added latency benchmark & GIF illustration of CAM on a video in README #95 (@frgfm)
- Added documentation of
.fuse_cams
method #93 (@frgfm) - Added ref to HF Space demo in README and documentation #96 (@frgfm)
- Added tutorial notebooks and reference page in the documentation #99 #100 #101 #102 (@frgfm)
Others
Other tools and implementations
- Added
class_idx
&target_layer
selection in the demo #67 (@frgfm) - Added CI jobs to build on different OS & Python versions, to validate the demo, and the example script #73 #74 (@frgfm)
- Added LayerCAM to the demo #77 (@frgfm)
- Added an environment collection script #78 (@frgfm)
- Added CI check for the latency evaluation script #95 (@frgfm)
Bug fixes
CAMs
Documentation
Others
Improvements
CAMs
- Improved weight broadcasting for all CAMs #77 (@frgfm)
- Refactored hook enabling #80 (@frgfm)
- Improved the warning message for target automatic resolution #87 #92 (@frgfm)
- Improved arg type checking for CAM constructor #88 (@frgfm)
Scripts
- Improved the layout option of the example script #66 (@frgfm)
- Refactored example script #80 #94 (@frgfm)
- Updated all scripts for support of multiple target layers #89 (@frgfm)
Test
Documentation
- Added latest release doc version & updated README badge #63 (@frgfm)
- Added demo screenshot in the README #67 (@frgfm)
- Updated instructions in README #89 (@frgfm)
- Improved documentation landing page #91 (@frgfm)
- Updated contribution guidelines #94 (@frgfm)
- Updated documentation requirements #99 (@frgfm)
Others
- Updated package version and fixed CI jobs to validate release publish #63 (@frgfm)
- Updated license from MIT to Apache 2.0 #70 (@frgfm)
- Refactored CI jobs #73 (@frgfm)
- Improved bug report template #78 (@frgfm)
- Updated streamlit syntax in demo #94 (@frgfm)
- Added isort config and CI job #97 (@frgfm)
- Added CI job for sanity check of the documentation build #98 (@frgfm)
Compatibility with 3D inputs and improved documentation
This release extends TorchCAM compatibility to 3D inputs, and improves documentation.
Note: TorchCAM 0.2.0 requires PyTorch 1.5.1 or higher.
Highlights
Compatibility for inputs with more than 2 spatial dimensions
The first papers about CAM methods were built for classification models using 2D (spatially) inputs. However, the latest methods can be extrapolated to higher dimension inputs and it's now live:
import torch
from torchcam.cams import SmoothGradCAMpp
# Define your model compatible with 3D inputs
video_model = ...
extractor = SmoothGradCAMpp(video_model)
# Forward your input
scores = model(torch.rand((1, 3, 32, 224, 224)))
# Retrieve the CAM
cam = extractor(scores[0].argmax().item(), scores)
Multi-version documentation
While documentation was up-to-date with the latest commit on the main branch, previously if you were running an older release of the library, you had no corresponding documentation.
As of now, you can select the version of the documentation you wish to access (stable releases or latest commit):
Demo app
Since spatial information is at the very core of TorchCAM, a minimal Streamlit demo app was added to explore the activation of your favorite models. You can run the demo with the following commands:
streamlit run demo/app.py
Here is how it renders retrieving the heatmap using SmoothGradCAMpp
on a pretrained resnet18
:
New features
CAMs
Implementations of CAM method
- Enabled CAM compatibility for inputs with more than 2 spatial dimensions #45 (@frgfm)
- Added support of XGradCAM #47 (@frgfm)
Test
Verifications of the package well-being before release
Documentation
Online resources for potential users
- Added references to XGradCAM in README and documentation #47 (@frgfm)
- Added multi-version documentation & added github star button #53, #54, #55, #56 (@frgfm)
- Revamped README #59 (@frgfm) focusing on short easy code snippets
- Improved documentation #60 (@frgfm)
Others
Other tools and implementations
- Added issue templates for bug report and feature request #49 (@frgfm)
- Added option to specify a single CAM method in example script #52 (@frgfm)
- Added minimal demo app #59 (@frgfm)
Bug fixes
CAMs
- Fixed automatic layer resolution on GPU #41 (@frgfm)
- Fixed backward hook warnings for Pytorch >= 1.8.0 #58 (@frgfm)
Utils
- Fixed RGBA -> RGB conversion in
overlay_mask
#38 (@alexandrosstergiou)
Test
- Fixed
overlay_mask
unittest #38 (@alexandrosstergiou)
Documentation
Others
- Fixed CI job for conda build #34 (@frgfm)
- Fixed model mode in example script #37 (@frgfm)
- Fixed sphinx version #40 (@frgfm)
- Fixed usage instructions in README #43 (@frgfm)
- Fixed example script for local image input #51 (@frgfm)
Improvements
CAMs
Test
- Added NaN check unittest for gradcam #37 (@frgfm)
- Switched from
unittest
topytest
#45 (@frgfm) and split test files by module
Documentation
- Updated README badges #34, illustration #39 and usage instructions #41 (@frgfm)
- Added instructions to run all CI checks locally in CONTRIBUTING #34, #45 (@frgfm)
- Updated project hierarchy description in CONTRIBUTING #43 (@frgfm)
- Added minimal code snippet in documentation #41 (@frgfm)
Others
Automatic target layer resolution and support of IS-CAM
This release adds an implementation of IS-CAM and greatly improves interface.
Note: torchcam 0.1.2 requires PyTorch 1.1 or newer.
Highlights
CAMs
Implementation of CAM extractor
New
Improvements
Fixes
Test
Verifications of the package well-being before release
New
Improvements
- Removed pretrained model loading in unittests #25 (@frgfm)
- Switched all models to eval, removed gradient when not required, and changed to simpler models #33 (@frgfm)
Documentation
Online resources for potential users
New
Fixes
Others
Other tools and implementations
New
- Added annotation typing to the codebase & mypy verification CI job #19 (@frgfm)
- Added package publishing verification jobs #12 (@frgfm)
Improvements
Fixes
Support of SmoothGradCAM++, Score-CAM and SS-CAM
This release adds implementations of SmoothGradCAM++, Score-CAM and SS-CAM.
Note: torchcam 0.1.1 requires PyTorch 1.1 or newer.
brought to you by @frgfm
Highlights
CAMs
Implementation of CAM extractor
New
- Add a SmoothGradCAM++ implementation (#4)
- Add a Score-CAM implementation (#5)
- Add a SS-CAM implementation (#11).
Improvements
- Refactor CAM extractor for better code reusability (#6)
Test
Verifications of the package well-being before release
New
Documentation
Online resources for potential users
Improvements
- Add detailed explanation of CAM computation (#8, #11)
- Add websearch referencing of documentation (#7)
Others
Other tools and implementations
- Fixed conda upload job (#3)
Class activation maps for CNN in PyTorch
This release adds implementations of CAM, GradCAM and GradCAM++.
Note: torchcam 0.1.0 requires PyTorch 1.1 or newer.
brought to you by @frgfm
Highlights
GradCAM
Implementation of gradient-based CAM extractor
New
Test
Verifications of the package well-being before release
New
Documentation
Online resources for potential users
New
- Add sphinx automatic documentation build for existing features (#1, #2)
- Add contribution guidelines (#1)
- Add installation, usage, and benchmark in readme (#1, #2)
Others
Other tools and implementations
- Add ̀overlay_mask` to easily overlay mask on images (#1).