NAPS: Normalized Areal Perceptual Similarity Metric

This repository provides the official TensorFlow 2.x implementation for the NAPS (Normalized Areal Perceptual Similarity) metric.

NAPS is a novel evaluation framework that assesses visual explanations by measuring feature-level information preservation while normalizing for saliency area. It aims to provide a more stable and consistent evaluation for saliency-based explanation methods compared to traditional metrics like Drop% or Increase%.

Overview

The NAPS metric is calculated based on the following formula:

$$NAPS = NAPS_+ - NAPS_-$$

where $NAPS_+$ and $NAPS_-$ are defined as:

$$NAPS_+ = \frac{1}{N} \sum_{k=1}^{N} \left( (1 - \text{Area}^k) - \frac{1}{|L|} \sum_{l \in L} \text{NPS}_{l+}^{k} \right)$$

$$NAPS_- = \frac{1}{N} \sum_{k=1}^{N} \left( \text{Area}^k - \frac{1}{|L|} \sum_{l \in L} \text{NPS}_{l-}^{k} \right)$$

• $\text{Area}^k$: The area of the saliency map $S^k$ for the $k$-th image.
• $L$: The set of layers from a backbone network used for feature extraction.
• $\text{NPS}_{l+}^{k}$: Measures the feature-level similarity difference for image $k$ at layer $l$ when the salient region, defined by $S^k$, is preserved and the non-salient region is replaced by a blurred version.
• $\text{NPS}_{l-}^{k}$: Measures the feature-level similarity difference for image $k$ at layer $l$ when the non-salient region, defined by $1-S^k$, is preserved and the salient region is replaced by a blurred version.

The core idea is to compare feature representations of an original image $X_{org}$, a blurred version of the image $X_{blur}$, and a masked version $X_{mask}$ where parts are taken from $X_{org}$ and other parts from $X_{blur}$ based on the saliency map $S$. Features are extracted from multiple layers of a pre-trained CNN, and their differences are aggregated. Gradient masking is used to focus on class-relevant features.

Features

• Evaluates saliency maps at the feature level.
• Normalizes for saliency map area.
• Uses gradient masking to consider class-discriminative features.
• Designed to be robust against superficial manipulations that can fool traditional metrics.
• Assigns near-zero scores to random saliency maps.

Requirements

• Python 3.x
• TensorFlow 2.x
• NumPy
• OpenCV-Python (for Gaussian Blur, consider replacing with TensorFlow native operations for better graph compatibility if needed)

Usage

The core logic is implemented in the NAPS_Metric class in metric.py.

import tensorflow as tf
import numpy as np
# from naps_metric import NAPSMetric # Assuming the class is in this file

# --- Example Setup (replace with your actual model and data) ---
from tensorflow.keras.applications import ResNet50V2
from tensorflow.keras.applications.resnet_v2 import preprocess_input as preprocess_resnet_v2

# 1. Load your backbone model and define preprocessing function and layer names
backbone_model = ResNet50V2(weights='imagenet', include_top=True)
preprocess_fn = preprocess_resnet_v2

# Example layer names for ResNet50V2 (These are common choices, verify with your model summary if needed)
# Typically, these are the outputs of residual blocks or activation layers before downsampling.
layer_names = [
    'conv2_block3_1_relu',         # Initial convolution
    'conv3_block4_1_relu',   # Output of the 2nd stage
    'conv4_block6_1_relu',   # Output of the 3rd stage
    'conv5_block3_out',   # Output of the 4th stage
    # Alternatively, you might pick layers like 'conv5_block3_out' if available and suitable
]
# You can print `backbone_model.summary()` to see all available layer names.

# 2. Initialize the NAPS_Metric calculator
naps_calculator = NAPS_Metric(
    backbone_model=backbone_model,
    layer_names=layer_names,
    preprocess_function=preprocess_fn,
    blur_sigma=16 # Default sigma for Gaussian Blur
)

# 3. Prepare your data
# Ensure images are in the format expected by your preprocess_function (e.g., 0-255 or 0-1, RGB)
# ResNetV2 preprocess_input typically expects pixels in the range [-1, 1].
# Saliency masks should be normalized (0-1 range) and broadcastable to image shape if single channel.
# Example: batch_size = 4
# Dummy images (0-255) - preprocess_fn will handle the scaling to [-1, 1]
image_batch = np.random.randint(0, 256, size=(4, 224, 224, 3), dtype=np.uint8).astype(np.float32)
saliency_mask_batch = np.random.rand(4, 224, 224, 1).astype(np.float32)

# 4. Calculate the final NAPS score for the batch
final_naps_score = naps_calculator(image_batch, saliency_mask_batch) # Uses the __call__ method

print(f"Final NAPS Score for the batch (ResNet50V2): {final_naps_score.numpy()}")