This repository contains the implementation and evaluation for the paper "Consistency-based Abductive Reasoning over Perceptual Errors of Multiple Pre-trained Models in Novel Environments"

Overview

The project implements and compares three main algorithmic approaches for solving constraint satisfaction problems in knowledge representation:

1. Integer Programming (IP) - Exact optimization approach
2. Majority Voting (MV) - Baseline heuristic approach
3. Heuristic Search (HS) - Approximate search-based approach

Main Techniques

Integer Programming (IP)

The IP approach formulates the reasoning problem as an optimization problem with integer constraints. Key features:

• Exact solutions guaranteed
• Higher computational cost
• Handles complex constraint relationships
• Implemented in intprogramming_per_image.py

Heuristic Search (HS)

The heuristic search approach uses approximation algorithms to find good solutions efficiently:

• Faster execution compared to IP
• Trade-off between solution quality and runtime
• Implemented in heuristic_search/heuristic_solution.py
• Includes tie-breaking mechanisms for handling inconsistencies

Majority Voting (MV)

Simple baseline approach that aggregates predictions:

• Fastest execution time
• Serves as performance baseline
• Implemented in majority_vote/ directory

Repository Structure

├── intprogramming_per_image.py    # Main IP implementation
├── heuristic_search/              # Heuristic search algorithms
│   └── heuristic_solution.py      # Core HS implementation
├── majority_vote/                 # Majority voting implementation
├── tie_breaker/                   # Inconsistency resolution methods
├── pyreason/                      # PyReason framework integration
├── runtimes/                      # Performance analysis scripts
│   ├── runtime2.py               # IP vs MV comparison
│   ├── runtime3.py               # IP vs MV vs HS comparison
│   └── runtime4.py               # Advanced runtime analysis
├── comparison_plots_clean_style/  # Generated comparison visualizations
├── comparison_plots_ordered/      # Ordered dataset comparisons
├── notebooks/                     # Jupyter analysis notebooks
├── test/                         # Test datasets and utilities
├── context_handlers.py           # Context processing utilities
├── utils.py                      # Common utility functions
└── paths.py                      # Path configuration

Quick Start

Running Integer Programming Analysis

python intprogramming_per_image.py

This generates output.json with per-image metrics.

Running Heuristic Search

cd heuristic_search
python heuristic_solution.py

Performance Comparison

cd runtimes
python runtime4.py  # For comprehensive runtime analysis

Workflow

1. Run Integer Programming: Execute intprogramming_per_image.py
2. Process Results: Use notebooks in notebooks/ to aggregate results
3. Compare Algorithms: Run runtime comparison scripts in runtimes/
4. Analyze Performance: Review generated plots in comparison directories

Key Features

• Multi-algorithm comparison: Direct performance comparison between IP, MV, and HS
• Inconsistency handling: Tie-breaking mechanisms for resolving conflicts
• Scalability analysis: Runtime performance across different dataset sizes
• Visualization: Automated generation of comparison plots and charts
• Sensitivity analysis: Parameter exploration for epsilon and delta values

Dependencies

• PyReason framework
• pandas, numpy, matplotlib
• Excel file processing (openpyxl)
• Integer programming solvers
• multiprocessing support

Output

The system generates:

• Performance metrics (F1, Accuracy, Runtime)
• Inconsistency analysis
• Comparative visualizations
• Excel reports with detailed statistics
• 3D sensitivity plots for parameter analysis

Configuration

Main configuration parameters:

• epsilon: Confidence threshold parameter
• delta: Algorithm-specific tuning parameter
• Dataset paths in paths.py
• Cloud folder configuration for data access