Feat/prefilter logic v2 335

[ZohaibHassan16]

Description

This PR implements Two-Stage Scoring to fast-fail weak candidate pairs before they reach the computationally expensive multi-factor semantic scoring logic.

Type of Change

   
• ✅ New feature (non-breaking change which adds functionality)
   
• ✅ Performance improvement

Related Issues

   
• Resolves [Sub-Issue 2] Two-Stage Scoring: Fast Prefilter + Full Semantic Score #335
   
• Part of [EPIC] Deduplication v2: Higher Accuracy, Lower Latency, Backward Compatible #333 (Deduplication v2 Epic)

---

Changes Made

1. Prefilter Gates

Added _prefilter_pair to evaluate type mismatch, name length-ratios, and token overlaps instantly.

2. Scoring Orchestration

Updated calculate_similarity to respect the prefilter_enabled configuration and bypass string/property/embedding checks if the prefilter fails.

3. Explainability

Added opt-in score_breakdown to the metadata output for better debugging and transparency.

---

Benchmark Results

V1 Candidate Gen vs. V2 Prefilter

                                                                                                                                                                                                               

Metric	V1 (Legacy)	V2 (Prefilter)	Improvement
Total Suite Runtime	11m 11s	9m 26s	~15% Faster
Hierarchical Clustering	124.3s	103.8s	-20.5s

---

Testing & Quality Assurance

   
• ✅ Tested locally: Benchmarks verified on local environment.
   
• ✅ Package builds: python -m build successful.
   
• ✅ Style Guidelines: Code adheres to project standards.
   
• ✅ Breaking Changes: None. Prefilter defaults to False to preserve legacy behavior.

---

---

Additional Notes

This PR is part of a stacked sequence for the Deduplication v2 Epic. Please note the following for review:

• Current Status: This PR is built upon the changes in PR #334.
• Diff Content: Because the previous PR is not yet merged into main, this PR currently displays the combined changes of both Sub-Issue 1 and Sub-Issue 2.
• Review Strategy: It is highly recommended to review and merge PR [Sub-Issue 1] Candidate Generation v2 to Cut Pair Explosion #334 first. Once that is merged, I will rebase this PR against main so that it only displays the unique changes for Sub-Issue 2 ([Sub-Issue 2] Two-Stage Scoring: Fast Prefilter + Full Semantic Score #335).