[Research][Testing] Real Market Fat Tails Validation #542
Description
Summary
Validate exit strategy conclusions using real historical data with known crash periods and fat-tail events.
Background
Current analysis uses synthetic normally-distributed data which underestimates extreme events:
Synthetic Data: Uses normally distributed random data. Real markets have "fat tails" (extreme events) that occur more frequently than this model predicts.
The "Balanced" strategy recommendation may not hold during market crashes or flash crashes.
Objective
Create real_data_fat_tails.py to:
- Test strategies on real historical data including crash periods
- Validate that conclusions hold with actual market fat tails
- Identify regime-specific strategy modifications
Historical Periods to Test
| Period | Event | Characteristics |
|---|---|---|
| Mar 2020 | COVID Crash | -34% in 23 days, high VIX |
| Oct-Dec 2018 | Q4 Selloff | -20% correction |
| Feb 2018 | Volmageddon | Flash crash, VIX spike |
| Aug 2015 | China Fears | Flash crash |
| 2022 Full Year | Bear Market | Sustained downtrend |
| 2023-2024 | Bull Rally | Strong uptrend |
Key Questions
- Does 5% stop loss get triggered too often in crashes?
- Does 8% take profit get hit less often in high-vol?
- Do conclusions change during extreme volatility?
- Is "Balanced" still optimal during bear markets?
Metrics to Capture
- Performance during crash periods vs normal periods
- Stop-out rate in high VIX environments
- Comparison of synthetic vs real win rates
- Kurtosis/skewness of actual returns
- Maximum single-day adverse move vs stop loss level
Test Matrix
| Strategy | Normal (2024) | Crash (Mar 2020) | Bear (2022) |
|---|---|---|---|
| Conservative | ? | ? | ? |
| Balanced | ? | ? | ? |
| Aggressive | ? | ? | ? |
Acceptance Criteria
- Script created at
scripts/research/exit_strategy_analysis/real_data_fat_tails.py - Test on minimum 3 crash/high-vol periods
- Compare synthetic vs real data statistics
- Document if strategy ranking changes during extremes
- Provide regime-specific exit recommendations
Data Sources
- Use cached historical data (SQLite cache)
- Symbols: SPY, QQQ, AAPL (most liquid, good data quality)
Related
scripts/research/exit_strategy_analysis/README.md(Tool Limitations - Synthetic Data)- Issue [Research] Path-dependent backtesting engine for simulation fidelity #528 (Path-dependent simulation)
docs/08_research/README.md(Wick Risk gap)