A powerful real-time news monitoring and analysis system that uses AI to track, cluster, and alert on breaking news events. Built with Python, Flask, and powered by local LLMs via Ollama.
- Real-time Google News RSS feed monitoring
- Custom search topics with 24-hour lookback
- Automatic story deduplication and clustering (IN PROGRESS)
- Local LLM integration via Ollama (privacy-first)
- Intelligent event extraction and categorization
- Context-aware story summarization
- Status detection (ongoing, announced, escalating)
- Multi-story event clustering with similarity scoring
- Freshness, specificity, and impact scoring
- Cross-source corroboration validation (TODO)
- Alert cooldown system to prevent spam (TODO)
- Responsive Flask web application
- Real-time search progress with live updates
- Interactive dashboards for event monitoring (TODO)
- Configurable scoring thresholds (TODO)
- High-priority event highlighting
- Timeline tracking with persistent storage (TODO)
- Status-based alert triggers
- Python 3.8+
- Ollama installed and running
- Basic familiarity with command line
-
Clone the repository
git clone https://github.com/cas1m1r/Scout.git cd news-monitor -
Install dependencies
pip install -r requirements.txt
-
Configure environment
cp .env.template .env # Edit .env with your settings -
Start Ollama and pull a model
ollama serve ollama pull gemma2:2b # or your preferred model -
Run the application
python flask_news_frontend.py
-
Open your browser
http://localhost:5000
Scout/
├── 📄 flask_news_frontend.py # Main Flask web application
├── 🧠 llama_utils.py # Ollama/LLM integration utilities
├── 📰 main.py # Core news processing logic
├── 📊 tracker.py # Event clustering and tracking system
├── ⚙️ template_setup.py # Template installation script
├── 📋 requirements.txt # Python dependencies
├── 🌍 .env.template # Environment configuration template
├── 📝 prompts/
│ └── news_prepper.txt # LLM prompt for news analysis
├── 🎨 templates/ # Flask HTML templates
│ ├── base.html # Main layout template
│ ├── index.html # Homepage with search
│ ├── results.html # Search results display
│ ├── historical.html # Historical data browser
│ ├── event_store.html # Event tracking dashboard
│ └── error.html # Error pages
├── 💾 logs/ # Application logs
└── 📊 event_store.jsonl # Persistent event tracking data
- Enter a search topic:
"climate protests" - Watch real-time progress as stories are analyzed
- Review clustered events with AI-generated summaries
- Monitor alerts for high-priority developments
# Customize scoring parameters
tracker = EventTracker(
cooldown_minutes=30, # Time between alerts for same event
min_score=60 # Minimum score for high-priority alerts
)# Get system status
curl http://localhost:5000/api/status
# Export topic data
curl http://localhost:5000/api/export/climate_protests
# Get cluster details
curl http://localhost:5000/api/cluster/ice-enforcement-chicago# Required: Ollama host (without http://)
URL=localhost
# Optional: ComfyUI integration (for future image analysis)
COMFY=localhost
# Flask settings
FLASK_ENV=production
FLASK_DEBUG=False
# Model configuration
DEFAULT_MODEL=gemma2:2bThe system scores events on multiple dimensions:
- Freshness (0-40): How recent the story is
- Specificity (0-30): Concrete locations, actors, actions
- Corroboration (0-20): Multiple credible sources
- Impact (0-10): Public safety implications
Edit tracker.py to adjust scoring weights for your use case.
Compatible with any Ollama model. Recommended options:
| Model | Size | Speed | Quality | Best For |
|---|---|---|---|---|
gemma2:2b |
1.6GB | ⚡ Fast | Good | Development/Testing |
llama3.1:8b |
4.7GB | 🔄 Medium | Better | Production |
qwen2.5:14b |
8.7GB | 🐌 Slow | Best | High Accuracy |
- Browse all previously monitored topics
- Export complete datasets as JSON
- Timeline analysis of event evolution
- Cross-topic pattern recognition
- WebSocket-based live updates
- Configurable refresh intervals
- Background processing with progress tracking
- Multi-topic concurrent monitoring
- RESTful API endpoints
- JSON export functionality
- Webhook support (coming soon)
- Database integration options
The web interface provides several views:
- Start new topic searches
- Real-time progress tracking
- Quick access to recent searches
- All tracked events with scores
- Alert history and patterns
- Cluster analysis and trends
- Previously searched topics
- Data export capabilities
- Timeline views
- Local Processing: All AI analysis runs on your hardware
- No Data Sharing: Stories and analysis stay on your system
- Configurable Sources: Control which news feeds to monitor
- Rate Limiting: Respects news source rate limits
"URL environment variable not set"
# Check your .env file
cat .env
# Should contain: URL=localhost"No news articles found"
- Try broader search terms
- Check internet connectivity
- Verify Google News access
LLM Processing Errors
# Check if Ollama is running
curl http://localhost:11434/api/tags
# Pull the model if needed
ollama pull gemma2:2bTemplate Errors
# Regenerate templates
python template_setup.py- Use smaller models for faster processing
- Adjust
cooldown_minutesfor your monitoring needs - Clean up old data files periodically
- Monitor disk space usage
We welcome contributions! Here are some areas where help is needed:
- Additional news source integrations (Reddit, Twitter, etc.)
- Advanced clustering algorithms (semantic similarity)
- Email/SMS alerting system
- Mobile-responsive improvements
- Database backend
- Docker containerization
Please include:
- Operating system and Python version
- Ollama version and model used
- Complete error traceback
- Steps to reproduce
- Use case examples
- API documentation
- Deployment guides
This project is released under the MIT License. See LICENSE for details.
- Ollama for local LLM infrastructure
- Flask for the web framework
- feedparser for RSS processing
- The open source community for inspiration and tools
⭐ If you find this project useful, please star it on GitHub!
Built with ❤️ for journalists, researchers, and anyone who needs to stay informed about rapidly evolving news events.