Merge pull request #3 from AryanBV/enhanced-features

Integrate weather prediction feature into dashboard

README.md

@@ -2,23 +2,32 @@
 
 ## Overview
 
-This Weather Monitoring System is a Python-based application that fetches real-time weather data for multiple cities, processes and stores this data, generates alerts for extreme weather conditions, and creates visualizations of weather trends.
+This Weather Monitoring System is a Python-based application that fetches real-time weather data for multiple Indian metro cities, processes and stores this data, generates alerts for extreme weather conditions, and creates visualizations of weather trends.
 
-## Features
+## Core Features
 
-- Fetches weather data from OpenWeatherMap API(temperature data is received in Celsius)
+- Fetches weather data from OpenWeatherMap API for Delhi, Mumbai, Chennai, Bangalore, Kolkata, and Hyderabad
 - Processes and stores weather data in MongoDB
-- Generates daily weather summaries
-- Alerts for high and low temperature thresholds
-- Visualizes temperature trends and weather conditions
-- Logs system activities and errors
+- Generates daily weather summaries including:
+  - Average temperature
+  - Maximum temperature
+  - Minimum temperature
+  - Dominant weather condition
+- User-configurable alerts for temperature thresholds
+- Visualizes daily weather summaries, historical trends, and triggered alerts
 
-## New Features 
+## Additional Features
 
-- Extended weather parameters: Now includes humidity and wind speed in addition to temperature
+- Extended weather parameters: Includes humidity and wind speed in addition to temperature
 - 5-day weather forecast retrieval and processing
-- Enhanced daily summaries with additional weather parameters
-- Forecast summaries based on predicted conditions
+- Web dashboard for real-time weather visualization
+- Machine learning-based weather prediction for the next 24 hours
+
+    ### Web Dashboard
+    The web dashboard provides a user-friendly interface to view current weather data, forecasts, and predictions for all monitored cities.
+
+    ### Machine Learning Weather Prediction
+    Trained machine learning models for 24-hour weather prediction are stored in the `models/` directory.
 
 ### Weather Monitoring System
 
@@ -74,63 +83,82 @@ The dashboard displays the latest weather data for all monitored cities in an ea
 
 5. Configure the application:
    - Open `config/config.yaml`
-   - Update the list of cities if needed
    - Adjust the MongoDB configuration if necessary
    - Modify alert thresholds and update intervals as desired
 
 ## Usage
 
-To run the Weather Monitoring System:
+To run the core Weather Monitoring System:
 
 ```
 python main.py
 ```
 
+To view the real-time weather dashboard (additional feature):
+
+```
+python app.py
+```
+
+Then open a web browser and navigate to `http://localhost:5000`
+
 For a short demonstration run:
 
 ```
 python run_demo.py
 ```
 
-The application will start fetching weather data at regular intervals, process it, store it in the database, check for alerts, and generate visualizations.
-
 ## Project Structure
 
 ```
 weather_monitoring_system/
+├── config/
+│   └── config.yaml
+├── logs/
+├── models/
 ├── src/
+│   ├── alerts/
+│   │   ├── __init__.py
+│   │   └── alert_manager.py
 │   ├── api/
+│   │   ├── __init__.py
 │   │   └── weather_api.py
 │   ├── data_processing/
-│   │   ├── data_processor.py
-│   │   └── aggregator.py
+│   │   ├── __init__.py
+│   │   ├── aggregator.py
+│   │   └── data_processor.py
 │   ├── database/
+│   │   ├── __init__.py
 │   │   └── db_handler.py
-│   ├── alerts/
-│   │   └── alert_manager.py
-│   ├── visualization/
-│   │   └── visualizer.py
-│   └── utils/
-│       ├── logger.py
-│       └── config_loader.py
+│   ├── ml/
+│   │   └── weather_predictor.py
+│   ├── utils/
+│   │   ├── __init__.py
+│   │   ├── config_loader.py
+│   │   └── logger.py
+│   └── visualization/
+│       ├── __init__.py
+│       └── visualizer.py
+├── templates/
+│   └── dashboard.html
 ├── tests/
-│   └── test_system.py
-├── config/
-│   └── config.yaml
-├── logs/
-│   └── ...
+│   ├── test_aggregator.py
+│   ├── test_alert_manager.py
+│   ├── test_data_processor.py
+│   ├── test_db_handler.py
+│   └── test_weather_api.py
 ├── visualizations/
-│   └── ...
-├── requirements.txt
+├── app.py
 ├── main.py
+├── README.md
+├── requirements.txt
 ├── run_demo.py
-└── README.md
+└── test_system.py
 ```
 
 ## Configuration
 
 You can modify the following in `config/config.yaml`:
-- List of cities to monitor
 - Database settings
 - Alert thresholds (user-configurable)
 - Data processing and visualization update intervals
@@ -145,28 +173,41 @@ Logs are stored in the `logs/` directory. Check these for application activity a
 
 Weather trend visualizations are saved in the `visualizations/` directory.
 
+## Testing
 
-## Test Cases
-
-The `test_system.py` script covers the following test cases:
-
-1. System Setup: Verifies successful connection to the OpenWeatherMap API.
-2. Data Retrieval: Simulates API calls and checks correct data parsing.
-3. Daily Weather Summary: Verifies correct calculation of daily summaries.
-4. Alerting Thresholds: Tests alert triggering based on configured thresholds.
-
-To run the tests:
+To run the system tests:
 
 ```
 python test_system.py
 ```
 
+
+Individual test files for different components can be found in the `tests/` directory.
+
+## Test Cases
+
+1. System Setup:
+   - Verifies system starts successfully and connects to the OpenWeatherMap API using a valid API key.
+2. Data Retrieval:
+   - Simulates API calls at configurable intervals.
+   - Ensures the system retrieves weather data for the specified locations and parses the response correctly.
+3. Temperature Conversion:
+   - Tests conversion of temperature values from Kelvin to Celsius.
+4. Daily Weather Summary:
+   - Simulates a sequence of weather updates for several days.
+   - Verifies that daily summaries are calculated correctly, including average, maximum, minimum temperatures, and dominant weather condition.
+5. Alerting Thresholds:
+   - Defines and configures user thresholds for temperature conditions.
+   - Simulates weather data exceeding or breaching the thresholds.
+   - Verifies that alerts are triggered only when a threshold is violated.
+
+
 ## Troubleshooting
 
 - If you encounter any issues with API key authentication, ensure that you've correctly set the `OPENWEATHERMAP_API_KEY` environment variable.
 - For database connection issues, check your MongoDB service is running and the connection details in `config.yaml` are correct.
 - If visualizations aren't generating, ensure you have write permissions in the `visualizations/` directory.
 
-## Note 
+## Note
 
-This project is an assignment submission. The implementation focuses on demonstrating understanding of system design, API integration, data processing, and visualization techniques. 
+This project is an assignment submission. The implementation focuses on demonstrating understanding of real-time data processing, API integration, data aggregation, alerting systems, and visualization techniques in a weather monitoring context. Additional features showcase further capabilities beyond the core requirements.

app.py

@@ -2,6 +2,8 @@
 from src.data_processing.data_processor import DataProcessor
 from src.database.db_handler import DBHandler
 from src.utils.config_loader import load_config
+from src.ml.weather_predictor import WeatherPredictor
+from datetime import datetime, timedelta
 import os
 
 app = Flask(__name__)
@@ -14,12 +16,30 @@
 def dashboard():
     cities = db_handler.get_cities()
     latest_data = {}
+    predictions = {}
     for city in cities:
         city_data = db_handler.get_recent_weather_data(city, limit=1)
         if city_data:
             latest_data[city] = city_data[0]
+            predictor = WeatherPredictor(city)
+            try:
+                predictor.load_model()
+            except FileNotFoundError:
+                historical_data = db_handler.get_historical_weather_data(city)
+                predictor.train_model(historical_data)
+
+            next_day = datetime.now() + timedelta(days=1)
+            prediction = predictor.predict(
+                next_day.hour,
+                next_day.weekday(),
+                next_day.month,
+                latest_data[city]['humidity'],
+                latest_data[city]['wind_speed'],
+                latest_data[city]['weather_condition']
+            )
+            predictions[city] = round(prediction, 1)
 
-    return render_template('dashboard.html', latest_data=latest_data)
+    return render_template('dashboard.html', latest_data=latest_data, predictions=predictions)
 
 if __name__ == '__main__':
     app.run(debug=True)

src/database/db_handler.py

@@ -58,6 +58,13 @@ def get_data_for_alerts(self, city, hours=24):
             'city': city,
             'timestamp': {'$gte': start_time}
         }).sort('timestamp', 1))
+
+    def get_historical_weather_data(self, city, days=30):
+        start_date = datetime.utcnow() - timedelta(days=days)
+        return list(self.weather_collection.find({
+            'city': city,
+            'timestamp': {'$gte': start_date}
+        }).sort('timestamp', 1))
 
     def close(self):
         self.client.close()

src/ml/weather_predictor.py

@@ -0,0 +1,49 @@
+import pandas as pd
+from sklearn.model_selection import train_test_split
+from sklearn.ensemble import RandomForestRegressor
+from sklearn.preprocessing import LabelEncoder
+import joblib
+import os
+
+class WeatherPredictor:
+    def __init__(self, city):
+        self.city = city
+        self.model = None
+        self.le = LabelEncoder()
+        self.model_path = f'models/{city}_weather_model.joblib'
+
+    def prepare_data(self, data):
+        df = pd.DataFrame(data)
+        df['timestamp'] = pd.to_datetime(df['timestamp'])
+        df['hour'] = df['timestamp'].dt.hour
+        df['day_of_week'] = df['timestamp'].dt.dayofweek
+        df['month'] = df['timestamp'].dt.month
+        df['weather_condition'] = self.le.fit_transform(df['weather_condition'])
+        return df
+
+    def train_model(self, data):
+        df = self.prepare_data(data)
+        X = df[['hour', 'day_of_week', 'month', 'humidity', 'wind_speed', 'weather_condition']]
+        y = df['temperature']
+        X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
+        self.model = RandomForestRegressor(n_estimators=100, random_state=42)
+        self.model.fit(X_train, y_train)
+        self.save_model()
+
+    def predict(self, hour, day_of_week, month, humidity, wind_speed, weather_condition):
+        if self.model is None:
+            self.load_model()
+        weather_condition_encoded = self.le.transform([weather_condition])[0]
+        prediction = self.model.predict([[hour, day_of_week, month, humidity, wind_speed, weather_condition_encoded]])
+        return prediction[0]
+
+    def save_model(self):
+        if not os.path.exists('models'):
+            os.makedirs('models')
+        joblib.dump(self.model, self.model_path)
+
+    def load_model(self):
+        if os.path.exists(self.model_path):
+            self.model = joblib.load(self.model_path)
+        else:
+            raise FileNotFoundError(f"No trained model found for {self.city}")

templates/dashboard.html

@@ -12,6 +12,7 @@
         .city-name { color: #0066cc; margin-top: 0; }
         .weather-icon { font-size: 48px; margin: 10px 0; }
         .weather-details { margin: 5px 0; }
+        .prediction { color: #ff6600; font-weight: bold; margin-top: 15px; border-top: 1px solid #eee; padding-top: 10px; }
     </style>
 </head>
 <body>
@@ -33,6 +34,7 @@ <h2 class="city-name">{{ city }}</h2>
             <p class="weather-details"><strong>Wind Speed:</strong> {{ "%.1f"|format(data.wind_speed) }} m/s</p>
             <p class="weather-details"><strong>Condition:</strong> {{ data.weather_condition }}</p>
             <p class="weather-details"><strong>Last Updated:</strong> {{ data.timestamp.strftime('%Y-%m-%d %H:%M:%S') }}</p>
+            <p class="prediction"><strong>Predicted Temperature (24h):</strong> {{ "%.1f"|format(predictions[city]) }}°C</p>
         </div>
     {% endfor %}
     </div>