Assignment2_part2_2_(iii) copy.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "49a0b40220d3b5b2",
   "metadata": {},
   "source": [
    "# (iii)\n",
    "\n",
    "## (a) Effect of Learning Rate and Momentum on Training Performance\n",
    "\n",
    "### 1. Impact of Learning Rate\n",
    "\n",
    "#### For SGD:\n",
    "- Lower learning rate (0.1) consistently produced better results across different numbers of hidden neurons\n",
    "- Higher learning rates (0.5, 0.9) generally led to poorer performance\n",
    "- The best model used a learning rate of 0.1, achieving:\n",
    "  * Test MSE: 0.002905\n",
    "  * Test R²: 0.958846\n",
    "\n",
    "#### For Adagrad:\n",
    "- The adaptive learning rate approach failed to match or exceed the performance of fixed learning rates\n",
    "- Despite automatically adjusting the learning rate, the model showed poor convergence:\n",
    "  * Test MSE: 0.3095\n",
    "  * Test R²: -3.3841\n",
    "\n",
    "### 2. Impact of Momentum\n",
    "\n",
    "- High momentum (0.9) combined with low learning rate (0.1) produced the best results\n",
    "- This combination likely helped the model:\n",
    "  * Overcome local minima\n",
    "  * Maintain stable convergence\n",
    "  * Achieve better generalization across all datasets\n",
    "\n",
    "### 3. Interaction Between Learning Rate and Momentum\n",
    "\n",
    "- The relationship between learning rate and momentum appears to be crucial:\n",
    "  * Best performance: Low learning rate (0.1) + High momentum (0.9)\n",
    "  * Poor performance: High learning rate + High momentum\n",
    "- This suggests that the balance between these parameters is critical for optimal training\n",
    "\n",
    "### 4. Stability and Convergence\n",
    "\n",
    "- SGD with appropriate learning rate and momentum showed stable convergence\n",
    "- The combination of parameters affected not just final performance, but also:\n",
    "  * Training stability\n",
    "  * Convergence speed\n",
    "  * Generalization ability"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce69c111efbc7d6a",
   "metadata": {},
   "source": [
    "## (b) Comparison of Performance between Best Networks from Two Training Methods\n",
    "\n",
    "### Direct Comparison of Results\n",
    "\n",
    "| Metric | SGD (Best Model) | Adagrad |\n",
    "|--------|------------------|---------|\n",
    "| Test MSE | 0.002905 | 0.309470 |\n",
    "| Test R² | 0.958846 | -3.384062 |\n",
    "| All Data MSE | 0.002996 | 0.318307 |\n",
    "| All Data R² | 0.944083 | -4.940097 |\n",
    "\n",
    "### Analysis of Performance Differences\n",
    "\n",
    "1. **Mean Squared Error (MSE)**\n",
    "   - SGD significantly outperformed Adagrad in terms of MSE\n",
    "   - The difference is substantial:\n",
    "     * Test data: SGD's MSE is ~106 times lower than Adagrad's\n",
    "     * All data: SGD's MSE is ~106 times lower than Adagrad's\n",
    "   - This indicates that SGD predictions are much closer to actual values\n",
    "\n",
    "2. **R-squared (R²)**\n",
    "   - The contrast in R² values is even more striking:\n",
    "     * SGD achieved positive R² values close to 1 (ideal)\n",
    "     * Adagrad produced negative R² values, indicating worse performance than a horizontal line\n",
    "   - This suggests:\n",
    "     * SGD: Successfully captured the underlying patterns in the data\n",
    "     * Adagrad: Failed to learn meaningful relationships between inputs and outputs\n",
    "\n",
    "3. **Consistency Across Datasets**\n",
    "   - SGD showed consistent performance between test data and all data\n",
    "   - Adagrad also showed consistency, but consistently poor results\n",
    "   - This suggests that the performance differences are systematic rather than random\n",
    "\n",
    "### Conclusion\n",
    "\n",
    "The performance comparison clearly favors the SGD model with fixed learning rate and momentum over the Adagrad model with adaptive learning rate."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "129feee00bf8e345",
   "metadata": {},
   "source": [
    "## (c) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "59adb41483e59ec3",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-10-02T07:56:38.038053Z",
     "start_time": "2024-10-02T07:56:35.904340Z"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "from tensorflow.keras.models import load_model\n",
    "from sklearn.metrics import mean_squared_error, r2_score\n",
    "from IPython.display import display\n",
    "\n",
    "# Setting the Seaborn style\n",
    "sns.set_style(\"whitegrid\")\n",
    "\n",
    "# Load data\n",
    "def load_data():\n",
    "    file_path = 'Heat_Influx_insulation_east_south_north.csv'\n",
    "    data = pd.read_csv(file_path)\n",
    "    \n",
    "    features = ['Insulation', 'East', 'South', 'North']\n",
    "    target = 'HeatFlux'\n",
    "    \n",
    "    from sklearn.preprocessing import MinMaxScaler\n",
    "    scaler = MinMaxScaler()\n",
    "    data_normalized = pd.DataFrame(\n",
    "        scaler.fit_transform(data[features + [target]]),\n",
    "        columns=features + [target]\n",
    "    )\n",
    "    \n",
    "    X = data_normalized[features].values\n",
    "    y = data_normalized[target].values\n",
    "    \n",
    "    return X, y\n",
    "\n",
    "# Load models and make predictions\n",
    "def get_predictions():\n",
    "    X, y_true = load_data()\n",
    "    \n",
    "    sgd_model = load_model('best_heat_flux_model.keras')\n",
    "    adagrad_model = load_model('best_heat_flux_model_adagrad.keras')\n",
    "    \n",
    "    y_pred_sgd = sgd_model.predict(X).flatten()\n",
    "    y_pred_adagrad = adagrad_model.predict(X).flatten()\n",
    "    \n",
    "    return y_true, y_pred_sgd, y_pred_adagrad\n",
    "\n",
    "# Computational performance metrics\n",
    "def calculate_metrics(y_true, y_pred):\n",
    "    mse = mean_squared_error(y_true, y_pred)\n",
    "    r2 = r2_score(y_true, y_pred)\n",
    "    return mse, r2\n",
    "\n",
    "# Create comparison charts\n",
    "def plot_comparison():\n",
    "    y_true, y_pred_sgd, y_pred_adagrad = get_predictions()\n",
    "    \n",
    "    # Computational performance metrics\n",
    "    mse_sgd, r2_sgd = calculate_metrics(y_true, y_pred_sgd)\n",
    "    mse_adagrad, r2_adagrad = calculate_metrics(y_true, y_pred_adagrad)\n",
    "    \n",
    "    # Create charts\n",
    "    fig, axs = plt.subplots(2, 2, figsize=(20, 20))\n",
    "    \n",
    "    # Scatterplot of predicted vs. actual values\n",
    "    axs[0, 0].scatter(y_true, y_pred_sgd, alpha=0.5, label='SGD', color='blue')\n",
    "    axs[0, 0].scatter(y_true, y_pred_adagrad, alpha=0.5, label='Adagrad', color='red')\n",
    "    axs[0, 0].plot([y_true.min(), y_true.max()], [y_true.min(), y_true.max()], 'k--', lw=2)\n",
    "    axs[0, 0].set_xlabel('Actual Values')\n",
    "    axs[0, 0].set_ylabel('Predicted Values')\n",
    "    axs[0, 0].set_title('Predictions vs Actual Values')\n",
    "    axs[0, 0].legend()\n",
    "    \n",
    "    # Error maps\n",
    "    error_sgd = y_pred_sgd - y_true\n",
    "    error_adagrad = y_pred_adagrad - y_true\n",
    "    axs[0, 1].plot(range(len(y_true)), error_sgd, label='SGD Error', color='blue', alpha=0.7)\n",
    "    axs[0, 1].plot(range(len(y_true)), error_adagrad, label='Adagrad Error', color='red', alpha=0.7)\n",
    "    axs[0, 1].axhline(y=0, color='k', linestyle='--')\n",
    "    axs[0, 1].set_xlabel('Sample Index')\n",
    "    axs[0, 1].set_ylabel('Prediction Error')\n",
    "    axs[0, 1].set_title('Error Graph')\n",
    "    axs[0, 1].legend()\n",
    "    \n",
    "    # Residual plots\n",
    "    axs[1, 0].scatter(y_pred_sgd, error_sgd, alpha=0.5, label='SGD', color='blue')\n",
    "    axs[1, 0].scatter(y_pred_adagrad, error_adagrad, alpha=0.5, label='Adagrad', color='red')\n",
    "    axs[1, 0].axhline(y=0, color='k', linestyle='--')\n",
    "    axs[1, 0].set_xlabel('Predicted Values')\n",
    "    axs[1, 0].set_ylabel('Residuals')\n",
    "    axs[1, 0].set_title('Residual Plot')\n",
    "    axs[1, 0].legend()\n",
    "    \n",
    "    # Histogram of predicted vs. actual values\n",
    "    axs[1, 1].hist(y_true, bins=30, alpha=0.5, label='Actual', color='green')\n",
    "    axs[1, 1].hist(y_pred_sgd, bins=30, alpha=0.5, label='SGD', color='blue')\n",
    "    axs[1, 1].hist(y_pred_adagrad, bins=30, alpha=0.5, label='Adagrad', color='red')\n",
    "    axs[1, 1].set_xlabel('Values')\n",
    "    axs[1, 1].set_ylabel('Frequency')\n",
    "    axs[1, 1].set_title('Distribution of Actual and Predicted Values')\n",
    "    axs[1, 1].legend()\n",
    "    \n",
    "    plt.tight_layout()\n",
    "    \n",
    "    # Display performance indicators\n",
    "    performance_text = f\"\"\"\n",
    "    Performance Metrics:\n",
    "    SGD Model:\n",
    "        MSE: {mse_sgd:.6f}\n",
    "        R²: {r2_sgd:.6f}\n",
    "    Adagrad Model:\n",
    "        MSE: {mse_adagrad:.6f}\n",
    "        R²: {r2_adagrad:.6f}\n",
    "    \"\"\"\n",
    "    fig.text(0.5, 0.01, performance_text, ha='center', fontsize=12, bbox=dict(facecolor='white', alpha=0.8))\n",
    "    \n",
    "    return fig\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    # Generate and display charts\n",
    "    fig = plot_comparison()\n",
    "    display(fig)\n",
    "    \n",
    "    # Save the chart\n",
    "    fig.savefig('model_comparison.png')\n",
    "    plt.close(fig)\n",
    "    \n",
    "    print(\"Comparison plot has been generated, displayed, and saved as 'model_comparison.png'\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "744aac6f8214987c",
   "metadata": {},
   "source": [
    "## (d) Recommended Model for the Architect\n",
    "\n",
    "Based on the comprehensive analysis of both the SGD (Stochastic Gradient Descent) and Adagrad models, I strongly recommend the SGD model to the architect for predicting heat influx in home designs. Here's why:\n",
    "\n",
    "1. **Superior Predictive Accuracy**: The SGD model demonstrates significantly better performance across all metrics. With a test MSE of 0.002905 compared to Adagrad's 0.309470, and an R² of 0.958846 versus Adagrad's negative R², the SGD model clearly provides more accurate and reliable predictions.\n",
    "\n",
    "2. **Consistent Performance**: The SGD model shows consistent performance across both test data and the entire dataset. This consistency indicates that the model has good generalization capabilities and is likely to perform well on new, unseen data – a crucial factor for practical applications in architectural design.\n",
    "\n",
    "3. **Visual Confirmation**: The \"Predictions vs Actual Values\" plot clearly shows that the SGD model's predictions (blue dots) closely align with the ideal prediction line, while Adagrad's predictions (red dots) show a scattered, inaccurate pattern. This visual representation reinforces the quantitative metrics.\n",
    "\n",
    "4. **Error Distribution**: The \"Error Graph\" and \"Residual Plot\" both demonstrate that the SGD model's errors are much smaller and more evenly distributed around zero compared to Adagrad's large, systematic errors.\n",
    "\n",
    "5. **Practical Usability**: Given the architect's goal of designing the warmest possible homes for clients, the SGD model's ability to accurately predict heat influx based on insulation and orientation factors is invaluable. It will allow for more informed decision-making in the design process.\n",
    "\n",
    "6. **Reliability for Design Iterations**: The \"Distribution of Actual and Predicted Values\" histogram shows that the SGD model's predictions (blue) closely match the distribution of actual values (green), unlike Adagrad's predictions (red). This suggests that the SGD model will be reliable across various design scenarios the architect might consider."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}