sagemaker/sagemaker-model-monitor/black-belt-part2/model-quality-step-B.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# モデル品質モニタリングのステップB"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "このノートブックを実行する時のヒント：  \n",
    "- このノートブックは大容量のRawデータを読み込むため、<span style=\"color: orange; \">メモリー8GB以上のインスタンス</span>で実行してください\n",
    "- KernelはPython3（Data Science）で動作確認をしています。\n",
    "- デフォルトではSageMakerのデフォルトBucketを利用します。必要に応じて変更することも可能です。\n",
    "- 実際に動かさなくても出力を確認できるようにセルのアウトプットを残しています。きれいな状態から実行したい場合は、右クリックメニューから \"Clear All Outputs\"を選択して出力をクリアしてから始めてください。\n",
    "- 作成されたスケジュールはSageMaker Studioの`SageMaker resource` （左側ペインの一番下）のEndpointメニューからも確認可能"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "複数のノートブックで共通で使用する変数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# エンドポイント名を指定する\n",
    "endpoint_name = 'nyctaxi-xgboost-endpoint'\n",
    "\n",
    "# エンドポイントConfigの名前を指定する\n",
    "endpoint_config_name = f'{endpoint_name}-config'\n",
    "\n",
    "# データ品質のモニタリングスケジュールの名前を指定する\n",
    "model_quality_monitoring_schedule = f'{endpoint_name}-model-quality-schedule'\n",
    "\n",
    "# SageMaker default bucketをModel Monitorのバケットとして使用\n",
    "# それ以外のバケットを使用している場合はここで指定する\n",
    "import sagemaker\n",
    "bucket = sagemaker.Session().default_bucket()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "モニタリング結果を保管するための、ベースラインやレポートのS3上のPrefixを設定します"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## B1（オプションA） 推論を実行してGround TruthをS3にアップロードする\n",
    "推論の実行後に次の周期のモニタリングジョブを待つ必要があります"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ベースラインの出力先Prefixを設定する\n",
    "baseline_prefix = 'model_monitor/model_quality_baseline'\n",
    "\n",
    "# 時系列での可視化のために、複数のレポートに共通するPrefixを設定する\n",
    "report_prefix = 'model_monitor/model_quality_monitoring_report'\n",
    "\n",
    "# Ground Truthをアップロードする先のPrefixを指定します\n",
    "ground_truth_prefix = 'model_monitor/model_quality_ground_truth'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Inference IDを指定して推論を実行する"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 推論を実行する日付を指定する\n",
    "prediction_target_date = '2021-09-15'\n",
    "\n",
    "# データのサンプリングレートを指定する（モデル作成時の設定に合わせる）\n",
    "sampling_rate = 20\n",
    "\n",
    "# 推論結果を保存するディレクトリ名を指定する\n",
    "result_dir = 'prediction_results_model_quality'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import boto3\n",
    "import pandas as pd\n",
    "import time\n",
    "from datetime import datetime\n",
    "import model_utils"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_data_for_pred(target, sampling_rate):\n",
    "    previous_year, previous_month = model_utils.get_previous_year_month(target.year, target.month)\n",
    "    df_previous_month = model_utils.get_raw_data(previous_year, previous_month, sampling_rate)\n",
    "    df_current_month = model_utils.get_raw_data(target.year, target.month, sampling_rate)\n",
    "    df_data = pd.concat([df_previous_month, df_current_month])\n",
    "    del df_previous_month\n",
    "    del df_current_month\n",
    "\n",
    "    # Extract features\n",
    "    df_features = model_utils.extract_features(df_data)\n",
    "    df_features = model_utils.filter_current_month(df_features, target.year, target.month)\n",
    "    \n",
    "    return df_features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading data for 2021-09\n",
      "Predicting 2021-09-15 00:00:00 nyctaxi-xgboost-endpoint\n",
      "Prediciton completed. Result file:  prediction_results_model_quality/prediction-result-2021-09-15.csv\n"
     ]
    }
   ],
   "source": [
    "# Create result directory if not exist\n",
    "if not os.path.exists(result_dir):\n",
    "    os.makedirs(result_dir)\n",
    "\n",
    "target_date = pd.to_datetime(prediction_target_date)\n",
    "print('Loading data for', target_date.strftime('%Y-%m'))\n",
    "df_features = get_data_for_pred(target_date, sampling_rate)\n",
    "        \n",
    "# Exec prediction for the target date\n",
    "print('Predicting', target_date, endpoint_name)\n",
    "df_pred = df_features[df_features.index == target_date].copy()\n",
    "df_pred[['pred', 'inference_id']] = model_utils.exec_prediction(endpoint_name, df_pred)\n",
    "\n",
    "# Save prediction result\n",
    "result_file = f'{result_dir}/prediction-result-{prediction_target_date}.csv'\n",
    "df_pred.to_csv(result_file, index=False)\n",
    "print('Prediciton completed. Result file: ', result_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 推論時に取得したInference IDとGround TruthをマージしてS3にアップロードする"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sagemaker\n",
    "import boto3\n",
    "import pandas as pd\n",
    "import io\n",
    "import json\n",
    "import sagemaker\n",
    "from sagemaker.s3 import S3Uploader\n",
    "from datetime import datetime\n",
    "\n",
    "s3r = boto3.resource('s3')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Ground Truth（今回はpickup_countカラム）および、inference id（今回はinference_id）が格納されたファイルを取得する\n",
    "# ここではローカルファイルのcsvに1日分の推論実行結果が保存されており、Ground TruthとInferenceが同一ファイルに格納されていると想定する。\n",
    "ground_truth_colname = 'pickup_count'\n",
    "inference_id_colname = 'inference_id'\n",
    "\n",
    "# Ground TruthをアップロードするPrefixを設定\n",
    "# create_monitoring_scheduleを実行した際のモニタリングジョブの設定と一致させる\n",
    "bucket = sagemaker.Session().default_bucket()\n",
    "ground_truth_path = f's3://{bucket}/{ground_truth_prefix}'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>pickup_count</th>\n",
       "      <th>history_12slots</th>\n",
       "      <th>history_16slots</th>\n",
       "      <th>history_20slots</th>\n",
       "      <th>history_24slots</th>\n",
       "      <th>history_28slots</th>\n",
       "      <th>history_32slots</th>\n",
       "      <th>history_36slots</th>\n",
       "      <th>history_40slots</th>\n",
       "      <th>history_44slots</th>\n",
       "      <th>...</th>\n",
       "      <th>tolls_amount_mean_20slot</th>\n",
       "      <th>tolls_amount_mean_96slot</th>\n",
       "      <th>tolls_amount_mean_100slot</th>\n",
       "      <th>tolls_amount_mean_104slot</th>\n",
       "      <th>tolls_amount_mean_192slot</th>\n",
       "      <th>tolls_amount_mean_196slot</th>\n",
       "      <th>tolls_amount_mean_200slot</th>\n",
       "      <th>time_slot</th>\n",
       "      <th>pred</th>\n",
       "      <th>inference_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>113</td>\n",
       "      <td>264.0</td>\n",
       "      <td>324.0</td>\n",
       "      <td>350.0</td>\n",
       "      <td>359.0</td>\n",
       "      <td>369.0</td>\n",
       "      <td>362.0</td>\n",
       "      <td>352.0</td>\n",
       "      <td>301.0</td>\n",
       "      <td>294.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.352771</td>\n",
       "      <td>0.440594</td>\n",
       "      <td>0.674265</td>\n",
       "      <td>0.203778</td>\n",
       "      <td>0.169022</td>\n",
       "      <td>0.544014</td>\n",
       "      <td>0.699157</td>\n",
       "      <td>0</td>\n",
       "      <td>88</td>\n",
       "      <td>f7642aa4-4496-43be-9185-89e9cfe14c53</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>91</td>\n",
       "      <td>249.0</td>\n",
       "      <td>274.0</td>\n",
       "      <td>330.0</td>\n",
       "      <td>391.0</td>\n",
       "      <td>351.0</td>\n",
       "      <td>320.0</td>\n",
       "      <td>343.0</td>\n",
       "      <td>340.0</td>\n",
       "      <td>279.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.277879</td>\n",
       "      <td>0.786000</td>\n",
       "      <td>0.751639</td>\n",
       "      <td>0.463443</td>\n",
       "      <td>0.883146</td>\n",
       "      <td>0.681200</td>\n",
       "      <td>0.872543</td>\n",
       "      <td>1</td>\n",
       "      <td>71</td>\n",
       "      <td>a0911496-6e3e-4984-84e2-349152581687</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>81</td>\n",
       "      <td>273.0</td>\n",
       "      <td>270.0</td>\n",
       "      <td>327.0</td>\n",
       "      <td>405.0</td>\n",
       "      <td>376.0</td>\n",
       "      <td>363.0</td>\n",
       "      <td>333.0</td>\n",
       "      <td>316.0</td>\n",
       "      <td>298.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.260398</td>\n",
       "      <td>0.777119</td>\n",
       "      <td>0.564286</td>\n",
       "      <td>0.536010</td>\n",
       "      <td>1.105844</td>\n",
       "      <td>0.661009</td>\n",
       "      <td>0.595752</td>\n",
       "      <td>2</td>\n",
       "      <td>72</td>\n",
       "      <td>bed71a18-4006-461d-863d-e9c1ec7e65d6</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3 rows × 145 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   pickup_count  history_12slots  history_16slots  history_20slots  \\\n",
       "0           113            264.0            324.0            350.0   \n",
       "1            91            249.0            274.0            330.0   \n",
       "2            81            273.0            270.0            327.0   \n",
       "\n",
       "   history_24slots  history_28slots  history_32slots  history_36slots  \\\n",
       "0            359.0            369.0            362.0            352.0   \n",
       "1            391.0            351.0            320.0            343.0   \n",
       "2            405.0            376.0            363.0            333.0   \n",
       "\n",
       "   history_40slots  history_44slots  ...  tolls_amount_mean_20slot  \\\n",
       "0            301.0            294.0  ...                  0.352771   \n",
       "1            340.0            279.0  ...                  0.277879   \n",
       "2            316.0            298.0  ...                  0.260398   \n",
       "\n",
       "   tolls_amount_mean_96slot  tolls_amount_mean_100slot  \\\n",
       "0                  0.440594                   0.674265   \n",
       "1                  0.786000                   0.751639   \n",
       "2                  0.777119                   0.564286   \n",
       "\n",
       "   tolls_amount_mean_104slot  tolls_amount_mean_192slot  \\\n",
       "0                   0.203778                   0.169022   \n",
       "1                   0.463443                   0.883146   \n",
       "2                   0.536010                   1.105844   \n",
       "\n",
       "   tolls_amount_mean_196slot  tolls_amount_mean_200slot  time_slot  pred  \\\n",
       "0                   0.544014                   0.699157          0    88   \n",
       "1                   0.681200                   0.872543          1    71   \n",
       "2                   0.661009                   0.595752          2    72   \n",
       "\n",
       "                           inference_id  \n",
       "0  f7642aa4-4496-43be-9185-89e9cfe14c53  \n",
       "1  a0911496-6e3e-4984-84e2-349152581687  \n",
       "2  bed71a18-4006-461d-863d-e9c1ec7e65d6  \n",
       "\n",
       "[3 rows x 145 columns]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Ground Truthの収集\n",
    "df_prediction = pd.read_csv(result_file)\n",
    "df_prediction.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Ground TruthとInference IDを開発者ガイドで定義されたフォーマットに変換する\n",
    "# https://docs.aws.amazon.com/sagemaker/latest/dg/model-monitor-model-quality-merge.html\n",
    "def ground_truth_with_id(ground_truth, inference_id):\n",
    "    return {\n",
    "        \"groundTruthData\": {\n",
    "            \"data\": str(ground_truth),\n",
    "            \"encoding\": \"CSV\",\n",
    "        },\n",
    "        \"eventMetadata\": {\n",
    "            \"eventId\": str(inference_id),\n",
    "        },\n",
    "        \"eventVersion\": \"0\",\n",
    "    }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "推論したデータの前半に対しては、すぐにGround Truthが付与されたと仮定して、Ground Truthをアップロードする"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'s3://sagemaker-ap-northeast-1-370828233696/model_monitor/model_quality_ground_truth/2022/12/16/10/ground_truth.jsonl'"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Prediction結果をjsonに変換し、jsonlとしてstring化する\n",
    "gt_records = df_prediction.iloc[:int(df_prediction.shape[0]/2)].apply(lambda x: json.dumps(ground_truth_with_id(x[ground_truth_colname], x[inference_id_colname])), axis=1)\n",
    "gt_jsonl = '\\n'.join(gt_records)\n",
    "\n",
    "# StringをS3のground truth inputにアップロードする\n",
    "upload_time = datetime.now().strftime('%Y/%m/%d/%H')\n",
    "target_s3_uri = f'{ground_truth_path}/{upload_time}/ground_truth.jsonl'\n",
    "S3Uploader.upload_string_as_file_body(gt_jsonl, target_s3_uri)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "推論したデータの後半に対しては、Ground Truthの付与に1時間かかったと仮定して、3600秒後にGround Truthをアップロードする"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'s3://sagemaker-ap-northeast-1-370828233696/model_monitor/model_quality_ground_truth/2022/12/16/11/ground_truth.jsonl'"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "time.sleep(3600)\n",
    "# Prediction結果をjsonに変換し、jsonlとしてstring化する\n",
    "gt_records = df_prediction.iloc[int(df_prediction.shape[0]/2):].apply(lambda x: json.dumps(ground_truth_with_id(x[ground_truth_colname], x[inference_id_colname])), axis=1)\n",
    "gt_jsonl = '\\n'.join(gt_records)\n",
    "\n",
    "# StringをS3のground truth inputにアップロードする\n",
    "upload_time = datetime.now().strftime('%Y/%m/%d/%H')\n",
    "target_s3_uri = f'{ground_truth_path}/{upload_time}/ground_truth.jsonl'\n",
    "S3Uploader.upload_string_as_file_body(gt_jsonl, target_s3_uri)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- ここまでの実行で推論の実行とGround Truthのアップロードが完了しました\n",
    "- 次の時間の0分から20分の間にモデル品質のモニタリングジョブが稼働し、モニタリングレポートが出力されます"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## B1 （オプションB） モニタリングの実行をまたずにサンプルで分析してみる\n",
    "モニタリング結果の分析では、複数のモニタリング周期にまたがる精度の推移を可視化します。複数のモニタリング周期にまたがってレポートを出力するには時間がかかるため、サンプルコードに含まれるレポートで分析や可視化を試したい場合は、以下のセルを実行してS3バケットにサンプルのレポートをアップロードしてください  \n",
    "ご自身のレポートで可視化を行う場合は、このセルはスキップしてください"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "sagemaker.s3.S3Uploader.upload('model_quality_samples', f's3://{bucket}/model_monitor/model_quality_samples')\n",
    "\n",
    "baseline_prefix = 'model_monitor/model_quality_samples/baseline'\n",
    "report_prefix = 'model_monitor/model_quality_samples/reports'\n",
    "specific_report_prefix = 'model_monitor/model_quality_samples/reports/2020/03/16/01'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# B2. モニタリング結果の分析"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "分析の実行"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sagemaker\n",
    "from sagemaker import model_monitor\n",
    "import boto3\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_reports(report_bucket, report_location, file_type):\n",
    "    s3 = boto3.client('s3')\n",
    "    assert file_type in ('statistics', 'constraints', 'violations')\n",
    "    \n",
    "    resp = s3.list_objects(Bucket=report_bucket, Prefix=report_location)\n",
    "    report_files = [x['Key'] for x in resp['Contents'] if x['Key'].endswith(f'{file_type}.json')]\n",
    "\n",
    "    monitoring_reports = {}\n",
    "    for key in sorted(report_files):\n",
    "        report_s3uri = f's3://{report_bucket}/{key}'\n",
    "        \n",
    "        if file_type == 'statistics':\n",
    "            body_dict = model_monitor.Statistics.from_s3_uri(report_s3uri).body_dict\n",
    "        elif file_type == 'constraints':\n",
    "            body_dict = model_monitor.Constraints.from_s3_uri(report_s3uri).body_dict\n",
    "        elif file_type == 'violations':\n",
    "            body_dict = model_monitor.ConstraintViolations.from_s3_uri(report_s3uri).body_dict\n",
    "        else:\n",
    "            print('Unexpected file type')\n",
    "            return \n",
    "        \n",
    "        report_time = pd.to_datetime('-'.join(key.split('/')[-5:-1]), format='%Y-%m-%d-%H')\n",
    "        monitoring_reports[report_time] = body_dict\n",
    "\n",
    "    return monitoring_reports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def extract_stats_value(statistics_reports, metric_type):\n",
    "    \n",
    "    all_values = {}\n",
    "    for key, report in statistics_reports.items():    \n",
    "        report_values = {}\n",
    "        for metric_name, metric_value_dict in report[metric_type].items():\n",
    "            report_values[metric_name] = metric_value_dict['value']\n",
    "                \n",
    "        all_values[key] = report_values\n",
    "\n",
    "    df = pd.DataFrame(all_values).transpose()\n",
    "    return df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 精度メトリクスを時系列でDataFrame化する"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mae</th>\n",
       "      <th>mse</th>\n",
       "      <th>rmse</th>\n",
       "      <th>r2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2020-01-06 01:00:00</th>\n",
       "      <td>48.736082</td>\n",
       "      <td>3559.985828</td>\n",
       "      <td>59.665617</td>\n",
       "      <td>0.915222</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-01-13 01:00:00</th>\n",
       "      <td>33.640843</td>\n",
       "      <td>2203.995053</td>\n",
       "      <td>46.946726</td>\n",
       "      <td>0.959612</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-01-20 01:00:00</th>\n",
       "      <td>56.177981</td>\n",
       "      <td>5339.805945</td>\n",
       "      <td>73.073976</td>\n",
       "      <td>0.840637</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-01-27 01:00:00</th>\n",
       "      <td>32.058851</td>\n",
       "      <td>1598.723379</td>\n",
       "      <td>39.984039</td>\n",
       "      <td>0.966500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-02-03 01:00:00</th>\n",
       "      <td>39.323701</td>\n",
       "      <td>2523.980451</td>\n",
       "      <td>50.239232</td>\n",
       "      <td>0.948893</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-02-10 01:00:00</th>\n",
       "      <td>41.485255</td>\n",
       "      <td>2958.223253</td>\n",
       "      <td>54.389551</td>\n",
       "      <td>0.949442</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-02-17 01:00:00</th>\n",
       "      <td>47.202256</td>\n",
       "      <td>3553.185067</td>\n",
       "      <td>59.608599</td>\n",
       "      <td>0.865158</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-02-24 01:00:00</th>\n",
       "      <td>39.585234</td>\n",
       "      <td>2842.260041</td>\n",
       "      <td>53.312851</td>\n",
       "      <td>0.942540</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-03-02 01:00:00</th>\n",
       "      <td>41.083948</td>\n",
       "      <td>2872.077381</td>\n",
       "      <td>53.591766</td>\n",
       "      <td>0.942076</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-03-09 01:00:00</th>\n",
       "      <td>61.858424</td>\n",
       "      <td>7275.347270</td>\n",
       "      <td>85.295646</td>\n",
       "      <td>0.812754</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-03-16 01:00:00</th>\n",
       "      <td>101.941843</td>\n",
       "      <td>14074.934422</td>\n",
       "      <td>118.637829</td>\n",
       "      <td>-1.424614</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-03-23 01:00:00</th>\n",
       "      <td>220.142144</td>\n",
       "      <td>57067.685159</td>\n",
       "      <td>238.888437</td>\n",
       "      <td>-176.484370</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-03-30 01:00:00</th>\n",
       "      <td>190.397829</td>\n",
       "      <td>45052.495289</td>\n",
       "      <td>212.255731</td>\n",
       "      <td>-274.076476</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-04-06 01:00:00</th>\n",
       "      <td>92.610319</td>\n",
       "      <td>9805.263248</td>\n",
       "      <td>99.021529</td>\n",
       "      <td>-85.943666</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-04-13 01:00:00</th>\n",
       "      <td>63.392568</td>\n",
       "      <td>4285.378399</td>\n",
       "      <td>65.462802</td>\n",
       "      <td>-37.234659</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-04-20 01:00:00</th>\n",
       "      <td>62.310475</td>\n",
       "      <td>4175.124192</td>\n",
       "      <td>64.615201</td>\n",
       "      <td>-34.572366</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-04-27 01:00:00</th>\n",
       "      <td>61.910881</td>\n",
       "      <td>4149.062384</td>\n",
       "      <td>64.413216</td>\n",
       "      <td>-25.887938</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-05-04 01:00:00</th>\n",
       "      <td>59.307794</td>\n",
       "      <td>3856.049834</td>\n",
       "      <td>62.097100</td>\n",
       "      <td>-18.118762</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-05-11 01:00:00</th>\n",
       "      <td>60.788436</td>\n",
       "      <td>4143.170986</td>\n",
       "      <td>64.367468</td>\n",
       "      <td>-18.026519</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-05-18 01:00:00</th>\n",
       "      <td>59.132732</td>\n",
       "      <td>3848.765692</td>\n",
       "      <td>62.038421</td>\n",
       "      <td>-16.037470</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                            mae           mse        rmse          r2\n",
       "2020-01-06 01:00:00   48.736082   3559.985828   59.665617    0.915222\n",
       "2020-01-13 01:00:00   33.640843   2203.995053   46.946726    0.959612\n",
       "2020-01-20 01:00:00   56.177981   5339.805945   73.073976    0.840637\n",
       "2020-01-27 01:00:00   32.058851   1598.723379   39.984039    0.966500\n",
       "2020-02-03 01:00:00   39.323701   2523.980451   50.239232    0.948893\n",
       "2020-02-10 01:00:00   41.485255   2958.223253   54.389551    0.949442\n",
       "2020-02-17 01:00:00   47.202256   3553.185067   59.608599    0.865158\n",
       "2020-02-24 01:00:00   39.585234   2842.260041   53.312851    0.942540\n",
       "2020-03-02 01:00:00   41.083948   2872.077381   53.591766    0.942076\n",
       "2020-03-09 01:00:00   61.858424   7275.347270   85.295646    0.812754\n",
       "2020-03-16 01:00:00  101.941843  14074.934422  118.637829   -1.424614\n",
       "2020-03-23 01:00:00  220.142144  57067.685159  238.888437 -176.484370\n",
       "2020-03-30 01:00:00  190.397829  45052.495289  212.255731 -274.076476\n",
       "2020-04-06 01:00:00   92.610319   9805.263248   99.021529  -85.943666\n",
       "2020-04-13 01:00:00   63.392568   4285.378399   65.462802  -37.234659\n",
       "2020-04-20 01:00:00   62.310475   4175.124192   64.615201  -34.572366\n",
       "2020-04-27 01:00:00   61.910881   4149.062384   64.413216  -25.887938\n",
       "2020-05-04 01:00:00   59.307794   3856.049834   62.097100  -18.118762\n",
       "2020-05-11 01:00:00   60.788436   4143.170986   64.367468  -18.026519\n",
       "2020-05-18 01:00:00   59.132732   3848.765692   62.038421  -16.037470"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Statisticsレポートを取得\n",
    "statistics_reports = get_reports(bucket, report_prefix, 'statistics')\n",
    "\n",
    "# 特定のmetricを取得\n",
    "df_regression_metrics = extract_stats_value(statistics_reports, 'regression_metrics')\n",
    "df_regression_metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 取得した精度メトリクスを時系列でグラフ化"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x7f0c0066ee10>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA14AAAEyCAYAAADqcQH6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAPYQAAD2EBqD+naQAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nOzdeXhTZd4+8PskTdI9ULrThRZaRFugUISCLCNQZFNQAUeHERdwRREdlHd+jjDjyOu4viMqoGwiIjhsMiCCIi3IVnaKrF1oC11pmzZdsp7fH2lDC2WV9OQk9+e6ciU550nyPeSQ5s7znOcIoiiKICIiIiIiIodRSF0AERERERGRq2PwIiIiIiIicjAGLyIiIiIiIgdj8CIiIiIiInIwBi8iIiIiIiIHY/AiIiIiIiJyMAYvIiIiIiIiB2PwIiIiIiIicjAGLyIiIiIiIgdj8CIiIre1ZMkSCIIAQRCwffv2K9aLoohOnTpBEAQMGjToivVlZWXQaDQQBAH79+9v8TUmTZpkf42WLkRE5B48pC6AiIhIan5+fli4cOEV4SotLQ1ZWVnw8/Nr8XHLli2D0WgEACxcuBDJyckttvPy8sK2bdtua81ERCQvDF5EROT2JkyYgOXLl+PTTz+Fv7+/ffnChQuRkpKCqqqqFh+3aNEiBAcHIzo6GitWrMCHH34ILy+vK9opFAr06dPHYfUTEZHz41BDIiJye3/84x8BACtWrLAv0+l0WL16NZ588skWH7N3715kZmZi4sSJmDx5sr09ERFRSxi8iIjI7fn7++Phhx/GokWL7MtWrFgBhUKBCRMmtPiYhQsXAgCefPJJPPLII/D29rYva4nZbL7iYrVab++GEBGR02LwIiIigi1A7du3D8ePHwdgG0Y4bty4Fo/vqq2txcqVK9GnTx/ceeed8PPzw7hx4+zHhF2upqYGKpXqiktqaqrDt4uIiJwDj/EiIiICMHDgQHTs2BGLFi3CpEmTkJGRgQ8++KDFtqtWrUJVVVWzYYhPPvkkli5disWLF+Ptt99u1t7Lywvp6elXPE/T48mIiMi1MXgREREBEAQBTzzxBP7973+jvr4e8fHx6N+/f4ttFy5cCE9PT9x3332orKwEAHTt2hUdOnTAkiVLMHv2bCiVSnt7hUJx1RkPiYjIPXCoIRERUYNJkyahrKwM8+bNwxNPPNFim9OnT2Pnzp2or69HVFQU2rZta7/k5ubi/Pnz+PHHH1u5ciIicnbs8SIiImrQvn17/OUvf8HJkyfx+OOPt9imcQKNL774Ap06dWq2rq6uDg888AAWLVqEESNGOLxeIiKSDwYvIiKiJv73f//3quvMZjO++uordOnSBU8//XSLbUaPHo3vv/8epaWlCAoKAgBYrVbs2bOnxfZJSUnQaDS/v3AiInJqDF5EREQ3aOPGjSgqKsIbb7xx1TZTpkzBmjVrsGzZMkyfPh2ArScsJSWlxfZnzpy5oueMiIhcjyCKoih1EURERERERK6Mk2sQERERERE5GIMXERERERGRgzF4EREREREROZisgtecOXPQq1cv+Pn5ITg4GGPGjMGpU6eatRk0aBAEQWh2eeSRR5q1qaiowMSJE6HVaqHVajFx4kT7CTCJiIiIiIhuN1kFr7S0NLzwwgvYs2cPtm7dCrPZjNTUVNTU1DRrN3nyZBQWFtov8+fPb7b+0UcfxeHDh7F582Zs3rwZhw8fxsSJE1tzU4iIiIiIyI3IelbD0tJSBAcHIy0tDQMGDABg6/Hq3r07Pv744xYfc+LECdx5553Ys2cPevfuDQDYs2cPUlJScPLkSXTu3LnV6iciIiIiIvcg6/N46XQ6AEBAQECz5cuXL8fXX3+NkJAQDB8+HG+99Rb8/PwAALt374ZWq7WHLgDo06cPtFotdu3a1WLwMhgMMBgM9vtWqxXl5eVo164dBEFwxKYREREREZEMiKKI6upqhIeHQ6G4+oBC2QYvURQxffp03HPPPUhISLAvf+yxxxATE4PQ0FBkZmZi5syZOHLkCLZu3QoAKCoqQnBw8BXPFxwcjKKiohZfa86cOZg9e7ZjNoSIiIiIiGQvPz8fERERV10v2+D14osv4ujRo9i5c2ez5ZMnT7bfTkhIQFxcHJKTk3Hw4EH06NEDAFrspRJF8aq9VzNnzsT06dPt93U6HaKiopCfnw9/f//bsTlERERERCRDVVVViIyMtI+wuxpZBq+pU6fi+++/R3p6+jVTJQD06NEDKpUKZ86cQY8ePRAaGori4uIr2pWWliIkJKTF59BoNNBoNFcs9/f3Z/AiIiIiIqLrHoIkq1kNRVHEiy++iDVr1mDbtm2IiYm57mOOHz8Ok8mEsLAwAEBKSgp0Oh327dtnb7N3717odDr07dvXYbUTEREREZH7ktWshs8//zy++eYbrF+/vtkkGFqtFl5eXsjKysLy5csxYsQIBAYG4rfffsOrr74KLy8vZGRkQKlUAgCGDx+OCxcu2KeZnzJlCqKjo7Fhw4YbqqOqqgparRY6nY49XkREREREbuxGs4GsgtfVuu8WL16MSZMmIT8/H3/605+QmZkJvV6PyMhIjBw5Em+99VazmQ/Ly8vx0ksv4fvvvwcA3H///Zg7dy7atGlzQ3UweBEREREREeCiwctZMHgRERERERFw49lAVsd4ERERERERyRGDFxERERERkYMxeBERERERETkYgxcREREREZGDMXgRERERERE5GIMXERERERGRgzF4ERERERERORiDFxERERERkYMxeBERERERETkYgxcREREREZGDMXgRERERERE5GIMXERERERGRgzF4ERERERERORiDFxERERERkYMxeBERERERETkYgxcREREREZGDMXgRERHRFTZnFiH9dKnUZRARuQwPqQsgIiIi53LgXDme/foAAGB4QihmP3AXgv08Ja6KiEje2ONFREREzazMyLff/iGzCEM/TMeagwUQRVHCqoiI5I3Bi4iIiOz0BjP+e7QQAPD3B+7CXeH+0NWZMH3VETy5JAMXKuskrpCISJ4YvIiIiMhu09FC1BotiAn0wcQ+0Vj3Qj/8ZVhnqJUK/HKqFKkfpeObvXns/SIiukkMXkRERGS3ar9tmOG45AgIggCVUoEX/tAJm16+B0lRbaA3mPE/a4/h0S/2Iu9ircTVEhHJB4MXERERAQDOluix/1wFFALwUI+IZus6BfvhP8/2xZuj7oSnSoHd2Rcx7ON0LNqZA4uVvV9ERNfD4EVEREQAgO8O2Hq7/tA5GCH+V85iqFQIeOqeGPw4bQD6xAagzmTB3//7G8bN24WzJfrWLpeISFYYvIiIiAgmixWrD5wHAIxLjrxm2+h2Pvjm6T7459gE+Go8cDCvEiP+vQOf/nIWZou1NcolIpIdBi8iIiLC9lOlKNMb0M5HjXvvCL5ue4VCwGO9o7HllQEY1DkIRrMV7/14CmM++xW/XahqhYqJiOSFwYuIiIjsk2o82KM91B43/vUgvI0XFk/qhQ/GdYPWS4XM81W4f+5OfLjlFAxmi6PKJSKSHQYvIiIiN1dSXY9tJ0sAAOOvM8ywJYIg4KGeEdg6fQDuuysUZquIf287i9Gf7MTh/MrbXS4RkSwxeBEREbm5tQfPw2IVkRTVBnEhfrf8PMF+npg3sSc+e6wHAn3VOF2sx4Of/Yp3Np1AnZG9X0Tk3hi8iIiI3JgoiljZMMzwVnq7WjIiMQxbXxmIsUntYRWBBenZGP5/6dibffG2PD8RkRzJKnjNmTMHvXr1gp+fH4KDgzFmzBicOnWqWRuDwYCpU6ciMDAQPj4+uP/++1FQUNCsTV5eHkaPHg0fHx8EBgbipZdegtFobM1NISIicgoH8yqQXVoDL5USo7qG3bbnbeujxkcTumPh48kI9fdE7sVaTFiwB2+uy4TeYL5tr0NEJBeyCl5paWl44YUXsGfPHmzduhVmsxmpqamoqamxt5k2bRrWrl2Lb7/9Fjt37oRer8eoUaNgsdiGOFgsFowcORI1NTXYuXMnvv32W6xevRqvvvqqVJtFREQkmZUZtt6uEYlh8PNU3fbnH9wlBFumD8Af77b1pi3bcw7DPkpH+unS2/5aRETOTBBFUbanmy8tLUVwcDDS0tIwYMAA6HQ6BAUFYdmyZZgwYQIA4MKFC4iMjMSmTZswbNgw/PDDDxg1ahTy8/MRHh4OAPj2228xadIklJSUwN/f/7qvW1VVBa1WC51Od0PtiYiInFGNwYxe//wJtUYLVk7pg96x7Rz6er+eLcPrq4+ioKIOADCuZwT+38g7ofW+/YGPiKi13Gg2kFWP1+V0Oh0AICAgAABw4MABmEwmpKam2tuEh4cjISEBu3btAgDs3r0bCQkJ9tAFAMOGDYPBYMCBAwdafB2DwYCqqqpmFyIiIrnbeKwQtUYLOrTzxt0xAQ5/vX6dAvHjtAGY1LcDBAH47kABhn6Uhi3Hixz+2kREUpNt8BJFEdOnT8c999yDhIQEAEBRURHUajXatm3brG1ISAiKiorsbUJCQpqtb9u2LdRqtb3N5ebMmQOtVmu/REbenoOPiYiIpLSqYZjhuORICILQKq/po/HArPvvwnfPpCA20Acl1QZMWXYAU1ccwkW9oVVqICKSgmyD14svvoijR49ixYoV120rimKzPygt/XG5vE1TM2fOhE6ns1/y8/NvvXAiIiInkFWqx/5zFVAIwMM9I1r99ZM7BGDTy/3x7MCOUCoEbDhyAUM/Ssf3Ry5AxkdBEBFdlSyD19SpU/H999/jl19+QUTEpT8WoaGhMBqNqKioaNa+pKTE3ssVGhp6Rc9WRUUFTCbTFT1hjTQaDfz9/ZtdiIiI5GxVwxTygzoHI8TfU5IaPFVKvDH8Dqx7vh/uCPVDeY0RL604hCnLDqC4ql6SmoiIHEVWwUsURbz44otYs2YNtm3bhpiYmGbre/bsCZVKha1bt9qXFRYWIjMzE3379gUApKSkIDMzE4WFhfY2W7ZsgUajQc+ePVtnQ4iIiCRkslix+sB5ALfv3F2/R2KEFt+/eA9eGRIPlVLA1t+KMeTDNKzan8/eLyJyGbKa1fD555/HN998g/Xr16Nz58725VqtFl5eXgCA5557Dv/973+xZMkSBAQE4LXXXsPFixdx4MABKJVKWCwWdO/eHSEhIXjvvfdQXl6OSZMmYcyYMfjkk09uqA7OakhERHL202/FePqr/Wjno8bumYOh9nCe32FPFVVjxn+O4EiBbQKt/nGBmPNgIiLaektcGRFRy1xyVsPPP/8cOp0OgwYNQlhYmP2ycuVKe5uPPvoIY8aMwfjx49GvXz94e3tjw4YNUCqVAAClUomNGzfC09MT/fr1w/jx4zFmzBi8//77Um0WERFRq1rZMMxwbFJ7pwpdANA51A+rn+uLmcPvgMZDgR1nyjDso3Qs250Lq1U2vxUTEV1BVj1ezoI9XkREJFcl1fVImbMNFquILa8MQHyIn9QlXVV2qR6vrz6KjFzbsdt3xwTg3Ye6IibQR+LKiIgucckeLyIiIvp91h48D4tVRPfINk4dugAgNsgXK6ekYPb9d8FbrcS+nHKMm7cLeoNZ6tKIiG4agxcREZGbEEXRPpvhhF7ST6pxIxQKAY/37YAfpw1AVIA3yvRGfLsvT+qyiIhuGoMXERGRmziYV4Gs0hp4qZQY1TVM6nJuSmSAN54b1BEA8OWOHBjNVokrIiK6OQxeREREbmJVRgEAYERiGPw8VRJXc/PGJrVHkJ8GRVX1WH/4vNTlEBHdFAYvIiIiN1BjMOO/Ry8AAMYnR0hcza3xVCnxZD/bOTwXpGdzlkMikhUGLyIiIjew8VghaowWdGjnjbtjAqQu55Y91icKfhoPnCnRY9vJEqnLISK6YQxeREREbuC7hkk1xiVHQhAEiau5df6eKjzaJwoAMC8tS+JqiIhuHIMXERGRi8sq1SMjtwIKAXiohzyHGTb1VL8YqJUK7D9Xgf255VKXQ0R0Qxi8iIiIXNx3+22TagzqHIxQrafE1fx+wf6eeLBHewDs9SIi+WDwIiIicmFmixWrD9qCl1wn1WjJlAGxEATgpxMlOF1cLXU5RETXxeBFRETkwrafKkVptQHtfNS4944Qqcu5bWKDfDHszlAAwPy0bImrISK6PgYvIiIiF7aqYVKNsUntofZwrT/7zzacUHn94fO4UFkncTVERNfmWp/AREREZFdabbBPuT6+V6TE1dx+3SPboE9sAMxWEQt35khdDhHRNTF4ERERuai1hwpgtoroHtkG8SF+UpfjEM8OtPV6rdiXh8pao8TVEBFdHYMXERGRCxJFESszbMMMxye7Xm9Xo4HxQegS5o9aowXLdp+Tuhwioqti8CIiInJBB/MqkVVaA0+VAqO7hUldjsMIgoBnB8YCAJbsykW9ySJxRURELWPwIiIickHfNUyqMSIxDH6eKomrcayRiWGIaOuFizVG+3YTETkbBi8iIiIXU2MwY8ORCwCACS48zLCRh1KByf1tvV4LdmTDbLFKXBER0ZUYvIiIiFzMpmOFqDFa0KGdN+6OCZC6nFYxPjkSAT5q5JfXYVNmkdTlEBFdgcGLiIjIxTSeu2tcciQEQZC4mtbhpVbi8ZQOAIB527MgiqK0BRERXYbBi4iIyIVkl+qRkVsBhQA81CNC6nJa1Z9TouGlUuK3wirsOFMmdTlERM0weBEREbmQVfsLANimWQ/VekpcTetq66PGI3fbjmmbl5YlcTVERM0xeBEREbkIs8WK1QdtwWtCL9efVKMlT/ePhYdCwK6sizhaUCl1OUREdgxeRERELiLtdClKqw1o56PGvXeESF2OJNq38cL93cIBsNeLiJwLgxcREZGLWJlhm1RjbFJ7qD3c90/8MwM7AgB+yCxCTlmNxNUQEdm476cyERGRCymtNmDbyRIAttkM3VnnUD/ce0cwRBFYkJ4tdTlERAAYvIiIiFzC2kMFMFtFdItsg86hflKXI7lnG3q9Vh8sQEl1vcTVEBExeBEREcmeKIr22QwnuHlvV6NeHdqiR1QbGM1WLP41V+pyiIgYvIiIiOTuUH4lzpbo4alSYFS3MKnLcQqCINh7vb7ecw7V9SaJKyIidyer4JWeno7Ro0cjPDwcgiBg3bp1zdZPmjQJgiA0u/Tp06dZG4PBgKlTpyIwMBA+Pj64//77UVBQ0JqbQUREdFutaphUY0RiGPw9VRJX4zyGdAlBxyAfVNeb8c3ePKnLISI3J6vgVVNTg27dumHu3LlXbXPfffehsLDQftm0aVOz9dOmTcPatWvx7bffYufOndDr9Rg1ahQsFoujyyciIrrtao1mbDhyAQAwnsMMm1EoBDwzwNbrtXBnDgxm/q0nIul4SF3AzRg+fDiGDx9+zTYajQahoaEtrtPpdFi4cCGWLVuGIUOGAAC+/vprREZG4qeffsKwYcNue81ERESOtPFoIWqMFnRo543eMQFSl+N0HkgKxwdbT6G4yoB1h85jQq8oqUsiIjclqx6vG7F9+3YEBwcjPj4ekydPRklJiX3dgQMHYDKZkJqaal8WHh6OhIQE7Nq1S4pyiYiIfpfvGibVGJccCUEQJK7G+Wg8lHjqnhgAwPz0bFitosQVEZG7cqngNXz4cCxfvhzbtm3DBx98gIyMDNx7770wGAwAgKKiIqjVarRt27bZ40JCQlBUVHTV5zUYDKiqqmp2ISIiklp2qR77csuhEICHekRIXY7T+uPdUfDz9EB2aQ22niiWuhwiclMuFbwmTJiAkSNHIiEhAaNHj8YPP/yA06dPY+PGjdd8nCiK1/yVcM6cOdBqtfZLZCTH0BMRkfS+O2Dr7RoYH4RQrafE1TgvP08VJvaJBgDMS8uCKLLXi4han0sFr8uFhYUhOjoaZ86cAQCEhobCaDSioqKiWbuSkhKEhIRc9XlmzpwJnU5nv+Tn5zu0biIiousxW6xY3RC8OKnG9T3RLwZqDwUO5VViX0651OUQkRty6eB18eJF5OfnIyzMdk6Tnj17QqVSYevWrfY2hYWFyMzMRN++fa/6PBqNBv7+/s0uREREUko7XYqSagMCfNQY3OXqPx6STZCfBg/3tA3HnJeWJXE1ROSOZBW89Ho9Dh8+jMOHDwMAcnJycPjwYeTl5UGv1+O1117D7t27kZubi+3bt2P06NEIDAzE2LFjAQBarRZPPfUUXn31Vfz88884dOgQ/vSnPyExMdE+yyEREZEcrNpvG30xNqk91B6y+nMumSn9Y6EQgF9OleJkEY/XJqLWJatP6v379yMpKQlJSUkAgOnTpyMpKQl/+9vfoFQqcezYMTzwwAOIj4/H448/jvj4eOzevRt+fn725/joo48wZswYjB8/Hv369YO3tzc2bNgApVIp1WYRERHdlNJqA34+YZu1l8MMb1yHQB8MT7CNgpmfli1xNUTkbgSRR5jetKqqKmi1Wuh0Og47JCKiVvdFejb+uekEukW2wfoX+kldjqwcK9Bh9NydUCoEpP1lECLaektdEhHJ3I1mA1n1eBEREbk7URSxsmGY4QT2dt20xAgt+nVqB4tVxJc7cqQuh4jcCIMXERGRjBzKr8TZEj08VQqM6hYmdTmy9OzAjgCAlRn5qKgxSlwNEbkLBi8iIiIZ+a6ht2tEQhj8PVUSVyNP93QKxF3h/qgzWbB0d67U5RCRm2DwIiIikolaoxkbjhQCAMb34jDDWyUIgr3Xa+muXNQazRJXRETugMGLiIhIJjYdK4LeYEZ0O2/0jgmQuhxZG54QiqgAb1TUmrAqI1/qcojIDTB4ERERyURjQBifHAlBECSuRt48lApMHhALAPhiRw5MFqvEFRGRq2PwIiIikoHsUj325ZZDIQAP9YiQuhyXMK5nBAJ91ThfWYeNRwulLoeIXByDFxERkQz850ABAGBgfBBCtZ4SV+MaPFVKTOrbAQAwLy0LPLUpETkSgxcREZGTM1us9uA1nufuuq0m9ukAH7USJ4uqsf10qdTlEJELY/AiIiJyculnSlFSbUCAjxqDu4RIXY5L0Xqr8Me7owAA87ZnSVwNEbkyBi8iIiInt7JhUo2xSe2h9uCf7tvtqf4xUCkF7M0px6G8CqnLISIXxU9vIiIiJ1amN+DnEyUAOMzQUcK0Xnige3sAtmO9iIgcgcGLiIjIia09eB5mq4hukW3QOdRP6nJc1rMDbVPLb/mtGFmleomrISJXxOBFRETkpERRxKr9jefu4hTyjtQp2A9DuoRAFIEFadlSl0NELojBi4iIyEkdzq/EmRI9PFUKjO4WLnU5Lu+5QbZer7WHzqO4ql7iaojI1TB4EREROanG3q4RCWHw91RJXI3r6xkdgF4d2sJosWLRzhypyyEiF8PgRURE5IRqjWZsOFIIABjHSTVazbMDOwIAlu/Ng67OJHE1RORKGLyIiIic0KZjRdAbzIhu540+sQFSl+M2/tA5GPEhvtAbzFi+95zU5RCRC2HwIiIickKNwwzH9YyAIAgSV+M+FAoBzwyw9Xot2pmLepNF4oqIyFUweBERETmZnLIa7Msph0IAHurJ2Qxb2+hu4QjTeqJMb8Cag+elLoeIXASDFxERkZP5rqG3a0B8EMK0XhJX437UHgo8dU8MAGBBehYsVlHiiojIFTB4ERERORGzxYrVBwsAABM4qYZk/nh3FLReKuRerMWW40VSl0NELoDBi4iIyImknylFcZUBAT5qDO4SInU5bstH44E/p0QDAOalZUEU2etFRL8PgxcREZETWZVh6+0a07091B78My2lx/t2gMZDgSMFOuzOvih1OUQkc/xEJyIichJlegN+OlEMAJjQi8MMpRboq8H4huGe89KyJa6GiOSOwYuIiMhJrDt0HmariG4RWnQO9ZO6HAIwuX8sFAKQfroUxy/opC6HiGSMwYuIiMgJiKKIlRm22QzHs7fLaUS188bIruEAgPns9SKi34HBi4iIyAkczq/EmRI9PFUKjO4WLnU51MQzA2IBAP89egH55bUSV0NEcsXgRURE5ARW7bdNqjEiIQz+niqJq6GmEtpr0T8uEFYR+GIHe72I6NYweBEREUms1mjGhiMXAADjeO4up/TcwI4AgFX783FRb5C4GiKSI1kFr/T0dIwePRrh4eEQBAHr1q1rtl4URcyaNQvh4eHw8vLCoEGDcPz48WZtKioqMHHiRGi1Wmi1WkycOBGVlZWtuRlERETN/HCsCHqDGVEB3ugdEyB1OdSClI7t0DVCi3qTFUt35UpdDhHJkKyCV01NDbp164a5c+e2uP5f//oXPvzwQ8ydOxcZGRkIDQ3F0KFDUV1dbW/z6KOP4vDhw9i8eTM2b96Mw4cPY+LEia21CURERFdYub9hUo3kCCgUgsTVUEsEQcCzDb1eS3efQ43BLHFFRCQ3HlIXcDOGDx+O4cOHt7hOFEV8/PHH+Otf/4oHH3wQALB06VKEhITgm2++wTPPPIMTJ05g8+bN2LNnD3r37g0A+OKLL5CSkoJTp06hc+fOrbYtREREAPDziWLsyymHQgAe6hkhdTl0DcPuCkVMoA9yymrwbUY+nronRuqSiEhGZNXjdS05OTkoKipCamqqfZlGo8HAgQOxa9cuAMDu3buh1WrtoQsA+vTpA61Wa29DRETUWr7Zm4fJX+0HAIzp3h5hWi+JK6JrUSoETO5vm+Fw4Y5smCxWiSsiIjlxmeBVVFQEAAgJCWm2PCQkxL6uqKgIwcHBVzw2ODjY3qYlBoMBVVVVzS5ERES3ShRFfLDlFP5n7TFYReDhnhF49+GuUpdFN+DBHu0R6KvBBV09vj98QepyiEhGXCZ4NRKE5mPjRVFstuzy9S21udycOXPsk3FotVpERnLGKSIiujVGsxWvfncEn2w7CwB4aXAc3nu4K1RKl/uT7JI8VUo8eU8HAMD89CxYraK0BRGRbLjMp3xoaCgAXNFzVVJSYu8FCw0NRXFx8RWPLS0tvaKnrKmZM2dCp9PZL/n5+bexciIichfV9SY8tTQDaw6eh1Ih4N2HEjF9aPw1f/wj5/NY72j4ajxwuliPX06VSF0OEcmEywSvmJgYhIaGYuvWrfZlRqMRaWlp6Nu3LwAgJSUFOp0O+/S42DsAACAASURBVPbts7fZu3cvdDqdvU1LNBoN/P39m12IiIhuRpGuHuPn78GOM2XwVivx5ePJmNArSuqy6BZovVR4rLftvZuXliVxNUQkF7Ka1VCv1+Ps2bP2+zk5OTh8+DACAgIQFRWFadOm4Z133kFcXBzi4uLwzjvvwNvbG48++igAoEuXLrjvvvswefJkzJ8/HwAwZcoUjBo1ijMaEhGRw5wursakRftwQVePQF8NFk/qhcQIrdRl0e/w5D0xWPxrLjJyK3DgXDl6RvP8a0R0bbLq8dq/fz+SkpKQlJQEAJg+fTqSkpLwt7/9DQAwY8YMTJs2Dc8//zySk5Nx/vx5bNmyBX5+fvbnWL58ORITE5GamorU1FR07doVy5Ytk2R7iIjI9e3OuoiHPt+FC7p6xAb5YO3zfRm6XECIvyfGJrUHAHz2SxZEkcd6EdG1CSI/KW5aVVUVtFotdDodhx0SEdFVfX/kAl5bdQRGixXJ0W3xxZ+T0dZHLXVZdJtkleox5MM0iCIwID4I/xyTgMgAb6nLIqJWdqPZQFY9XkRERHIgiiLmp2XhpRWHYLRYMTwhFF8/3Zuhy8V0DPLF7PvvgtpDgfTTpRj2cToW7cyBhTMdElEL2ON1C9jjRUREV2Oxivj7huNYuvscAOCJfh3w/0beCaWCMxe6quxSPd5Ycwz7csoBAN0j2+Ddh7qic6jfdR5JRK7gRrMBg9ctYPAiIqKW1JsseGnFIWz5zXbqkv83sgue7h8rcVXUGqxWEd9m5GPOphOoNpihUgp4bmBHvHBvJ2g8lFKXR0QOxODlQAxeRER0ufIaI55emoGDeZVQKxX4cEI3jOoaLnVZ1MqKdPV4c30mtjaE745BPnj3oa5I7sBZD4lcFYOXAzF4ERFRU3kXa/H44n3IKauB1kuFL/6cjLtj+EXbXYmiiB8yi/C39cdRpjdAEICJfaIx47474KuR1Zl8iOgGcHINIiKiVnAkvxIPfv4rcspq0L6NF1Y/l8LQ5eYEQcCIxDD8NH0AxidHQBSBr3afw9AP07DtZLHU5RGRRNjjdQvY40VERADw84livPjNIdSZLLgr3B+LJ/VCsL+n1GWRk/n1bBlmrjmGvPJaAMD93cLx1ug70c5XI3FlRHQ7sMeLiIjIgZbvPYfJX+1HncmCAfFBWPlMCkMXtahfp0D8OG0ApgyIhUKwnd9tyIdpWHOwgCdeJnIj7PG6BezxIiJyX6Io4oMtpzH3l7MAgHE9I/DOg4lQKflbJl3fsQIdZqw+ihOFVQB44mUiV8DJNRyIwYuIyD0ZzVa8sfoo1hw6DwB4eXAcpg2JgyDwHF1040wWKxakZ+P/fj4Do9kKb7USr6V2xuN9O/B8b0QyxODlQAxeRETup6rehOe+PoBfz16EUiHgnbEJmNArSuqySMZ44mUi18Dg5UAMXkRE7qVIV49Ji/fhZFE1vNVKfPpYD/yhc7DUZZELsFpFrMjIw/9uOnnpxMuDOuGFP3TkiZeJZILBy4EYvIiI3MepompMWrwPhbp6BPpqsOSJXkhor5W6LHIxl594uVOwL959KBE9o3lqAiJnx1kNiYiIfqfdWRfx8LxdKNTVIzbIB2uf78vQRQ4RqvXEgok98dljPRDoq8HZEj0enrcbf1ufCb3BLHV5RHQbMHgRERG1YP3h83h80T5U15uRHN0Wa57ry5nnyKF44mUi18ahhreAQw2JiFyXKIqYn56N//3hJABgRGIoPhzfHZ4qHm9DrYsnXiaSBw41JCIiukkWq4i3vj9uD11P3RODuX/swdBFkuCJl4lcC3u8bgF7vIiIXE+d0YKXvj2Erb8VQxCAv47ogqf7x0pdFhEA4GhBJV5ffazZiZffGZuAiLYc/kokNc5q6EAMXkRErqW8xoinlmbgUF4l1B4KfDS+O0Z2DZO6LKJmeOJlIufE4OVADF5ERK7j3MUaTFqcgZyyGmi9VPjy8WT06sApvMl58cTLRM6Fx3gRERFdx+H8Sjz42S7klNWgfRsvrH6uL0MXOb3YIF98O7kP/jk2AX4aDxzOr8SoT3bgw62nYTBbpC6PiK6CwYuIiNzST78V45EFu3GxxoiE9v5Y+0JfdAr2lbosohuiUAh4rHc0tk4fiKF3hsBkEfHvn89g5L934sC5cqnLI6IWcKjhLeBQQyIieVu+9xzeXJcJqwgMjA/Cp4/1gK/GQ+qyiG6JKIr4IbMIf1t/HGV6AwQBuO+uULTzVUOlVECtVEDVePEQoFYqoPZoskwpXGrjcdn9hrbqhsfalzU8TqkQIAg8vozcG4/xciAGLyIieTKarfj4p9P4bHsWAGBCciTeHpsAlZIDQEj+KmuNeGfTCazaX9BqrykIaBbEmgW1y8KaSilAITReAIVgC21KBezLBQFQKi7dVggClIIAhcJ2gmmFACgbHqdo8lj7OsWl27b1l55HIaDZ6ysVApQKBXw9PeDn6QF/Tw/4alTwa7jvo/aAgpOW0A1g8HIgBi8iInmpqjdhxd48LP41F0VV9QCAV4bE46XBnfhrPbmc/bnl2JtTDpPF2nARYTRbYbRYYTI3WWZfb4XJLMLQbP1lbRofb3Gfr42CAPiqbSHMFs5socxXY7vtb7/tAV/PS4HNr0l48/X0gMbDOc4DaLWKMJitqDdZYDBbYTBbUG+yXduXm6wttjFbrFB7KOCpUsJLpYRGdem2p0oJT5XCflvT5La7/Kh1o9mA4yqIiMhlFerqsPjXXHyzNw96gxkAEOSnwf+MuANjkyIkro7IMZI7BCDZQZPEiKIIk0W0h7PGMNYY2AxNgp19vfnSfasowioCVlGEKIqwWC/dblxusYoQG243LrNam9y2X65cJ4q2E6E3rhcb2lqsl25bRcDS8JpGswi9wYTqejP0BjOq682orjfBZLE9V7XBjGqDGdDd+r+Z2kPRJKQ1D2/2sNYQ7Hw1HvBWK2E0txyADGZbOKo3Xz0kNd42XNZeitCsVAgNIUwBjYcSXmrbbc+G2xqPhvv2EKdoCHJXBrpm6xoe39ZbhTbe6lbfrlvF4EVERC7nZFEVFqRn4/vDF2C22r5sdAr2xZT+sXggKdxpfoEmkhtBEKD2EKD2cN2eDFG09Qw1hrCmgcx2bW4Iak3uGxra1l9qW2O0zTBpNFtRpjeiTG+UeMsu8VAI0HgooFEp4dlwrWl63dC7ZbuthEopwGi2Bbh6kxV1Rov9dr3J0uRiRZ3p0syaFqsIvcEMvcEx2/FY7yj8c2yiY57cARi8iIjIJYiiiN1ZFzE/PRtpp0vty++OCcAzA2Lxh87BPF6DiK5LEAR7z0qQn+aWn6cxdNxweGtYX2eyQK1sEnwaeog0Db1GjdeNvUjNQ1LTx129jYcDhwA2BldDQwirN9lCWp2xIaiZLahvEtyuFuLqWgh1tuUNPXsmi+wmRZJXtUS34KLegEW/5qBIZ8AzA2MRH8ITTBK5ErPFik2ZRViQnoXM81UAAIUA3JcQiikDOqJ7ZBuJKyQid6RUCNB6qaD1UkldSqtqGly1cK9tvx4GL3JZZXoDvkjPxle7z9m7vdceKsDYpAi8MjQOEW29Ja6QiH6PWqMZKzPysXBnDgoq6gAAnioFxvWMxNP9YxDdzkfiComIiC5xqQG6s2bNgtAwpWjjJTQ01L5eFEXMmjUL4eHh8PLywqBBg3D8+HEJKyZHKK024J8bf0P/d3/B/PRs1JksSGyvxdA7Q2AVgdUHC3Dv+2mYvcF2vhMikpfSagPe//EUUuZsw+wNv6Ggog4BPmpMGxKHXW8Mxj/GJDB0ERGR03G5Hq+77roLP/30k/2+UnnpAOp//etf+PDDD7FkyRLEx8fj7bffxtChQ3Hq1Cn4+XH4mdyVVNVjfno2lu89h3qTFQDQLUKLl4fE4Q+dgyEIAg7lVeBfm09hd/ZFLP41F6sy8vF0/1g83T8Gfp7sDidyZlmleny5IxurD56H0Wz7Px7dzhtP94/Fwz0i4KXmhBlEROS8XOo8XrNmzcK6detw+PDhK9aJoojw8HBMmzYNr7/+OgDAYDAgJCQE7777Lp555pkbfh2ex8u5FFfV4/PtWVixLw+Ghi9j3SPb4OUhcRgUH3TFOXpEUcTOs2X41+ZTOHbeNj9sgI8aL/yhEx7rHQVPFb+8ETmT/bnlmJ+ejZ9OFKPxL1b3yDZ4ZkAsUu8KhZITZhARkYTc9jxeZ86cQXh4ODQaDXr37o133nkHsbGxyMnJQVFREVJTU+1tNRoNBg4ciF27dl0zeBkMBhgMl4akVVVVOXQb6MYU6erx+fazWJGRb//1u0dUG7w8JB4D4gKvelJUQRDQPy4I93QKxA+ZRXj/x1PILqvBP/77GxbuyMa0ofF4MKm9Q2f8IaJrs1pFbPmtGAvSs3Awr9K+fEiXYEwZ0BG9OrTliY+JiEhWXCp49e7dG1999RXi4+NRXFyMt99+G3379sXx48dRVFQEAAgJCWn2mJCQEJw7d+6azztnzhzMnj3bYXX/Hk8tyUBskA9GJIahe2Qbt/gicqGyDp9vz8LKjHwYLbbAlRzdFi8PicM9na4euC4nCAJGJIYh9c4Q/OdAAT7+6Qwu6Oox4z9HsSA9G6+ldsawu0Lc4t+UyFnUmyxYc/A8vtyRjeyyGgCAWqnA2KT2mDwgBp2COSyciIjkyaWGGl6upqYGHTt2xIwZM9CnTx/069cPFy5cQFhYmL3N5MmTkZ+fj82bN1/1eVrq8YqMjJR8qGFWqR6DP0iz32/fxgvDE0IxomsYklwwhJ2vrMNnv5zFd/sL7IHr7g4BeHlIHPp2bPe7t7feZMFXu3Px6S9Z0NWZAADdItvg9fs6o2/HwN9bPrmAGoMZ5yvrUFBRizK9EZFtvREf4ot2vrd+nheyqagx4us957B0d679JKP+nh74U59oTOrbAcH+ntIWSEREdBU3OtTQpYMXAAwdOhSdOnXCX/7yF3Ts2BEHDx5EUlKSff0DDzyANm3aYOnSpTf8nM5yjFe9yYLtp0rw36OF2HayBLXGS2cKD9d6YnhiGEYk2kKYnE8aml9ei8+2Z+E/B/Jhsth2194xtsCVEvv7A9fldHUmfJGejYU7c+zT0PePC8SMYXcgMUJ7W1+LnEud0YLzlbXIL7eFq4KKOhRU1CG/4XZ5jbHFx7XzUaNTsC/iQ/wQH+KLuBA/xIf4IcBH3cpbID/55bVYuDMHKzPy7f/f2rfxwpP3xGBCr0jZnRyTiIjcD4MXbD1VHTt2xJQpU/Dmm28iPDwcr7zyCmbMmAEAMBqNCA4OdonJNRpD2MZjRfj5RHGzEBam9cTwhDCM7BqKpMi2sglh+eW1+PSXs/jPgQKYrbbdNCW2HV4eEoc+se0c/vol1fWYu+0sVuzLswe+kV3D8OrQeMQG+Tr89en2qzdZGsLUlaHqfEMv1vX4e3ogMsAbAT5qnLtYi/yKWlztUzTQV4244OZhLD7EF228GciOFegwPz0Lm44VouG/N7qE+eOZAbEY2TUMKh5jSUREMuGWweu1117D6NGjERUVhZKSErz99ttIS0vDsWPHEB0djXfffRdz5szB4sWLERcXh3feeQfbt2+/6enknTF4NWULYaXYdKwQP58oRs1lIey+hFCMTAxDjyjnDGF5F2sx95czWHPwvD1w9evUDi8PjsfdMQGS1PPRT6ex7vB5iKLtTPTjkyPx8uA4hGo5/MmZ1JssuFDZPFA1DVql1dc/b5ufxgMRAd6IaOuFiLZeiGzbeNsb7dt6QevV/LQDdUYLzpbocbq4GmdK9DhTXI3TJdXIL6+76msE+WkQ19BDFhfS0FMW7Aett2uf0kAURWw/XYoFadnYnX3Rvrx/XCCmDIi9qWM0iYiInIVbBq9HHnkE6enpKCsrQ1BQEPr06YN//OMfuPPOOwHY/ujPnj0b8+fPR0VFBXr37o1PP/0UCQkJN/U6zh68mqo3WZB22hbCfvqteQgL9beFsFFdnSOE5ZbVYO4vZ7H20HlYGgJX/7hAvDw4DskdWj9wXe5EYRXe//EUfj5ZAgDQeCgwqW8HPDeoI3swWonBbMGFyvomPVa26/xy23XJDQQrH7USkQGXwlTT68gA7yuC1a2qNZobAllDGCuuxuliPc5XXj2QBftpmoexEF90Cva7bTVJxWi24vsjF/BFejZOFVcDsP2AMbprGCYPiMVd4RzCS0RE8uWWwau1yCl4NVVvsiC9MYSdKIHeYLavawxhI7uGoWcrh7Ccshp8su0M1h++YA9cA+KD8PLgOPSMbttqddyojNxy/GvzSWTkVgAA/Dw98OzAjniiXwd4q3k8SktEUYTBbIXeYEatwYIaoxm1RjNqDBbUGs3QN1zXXHatN5hRa7RAV2fC+Yo6FFfXX3VYXyNvtbJJL1XzUBXR0GMlZa9KjcGMMw09ZPaesusEshD/hkDWZNhiXIgv/CU46bfZYkW92QqDyQKD2Yr661wX6erxzd48FFXVA7AF30fujsKT98SgfRuvVq+fiIjodmPwciC5Bq+m6k0W7DhTZu8Jq24SwkL8NRieYJuYIznacSEsq1SPudvOYv3h8/ZjPAZ1tgWupCjnC1xNiaKI7adK8e7mkzhZZPsFP9BXg5cHd8KEXlFQe8j3+BRRFFFvstrCUZOQpDdYUGswo8bYQjiyt7OgxnDpuulzWG/TJ42XSnnVUBXR1httvaUNVrdKbzDjTEMIO11cjdMNwxYLdfVXfUyY1tN27FjDsMXYIB8IgmAPRQazBfWmG7m+FJQMJkuzYNX0fr3Zav9x5GYF+WnwRL8OeKx3tOx78IiIiJpi8HIgVwheTRnMFuw4bQthWy8LYcF+GtsU9YlhSO4QAOVtCGFnS6rxybaz2HDkgv3L+L13BOOlwXHoHtnmdz9/a7JaRWw4egEfbDmNvPJaAEBUgDdeTY3H6K7hkg/fvBZdrQlnS229LmdL9MgqrcHZElvPy61+ub4RXiolfDQe8NEo4a32gI9aCW+N7dpH0/y+t9rWzs9ThfA2trDVzkcty2B1q6rqTThjH66ox5kSWzhr7EGSklqpgMZDAY1KCY2HAp4qBTQeSmhUCng2XHurlRgUH4wHksKh8VBKXTIREdFtx+DlQK4WvJoymC3YeaYMGxtDWP2lEBbUJIT1uoUQdrq4Gv/++Qw2Hiu0Dxcb0sUWuLpGyCtwXc5otmJlRh7+7+ezKNPbjjO6I9QPM+7rjD90DpYsKIiiiOIqQ0O4qsbZUn3D7Rp7ndfi3ST8eKs94Ktpfr8xJPlqPOCtVsJH7QFvTcN1Y5BqEqa8VMrbEt7JdtqDsyW2MNY4XDH3Yg0UgtAQgpQtXmtaCEeXX2s8lPYQdbVrtYeC7yUREREYvBzKlYNXUwazBb+eLcPGo0XY8lvRFSHsvrtsIezumGuHsFNFtsC1KfNS4Bp6ZwheHhyHhPaudVB9rdGMxb/mYt72LHvPYa8ObTHjvjvQy4EThJgtVuRX1Nl7r86W6HG2VI/sEn2zHszLhfp7olOwLzoF+6JjsC86BfmiQ6A3/D1V8FIpnbrHjoiIiMgZMHg5kLsEr6aMZqsthB0rxJbjRahqEsICfTW4LyEEIxLD0DumnT2EnSiswr9/PoMfMovsbYfdFYKXBse5/CxmlbVGfJ6WhSW/5sJgtgIABt8RjNeGdUaXsFvfZ+pNFmQ19FplNYSrsyV65JbVwmixtvgYpUJAdIC3LVg1hKvGoMWT0xIRERH9PgxeDuSOwaspo9mKX7PKsOloIbb8Vgxdncm+LtBXjWF3heKi3ojNxy8FrhGJoZh6b9zvCh1yVKSrx//9fAar9ufDYhUhCMCY7u3xypB4RLXzvurjLj/+qrEHq6Ci7qqz+nmqFIgN9LX3YDVeott589gaIiIiIgdh8HIgdw9eTRnNVuzKsk3M8ePx5iFMEIARiWF46d44dA698RNUu6LsUj0+2HoaG48WAgBUSgF/vDsKf+oTjZIqw00df9XGW4VOQb7oGNQ8YLVv48WhgUREREStjMHLgRi8WmayWLEr6yI2ZxZBEIBJfTsgPsS9A9fljhXo8K8fT2LHmbLrtg3T2o6/ujxgudusfkRERETOjMHLgRi86PfalVWG9388haMFOkTx+CsiIiIi2brRbMBvdkQS6NsxEGueD4Qoiuy9IiIiInIDCqkLIHJnDF1ERERE7oHBi4iIiIiIyMEYvIiIiIiIiByMwYuIiIiIiMjBGLyIiIiIiIgcjMGLiIiIiIjIwRi8iIiIiIiIHIzBi4iIiIiIyMEYvIiIiIiIiByMwYuIiIiIiMjBGLyIiIiIiIgcjMGLiIiIiIjIwRi8iIiIiIiIHIzBi4iIiIiIyMEYvIiIiIiIiByMwYuIiIiIiMjBGLyIiIiIiIgcjMGLiIiIiIjIwRi8iIiIiIiIHIzBi4iIiIiIyMHcNnh99tlniImJgaenJ3r27IkdO3ZIXRIREREREbkotwxeK1euxLRp0/DXv/4Vhw4dQv/+/TF8+HDk5eVJXRoREREREbkgQRRFUeoiWlvv3r3Ro0cPfP755/ZlXbp0wZgxYzBnzpzrPr6qqgparRY6nQ7+/v6OLJWIiIiIiJzYjWYDj1asySkYjUYcOHAAb7zxRrPlqamp2LVrV4uPMRgMMBgM9vs6nQ6A7R+ZiIiIiIjcV2MmuF5/ltsFr7KyMlgsFoSEhDRbHhISgqKiohYfM2fOHMyePfuK5ZGRkQ6pkYiIiIiI5KW6uhparfaq690ueDUSBKHZfVEUr1jWaObMmZg+fbr9vtVqRXl5Odq1a9fiY6qqqhAZGYn8/HwORZQhV3v/XG173I1c3j+51ElXcsX3zhW3yZ3I+f2Tc+106++fKIqorq5GeHj4Ndu5XfAKDAyEUqm8onerpKTkil6wRhqNBhqNptmyNm3aXPe1/P39+Z9Oxlzt/XO17XE3cnn/5FInXckV3ztX3CZ3Iuf3T8610629f9fq6WrkdrMaqtVq9OzZE1u3bm22fOvWrejbt69EVRERERERkStzux4vAJg+fTomTpyI5ORkpKSkYMGCBcjLy8Ozzz4rdWlEREREROSClLNmzZoldRGtLSEhAe3atcM777yD999/H3V1dVi2bBm6det2215DqVRi0KBB8PBwy2wre672/rna9rgbubx/cqmTruSK750rbpM7kfP7J+faybHvn1uex4uIiIiIiKg1ud0xXkRERERERK2NwYuIiIiIiMjBGLyIiIiIiIgcjMGLiIiIiIjIwRi8iIiIiIiIHIzBi4hIApWVlcjOzkZVVRUAgBPMkiNwPyNnwv2R3B2D103Q6/XQ6XQA+GEhR+Xl5Th27BiKi4ulLuW2qaurg8FgkLoMuklvvPEGkpKS8NBDD6Fnz57YsWMHBEGQuqwWcR+TLzntZzeD+6Q8yX1/5HdAeXOW74AMXjdo1qxZSEhIwNq1awFAVh8WZPvAT0hIwOOPP46EhASsWrUKdXV1Upf1u7z55ptITk7G3r17pS6FbtDp06cxdOhQbN26FYsWLcJ7772Hrl274qmnnrL/AuxMuI/Jk9z2s5vBfVJ+XGF/5HdAeXOm74A8pfZ1lJeXY8aMGTh06BAAYNOmTejXrx/i4uIgiiL/8zm53NxcTJ06FYWFhfj222+h1Wrx+eef4/XXX0eXLl2QmJgodYk3raioCDNmzEBmZiZyc3OxZMkSdOvWDVqtVurS6Dq2b98OQRCwevVqdOjQAQCQkpKCwMBAHDt2DP369ZO2wAbcx+RNLvvZzeA+KV9y3h/5HVDenPE7IINXC5r+ZzKbzQgLC8PYsWPh5eWFiRMn4scff0SHDh2gUqkkrpSuZ//+/RAEAUuWLEFCQgIAYN68edBqtcjJyUFiYqLsPjx1Oh2CgoLw8ccfo6qqCg888AAefPBBjBo1SurS6DJWqxUKxaWBBSNGjEBYWJj9ywdg+0LZvn17p/o84T4mL3Ldz24G90n5kPv+yO+ArsMZvwMyeF3GYDDAarXCy8sLANC2bVtMnToVwcHBAIDU1FSsWLECvXv3Rq9evaQslVpgNpuhVCrt/4n69esHf39/+384wHZwb1RUlL2Ns4cuk8kEQRDg4WH77xodHY2XX34ZUVFRAIB7770X7777Lnr06IHw8HApS6Um3nvvPRw6dAgBAQGYNGkSunbtioiICERERAAALBYLlEolSkpKUFVVhcjISMlq5T4mX3Laz24G90l5kvv+yO+A8iaH74A8xquJWbNmISUlBaNGjcIHH3yAsrIyqFQqBAcHw2q1AgDefvttnD9/HuvWrUNlZSUAHmTpLObMmYOxY8fi0Ucfxfr166HX6xEWFobU1FQAsL+HJSUlyMvLQ1xcnJTl3pC///3vGDx4MMaNG4f//Oc/0Ol08PT0RFRUlH17FixYgF9//RXr16+HyWSSuGLatWsXEhMT8fXXX6NTp07Yvn07pkyZgo0bNzZr1/hhv337diQmJiIsLEySzxLuY/Ikt/3sZnCflB9X2B/5HVDe5PIdkMELtl9gJk2ahOXLl+O1115DbGwsli5dinHjxtnbKBQKWCwWtG/fHk8//TTWrFmDPXv2ALB9kPA/nnT27duHpKQkfPPNNxgyZAgKCwsxe/ZsfPnll83aNX7g79y5E506dcIdd9zhtO9bXV0dHnroIXz99dd47LHHYDKZMHv2bDz33HP2No37ZExMDF544QW8++67yMrKkrBqysvLw9y5c3Hvvfdiz549+Pvf/47MzEwIgoAjR44AuPRHunEozq5duzBkyBAAtn300KFDOHnypMNr5T4mX3Laz24G90l5kvv+yO+A8ia774AiidnZ2eIdd9whrlmzxr4sIyND9Pf3F9966y3RYrGIoijar61Wq9itWzfxqaeeErOzs8V169aJn3zyiSS1u7vS0lJxypQp4uTJk8Xq6mr78gkTJohTpkwRjUbjFY955plnUNq4KgAAGbRJREFUxGnTptnv7927V0xPT2+Vem/UoUOHxLi4OHHnzp32ZatWrRI1Go345Zdf2pc13Se1Wq34+uuvixUVFeKGDRvElStXtnrd7q6goECcOnWqePz4cVEURbG+vl4URVH885//LI4ePfqK9oWFhWJMTIy4d+9eMTs7Wxw+fLioUCjE7du3O7xW7mPyJaf97GZwn5Qnue+P/A4oX3L8DsjgJYriqVOnREEQxNzcXFEUbf+pRFEUP/30U9HHx0c8evSova3ZbBb/f3v3HhVVtccB/DfDS0qBRBBvKiJIijBLKVRQidQUH7erouEDn7dIEzVKr1Sa0FJL80ayXHXhYksvapqRucT0SgS+46FcM0ERcaGiIJpE4Cgg3/sHzZERH4zNa5/5ff5peWbG9l6/r3v2PrPPOUDTl4GLiwu6du0Ka2trJCQkGL/hDJWVlYiNjcXx48cBQPpHtnjxYvTr16/F+2tqauDr64u0tDSUlpZKA/53331n1HY/zuHDh2FjY4PKykrp2J07dxATE4MOHTqguroaQFNWNZlMTEyEra0tevXqBWtra/znP/8xSdst3YMG+uHDh2PNmjUtjmdkZMDNzQ1vvPEGbGxsMG7cOK2aGxJnTGyi5EwXnElxiZxHngOKS8Q5IG81pKafmVUqFW3fvp2I7v0c+eabb5K7uzutX79eep+VlRWVlpbSjz/+SNevX6ehQ4dSRUUFzZ8/32Ttt2QdOnSg999/n/z9/YmIpAuxr1+//sBb1J45c4bKyspoy5Yt1KNHD7Kzs6OKigr629/+ZtR2P87t27epV69elJ6eLh2ztbWlN998k+zs7Gjt2rVE1LRnWZPJkydPUn19PQ0YMIDKy8tp2rRppmq+rOERWxMAkI2NjdZ77ty5Q5WVldSnT58W7z948CBVVFRQQUEBZWVl0bfffksdOnQwSLvvxxkTl0g50wVnUkyi55HngOIScg5otCWeGaupqUF4eDjGjx+PCxcuAADq6+sBAElJSXB2dsatW7ek90dHR8PNzQ05OTmmaC5rRnNmSvNfjUGDBklnPpu/lpycDIVCgcGDB+Po0aPGa+h97m/v/aqqqtC/f39ERUVpnQmsq6vD0qVL4efnB7VaDQCora3FzJkz8eyzz3ImDay6uhp37tyR/qzZegLcGzM0NGdG8/Ly4OjoiNLSUum1iooKAMClS5ewdetWQzb5oThj5uvGjRsoLy+XsqbJEiBeznTBmTRPFy9exN69ex/4mhzyyHNAcYk4B5T9wkszMWr+xaXRfMD46quvoFKp8PHHH2u9Z9u2bfD29sbZs2elY833kTLDam39mr/nwoULcHBw0NoeUFZWBqBpb/nOnTsN1dxWae3kfd26dfD09GxxzcKaNWsQEBCAGzduSMeuXr1qwBazxsZGvPXWWwgMDMTQoUOxbNky6Yu4+RabxsZG7Nq1S+uzy5cvR0hICICmayHCw8MxYcIEVFVVGay9rZ24c8bMS2NjI+bPnw9vb2/069cPQ4YMwc2bNwGYZ8500drJO2fSvBQVFUGhUMDe3h5FRUXS8eZjijnnsbi4GIsXL8apU6davMZzQPPX2voB4swBZb3V8O2336aIiAgiIrKyspKO44+fw62trenu3bu0ZcsWmjRpEgUFBdHOnTspLS1Nem9ZWRm1b9+ePDw8pGNt27Y1Ug8sW2vqB4A2bdqk9Z69e/dS9+7dyc/Pj8rKyig8PJzGjh1LN27cIDc3Nxo7dqyRe3Kv3dHR0TRixAgaNWoUffDBB6RWq0mpVEq3Q9Zkct++fbRgwQLy9PSkjRs30rFjx6S/p6qqihwdHcnJyUk65ubmZvT+WIqDBw+Sr68v/fTTT9Idr1JTU+ndd98lIpIeopmUlESurq70zTffSLcZJiIqKSmhUaNG0UcffUQ9evSg8vJySkhIIEdHR723FQAtWLCAAgMD6ZVXXqGRI0dSVVUVWVlZccbM3J49e8jHx4fy8vJo/fr1FBkZSeXl5dIWJnPKma7OnTtH7u7uNH78eDp37px0/O7du0TEmTRn9fX1NGLECHJ2dqaVK1dKxzXft+aaRwA0d+5c6tGjB928eZM8PT21XiPiOaA5a239RJoDSky14jOkEydOYNiwYXBxcYFSqcS+ffsAtFwdJyUlwdXVFcOHD0ddXR0KCwsxe/ZsWFlZ4fXXX8e8efPg4OCAf/7zn2hsbHzs9jCmH7rWb9SoUbh27Zp0PDo6GgsXLsSqVatgb2+PIUOG4PLly0btw/0OHDgAHx8fDBgwAKmpqXj99dfh4+ODhQsXar0vMTERHTp0wKRJkwAAx44dw+jRo9G+fXt88MEHeOutt+Dk5IQvv/zSFN2wOLW1tYiKisLf//536ReuhoYGrF69GiEhIdLZ24SEBLRp0wZffvml1png8vJytGvXDgqFAp6enti/f7/B2pqWloaePXsiMDAQ+/fvR3JyMnx8fBAREaH1Ps6YeVq0aBEWLVqk9Wv4vHnzEBkZKf05Pj7e5Dl7EqdPn0ZoaCg6d+6MGTNmtHidM2m+tm7dildeeQXHjh2DUqlEZmam9Npnn31mlnncunUr2rdvD39/f+Tm5mq91nwex3NA86Rr/USYAzYny4VXYmIiZsyYgT179iAiIgK+vr4t3pOSkoIuXbpgw4YNLSb0a9euxRtvvIHQ0FBkZGQYq9nsD7rWr/mAf+vWLXTr1g0KhQLe3t5mMQF5ksl780zeuHEDS5YsQXh4OIKDgzmTBtZ8YK+qqkJKSoq0l1+zLXTlypV44YUXtG4vrNkS1lxZWRlCQkKwefNmg7db14k7Z8y07p/EXbt2Tbq+BGiavAYEBGDFihU4dOiQ9JkHbdUyZs6eRGsn75xJ02mex+bb33fs2IH58+cDAF5++WUEBwcDuHciVHOnyeZMnccRI0agW7duuHLlCgDg559/RlpaGs6cOYOamhoAwMaNG3kOaKZ0rZ+5zwHvJ4uF1/1fYOXl5dLezszMTHTq1AmffvopAO198vcPGHw2wzT0VT+g6ct62rRp2LJliwFb/Hj6nLxr3P/lwPRv5cqVmDdvHuLj46UBvjnNAL9kyRJMmDABwMPHjYdd9Ksv+py4a3DGjONxOUtOToa1tTUGDBiAkJAQODs745133kFtbW2L9xo6Z7rQ5+RdgzNpeI/K47Jly6RfKSsrK2FjY4PQ0FD4+/sjPz+/xd9lijze//86efIkunfvjqVLlyIsLAzdunWDSqWCq6ur1nPFeA5oHvRVP8B85oCPIvzC63FfYDdv3kRMTAxcXV2lIj3oRg3MNPRZP3MZNPU5eWfGcfbsWfj6+sLPzw8zZsyAs7MzgoKCpLNl9z9A86WXXpJOBpiidvqcuDPjeVzONFJSUpCRkSFla/fu3bC2tkZhYSEA8xwv9Dl5Z8bRmjzOnj1berDwli1b0LZtW1hZWZnN89IelruoqCjY2tpiypQpyM3NxS+//ILvv/8eDg4OeOedd0zYYtacPutnjuPigwi78HrcgNG8APn5+fD19ZW23DQ/E8dMQ471E23yzu6Jj49H//79pW16N2/eRHBwMEaOHImCggIA9+pWUVEBV1dX5OXlSZ8vLi4GYPiTOnKeuFuC1uQMaFmf0tJS2NraIjU11ajtbQ05TN4t1aPyqNl18tprr2HSpEkYPHgwnnnmGcTGxsLV1RWxsbGmbPpDc6e5c+Zvv/2G999/X2sXANB0XVDbtm0feMKKGY8l10/Yuxp+//339PTTT1NeXh5t3LiRiouLydramuLj46mwsJAUCgU1NDQQEVGvXr1ozpw59NVXX1FBQQEplUrKysrSugMPMy451u9xfVIqldTY2EhKpZKuXbtGp0+fpuDgYCJqemDj+fPniejenb6YcTQ0NNDPP/9MLi4u0kNAnZycKCYmhmpqaqSHZyqVTcNleno6OTs70/PPP08FBQUUEhJCKpWK1Gq11t03DeFxGdOYOnUqDRkyRHoQqEqlIqVSSQUFBUR07wGhzHhamzOilvXZtWsXBQUF0dChQ43d7Md6VCZPnTpFRE3/dr7++msKDg6mqKgoWrRoETk7O1NJSYmJW2+5HpfHxMREIiJSq9W0d+9eeu655yg/P5+WL19Oy5cvp7i4ODpz5ozJ2v+w3CUkJNAvv/xCDg4O9I9//IO6deum9blOnTqRlZUVnT592jQNZ0Rk2fUTcuHV2i8wza0m7ezsaNSoUTRo0CCaOnUqDRo0iEaPHk0VFRUm7ollkmP9RJq8M23W1tZUV1dHd+7cobt371JjYyMREY0cOZKCg4MpPz+fjh49Kr2/oKCAvLy8KCYmhlQqFXXu3JmuXr1K9vb2Bm2nXCfulkLXnF26dIkuXLhA8+fPp1WrVtGrr75Kjo6O0q2UzYHok3dL9rg85ubmUmFhIcXGxlJWVhYlJSWRu7s7ERHNmTOHVq9eTd27dzdJ2x+Xuy+++IKIiBwcHFp89sCBAxQUFEQBAQHGbjb7g6XXT8iFly5fYJovqYaGBvr111/p5MmT1LNnTyovL6fnnnvOZH2wZHKsnyiTd6ZN8+via6+9RhkZGXTq1CmysrKSjv/1r3+lmpoarbNru3fvprS0NDp06BDl5OTQ5s2bH/gFoW9ynLhbCl1zdv78efrXv/5FISEhdOLECfrvf/9Lc+fOJSLz+rVS5Mm7JWtNHmtra+nIkSPk5eVFffr00cqdUqmkxYsXk62trUnar+tYePHiRbpw4QJFRUXRtm3baPr06aRQKHgsNBGLr5+x9zb+WZprKDIzM6FUKnHixAmt4z/99BP8/PyQlJQkfSY3Nxfe3t7o06cPTp8+bfxGM4kc6/ckffLz84NCoUBQUBCOHz9u/EZbkPtvKNH8+hnNHdPUajVefPFFDBs2rMV7fH19ERMTA6DpLkpr167F7t27Dd1sLbpmrLi4GO+99x66du2KoKAgnDx50qjttUT6zJlarcaRI0dw4MABQzf7ibUmk71798a///1vk7XRkukjj++++26L46am61h47tw5vP3223Bzc0NgYKB07RozDa6fmd5cQ58DBgBcv34dhw8fNmSTWTNyrJ8cJu+Wpq6uDnPmzMHQoUMRHh6O9PR0qSbNn3XV0NCA8vJyZGVlwcbGBp9//rl0I42amhoEBATgk08+MXh7LW3iLhei5UwXcp28y5kc8qjPsbC2thaZmZn48ccfDd1s9geu36OZ1cJLDgOGJZNj/eTYJ0tw9epV+Pv7Y+DAgUhOTkZQUBB69+6NpUuXar1v3bp1sLW1xcaNGwEAK1asgIuLC2bNmoXMzExER0eja9euBj3LxhkTl0g50wVnUkyi55FzJzauX+uYzcJL9AHD0smxfnLsk6VITU1Fr169UFpaCqDprFl8fDwUCgWOHj2K+vp6TJ48GX/5y1+wadMmrbNtCQkJGDx4MFQqFfr27Yvs7GyDtZMzJjZRcqYLzqS4RM4j505sXL/WM5uFl8gDBpNn/eTYJ7nTnDX74osv0LFjR63XqqurMXbsWPTr1w93795FdnY2fvvttxafBZq2Pdz//BBD4IyJSbSc6YIzKR455JFzJzauX+uZfOElhwHDksmxfnLsk5zt2LED6enpuHLlinQsKSkJffr0wcGDB7Xem5OTA2tra+zatQuA4R94/DCcMfGImDNdcCbFIpc8cu7ExvXTnVVsbGysse+k+M0339CVK1fI3t5eug1zfn4+nTlzhnr37i3dbtbOzo48PT1pxYoV1LdvXxoyZAhZW1tLz0JqfntThUJBTk5Oxu6KRZJj/eTYJ7lLSUmh4cOHU0FBAW3YsIEyMzPJ0dGRfHx8SKlU0o4dO+iZZ56hfv36Sc9Gc3BwoOLiYsrOzqbJkydLdTMGzpiYRMuZLjiT4pFDHjl3YuP6/TlG/deXkpJCHTt2pE8++YSmTJlCr776KqWmphIR0QsvvEB1dXV05MgRqqurkz7Ts2dPCgsLo+TkZCIifrisCcmxfnLsk9w1NDTQunXr6KOPPqJVq1bRoUOH6LvvvqPu3bvThg0bSK1WU9++fWnQoEG0c+dOreeBtGvXjp566ilq06YN1dfXG6W9nDExiZYzXXAmxSOHPHLuxMb10w+jLLzkMGBYMjnWT459shS1tbVUWVlJ06dPp1mzZpGtrS0FBQWRj48PVVdXSzWJi4uj+vp6SkxMpLKyMunzVVVV5OLiQjY2NgZtJ2dMbKLkTBecSXGJnEfOndi4fvpllIWXyAMGk2f95NgnOSssLJRq4ujoSBMnTqTFixeTUqmUnnrftWtXunXrFrVp04aIiNzc3Oi9996jS5cuUWBgIK1Zs4amTZtGhw8fpgkTJhi8zZwx8YiYM11wJsUilzxy7sTG9dMzQ108VlBQgLq6OunP//vf/6QLOjUX1G3duhV9+/bVur//jh07MHDgQHTp0gWrV69GREQEXFxcZPXwNBHIsX5y7JPcff3113B3d4e3tzdUKhUSEhK0LgxvfnFuREQEZs2aBUD7mSGXL19GZGQkxo0bh9GjR+PMmTMGay9nTEyi5UwXnEnxyCGPnDuxcf0MR+8LLzkMGJZMjvWTY58swQ8//AAPDw+sX78eWVlZWL58Oezs7LBmzRr8/vvvAJruhNTY2Ai1Wg2VSoWUlJSH/n1qtdpgbeWMiUuknOmCMykm0fPIuRMb18/w9LrwEn3AsHRyrJ8c+yR3mud7fPjhh+jTp4/WgB4TE4OAgADp4YsaZWVl6NatG4qKigAARUVFiI6ONkp7OWNiEi1nuuBMikcOeeTciY3rZxx6WXjJYcCwZHKsnxz7ZGnCw8MxYcIEAJC2PFRVVWH06NEICwuTHtQIAJs2bcLgwYPx+++/Y8GCBbC2tsb48eNRV1en9aBGfeKMyYO550wXnEnxiZhHzp3YuH7GpZeba2juxX/69Gny8vIiW1tb6WK7mJgYcnV1pd27d9PFixelz/zwww/UpUsX6tSpEy1cuJB8fHyotLSU6uvrCYA+msVaSY71k2Of5Co9PZ0WLFhA69ato5ycHOn40KFDad++fdTQ0EA2NjZUX19Pjo6ONHnyZCoqKqITJ04QEREASktLo1OnTpG7uztlZGTQsWPHKDU1lWxsbLSeFaJPnDGxiJozXXAmxSGnPHLuxMb1M7InWa3t378f8+fPx2effYbs7GzpeFJSEtq2bYv6+noA987WbN68GX5+fti5cyeAptX1xIkT4eTkhPbt26N3797Izc19sqUj05kc6yfHPsndlStXMGbMGLi6umLq1Knw8/ODo6OjVL+zZ8/i2WefxbJlywBo7yH39PTEhx9+CACora3FmDFj0LlzZ2zfvt1g7eWMiUm0nOmCMykeOeSRcyc2rp9p6bTwksOAYcnkWD859skS1NbWYsaMGQgPD0dJSYl0PCAgADNnzgQAVFdXY8WKFbC3t8fFixcB3LuwNzQ0FJMnT5Y+l5eXZ7C2csbEJVLOdMGZFJPoeeTciY3rZx5avfASfcCwdHKsnxz7ZEkiIyOxd+9eAJDOsMXFxaF///7SnvOSkhIMHDgQ/fv3l2p86dIl+Pr64ttvvzV4Gzlj4hMhZ7rgTIpN1Dxy7sTG9TMfrb7G66mnniI7OzuaOXMmeXh4UENDAxERjRkzhgoLCwkAtWvXjqZMmUL+/v40ceJEunDhAimVSrp8+TJdvnyZJk6cKP19zz//vP73TbKHkmP95NgnS7J+/XoKDQ0lIiIrKysiIjp37hypVCppz7mHhwdt376dfv31V3rxxRcpLCyMBgwYQM7OzhQYGGjwNnLGxCdCznTBmRSbqHnk3ImN62c+FEDrr4Krr6+XnjwNgBQKBU2bNo3s7e0pKSlJel9ZWRm99NJLdPv2bQoICKDs7Gzy8vKibdu2kZubm/57wVpFjvWTY58sWXBwMM2ePZtmzpxJjY2NRESkVCqpuLiYjh8/Trm5uaRSqWj69OlGaxNnTH7MMWe64EzKiyh55NyJjetnHnRaeD2IKAMGezA51k+OfbIEJSUlFBQURHv27JHOptXV1ZGtra2JW9YSZ0xcIuVMF5xJMYmeR86d2Lh+JvBn9imeP38eHTt21Nrr2fxiPGbe5Fg/OfZJ7jTXNWzatAmenp7S8djYWMyZMwcVFRWmatoDccbEJFrOdMGZFI8c8si5ExvXzzSe6Dle+ONHssOHD1Pbtm2lszRxcXG0cOFCunbtmv5Whkzv5Fg/OfbJUmiua8jJyaGwsDBKT08nDw8P+vzzz2ncuHHk6upq4hY24YyJTZSc6YIzKS6R88i5ExvXz7T+1FbDqKgoevrpp2nYsGEUGRlJt27dopSUFBo+fLg+28gMRI71k2OfLMHt27fJz8+Pzp8/T7a2thQXF0dLliwxdbMeiDMmLpFypgvOpJhEzyPnTmxcPxN50p/K1Go1vLy8oFAoYGdnh48//lgPP8AxY5Fj/eTYJ0sybNgwzJ07F2q12tRNeSjOmPhEyJkuOJNiEzWPnDuxcf1M50/94vXyyy9Tjx496NNPP6U2bdrocz3IjECO9ZNjnyzF3bt3pdsrmzPOmNhEyZkuOJPiEjmPnDuxcf1M408tvEQeMJg86yfHPjHzwhlj5oYzyUyBcyc2rp9p/OnbyTPGGGOMMcYYe7QnuqshY4wxxhhjjLHW44UXY4wxxhhjjBkYL7wYY4wxxhhjzMB44cUYY4wxxhhjBsYLL8YYY4wxxhgzMF54McYYY4wxxpiB8cKLMcYYY4wxxgyMF16MMcYYY4wxZmC88GKMMcYYY4wxA+OFF2OMMcYYY4wZ2P8Bq0babPIXzOAAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 1000x300 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "metric = 'mae'\n",
    "df_regression_metrics[metric].plot(ylim=(0,250), figsize=(10,3), title=metric.upper())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 同じ期間のGround Truthと予測結果を比較する\n",
    "※ 「B1 オプションB」を選択した場合は推論結果がローカルに保存されていないため、実行できません。スキップしてください"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import glob\n",
    "\n",
    "values = {}\n",
    "prediction_result_files = glob.glob(f'./{result_dir}/*.csv')\n",
    "for file in prediction_result_files:\n",
    "    df_pred = pd.read_csv(file)\n",
    "    pred_date = pd.to_datetime('-'.join(file.split('.')[-2].split('-')[-3:]))\n",
    "    values[pred_date] = df_pred[['pickup_count', 'pred']].mean().to_dict()\n",
    "    \n",
    "df_gt = pd.DataFrame(values).transpose().sort_index()\n",
    "df_regression_metrics = pd.merge(df_regression_metrics, df_gt, how='inner', left_index=True, right_index=True)\n",
    "df_regression_metrics.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_regression_metrics[['pickup_count', 'pred']].plot(figsize=(10,3), title='Taxi Pickup Count Per Hour (mean)')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 参考コード"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 参考｜キャプチャしたデータからinferenceIdを収集する\n",
    "データキャプチャ時にサンプリングしている場合は、キャプチャ対象となった推論リクエストを特定するためキャプチャされたレコードを取得する  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bucket_name = sagemaker.Session().default_bucket()\n",
    "capture_data_key = 'model_monitor/endpoint-data-capture/nyctaxi-endpoint/AllTraffic/2022/12/11/07/27-10-902-c1277fdc-38a1-407e-b96d-3921ed423787.jsonl'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>eventId</th>\n",
       "      <th>inferenceId</th>\n",
       "      <th>inferenceTime</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>dcbf4055-1196-4b71-ab54-e290b4b3530e</td>\n",
       "      <td>50f38bbc-1c15-4d81-ac2a-7a922d8adef6</td>\n",
       "      <td>2022-12-11T07:27:10Z</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>53b77003-449e-4ee9-9eea-88ac0b66df40</td>\n",
       "      <td>b5d9a7ac-9eaf-4fa4-8549-c4b1e2c78492</td>\n",
       "      <td>2022-12-11T07:27:10Z</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>07e28837-40c6-4731-a1fa-dcdfbf543ec6</td>\n",
       "      <td>20b5e6f6-0b84-44fc-8977-68da623391d1</td>\n",
       "      <td>2022-12-11T07:27:11Z</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3004ec50-6b30-4043-b149-76e3322016cd</td>\n",
       "      <td>37b11b6d-1087-4378-948d-247a86da68f6</td>\n",
       "      <td>2022-12-11T07:27:11Z</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>9336b2ff-dea3-4095-9d9c-f882cab4e7b3</td>\n",
       "      <td>f6bed701-025d-4aac-9941-c90e9ede4509</td>\n",
       "      <td>2022-12-11T07:27:11Z</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>83</th>\n",
       "      <td>eb42e34b-93e5-46e5-9f28-68807f93348c</td>\n",
       "      <td>db5843e2-a983-4a19-b6fe-ae6c8a321ce9</td>\n",
       "      <td>2022-12-11T07:27:15Z</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>84</th>\n",
       "      <td>72846ebd-4f6b-4571-9f09-554d792f5609</td>\n",
       "      <td>190dbd03-4479-4706-bbe0-0b101e1a6fe2</td>\n",
       "      <td>2022-12-11T07:27:15Z</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>85</th>\n",
       "      <td>bddc4048-e81c-4bbf-be19-d2359f5d8a0b</td>\n",
       "      <td>bf1842fd-2884-4f2b-9932-8a2acb9b754c</td>\n",
       "      <td>2022-12-11T07:27:15Z</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>86</th>\n",
       "      <td>38c518b2-5d79-4cd6-94b6-6497bb9e992b</td>\n",
       "      <td>fb6c7636-cce2-4727-a79e-599ea8c1fd1d</td>\n",
       "      <td>2022-12-11T07:27:15Z</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>87</th>\n",
       "      <td>1d62673c-fb3a-4dad-94cf-50497d76723a</td>\n",
       "      <td>ad8d3c31-64ef-4b0a-92b0-e34e870d9fbf</td>\n",
       "      <td>2022-12-11T07:27:15Z</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>88 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                 eventId  \\\n",
       "0   dcbf4055-1196-4b71-ab54-e290b4b3530e   \n",
       "1   53b77003-449e-4ee9-9eea-88ac0b66df40   \n",
       "2   07e28837-40c6-4731-a1fa-dcdfbf543ec6   \n",
       "3   3004ec50-6b30-4043-b149-76e3322016cd   \n",
       "4   9336b2ff-dea3-4095-9d9c-f882cab4e7b3   \n",
       "..                                   ...   \n",
       "83  eb42e34b-93e5-46e5-9f28-68807f93348c   \n",
       "84  72846ebd-4f6b-4571-9f09-554d792f5609   \n",
       "85  bddc4048-e81c-4bbf-be19-d2359f5d8a0b   \n",
       "86  38c518b2-5d79-4cd6-94b6-6497bb9e992b   \n",
       "87  1d62673c-fb3a-4dad-94cf-50497d76723a   \n",
       "\n",
       "                             inferenceId         inferenceTime  \n",
       "0   50f38bbc-1c15-4d81-ac2a-7a922d8adef6  2022-12-11T07:27:10Z  \n",
       "1   b5d9a7ac-9eaf-4fa4-8549-c4b1e2c78492  2022-12-11T07:27:10Z  \n",
       "2   20b5e6f6-0b84-44fc-8977-68da623391d1  2022-12-11T07:27:11Z  \n",
       "3   37b11b6d-1087-4378-948d-247a86da68f6  2022-12-11T07:27:11Z  \n",
       "4   f6bed701-025d-4aac-9941-c90e9ede4509  2022-12-11T07:27:11Z  \n",
       "..                                   ...                   ...  \n",
       "83  db5843e2-a983-4a19-b6fe-ae6c8a321ce9  2022-12-11T07:27:15Z  \n",
       "84  190dbd03-4479-4706-bbe0-0b101e1a6fe2  2022-12-11T07:27:15Z  \n",
       "85  bf1842fd-2884-4f2b-9932-8a2acb9b754c  2022-12-11T07:27:15Z  \n",
       "86  fb6c7636-cce2-4727-a79e-599ea8c1fd1d  2022-12-11T07:27:15Z  \n",
       "87  ad8d3c31-64ef-4b0a-92b0-e34e870d9fbf  2022-12-11T07:27:15Z  \n",
       "\n",
       "[88 rows x 3 columns]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s3r_bucket = s3r.Bucket(bucket_name)\n",
    "capture_data = s3r_bucket.Object(capture_data_key).get()['Body'].read().decode('utf-8')\n",
    "\n",
    "df_capture = pd.read_json(io.StringIO(capture_data), lines=True)\n",
    "df_inference_id = pd.json_normalize(df_capture['eventMetadata'])\n",
    "df_inference_id"
   ]
  }
 ],
 "metadata": {
  "instance_type": "ml.m5.large",
  "kernelspec": {
   "display_name": "Python 3 (Data Science)",
   "language": "python",
   "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:ap-northeast-1:102112518831:image/datascience-1.0"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}