A&B Testing ML models in Production using AWS SageMaker.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# A/B Testing ML models in Production using AWS SageMaker"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook, we will evaluate models by invoking specific variants of the model, and release a new model by specifying traffic distribution\n",
    "\n",
    "In production ML workflows, we frequently have to our their models, such as by performing Automatic Model Tuning, training on additional or more-recent data, and improving feature selection. **Performing A/B testing** between a new model and an old model with production traffic can be an effective final step in the validation process for a new model. In A/B testing, you test different variants of your models and compare how each variant performs relative to each other. You then choose the best-performing model to replace a previously-existing model new version delivers better performance than the previously-existing version.\n",
    "\n",
    "Amazon SageMaker enables you to test multiple models or model versions behind the same endpoint using production variants. Each production variant identifies a machine learning (ML) model and the resources deployed for hosting the model. You can distribute endpoint invocation requests across multiple production variants by providing the traffic distribution for each variant, or you can invoke a specific variant directly for each request."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Testing models by distributing traffic to variants\n",
    "\n",
    "To test multiple models by distributing traffic between them, specify the percentage of the traffic to route to each model by specifying the weight for each production variant in the [endpoint configuration](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateEndpointConfig.html). Amazon SageMaker distributes the traffic between production variants based on the respective weights that you provided. This is the default behavior when using production variants. The following diagram shows how this works in more detail. Each inference response also contains the name of the variant that processed the request."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/gif": "R0lGODlh9AF1AffiAAEBAAsGAgwLChMLAxgOBB0SBh8UCBgYGCMWCSscDCUYCDAfDjIhDjsmDzUkEDkmEj0qFC8gDSQkJCsrKzQ0NDs7Ozk1Mj8zJRY/KUItFkUvGEYxGEs0Gko0GVM6Hlk+H1E2G1Q+Jlk/IFI/KkI9NV1BIlZDJlBBMVBGO1BIPkdAOGJFJGZIJmtMKWhKJnBPKnRSLHlWL3pYLn1aMXhXMGxRK3tSGyhkTiloUy5yXTB2XS5zYTJ6ZTJ6ZyJZREREREtLS01IQ1NNSFJMQ1RUVFtaWlhYW2RkZGZqbmxsbGBlaXR0dHt7e3Z3eGV7d1RnZiJYMINdM4ZhNYliNo1lOIppOpRqO5puPZBnOZ1xPqtxKaFyP7N1KJpdC5x0QpVwQKN1Qqt8Rqd5RLB/RzuKADiFAECPAESTAFCbCGGmMDaDbTeFbDqLdDmJdD2TfDyQdUSGdkCTfnKoV8KEOMOIN7SCSbmGS72JTbOESrSLT42+b8GMT9aUR9eaRsSOUcmPUcyTVMeTVNCXVtKYV9WbWdqeWtmcV8SQQeGeT92hXNahXM+jWeqmVeKkXuesVvWvXuyvV92rYuWnYOapYeusY+inYPGvZeu0aPWzZ/a4Zva2afa7bJPBePfBb/fCcfjGc/jKdfnOePnSe/rZf+3Bbz+WgEGchEiZhliajGuQiUWjjESjiUmsk0inkVOsmGWnmU+2oFq3pGe3p3e3qnDDs4ODg4uMjIeIiJSUlJycnIi5sKSkpKurq7Ozs7u7u6y5t5rFg67Rm6bMkorGupfDu7jXqKTHv/rbgfzkhcXeuMjgvZjOxK/Xz7vj3MTExMzMzM7Fx9PT09ra2sLc18/l0NXc5N3j6tfq59rx7c7o4+Tk5O3t7eLt6+rz6Orx9On28/P09PX5/f////X69O708OTw3gAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACH5BAAAAAAALAAAAAD0AXUBAAj+AMUJHEiwoMGDCBMqXMiwocOHECNKnEixosWLGDNq3Mixo8ePIEOKHEmypMmTKFOqXMmypcuXMGPKnEmzps2bOHPq3Mmzp8+fQIMKHUq0qNGjSJMqXcq0qdOnUKNKnUq1qtWrWLNq3cq1q9evYMOKHUu2rNmzaNOqXcu2rdu3cCuC20ZXG7iJdCeCk3Y3It2+4uYCRqhNW9zDiIvyAsAYgIRdEbUdkPDwmS1p4nABgBxxcZGBFABEUxgOgIDBCsHh4py4tWuUiw8sqcD4GURwAg48zLVZnLMktiMDCGD4GQDdCMMJXFILdcJtx19Lnx5ycRKBSQDwkpbE1xLIvSr+TKi1TaC2JUBwSZggbteS8u4xS1vyw9Y2abQrSIuWZPS2JRP84ItAudTSCxBMYEZQEQAMyBsu4vgCxA+4wMdEMkdEkwuE4vDyww+53IVLLrsQwUR5TDCWizjPADGBLYZRJ+OMEFm3HAC9+NIYLr0wFgAAP4gTjgWNARnODwAYhuQz4EhwHAAT7OLkAc7s0mA4tAnAWC/iZKnlBIP1aIs4RwCwjTPDhQaEOEY09sxk4vAmQGhMiBOaAFr+sA2DADQhzQEHIAmAczQWamiHxyFBm5k6StCLZAAwCUSDaAJRDW8ViDOpYZtqtgQ4S0gQTY+Q8eZLj0eA0+MER+IIzgH+kRKkjQBASGaBb01EY9ygZRrBpGPj9JKgjnrS5sw2Tm6DG3uaEaENLjAeKu2hiw1H64BiiiMNlAIthotmXC6rqZnjasNgcMrxxkucON4CgDMCOakNkndNGtxAZWoGIThJVADrAeCUqSC32xxBAaxAbGMBcmUqO6c48zIGxIDTVjzjYqkCttiK1UQnji0A5MLbitoAcKsQg3ZpJoMDSrPLNlZyyVsvmkEWDqzb0DvuvQJZ+eNo2dUSzQS8JinOOFCCg2QuxiVMgQR3MUhXAOxZ4wsvZb5r8dbS2UjQxgLRlkQusEqzrQS4IJlpmUmADEA1PVZAdpIYw9zg2bmU+Rn+beXZWxCkJAhERINWChAOg4aVNsE2BwTgDMgJTwCwOIMrG4AA/AGwRDS0Ucz154dhXBDY2i7amzgpOrYei7BSHYBtWfcWDay9LMalZozpqTI5Oxs06Zi+MSaBBQFEI/DR3FY7gQQH4Ae1OIhDD0CtgzN2HejYwwWONuURtD1g4DzjTIwC6QrO9gJtE42y2oxTvjMKQhzN+XYJJI0zzygnzjYx8q8/Qf4biDWgwT5weKN7EOueNKAxF2vMpXvk0IZy9qKgUY0mexjMoAY3yMEOevCDIAyhCEdIwhKa8IQoTKEKV8jCFrrwhTCMoQxltI1q2PCGOMyhDnfIwx768If+QAyiEIdIxCLm0IEzTGJBkGUCE4QgBE18ohSnGAIRPNGKUjSBFZ0IRSpGMYte3OITnUjGKJIRimbsYhPPyEYqjpGLYwxjF+eYRTjScYxivCMap2hHOjYRAk1QoiDFYY0QdCITm9gEIhPJyEY68pGKZOQiE5mJSSrSkpJ05CIRuUlKRvKTnPQkJClpyUpCcpKlxCQmRflJUjbSlK9k5Ce4oIRBKtEaI8gEJnaJCV3yspe/7KUvfznMXRZTmMQM5jGDycxmOpOZx6xkMqfJy2gWM5rKVOY1q8nNTmgBCbZMIi6XSc1nNpOc5TSnOtfJzmyaE53cfKc64elNcIZThuP+bKc+98nPfvrznwDdZz3vic9cBvSgCE2oQhf6zIESFIb5ZKhEJ0rRigbToQ91YUQtytGOelSg38zoCzf60ZKa1KQYFekKSXrSlrpUoilVaQpZ+tKa2tSfMZXpCWl60576tKEh1akKefrTohY1p0IlIVGNylSbIjWpIlxqU6d60qdCFYRSpapWO2rVq3owq1sN60S76lUOglWsE9XES+GJCbKWVYNETaQlMKGJTqiVmZu4qzM1oddNeEKtmshrQjXxiU94QpHqTKQ5+RpYcyZSrYjU6y8Vu0vJVvaujAWqPd8aVYNmMxCN0IQu90AJyWrCEneYhGX3WghA8JUSgwD+JjvRqYlJ2MELYGjEJhabCEKslpd8bcQdaFsIQUB2EIWwrCYA4dvKQhOwm6BEaZ3pVs5mj6Z1FYEHPPGJMAyAEp8Q7CY+oYkoEKIT6FVrIg/7iSvgwROY2EQUAAAIUdwhA43wRHoR2YnD7tevdo2vXjPRCSoEoAYggAAgwptXv4r3GFFgQXjju1sHbwIMeEjEFALrib/WdROd+EQdCqBbShRADKE4LF/9GoYwrHe3Hw7vIKqwCUuMYQODQCwzq2td0PH0E3dIQCNEsQIYAKLIidCEF2LgBTtQAhAlaAEgNpEFKnggDJcYQAImkcgrBOADnQCEByjhhxLEIBGYuML+DFZgBxiwoBGdqAMHpGCJTVzhD32VQQ2QIYoNuEARVpiCJK7AASrwVQow2IAUABEGT9yh0WLgwBQUkYAEBKIOnrADB0rQiElgIQoeCMQmIACG7jKAElPwwBY08QcqTCEMU66CB6ygCUJMYQYlSAQNALCHQkAAAIPoBHWD2uMP8pSvDqjDJhggBhnUIQor0IQCFHCHD9ghCmGowgY8gYAOVAEBidj0im+9gi0UogSAaIEdaNACTBCABS0AwBQgMAVAQCAPNYjBJ2Zwh75S4d5eSEAYrgAAL1jBAWLYQBjAkAArBMAKV2BBKKgAA0IwwAsbsMIKVlAHFiQiAohewR3+AIAFD3AgFFEogShcMAU7tMAPELDDFAAABhpk4Q9ShkAdRu4FCMAgDAkwxCYAAYIpD3uzxe4gTzOx7xnUIQOonQIHIKCJB0x5A38ARKoTgIkO3CEUGxAEDabwV/lKIREZCMMKGiHzDWgAEw94cgY6kYUVzCACX/AAAh7b1ysQ4AMeAIN8XyAKDYAhFGHYQAnEcAwYzMALMAhFoKfwgk9MQtdWGAQMrCAC8mZgBh74hCAgMPQHTAICpLWCFAbgBSrMQBRRsEImrjCFAmQhDBwQhR2KXoLANiLUu20mj5NuMaJqYhAb2MAUKAEBKdAABJrYQHKv7IEoTOHtHFhw9l/+QAXylhcGopACAVpwh4zHwAOa6AAlcN6JK7RABhkw+BUE7EtEQnsUHS4vFTwhgjCIog4lwAFhgAwyMANXAAMQNgVYIAOgMAmEAANX8IBXsAKfYAkiQAUl8AmAkAGitQIi0AKbgAXx9wBg4AVXsG9gEAYQ4AUaIAZhIAL/5wF7sAJ5lQjAd3TE91WexUyBpQAAIAl/MABiRwCJkACCMGpUoACB4GWEAAF+8AkL8Ac0sAGXEFgwsAKhQAkAoABgoACKAAMFQAgPIFwc0AmppoKKQANS8AlTsAeitUub8AIbEAqABQMMeAX3xgF38G94AABRgAcIUAcEsAKK4ACJFwb+UuABdeABkkCCMhADY+AAn+AHCMB0YAAAVyAKHLACiUAAjjcFoNACXkADHFCIm5cBoiAGDQAIChBsitAAeyBswkdsOQhXOxhMgRUGUZBXqUYFNGAHWRBaVqAIYOABUwADYwAGhbAJVlAIg1AChsBXdeBicTYDlmBlvmgHYEAJhJAFm3AHgmcFG0CI8tVvvLQJY+AFwacJdaBslJBoM6AJlLACHAADbggDHiADjUYFIBAFtVYCC+cJYLABHlAIiWAFm6AIVBBfjbAChvAJg/CBVRAFYuAHm4AHdyAJJbACqlcHWaCBVoAJLbAFn9AIw/iGO0aLtYhBUuVXu9Rf3KX+CX/VVoEFkzKpVmU3kxQ2V+NVk/nnYcIGYphwWDNJXsHkV3MFhzBGlHzVCZsACrvVYRPWYR7GYIHlYHVFV/BFV6BwWn7VYYxEYTJpYcJWV0wHWQGGgytpi2xlTHQFWMQkWr60WpiFi29JV4s1YOuEWXrFV88kWcPkl3u5WLKVTpq1lmzpTDFpCSEGWXNJV0bpWMGHCYVVZ12pCaAgi7w0YXi5Sx02l4LFTNw1mYQllb8lVsOHmDSCXZqQBdbYXS1ACXkFWIEVCDCQXIFFm7m5XIkgWFKABZ4wBi/QCJZAAy6Wm5bQAsfZYJpgBYLHV5bgB6UlmIFVBzBgBxUmCVH+gAVWgAVGh1a/lJqqKSM85QkyAACFEAoPAAC1FVryiAmXUAcgEGyUgJuUgAnSFV0KsAJ/1QkJAACfwAEAcAeAAAAfKQm6lQkacIKagGZ8pQAZIF2iBwBi4GFwGAaNoWybEIQO4INe8AngGZ4qOZ5bU56EAABhQAkBUAViEAAEMAObEAMckABuJlwEEABrRwUbkAAEAAh2wBh/YFgOBwgaQHOXaAl1wBi0tqCeEG8Z4AFQVgAuGggzAAAOQAlR4I+BBQER4AkEsG27VFoekACyGaK8JJ4kKh3HNmol8KN/AAPQhp4eAACbRwCFQAUsZ6DxRgUJwAGSMAAQMJ2EgAD+LaABGQADLcABhQAALSAFAFAHBjAFgeCHG8BrJRAAfsdxACAFmwADMcBYBNABo8ClW9kJI3cHW2mmbTWiaTotRMWGAeABpGcFHlCpgjCm/xcAg1AHBImJJeABoiADDqAJNCoKlfVrIiAGCRAAsAYAH1gAV6AAVxAFAXAMghAAd+ABKkePjcBroUBZm/Clx8AACZBidaUBDjCZqoqmrdoacWUIjHEFlgAAMeBwg1CroBAGBeAHD7ABdRAAYFACItCkDoAJBVACoACHVQoDlcAYkrCoMnAHUeAHBGAFYgAAtgYAfuAB/PmrJzoGn0AIipAJgZUBCXAJBOABkWCRiRD+AGSnqmfKqu1qKEu1CQiAnpYwAAzgAAOqAQnwCZcoZgSQAQAwA27nCSVQAJogArz2CYg0cmQXAAMQX5eKAARwojGwCes5AAEAcxzwCRvgAJIQAAVACQkQASv2o1ybCCsAAJuQpHegrmbKrjN7GEulCYKwByQ7CFdwB3fQCIIwZcKFCY0ABmPgB4AACMY1CHjWCGGQY3RlCXagW1nXWAOXZHegCIoQBWDwAgAgCIIwCHgbCHi7BU/mWozlB3fWCYNgB5rAdtMFs6uKdHV7KC1plF/JYEvJYaPpkr6rk1pZYx2GCXUGlqVJCBsAAQjQe4cVX/DlV/wFXPoVlCBKWKf+iVZ0W7tvcVZ3aZfO9b3X+5e/pAmSUAduCJjjO5iyG7O0q72reYtTdZXhu778lL3uyxbce1MoSb8IZb/3qxb5y78C/JIy+7+vEcADzL/+a8BngcAJvL4LzMBl4cAPDLMRLMFjQcEVPLcFjMF2C78bHMIX7MFgocEhjL0dTMLbC8ILNb8J3JbypE8jrMJdYXyR6Ve/FVlHuY7NS2ElJZOFxZkABbyeKYuZ0MPQpJmVJVi5qU4zTMNbwZqwG1yk9VuDIAl9iVyMdQe9WWvJ5VHyWAdXkAVZ4Ad31Za/BQj3+Ut+ILp0FQiu1UyUEAidKY+AIGCTsE5PDMVZwVP5iqL+HeYC9HVYlDSWKxAG4XVYoWBmAWalT1kCLwCVisVIRAlj6AVjcjtPm3CxLlACH0CSzPlhFdZIHvZiZptjl+xuCCCT3doBieSUgBWyDBBf6DWJ+HUJUhAClhC+e8zHU8E9AvFjGNoCPTgAhTCPGwCybNsCGxC3MxACYRAKLRAFdkUJlXoFo9CopCAFInAFm+AH0PZqHODN7ygCddAJgkAFSdlOrekBoxDEWAAGnjAFfpAILPAB56yLJQAGVNABgDAJvsgBZoxfz1gCvlUCvDYKBCcDQKsBLdAIn2AHIrACG7AJhFACuUYIY2YHBQABu2xOvezLTqENvlAmXEJILPz+CVsAARrwZMlXCCvwAWLgAH5QAyIwX4BABS0QCBwwCDEgBXY1CSVwBVeqhg84CBkACF4AAFaQAA8wBQ9ACDQAA36wAYAwCFEQu+y0CWGAAFIgAzJQCHjgAFTgAJNwBVYQBi3tAQzgZSyAqH7A1DQwZlDqAbf3AbbmAFbwCS3gAGCwBwrgBSIAA0VYBR0Qd+OMByWg2JNACmEwZrycwiL9Fe4BK8MRIz82gTGwB1EAAz+nAYkgClPYAfFlzhAgAnenZtRcWxvQCDEAAzTgj1lAAwZQB50bCiwwBZ+wzw6wAlKAALqdyYkVBgowkea1b7wGCoIwBSsQdD/nCRAgCGz+G2l5xXF1OgPzpYtYEAN3wAJSYAUtAAOjoNE10AKi8AceYAcEENsEQAUsMAmhINGqBdKSPdldUT2MsQQClNITeAcdcMgwIJ+NMApXuAGBpYiHKgZUUAhwGtQZMAiWsAEV+9h44HVbMAOfsALeWAI9NwNiQM+hqU92VgJ8JgqHBQMD6glvnQUbkAgxoGygTQgsIAYg8FcQEAY0QAUGMMZSIAgskG4Q8IsUJwPHcGRXOAqBoIgIYAUGB4CNAAoSrdWz2L72PRYFwxjrEswsDAo6igkAsAHMRwgbVwcPIAgrUANUwGszIGUikAglAAN/1QgI8Ad/TAUysAKDoABToNP+oCBp0B0G+fYHIkAItrbOW50FAVBlVLAHWLCCClAHfj0GAdBxYmCzhAAIHFAHA2AFNDCGGWAHhhcGt7kBgWAFGgsGVJ0AeGCMiYAAYqB3ibCC1okH+BUKdQABukXfVF7lYHE+yEIc+01Om7AH3pjWVPawIiCAGugBF1gIlsACGecJYYCdmUAJVtCbmDADe9CQLEAFVmAHdtAJf92ahSDUHeCNgEBnMAxchNACG7cCYPDtn0AFdnAHNx0FXjAGoosFCCoGduAAGE26IekHHOABZuwFBhkFmOAHLqZtLTAJnhBpLeDNGwsCfzAJWUAJ6JzxTlzfvJ4VvjAgTEAEBMH+U8U7lFHJV2BJyzE5lvDVw9YLmXlFlU65W045lHwVXocU8/uEmZ8AClDZvIRsmorlYboHAoW1W+T1yjOf8ytGYDhMYYVFy7DsCboklB2/6x+/FdvgC+RwF3NR8ix8TpcFXH3pwsYEWcCFi/v7T/Akl820CYUQBp1p9muf9m959mmP9imp9VufFdvQC/9TEMank3SpY87Ulp4ACoUFlYZe9wp1k48/vpUMTHVFxD4M+STLUCH990CRC/FzEDy1CQcpCYwEXXllCbjZxPLLWExMWbskAyXgAitQAnVQva6PWaEpmHk5m4lECVcgdG/5WA02aEyOxSSrYVcgm4DlaGH+QLKzmfe7SVeUhFMe7/kSEQ5mIw2GEQ0XZB53EQ4ItBB8YRB7MRDcw/3jjxD7IQ3rT/gs7Al7kACEsAmVcMd8NQiE4GjS5wmNgLoAQYmSIEqbNiUCpEkhoUKbNGHCVCdMgA9h7DjEJAkTJUCYNG2iFAgTSEqYMkFEmRKlJUKUMiWkpOGOJ0yWLBWSpIngpysbwlxJ9HESBBEwShS65PCSlBUOATXSZAlTI0qaBpX8OKihSq5dMXXSgkTcWLJlzZ5Fm1btWrZt3b6FG1fuXLp17d5duy0Xrgq4djU50muss125tPl6RlabM17gxElzBq6X4F23dnl7nGsbuCS5nIn+8+Xsli9ftpZIExcu2llnuNpaG3GSa6dAIAC1+KCgxSYwHxKEwQJAzJ0MHGZ8ipKhwwNKhB44gOHJCoMHYRxq8gQKRJhRLbaIkkIjhgcITRNxYODiUqIVijbJ7sqbQwcQNRLMmLTCT6dNdzhw2OCFB0SwxIoV7vhoJEBgUCiCKzygBIwovIhCkhYg8ACQSTxIYI8sHvBgBU2uWACCMTDyyiuwxMKrRRdfhDFGGWek0S5tlhiHiMeQ2EacXpJoYrKylrCgCFy0kcCWXIyooJdcirwlmiOSYAKcCohochdf/MJlCSKSgEbKsraxpcfXYuuKNtsWuMK5RhK4YpAw9uD+oBEw7mgEgUZc0OASCKZYwYpJaBjDAUIAUYCQTTzahIMsRolCxA3E2GCGS0AIZIXoIJACkzuqSnEkGiCYJAETEnFgjxb2402BRERIIJIHxogBAQ88cO8kg0rwYBMZHICgEAkFiUITFlqgJAAwBtlAkSkiCEOBSPBgQJJFQ50trBq35bZbb78Fty4meBHHiHFuxMxHaXjZxZbVxDliF3GY2MUIeLXZZoklyC0CGl5yAUKaI3xcwhZf9sqll2hwqWUXecfaZl3HzoQvpU4AAWEPFgoqQRA7HPCAkEE+wOQPD0JwtYUrkJmhhA0aAQWUKgpYwQUHBln0Iw6uACURD7b+4GCSEggB5YoYADQBBCk2+eShFDPRZAoqRFmhDk86sKOFPT7xBAwYRIliClE8yCKSRjCBLpSPEskABElCoQSAFY65YgZKoighghYS6SCUK2T4hBIWohjAhRU2UBTbbFkMt3HHH4c8crfAAaJHIMSJ5otwyJIml1t6scWxIz7rrBZxdBRniSY+KziJXoqQZglxnGHCYFxwyWWXZ7RJ4gczxcGF3LdgqxilTQLxYI8VGrHkQD8seXCMEjKBAIxJmHPBilFgMOqPT4B7oJFGrMhJoU02uMITT0QgQAZPINgjFA+w2AAL3hAsCduop/hkhTA0wZrWHBIGGHyCBvzzwBX+6oAJUQzOE5qgBAeoAIpQfCILBIDAJcAgBSpkwBPco8QGOtGTTwSiA1NIACUKYYWCKE4lK5JcDGU4QxrKqBeyAx4RSCAYiOFCGrs4Qi4cswQKFIEJYsLFEX6wC1yQ4Ai22AUQaiGBXgDhCE3CBehskcUiiKMIlxPHNnyxGbgQL01/yIAdPFCICAJCBB9gAQ0IwQArsKAoymLBFEaxgiiEwVYi2JAINrCCSzzkIw+wgic6AQYA+GETDmDAChwQIQTAoACBSMQHEnEtr2xiBjPQjhg08YA6lMAPVshCHVbwiRYcp3pWSMAKHjCJMUxhDADwQAlKAAYEEAIGK6hbGBL+QIMIMMAOGdCEJHT5AAY0YpAQYAGKXAgRGNbQmtfEZjYVo42xhGMXPByLN1AjmYkdITDkeExq2BUOcBzMMb5QmDR+6Atx4Esb92wnOIpArnBIwzVxMSNXIEgQQkhFEBmhwhUKsoc9TOIKeAiEHxIRFEJA5SeT6IQksGCFkUBEIU9RiBgyABIOTOEK7tHEGKIACE9Q4lNO84omElEITRAiJ4JoREsAcRVCaKIQQQFEVcSABUUdqhF2CIMXxAAIQHRiEhINSh2sMIg7DKKnlEhqFD6giUYk1CPTtJi2tDlWspZ1rLdITIx60YSx6G4uARWoQxySCbl24oGZ4M8mPKH+V4Ms6jqd6MRDDPJAlez1eBmwDiVKkIhP5Ex9nWBU8eLj14cEVq5yHQllNWHXnDnEE5/g2iY6AbW+blaRgQ1sfurohc0CFqZgxYQnxGpW2tbWtrddizaiwU25wFUlUOuKJmTjtNeiBD7DLZ5CPlKIhDyEjQpBSXElG1zjmuSr1f1qJrT7VehSt7gQgdokwtDck0zXhdXEbXrVu95rPuNhb0WTQGuC3ZR0t76wXZtEFSkbaQZXuE9TLmwFrLjNcnLAYWUcexW8YAZvaxy+kAY66eJb47mRsZ0ARc4+kWFK1EEh2fHrZ2mCCdBquGke3YQVHNABOClkw5uwRGAzK9r+TwR2EILoa3CPh5/vHtjHP37hbBs8ZCIXWS7h4MXv4DtdT6BREIm4AxUURYkrWEETYiDAHTYRBio0ZBCn1PKWp9AQKnvhq1suwCI+Qa1EdC0KjeiERI9XiEL4gQqA2AQIICCJQgjiu5q4AwIIUBDzAtnQAkavkRW9aEbzApx1oTA1kQeIB3QgA8hsAQs8kFAC1AEMIpgBBzbRggG0IACCCAMDRMCBS7BAlzGQKwuOo5BJWCIKCRDkJTwABlGI4AotAJED3BgBhxbrt5pQhBVeZuBDN3vAiWZ0tKW93m08Qxq8tUukv1IbNN4BExsoRAZYUFU3WsIPYQgDARJxLFH+lOAFHNByHsBQAIkAoKmaKAEVHDuIXuZ5DCsQQyhKcIUVfGF9EpkCSbzyiUJ4YJPOhvizhTxtile8rNr4J160TRsP3IEFk8A3INadgKWKABNegEALFpAIlR1DBisAQSFAiwUErCDTmxy1CDIctZ4YZAZiaMEYBG6FGdzhEy8IgxiWZhAVBYIDD4941M87cYtX3eoy5AVqXKTtJstksZTwgB2ogAk7bKAOHShEAuyAiUStgArHaAH3sFCIF4iUEpKYAsgBPYCLzGDPHKiDJDQgCT4mIgBegIHV/AeG3QgiIV1pcga2InXKpwjaV8d85r0Fjm04Q8l30fYmBOGCP8j+QBKUgEEiogAyMYCdCspegQauMAUvgG1ZJYrCJqLwHH2b7woJ2MACAgEKYWZgBp4YQwI+kIE6UMGRM8CDIAyVhSn0WPQugErltb84zXff+zXyhfBepG2PXtcjPoVKVCoh0/Xr/fyaqET6fdoQ4m5CEgR5j0x7SuucmF8hlJAKqUiRHtu+yru870PABISLcPC8iRm/+KIugSIuQwowj6JAC/yz7qpAqHkIySJAlfjAAoy6A1TAEjRBs9iG94IR8hPBFnTBFiTBE5TBBNQSGmHBF8TBHIS4GJzBHsS8frI2G4RAHSTCIjw0HvTBJKw4aWCCbblBI4TCKFQRqlPCKqT+OF6Ahs+LkSeUwi6UQiS0wjAkMlxIqxrhQi9EQyIEQzFkw/XqhYxzwiFMwzmMwjVswzukLXCIBml4F245QzoExO2zQzwkRG0KhyP5lj8MxEUcQSosxEckq2SAhnBRREa0xCN0REjUxBoah22Ihj70lkq8xFE8sEHcxFMMlx/aHEqUQ1J0RWczRVSUxW2Jhm3Qwm4RxVfUxVCJxVn0xRjJusfJxV0kxiBLsF9Exm5xhkdrHDOSjUKbJmjsimf8LetSnPJyIWyMxmrkCm3EFm8MFe16xkxMxnJskXDQBsiIHGswgcv6iI/oq3gsLXmUR/Ohx3iEx9KCx3d8x3u8LIP+6Md7zEe5Gkh9nEeApEd7vI5/rEd8ZMh8HMh+FK5P4IJjNMeLbJF8uUVwsQYFmIOPBMmQFMmRJMmSNMmT/Eg6QMmQVMmVTEmXhEmWjMmZDMlDsAF7wcicxItdSLIYAocjKIKgFMqhLAIiIEqhNIKgJAIjYMqgTMqhNMqjHEqmbEqkpMqrhMqr1MqqVMqt1EqkdMqn7EqifEqvpMqsrMq0FEuhXMqmPEsjyAWdlEu6OEQVnMu7xEtpiwYvyMu+9EtFkydm+MvBJEz2IsPCRMzELCtpoKdVVMzHhEyfZMzIpMzKbJxxMAzLVAvH1MzOnIt7kgYHtKZx8IbSNM3TRM3+1FTN1WTN1izNbXDN04TN2HxN2rRN2bxN1pxN3DRNzvTMU+yFz8gmcLCAEzDO4zwBEhiBEUAB5DzO5WTOEXDO54RO5FzO6YROFKjO6eTO7vTO7wTP7JRO8CRP6xxP8hwBEjiBCzCd39zEdfFNa7KGDXiE+rTP+8TP+rSER9hP/sxP/txPm7hPAcVPAiXQ/0TQBFXQBS1Q/3TQBmXQ/+zP/mTQTNACJXBPTdyGwxDNayKeECzGEJ1Ci8zQNqyFSSQr2ABREWVRBCtRQowGX+jQbBrGFtXFXnzRBPQFu9SmGrVRV8TRHPW+MqktH/3RUQxSIcU8eRJO2jLSI7XEJFX+UqtLmNt6UihdRCmdUorbhl4gI9u6UiwFRC3dUmnTBmZ00lYU0xYl0zJdtFwoQytV0zUV0TZ1UyKLGGnYyLIKUzpFQzu90wYjDPbqUz/tQkANVPbSiz1NU2k0VCAlx0SlOHA4U0Y1q0J9VChEVEnFrRpcMEzN1CLcVE6lLTEahwYD1VDVwVFFRs6zxVf1hle1RVeV1Vq11VvF1VzV1V3l1VmFC2lA0/WCDVVlUVb1xSb4ASJQ1mUlAiFg1meF1miV1mml1mq11mmtAB5Fi13QOlQdAWKt00gtTCKQ0XEIh3EwV3M913Q913BYV3d1V3BgV3hNV3EAh3Y91XtFV3b+5Vd89dd+Bdh1Rdd2fddxMA222NBomFFCnVNwvURjncUiaNKqw4UmXIvwM7JUdVgYFFfClFjMY5i12IUvJTKN3dgChFhZLALBvDpc+IK0CAdv2IWF/dSGXdEDu1kgA9GcTcOURcWPbVmLPQsoYjTyUy7ziantki/oOol3ZJToiqkIjK6FtK/ye78QhK4/S4n3YJQeY1prrC/hUq7+clHNBFqrq9iz6LwgXLRIUwgYoILGEgS3AQXCYhpNiAJF4RrBYhqDAISt0AQv4ABKAIUFsRb1ESy7wg67bSyI6K9NqAMIAIENiAJKaCy5atyROAYw0LfOGonG0omcqj5N2Fv+eOQaO1gBS3gIFggDUGjcj/CEMDiRvmWUTmgaSvgDkKABFxiDqkUJnz3F0QFZoSWLaMCholXTT0AhTQiFKIAATMADO3iIRBiEQKiDqgiDBdK/1tuEDCiBQtqE4IgCUfADDaiELUMQTGAuD7sDBOkESvACPNMEQ1g/4/G7SJC+KbgESfCDTGiELMCZTSCEO2iBF1iPZGqISQCDnsolRagDGBODB76EQviD3k0EAtiDiwmAn/KCTZKERggEPPgDTyiELVCUSVCEPdAyK1iASGABCMgDBgADZqOmjh3MIqCnq8sF4gUeUGzbVqSrUvEDUIAAovMADrCyDQgAK4CBQJj+Ag0AAcCBgCl+OgWYJIOwAkEiBEIQAbo74ivAhClmgA3wAAKwg0v4D7NbH+swnihggVE4nxkIgwIYNwjYAAg4FAUAAQCwAq8JBSbWBA7QAAcIAwhwADGggU0QgQVwgKkigBJon1BYARkAhShoAUDIgA/gAELwAgJopTAgBA2IPULwgwFYgQEAgykAgDvIgkQ4hqIg26+w4b88W4oVWs7rhW5F3ukiXRGwAkK4GTEYhCl4gExIACvIhA24g/G6AgXQBAe4gk2IAEI4oKbRPSwIgw/wgxXgZELwO7bzAz8IAEqYAhaYjkiggQQQ4EnYWi8YAA8AEUqQgge4hBKAAVD+KOcWcJ8NkAIvaIE/jgKgid06mAEruI0wcICpaOQEmIQ7cIBOGAMI+ASfAAQwSAQ4mYIN8CQrKOFGoCM7UIBNyAIOAIQSIF3dSxSuNcbO7BfM22GyGFSKIz/kW4EZaIoo4IDJxQQNaIRG8YOeIA9LyBD5AQQaiNuPiALA8QARaIFCiAIPoA9L2ABMEIRV2gIX+BUTWIEYyIQM21op6ABB+IORiIIqEAUNsANQII4OuANkmAIZ8AIZ+GMqmIEo4JpLgIGpOqB/zjMaKIFQIAQIgKANAIMOmARA2AATUIAsuIIrCIUZAAoPMAEG2IIwKIFRUKOSBgkTyABCgCzJAt7+TXQCYsCG0jbt00bt1L6G1GZtbMiGU62htB2HyKDZjJ1ThVCARvqDAvgewcZjTRCB5QUEB9CEDfCDib4NKfgEj4iCFwiFPwAAX2MAUBADDfi2SvCDEugEK3CBKSgB8o3mQgA5+/1r5QqbT4CBKAiFKTAQ9aZcL9A5EYiBUtpuMIiBX2YBP+AAq1gBMAiBEnqAzaICAIhbfh4FCLiCLLCCT5DjFmiBURCBLAgDDxCFCfcDnSsBE/DdsrXMVzCFWICFEBfxESfxEjfxEmcFWohPyUlbcOjJimPBTWABBEgmDjABDzAmEegpEcgaESgBBKiDFvAe2xADBECpKTgOT2j+AUORbA9wANTF7t24AhiYhA3ggATIgk8oATZ23HL+BA6Umk8QBAbggAyghEFQgA4gADB4Ew5QgESeXAhIhDqAkxfYhBXIAAYIgz/4b8QmsUAAADsQhSwojwfgIy9ADjCwA0PfgOxZAVHAAw+YBGgGgUrLADD4r5XWzFRoA1bwdE9vhU8X9VAX9VI39VJfBVq4JoaRhjiV6Yatiargqjv44ETIn6qgBDuYKFtX3VrLhKv4P6zIhJxohDuYhESgiljPBIEAiTFQlEywdZiCoEmAD4FYrjrYpJqyg9M7CD/g9oa2Fp2QhPOtBDsIhIfQCEvAiqk4v0D4g+aotY2oCkD+8ANKMASqyAR11z8VAgRByAnQnmUZAYdkCIaCN/iDR/iEL3hlqLpXYANYOPVSJ/WIp3hPVwUVt6aKjRiru8EE6doKHNujNSQOnEAUu1oNPD9GuRbf7a/uyrEZMySAJK2Yz1qAVAjVncCmFXkK5Mf8S/mxvY4RhZxiSAMzIIOjR/qkR/oyUPqkP4M0oIaK43SIp/hVWAVZiIVVUAVPX4UUh4Wtr/hUX/HIyQXU4fg5BUhL0CurndqdhSmmEy2F4A+VYDq6N7BYVvn6Akjtu1lHTZOAf5FiIIM0SIZuMPzDR/zEV3zDL4YzMIOonzZOp/hWMAVWYAZ3XQZWMAVToIX+bwgHbJCFU9h6Up94Vkj1a8qF46068qOzkZCEoJIuSdgDUBlAQ2KJTWqEQfAIQgiK8rsDmoLanwLRRWmJlZepnubZPwX8FukGNEgD2MaLbnh66Gc0yW+F6w/1618Fzv+Gb6CFZeh+WhBMZpCFZhAHZtB87M/+UD/9jOdhGG/Fj2gBAJAEqgEATHBdv+qEULiDB8CzzwKITZo2eSo48AUYUAMXDAjFAoCgRACifKq4iRIDK502fer0aZODEqA+UhIx6OMmTJg0NYIAYEMjTQPBAAhwRabKnDp38uzp8yfQoJ20IBFn9CjSpEqXMk0a7Ay1plKZFjMTdSrWrFqZgsv+BW5r0lRrWLVS1ersKlOwmolr1srUKlbMxGGLZeourW/faK3qe5aVKlWmaI0Da/gwLiaHFzNuPNXaiEw9P90BsGeTghaJWrTY0ylMlipXZiSiNGNFHU11vEyZQQlQzUQcYQCYxAFAmDoACk1qsWKQJhZhPtUpkUVMIg8QaNCg5AVAiUt79sjctCIAHgAtPG1qBADGFQB/PAUtb/48+qFFHR/mdKYw+6PdyhSLb1+qLwAVvh4WCzhwK6uossw34YwDSxxllWIKNsO4cZYqDzITTjawCBhYYIPdd18uii0WTjgbiogVZJLxdFEBURACQB0tOJDAAJOAAEABMDCQCBX+BmwAgB0tAJDARH4AABF3dxRwRQYETDFFAZRwgMACChBiwBSUBFATAH6sAIACE1EBAAeWePBBdQVwgMwDEXiiCSVihCLkHp+gNyeddKo3IlbuiThffXiK6MsEFUgQTX9jYdgXM98MA8s3dvXVVzbLmIKhKW+8EQs237gyaWBpyRKin4sl1hiooRoFnzj8xVdiT5l8skICU8QIhgw7EjLjJGEEQMgdL2x5xQoOaBKsJgmsAMpKlCSQAAc0cABBFIIAsAJtViRgRXia0HRHCBBsAgIIhbD4iR+ByKTJAB2I8gAD5GUCSmUerFQnvfX6dKepS7lXanzl0Jevfb5YIE7+LQDwYlgqagTYF1zXLLMGLOHIkiArcGEjqYCmxNLMMm6sEg4tpTy6SimyJKUqwEx1uFUvTeTijTW13KKNUdvw4ow40hCaVS/PKEWOV0Y9E00v0EQTjTSrRnavbgHIoIkDELSwqwcagBIGAYDAMAAMAWRRggiewOAAJgi0EMpK1k07yJWADAKAB1HMcAcBV3z5iR0B3OHBCp+U4MGKdXxSkEqbJACBKAU4AApBYQDwwiYp2Tv55PimfJR7qMbH5+WO+SLBV7kA4KFWqfCgSl+qiJwNM2u44k0sayyTzV0NilyKKpmGc0024MiC+6Mlg7rNEkAAkQtYu1hAszi2AIH+sjhM+IzVqFlps0Q0tTDxRS+JhbhLLUs4w4uquxxBBM5L2PLMEUBIs8sPQvgCTi1H9BLODz8coU0vvtTCyy6KMIFcbEMbtmAMq3oikyH5gRIAqAFt/ACBBIDiOXtQAASsAAApbCADnihBATARgQFIQiCboMkMNjEkTFBiAB6YAgRgE4NIBKABQPoDBDrwiQ04QCIl2EQJVlCdKXwHAGKgggMUMaQVlMAPkqMcFOdkuc7ta0Oc6xxjejGBbRjFGRJYwlYSprq+LIh1bFCFK1TRhmVgoxSluJjHcPcNbOwuG9/4HcnIaDKjJKEW4ejF8sQBDfIZJRq+0AYXbZY0Wxj+zCgCEAAXfdELowABGuLAmThuJo5xbANpRqkeVkIEDiLgognhsMYRuCiNJOSif9EAlRKU4AsmQEMC0FjCLnJhi1okwRlHgEYuckEEbfzAF+rDhS9wkQtl7oIJu3BGExC4NJ9sIgorsIQmpsCBF3jgD1SIwib2sIJGhKEDK2DBFpa0iSvAQBNh2MAdyKOJ5NhBbDMYiB86AIEpYKIEYLjEFFhwnT9MgQqb+CYmZrABSdDgaTKhRAseAINNWAFwHvAABzQQzyhy9DxTvFzmNuSvPmHxMFrkolHIESjmTSUVPRijG93AujXE4S6ya+MbafEGU5TCDbFgxjLeUArfCTX+j8IzChEsgDNwdAUIRzjCJSdQhAPwYhv6cx8uDgDVaEiACNuwBRGK4McjPOMWSRgHEogAhF5swwIS8MUnSYeVZxBhF+EgAjiqoQdVhUOXuDQKEnDmBVykQhxG+AoTlmDJJPjCFksogjQU84wm3CKZyuQFNGa5hOkdJoH3+oQmMBE5j2iiE57AJmhH+xHCEUS0AsmJJj6CiYJIhiCRi+0n/rABKWggAJPoiGg7MVuBcAe2nShtJjiC24qEtqPOBcpHUxbS+1xRKeAIB1NBZKBxGMhA1x0Hd7W7SX6x53MoNUo4hCAATErFdGkxBRta0QymMkMVOyUQM9xQimtgoxX+buApG9zAhgVd4xuz6ClP3bBHcYRjCROgAC7E0b9nHCAazgzHAHHhviMsIRdFIEI4cJGEImyjaLzYYhIOQAQJVzgXE4jGASYZV6201Rbb2Ib9ahHho+yiF7YAnzfEgYRa+MJ+SRCH/XKhPSZEowlN+AEgnwEEZzDBFrboxS2svIsIH2ECjfFseZrrE02YSIHmyYSYNWGJK5gTEGLWyZtzguY4P7fOQiFKScVRRer+KynaYAKgt7dkQH8B0LUgwhFqUYtUeEHRjn60LW4haStTutK2wAWmM43pXRDhANAjGABk3JRU6MAUNdVFOLAhC1l8YxzD4IaiYvoKOs7CDaf+cMN/TWHrCTXDFW5kwx7DQagSH8AZvvgBEizQiyTQjAm5WMIPinCELVe5lLlIZS70t58iCAAIzZPAEZAgvSOoCpRT0QaiicALbQChCCzlxsGgkQRLiqMJFDCCL7YR4WcUAQjR8CURmLFKJjDWefvzhTN6ofBogDEJR5Zmme1MuYEM5IkSvzhHowswTphBc+wZqVJE7IyRk5zkz2CGM6DRP2Mb2xeSlKTCe8GLmc98Fza/eTCDqWlMU7oWok5Kfg446lK/wWTLYMMa2BCHZYiDGadgQ4BL8QY3MD0b13D6TnXNBldg4xq4BrYoD7ALo/xgFxXA2QScUVdxTGAXtlD+jC+icYtaPAMAtYhGEQAFDm1sMe9FaOUPxKGNXUjD3UdZmamMcDAROcN9WGFpUsCM8Xqdi86Tv7yd8FzSPdsH5EnJxdibQt48JyUaAsArU1KRA/gO4xquG0Yp1nCKcbRiDW4YRiyg/gY1mKwZ4Ti6gIW6htb3lA2vOIoWgfCDJdCPCEmgQC7wzu1dbKMCRKiANHBRC3FIYBfkcPdjj2ABXjRBGtKYgDSKN0BtIIGl5sZTNCBvn2fQeyoUcHzkp4n5/fO//xrP13TdR58hRTCRXmNoAwVMgPwZheoJ2Cw4jDhMQ+ydwjfUnhuwRccEGBv0jjjMQhuwAXy9ARu0HtT+ARtS8M/OhMMzSMPe+cIzAQFcbcMhpcpX8EcNOsONaQN/oFTcGQXK4MItGODl4MKQJEHSHIXk9Z8SLqFz/Z+pBKBRFEMwTCEVVqEVXiEwDODhIY8QMsYSSMB5HYXqxYEJGhgbtIEazII46MIafOAsfMM1qFEbtAIzzALUvdrRtUHrCRjY8Yzz1cKnbUguCAE4HNLRaEO+sZUzmB8OspUhxl8ibsMiSgMO5tshDQ0ilpgkmh80aOIlxl8jSqI1SEMnWuI2YGIoTmIpaoMADAkAMJ9RJCETziIt1okThgrHaU44nAEZ9KIv/iIZmIEZ/KIwBmMvktQWduFioBsQhCH+A+JAHLTBG56CGhifHU1DODCDKawBD5SCq9neGvSAGqiBKzSdxOgACULdgmFFJwGMh6ngNpjfKYJDJ8afNkSDJILDM8SjNMyjM4CDPeLjP3aiPD4DOPxjQG5DJxKkNfgjOTyDNtyjN+AgQfbjPjoDOcSfNeAjRLYiAAiALfCHLNYiSZbkvWgeFkGhOJyBHnSDS3YDOcTkS84kTXYDNZABMhoF6CmjYTjD6QViKkDjGujCNaAhKsDhK7QBq3nDK6zBGmTDMKyBCFqK7FGIOHjDKehCNkCdGryCx/ngM7icLxyhdZHlUujj6DWF9vGkqTABAEiAjSHFSJokXZrkLfr+CecZxRlwAmN0A06GHPYMTTQMjWAS5r85w9EcjfktZkQ2ZkTe2I0xlWQyFXcdxi4AgNApRTgEpRsMJTaEoym8ghrwwBqowVNeQw+sATewYQl+YBu8AjPownzxThu0wRp45VLwwukRwdotBTRUgFRowy1AT9Asxfux5X3gDyzmX8TVpXPaJUpS0XsgxS7y5WKUw18mRfotwRI4WXdyp3cuQS20W2Jxp3mep8MlwRKkJ3u2p8M91REkwVMlARH40VIsAQCw11Kkwg24QVdG4Cn0gA64QjbMQg/kAFG+wjR0oBrUZm1ypQ7kgA7MRTO04W0yxS48nFEAk/Z8hS+oJ4n+8QIv0Qw0FAETMJUj4kIRPIMzBMD2GadcIeeGBKI4zOVz3qj/RSdITSdS7GVfZmdShMM2gMM2xOSQEimSEumQQiZkOqZjSkNjLqaU6ozOJGZiLoGnJUVbSYBZ7ucN7F5XYgM4DIMxjMM1oAIP3MAwbFIzYIMu9ICDniFXnsI0OIx/ouHxLYUWXRou9BUA4IIEEBBcSoARREMA2MIRpMIzTMAtxKc2QBa4JQEMVkDohVyMymi+2CiOburk3SWe5OUu6gFjeB6mKsUuUADKGOphYQV/1qYaiCOqbYOB6oAa6MI4ZMMr9EA2vOk49gCcliYbeMOZ8kCDvipuKgUvSED+oOkbVF3bygSQlBlFEQjBkUlDMSVBNAABzaCPEarMpZaqn8xlc/6E5enEuCphuc7JufKfp46ISu4lNXSDOChDMHCCvd5rMiSDHgiDOJQDJ/BrdYHr4aEqj/1p6fjAOJbmaGYDMdxAGlrDGgroGlyDLuRADpQCMeRqacYBU+aAU5ZmD+SpqYLRUYSPOFzbpUkYWT1cBQDB9kkDEchbNKQSkn1olx7FcQrsiIxk5a0ETvDEuUTcz4oWTgztunLU0JornEkGnaVZ074Wxe1EcyWtz1bea/1Eu4oIFO6iMABDOehZMgCDHFADNcgBJ/hrMfwrJ6QtNYwDkOrsyVoAfxz+QSOVjsOO46vqqjHcABtcQzO0QQ6Ioxpc3SzMQjZ4A1P6ahxsQ67i7WiK7OdJAM39WRE0DxE8g1L9APbFWC3gGwVQWRJYVeNpwzgUk7oxRc7C7X3IoifcwQIAQuR4wQMQAkGAFmptghTYgWl9xCdEwRW8lgNQRCdEgRV0hGwNREFwhOR4AmilTbr2xCb4gQNsgAZsgBUwjmtpAvOGVuSgBGghL2hRwgoUAkd8hJi0wEBgwgY8DUesyUqAgiS0gCUQREFoDSYQQgnAQCBYHE9k7YZwAhl43BkIgzBwQjLIgThQA7/qWVSsrTAEQzE8cExqoc7mAgUwWAVMAGdlRSr+ICze+irDOqwb9MAOEKs4SgMcZkPTiQMxBC4bMK4Jj2MOvEJa8psRGAERKNzY+QLypIJTRZ8FGA+h4MIPGAEL4oI14MJ14YI25EI0wajqhoosXg0ATAFBOAAAAAJlXAElHNcd1MEKhEEoAIIXTIIolAANuO+VDMIZw0AoDIIX0G4jFEIdDIIkiAH5Ri8YUMImSAIh0Mk6MQAlJEIiSEIgJEInEEIhaIIdZAElzBMhhIEkCIIYqEQh3EElU0IEkG8dhIFARAAAJIIoVIYIfEIjXMEdRI4l4AEYPIBodfInAAIHFIIDzIAYLAAhpKv/3oeeJMUAA4Mw6Os4KIN1GrD+OOiBAY8DNSiDHhTDOFCwwGYbNFDAEaRle3mwOIIwM9wADxyoGqBCD3QzxXazLnhDhe5ADsTBOLyCCfvqDMfHsnHIt0ax0pzrJ4CBmWRCIzCAAzTCFigAB3jAJVwBAmjAAABCIEDA31wCDETBmojvRX1CDExBJGRAC3TAJIBBAGwAA3TA9F5CHUCACIiAdIjAI6PHCUGAIgCCIGhCFHiAIjBAIIQBBHgAC1hHAnB0CCgAFQwCAZRABMSAJmxAIkRBBmxAFFACDQyAF4jC1lzBJGSABiQAGGyCBzxAAjxANWkAB1ABIJQAIGTBmiSAHeiyjkoXj6LXAAeDvAKDTS7+MDBEhTDABwFzgjK4bU4KLBEGwDy3lA/4KmDzgC6AAzMAbipswzWwQQ7wwK7iQIS2QQ/gwAyDgzXAAWCL4w5ALmNIw/3YR+rSs2PI4idcwUgXghec0x04wB2EAgi8AAf8QSMkwB5wQAzENBjMgENrgowUwgaEQRQktRgYggOMARhkgFWXgCZ0AE1PwSIogBVoQiGo6ybUwQCUAAe89iaIwABMgSfYgR2MQQIUwgrcEwGEgSCUQBgYgCRIAggAQguIQQHYwR4EQB1MwQtEQSOsQAuAQRRsACjUwQYgySZ4wQYAAgHUgR8QABaMkyg0QgfIC3SdNcA8xVUchY8yxVf+jkM5FEYyQDO49gIFVCpYhAMc/PVlH+grcMM2MMOKB6hkryY3R/YNwMFcMMNonvgMW7MBfjZoQxxPuAqwsEYLYAENgIEHUIIoSAEEbABBtABNg4AJsIAfyAA46XYGNMIeHDUVNMIHlMADNHIUOEQW+E0WOIAIBFEYGMScaMIWbMAlUAIlYMInEFEigAIYdEAJQAAhyIAdbMJCJYILnDZ3iIAY0MAUEMAKuIAI+AEMtIgMOPmCT8ExFALYyMAo4G96lwAL7PcKYEIhMEA7Pe8u20c3oAEC96h1GsY4pIGH97hWNIGJ4/gO6EKqnMIN4AAcLOg4VOwNIOgpoYJkn3j+ZGs2T/K4qxuGaJO2d3wA/sqyHYACB2SBBwDCJBjA3ogBKGTBZlT5JMTQJ8gAAFzBC0xBKHDAGGSBDPiNRniAGMDTaAOCJfzxnGxCFmiAcD0yIVSvZjAAIViCAxQCDISBJkBAIRBCGCcAJQyCAvwBC8juI1vBIJSAIEQBAIBBfZO2nbdAHTiAJ2QBBPyBA/DxFYxBCRDCA2RBJxytSoy6fQQDGaRB244DObAkeNW8zd+8zSdDGvBiXh87B/sADwS9r+5ACefAjDMDOBjDL+wOHNjqrlZDOPB6OPMA0ZMwOqOCjpOesfu8Vog2GKzAJkBAO5VAIoSBAnRAC3SCF2D+kAMAwh9EgD5dwgwYFJtwACB4AiUUwBbYgUJDgAxcwRR8QgtkgXX4wR3k9AZMgh+Y9PPuxCbYAQFoAARkUAukEATwNwfAihVEQVnPsiG8AORrgANQgSZ4QCNEwQM4QHO0ACDsAQJUghhEgSV0gAYwgB98UAZEAAhUkwM8wAwEAugzi1TDLtZKeMoEAxqcwRkUYzAKo/M/P/Q7PxmUwfQHQ9ZzvVI4wV9TPdWXcAnzgGS/gjdsQ8XiwIxPAzhMAxz0uvd7f9CDP9Zjai5oKPb7eE9YwiNbAjbFuSYAwh0QLUAECkQJ0yZCdyhtsmQJkyZMlCxp0gRx0yBAlAotnGj+KRPBinYiUmqEiWRJkydJZsLUaBAhQpJGbqJE0E8hjAsfktxURwOgQRIpSfTjByjJoJYSUrJTaJPCO4mCavKzB6glSS4JEURZspMWJOLAhhU7lmxZs2a7BePECZgwYG/hxpU71y21s3fx5tW7V28qDD12BBYcOEdhHD163MhBGAecHYsHD16MAxVfy5fF5iqCmXNnz3itjVC5VWJDh6clok6dSdOm05g0li7osGlq1A0b1ja9lXdJia5dl7Y93LQmT35gtF7dOrhD3MJbk1R+unlq4M55d/36mXt379/Bm00FJbLkx4Yl58BRnj2OVOPCc8fFZNu2+PfDhx6NUnn+74b7UzoJu6028cTApvxLUEHfXDMJwNya0krAARfc6kEFtcNPww05/M4JDHC4QcQRSSzRxBNN9KGyDi3DpRZopGFRxr30I40SQyLijbUKe7MjDD/uCEOQTTC5kMcEM7HEJgp9w6QQQQqRraRMGmmEySOxJCnDGbnscsZUboBDzDHJLNPMM9FEBRwv75pPHGlyWZNNNmtEyZNAHijkk04+IdK4PROxojVP+izIQE9IIpTIAj15DYYPAEjggzpAiaipTQotqM/frjypqQM/+aQRD/bwhLVNOvEE01TtgOAFGAjx09QCZ4iiE0wRNa5RRv38pNEsUdpyzmGJ/UyJX8D+2cabZZVd1ptummXWm/qc7WZaaZe1dllt4CtWLDe14UVOb1ms86ROAOEAEDus+GCMTgiRYQVFrgAgDEpmYOEOT+qwooUpNslkhhLq8GQQF1pohEhMPgEFhDBEASMQUO4Qo44rSgjDNStEwMMTSa6QpFPp/rjihSvq+AChEvzoRJNBvIBhijtW8GITKmCYSac9qOhkDzCmoGKTLErIYpNCrKAikTtcoKIOg1yAwUojeRSW3KuxJqsIaLKe0U2wcHmmaw3NNWmTQDwAJIESYGCAEg2imMKKLSLww4oo6nDAkhUGsCIAO6jYgAoOCOHACitKiKo1DrYYRQQYQvFghgz+NrDCgUk4zgOCPCaBIZGFedtkBgKsGACCyQdhoeVNxAhACgYMiEIBQGhAwIUwNMlkE0UUoOKBMK64AhB9IagDEABo8AMCKhLYgJINrkA8KGC58mrs668uwhnsOfz6TW1i5P67slM6G4Q9PIjSA0JYgCALljzAhBArRk+khSiOqYGFDgD5JJErCkAFKQAgEJ3IDQeuEAo7lCBdgBDBHT4RhStwwANeUIALEJSgTUQBBqLwgBU8kQE7tMAPqgLDB0ARgxaAogNgAF4dCnAFRPELAC34RNAwkQUrGOAKdnCAJ67QgmOIQQR2IAAVogAAQYxsQVYT3xNldITtQTE+3hP+xzNyQUXukE9LaNsDCxphiRIEohFZyMAM/CACTMDggw6QRAuscAwZsGB9n6CEFRAgBSrMQBJ+2gQCG+UBD5RgEx4QBChoEIMNrMAKUsCDbvyjiaCBYgV40EQH7OCCPfhKDDD4xAyo8AkQeIESnghFC1oQCuPYAQA1CMUUsoAHB3xhA2Cogwg+AQMYHOMOIgiDAoIWBZFRT0vW8wwTKnCZI9TiLuM4ggSgSQH7lAUaFAjfWWqRBLJoYxdaDEsRfOFN8FgxHODwhTbEiRkuYsITe3iAHTiQkQ3cAQY/wyQE7sCALABiAHcogRSOwTYprCAMItCcH+wQg0lQhwFYMFD+EsHgCQa0AA8KCEQLWDCJEiAEC8PszSZkIAMWekETDKiDB+5wBz+EoQSfaIEMPvGAK9AABnh4gB8EEQZFIIAKChADLGHAgUlAAAZggAAowpCAPHAgBIS4aR1aMD1iOvEyRABAOKKhDWfE6Bkx2obYwtGLcIpjAkcQB4zEEY1eoBMs4ZDABHYRV3Bkdatg0QYzcAGAaFRDq9EIiy96EQ5x4MIWc9WqfWoBgLFqsQjMSKd3rAiWXfj1sXxZ59GoMIiQWcIKjRDDBkRAFBhYwQ4e0KUXwIAHUGTBDpPwAAfC4IkwbGADj6TOzFDlBwNMAhMd2MAHN9EIEWwgBppohAz+rBTJMMTWCn+QJCCyoAgXEsILnRCDxqYAiEJ4QHCdsEMWxkAD2eLtKSYwwRRmMAYrEEkMNQgBBz4x2w3YQUrAoqpljnCAbVTgAAAQgDSK8ANxHKEC4BACAABQhHAAgQl5fYYvEGxNsICDAkBwhi+kAQ4LCMC/2gBHfwNwgGj8AMThtCoAfgCOJSThGQLo7w+0AQQAaNOb4KysfJhQFmfk+MZ5WWdDdqUSlwHHNUNGVZ+aYonaFGhRlzqJqjSRiBLEoBPPI4SvGjJkTvlHJQhyGSZcZuToIAjKn4IQolAVnFshWVV+eAEhXjADJCOIiRgyZmfyu40DECEaAsAFLwD+II0D4GIXAIBGLwDAiyNwuJsCoACgNyOOcVQAAAcQQBP0zGcB2AIXAnjGEQIQjQMAoc9EQLQvnAGAXSSBCM8AAC4g7IxeiFic2uuxZyILlmjwYhvjujVZftwbJjnnQlKqr2864YcoVGIiYEjEgOrMG+xQzULV9s1rGnLHF1DhErchZvW2g2f9SqCbFagFOABghAls4wgAAMsBmJAEBIcTABNoQhFwAZZxTGAJYflwN3+QBCAYQRypfoYEbCEOIjDY3eKQQBKSsDUA2AcAzniGfms9xV9fJtdhqQVlNz6WH2viE8Qm0Ld14itQpEY6xzaJy4P1K5RXqDURiTaW7sv+FyMIQM9Z5Lc45C3gvPLCFopdeBEEoA0J/EAaS+iFvifwaF70QtD5pkCDA9CLIuhVAhSAsBQBkItC8wIJREi1NLah2FRzzZtSDPllcsHjsoAP5G8XxzqNmwXFOSc1IiEOa2TDctuUZBOJYAEHXNDH3GA7d8OZCcsnpAkxkPLmM9+N5c91Z87UAggKD+cR8v0MCjxdHEyYgATynQReOJzXRPhBBbgRliRAc+kA/nw3gTCBHwBBGhUIwAQqYB9bvDXHtqgFNyggWAlToAniBEIupBF96U+f+tWnfjSsn33tSx/71e/+9bcvje9Lfwn9Pks08m13sKzTZwmIFSYU8Zv+Kh1HBI2ocpScM5LWUCL+EmlEH2mDECKgpkLgAUpJEvCPIWADNxKhIKJgBi4hJ1BiE1bAAWagA94P8zTwSHJOL7ZhF8QuF0JQ7EAwrniBF5iACU5QBHkhF3ChF3JhCXKBF+IKBF2Q00rQBluQCURQ1GzBF/7sBFNwBUMwrsSOF3ABF3ztiWyBCJzwCaEwCqVwCqmwCqWwCKYQC68wC60QCoGA9M4iHHJBbOyO/QIBBABBBkogAcSroBIgDKwAAMDADzaAA0JpcHwLExJBAx7AVsDAAYoHOGQABD4hFDDBCiYhEBjAAaTAE2KAUqIAeErAAVpAEhxgABIBDKiAQib+Ql1CwQGuAHQ2cBT9owPzQhuYYMXKLwlWkRVXzBVT0RVlsfxeEeJosQlUMRdZMRWXwBnAyRlecQlwMRdb8RWZoLDULxktYxeeQbDebp3uBA0TgAr8wAESIQHAQBDEAJ7KyA4aQQEawQUeIBEYwF+iQBFaoA4eQBAAwQGYYhNKoApy5RMsIQG+IBEcwIHAQBRE4ApEoAUIAZ+uAAY24Q/uoFOQqgAIofJIEfNMUYt6TRklkkN8wfxC7rLS5Q5YIChKIKcWoAQKYRDUSBBWQAQUwH6uABkGZgMaIVSsoABQKQGGhAI9SSjEAAIKRAayoAXCIBQYqZ5AAQaWKwoKxID++EMTYCAB7kAUG7IpTeIhJzIqpRIvtAFGyOEiRWMr7sQD/ACMKGEF9gAQJGEKRKAOSgATKicRIKARWMAKRkGFGOgT7AALFsAlqqCPNuElNWEUAOEB7oADLOETSuAPfCkU3GcG7GATKMoLqABRQucFVkATRCF3nLIywW0qMTMz+cIWyPDXoNEP3skDoGIDHKgFXuAFBiEBskAEViCJxGAFpiCgZkAMEmAFQiASNqAFSmAFuk0TLoEDLJAApiCXIKAGNmCDFOAF5LAFxsATREAMxkABIuEKpIApowwAcpNlmNIySREqNfM7pRIctGrj8I4S/EASLgIT/MASEiFuJmH+J+wgEaRADFKqJTZBEBbyCqbgcwphBqQgIaSDs4SyKSYBvWKlEoLHDxJhEEQGEBqBErygEACBKASEEupAh6zgJ7izMr0TPD1UGaWhmzwzK0ljVxxiVw7EOD7lQBDkUhTFUFTFbAyk5GA0V2bUyYIDRUkjVEJlOzd0Azv0Q4XU7nBB4yor2HrjQfiONFpO2gRkQSqP2n7U8oJ0SK301nwhGpZQnJB0Sr30Sx1S8650TJXRGRLuxroUTNV0TTlQTMn0TUMOHMRvmtIpTdn0TvE0WNwUTvn01ibrSEk0TwV1ULeiSvv0ULEnHLQhGR7LTgn1Ub3UUBF1UrNGGlavTgP+FVI19U4llVI9lVw4k0szdVNJ9Us79VNRlU3CAUboFIoctVRhVQNPNVVplUuiQUSpSD+wY0lxo0me1FcjCViZ9FelA1ijLfCCtVeLdVmhrSE1YVZrNVo7ZK5alXuswQSGI1u1dVu5tVuHozmqw1vFdVyto1zJ9Vy/1VynY12D4xP2VFrhtVh44ZrExxoYoA/4IF/1dV/5tV/99V8BNmAFdmAJtmAN9mARVl8RoQsILl4dllzCQRqMFHsMTAgs9mIxNmM1dmM51mKBoGNBNmRFNmM/dmRN9mQxtmRR9mKHYAh+IP0eNmaHJRoWS2Zt9mZxljtsgV5ztmd99mfDImL+pYGtgLZojdZmnwFXj3ZpmbZWqbVpoTZqEXUceIFopfZqsXZIaTZrubZrM3NrvTZsxTYZbUEbnJFcwCEIUmBt2bZt3RYFUMBt5TZu5ZZt4dZu6bZu9fZu9XZt87Zu+TYFArdv3/ZvAZdwA/du4RYF1HZu2VYFznRsu1ZRtcFsr8YaGgARNHdzObdzDcEQOjd0RZdzP3d0TddzQfd0Vbd0Vfd0Wbd1R/d1YRcRUjcRGFZyxTYZwNBbdLXxKPM1AC8ThBfwWo7vSoN4V0MliLdILk84cCN4hzfwpNchkFcihLdIZEMlpPc5sDd3otd4wbd7i+N6lzd4k1d4vdcT3hX+d6M2HLbBGc6WWBxVSjeUfrnTfp0SWtn3ZsOB18jlVWM1gL9Nf/f3Zp0hi4oFgAV4gXFufQsYaqXBF7ahW+ZEgRn4gu0s3B64a2+hWrvEgjE4hAvVgTeYad0XfIYFhEV4hYtJg0s4awErhUeVhWk4g1/Ya8EBHMSFTma4hn14hF34hrG2ankYf3+YhglYiGWW6LxEhY94gZNYiR/2GZzBw7jEiZ84gKNYih+WPq64h7P4h7eYi+N1G6ShMzsEi8O4VMeYjOHVFy61XMB4jZGYhN14acfhFqyWbOaYjle4je9YWuf1KjlEjf0YUgE5kKMVF24hjfv4kDE4kRWZVq3+IRq0dEMMGZIHVZInmVZrwYPBI5M1OU85uZNRdRzA9j5EeZQ51Y5N2WihoWbz45FZWYtd+ZWNlgl41jt+LIOMg0K6jImkpEA0wVJ41TeMWFOYDOZUQsl89NqiA0CuN8tO48ukQ3szSFmfo5pdhjL1NIhxGWrNKRpA+TO4SCIkSFUIYQVaskFQRZJipRO+7FOaghA+RyJes5QEIQoSAsoKgph1Y5ln4+X+8r2swJkXJUZ1JxSAp0VpA8peghCqQCKgjDnkmZ5w4xFBIUaZIwzqYFD8iJgpQRA6oUBlABBgrpTD+VPjKj64KBM+QQYSQBNeyQHUMxAcghIWNAwUZg/+iCITJCER7KAgQMAFum0TpgAA3FJ5KmET7gClH4IQiCIQUJpI6oApVkKqCoIGHGAQ8sABsEAmfoISxuDZgmsRXoAFLkFkKqGPLNRKSqAFFOFpMuEO/oBIGsEQiIIQBqBlACEABmES6kBkZmIQ6qCEGmGwZWISAAEQhgYCIoEDNsALIMCxgXiluTZZnmFLPeOyGiEBAAEUNIAKpqAOwUATPGAAkGgQsOABOiCUOiADGKAEGsEAEmCh8tK3EoEQQFIGNqADwqC3IGABTAAECsAPNkEEHmADSiVjQKcTZqAFkAEUOAAG6qA26RACMiARFIEBOCAArEAMZgAUvGAGkvv+AYrHAd6QKFtAARJADAhhbQZgCkDhA6IAFKigIylHBDIxAVxACupAEeqQAxLBDxRABAggDJKoDrxgEighAezAMTMPnDEbanOhnDljnTLBE0LACvARELwgUCBAExRACiwBk+AbDBRgExiA2xJAEWKgEVsjCrAADFYAEEpgEDx8CjIAExDADlhpEqLbCxzgEqKAATzBQbFjaArABDRAAyRBChbgElbAhqIbBl7gEuDGC1Lpbj5rE7wgDEAJEFygDtyGEBwgCxAgEczcE8RAAz4BuP2AlFRTCh7AyK2AEMCAEkDRDgzgEqwABATBAySTEBAgAQD0myu8a1N5i8BYtlj+IApEwBOqgLs2QBM2oI864A/AAAQ6AAIowZA8iHaGkzWiQAY6YQNYoAWQhrs8wBI2ABNI8hO8gAW42gREwAUwARRAR3eqAJ/qoKlpXBREKBTuAAI4wA+QgQpkwAtiIBSuYI+iwFcuIQYyNAamIJU2AQRoYAVCISCNawNmqxEIQQRKAAGyIHhEYQZCpgRMABtZahTuAAQCgQUwQRJGunJ8VKUXnVKhYWI7G4wnIgEGwA4AAQE2YbY2IQMW8gOqIAIw4Q8gIBM44A8+QV1aIJRmvIPuAABq5ofAgAMw3TxZAIjupwRAwQ9C0cH3A7pXQBRqI25cigpGwQqmoASw4Bj+RECCSkAUVoAGwmAFPuG6aMALCMEF7sADPEHKvOADPgEQRrwTpIAAGpEDZsAwwcALrOCTwgAGWIAfsyAMPkAU6gAEmEYTegIZiKtBJrzfuTYcamGPM3yOkzsANEESIODwFOAvfwIEzLIESqDgWeYTSFPNzToKXoDX1xAQOmB/INwDJsEPCMkKWEAtRSABtt6X+E50cAnwZN4P1gYCEuEPDAAEAOAKCgEBEJwgNQAEujqproAFCmkDHGAOO+ATAmHEP2EPAKAORIEKIKAF1obdP4EGsiBvWOABGGns66ADahsPqAABPoAl9/2W3d5nwaGrMJxG+tjvDEIMQvIlIiL+EagEvlvCwaOMIfZgmGCiNBaSEMRgQV/iIaxkEvqoEbwAuTVBEDyqSKoEOwBC0iRNmgBlEaRpkx8xhChtEhQmkaRNicIU2qTpDqFJmxqJqaMJkyJNlBJhCknopCY7dSQFSlRJk8BNdsZIApQoEclEmwAN2oRHjENMRIsS7aQFibilTJs6fQo1qtSpVKtavYo1q9atXLt6/Qo2rFVfvMBaG5HJqNqTm4hueosRo0q4GAliqiv3bsi7bTd1epswU0K2fOXmJUqwbVG7Cf+yBVy471u2geESDDmY8mPAIfHSbTsYMNy1RZEqFYs6terVrFu7fg17Kzho4LyeTUs6MOH+3Bhx71Vr1y3musGLHjb+WyVuowmTa17rnHTR5dKjS78u3XTs7dy7e/8OPvzTcLy82UYrvVEjgpQKWXKuadIgStYX/05En1Kjk40kJRfkn1oCMRddW+v99lYl2C3IYIMOZpeUeBJOSGGFFooHjTNd3QbdJi0MMIkoLQBACSif9OUJKHdAAIgmn5yYkCeefJJQFGHQqMkDCYDyAgCDNDKAFDOCsoklD2DxFigzbrLBCiZuQkkLhXgy2V2SgABACfRhksklXjhASH0PjkkmdtpdiGaaaq7J5lPPPLMhemt9UgcAfmySgJZSSCFIJyyJAcYMJl0Rwx6a+FGHF1ZgMgj+AAVEssknI1LCAQBj4AEAIZZIIaiHN/ohQx14SMLBBlRYoUkYALCgySCCsNVCAFYAAIMnmlgyYqaKlclrr6Sd2Wawwg5LLGvg1FKbVhyuBSUBUxQCQB0wIFAAApRgGUANDyRiRQAMAHBHDY4CQIUfAAAACJV4FLBFBgFQMQUBlIgQAAEOJGJAFJcQcK6dKwAwAABYSAGABvOKkFgBHCDzAAO2YiKIDAP85GvFFgNbbMYab5yVNtqIE04vvoTDVDTSiAPOx1Y9s81T4Gi4lDTaPCPzU9rUnNWya3lSAgPdNpLFDJUOgqUiYQRACB4wjJgFCwpIUoIDmCiwAo6UJPD+AAcvhKABDYIA4AINAFihgBVgADDJrHeI4MAlHHBACAA34mEHWwNwIMoDC3jC5SdhDCDIrhYPPibGHB+O+OHOWICLOEzooUQuS/WSiy3S8BJNybvc8rEvvWyDiy3h7JJELZ0zIQ04RSyxS8jO1NJLL0sQ4Uw44Pgiji+45yznzqoSAIMmHTCwAtIgZBBKGAQIMgMBLgSQRQklfAKDA5ko0EIoXG4iAgArBBIAuo1yMIMMdhBwBcGh+BHAHh5UvQII0Nph4ieIKZDBMQgkEArfm2QxMcERboALMlziDojANmmjCZIDQjiscYSW+SIJt4hdU5JQASbUQhoHqFwSiLD+C1z8gAlLeMYSmJAEcPxgCT/YxS58gYtc4KIWSUiCL55xhKW8LFlW0ZlRBIOJc92BEgCQQRQAEAgNLOATZ7uDATZwtilwQAOdKAECMJEAAwxlE14AwAw2cS5MEHEFYODA12YALQ88wE4ZAMEnNgCBuLVgEzCIQWLERgUAXMELGcBEJ2YFCAEScJBrMWACD4lICvEiF+EwQsqY0DJxPEMbu8gFE2B2hLIwIRc5LMI2wLGEJeDuCM/YRS2IEI0kiMMZS6gFDGPYi2jkoha46MVStlEWrPhQLXTkAH1iAIEVQGAPUXiBJ+wAgkSAAQIe8MAVZvBFKrDqS3bwXyEgMAb+T6ygBQmxg45gcC0vbCIKHPDAAP5gxE3MAAaYaMEDGsECVhFkEiKg2jgT4JAxcKAQYiLkIA2ZyIAKdDu5YCQRtgGNWpBsKdDYhS1e17Ik2PIIphQHEVLWhCbY8oRFcEYRpJHDXjChlriI4Qu3cQQL8DB3u6vKLpmziYfFpROJeU9cGpMQyagEppLpTF8aEwgPWKFSjaASX5qTU8FcRi+a6ARiaOrPqBYyQgOtqlVfU0ny/KAC0GCKNnKxjV5kchziWMIEiIALaSxBHLwgQgV6sQsVECEXvjACEyjQiyIA4QfOeGEM6VoEcRyBCE6JxjNWKpWXXqefmqAOcPq5mMX+UCIKD/DAHYoTWehEB7JSHSBArwra0GoFHMlyRuaYkjKQndaiTIgGySL5jK6KQxq1m200tqGNl52MtLydrTaI0FKmbMMWZKWKYjtLpsE0x7HIbW6ZPiva6EpXLLtYLWpgCBXb4SyxvXMur5bq3fDyyhNUna55z3u4XsiWu8wVr4M46974lqa86K2vfdsUjm0E9ynHla9//8sr6N53wAT+Di6n0l8AK3jB1xFwgR8MYdfwIpf87S6DL4xhtTg4whzu8FcMq7Jo8DDBGS6xfzfs4RSr2CrasIU4tCEAmImDxCaucXhRvOIc69gp2jgCAChMYxsLubM43rGRU7yNaPj+GAC1YEqQhwzlf9L3yFQ+8hL6BQASJOvJUf4vfAlX5CqLecDa4MWSAXCyGVt4MQ+7C1SZexy+FMWocgaOWnDT3gdpgkp/sc5eGoudpjrHMUShUnszwyXC5DTOGp7ymB+t4mjYQgC5VKwn/KCBQKZKfm+hUd82YYU7/IWmnqgCGOTCASucCAtXoJJRP2PUGA0nuYHYgAdAwIErEEkvmYipT6nka77YihIw4ImM2rKCGdilBFGI0cNcJAkZvOfYhIgCJghRAynwxEyOhrS3O6yNSCrWb+SK1BoBAQpAhMESNA2EH0oghlAoog6WEIUIYgDVcyXC3i8IRUUasYlJSGL+D/ixA8A3MYi6QWk/Y9rEFRRACHcnoiEUWU8g6B2fRgyxEHUTIyAuSwmIb2IPd8AIAgDQCFGYywOfoEQYAtmWO4xBape+wycAkWkEzIAKIHhPg7v97aBzeNxiGIAHSJKACDQiDAh4wAo2AQYEQMAAfyAEBDpQAjpGgaaTKEEGVhAKGExBEhkQgQcoIQYDaAACIkiABy6xIg6s4BJ7EAElGu6FDVxCEhORwQomAYFAjIEBG4CBhxgAgbIj4AqFcLsCmr0BSVAB8VWwRAwCIIZRyCAAWLgECBiwgDpsYgUMwNomsOAACBzke17wBB4csCUInUbotO/wuK9gdmWuYAX+e3BAHT7RARlw4A6KQID7aAB4MdBACjSVhAcGkYE6RCEKHqnEA+qwhQa47QOU2EAdNICFRCzgCpcABJk2UYcBiKBUgfC8AmTQiTHsYQ8JaMQKYGCJAYAhECKoQwEUkQhw4wJhoAB+AAgFYAdTwAJRMAkuQEZUkAF+wwFg8ACXYAUZMAgFcAeDgABWsAKYMApeEADy9HOzV3snWGAvlQmfcAUrYAVWIHYyUEaUIApT8AAb0Ame0AJhAAEcYAIlYAdGxHUZ0Ajf5wFUQAkrwDZ2kAVSICJZ8AkrkAUM4AEl8AFg8CLnBwYZEAk5QQmhkEeNkDzNBAGDEAPVtAGNkAj+LAAGK0AlJRAoUkAAJbACwwcDddACUdACYIAFJTAFx9AIIvABM3AMhCACYWAAJcACUrgChnAJlIAHEHAnJYiClUhgt1cChTAAJQAILPAHHRAIouABWOABioBFyCQGoAAGiQADWxcfGZAuPXIFrRgKIsCENAAKK3AF3CMGGiBqWWB+DPcg/9MBoQAXidABDkADlPAAgEAJEMCKY6AJEFAIhMACYcAAmtAICeAHLXAFsIcJVzAIJSAIR3QFdXAqKyAKeLiFyZMBfsAAA5EFdcACYuABJndZlGiJ+4het/cBeMJNHlAIWaAA7wNqCZABChAId+B2DhAJrTiEgPAJjaD+R2MAASyQADFgBTMQhbvoAXsQBgnAARAQCX5gd192F3UQABuwdlfQAjVwCQyQBTDwPgVgBTEgehlQjS5gBwOwAQygbBvAihAAATUwCd9jBwHQCGAgA5TQRhFgB5vAAR2QABCgCTBAlDDwByXQCB+QeDQAaLLHj2M5XYpFEusxCfTRCPRxB2EwFHVgB5NgCQoBBgAnCfSRFonwHpqAHxlhB5IgCGupjWo5l3cgBhxBCYNQJpZACPPnB4SAHx2RlmOAE4OQltqICZZACXYAAXZwWXwZEmEAEoOJCeuhHxQhBoGQEC4HCAFSB2GQCZmgE5RQB5/JbSZIlrlZVcdlF3L+cRmQkVON1Wu+iSDIsWh1oRyeMWvJRRfNwVSc0Zt9042R0hmJQRyCsQlKVRjDMRmWcRnBWUBAp5vjiUC8mRxh+WeOhZJ8MWp+ZjHL0ViMQRTq2RyTkGgEIl9hRp77GSxmWZrYAUS9EhKAsAeEQKAjcWd55iDMJQk+Vx2ToAgHcmeUEHvipZ/8iaFpolg9IZREsmsu8gmd0AheQBBL4mafwDeZMCOY8SKY0U4gEABuRz9zOWonchQfymjQMSN/oSRQ4wdO1VQoGil/4Qcb0AIwECYn4T+fEAVT0AmdAApAKiMJIaQn0aKDc6EZqqUUolidUGuA4Adh0AKilghRAAP+k5AFAQASVAADgNAJbSkDWYARWNACgeAJiWBEQ3EXn8ABqRgGhPAJF9eWY4oRbegHniAJYTAQCtoqYTADYbAHLeCMJfCjmlAIdSAFVwAIMBAGnsCml6CXCbEHYNAJ6iYFVOA317gJhRAGdbmpYLAHmxBtUaCnvpKlW4qr4NGlQQUIbrcCUuMBMgADWOAFCnAHV/ACWWCVxRMFAXCsEBADHBAJg0gDLDCX2AkCWyCKMRAKwZoBGzADDGAJXtABXiCJksAC23Yd6kQAzXORGmCNP9oJYhAAMZAACgADCcA8DAAD+dgJhIAAfJQFV3AFgmACYgABdwAIANACfpABNKD+ABugCSAQBTOwAg7aK7eaqxu7HbvqAX4AAi0CNx6wAWHQCIDgAYySBVgwAI3QAjQwCi3QAh1wqIFwBQWQBYAEGlJ5BaIwBh9ACBzwBx7wezRAriaQB/bEIAkxA9nTAVSwCZ3pAocKdRxAPdMDRYECBgYABp6QCfPqPZ8wBVQwrmLQgQz5Py0gCl5gWQOwsj4iSIUjnhxLtxKyoSh7By6wH5w4WQtABfyHCVGgATPgAC57BcfgdwJJJViAADAwAzEAcGxRrp/gNkqoCc8HCuvEASJAAzMAEjnKHFMwBbkIEh2ABw37ImAAA5+wc5/gTHsmKdmzZ+YiA6EgBVlwBw7+MAOqhwciQD2EeAclwHQxYETrUTEaW7fJmxoq2Al/0JkCaQnDJwOFMAbelwGBkAARQQCAUAJSMAr3Z6Z3sAJ1UIaAMAMDwRYOQAUyIgMA0LUL4DUJIL4xQEdDBAYBUh2aAE2gAAJioAm+VwJ3EAjqVjUtwJEQ8ExSAAgZYAeEcAfbGAMKgAdUkAUvIAKb0AEzgLChAAYMIAgl4AGC8ACE8Af4t55ThZvKq8LfsaGEMAOAMHaWEAWJcAXMhAeYwAJYEAYc0AIlMLBhEApWEBEQsAGtV8MQIAa+uQk0IHqXRgD+AUcbwHyEsAFfB6otELmLBQZdOwWXJQOBUAWDkAX+YAAIVtAJV3BqNPAHgAABGiADmzAGQswCnnAFMhAGDVxONMCpM9AJlnAFLLABHcCCRCkGCuogyLvCibwVvHlTTGVTzbGzcgadfbEbxoERkiCzULIBgXNTc5kYnNWd1XlTodEWNAXJ30lT3vnJTbUJgUADigADqwsaJ/wrc6vIt7y8ayYdvuEcaYFZiPEbv8wlTXUHMGCajHeetIyfiwXMS+uiJzEJUuACDIgZyozCuIzNr8GbnrauFdMZKAoKiYEYhizMHWLNC7oYAYdonmXL2ezOi6zL2uiW4HUSJ6GZzawSweHL1WwUATcDJTADlzDLe+HLibYXFFrP0oER83H+zslVzhazCe38zhPdQ2v2tX6QAIVAU/SxmpbgCYCwAkPB0eyxVBwNJQ6BGYkQTHyUASJtFwVN0g5HBc9MIKzIABPRZQtCXilM0T2tS7pcqiFLBS3AAE5qB1FzB1gAAGKAsrnmCVZQAiCAMFcCAVQACpG4AZOoThEYCpPVCFb3AFfApDY3BXUgBu4kA5PgAANQEagCHStSf3Gb0/PF0z5t1wimy54QCCGbAI/KAI0gk3cwjzp5BRGBTxiJvTMJA++2In9wBxowEaOHBXzjIpjwAFHwBwwwCB4ABqKIewH8AHcw1BkhjaThCYKQTNk512J5163tUrrcyiAgvutRAoP+cAUJ8AKNsNnXBgMiYACJ4I3HQAMl0AGJgKJWYAAyu71vsQIxgCPqBgF+Eac7GAq6CAN7wCNhEAZTMBqk0cockMWrfc2uTd5RoYKt/JEssB+8RwiEMNxHPbFRAAgPMAnBHQMtIAISGQhYEAF74AdUENlTgACbMArQVwcbgAlRuAclUAehkAFBI2o76AXr+9BeCgLhLd5GgcjlvcKW5gcOYAf7ZAlZzWwWCwgRUEYwcAVyw2zHUAJx+gAwAAKKkAFREAMswNHPuAFkQwM6yAFRkAGbIAMMcERZ0ALZJAJiEAYLIAl8aB2esAcMcBENzWAbzuHKa5aNYHBDhAl2kJj+LSADPAEGfwoDQlwHgJAuewAImWCmhPDRR4rTCTEJFosqHSEDddoRTWoRfsBPe1AIlDAFhuAHduBnRVihGU7XV67oapZne+bomGArMWVUwIaiJ/IW7IYRL4IiNmoclX4rkRJrL3IiNOXRJRoji9VmiK7hEr3o7sxlw0wgLw0dy0zrsD4dqk5IVt7qG/vquO7rib7r5N3rv+7ruh7sWzrsxK7qxn7sGZrsyp7hzN7s/Pns0L7a0j7t5Fnt1p7T2J7turnt3B5l3v7tZBnu4j5k5F7u/HgWCmYJvPLuRBHvFzbv805AO73uFO1D8Enl73VnzuwrmxUS8V4cBP8b9o7bzwl969+1Z6ye73RrDRDgCIxA8RVv8ReP8Rmv8RvP8R3v8R8P8iEv8iNf8ZDQBYH18O4MDiSAAi3v8i8P8zEv8zNP8zVv8zeP8zmv8zvP8zBPAreQ8hMdDkNP9EVv9ONg9EmP9EnP9E0fDkvv9FDf9FLv9ErP9FQ/9Fj/9FW/9Vzv9UMf9GEv9mNP9mVv9meP9mmv9mvP9m3v9m8P93Ev93NP93Vv93eP93mv93vP933v938P+IEv+INP+IVv+IeP+Imv+IvP+I3v+I8P+ZEv+ZNP+ZVv+Zev6AEBADs=\n",
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import Image\n",
    "file_name1 = './img/variant.gif'\n",
    "Image(file_name1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Solution \n",
    "\n",
    "You use a Jupyter notebook in Amazon SageMaker to create an endpoint that hosts two models (using ProductionVariant). Both models were trained using the Amazon SageMaker built-in XGBoost algorithm. For more information about how the models were trained. In the following use case, we trained each model on different subsets of the same dataset and used different versions of the XGBoost algorithm for each model\n",
    "For detailed instructions, please refer to this [official notebook](https://github.com/awslabs/amazon-sagemaker-examples/blob/master/introduction_to_applying_machine_learning/xgboost_customer_churn/xgboost_customer_churn.ipynb) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np                                # For matrix operations and numerical processing\n",
    "import pandas as pd                               # For munging tabular data\n",
    "import matplotlib.pyplot as plt                   # For charts and visualizations\n",
    "import matplotlib.ticker as ticker\n",
    "%matplotlib inline\n",
    "\n",
    "from IPython.display import Image                 # For displaying images in the notebook\n",
    "from IPython.display import display               # For displaying outputs in the notebook\n",
    "from time import gmtime, strftime                 # For labeling SageMaker models, endpoints, etc.\n",
    "import time\n",
    "from datetime import datetime, timedelta\n",
    "import sys                                        # For writing outputs to notebook\n",
    "import math                                       # For ceiling function\n",
    "import json                                       # For parsing hosting outputs\n",
    "import os                                         # For manipulating filepath names\n",
    "import boto3\n",
    "import botocore\n",
    "from botocore.exceptions import ClientError\n",
    "\n",
    "import csv\n",
    "import io\n",
    "import re\n",
    "import s3fs\n",
    "import mxnet as mx\n",
    "import seaborn as sns\n",
    "import pickle\n",
    "import gzip\n",
    "import urllib\n",
    "import csv\n",
    "import shutil\n",
    "import tqdm\n",
    "import urllib.request\n",
    "from tqdm import notebook\n",
    "from multiprocessing import cpu_count\n",
    "from tqdm.contrib.concurrent import process_map\n",
    "\n",
    "import cv2\n",
    "\n",
    "import sagemaker  \n",
    "from sagemaker.s3 import S3Downloader, S3Uploader\n",
    "from sagemaker.predictor import csv_serializer \n",
    "from sagemaker.predictor import json_deserializer\n",
    "import sagemaker.amazon.common as smac\n",
    "from sagemaker import get_execution_role"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Preparation (Specifying Sagemaker roles)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sagemaker session : <sagemaker.session.Session object at 0x000001BBAD316F08>\n",
      "S3 bucket : sagemaker-us-west-2-570447867175\n",
      "Prefix : ML-ABtesting\n",
      "Region selected : us-west-2\n",
      "IAM role : arn:aws:iam::570447867175:role/SageMakerNotebookRole\n"
     ]
    }
   ],
   "source": [
    "sm_session   = sagemaker.Session()\n",
    "bucket = sm_session.default_bucket()                     \n",
    "prefix = 'ML-ABtesting'\n",
    "region = boto3.Session().region_name\n",
    "role = 'arn:aws:iam::570447867175:role/SageMakerNotebookRole' # pass your IAM role name\n",
    "sm = boto3.Session().client(\"sagemaker\")\n",
    "sm_runtime = boto3.Session().client(\"sagemaker-runtime\")\n",
    "\n",
    "print('Sagemaker session :', sm_session)\n",
    "print('S3 bucket :', bucket)\n",
    "print('Prefix :', prefix)\n",
    "print('Region selected :', region)\n",
    "print('IAM role :', role)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Upload our pre-trained models to Amazon S3. Create Variant models and deploy those"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This code uploads two pre-trained XGBoost models that are ready for you to deploy. These models were trained using the XGB Churn Prediction Notebook in SageMaker. You can also use your own pre-trained models in this step. If you already have a pretrained model in Amazon S3, you can add it by specifying the s3_key. \n",
    "\n",
    "The models in this example are used to predict the probability of a mobile customer leaving their current mobile operator. The dataset we use is publicly available and was mentioned in the book Discovering Knowledge in Data by Daniel T. Larose. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Define where the models are located in S3 bucket. Then upload it to S3\n",
    "You use these locations when deploying the models in subsequent steps."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As stated above, we trained each model on different subsets of the same dataset as **'xgb-churn-prediction-model.tar'** and **'xgb-churn-prediction-model2.tar'**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('s3://sagemaker-us-west-2-570447867175/ML-ABtesting/xgb-churn-prediction-model.tar.gz',\n",
       " 's3://sagemaker-us-west-2-570447867175/ML-ABtesting/xgb-churn-prediction-model2.tar.gz')"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_url = S3Uploader.upload(local_path=\"./model/xgb-churn-prediction-model.tar.gz\",\n",
    "                              desired_s3_uri=f\"s3://{bucket}/{prefix}\")\n",
    "model_url2 = S3Uploader.upload(local_path=\"./model/xgb-churn-prediction-model2.tar.gz\",\n",
    "                              desired_s3_uri=f\"s3://{bucket}/{prefix}\")\n",
    "model_url, model_url2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "'get_image_uri' method will be deprecated in favor of 'ImageURIProvider' class in SageMaker Python SDK v2.\n",
      "WARNING:root:There is a more up to date SageMaker XGBoost image. To use the newer image, please set 'repo_version'='1.0-1'. For example:\n",
      "\tget_image_uri(region, 'xgboost', '1.0-1').\n",
      "WARNING:sagemaker.amazon.amazon_estimator:'get_image_uri' method will be deprecated in favor of 'ImageURIProvider' class in SageMaker Python SDK v2.\n",
      "WARNING:root:There is a more up to date SageMaker XGBoost image. To use the newer image, please set 'repo_version'='1.0-1'. For example:\n",
      "\tget_image_uri(region, 'xgboost', '1.0-1').\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'DEMO-xgb-churn-pred2-2020-07-29-15-18-50'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# You use these model objects to deploy on production variants on an endpoint.\n",
    "# You can develop the models by training ML models on different datasets, different algorithms, different ML frameworks, and different hyperparameters\n",
    "from sagemaker.amazon.amazon_estimator import get_image_uri\n",
    "\n",
    "model_name = f\"DEMO-xgb-churn-pred-{datetime.now():%Y-%m-%d-%H-%M-%S}\"\n",
    "model_name2 = f\"DEMO-xgb-churn-pred2-{datetime.now():%Y-%m-%d-%H-%M-%S}\"\n",
    "\n",
    "image_uri = get_image_uri(boto3.Session().region_name, 'xgboost', '0.90-1')\n",
    "image_uri2 = get_image_uri(boto3.Session().region_name, 'xgboost', '0.90-2')\n",
    "\n",
    "sm_session.create_model(name=model_name, \n",
    "                        role=role, \n",
    "                        container_defs={\n",
    "                            'Image': image_uri,\n",
    "                            'ModelDataUrl': model_url\n",
    "                        }\n",
    "                       )\n",
    "\n",
    "sm_session.create_model(name=model_name2, \n",
    "                        role=role, \n",
    "                        container_defs={\n",
    "                            'Image': image_uri2,\n",
    "                            'ModelDataUrl': model_url2\n",
    "                        }\n",
    "                       )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create variants\n",
    "We now create two variants, each with its own different model (these could also have different instance types and counts).\n",
    "\n",
    "We set an initial_weight of “1” for both variants: this means 50% of our requests go to Variant1, and the remaining 50% of all requests to Variant2. (The sum of weights across both variants is 2 and each variant has weight assignment of 1. This implies each variant receives 1/2, or 50%, of the total traffic.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "({'ModelName': 'DEMO-xgb-churn-pred-2020-07-29-15-18-50',\n",
       "  'InstanceType': 'ml.m4.xlarge',\n",
       "  'InitialInstanceCount': 1,\n",
       "  'VariantName': 'Variant1',\n",
       "  'InitialVariantWeight': 1},\n",
       " {'ModelName': 'DEMO-xgb-churn-pred2-2020-07-29-15-18-50',\n",
       "  'InstanceType': 'ml.m4.xlarge',\n",
       "  'InitialInstanceCount': 1,\n",
       "  'VariantName': 'Variant2',\n",
       "  'InitialVariantWeight': 1})"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sagemaker.session import production_variant\n",
    "\n",
    "variant1 = production_variant(model_name=model_name,\n",
    "                              instance_type=\"ml.m4.xlarge\",\n",
    "                              initial_instance_count=1,\n",
    "                              variant_name='Variant1',\n",
    "                              initial_weight=1)\n",
    "\n",
    "variant2 = production_variant(model_name=model_name2,\n",
    "                              instance_type=\"ml.m4.xlarge\",\n",
    "                              initial_instance_count=1,\n",
    "                              variant_name='Variant2',\n",
    "                              initial_weight=1)\n",
    "\n",
    "variant1, variant2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Deploy the models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "EndpointName=DEMO-xgb-churn-pred-2020-07-29-15-19-23\n",
      "----------------!"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'DEMO-xgb-churn-pred-2020-07-29-15-19-23'"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "endpoint_name = f\"DEMO-xgb-churn-pred-{datetime.now():%Y-%m-%d-%H-%M-%S}\"\n",
    "print(f\"EndpointName={endpoint_name}\")\n",
    "\n",
    "sm_session.endpoint_from_production_variants(\n",
    "    name=endpoint_name,\n",
    "    production_variants=[variant1, variant2]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Invoke the deployed models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can now send data to this endpoint and get inferences in real time. For this post, we use both approaches for testing models supported in Amazon SageMaker: distributing traffic to variants and invoking specific variants.\n",
    "\n",
    "Amazon SageMaker distributes the traffic between production variants on an endpoint based on the respective weights that you configured in the preceding variant definitions. See the following code where we invoke the endpoint:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sending test traffic to the endpoint DEMO-xgb-churn-pred-2020-07-29-15-19-23. \n",
      "Please wait...\n",
      "........................................................................................................................Done!\n"
     ]
    }
   ],
   "source": [
    "# get a subset of test data for a quick test\n",
    "print(f\"Sending test traffic to the endpoint {endpoint_name}. \\nPlease wait...\")\n",
    "\n",
    "with open('./data/abtest/test_sample_tail_input_cols.csv', 'r') as f:\n",
    "    for row in f:\n",
    "        print(\".\", end=\"\", flush=True)\n",
    "        payload = row.rstrip('\\n')\n",
    "        sm_runtime.invoke_endpoint(EndpointName=endpoint_name,\n",
    "                                   ContentType=\"text/csv\",\n",
    "                                   Body=payload)\n",
    "        time.sleep(0.5)\n",
    "        \n",
    "print(\"Done!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Invocations per variant\n",
    "Amazon SageMaker emits metrics such as Latency and Invocations for each variant in Amazon CloudWatch. Let’s query CloudWatch to get number of Invocations per variant, to show how invocations are split across variants:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "cw = boto3.Session().client(\"cloudwatch\")\n",
    "\n",
    "def get_invocation_metrics_for_endpoint_variant(endpoint_name,\n",
    "                                                variant_name,\n",
    "                                                start_time,\n",
    "                                                end_time):\n",
    "    metrics = cw.get_metric_statistics(\n",
    "        Namespace=\"AWS/SageMaker\",\n",
    "        MetricName=\"Invocations\",\n",
    "        StartTime=start_time,\n",
    "        EndTime=end_time,\n",
    "        Period=60,\n",
    "        Statistics=[\"Sum\"],\n",
    "        Dimensions=[\n",
    "            {\n",
    "                \"Name\": \"EndpointName\",\n",
    "                \"Value\": endpoint_name\n",
    "            },\n",
    "            {\n",
    "                \"Name\": \"VariantName\",\n",
    "                \"Value\": variant_name\n",
    "            }\n",
    "        ]\n",
    "    )\n",
    "    return pd.DataFrame(metrics[\"Datapoints\"])\\\n",
    "            .set_index(\"Timestamp\")\\\n",
    "            .sort_values(\"Timestamp\")\\\n",
    "            .drop(\"Unit\", axis=1)\\\n",
    "            .rename(columns={\"Sum\": variant_name})\n",
    "#            .drop_duplicates(subset='Timestamp', keep=\"last\")\\\n",
    "\n",
    "def plot_endpoint_metrics(start_time=None):\n",
    "    start_time = start_time or datetime.now() - timedelta(minutes=60)\n",
    "    end_time = datetime.now()\n",
    "    metrics_variant1 = get_invocation_metrics_for_endpoint_variant(endpoint_name, variant1[\"VariantName\"], start_time, end_time)\n",
    "    metrics_variant2 = get_invocation_metrics_for_endpoint_variant(endpoint_name, variant2[\"VariantName\"], start_time, end_time)\n",
    "    metrics_variants = metrics_variant1.join(metrics_variant2, how=\"outer\")\n",
    "    metrics_variants.plot()\n",
    "    return metrics_variants"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Waiting a minute for initial metric creation...\")\n",
    "time.sleep(60)\n",
    "plot_endpoint_metrics()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/jpeg": "/9j/4Q5fRXhpZgAATU0AKgAAAAgADAEAAAMAAAABAgkAAAEBAAMAAAABARQAAAECAAMAAAADAAAAngEGAAMAAAABAAIAAAESAAMAAAABAAEAAAEVAAMAAAABAAMAAAEaAAUAAAABAAAApAEbAAUAAAABAAAArAEoAAMAAAABAAIAAAExAAIAAAAiAAAAtAEyAAIAAAAUAAAA1odpAAQAAAABAAAA7AAAASQACAAIAAgADqYAAAAnEAAOpgAAACcQQWRvYmUgUGhvdG9zaG9wIENDIDIwMTkgKFdpbmRvd3MpADIwMjA6MDY6MTIgMDk6MTU6NDgAAAAABJAAAAcAAAAEMDIzMaABAAMAAAAB//8AAKACAAQAAAABAAAB9KADAAQAAAABAAABCQAAAAAAAAAGAQMAAwAAAAEABgAAARoABQAAAAEAAAFyARsABQAAAAEAAAF6ASgAAwAAAAEAAgAAAgEABAAAAAEAAAGCAgIABAAAAAEAAAzVAAAAAAAAAEgAAAABAAAASAAAAAH/2P/tAAxBZG9iZV9DTQAC/+4ADkFkb2JlAGSAAAAAAf/bAIQADAgICAkIDAkJDBELCgsRFQ8MDA8VGBMTFRMTGBEMDAwMDAwRDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAENCwsNDg0QDg4QFA4ODhQUDg4ODhQRDAwMDAwREQwMDAwMDBEMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwM/8AAEQgAVQCgAwEiAAIRAQMRAf/dAAQACv/EAT8AAAEFAQEBAQEBAAAAAAAAAAMAAQIEBQYHCAkKCwEAAQUBAQEBAQEAAAAAAAAAAQACAwQFBgcICQoLEAABBAEDAgQCBQcGCAUDDDMBAAIRAwQhEjEFQVFhEyJxgTIGFJGhsUIjJBVSwWIzNHKC0UMHJZJT8OHxY3M1FqKygyZEk1RkRcKjdDYX0lXiZfKzhMPTdePzRieUpIW0lcTU5PSltcXV5fVWZnaGlqa2xtbm9jdHV2d3h5ent8fX5/cRAAICAQIEBAMEBQYHBwYFNQEAAhEDITESBEFRYXEiEwUygZEUobFCI8FS0fAzJGLhcoKSQ1MVY3M08SUGFqKygwcmNcLSRJNUoxdkRVU2dGXi8rOEw9N14/NGlKSFtJXE1OT0pbXF1eX1VmZ2hpamtsbW5vYnN0dXZ3eHl6e3x//aAAwDAQACEQMRAD8A9VSSQcvGblY78d7nMbYILmHa4fApKXuyKqA02EjedrQAXEmC+A1gc76LHOUPt+P/AMJ/21Z/5BU/2ezFGFQy610XvIsc6XSab9Xaem/+3X9P9J9NXN2XV9IeuzxbDX/5jv0b/wDPr/4tAmuh80gX1V9vx/8AhP8Atqz/AMgl9vx/+E/7as/8gp1ZFVpLWuh45Y72uHxY6HIkhIEHY2ggjfRB9vx/+E/7as/8gl9vx/8AhP8Atqz/AMgrCSKmv9vx/wDhP+2rP/IJfb8f/hP+2rP/ACCsJJKa/wBvx/8AhP8Atqz/AMgl9vx/+E/7as/8grCSSmv9vx/+E/7as/8AIJfb8f8A4T/tqz/yCsJJKa/2/H/4T/tuz/yCX2/G/l/9t2f+QUMkxZ/Sm4/8lwaZ0Gv6RCuftwbnjPY1wI25Ja0trMt2+qxm3dW538776/0f+ErSsdxfbqrXsa79E/2/G/l/9t2f+QT/AG/H/l/9t2f+QWQzJtNUt65jk7hDn1sgN3gNZZ+kZ+sf4F/0P+K9RWMC82ZIH7Qbmt22wGtAEj7M5vuqHo7qWudv/P8A1r/g9iSnUpururFtR3MdweONDypqt0/+jf8AXLf/AD5YrKSn/9D1VByhkOx3jGcGXx+jc4SJ80ZAzMpuJi2ZLmWWtqbuNdLS958mVt+kkpqNHUA7B+2ms2/aH7vTmI9G/aGzt/6n+b/lrSWZbkNzm4FlRto9S98OgBw21ZDfduFtfuViOo1f6PJaPjU//wBG1WO/7YRpBKe2iq4RY0OA4nkHxafzUL0sio/oX+o3/R26/wCbcPf/ANuesm/aFLNMhr8Y+Nohv/b7N9H/AIKrDHse0PYQ5rtQ4GQfmE0wG+x7pEul34IRlsadtzTQ7+X9E/1bR+j/AO/o4MpEAggiQdCED7Ls1x3Gk/ujVn/bR+j/ANa9ND1D+t+Ek+k+H4hsJKuLrqv5+skf6SuXD+1X/Os/8ERa7qrG7q3B7eJbqiJA+fZRBGv4s0kkkUKSSSSUx26khxE88KL6W2MdXZ72PBa5rg0gg6FrmlqIkkprHp+EW7TTWWlxeQWMjcdS/wCh9NTbiY7LTa1jW2EFpeGtDiCdzm7w3dtc4IySSmr07+ij/jLf/PlitKr07+ij+vb/AOfLFaSU/wD/0fVVF72VsNljgxjRLnOMAAd3EqShbUy6t1VgljwWuEkSDz9FJTWutqutwn0vbYz13Dcwhw0pyAfc1XFnuwsXFfgV49Ta2MudsAHE03iG/wBlrWf8X+jWgkpZV39PxXOL2M9Kw8vqJrJ/rent9T/rispJWogFqeln1fzdzb2/u3N2u/7epG3/ANl0/wBtfX/SKLK/F7R6jP8AOp3Wf9u1Vq0kjfdFdijpvovbupsbYByWEGD8lGzGqe7fBbZ/pGHa7/Ob9L+2muw8a5wfZW02DiwaPH9W1m2xv+ch/Zcmv+YyTHZlw9QfJ/6O/wDz7bEDGJ3/ABSDIf2M5y6vDIb8mP8A/ST/APwFSryarHbJ22f6Nw2u/wA130v7KF9pya/5/HJHd9J9QfNjvTu/zK7U4uwsv9FuY9w1NTxDx5+lZtsZ/moGMhsb89f+criid9PLT/mtlOq3oXVa02bh/o7ZcP7Nn863+36qcZbWaXtNJ/eOrP8At0e3/tz00OKvm0/L7U8PbX8/sbCSYEEAjUHgp05CkkkklNXp39FH9e3/AM+WK0qvTv6KP69v/nyxWklP/9L1VCyq7bceyul/p2OaQ15k7T+97HMd/wBNFQcu6yjHfbVWbnsEisTLtfotgO9ySmm3HyKHYTL8h2Q4ZDzvIAkGm+N3/mH/AJ69i0ln+vfc7AfbSaXOveHMJmIqyA3936a0ElKSSSSUpJJJJSkkkklKQ7qKL27bq22NGoDwHD/pIiSSmp9idX/Rr7Kv5BPqM/zbtz2t/wCKsrS359X85S29v71J2uP/AFm47P8A2ZVtJG++qK7aOcLcMOArsdhWk6VvHpgn/irf0Vn/AFn/ALcVj1civ+dr3t/0lUn/ADqT+k/7b9ZHcxr2lrgHNPLTqCq/7PpZrjufjHwqMN/7Yfvx/wDwJN4R0uP/AEU8R6+r/pJq76rRNbg4DQx2Pg791TVGzGywQ5wryHAQHtJotA/rs9Rr/wDwGtRGbZQYvDmt/wCGaGED/wAMVb8N/wDV30peobi/GP8A3nzKuPev73/ffKn6d/RR/Xt/8+WK0qnS3B+E144c6wjg6Gx5/NLmq2ip/9P1VRe9lbHPe4NYwFznEwABqXEoGc/LbjOOG0Pu7NMHT+QHOrY57fpbH2Vsf/pK/wCcVU9Mdk31ZWbY4Prhxore7094Dq93u/Msos9O7H/mbP6/6RJST16OpPb9jyNrsS3c8hszLLafbv8Azfc79J/waN9nyv8AuU7/ADGf+RRKaaaGCulgrYOA0Qqt56jZmGlrWjBcyHWBxa/3CHbXsd6td1cezZX/AIT+eqsp/SpS2VccRm6/NLdC4NDGlxaC1r3hjW7trPUZv/cRa6sp7A8Zm5r/AHMc1jANp1Z+9u9v5yFidJxsdkWudk2Gz1nPtMj1dvpOtrr+hTvZ9L0/6/8AOWWK5YX+m70tpsg7A6du6Pbu2+7akpF9nyv+5Tv8xn/kVWZk+tkHFpz5uA3fzbSCGu22ta6Gt3t/rfo0I4GX1LHYOoudj6y6up5Y4t9trGv9J72U30XtZ6d9Nz7P0P8Ag/XfWtGuimpz3saA+x257+5MNZJP9WtiSmH2fK/7lO/zGf8AkUPIL8ao25Gaa6wQNzmsiT9Fv0fpOT5r+otfUMJjHtJPqF/aBLd2rdtb/o+pX61rH/4Cz9J6QMfpmKcjItde7KNgNNlbnlzW17jb6FrNzt+yyy7+d/Selb6H80kpLim/KqF9OYXUvHs/RtBBBc2zfP8AK9u3Z7EX7Plf9ynf5jP/ACKOA1vtAgdgNFmmnqmWLqskjGYHg1WVHXSPzNzm3UWs37vtHp+/9H6NnpsvsSmV2SKchuK/P232QGt2NMF3836kN9vqO/m9384rIx8yBuyiXQJIYwCe8CHKOH07Cw2MbTWC6pgqba/327G/QrdfZute1n5u96fPfmtpBwmtdYXAO3QSGnQva1z6mO2fn/pGfo/ofpP0ViUxubbRW627N9KtmrnvbW1o/rOcEPGsty9zqM0ltTnV2tNbQ7cI28hu3/M96ZnTN+W3NybHm0NIFLXuNbTY0MyWCdvq02bKttT2emyyr1/5xXaq6aa21VNFdbAA1rRAAA2hJSL7Plf9ynf5jP8AyKrZeUzCLRk55YTHt2MJgz7y1rfaz2u96e1vUr77qHbasRzYZcxxa8E7tr63Mdv9Vr9vqV7GVbP8NZ+korJidMxcZjAR6r6/oOfrtkBrm1T/ADdf8hJSfDxzjY7aXO3kFxLgA2dznP8AoN9rfpIyjISkJKf/1PT/AHpe9fLaSSn6k96XvXy2kkp+pPel718tpJKfqT3pe9fLaSSn6k96xs3/AJu/a7Ptcfad3uj1Z3R+Z6X+F2f6P9KvnRJJT9A/9ifpiA/btZGw37tu6nZHpD1P537P6n8v0/WVrpv7C/aP+Tp9X0Tt27/S27j6mzd7N/7/AP1tfOiSSn6k96XvXy2kkp+pPel718tpJKfqT3pe9fLaSSn6k96XvXy2kkp//9n/7RX2UGhvdG9zaG9wIDMuMAA4QklNBAQAAAAAAA8cAVoAAxslRxwCAAACAAAAOEJJTQQlAAAAAAAQzc/6fajHvgkFcHaurwXDTjhCSU0EOgAAAAAA5QAAABAAAAABAAAAAAALcHJpbnRPdXRwdXQAAAAFAAAAAFBzdFNib29sAQAAAABJbnRlZW51bQAAAABJbnRlAAAAAENscm0AAAAPcHJpbnRTaXh0ZWVuQml0Ym9vbAAAAAALcHJpbnRlck5hbWVURVhUAAAAAQAAAAAAD3ByaW50UHJvb2ZTZXR1cE9iamMAAAAMAFAAcgBvAG8AZgAgAFMAZQB0AHUAcAAAAAAACnByb29mU2V0dXAAAAABAAAAAEJsdG5lbnVtAAAADGJ1aWx0aW5Qcm9vZgAAAAlwcm9vZkNNWUsAOEJJTQQ7AAAAAAItAAAAEAAAAAEAAAAAABJwcmludE91dHB1dE9wdGlvbnMAAAAXAAAAAENwdG5ib29sAAAAAABDbGJyYm9vbAAAAAAAUmdzTWJvb2wAAAAAAENybkNib29sAAAAAABDbnRDYm9vbAAAAAAATGJsc2Jvb2wAAAAAAE5ndHZib29sAAAAAABFbWxEYm9vbAAAAAAASW50cmJvb2wAAAAAAEJja2dPYmpjAAAAAQAAAAAAAFJHQkMAAAADAAAAAFJkICBkb3ViQG/gAAAAAAAAAAAAR3JuIGRvdWJAb+AAAAAAAAAAAABCbCAgZG91YkBv4AAAAAAAAAAAAEJyZFRVbnRGI1JsdAAAAAAAAAAAAAAAAEJsZCBVbnRGI1JsdAAAAAAAAAAAAAAAAFJzbHRVbnRGI1B4bEBYAAAAAAAAAAAACnZlY3RvckRhdGFib29sAQAAAABQZ1BzZW51bQAAAABQZ1BzAAAAAFBnUEMAAAAATGVmdFVudEYjUmx0AAAAAAAAAAAAAAAAVG9wIFVudEYjUmx0AAAAAAAAAAAAAAAAU2NsIFVudEYjUHJjQFkAAAAAAAAAAAAQY3JvcFdoZW5QcmludGluZ2Jvb2wAAAAADmNyb3BSZWN0Qm90dG9tbG9uZwAAAAAAAAAMY3JvcFJlY3RMZWZ0bG9uZwAAAAAAAAANY3JvcFJlY3RSaWdodGxvbmcAAAAAAAAAC2Nyb3BSZWN0VG9wbG9uZwAAAAAAOEJJTQPtAAAAAAAQAGAAAAABAAEAYAAAAAEAAThCSU0EJgAAAAAADgAAAAAAAAAAAAA/gAAAOEJJTQQNAAAAAAAEAAAAHjhCSU0EGQAAAAAABAAAAB44QklNA/MAAAAAAAkAAAAAAAAAAAEAOEJJTScQAAAAAAAKAAEAAAAAAAAAAThCSU0D9QAAAAAASAAvZmYAAQBsZmYABgAAAAAAAQAvZmYAAQChmZoABgAAAAAAAQAyAAAAAQBaAAAABgAAAAAAAQA1AAAAAQAtAAAABgAAAAAAAThCSU0D+AAAAAAAcAAA/////////////////////////////wPoAAAAAP////////////////////////////8D6AAAAAD/////////////////////////////A+gAAAAA/////////////////////////////wPoAAA4QklNBAgAAAAAABAAAAABAAACQAAAAkAAAAAAOEJJTQQeAAAAAAAEAAAAADhCSU0EGgAAAAADRQAAAAYAAAAAAAAAAAAAAQkAAAH0AAAACABpAG0AYQBnAGUAMAAwADUAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAfQAAAEJAAAAAAAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAAAAAAAEAAAAAEAAAAAAABudWxsAAAAAgAAAAZib3VuZHNPYmpjAAAAAQAAAAAAAFJjdDEAAAAEAAAAAFRvcCBsb25nAAAAAAAAAABMZWZ0bG9uZwAAAAAAAAAAQnRvbWxvbmcAAAEJAAAAAFJnaHRsb25nAAAB9AAAAAZzbGljZXNWbExzAAAAAU9iamMAAAABAAAAAAAFc2xpY2UAAAASAAAAB3NsaWNlSURsb25nAAAAAAAAAAdncm91cElEbG9uZwAAAAAAAAAGb3JpZ2luZW51bQAAAAxFU2xpY2VPcmlnaW4AAAANYXV0b0dlbmVyYXRlZAAAAABUeXBlZW51bQAAAApFU2xpY2VUeXBlAAAAAEltZyAAAAAGYm91bmRzT2JqYwAAAAEAAAAAAABSY3QxAAAABAAAAABUb3AgbG9uZwAAAAAAAAAATGVmdGxvbmcAAAAAAAAAAEJ0b21sb25nAAABCQAAAABSZ2h0bG9uZwAAAfQAAAADdXJsVEVYVAAAAAEAAAAAAABudWxsVEVYVAAAAAEAAAAAAABNc2dlVEVYVAAAAAEAAAAAAAZhbHRUYWdURVhUAAAAAQAAAAAADmNlbGxUZXh0SXNIVE1MYm9vbAEAAAAIY2VsbFRleHRURVhUAAAAAQAAAAAACWhvcnpBbGlnbmVudW0AAAAPRVNsaWNlSG9yekFsaWduAAAAB2RlZmF1bHQAAAAJdmVydEFsaWduZW51bQAAAA9FU2xpY2VWZXJ0QWxpZ24AAAAHZGVmYXVsdAAAAAtiZ0NvbG9yVHlwZWVudW0AAAARRVNsaWNlQkdDb2xvclR5cGUAAAAATm9uZQAAAAl0b3BPdXRzZXRsb25nAAAAAAAAAApsZWZ0T3V0c2V0bG9uZwAAAAAAAAAMYm90dG9tT3V0c2V0bG9uZwAAAAAAAAALcmlnaHRPdXRzZXRsb25nAAAAAAA4QklNBCgAAAAAAAwAAAACP/AAAAAAAAA4QklNBBEAAAAAAAEBADhCSU0EFAAAAAAABAAAAAE4QklNBAwAAAAADPEAAAABAAAAoAAAAFUAAAHgAACfYAAADNUAGAAB/9j/7QAMQWRvYmVfQ00AAv/uAA5BZG9iZQBkgAAAAAH/2wCEAAwICAgJCAwJCQwRCwoLERUPDAwPFRgTExUTExgRDAwMDAwMEQwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwBDQsLDQ4NEA4OEBQODg4UFA4ODg4UEQwMDAwMEREMDAwMDAwRDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDP/AABEIAFUAoAMBIgACEQEDEQH/3QAEAAr/xAE/AAABBQEBAQEBAQAAAAAAAAADAAECBAUGBwgJCgsBAAEFAQEBAQEBAAAAAAAAAAEAAgMEBQYHCAkKCxAAAQQBAwIEAgUHBggFAwwzAQACEQMEIRIxBUFRYRMicYEyBhSRobFCIyQVUsFiMzRygtFDByWSU/Dh8WNzNRaisoMmRJNUZEXCo3Q2F9JV4mXys4TD03Xj80YnlKSFtJXE1OT0pbXF1eX1VmZ2hpamtsbW5vY3R1dnd4eXp7fH1+f3EQACAgECBAQDBAUGBwcGBTUBAAIRAyExEgRBUWFxIhMFMoGRFKGxQiPBUtHwMyRi4XKCkkNTFWNzNPElBhaisoMHJjXC0kSTVKMXZEVVNnRl4vKzhMPTdePzRpSkhbSVxNTk9KW1xdXl9VZmdoaWprbG1ub2JzdHV2d3h5ent8f/2gAMAwEAAhEDEQA/APVUkkHLxm5WO/He5zG2CC5h2uHwKSl7siqgNNhI3na0AFxJgvgNYHO+ixzlD7fj/wDCf9tWf+QVP9nsxRhUMutdF7yLHOl0mm/V2npv/t1/T/SfTVzdl1fSHrs8Ww1/+Y79G/8Az6/+LQJrofNIF9Vfb8f/AIT/ALas/wDIJfb8f/hP+2rP/IKdWRVaS1roeOWO9rh8WOhyJISBB2NoII30Qfb8f/hP+2rP/IJfb8f/AIT/ALas/wDIKwkipr/b8f8A4T/tqz/yCX2/H/4T/tqz/wAgrCSSmv8Ab8f/AIT/ALas/wDIJfb8f/hP+2rP/IKwkkpr/b8f/hP+2rP/ACCX2/H/AOE/7as/8grCSSmv9vx/+E/7bs/8gl9vxv5f/bdn/kFDJMWf0puP/JcGmdBr+kQrn7cG54z2NcCNuSWtLazLdvqsZt3Vud/O++v9H/hK0rHcX26q17Gu/RP9vxv5f/bdn/kE/wBvx/5f/bdn/kFkMybTVLeuY5O4Q59bIDd4DWWfpGfrH+Bf9D/ivUVjAvNmSB+0G5rdtsBrQBI+zOb7qh6O6lrnb/z/ANa/4PYkp1Kbq7qxbUdzHcHjjQ8qardP/o3/AFy3/wA+WKykp//Q9VQcoZDsd4xnBl8fo3OEifNGQMzKbiYtmS5llram7jXS0vefJlbfpJKajR1AOwftprNv2h+705iPRv2hs7f+p/m/5a0lmW5Dc5uBZUbaPUvfDoAcNtWQ33bhbX7lYjqNX+jyWj41P/8ARtVjv+2EaQSntoquEWNDgOJ5B8Wn81C9LIqP6F/qN/0duv8Am3D3/wDbnrJv2hSzTIa/GPjaIb/2+zfR/wCCqwx7HtD2EOa7UOBkH5hNMBvse6RLpd+CEZbGnbc00O/l/RP9W0fo/wDv6ODKRAIIIkHQhA+y7NcdxpP7o1Z/20fo/wDWvTQ9Q/rfhJPpPh+IbCSri66r+frJH+krlw/tV/zrP/BEWu6qxu6twe3iW6oiQPn2UQRr+LNJJJFCkkkklMdupIcRPPCi+ltjHV2e9jwWua4NIIOha5paiJJKax6fhFu001lpcXkFjI3HUv8AofTU24mOy02tY1thBaXhrQ4gnc5u8N3bXOCMkkpq9O/oo/4y3/z5YrSq9O/oo/r2/wDnyxWklP8A/9H1VRe9lbDZY4MY0S5zjAAHdxKkoW1MurdVYJY8FrhJEg8/RSU1rrarrcJ9L22M9dw3MIcNKcgH3NVxZ7sLFxX4FePU2tjLnbABxNN4hv8AZa1n/F/o1oJKWVd/T8Vzi9jPSsPL6iayf63p7fU/64rKSVqIBanpZ9X83c29v7tzdrv+3qRt/wDZdP8AbX1/0iiyvxe0eoz/ADqd1n/btVatJI33RXYo6b6L27qbG2AclhBg/JRsxqnu3wW2f6Rh2u/zm/S/tprsPGucH2VtNg4sGjx/VtZtsb/nIf2XJr/mMkx2ZcPUHyf+jv8A8+2xAxid/wAUgyH9jOcurwyG/Jj/AP0k/wD8BUq8mqx2ydtn+jcNrv8ANd9L+yhfacmv+fxyR3fSfUHzY707v8yu1OLsLL/RbmPcNTU8Q8efpWbbGf5qBjIbG/PX/nK4onfTy0/5rZTqt6F1WtNm4f6O2XD+zZ/Ot/t+qnGW1ml7TSf3jqz/ALdHt/7c9NDir5tPy+1PD21/P7GwkmBBAI1B4KdOQpJJJJTV6d/RR/Xt/wDPlitKr07+ij+vb/58sVpJT//S9VQsqu23Hsrpf6djmkNeZO0/vexzHf8ATRUHLusox321Vm57BIrEy7X6LYDvckpptx8ih2Ey/IdkOGQ87yAJBpvjd/5h/wCevYtJZ/r33OwH20mlzr3hzCZiKsgN/d+mtBJSkkkklKSSSSUpJJJJSkO6ii9u26ttjRqA8Bw/6SIkkpqfYnV/0a+yr+QT6jP827c9rf8AirK0t+fV/OUtvb+9Sdrj/wBZuOz/ANmVbSRvvqiu2jnC3DDgK7HYVpOlbx6YJ/4q39FZ/wBZ/wC3FY9XIr/na97f9JVJ/wA6k/pP+2/WR3Ma9pa4BzTy06gqv+z6Wa47n4x8KjDf+2H78f8A8CTeEdLj/wBFPEevq/6Sau+q0TW4OA0Mdj4O/dU1RsxssEOcK8hwEB7SaLQP67PUa/8A8BrURm2UGLw5rf8AhmhhA/8ADFW/Df8A1d9KXqG4vxj/AN58yrj3r+9/33yp+nf0Uf17f/PlitKp0twfhNeOHOsI4OhsefzS5qtoqf/T9VUXvZWxz3uDWMBc5xMAAalxKBnPy24zjhtD7uzTB0/kBzq2Oe36Wx9lbH/6Sv8AnFVPTHZN9WVm2OD64caK3u9PeA6vd7vzLKLPTux/5mz+v+kSUk9ejqT2/Y8ja7Et3PIbMyy2n27/AM33O/Sf8GjfZ8r/ALlO/wAxn/kUSmmmhgrpYK2DgNEKreeo2Zhpa1owXMh1gcWv9wh217HerXdXHs2V/wCE/nqrKf0qUtlXHEZuvzS3QuDQxpcWgta94Y1u7az1Gb/3EWurKewPGZua/wBzHNYwDadWfvbvb+chYnScbHZFrnZNhs9Zz7TI9Xb6Tra6/oU72fS9P+v/ADlliuWF/pu9LabIOwOnbuj27tvu2pKRfZ8r/uU7/MZ/5FVmZPrZBxac+bgN3820ghrttrWuhrd7f636NCOBl9Sx2DqLnY+surqeWOLfbaxr/Se9lN9F7WenfTc+z9D/AIP131rRropqc97GgPsdue/uTDWST/VrYkph9nyv+5Tv8xn/AJFDyC/GqNuRmmusEDc5rIk/Rb9H6Tk+a/qLX1DCYx7ST6hf2gS3dq3bW/6PqV+tax/+As/SekDH6ZinIyLXXuyjYDTZW55c1te42+hazc7fsssu/nf0npW+h/NJKS4pvyqhfTmF1Lx7P0bQQQXNs3z/ACvbt2exF+z5X/cp3+Yz/wAijgNb7QIHYDRZpp6pli6rJIxmB4NVlR10j8zc5t1FrN+77R6fv/R+jZ6bL7EpldkinIbivz9t9kBrdjTBd/N+pDfb6jv5vd/OKyMfMgbsol0CSGMAnvAhyjh9OwsNjG01guqYKm2v99uxv0K3X2brXtZ+bvenz35raQcJrXWFwDt0Ehp0L2tc+pjtn5/6Rn6P6H6T9FYlMbm20VutuzfSrZq5721taP6znBDxrLcvc6jNJbU51drTW0O3CNvIbt/zPemZ0zfltzcmx5tDSBS17jW02NDMlgnb6tNmyrbU9npssq9f+cV2qummttVTRXWwANa0QAANoSUi+z5X/cp3+Yz/AMiq2XlMwi0ZOeWEx7djCYM+8ta32s9rventb1K++6h22rEc2GXMcWvBO7a+tzHb/Va/b6lexlWz/DWfpKKyYnTMXGYwEeq+v6Dn67ZAa5tU/wA3X/ISUnw8c42O2lzt5BcS4ANnc5z/AKDfa36SMoyEpCSn/9T0/wB6XvXy2kkp+pPel718tpJKfqT3pe9fLaSSn6k96XvXy2kkp+pPesbN/wCbv2uz7XH2nd7o9Wd0fmel/hdn+j/Sr50SSU/QP/Yn6YgP27WRsN+7bup2R6Q9T+d+z+p/L9P1la6b+wv2j/k6fV9E7du/0tu4+ps3ezf+/wD9bXzokkp+pPel718tpJKfqT3pe9fLaSSn6k96XvXy2kkp+pPel718tpJKf//ZADhCSU0EIQAAAAAAXQAAAAEBAAAADwBBAGQAbwBiAGUAIABQAGgAbwB0AG8AcwBoAG8AcAAAABcAQQBkAG8AYgBlACAAUABoAG8AdABvAHMAaABvAHAAIABDAEMAIAAyADAAMQA5AAAAAQA4QklNBAYAAAAAAAcAAwAAAAEBAP/hDcNodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvADw/eHBhY2tldCBiZWdpbj0i77u/IiBpZD0iVzVNME1wQ2VoaUh6cmVTek5UY3prYzlkIj8+IDx4OnhtcG1ldGEgeG1sbnM6eD0iYWRvYmU6bnM6bWV0YS8iIHg6eG1wdGs9IkFkb2JlIFhNUCBDb3JlIDUuNi1jMTQ1IDc5LjE2MzQ5OSwgMjAxOC8wOC8xMy0xNjo0MDoyMiAgICAgICAgIj4gPHJkZjpSREYgeG1sbnM6cmRmPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5LzAyLzIyLXJkZi1zeW50YXgtbnMjIj4gPHJkZjpEZXNjcmlwdGlvbiByZGY6YWJvdXQ9IiIgeG1sbnM6eG1wTU09Imh0dHA6Ly9ucy5hZG9iZS5jb20veGFwLzEuMC9tbS8iIHhtbG5zOnN0RXZ0PSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvc1R5cGUvUmVzb3VyY2VFdmVudCMiIHhtbG5zOmRjPSJodHRwOi8vcHVybC5vcmcvZGMvZWxlbWVudHMvMS4xLyIgeG1sbnM6cGhvdG9zaG9wPSJodHRwOi8vbnMuYWRvYmUuY29tL3Bob3Rvc2hvcC8xLjAvIiB4bWxuczp4bXA9Imh0dHA6Ly9ucy5hZG9iZS5jb20veGFwLzEuMC8iIHhtcE1NOkRvY3VtZW50SUQ9ImFkb2JlOmRvY2lkOnBob3Rvc2hvcDo5YWI5MzRlYS1jMzdmLWM4NGUtODE2Ny1mZTRjOTdjMmZhNWYiIHhtcE1NOkluc3RhbmNlSUQ9InhtcC5paWQ6NGQ3ZDc2MWMtYmRlZS0zYzQ5LWFlMDYtMjBkMTg1MTdlNmYzIiB4bXBNTTpPcmlnaW5hbERvY3VtZW50SUQ9IkJFRkQ3MDgxNDkxQ0JGOUVENjEwOEI4OTIyNzRGQTUxIiBkYzpmb3JtYXQ9ImltYWdlL2pwZWciIHBob3Rvc2hvcDpDb2xvck1vZGU9IjMiIHBob3Rvc2hvcDpJQ0NQcm9maWxlPSIiIHhtcDpDcmVhdGVEYXRlPSIyMDIwLTA2LTEyVDA5OjEwOjIwLTA3OjAwIiB4bXA6TW9kaWZ5RGF0ZT0iMjAyMC0wNi0xMlQwOToxNTo0OC0wNzowMCIgeG1wOk1ldGFkYXRhRGF0ZT0iMjAyMC0wNi0xMlQwOToxNTo0OC0wNzowMCI+IDx4bXBNTTpIaXN0b3J5PiA8cmRmOlNlcT4gPHJkZjpsaSBzdEV2dDphY3Rpb249InNhdmVkIiBzdEV2dDppbnN0YW5jZUlEPSJ4bXAuaWlkOmZmODY5MjI2LTgxOWYtMzI0MS04M2FhLWM5ZjI2MDAxOGNlMiIgc3RFdnQ6d2hlbj0iMjAyMC0wNi0xMlQwOToxNTo0OC0wNzowMCIgc3RFdnQ6c29mdHdhcmVBZ2VudD0iQWRvYmUgUGhvdG9zaG9wIENDIDIwMTkgKFdpbmRvd3MpIiBzdEV2dDpjaGFuZ2VkPSIvIi8+IDxyZGY6bGkgc3RFdnQ6YWN0aW9uPSJzYXZlZCIgc3RFdnQ6aW5zdGFuY2VJRD0ieG1wLmlpZDo0ZDdkNzYxYy1iZGVlLTNjNDktYWUwNi0yMGQxODUxN2U2ZjMiIHN0RXZ0OndoZW49IjIwMjAtMDYtMTJUMDk6MTU6NDgtMDc6MDAiIHN0RXZ0OnNvZnR3YXJlQWdlbnQ9IkFkb2JlIFBob3Rvc2hvcCBDQyAyMDE5IChXaW5kb3dzKSIgc3RFdnQ6Y2hhbmdlZD0iLyIvPiA8L3JkZjpTZXE+IDwveG1wTU06SGlzdG9yeT4gPC9yZGY6RGVzY3JpcHRpb24+IDwvcmRmOlJERj4gPC94OnhtcG1ldGE+ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPD94cGFja2V0IGVuZD0idyI/Pv/uAA5BZG9iZQBkAAAAAAH/2wCEAAoHBwcIBwoICAoPCggKDxINCgoNEhQQEBIQEBQRDAwMDAwMEQwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwBCwwMFRMVIhgYIhQODg4UFA4ODg4UEQwMDAwMEREMDAwMDAwRDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDP/AABEIAQkB9AMBEQACEQEDEQH/3QAEAD//xAGiAAAABwEBAQEBAAAAAAAAAAAEBQMCBgEABwgJCgsBAAICAwEBAQEBAAAAAAAAAAEAAgMEBQYHCAkKCxAAAgEDAwIEAgYHAwQCBgJzAQIDEQQABSESMUFRBhNhInGBFDKRoQcVsUIjwVLR4TMWYvAkcoLxJUM0U5KismNzwjVEJ5OjszYXVGR0w9LiCCaDCQoYGYSURUaktFbTVSga8uPzxNTk9GV1hZWltcXV5fVmdoaWprbG1ub2N0dXZ3eHl6e3x9fn9zhIWGh4iJiouMjY6PgpOUlZaXmJmam5ydnp+So6SlpqeoqaqrrK2ur6EQACAgECAwUFBAUGBAgDA20BAAIRAwQhEjFBBVETYSIGcYGRMqGx8BTB0eEjQhVSYnLxMyQ0Q4IWklMlomOywgdz0jXiRIMXVJMICQoYGSY2RRonZHRVN/Kjs8MoKdPj84SUpLTE1OT0ZXWFlaW1xdXl9UZWZnaGlqa2xtbm9kdXZ3eHl6e3x9fn9zhIWGh4iJiouMjY6Pg5SVlpeYmZqbnJ2en5KjpKWmp6ipqqusra6vr/2gAMAwEAAhEDEQA/AOzYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq/wD/0OzYq7FXYq7FXYq7FXYq1irsVbxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxVrFW8VdirsVdirsVdir//R7NirsVdirsVdirWKuxVxO3viqUTaxdDUbiwtLBrlrZUaST1EQfvByUAPviq79I6z/wBWdv8AkfHitu/SOs/9Wdv+R8eKu/SOs/8AVnb/AJHx4q79I6z/ANWdv+R8eKu/SOs/9Wdv+R8eKu/SOs/9Wdv+R8eKu/SOs/8AVnb/AJHx4q79I6z/ANWdv+R8eKu/SOs/9Wdv+R8eKu/SOs/9Wdv+R8eKu/SOs/8AVnb/AJHx4q79I6z/ANWdv+R8eKu/SOs/9Wdv+R8eKu/SOs/9Wdv+R8eKu/SOs/8AVnb/AJHx4q79I6z/ANWdv+R8eKu/SOs/9Wdv+R8eKu/SOs/9Wdv+R8eKu/SOs/8AVnb/AJHx4q79I6z/ANWdv+R8eKu/SOs/9Wdv+R8eKu/SOs/9Wdv+R8eKu/SOs/8AVnb/AJHx4q79I6z/ANWdv+R8eKu/SOs/9Wdv+R8eKu/SOs/9Wdv+R8eKu/SOs/8AVnb/AJHx4q79I6z/ANWdv+R8eKu/SOs/9Wdv+R8eKu/SOs/9Wdv+R8eKu/SOs/8AVnb/AJHx4q79I6z/ANWdv+R8eKu/SOs/9Wdv+R8eKu/SOs/9Wdv+R8eKu/SOs/8AVnb/AJHx4q79I6z/ANWdv+R8eKu/SOs/9Wdv+R8eKu/SOs/9Wdv+R8eKu/SOs/8AVnb/AJHx4q79I6z/ANWdv+R8eKu/SOs/9Wdv+R8eKtfpLWf+rQ3/ACPjxVv9I6z/ANWdv+R8eKu/SOs/9Wdv+R8eKu/SOs/9Wdv+R8eKu/SWs/8AVob/AJHx4q79Jaz1/RDf8j48VUzrd7Fc2sN3prQJdSiFJBKjgMVZ91X/AFMVTkDFW8VdirsVdirsVf/S7NirsVdirsVaxVJ/MuqT6bp6S26lp55o7eOi8iDKePIL+1xxVJoNZ1K7vn0u3vWiubWF7i5knhQSfC3BYGjX92qP9vnirING1CS/0e1v5EpJNHzZF8Rt8PzpiqT6dq0Q8waq1zBLa8ltwPVXbZW3+EtlUp0WwQJGzJo5opFDI6sD0INcnGYlyLCQrmvqPHJWh1Rii3VGKW8VdirsVdirsVdirsVdirsVdirsVdirsVdirsVdirsVdirsVdirsVdirsVdirsVdirsVdirsVdirsVUZmIKKDTk1CRscVd6Q/nf/gsUW36I/nf/AII4LS70R/O//BHG1a9Ifzv/AMEcKodLqze4mtlnb17cB5kqagHo3yxV1pc2d7Es1tcGSN/skN1p3AxVUleGIVaVvtBaBqmp8cVXhYyxAlJYdRy3H0YqtkaCON5HnIRAWdufQDqTiqCvNX0qzhhuJ7kiCdgsci8nUlvs1KfZxVHNHxQsHY0BIqa9sVSzWyTPopPe8Sv/ACLkxVOhireKuxV2KuxV2Kv/0+zYq7FXYq7FWsVQ95Z297Abe4TlGSDsaEFfssrD7LYqlzeVdHMYX0nDA1aZZGEr16rJLXnIn+S2KppDBHBEkMShI4wFRR0AHTFUpsUVvMWsBhUFLbYj/IbImAKRIhFS6PErepaO1tJ1PD7JPuuY88H808LdHNX1DjCz61qFrtdQetGOssPX5smPHOPMcSmMJ8jwoq2vrW6H7qQEjqh2Yf7E5OGaM+TCeKUeYROXtTlwA2mqXYVdirsVdirsVdirsVdirsVdirsVdirsVdirsVdirsVdirsVdirsVdirsVdirsVdirsVdirsVdiqhNu0f+t/DCEFLpNB9SRn+vXK8iTxV6AV/lyXEwIWf4eb/q4Xf/IzJeP/AEWvwPN3+Hm/6uF3/wAjMfH/AKIXwPNNLaA28Kxc2l4CnNzVj8zlTcGP+ZdK1Wa5hutHC+vOptL1mNKQsa+qv8zRtiySu58v6zHqdubO3KR2s8Hoyxsqp9WT4ZFkr+9L/wAy4qiU8t3kdjIyQ0v5NSFwz8zyMAlD/aJ+zw/ZxVSTR9fPmKK+MJjCzym4dWURPA60j+H+8Z/5+WKrbXynexadYw+gFuGt5odQYvXkXDemH3+L48VVrjT7+LyrFp1lpJSdXQNCjoB+7YM0gYn9unw4qymN5JLPnJEYXKbxsQSu3QlcVS7Wv77RP+YxP+TcmKp0MVbxV2KuxV2KuxV//9Ts2KuxV2KuxV2KuxV2KtHpiqTaf/yker/6lt/xBsaVOd8VWkN9GRpbQlzptvcNyZOMnaRPhYf7IZVPCJf0Wccso/0lD09VtP7thdQj9l6B/wDgsr/eQ5DibbhPn6FWHVbZm9OatvL/ACSbf8C/2WyY1A5S9EmP5cjePrijgykA12PQ5eDbSdm6jCrqjAJAqXAg4UAurituxS3irsVdirsVdirsVdirsVdirsVdirsVdirsVdirsVdirsVdirsVdirsVdirsVUZvtRf62AlXG6twaGRQfCoyByRHVmIHua+tW3+/U/4IYeMd7HgPc761bf79T/ghjxjvXgPcqq6OvJCCD0I6ZIG0EUlx1mBL66s5lMTW0Qn5tTi8dD8S/6tMKFKx8yabdJb82NvNcbxQyA1PL7Hx09P4/5eWKVzeY9JLMsNwsrRyiCSlQocnhw5kcC3L/KxVELq+nG8NiJwLpa/uiGH2RVuJYBW/wBjiqF/xToTwyyw3ayiKNpSFDVKp9rjUfF/scVU/wDEHrWCahYWrXVsVMkjFhDwAFf93BeWKo+1vIr7T1u4gRHNHyUMKHfxxVA61/faJ/zGJ/ybkxVOhireKuxV2KuxV2Kv/9Xs2KuxV2KuxV2KtVGKuqMVdXFUn0//AJSPV/8AUtv+INiqc4q7FWjgV1MKFGa2hmUpKgdT2YZWYBmJEII6bPB8VjOUp/up/iT8fiXKThI3BbRmEvqDQ1OS3NL6ExD/AH6vxJ+H2ciM8gakEnDGX0lHQzwzLyicOp7g7ZkicZNBhKPNVXp1wiNbBF23XDSLdhS3irsVdirsVdirsVdirsVdirsVdirsVdirsVdirsVdirsVdirsVdirsVdirsVUZvtR1/m/hjS2pNpti5LNAhY7kkdzlZxhmMhW/orTf+WdPuyvwcfcz8efe79Fab/yzp92Pg4+5fHn3omKJIkCIoVR0A6ZdEU1E2k3mLy6+sNbtFcG2eMlJ2Ar6kLfbh/DJFAQN55PuLnUUuBcIII54Z4iwYyoIaH0I2DcFRv5sVRX+GH/AEcbL1wCb366ZAtP2xLw64qhh5RujrEWoSXMbCGaSZXKsZSJF4+kzcuPBcVVYfKZSx0+0NxUWUUkTNx3YSKy+P8AlYqtufL+tzafa2AvYFtoDSaP03pKi/YR2WRW/wCMmKp7HHJFZCOTgGRKERgqmw/YU1xVLda/vtE/5jE/5NyYqnYxVvFXYq7FXYq7FX//1uzYq7FXYq7FWjiqVeYRqx08DSiwuvVjL8Cob0+X73h6nwcuGKpDBql5e3g0yC9ntntYZJrmWUIZTKjcfRbb0mjUfyYqyDQ7+W90W1v51pLLFydVHcV+yv8AlUxVB6JdLda9q8qxyRjjbDjKpRtkbscVZBirsVdirWBXU3w7opqhrkaSt4VqCAQeo7HDueaAKQUukQFvUgJt5f5o9gf9ZemY8tPHnEcEnIjnPI+oKfr6naGk8f1mMdXi2YD3TIXOB39bKoS5ekom21C0uTSN/jHVG+Fh9By+GYSaZ4TFFBgctBa7bqMVt1cUurireKuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KrHRXFGFRiq36uvi3/BHArX1dPFv+COGyjhDvq6eJ/wCCOPEUcIb9BfFv+COLJ3oJ4t/wRxV3oJ4t/wAEcVd6CeLf8EcVd6CeLf8ABHFXegni3/BHFXfV08W/4I4q76unufYk4qlWu/706NTp9dX/AJNyYqnIxVvFXYq7FXYq7FX/1+zYq7FXYq7FWsVQ95ZQXkJhuF5RmhoCV6e6kYqg38taK8EcBtE4RnklKhqnrVx8bcsVTGOJIo1jjAVEHFVGwAG22KpTp3/KR6x/qW3/ABBsVTrFXYq7FXYq7FXYq7FWsVW0+LBfRFdUNc6da3P94nxdmGxB/wBYZVPCC2wzEIb6tqNrvbyi4iH+6pftU/yXXKvDMNx6mzjjM7jgXxarFyEVyhtpT0DjY/7L7OShmJ+r0sZYa+n1I8OpFVNa9MvEgWqQLddsJLEN4Ut4q7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq1iqTa9/vVo//ADGr/wAm5cVTkYq3irsVdirsVdir/9Ds2KuxV2KuxV2KuxV2KtHFUm08/wDOx6v/AKlt/wAQbFU45DFW+QxVwauK26uKuxRbeKXYq7FWqYop1DgpLXHGlWSQRyqUkUMp6ggHIygJc0xkRyQLaXJCeVjO0A/30fijP+xP2f8AY5TLFIci3DID9QW/pC5tdr2A8f8Af0fxL932sjHKYfUE+GJfSUdBdW9wvKFw4707fPL45Iy5epqlAx5jhVq5YwLRYDFSabqMFq3hV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KtYqk2vf71aP/AMxq/wDJuXFU5GKt4q7FXYq7FXYq/wD/0ezYq7FXYq7FXYq1XFXV3Hvirj0xtaYx+i0vPMuqyiaWCZI7YI8TU/Yb7S/ZbJCTAxRnpeY7T+7liv4/CQenJ/wS/BkrBYHiHJtfMSRELqNrNZt05MvOP/kYmJh3JGTvTK1vbS6XlbzJKp7qQfwyBiQyEonkr1FcFM7bBxV2Kt4q7FXYq7FXYq7FWsAkpDXGu2JiCgWEDNpdrK3qx1hm7SRnj96jZsoOGJ+lvGaQ5/7JSL6ra/3gF5COrJ8Mg+a/ZysnJH/bf6jMDHL/AGr+ur2+p2lweAb05B1jk+FvuOWxziXL0tUsJjz9SL8MusNVFdhS3irsVdirsVdirsVdirsVdirsVdirsVdirsVdirsVdirsVdirsVdirsVdirsVdirsVdirWKpNr3+9Wj/8xq/8m5cVTkYq3irsVdirsVdir//S7NirsVdirsVaOKpH5l10aZbiGGWOK9nB9FpSoReI5M7Bj8XGn2cVSO08z6reWE2oQ3ERj0+KF7hERT67SDm/xfai4r8KccVZmk3OBZQp+JA/EddxypiqS6LcNca/rDmGSCi2w4ygAn4G3+FnxVPiK4q00asKHcHsemEFBFpdc+XtMmbmIvRl7SwkxtX/AGFMkJkMPDHRQNhrdpvaXwuEHSK6Wv0esnx4bBY8JDhrd5bD/cjp8kYH2poT6qfcP3mPAD1XxCOiNtNY0682guEZ+8ZPF/8AgGo2RMSGwSBRnL2yG7Kw3XG1dXG1dXFXYVbxVqmNIdilopXAYgo36NFMd+ikA81G4sba5FJowx7N0I+RGVTxiTZCZjyQn1O/td7Sb1EHSGXf6FftlYxyjvE/5jb4kZfUP89cmrRq3C8ja2fp8W6k/wCS4wjUAfUPDQcBP0/vEckqOoZDyU9GHTL4y4uTRIVzXch17YSaUbuDVwg2i9264pdireKuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxVrFUm17/erR/+Y1f+TcuKpyMVbxV2KuxV2KuxV//T7NirsVdirsVaO+KqU9rb3KFJ41lUilHAPX54qlv+FdFHpBYSiRKq8FYhWCGqeqv+7OP+XiqbcaCg6eGKpPp//KR6v/qW3/EGxVOsVdirsVaIxVqmKoO70jTrve4gRm/mA4t/wa8XyQyU1nHaCOi3lvvp1/JHTpFOPVj+W/Fsl4oPNgMVO/SGt2m13Yi4QdZbVqn6Yn4t/wANjwgp4iFa38waXO3Ay+jN/vuYGNq/7OmA4qSMqZBgRUbjxByB2bBRXA4pdXFW8VdirsVdirVMFK4jbrjSqbxRyLRxyU9Qd8jKI6+pRI9PSgX0oRvzs5WtnO/Ebp/wJzHlpzzifDbxnHKQ8Rb9cvrba9g5p3mi3FP8pOuAZZ4/qH/JT/jiThhP6T/yT/4+i7a7tbkc4HDgbddx/scvjOMvU1ShKOyIrlrVZbGCmVt4VdirsVdirsVdirsVdirsVdirsVdirsVdirsVdirsVdirsVdirsVdirWKpNr3+9Wj/wDMav8AyblxVORireKuxV2KuxV2Kv8A/9Ts2KuxV2KuxV2KuxV2KtHFUn0//lI9X/1Lb/iDYqnOKuxV2KtYq7FXUxQ6mKuK4rShc2NpdLxuYUlH+WAcIJCDEFLG8uRxEtp11NZN2RW5x/8AIt65MZO8NZxebhL5js9pIor+MftRn05Kf6h+FsNRP9FFyG1L4/MljyEd2sllKduE6lRX2cVTI8Hcy8TvTOK4imXnE6uvipBH4ZAgsxIKlcUuril2KuxQ7FLqYFdTFXUwqgrnS7Sc8+Jjm/37H8LfTTrlE8EZG/4m2OaURX8P81QH6VszQUvIh2PwyfR+zldyhy9TP0y/oK1vqttK/puTDL3jkHE/j9rLI5e9rljPT1I0NlwNtbuWNobrilwNcKAW8UuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxVrFUm17/AHq0f/mNX/k3LiqcjFW8VdirsVdirsVf/9Xs2KuxV2KuxV2KtVxV1fuxV1cVSfT/APlI9X/1Lb/iDYqnOKuxV2KuxV2KuxV2KuxVrFXUxVqm+NLayWGKVSsiB1PUEAj8ceKmJgClkvluwLepbc7OXrygYoPpT7GTGW2HhLPS8xWn91PHfRD9iVfTf/kYnwf8Jh9JRcg2PMAhNNRtJbM93K+pH/yMj/5pxOIdEDKe5Mba/s7teVtMko/yWBP3dciYtokFeuRpNt4pbxV2KuxVrFWiMaCLKjcWtvOvGaMOO1RuPllUscTzHE2RyGPI8KC+oXlsf9Dn/djcRS/Evy5fayg48gOx9LZxwl9Ud211QxHjewtbt3f7SH/ZD7OTGYDmFOGxYKOimjlUOjBlPQg1y6OQS5NBgY816mtckAQgG12FLeKuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KtYqk2vf71aP8A8xq/8m5cVTkYq3irsVdirsVdir//1uzYq7FXYq7FWsVQt+sz2riCSRJBuDDw5mm/BPW/d/F/lYqw+x1jW725l06W8NpLaRzTOxVGkrG4SOKfYxfZPx+lirKdHv5L3Sba9lQiSWLm4UdxXov0YqgdEu0utf1iREkQBbYUlRozsjfsuAcVZBirsVdirsVdirsVdirsVdirsVawK6mFXUwUruIxpWuIpQ74QggJfc6Bpdwxcw+nKf8AdsR9Nv8AgkpkxMhrOIFDfo7WbXeyv/WQf7qu15f8lE4tkgQeaCCOTjrN/a/8dDT3VB1mg/ep8+I+PAYjoomRzRlprWm3Z4wzqX7ox4sP9i2+RMCyGUFGhq08DkaLY3irq4q7Glt3XFDqYpWtGrCjAEeB3GRMQVBpASaTEGMlq7W8n+R9k/NMolg6hvjn6H1LBd6jaf71Q+tEOs0XX6Y8gMk4808EJcvSi7bULa5H7pxy7odmHzXL45oyapY5R5q4Y1ocsotXEG6muNsmwdsO627FXYq3irsVdirsVdirsVdirsVdirsVdirsVaxVJte/3q0f/mNX/k3LiqcjFW8VdirsVdirsVf/1+zYq7FXYq7FWsVULyxt72L0bhSyVrsxU/8ABIVbFUE/lnQ3iSI2q8UJYEFgxJ+1zcHnJy/y2xVMkjSNAiDiigBVHQAbYqlOn/8AKR6v/qW3/EGxVOcVdirsVdirsVdirsVdirsVdirsVdirsVdirsVaxVqgwK6gAOG0IO60rTrwf6Rbo7dmpRv+DWjYRMseEFAnRLm2306/lhA6Qy/vY/l8XxDJ8V82BxEb24X+u2u13ZC5Qf7ttmqaeJjfGgV4iFe38w6ZM/ptL6Ev++5wY2/4fAYFkJ96ZK4YVUgjxG4yuizsFsYU03irjgKtUxtaaNARgJCkWhbnTbS4PJ04ydpF+FvvGVywA8mcc0ghvS1S03icXcY/YfZ6ezd8pEckf6TdxY59OCSrDq1u7+nKDBL3SUU/4Fvs5bHMDs1ywkbj1BHKwI2NfA5aJNVLskrsQreKuxV2KuxV2KuxV2KuxV2KuxV2KuxVrFUm17/erR/+Y1f+TcuKpyMVbxV2KuxV2KuxV//Q7NirsVdirsVdirsVdirRxVJ9P/5SPV/9S2/4g2KpzirsVdirsVdirsVdirsVdirsVdirsVdirsVdirsVaxV1MVdQYq7iMUU6gxWlC5s7S5QpcQpKp7MAcIkQxMAUtPl2GJi+nzzWbdeKNySv/GN6rlniXzYeFXJr1PMlp9tIr+P+Zf3Un/AmqYfSUXML4vMlmGCXiSWUh2pMpC19pB8ORMO5kJplDcQzrzhkWRP5lII/DIUQzEgVXAyaoMUU7iK1wUGTdAcKFKe2gnThMgdPA5CUBLmyjIx5IE6dPbktYzlB2if4k/H7OUeAY/QW7xhL6g2upzQELfQmLt6qfEn4ZEZpR+sKcQP0niRsNxHMvKNw69iu+ZMZgiw0EEGqVAT3wgobySurgtW8KuxV2KuxV2KuxV2KuxV2KtYqk2vf71aP/wAxq/8AJuXFU5GKt4q7FXYq7FXYq//R7NirsVdirsVdirsVaxV2KpPp/wDyker/AOpbf8QbFU5xV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxVo4q1ii3UGBK2SKOVSsihlPVWFR+OESLExBSyby7pzNzgDWsv80DFP+E/u/wDhcsGRrOOuSn6HmG0/uLiO9iHSOccJP+RifB/wuGwV3DY8wNAeOo2c1r4ycfUjr/xkj/5pwcF8l465o+11Gyu1DW86SV8CK/8AA9ciYsxNFA5FkC7FK09cHD1WmiAwIO48MSNt1Eu5BS6Vbs3qQFreXryjNAT/AJS/ZOYx04J4o+luGeVVL1RUzNqdp/fRi5iH7cfwuB4lMBnOHP1shCE+XoRNvqdrcfCj0k7xt8LD/YtlsMwIa5YiEVXLhRa928VbxV2KuxV2KuxV2KuxV2KtYqk2vf71aP8A8xq/8m5cVTkYq3irsVdirsVdir//0uzYq7FXYq7FWj0xVC6h6/1ZzDci1K/E07KHAUbt8LUxVjFrrWv30v1KOVIHWKa5ju2j/vYo3EUf7tv7vnyxVkGj6k2o6RbagUKtNHzMY3NRUfD/AMDiqB0S5Nzr+sOYZYCFthxmAU7I2+xbFWQYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXY0rWKuxV1BgpXUwq0yqeoqMbKOEFL7vQtLuSXaAJIf92xfu3/4JKZMSLAwCF/RmsWv+8N+ZEHSG6HMf8jF+PJcUTzDAxl0Lv0xqNrUahYPwH+7rc+ov/AfbwcAPJPGRzRVprWmXbcYbhTJ3jb4WH+xahyJxkJGQFMBQ5CqbAbdt0wWmnUGEBBKGuLC1uB+8jHLs42Yf7IZXLED/WbIZSPchTbala/7zTCeMf7ql+1/sZMo8OY5Ft44y5hfHq8XIR3KtbSntIPhP+rJ9nDj1FmiOFicR6I5XDCoNR7ZlAjvajErt8KOTqnAi12FLsVdirsVdirWKpNr3+9Wj/8AMav/ACblxVORireKuxV2KuxV2Kv/0+zYq7FXYq7FWsVQOq6Taatam0u+RgYhiEcoSQa0JXFULJ5a02WKNHaYtFVVk9V+fE9YjJXkYv8AIxVNILeK3gSCFQkMa8UQdABiqV6f/wApHq/+pbf8QbFU5xV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KtYq6mK20Rg3RQQl1pWn3gpcW6P4GlGH+yGSE5BBhFAnQ7i2302+lgA6RSH1Y/ufdcmJjq1mB6O+u69a/702iXUY6yWzEN8/Sk/41w1E8kXIc1WHzDpsjenJI1tMdhHOpib/h/hbInGejMZB1TJHRxVSGHiCDlfCQzsFdTbE7pApTeGKVCkiB1PUEVGRIEhukSKAbS2iPKxmMB/32fiT/gD9nKDhr6S3DIOrv0jc2+19AQn+/o/iT/ZftLgGaQ+oL4UZfSUZDdW9woaGRXHsd/uy6M4y6tUoSj0RGWsXYq3irsVdirWKpNr3+9Wj/8AMav/ACblxVORireKuxV2KuxV2Kv/1OzYq7FXYq7FXYq1irsVccVSfT/+Uj1f/Utv+INiqc4q7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FWsVdQYq7FXYqoz2tvcKUmiSRTsVcA4RIhjwjuSxvLtvG3Kwnlsn6gRNWP6Yn+HJjN0LX4XUFqvmO06iK/jHcfupPuP7vCBE/0UEkNp5jtUbhexS2UncTLRf+Ri1TAcaRm8kzhubedecMiyL4qQR+GVmNMxMFUIBG4yNXzDPkgptLtpW5x/uZu0kZ4n6Qu2VSwDp6W6Oc9fUpF9VtNiPrcI/aXaQfR+1lRlPH/tjMCGT/AGsq0GpW0/wBzHKOsb/C1fpy2GaMuZ4ZNU8Uo8vXFGVNBvl+7X73VNQMjxC6Vdklt2KpNr3+9Wj/APMav/JuXFU5GKt4q7FXYq7FXYq//9Xs2KuxV2KuxV2KuxV2KtHFUn0//lI9X/1Lb/iDYqnOKuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxVrFXYNkbtHHdOy1kSRSrhWU9VIqPxwglGyWT+XdNd/UgVrSbs8DFP+FX4MmMtMDjtS+q+YbUVt7pLyMdI7gcX/wCRqf8ANOGwWNFw197f4dSsprbxlUepH/wSY8BPJfEA5pha6hY3qg21wkh8FO/0r9rIGBjzDISjLkXXNnbXG00YY9j0I/2QzHnihLm3Qyzjy+lC/VL613tJ/Uj/AN8y7/c+UnFkj9J4g3eJCXNcmrIjBLyNraQbVO6H5PkhnF0R60HEasfSmEbq6hlIZT0YGtcyYytoIpfklSbXv96tH/5jV/4hJiqcjFW8VdirsVdirsVf/9bs2KuxV2KuxVo9MVSnzJe6hYaXLe2Jj5W45yLKCwZR+yvEr8WKpff6/qMcl89ssQt9LjjkulcMWk9Qep+6ao48UxVkMcyvbrOAeLIHA6mhHKmKpLol1Hda/rEiLIgC2wpIjRn7DdnxVkGKuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxVrFW8VWkAihG2KKS660LTLpi0kCrJ2kjqjD35LkxkLA4wh/0TqtrvYagzqOkNyPUWnhz+2uS4weYYmBHJ36V1O22v7Bin+/rU+qPmU/vFx4I9ColIc1eHV9Jv19NJkL9DFJ8Lf8AAvlc8JrcbM45R0PqbbSVRvUspWtm6kDdD80O2YksFbxPA5cc97SHG0L2+tv97IOa9PWh3H+yQ/EuHxJx5oOOMtwfV/NQWq3cFzcaO0LhgL1agdR8EuWxyW1nHTIBkwwbwq7FXYq7FXYq/wD/1+zYq7FXYq7FWsVS/WNKj1WzeylmlhhkBEnosFZh4FmDYqg38r2TsDJPcEMqpcAOAJljoE9fivx8RiqdIqqoVRRVAAA7AdMVSnT/APlI9X/1Lb/iD4qnOKuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KuxV2KtYocemNrTXbAPJfehLrTbG72ubdJPdgK/8FkozkOTAwgUD+gpoDXTr6a3HURuRLH/wMnxL/scs8X+cOJj4XceFzXOv2o/0q0jvEH+7LduLf8i5MBxwkOfqXxJx5j0pXe3+m3ep6T6cLQ3q3g5pIhR6cJK8v2TlZw8LYM3Ey4Ysm8VdirsVdirsVf/Q7NirsVdirsVdirsVdirsVSSykjTzHq/N1X4LbqQP2GxVN/rEH+/U/wCCGKu+sQf79T/ghirvrEH+/U/4IYq76xB/v1P+CGKu+sQf79T/AIIYq76xB/v1P+CGKu+sQf79T/ghirvXg/34n/BDFXfWIP8Afqf8EMVd9Yg/36n/AAQxV31iD/fqf8EMVd9Yg/36n/BDFXfWIP8Afqf8EMVd9Yg/36n/AAQxV31iD/fqf8EMVd68HX1Ep/rDFXfWIP8Afqf8EMVd9Yg/36n/AAQxV31iD/fqf8EMVd9Yg/36n/BDFXfWIP8Afqf8EMVd9Yg/36n/AAQxV31iD/fqf8EMVd68HX1Ep/rDFXfWLf8A36n/AAQxV31iD/fqf8EMVd9Yg/36n/BDFXfWIP8Afqf8EMVd9Yg/36n/AAQxV31iD/fqf8EMVd9Yg/36n/BDFXfWIP8Afif8EMVd9Yt/9+p/wQxV31iD/fqf8EMVd9Yg/wB+p/wQxV31iD/fqf8ABDFXfWIP9+p/wQxV31iD/fqf8EMVd9Yg/wB+p/wQxV31iD/fif8ABDFXfWIP9+p/wQxV31iD/fqf8EMVd9Yg/wB+p/wQxV31iD/fqf8ABDFXfWIP9+p/wQxV31iD/fqf8EMVaNxb/wC/U/4IYqk+uPE95o5V1Zher0IJ+xLiqeDFW8VdirsVdirsVf/R7NirsVdirsVdirsVdirR3GKpO3l2yn1W7v7uGO4FwsSoHWpX0wVP/BYqrf4d0P8A5YIf+BxV3+HdD/5YIf8AgcVd/h3Q/wDlgh/4HFXf4d0P/lgh/wCBxV3+HdD/AOWGH/gcVd/h3Q/+WGH/AIHFXf4e0PeljD7/AA4qhLXyppUV3eTSW0UkVwytDGV2QKvFh/smxVF/4d0P/lgh/wCBxV3+HdD/AOWCH/gcVd/h3Q/+WCH/AIHFXf4d0P8A5YIf+BxV3+HdD/5YIf8AgcVd/h3Q/wDlhh/4HFXf4d0L/lhh/wCBxVCJ5U0salNdNbRGCSJI1h47BlZi7/7IFcVRf+HdD/5YIf8AgcVd/h3Q/wDlgh/4HFXf4d0P/lgh/wCBxV3+HdD/AOWCH/gcVd/h3Q/+WGH/AIHFXf4d0P8A5YYf+BxV3+HtC/5YYf8AgcVQj+U9KbUoroW0QgSJo2h47FmZWV/9iFxVF/4d0P8A5YIf+BxV3+HdD/5YIf8AgcVd/h3Q/wDlgh/4HFXf4d0P/lgh/wCBxV3+HdD/AOWCH/gcVd/h3Qv+WGH/AIHFXf4d0L/lhh/4HFUJdeVNKlu7OaK2iSK3Z2mjC/b5LxUH/VbFUX/h3Q/+WCH/AIHFXf4d0P8A5YIf+BxV3+HdD/5YIf8AgcVd/h3Q/wDlgh/4HFXf4d0P/lgh/wCBxV3+HdD/AOWCH/gcVd/h3Qv+WGH/AIHFUJf+VNKufq3o20UXpTJI9F+0i/aj/wBliqL/AMO6H/ywQ/8AA4q7/Duh/wDLBD/wOKu/w7of/LBD/wADirv8O6H/AMsEP/A4q7/Duh/8sEP/AAOKu/w7oX/LDD/wOKu/w7oX/LDD/wADiqHn8tad9Zsri0gjgktZhKWUUJUK68P+GxVOhireKuxV2KuxV2Kv/9Ls2KuxV2KtYq6oxVbI6ojOxoFBY/Ib4qxa/wDNMkunubKGW2llUtayyKKSIrUkaL/i3j8So2KonR9UmW+is5boXsFzEZra4oFkHE8ZElC+DYqyLFW8VdirWKuqMVaJAFfDfFUk/T31xL23sopItQgRmgS4TgXp+2nL7S4ql+kazcxy2IluvrcF6TG6SACaGZR8aUX9iuKssHTfFW8VdirWKuqOmKuJxVJJ/MSRziL6rMqSOYILqRaRGYg8V/n9PkPt4qlNhrOo24iuLm69YtP9XvrOQKrxSE/D6XH7SYqzLFXYq7FWsVdXFXV2xVJNR8xwWE4jnt5hAXWJrkgLHzc0Xjy+J/8AK44qlEWtapame7uboSPBNwu7FgABE7Uhltz9orxxVmCMGUMOhFcVX4q7FXYq1irq4qles65BpLWvqxPIty/Dkm4RR9p3/wAnFUjl1bVI7q+n+tr/AKKRKlkwX05Lan243+3zxVldrOlxbRTp9mVQ4+kVxVWxV2KuxVqoxV1RiqUT+YbWHVTppUh41DzyyERxqh6MrP8A3n+xxVJJNW1OCe+na7HO1cOLFgvpPbN9iSF+uKsut5lnhjmUELIoYA9d8VVcVdirWKuxV1RiqV6nrS2M8dqlvJc3EitJ6cQ3CL9tsVSD9M3t1e3Mtte+iqRiazt3UCOSNR+9Rq7+srYqynTLwX1hBdgcfWQPx8CeuKovFXYq7FXYq7FXYq//0+y4q7FWqjFWpG4IzULEAniOpp2GKsfj8zrJdTafcW7Wd16DTQq7DkVp8PKn2JG/ZxVKPK2u3Usq6RePcXCSgy22oOhUMAOUtrIWHxPH8Sq/8mKpoPLlzJ6Mcdx6enpMLhIJErNGwNTGknT03xVObbS9PtbiS5t7ZI55PtuooTU1xVF4q3irsVaJoD39sVSJfM6rqcdhdWzWpuA31d5WFW4fzIPsc/2MVSDRNfvI9SfT55Lm5tb+ZxDdlCPq8vI/6Ozn7SFR8OKsg1bS77UpLWeyl+o3Fu55XLrykKd1RPs/HX9vFUfDpNjFOt16CG7AobjiAxPd9vh+LFUdirsVarirq9sVSO78zLZahFbXVq8FvNIYo7hyByb+cJ/vv/KxVIIvMV3Za9dLI9xd2N1P6TBUJW1b4UikSWnH0pMVT3V9O1S/Nmlm6wizm5tNOOXPitFdET7X2v2sVR8Wj2fOK4uY0uL6ICt0VAYsP26DbFUwxV2KuxVqu1R9+KpHq3mX9FzL9YtHFp6iRtcswALP/vtPtOq/tYqkWoeY7rTvMc1zE893YnhHLZxqXREZdruOT7P2/t4qyK9tpL21WGKGOaQANHPdqGQBxuVA+0yrirdn5c0+G1tYrmNbua0Xik8gq3Xl/wACP2cVTYCmKt4q7FXV2riqS65r7aQDK9o8ltHQzXBIVQCacUr9uT/JxVj+u+YLm11iG/sZJrm3SBHbT4lLrLEx/eybfYlhXFU5uNOi16C0l9a4ihdWmDjiCUlG8DV3XFVay8q6TbWUVlLF9ZhgYmEzfE6g78ef7WKp0qhVCgAAbADpTFW8VdirsVSPWvMJ0li8to72qcedwSFWrHiFiB+J2/mxVIda8xXNhr/1y1kmu7NYkElhEpdGib4nuEYdJIv2sVT64sodQhjuI7VLm79PlbXN0oKhZftI1P5F/YxVdY+W7KGxgtruNbuSAkrIw6cjyZEX9mMfspiqcgACnbsMVbxVqoxV21MVSjWtafS0MgtHmgRfUlm5KiKK9OTfakxVKUv9WN1Lq0R+taa8Ykty0qwxRqR8cTx/akuBiqN9BtYgtNQtzLb3LVlglYArHX7UUg/aR8VVrLy5AtvLHqSx3ck0rTt8PFFZuqxD7SriqcRxpGipGoVFACqNgAOgxVfirsVdirsVdirsVf/U7LirsVYvqGpavcSXGktH+j7qcFtOnVw3qhftxmv927LirrC7vILi20a6iRWkVn/cTNLKhTflcFx/u04qjb/RptUkiaTjbQKweZQqmZ2Q/B++FeKYqnAQCgAHw/Z26YqvxV2KuxVo4qx7VNV1GO7ewli+pW9yDFZairBv31KqrKdo+f7LYqhLG51PThbadeRoXu5PTdkmaW4DHczvyHFV/wCI4qmep6VcapEIarbx14TSMqvMyKQVMcn7HL4sVTWOFI40jVfhjAC9+mw6/tYqqDwxVvFXYq7FUj1bWL2wu1SS39PTZaR/pDlX05XFFZ4v998v2sVSm3utW0SELfCOdriUIzmcvcS8zx9WKOnBFVf5cVTu/wBOuby3+rRhY6fuzcTKJHMbD4/T/lfFUdZ2cdpaRWq/GsahatSpptVsVRHyxVvFXYq0emKpRrOr3mmvHILT1NPFPrVyGo0YP7Sx78+P7WKpKtxq2jrc39w0V3HK1YppZj6kqH7CW8KDgrLiqey21zcWohhQK5UNHPcAS05/bUKd/gxVX0vTI9OsY7QMZeAPN2A+Inc7fy/5OKowClBTYdPDFV2KuxV2KpVreo6hp8Kz2ln9aiT4rg8+JVB9plX9vFUj+s6lbTz60ssV1YSr6lrLPMUAQivoRQIvH1K4qnUJnvbFJI4aSXCesrXFJFjc/sen8LYqq6NpI0y2aIyetLI7SSSEBQS25VR2T/JxVMAKCgAA9sVXYq7FXYqlOt6pfaaqTw2f1m0T4ruQNxZEH7SJvzbFUkFxqunPdau0kV5aSD1YJ5ZSvKMj4IIIVqiuMVTvhcXdiPSipJKgmjkuKSKjsPsemfi+AHFVbSNKTTLMW4YysSzPIQBUseTAD9lf8nFUeAAKAUHQAYq3irsVdiqW6ze6jZW4lsbP64wJaVOfCiDrw/nf+XFUhW41E3T63BNFc6e8YaB7iYxpFt8cPpIP72v8+KpvZTzajYJcLAQ12vqUmo8cbjYfAaEpirVh5dgigkS943Tzy+tIOPGMN/xXH+yuKpwiqgCqOKgUAGwAxVdirsVdirsVdirsVdirsVf/1ex1xV3LFUt1nRotXhihmlkiSKQSViorkr9njJ1j/wBjiqJtLG0swfq8Sqx+09Ks3u7H4uWKomuKuqMVdXFXVxV1cVdXFUBrOlx6tYtZSStFG5BZ0A5ihr8DH7Df5WKrrLS7KxVfRTlIq8fXk+OUgfzSt8WKowGmKt1xV1cVdXFXVxV1cVQupWS39lNZs3BJl4M4AJAPhXFVDTdEsNPjjWNWmkiHFbic+pLT/XbFUxrirqjFXVxV1cVdXFXVxVTuI/WgkhrT1FKEkBhvt9k/axVLdK8uabpkKRqGuXj+zLcH1GH+ry+wv+riqal1XqQK9CdsVcJFNQrAkdQDU4q3y74q7liruWKuriq16FaVHQ+/bFUn0zyxp2nq5YvdMzmSs55IrE8v3UP2I8VToHbbtt92Kurirq4q6uKurirq4qskXmjpWnIEVpXr7YqlOl+WdN06IJ8V0wYuGn+MKSa/uk+zHiqcgimKurirq4q6uKurirq4q6uxGKpLYeWNPtJJpJC1008hlZZd4lY7/uYPsJiqcigAA2A6AbYq3UYq6uKurirq4q6uKurirq4q6uKurirq4q//1uwVxV1cVdXFXVxV1cVdXFXVxV1cVdXFXVxV1cVdXFXVxV1cVdXFXVxV1cVdXFXVxV1cVdXFXVxV1cVdXFXVxV1cVdXFW64qlfmKytbvSbkzxCVoo2aImtQ1P2eJxViyzT6VEn6MQQu2lCflxZyZfVCVbmT2/ZxVFS3/AJgga+Bv2dbGKCdSYlBdnAaSNqD+7xVaNd8wzamU9WCzVZUCWsxK+pEwDF1FOXI8v5sVU7zVdTlXU7Nr13n9GV4WtlVkjCGo5GnNJaYqt1DzBf2mnWwsb5ppY7X12mkACyMqg+iPh+OTFVW51nXporq8gufQSC2tJo7cRAgvPT1Vao5d8VdqWt6xYCe3a86SW7C6aIEqk3L1EC/Z+DjiqeeXNQursXiSzfWreGUJbXZXgZFK1bYUX4HxVOa4q6uKurirq4q6uKurirq4q6uKurirq4q6uKurirq4q6uKurirq4q6uKurirq4q6uKurirq4q6uKurirq4q6uKv//X67XFXVxV1cVdXFXVxV1cVdXFXVxV1cVdXFXVxV1cVdXFXVxV1cVdXFXVxV1cVdXFXVxV1cVdXFXVxV1cVdXFXVxV1cVdXFWq167jwxV1AeoG/XYdD2xV3w13FR9G/wA8VaIUsGKqWHRiATirgsYrxVRXrQDevXFXcY9vgU06bDb5bYq3t2A+4dug6Yqo3Vna3aqlxGHVHEig9mHRsVVhQCigKOwAoMVbrirq4q6uKurirq4q6uKurirq4q6uKurirq4q6uKurirq4q6uKurirq4q6uKurirq4q6uKurirq4q6uKurirq4q//0Ot4q7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq7FXYq/wD/2Q==\n",
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# It should look like this\n",
    "file_name2 = './img/targeted-routing-005.jpg'\n",
    "Image(file_name2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Evaluate variant performance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Invoke a specific variant\n",
    "Now, let’s use the new feature that was released today to invoke a specific variant. For this, we simply use the new parameter to define which specific ProductionVariant we want to invoke. Let us use this to invoke Variant1 for all requests.\n",
    "\n",
    "#### Evaluating Variant 1\n",
    "Using the new targeting feature, let us evaluate the accuracy, precision, recall, F1 score, and ROC/AUC for Variant1:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sending test traffic to the endpoint DEMO-xgb-churn-pred-2020-07-29-15-19-23. \n",
      "Please wait...\n",
      "........................................................................................................................Done!\n"
     ]
    }
   ],
   "source": [
    "predictions = ''\n",
    "\n",
    "print(f\"Sending test traffic to the endpoint {endpoint_name}. \\nPlease wait...\")\n",
    "with open('./data/abtest/test_sample_tail_input_cols.csv', 'r') as f:\n",
    "    for row in f:\n",
    "        print(\".\", end=\"\", flush=True)\n",
    "        payload = row.rstrip('\\n')\n",
    "        response = sm_runtime.invoke_endpoint(EndpointName=endpoint_name,\n",
    "                                   ContentType=\"text/csv\",\n",
    "                                   Body=payload,\n",
    "                                   TargetVariant=variant1[\"VariantName\"]) # <- Note this new parameter\n",
    "        \n",
    "        predictions = ','.join([predictions, response['Body'].read().decode('utf-8')])\n",
    "        time.sleep(0.5)\n",
    "\n",
    "# Convert our predictions to a numpy array\n",
    "pred_np = np.fromstring(predictions[1:], sep=',')\n",
    "        \n",
    "# Convert the prediction probabilities to binary predictions of either 1 or 0\n",
    "threshold = 0.5\n",
    "preds = np.where(pred_np > threshold, 1, 0)    \n",
    "print(\"Done!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### When we again check the traffic per variant, this time we see that the number of invocations only incremented for Variant1, because all invocations were targeted at that variant:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "time.sleep(20) #let metrics catch up\n",
    "plot_endpoint_metrics()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.9583333333333334\n",
      "Precision: 0.9411764705882353\n",
      "Recall: 0.8\n",
      "F1 Score: 0.8648648648648648\n",
      "AUC is 0.895\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd5xU1f3/8ddHugqogCggAgoo0kREMfYGViAqYo1Gg91ofibytfeeaExQgmiMDRJ7AzEWxNhBERHUEEBZCyCgNIEtn98f565snZ1d9k59Px+PeezcmTMzn7ss9zPn3HM/x9wdERHJX5ukOwAREUkvJQIRkTynRCAikueUCERE8pwSgYhInlMiEBHJc0oEIiJ5TolAcoqZLTCzn8xslZl9Z2YPmtnmFdrsZWavmdlKM/vRzJ43sx4V2rQws7vM7KvoveZG262r+VwzswvNbJaZrTazAjN73Mx6xbm/IvVBiUBy0VHuvjnQF9gV+L/SJ8xsIPAy8CzQDugMfAy8ZWZdojaNgVeBXYDBQAtgL2ApMKCaz/wz8FvgQmAroBvwDHBEbYM3s4a1fY3IxjBdWSy5xMwWAGe6+yvR9m3ALu5+RLT9JvCJu59b4XWTgCXufqqZnQncCOzg7quS+MyuwGfAQHd/v5o2U4BH3H1ctH1aFOfe0bYD5wMXAQ2BycAqd7+kzHs8C7zh7n8ys3bAX4B9gVXAne5+dxK/IpFK1COQnGVmHYDDgLnR9qaEb/aPV9H8X8Ah0f2DgZeSSQKRg4CC6pJALQwF9gB6AI8Bx5uZAZjZlsChwAQz2wR4ntCTaR99/kVmNmgjP1/ylBKB5KJnzGwlsBBYDFwdPb4V4W/+2ype8y1QOv7fqpo21alt++rc7O7L3P0n4E3AgX2i544F3nH3b4DdgTbufp27r3f3ecB9wIh6iEHykBKB5KKh7t4c2B/YiQ0H+OVACbBtFa/ZFvg+ur+0mjbVqW376iwsveNhzHYCcEL00InAo9H97YF2ZvZD6Q24DGhbDzFIHlIikJzl7m8ADwJ3RNurgXeA46poPpxwghjgFWCQmW2W5Ee9CnQws/4J2qwGNi2zvU1VIVfYHg8ca2bbE4aMnoweXwjMd/ctytyau/vhScYrUo4SgeS6u4BDzKxvtD0K+FU01bO5mW1pZjcAA4FrozYPEw62T5rZTma2iZm1MrPLzKzSwdbd/wvcA4w3s/3NrLGZNTWzEWY2Kmo2A/ilmW1qZjsCZ9QUuLt/BCwBxgGT3f2H6Kn3gRVmdqmZNTOzBmbW08x2r8svSESJQHKauy8BHgKujLb/AwwCfkkY1/+SMMV07+iAjruvI5ww/gz4N7CCcPBtDbxXzUddCPwVGA38APwPGEY4qQtwJ7AeWAT8gw3DPDUZH8XyWJl9KgaOIkyPnU8Y0hoHtEzyPUXK0fRREZE8px6BiEieUyIQEclzSgQiInlOiUBEJM9lXXGr1q1be6dOndIdhohIVpk+ffr37t6mqueyLhF06tSJadOmpTsMEZGsYmZfVvechoZERPKcEoGISJ5TIhARyXNKBCIieU6JQEQkzykRiIjkOSUCEZE8p0QgIpLnlAhERPKcEoGISJ5TIhARyXNKBCIieU6JQEQkz8WWCMzsATNbbGazqnnezOxuM5trZjPNrF9csYiISPXi7BE8CAxO8PxhQNfoNhK4N8ZYRESkGrElAnefCixL0GQI8JAH7wJbmNm2ccUjIpIJli+H6dNh6tR0R7JBOhemaQ8sLLNdED32bcWGZjaS0GugY8eOKQlORKQu1q2DL7+EefNg/vzw8+f7/3P2XDGZyQxihx2MuXPTHW2QzkRgVTzmVTV097HAWID+/ftX2UZEJBVKSuC77zYc5Cv+/Ppr8DJHqaZNoXNn2KPtAh5tOZKdV/ybd654kc2HH56+naggnYmgANiuzHYH4Js0xSIi8rOVK6v4Nh/9nD8f1q7d0NYM2rcPB/uDDoIuXcL90p/btClmk3tHw2WXhcb33MPAswZn1JzNdCaC54DzzWwCsAfwo7tXGhYSEalvhYWwcGHlb/Ol97//vnz7li3DgX3nneGII8of7LffHpo0SfBhxx4PTz4Jhx0GY8ZABg5vx5YIzGw8sD/Q2swKgKuBRgDuPgaYCBwOzAXWAKfHFYuI5Bf3cDCvauhm3ryQBIqLN7Rv2BA6dQoH9n79yh/ou3SBLbesZQCFheFno0Zw+ukwdCicdFLoEWSg2BKBu59Qw/MOnBfX54tIbluzBhYsqH4IZ9Wq8u3btg0H9b322nCALz3Yt28PDRrUU2DTp8Ovfw3HHx+Gg444op7eOD7pHBoSEalWcTF88001s2/mhRO2ZW222YYD+4EHlj/Yd+oUno/VTz/BNdfAH/8IW28NPXvG/IH1R4lARNJm+fLqZ98sWLBhhAVgk03C8HrnzuFLdtkTsl26QJs2aRx5ee89OOUU+O9/4cwz4fbbYYst0hRM7SkRiEhs1q/fMKe+qoP9Dz+Ub9+qVTiw77orHHNM+YN9x45hyD0jbRJNAXrllTB1KMsoEYhInblXPae+9H5BQfk59U2ahIN6584wcGD5E7KdO0OLFunbl1qbODH0BK69FnbfHebMqccTDamlRCAiCZXOqa/qYL9gQRgaL6t9+3BgP+CAyrNvttlmw5fnrPX993DxxfDII7DLLnDppbDpplmbBECJQCTvFRWF6ZRVDd3Mnw9LlpRv36JFOKjvtBMcfnjlOfVNm6ZnP2LnDo8/DuefH05uXHVVmBWU8CKC7KBEIJLj3GHp0upn33z1VeU59dtvHw7sw4ZVPac+Q6fDx2vx4jAtdOedw7mA3r3THVG9USIQyQE//RSGaaq7gKrinPqttw4H9YED4cQTK8+pb6gjQ+AOkyaFq4Lbtg0lQ3v3zrlfUG7tjUiOKinZMKe+qoP9txWKs2y66YYD+/77lz8h27lzCubU54J58+A3v4HXXoMXXwzjYP1yc/0sJQKRDPHDD9VfJbtgQZiKWWqTTWC77cJB/bDDKs+p33rrPB2+qQ/FxXD33XD55eGb/5gxMDjRGlvZT4lAJEVK59RXN9Vy+fLy7bfaKhzU+/QJY/UV59Q3bpye/ch5xx0HTz8drlobMwY6dEh3RLFTIhCpJ+6waFH1s28KCsIQT6nGjTcM1ey5Z+XyxS1bpm9f8s769aEL1ahRuDL42GPhhBPyplulRCBSC6tWlZ9TX/FgX3FOfbt24cC+336VZ99su20OzKnPBR98EGYDjRgRhoMOz5wFY1JFiUCkjKKi8M29utk3FefUN28eDurdu4dh5Ipz6ps1S89+SBLWrAnXAtx5Z8jKffumO6K0USKQvOIOy5ZVP/vmq69CMijVsGEYj+/SJZSUr1i+eKut8mb0ILe8804oEve//8FZZ8Gtt+b1WJwSgeSctWs3zKmv6mC/cmX59m3ahIP6HnuE0YGyB/oOHXJuyrhA+Edt2DBMDT3ggHRHk3b6E5esUzqnvrrZN99UWPm6WbMNB/b99it/QrZzZ9h88/Tsh6TYiy+GInHXXReKxH36aVbXB6pPSgSSkX78MfGc+nXrNrQ12zCnftCgyhUt27bV8E1eW7IELroIHnsMevWCUaOyvkhcfVMikLRYvz6Mx1d3sF+2rHz7LbcMB/ZevWDIkPIHe82plyq5wz//CRdcEL5ZXHttSAL6Y6lEiUBi4R5qdFVX6KyqOfWdOoUD/IABlefUZ9FiT5IpFi8O1wTssgvcf39WLR2ZakoEUmerV1e/zOD8+WF2XlnbbhsO7PvuW3n2Tbt2mlMv9aCkJJwLOPLIMCb4n/+EbqSGgRJSIpBqFRdXnlNf9v7ixeXbb755OKh37QqHHlr+QN+pk+bUS8zmzg1F4qZM2VAkLo+vDagNJYI8VjqnvrrZN19+WX5OfYMGG+bUH3105W/1rVrppKykQVER3HUXXHllWCRm3LhQiU+SpkSQ40rn1Fc3hLNiRfn2bdqEA/vuu8Pw4eUP9tttpzn1koGOOw6eeSbMIrjnnjDOKLWi/9ZZrqQk1KKvbvbN11+Xb9+06YZv8PvsU3lOffPm6dkPkVpZty6cVGrUCEaODFcCDh+uLmkdKRFkgRUrqq99U9Wc+g4dwkH9kEMqz77ZZhv9X5Es9957cMYZ4eB/xRUaBqoHSgQZoLAwzKmv7mBfcU79FluEA3vPnmGsvuKc+hxYS1ukstWrw3mAu+4K62nm6Gph6aBEkALu4eLG6mrfLFxYfk59o0Zhlk2XLtC/f+Vv9VtumbZdEUmPt98OReLmzYNzzoFbboEWLdIdVc6INRGY2WDgz0ADYJy731Lh+ZbAI0DHKJY73P3vccYUlzVrqp99M29e5Tn122wTDux77131nHpNexYpo3HjcJsyJRSMknoVWyIwswbAaOAQoAD4wMyec/fZZZqdB8x296PMrA3wuZk96u7rq3jLtCouDideq/tWv2hR+fabbRYO6jvsAAcfXHlO/aabpmU3RLLHc8+F8wE33hi6xrNm6RtSTOLsEQwA5rr7PAAzmwAMAcomAgeam5kBmwPLgKKKb5QJBg+GV17ZsN2gQZhO2aULHHVU5cXDW7fWSVmROlm8GC68MNQJ6tMnrBqmInGxijMRtAcWltkuAPao0OavwHPAN0Bz4Hh3L6nQBjMbCYwE6NixYyzB1uSLL8J0y6uv3jCnvlGjtIQikpvc4dFH4be/DWuC3nAD/OEP+o+WAnFWd6nq+7BX2B4EzADaAX2Bv5pZpTNA7j7W3fu7e/82bdrUf6RJKCwMyxEedFD4xq+/TZF6tmhROBHcvTvMmBF6AvqPlhJxJoICYLsy2x0I3/zLOh14yoO5wHxgpxhjqrPCQl1VK1LvSkrg2WdDb2CbbeCtt+DNN2HnndMdWV6JMxF8AHQ1s85m1hgYQRgGKusr4CAAM2sLdAfmxRhTnRUW6suJSL364ouwTOTQoTBpUnisd2+dC0iD2BKBuxcB5wOTgTnAv9z9UzM728zOjppdD+xlZp8ArwKXuvv3ccW0MZQIROpJURHcdls4ETxzJjzwgK4OTrNYBzvcfSIwscJjY8rc/wY4NM4Y6osSgUg9OeaYMDV02DAYPTosVCFppVHvJBUVKRGI1Nm6dWE+dePG4YTwKaeEhKA51hlBa0Ilobg4nMtSIhCpg7ffDgvE3Hpr2B48GI49VkkggygRJKGwMPxUIhCphVWrwjUBe+8daqwMGJDuiKQaGhpKghKBSC395z9w8slhmbvzz4ebbtJiFxlMiSAJSgQitdSsWVjE+s03Q49AMlrSQ0NmtlmcgWQyJQKRJDz9NFx2Wbi/225haqiSQFaoMRGY2V5mNptwLQBm1sfM7ok9sgyiRCCSwHffhXWDf/lLeOkl+Omn8PgmOgWZLZL5l7qTUBNoKYC7fwzsG2dQmUaJQKQK7vDQQ9CjBzz/fDgP8N57YVhIskpS5wjcfaGVn+pVHE84mUmJQKQKixbBeeeFK4THjYOdMrJMmCQhmR7BQjPbC3Aza2xmlxANE+ULJQKRSEkJPPXUhiJxb78NU6cqCWS5ZBLB2YSVxNoTKor2Bc6NM6hMU5oIVH1U8tpnn8G++4Yrgl96KTzWq5fOBeSAZP4Fu7v7Se7e1t23dveTgbyqEasegeS1wsIw/t+nD8yeDf/4R7g6WHJGMongL0k+lrOUCCSvHXNMWCTm6KNhzhw49VSVh8gx1Q52mNlAYC+gjZn9rsxTLYC8KhheFK2irEQgeWPt2jDk07hxuDL4tNPC9FDJSYl6BI0JC8o3JKwnXHpbARwbf2iZQz0CyStvvRWKxN1yS9g+9FAlgRxXbY/A3d8A3jCzB939yxTGlHGUCCQvrFwZrgwePRq23x4GDkx3RJIiycyDWWNmtwO7AE1LH3T3A2OLKsMoEUjOe/PNUCRu4UK44AK48cZQK0jyQjKJ4FHgn8CRhKmkvwKWxBlUplEikJy32WbQokWoGrrXXumORlIsmVlDrdz9fqDQ3d9w918De8YcV0ZRIpCc4w5PPAGXXhq2+/WDjz9WEshTySSC6DDIt2Z2hJntCnSIMaaMo0QgOeXbb8OU0OOOg1dfVZE4SSoR3GBmLYH/B1wCjAMuijWqDKNEIDnBHR54IBSJmzQJbrsN3n1XReKk5nME7v5CdPdH4AAAM/tFnEFlGiUCyQmLFsFFF8Guu8J990G3bumOSDJEtT0CM2tgZieY2SVm1jN67Egzexv4a8oizABKBJK1iovDuYDSInHvvAOvv64kIOUkGhq6HzgTaAXcbWZ/B+4AbnP3XVMRXKZQIpCsNGcO7LNPOBdQWiRul110LkAqSTQ01B/o7e4lZtYU+B7Y0d2/S01omUPVRyWrFBbCrbfC9deHBeMfeURF4iShRIe29e5eAuDua83si3xMAqAegWSZYcPgxRfh+OPh7rth663THZFkuESJYCczmxndN2CHaNsAd/fesUeXIZQIJOP99BM0aBCKxF14IfzmNzBkSLqjkiyRKBFs9JoDZjYY+DOhWuk4d7+lijb7A3cBjYDv3X2/jf3c+lZUFKruNsirmquSNd54A848M5SIuPrqUCROpBYSFZ3bqEJzZtYAGA0cQljZ7AMze87dZ5dpswVwDzDY3b8ys4zswxYWqjcgGWjFinBl8Jgx0KVLODEsUgdxTh8YAMx193nuvh6YAFTsq54IPOXuXwG4++IY46kzJQLJOFOnhhlAY8fC734HM2fCgXlTB1LqWZyJoD2wsMx2QfRYWd2ALc1siplNN7NTq3ojMxtpZtPMbNqSJamvd6dEIBmneXNo1SosHv/HP4aicSJ1lFQiMLNmZta9lu9d1Vp2XmG7IbAbcAQwCLjSzCpd6eLuY929v7v3b9OmTS3D2HhKBJJ27vDPf8If/hC2d90VPvoI9tgjvXFJTqgxEZjZUcAM4KVou6+ZPZfEexcA25XZ7gB8U0Wbl9x9tbt/D0wF+iQTeCopEUhaffMNDB0KI0bAlCkbisRp3WCpJ8n0CK4hjPf/AODuM4BOSbzuA6CrmXU2s8bACKBiAnkW2MfMGprZpsAewJzkQk8dJQJJC3cYNy4UiXv5ZbjjjjAUpCJxUs+SuVa2yN1/tFp++3D3IjM7H5hMmD76gLt/amZnR8+Pcfc5ZvYSMBMoIUwxnVW7XYifEoGkxaJF4URwv34hIey4Y7ojkhyVTCKYZWYnAg3MrCtwIfB2Mm/u7hOBiRUeG1Nh+3bg9uTCTQ8lAkmZ0iJxw4eHInHvvQfdu6s+kMQqmb+uCwjrFa8DHiOUo8679QiUCCR2s2aFFcJGjNhQJG7nnZUEJHbJ9Ai6u/vlwOVxB5OplAgkVuvXw803hwXjW7aExx5TkThJqWQSwZ/MbFvgcWCCu38ac0wZp7BQlUclRsOGwcSJcOKJcNddkIYp0pLfauxzuvsBwP7AEmCsmX1iZlfEHVgmUY9A6t2aNbBuXbh/8cXw/PPw6KNKApIWSQ0+uvt37n43cDbhmoKrYo0qwygRSL2aMgV694abbgrbBx8MRx6Z1pAkvyVzQdnOZnaNmc0iLFH5NuHisLxRVKREIPXgxx/hrLPggAPCdulPkTRLZuT778B44FB3r3hlcF5Qj0A22pQpoUz0t9/CJZfAtdfCppumOyoRIIlE4O57piKQTKZEIBttiy3CSmFPPw27757uaETKqTYRmNm/3H24mX1C+WJxeblCmRKB1Io7jB8P06eH6qB9+4b7qg8kGShRj+C30c+8P4ulRCC1snAhnHNOWDd4zz1DkbhmzZQEJGNVe7LY3b+N7p7r7l+WvQHnpia8zKBEIEkpKYG//S0sGPP663DnnfCf/6hInGS8ZKaPHlLFY4fVdyCZTIlAkrJ4Mfz+9zBgAHzyCVx0kRa6lqyQ6BzBOYRv/l3MbGaZp5oDb8UdWCZRIpBqFRXB44+H+kDbbAMffADdumkYSLJKonMEjwGTgJuBUWUeX+nuy2KNKsMoEUiVZs6EM86AadNgyy1DfaDutV3ITyT9Eg0NubsvAM4DVpa5YWZbxR9a5lAikHLWrYOrr4bddoMvvwxLSA4alO6oROqsph7BkcB0wvTRsn1dB7rEGFdGUSKQcoYODWWiTz45FIlr1SrdEYlslGoTgbsfGf3snLpwMo+7EoEAq1eHErRNmoQrgy+4AA4/PN1RidSLZGoN/cLMNovun2xmfzKzjvGHlhmKi8NPlaHOY6++Cr16hfUCAA46SElAckoy00fvBdaYWR/gD8CXwMOxRpVBCgvDT/UI8tAPP8CZZ4bqoA0bwiFVzaQWyX7JJIIid3dgCPBnd/8zYQppXigqCj+VCPLM669Djx7w4IMwahR8/DHss0+6oxKJRTIDHivN7P+AU4B9zKwBkDeHRfUI8tRWW0H79mHBmN12S3c0IrFKpkdwPGHh+l+7+3dAe+D2WKPKIEoEecIdHn4Yfve7sN2nD7z/vpKA5IVklqr8DngUaGlmRwJr3f2h2CPLEEoEeeCrr+CII+DUU+G990KRONDVwZI3kpk1NBx4HzgOGA68Z2bHxh1YplAiyGElJXDvvaFI3NSpcPfd4aeKxEmeSeYcweXA7u6+GMDM2gCvAE/EGVimUCLIYYsXhxPBAwfC2LHQqVO6IxJJi2TOEWxSmgQiS5N8XU5QIsgxRUXwyCPhnEBpkbjJk5UEJK8l0yN4ycwmE9YthnDyeGJ8IWUWJYIcMmNGKBL34YfQunUoEtetW7qjEkm7ZE4W/x74G9Ab6AOMdfdLk3lzMxtsZp+b2VwzG5Wg3e5mVpyJ5x6UCHLA2rVw+eXQvz98/TU88URIAiICJF6PoCtwB7AD8Alwibt/newbR9cbjCYsbFMAfGBmz7n77Cra3QpMrn348VMiyAFDh4bhn1/9Cv70p3CNgIj8LFGP4AHgBeAYQgXSv9TyvQcAc919nruvByYQrk6u6ALgSWBxFc+lnRJBllq1KpSLBrj00lAt9MEHlQREqpAoETR39/vc/XN3vwPoVMv3bg8sLLNdED32MzNrDwwDxiR6IzMbaWbTzGzakiVLahnGxlEiyEIvvww9e8INN4TtAw7QegEiCSRKBE3NbFcz62dm/YBmFbZrUtXVOF5h+y7gUncvTvRG7j7W3fu7e/82bdok8dH1pzQRqPpoFli+HE4/PRz0mzbVwV8kSYkOb98Cfyqz/V2ZbQcOrOG9C4Dtymx3AL6p0KY/MMHCFZytgcPNrMjdn6nhvVNGPYIs8eqrYaGYJUvg//4PrroqJAMRqVGihWkO2Mj3/gDoamadga+BEcCJFT7j50VvzOxB4IVMSgKg6qNZo3Vr6NgRJk6EXXdNdzQiWSW2AQ93LzKz8wmzgRoAD7j7p2Z2dvR8wvMCmUI9ggzlDg89FK4J+POfQ5G4d99VfSCROoh15NvdJ1Lh4rPqEoC7nxZnLHWlRJCBFiyAs84KJ4X33jsUiWvWTElApI7yplREXSkRZJCSEvjLX8KMoLffhr/+Fd54Q0XiRDZSMtVHLVqr+Kpou6OZDYg/tMygRJBBFi+GK64IK4XNmgXnnQeb6LuMyMZK5n/RPcBA4IRoeyXhiuG8oESQZoWF4VxAaZG46dPDCeHtt093ZCI5I5lEsIe7nwesBXD35UDjWKPKIEoEafThhzBgQCgN8e9/h8d23FHnAkTqWTKJoDCqB+Tw83oEJbFGlUGUCNLgp5/CtQADBsB338FTT8Ghh6Y7KpGclcysobuBp4GtzexG4FjgilijyiBKBGkwdGiYEfTrX8Mdd8CWW6Y7IpGcVmMicPdHzWw6cBChbMRQd58Te2QZQokgRVauDL/kpk1Db+CSS+CQQ9IdlUheSGbWUEdgDfA88BywOnosLxQWhokpmpwSo0mTwrrBpUXi9t9fSUAkhZIZGnqRcH7AgKZAZ+BzYJcY48oYhYXqDcRm6VK4+GJ4+GHYeWc44oh0RySSl5IZGupVdjuqPHpWbBFlmMJCVR6NxSuvwEknwbJlcOWVYQWxJk3SHZVIXqr1Ic7dPzSz3eMIJhOpRxCTtm1hhx3CSeE+fdIdjUheqzERmNnvymxuAvQDUrs6TBoVFSkR1At3+Pvf4aOPQpmIXr3grbd0TYBIBkjmFGjzMrcmhHMGVS05mZPUI6gH8+eH6wDOOANmzgzXCYCSgEiGSNgjiC4k29zdf5+ieDKOEsFGKC4OheEuuwwaNIB774WRIzUFSyTDVJsIzKxhtKZAMstS5iwlgo2wZAlcfTXstx/87W+w3XY1v0ZEUi5Rj+B9wvmAGWb2HPA4sLr0SXd/KubYMoISQS0VFsKjj8Kpp4YicR9+CJ07axhIJIMlM2toK2ApYY3i0usJHFAikPKmTdtwHqBdu3BeoEuXdEclIjVIlAi2jmYMzWJDAijlsUaVQZQIkrBmDVxzDfzxj6EX8MwzKhInkkUSJYIGwOaUTwCllAhkg6FDQ5no3/wGbrsNttgi3RGJSC0kSgTfuvt1KYskQykRVGPFCmjcOBSJu/xyGDUKDjww3VGJSB0kmsens3soEVTpxRdDkbjrou8J++2nJCCSxRIlgoNSFkUGUyIo4/vv4eST4cgjoWVLGJI31xWK5LRqE4G7L0tlIJlKiSDy8suhQui//hWuDfjwQ9hjj3RHJSL1QHU1a6Dqo5Ftt4Xu3cPVwb161dxeRLKGrvWvQd4WnXOHcePg/PPDdq9e8OabSgIiOUiJoAZ5OTT0v//BQQeF6aCffgpr14bHdXWwSE5SIqhBXiWC4uJwUVivXjB9OowdC6++GqaIikjOijURmNlgM/vczOaa2agqnj/JzGZGt7fNLONWKMmrRLBkCVx/PRx8MMyeHXoEqhQqkvNi+18elbAeDRwG9ABOMLMeFZrNB/Zz997A9cDYuOKpq5xPBOvXw/33Q0lJKA8xYwY8+yy0b5/uyEQkReL8ujcAmOvu89x9PTCBCgvauPvb7r482nwX6BBjPHWS04ng/fdht93gzDPDEBBAp046FyCSZ+JMBO2BhWW2C6LHqnMGMKmqJ8xspJlNM7NpS5akdpXMnEwEa9bAJZfAwIGwfDm88AIccki6oxKRNIlzhnzSxerM7ABCIti7qufdfdr4hF0AABDoSURBVCzRsFH//v1TWvAuJxPBkCHwyitw1llw663hKmERyVtx9ggKgLJLUnUAvqnYyMx6A+OAIe6+NMZ4as09hxLBjz9umAZ65ZXw+uswZoySgIjEmgg+ALqaWWczawyMAJ4r28DMOhIWuDnF3b+IMZY6KS4OP7M+ETz/PPTosaFI3L77wv77pzUkEckcsSUCdy8CzgcmA3OAf7n7p2Z2tpmdHTW7CmgF3GNmM8xsWlzx1EVhYfiZtYlgyRI44QQ4+mho1Qp++ct0RyQiGSjWKjruPhGYWOGxMWXunwmcGWcMGyOrE8HkyXDSSWHdgOuug0svDesHiIhUoHJqCWR1ImjfPqwZcO+9YVhIRKQaumw0gdJEkBXVR0tKwsnfc84J2z17whtvKAmISI2UCBIoKgo/M75H8N//hhXCzjkn3C+dHSQikgQlggQyfmioqAhuvx169w6lIe6/PywiryJxIlIL2TDokTYZnwiWLoWbboJBg+Cee6Bdu3RHJCJZSD2CBDIyEaxbB/fdF84JtG0begJPP60kICJ1pkSQQMYlgnffhX79YOTIDUXitt9eReJEZKMoESSQMYlg9Wq4+GLYay9YuRImTlSROBGpNzpHkEDGJIIhQ0IP4Nxz4eaboUWLNAckIrlEPYIE0poIfvgBfvop3L/66nBNwOjRSgIiUu+UCBJIWyJ45plwIdi114btffYJheJERGKgRJBAyhPBokUwfDgMGxZmBA0fnqIPFpF8pkSQQEoTwaRJoRfw7LNw441hGcl+/VLwwSKS73SyOIGUJoLttgtXCN97L+y0Uwo+UEQkUI8ggVgTQUlJuBr4rLPCds+eYdUwJQERSTElggRiqz76+edhhbDzzoP581UkTkTSSokggXqvPlpUBLfcAn36wCefwN//HhaQUZE4EUkjnSNIoN6HhpYuhVtvhcMPD9cEbLttPb2xiEjdqUeQQL0kgrVrw4IxpUXiZs6Ep55SEhCRjKFEkMBGJ4K33oK+fcOCMa+9Fh7bbrt6iU1EpL4oESRQ50SwahVceGG4Injt2nAe4OCD6z0+EZH6oHMECdQ5EQwZEqaCnn9+WDhm883rPTYRkfqiRJBArRLBsmXQrFm4XXddeOwXv4gtNhGR+qKhoQQKC2GTTcItoSefDOUhrrkmbP/iF0oCIpI1lAgSKCysoTfw3Xdw7LHh1q4djBiRsthEROqLEkECCRPBxImhF/DCC2GxmPfeg113TWl8IiL1QecIEkiYCDp1CtVBR4+G7t1TGZZI1igsLKSgoIC1KqOSMk2bNqVDhw40qsUsFyWCBMolgpKScNCfORPuuy/0Bl55Ja3xiWS6goICmjdvTqdOnTCzdIeT89ydpUuXUlBQQOfOnZN+XaxDQ2Y22Mw+N7O5ZjaqiufNzO6Onp9pZhlVgP/nRDBnTrgm4MILoaBAReJEkrR27VpatWqlJJAiZkarVq1q3QOLLRGYWQNgNHAY0AM4wcx6VGh2GNA1uo0E7o0rnrooWVfIhatvClcHf/YZPPRQODegInEiSVMSSK26/L7j7BEMAOa6+zx3Xw9MAIZUaDMEeMiDd4EtzCxjivA0Wb2MkSvugKFDYfZsOOUU0B+1iOSYOBNBe2Bhme2C6LHatsHMRprZNDObtmTJknoPtDrDzm7Lv//4Cfzzn6FgnIhkpaeffhoz47PPPvv5sSlTpnDkkUeWa3faaafxxBNPAOFE96hRo+jatSs9e/ZkwIABTJo0aaNjufnmm9lxxx3p3r07kydPrrLNxx9/zMCBA+nVqxdHHXUUK1asqNXrayvORFDVV2evQxvcfay793f3/m3atKmX4JIxaBAcd1GlvCQiWWb8+PHsvffeTJgwIenXXHnllXz77bfMmjWLWbNm8fzzz7Ny5cqNimP27NlMmDCBTz/9lJdeeolzzz2X4uLiSu3OPPNMbrnlFj755BOGDRvG7bffXqvX11acs4YKgLKlNjsA39ShjYjkgIsughkz6vc9+/aFu+5K3GbVqlW89dZbvP766xx99NFcU1oBIIE1a9Zw3333MX/+fJo0aQJA27ZtGT58+EbF++yzzzJixAiaNGlC586d2XHHHXn//fcZOHBguXaff/45++67LwCHHHIIgwYN4vrrr0/69bUVZ4/gA6CrmXU2s8bACOC5Cm2eA06NZg/tCfzo7t/GGJOI5JlnnnmGwYMH061bN7baais+/PDDGl8zd+5cOnbsSIsWLWpse/HFF9O3b99Kt1tuuaVS26+//prtypSi79ChA19//XWldj179uS558Lh8vHHH2fhwoW1en1txdYjcPciMzsfmAw0AB5w90/N7Ozo+THAROBwYC6wBjg9rnhEJL1q+uYel/Hjx3PRRRcBMGLECMaPH0+/fv2qnV1T21k3d955Z9Jt3SuNfFf5eQ888AAXXngh1113HUcffTSNGzeu1etrK9YLytx9IuFgX/axMWXuO3BenDGISP5aunQpr732GrNmzcLMKC4uxsy47bbbaNWqFcuXLy/XftmyZbRu3Zodd9yRr776ipUrV9K8efOEn3HxxRfz+uuvV3p8xIgRjBpV/vKpDh06/PztHsIFd+3atav02p122omXX34ZgC+++IIXX3yxVq+vNXfPqttuu+3mIpIdZs+endbPHzNmjI8cObLcY/vuu69PnTrV165d6506dfo5xgULFnjHjh39hx9+cHf33//+937aaaf5unXr3N39m2++8Ycffnij4pk1a5b37t3b165d6/PmzfPOnTt7UVFRpXaLFi1yd/fi4mI/5ZRT/P7776/V66v6vQPTvJrjqorOiUjOGj9+PMOGDSv32DHHHMNjjz1GkyZNeOSRRzj99NPp27cvxx57LOPGjaNly5YA3HDDDbRp04YePXrQs2dPhg4dysbOWtxll10YPnw4PXr0YPDgwYwePZoGDRoAYabQtGnTfo67W7du7LTTTrRr147TTz+9xtdvDPMqxpwyWf/+/b30lyUimW3OnDnsvPPO6Q4j71T1ezez6e7ev6r26hGIiOQ5JQIRkTynRCAiscq24edsV5fftxKBiMSmadOmLF26VMkgRTxaj6BpLSska2EaEYlNhw4dKCgoIJXFIvNd6QpltaFEICKxadSoUa1WypL00NCQiEieUyIQEclzSgQiInku664sNrMlwJcp/MjWwPcp/LxU0/5lt1zev1zeN0j9/m3v7lXWyMi6RJBqZjatusuyc4H2L7vl8v7l8r5BZu2fhoZERPKcEoGISJ5TIqjZ2HQHEDPtX3bL5f3L5X2DDNo/nSMQEclz6hGIiOQ5JQIRkTynRBAxs8Fm9rmZzTWzUVU8b2Z2d/T8TDPrl4446yqJ/Tsp2q+ZZva2mfVJR5x1UdO+lWm3u5kVm9mxqYxvYyWzf2a2v5nNMLNPzeyNVMe4MZL422xpZs+b2cfR/p2ejjjrwsweMLPFZjarmucz47hS3WLG+XQDGgD/A7oAjYGPgR4V2hwOTAIM2BN4L91x1/P+7QVsGd0/LFv2L5l9K9PuNWAicGy6467nf7stgNlAx2h763THXc/7dxlwa3S/DbAMaJzu2JPcv32BfsCsap7PiOOKegTBAGCuu89z9/XABGBIhTZDgIc8eBfYwsy2TXWgdVTj/rn72+6+PNp8F6hdHdv0SebfDuAC4ElgcSqDqwfJ7N+JwFPu/hWAu2fTPiazfw40NzMDNickgqLUhlk37j6VEG91MuK4okQQtAcWltkuiB6rbZtMVdvYzyB8S8kGNe6bmbUHhgFjUhhXfUnm364bsKWZTTGz6WZ2asqi23jJ7N9fgZ2Bb4BPgN+6e0lqwotdRhxXtB5BYFU8VnFebTJtMlXSsZvZAYREsHesEdWfZPbtLuBSdy8OXyqzSjL71xDYDTgIaAa8Y2bvuvsXcQdXD5LZv0HADOBAYAfg32b2pruviDu4FMiI44oSQVAAbFdmuwPh20dt22SqpGI3s97AOOAwd1+aotg2VjL71h+YECWB1sDhZlbk7s+kJsSNkuzf5vfuvhpYbWZTgT5ANiSCZPbvdOAWD4Pqc81sPrAT8H5qQoxVRhxXNDQUfAB0NbPOZtYYGAE8V6HNc8Cp0Vn+PYEf3f3bVAdaRzXun5l1BJ4CTsmSb5Klatw3d+/s7p3cvRPwBHBuliQBSO5v81lgHzNraGabAnsAc1IcZ10ls39fEXo7mFlboDswL6VRxicjjivqEQDuXmRm5wOTCbMYHnD3T83s7Oj5MYTZJocDc4E1hG8pWSHJ/bsKaAXcE31zLvIMqYyYSJL7lrWS2T93n2NmLwEzgRJgnLtXOV0x0yT573c98KCZfUIYSrnU3bOiPLWZjQf2B1qbWQFwNdAIMuu4ohITIiJ5TkNDIiJ5TolARCTPKRGIiOQ5JQIRkTynRCAikueUCCQjRVVCZ5S5dUrQdlU9fN6DZjY/+qwPzWxgHd5jnJn1iO5fVuG5tzc2xuh9Sn8vs6KKnFvU0L6vmR1eH58tuUvTRyUjmdkqd9+8vtsmeI8HgRfc/QkzOxS4w917b8T7bXRMNb2vmf0D+MLdb0zQ/jSgv7ufX9+xSO5Qj0CygpltbmavRt/WPzGzShVGzWxbM5ta5hvzPtHjh5rZO9FrHzezmg7QU4Edo9f+LnqvWWZ2UfTYZmb2YlQff5aZHR89PsXM+pvZLUCzKI5Ho+dWRT//WfYbetQTOcbMGpjZ7Wb2gYW69Gcl8Wt5h6hAmZkNsLCOxEfRz+7RlbrXAcdHsRwfxf5A9DkfVfV7lDyUjtrXuulW0w0oJhQamwE8TbgKvkX0XGvClZilPdpV0c//B1we3W8ANI/aTgU2ix6/FLiqis97kGidAuA44D1CIbdPgM0I5Y8/BXYFjgHuK/PaltHPKYRv3z/HVKZNaYzDgH9E9xsTKk82A0YCV0SPNwGmAZ2riHNVmf17HBgcbbcAGkb3DwaejO6fBvy1zOtvAk6O7m9BqEe0Wbr/vXVL700lJiRT/eTufUs3zKwRcJOZ7Usoo9AeaAt8V+Y1HwAPRG2fcfcZZrYf0AN4Kyqd0ZjwTboqt5vZFcASQgXWg4CnPRRzw8yeAvYBXgLuMLNbCcNJb9ZivyYBd5tZE2AwMNXdf4qGo3rbhtXTWgJdgfkVXt/MzGYAnYDpwL/LtP+HmXUlVK9sVM3nHwocbWaXRNtNgY5kT20iiYESgWSLkwirU+3m7oVmtoBwEPuZu0+NEsURwMNmdjuwHPi3u5+QxGf83t2fKN0ws4OrauTuX5jZboQaMTeb2cvufl0yO+Hua81sCqG08vHA+NKPAy5w98k1vMVP7t7XzFoCLwDnAXcT6vG87u7DohPrU6p5vQHHuPvnycQr+UHnCCRbtAQWR0ngAGD7ig3MbPuozX3A/YQlAt8FfmFmpWP+m5pZtyQ/cyowNHrNZoRhnTfNrB2wxt0fAe6IPqeiwqhnUpUJhOJi+xCKrRH9PKf0NWbWLfrMKrn7j8CFwCXRa1oCX0dPn1am6UrCEFmpycAFFnWPzGzX6j5D8ocSgWSLR4H+ZjaN0Dv4rIo2+wMzzOwjwjj+n919CeHAON7MZhISw07JfKC7f0g4d/A+4ZzBOHf/COgFvB8N0VwO3FDFy8cCM0tPFlfwMmEt21c8LM8IYR2I2cCHFhY6/xs19NijWD4mlG6+jdA7eYtw/qDU60CP0pPFhJ5Doyi2WdG25DlNHxURyXPqEYiI5DklAhGRPKdEICKS55QIRETynBKBiEieUyIQEclzSgQiInnu/wObqvUpXq4hBgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn import metrics\n",
    "from sklearn.metrics import roc_auc_score\n",
    "\n",
    "# Let's get the labels of our test set; we will use these to evaluate our predictions\n",
    "\n",
    "df_with_labels = pd.read_csv('./data/abtest/test_dataset_sample_tail.csv')\n",
    "test_labels = df_with_labels.iloc[:, 0]\n",
    "labels = test_labels.to_numpy()\n",
    "\n",
    "# Calculate accuracy\n",
    "accuracy = sum(preds == labels) / len(labels)\n",
    "print(f'Accuracy: {accuracy}')\n",
    "\n",
    "# Calculate precision\n",
    "precision = sum(preds[preds == 1] == labels[preds == 1]) / len(preds[preds == 1])\n",
    "print(f'Precision: {precision}')\n",
    "\n",
    "# Calculate recall\n",
    "recall = sum(preds[preds == 1] == labels[preds == 1]) / len(labels[labels == 1])\n",
    "print(f'Recall: {recall}')\n",
    "\n",
    "# Calculate F1 score\n",
    "f1_score = 2 * (precision * recall) / (precision + recall)\n",
    "print(f'F1 Score: {f1_score}')\n",
    "\n",
    "# Calculate AUC\n",
    "auc = round(roc_auc_score(labels, preds), 4)\n",
    "print('AUC is ' + repr(auc))\n",
    "\n",
    "fpr, tpr, _ = metrics.roc_curve(labels, preds)\n",
    "\n",
    "plt.title('ROC Curve')\n",
    "plt.plot(fpr, tpr, 'b',\n",
    "label='AUC = %0.2f'% auc)\n",
    "plt.legend(loc='lower right')\n",
    "plt.plot([0,1],[0,1],'r--')\n",
    "plt.xlim([-0.1,1.1])\n",
    "plt.ylim([-0.1,1.1])\n",
    "plt.ylabel('True Positive Rate')\n",
    "plt.xlabel('False Positive Rate')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Evaluating Variant 2\n",
    "Same evaluate the accuracy, precision, recall, F1 score, and ROC/AUC for Variant1:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sending test traffic to the endpoint DEMO-xgb-churn-pred-2020-07-29-15-19-23. \n",
      "Please wait...\n",
      "........................................................................................................................Done!\n"
     ]
    }
   ],
   "source": [
    "predictions2 = ''\n",
    "print(f\"Sending test traffic to the endpoint {endpoint_name}. \\nPlease wait...\")\n",
    "with open('./data/abtest/test_sample_tail_input_cols.csv', 'r') as f:\n",
    "    for row in f:\n",
    "        print(\".\", end=\"\", flush=True)\n",
    "        payload = row.rstrip('\\n')\n",
    "        response = sm_runtime.invoke_endpoint(EndpointName=endpoint_name,\n",
    "                                   ContentType=\"text/csv\",\n",
    "                                   Body=payload,\n",
    "                                   TargetVariant=variant2[\"VariantName\"])\n",
    "        predictions2 = ','.join([predictions2, response['Body'].read().decode('utf-8')])\n",
    "        time.sleep(0.5)\n",
    "\n",
    "# Convert to numpy array        \n",
    "pred_np2 = np.fromstring(predictions2[1:], sep=',')\n",
    "        \n",
    "# Convert to binary predictions\n",
    "thresh = 0.5\n",
    "preds2 = np.where(pred_np2 > threshold, 1, 0)    \n",
    "\n",
    "print(\"Done!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "time.sleep(60) # give metrics time to catch up\n",
    "plot_endpoint_metrics()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.9583333333333334\n",
      "Precision: 0.8947368421052632\n",
      "Recall: 0.85\n",
      "F1 Score: 0.8717948717948718\n",
      "AUC is 0.915\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd5xU9fX/8deRrlIUUCkiKAIiHUSxoMaGJQJREWs0MdjQqD/zhWjsxp5ojChBNHZI7A3EihgLCopUNcTGCghiA5a2cH5/fO6yy+7s7GyZ/n4+HvPYuTP3zpy7LPfc+/ncz/mYuyMiIvlrq3QHICIi6aVEICKS55QIRETynBKBiEieUyIQEclzSgQiInlOiUBEJM8pEUhOMbMvzWyNma0ys6Vm9oCZbVtmnX3N7HUzW2lmP5nZ82bWtcw6TczsDjP7OvqshdFyiwq+18zsQjOba2arzazAzB43s+7J3F+R2qBEILnol+6+LdAL6A38sfgNMxsAvAw8C7QGOgAfA2+b2a7ROvWB14A9gUFAE2BfYAXQv4Lv/Bvwe+BCYHugE/AMcHRVgzezulXdRqQmTCOLJZeY2ZfAWe7+arR8C7Cnux8dLb8FzHH388psNxlY7u6nm9lZwJ+B3dx9VQLfuTvwCTDA3d+vYJ2pwCPuPj5aPiOKc/9o2YGRwEVAXWAKsMrdLy31Gc8Cb7r7X82sNfB3YCCwCrjd3e9M4FckUo6uCCRnmVlb4EhgYbS8NeHM/vEYq/8bOCx6fijwUiJJIHIIUFBREqiCIcDeQFfgMeBEMzMAM9sOOByYaGZbAc8TrmTaRN9/kZkdUcPvlzylRCC56BkzWwksApYBV0Wvb0/4m18SY5slQHH7f/MK1qlIVdevyI3u/r27rwHeAhw4IHrveOBdd18M7AW0dPdr3X29u38O3AsMr4UYJA8pEUguGuLujYGDgC6UHOB/ADYBrWJs0wr4Lnq+ooJ1KlLV9SuyqPiJhzbbicBJ0UsnA49Gz3cBWpvZj8UP4DJgx1qIQfKQEoHkLHd/E3gAuC1aXg28C5wQY/VhhA5igFeBI8xsmwS/6jWgrZn1i7POamDrUss7xQq5zPIE4Hgz24XQZPRk9Poi4At3b1bq0djdj0owXpEtKBFIrrsDOMzMekXLo4FfR7d6Njaz7czsemAAcE20zsOEg+2TZtbFzLYys+ZmdpmZlTvYuvt/gbuBCWZ2kJnVN7OGZjbczEZHq80CfmVmW5tZR+C3lQXu7h8By4HxwBR3/zF6633gZzMbZWaNzKyOmXUzs72q8wsSUSKQnObuy4GHgCui5f8ARwC/IrTrf0W4xXT/6ICOu68jdBh/ArwC/Ew4+LYAplfwVRcCdwFjgB+B/wFDCZ26ALcD64FvgQcpaeapzIQolsdK7dNG4JeE22O/IDRpjQeaJviZIlvQ7aMiInlOVwQiInlOiUBEJM8pEYiI5DklAhGRPJd1xa1atGjh7du3T3cYIiJZZebMmd+5e8tY72VdImjfvj0zZsxIdxgiIlnFzL6q6D01DYmI5DklAhGRPKdEICKS55QIRETynBKBiEieUyIQEclzSgQiInlOiUBEJM8pEYiI5DklAhGRPKdEICKS55QIRETynBKBiEieS1oiMLP7zWyZmc2t4H0zszvNbKGZzTazPsmKRUREKpbMK4IHgEFx3j8S2D16jADuSWIsIiJSgaTNR+Du08ysfZxVBgMPubsD75lZMzNr5e5LkhWTiEg6rV8PS5fCkiWwbh0MHJjuiIJ0TkzTBlhUarkgeq1cIjCzEYSrBtq1a5eS4EREErVuXTjAL14cDvIV/fzuO+cIpjCFI+jY0fjvf9MdeZDORGAxXvNYK7r7OGAcQL9+/WKuIyJS29auDQfweAf3xYvh++/Lb1unDuy0E7RuDR06wC+7f8lvpo9gt/+9wvtXvUiDoUelfocqkM5EUADsXGq5LbA4TbGISB4pLCw5wMc7yP/wQ/lt69ULB/hWraBjRzjggHCwb9Vqy58tWsBWWwEbN8KYMXDZZWAGd99N/7MHZdQ9m+lMBM8BI81sIrA38JP6B0SkJlav3vJAXtFB/qefym9br17JQbxzZzj44PIH91atoHnz6ACfqBNPhCefhCOPhLFjIQObt5OWCMxsAnAQ0MLMCoCrgHoA7j4WmAQcBSwECoEzkxWLiGS3VasqP7gvWQI//1x+2wYNwgG8VSvo2hUOOST2Gfz224cT9lqxYUP4Wa8enHkmDBkCp5xSi19QuyzctJM9+vXr5zNmzEh3GCJSQ+6wcmViZ/CrVpXfvmHDkgN5rDP34p/bbZfi4+/MmfCb34QrgcsuS+EXx2dmM929X6z30tk0JCI5yD2cmZc9oMc6yK9eXX77Ro3CQbx1a+jVK7SoxDrIN22aYSfYa9bA1VfDX/4CO+wA3bqlO6KEKRGISELc4ccfEzuDX7Om/PbbbFNyIO/Xr+Iz+SZNMuwAn4jp0+G00+C//4WzzoJbb4VmzdIdVcKUCETynHu4OyaRM/i1a8tv37hxyYG8f/+Km2kaN079vqVMce/xq6+GTogso0QgkqPcYcWK2Af0ss/XrSu/fZMmJQfyffet+Ax+221Tv28ZYdKkcCVwzTWw116wYEEYPJCFlAhEssymTeEAX9GdM8XPly4NJQ3Katas5EC+//6xO1xbtQpNORLDd9/BxRfDI4/AnnvCqFGw9dZZmwRAiUAkY2zaBMuXJ3YGX1RUfvvttis5kB90UPkDe/HzRo1Svmu5wR0efxxGjgxtaVdeGe4KatAg3ZHVmBKBSJJt3BgO8JWdwX/7bewDfPPmJQfyPfYo3zRT/GjYMPX7lleWLQu3he6xR+gL6NEj3RHVGiUCkWoqKgrHhngDnJYsCQf4jRvLb9+iRcnBvFu32O3vO+2UEyec2csdJk8O97DuuCNMmxYSQN3cOnTm1t6I1IKionDwruwMftmy0JxT1g47lBzMe/aMfQa/005Qv37q902q4PPP4Xe/g9dfhxdfhKOOgj65OX+WEoHkjQ0bSmrBxzvIL1sWTgRLMwsH+OKDeZ8+sc/gd9wxVBWQLLZxI9x5J1x+eTjzHzsWBsWbYyv7KRFI1iue7KOyWvDLl5ffdqutwsG7VSto2zbcBRjrHvgddsi51gCpyAknwNNPw9FHhyTQtm26I0o6/WlLxlq3rvIiY2Gyj/LbFteCb9UKdtkFBgyIfQa/ww5Zfdef1Jb168NlX716YWTw8cfDSSdl4RDn6lEikJQrnuyjsjP4WJN91K1bMtnHrrvCfvtVXAteB3hJyAcfhLuBhg8PzUFHZc6EMamiRCC1pvRkH/HO4Cua7KO4I7VTJzjwwNhn8Jsn+xCpqcLCMBbg9tvDH1evXumOKG2UCKRSZSf7qOhnrMk+6tcvOZB36RIm+6ioFrwO8JIy774bisT9739w9tlw882hnGmeUiLIY8W14Cs7yK9cWX7bBg22vAf+sMNin8HX6mQfIrWlbt3weP31cHaS55QIckzxZB+JzOYUa7KPRo1KDuQ9eoS75mIVG2vWTAd4yTIvvhiKxF17bbg9bN48dSRFlAiyhHtoeknkDL6wsPz2W29d8T3wpZ9n3GQfIjW1fDlcdBE89hh07w6jR2d9kbjapkSQZsWTfSRyBh9vso/WrcNkHxVN3de4sQ7wkmfc4V//ggsuCGdR11wTkoCGdJejRJAk7uH2x8om+og32UfxgXyffSo+g8/pyT5EamLZsjAmYM894b77smrqyFRTIqiGlSvhiy8qP4OPVQu+adOSA/m++1ZcSTJvJ/sQqYlNm0JfwDHHhCHj//lPaA5SM1BcSgRVtGYN7Lxz+VslmzUrOZgPHBj7DppWrULTpIgkwcKFoUjc1KklReLyeGxAVSgRVNH334ckcN55cPLJ4SC/006a7EMkbYqK4I474Iorwn3N48eHstGSMCWCKirusN1nn1DeQETS7IQT4JlnYPBguPvucHYmVaJEUEXFiUBXACJptG5dGIperx6MGBHqBA0bplvjqkmD+quo+B59JQKRNJk+Hfr2DWUhIDQDnXiikkANKBFUUfEVgTp9RVJs9Wq45JJQU/ynn3J2trB0UNNQFalpSCQN3nknFIn7/HM491y46SZo0iTdUeWMpF4RmNkgM/vUzBaa2egY7zc1s+fN7GMzm2dmZyYzntqgpiGRNKhfPzymTg0dwkoCtSppicDM6gBjgCOBrsBJZta1zGrnA/PdvSdwEPAXM8vo8d9qGhJJkeeeCxPFQKifMndumKhCal0yrwj6Awvd/XN3Xw9MBAaXWceBxmZmwLbA90BREmOqMTUNiSTZsmXhLqDBg8PAsOLLcI0OTppkJoI2wKJSywXRa6XdBewBLAbmAL93901lP8jMRpjZDDObsTzWDOQppEQgkiTu8MgjsMceYfL4668P00jq8jvpkpkIYt3L5WWWjwBmAa2BXsBdZlau8c/dx7l7P3fv17Jly9qPtArURyCSJN9+GzqCO3eGWbNCs1C9eumOKi8kMxEUADuXWm5LOPMv7UzgKQ8WAl8AXZIYU40VXxE0bJjeOERywqZN8Oyz4Wpgp53g7bfhrbfCVYGkTDITwQfA7mbWIeoAHg48V2adr4FDAMxsR6Az8HkSY6qxNWtCEtD8uiI19NlnYZrIIUNg8uTwWo8e6gtIg6Qdzty9CBgJTAEWAP9293lmdo6ZnROtdh2wr5nNAV4DRrn7d8mKqTasWaNmIZEaKSqCW26Bnj1h9my4/34ViUuzpA4oc/dJwKQyr40t9XwxcHgyY6hthYVKBCI1ctxx4dbQoUNhzJhQn13SSiOLq2jNGt3EIFJl69aFWkD164cO4dNOCwlB9YEyglq6q0hNQyJV9M47YYKY4iJxgwbB8ccrCWQQJYIqUtOQSIJWrYLf/x723z/8x+nfP90RSQXUNFRFahoSScB//gOnngpffQUjR8INN0DjxumOSiqgRFBFa9ZAmse0iWS+Ro1g223DmID99093NFKJhJuGzGybZAaSLdRHIFKBp5+Gyy4Lz/v2DbeGKglkhUoTgZnta2bzCWMBMLOeZnZ30iPLUOojEClj6dIwb/CvfgUvvVQy/F6jLrNGIv9StxNqAq0AcPePgYHJDCqTqY9AJOIODz0EXbvC88+HfoDp03WmlIUS6iNw90W25a1eG5MTTuZT05BI5Ntv4fzzwwjh8eOhS0aXCZM4ErkiWGRm+wJuZvXN7FKiZqJ8pEQgeW3TJnjqqZIice+8A9OmKQlkuUQSwTmEmcTaECqK9gLOS2ZQmWrjRli/XolA8tQnn8DAgWFE8Esvhde6d1dfQA5I5F+ws7uf4u47uvsO7n4qYTKZvKNpKiUvbdgQ2v979oT58+HBB8PoYMkZiSSCvyf4Ws7T7GSSl447LkwSc+yxsGABnH66ykPkmAo7i81sALAv0NLMLin1VhMgLwuGa3YyyRtr14Ymn/r1w8jgM84It4dKTop3RVCfMKF8XaBxqcfPwPHJDy3zqGlI8sLbb4cicTfdFJYPP1xJIMdVeEXg7m8Cb5rZA+7+VQpjylhqGpKctnJlGBk8ZgzssgsMGJDuiCRFEhlHUGhmtwJ7Aptn6nX3XyQtqgylRCA56623QpG4RYvgggvgz38OtYIkLySSCB4F/gUcQ7iV9NfA8mQGlanURyA5a5ttoEmTUDV0333THY2kWCJ3DTV39/uADe7+prv/BtgnyXFlJPURSM5whyeegFGjwnKfPvDxx0oCeSqRRLAh+rnEzI42s95A2yTGlLHUNCQ5YcmScEvoCSfAa6+pSJwklAiuN7OmwP8DLgXGAxclNaoMpUQgWc0d7r8/FImbPBluuQXee09/0FJ5H4G7vxA9/Qk4GMDM9ktmUJlKfQSS1b79Fi66CHr3hnvvhU6d0h2RZIgKrwjMrI6ZnWRml5pZt+i1Y8zsHeCulEWYQdRHIFln48bQF1BcJO7dd+GNN5QEZAvxmobuA84CmgN3mtk/gduAW9y9dyqCyzRqGpKssmABHHBA6AsoLhK3557qC5By4jUN9QN6uPsmM2sIfAd0dPelqQkt8xQWhv9D9eqlOxKRODZsgJtvhuuuCxPGP/KIisRJXPESwXp33wTg7mvN7LN8TgJQMjuZ6m1JRhs6FF58EU48Ee68E3bYId0RSYaLlwi6mNns6LkBu0XLBri790h6dBlGk9JIxlqzBurUCUXiLrwQfvc7GDw43VFJloiXCGo854CZDQL+RqhWOt7db4qxzkHAHUA94Dt3P7Cm35ssSgSSkd58E846K5SIuOqqUCROpAriFZ2rUaE5M6sDjAEOI8xs9oGZPefu80ut0wy4Gxjk7l+bWUZfwxYWKhFIBvn55zAyeOxY2HXX0DEsUg3JvH2gP7DQ3T939/XARKDsterJwFPu/jWAuy9LYjw1VtxHIJJ206aFO4DGjYNLLoHZs+EXeVcHUmpJMhNBG2BRqeWC6LXSOgHbmdlUM5tpZqfH+iAzG2FmM8xsxvLl6at3p6YhyRiNG0Pz5mHy+L/8JRSNE6mmhBKBmTUys85V/OxY99Z4meW6QF/gaOAI4AozKzfSxd3HuXs/d+/XsmXLKoZRe5QIJG3c4V//gv/7v7Dcuzd89BHsvXd645KcUGkiMLNfArOAl6LlXmb2XAKfXQDsXGq5LbA4xjovuftqd/8OmAb0TCTwdFAfgaTF4sUwZAgMHw5Tp5aMbNR9zFJLErkiuJrQ3v8jgLvPAtonsN0HwO5m1sHM6gPDgbIJ5FngADOra2ZbA3sDCxILPfXURyAp5Q7jx4cicS+/DLfdFpqCdDYitSyRiWmK3P0nq+LZh7sXmdlIYArh9tH73X2emZ0TvT/W3ReY2UvAbGAT4RbTuVXbhdRR05Ck1Lffho7gPn1CQujYMd0RSY5KJBHMNbOTgTpmtjtwIfBOIh/u7pOASWVeG1tm+Vbg1sTCTS8lAkm64iJxw4aFInHTp0PnzqoPJEmVyF/XBYT5itcBjxHKUeflfASFhWoakiSaOzfMEDZ8eEmRuD32UBKQpEvkiqCzu18OXJ7sYDKZu64IJEnWr4cbbwwTxjdtCo89piJxklKJJIK/mlkr4HFgorvPS3JMGWnDBti0SYlAkmDoUJg0CU4+Ge64A9J4i7Tkp0qvOd39YOAgYDkwzszmmNmfkh1YptHsZFKrCgth3brw/OKL4fnn4dFHlQQkLRJqfHT3pe5+J3AOYUzBlUmNKgNpdjKpNVOnQo8ecMMNYfnQQ+GYY9IakuS3RAaU7WFmV5vZXMIUle8QBoflFc1OJjX2009w9tlw8MFhufinSJol0kfwT2ACcLi7lx0ZnDeUCKRGpk4NZaKXLIFLL4VrrtHlpWSMShOBu++TikAynfoIpEaaNQszhT39NOy1V7qjEdlChYnAzP7t7sPMbA5bFovLyxnK1EcgVeIOEybAzJmhOmivXuG56gNJBop3RfD76Kd6sVDTkFTBokVw7rlh3uB99ikZgKIkIBmqws5id18SPT3P3b8q/QDOS014mUOJQCq1aRP84x9hwpg33oDbb4f//Ed/NJLxErl99LAYrx1Z24FkOvURSKWWLYM//AH694c5c+Cii8KE8iIZLl4fwbmEM/9dzWx2qbcaA28nO7BMoz4CiamoCB5/PNQH2mkn+OAD6NRJzUCSVeL1ETwGTAZuBEaXen2lu3+f1KgykJqGpJzZs+G3v4UZM2C77UJ9oM5VnchPJP3iNQ25u38JnA+sLPXAzLZPfmiZRU1Dstm6dXDVVdC3L3z1VZhC8ogj0h2VSLVVdkVwDDCTcPto6WtdB3ZNYlwZR1cEstmQIaFM9KmnhiJxzZunOyKRGqkwEbj7MdHPDqkLJ3OtWQP166vvL2+tXg1160KDBmFk8AUXwFFHpTsqkVqRSK2h/cxsm+j5qWb2VzNrl/zQMovmIshjr70G3buH+QIADjlESUBySiK3j94DFJpZT+D/gK+Ah5MaVQYqLFQiyDs//ghnnRWqg9atC4fFupNaJPslkgiK3N2BwcDf3P1vhFtI88qaNbp1NK+88QZ07QoPPACjR8PHH8MBB6Q7KpGkSKT66Eoz+yNwGnCAmdUB6iU3rMyjpqE8s/320KZNmDCmb990RyOSVIlcEZxImLj+N+6+FGgD3JrUqDKQEkGOc4eHH4ZLLgnLPXvC++8rCUheSGSqyqXAo0BTMzsGWOvuDyU9sgyjPoIc9vXXcPTRcPrpMH16yb3CGh0seSKRu4aGAe8DJwDDgOlmdnyyA8s06iPIQZs2wT33hCJx06bBnXeGn8r4kmcS6SO4HNjL3ZcBmFlL4FXgiWQGlmnWrIHWrdMdhdSqZctCR/CAATBuHLRvn+6IRNIikT6CrYqTQGRFgtvlFDUN5YiiInjkkdAnUFwkbsoUJQHJa4lcEbxkZlMI8xZD6DyelLyQMpOahnLArFmhSNyHH0KLFqFIXKdO6Y5KJO0S6Sz+A/APoAfQExjn7qMS+XAzG2Rmn5rZQjMbHWe9vcxsYyb3PeiuoSy2di1cfjn06wfffANPPBGSgIgA8ecj2B24DdgNmANc6u7fJPrB0XiDMYSJbQqAD8zsOXefH2O9m4EpVQ8/dZQIstiQIaH559e/hr/+NYwREJHN4l0R3A+8ABxHqED69yp+dn9gobt/7u7rgYmE0cllXQA8CSyL8V5G2LQpnFQqEWSRVatCuWiAUaNCtdAHHlASEIkhXiJo7O73uvun7n4b0L6Kn90GWFRquSB6bTMzawMMBcbG+yAzG2FmM8xsxvLly6sYRs2tXRt+qo8gS7z8MnTrBtdfH5YPPljzBYjEES8RNDSz3mbWx8z6AI3KLFcm1mgcL7N8BzDK3TfG+yB3H+fu/dy9X8uWLRP46tqluQiyxA8/wJlnhoN+w4Y6+IskKN5dQ0uAv5ZaXlpq2YFfVPLZBcDOpZbbAovLrNMPmGhhBGcL4CgzK3L3Zyr57JRSIsgCr70WJopZvhz++Ee48sqQDESkUvEmpjm4hp/9AbC7mXUAvgGGAyeX+Y7Nk96Y2QPAC5mWBEDTVGaFFi2gXTuYNAl69053NCJZJZFxBNXi7kVmNpJwN1Ad4H53n2dm50Tvx+0XyCTFVwTqI8gg7vDQQ2FMwN/+ForEvfee6gOJVEPSEgGAu0+izOCzihKAu5+RzFhqQk1DGebLL+Hss0On8P77l9zbqyQgUi15VyqiOtQ0lCE2bYK//z3cEfTOO3DXXfDmm/qHEamhRKqPWjRX8ZXRcjsz65/80DKHmoYyxLJl8Kc/hZnC5s6F88+HrXQuI1JTifwvuhsYAJwULa8kjBjOG2oaSqMNG0JfQHGRuJkzQ4fwLrukOzKRnJFIItjb3c8H1gK4+w9A/aRGlWGUCNLkww+hf/9QGuKVV8JrHTuqL0CkliWSCDZE9YAcNs9HsCmpUWUY9RGk2Jo1YSxA//6wdCk89RQcfni6oxLJWYncNXQn8DSwg5n9GTge+FNSo8ow6iNIsSFDwh1Bv/kN3HYbbLdduiMSyWmVJgJ3f9TMZgKHEMpGDHH3BUmPLIOoaSgFVq6EevXCaOA//hEuvRQOOyzdUYnkhUTuGmoHFALPA88Bq6PX8saaNaFZukGDdEeSoyZPDvMGFxeJO+ggJQGRFEqkaehFQv+AAQ2BDsCnwJ5JjCujFBaGE1X1UdayFSvg4ovh4Ydhjz3g6KPTHZFIXkqkaah76eWo8ujZSYsoA2mayiR49VU45RT4/nu44oowg5guuUTSosolJtz9QzPbKxnBZCrNTpYEO+4Iu+0WOoV79kx3NCJ5rdJEYGaXlFrcCugDpH52mDQqLFQiqDF3+Oc/4aOPQpmI7t3h7bfV3iaSARIZR9C41KMBoc8g1pSTOUtNQzX0xRdhHMBvfwuzZ5fchqUkIJIR4l4RRAPJtnX3P6QonoykpqFq2rgxFIa77DKoUwfuuQdGjFB9IJEMU2EiMLO60ZwCiUxLmdOUCKpp+XK46io48ED4xz9g550r30ZEUi7eFcH7hP6AWWb2HPA4sLr4TXd/KsmxZYzCwlDvTBKwYQM8+iicfnr4pX34IXTooGYgkQyWyF1D2wMrCHMUF48ncCBvEoH6CBI0Y0ZJP0Dr1qFfYNdd0x2ViFQiXiLYIbpjaC4lCaCYJzWqDKOmoUoUFsLVV8Nf/hKuAp55RkXiRLJIvERQB9iWLRNAMSUCKTFkSCgT/bvfwS23QLNm6Y5IRKogXiJY4u7XpiySDKZxBDH8/DPUrx9qb1x+OYweDb/4RbqjEpFqiHcfn3r3IuojKOPFF0ORuGuj84QDD1QSEMli8RLBISmLIoNt2ABFRboiAOC77+DUU+GYY6BpUxicV+MKRXJWhYnA3b9PZSCZSnMRRF5+OVQI/fe/w9iADz+EvfdOd1QiUguqXHQu32h2skirVtC5cxgd3L175euLSNbQWP9K5O0VgTuMHw8jR4bl7t3hrbeUBERykBJBJfIyEfzvf3DIIeF20HnzYO3a8LpGB4vkJCWCShQWhp95kQg2bgyDwrp3h5kzYdw4eO21cIuoiOSspCYCMxtkZp+a2UIzGx3j/VPMbHb0eMfMMm6GkrzqI1i+HK67Dg49FObPD1cEqhQqkvOS9r88KmE9BjgS6AqcZGZdy6z2BXCgu/cArgPGJSue6sr5pqH16+G++2DTplAeYtYsePZZaNMm3ZGJSIok83SvP7DQ3T939/XARMpMaOPu77j7D9Hie0DbJMZTLTmdCN5/H/r2hbPOCk1AAO3bqy9AJM8kMxG0ARaVWi6IXqvIb4HJsd4wsxFmNsPMZixfntpZMnOyj6CwEC69FAYMgB9+gBdegMMOS3dUIpImyRxHkHCxOjM7mJAI9o/1vruPI2o26tevX0oL3uVkH8HgwfDqq3D22XDzzWGUsIjkrWReERQApaekagssLruSmfUAxgOD3X1FEuOplpxpGvrpp5LbQK+4At54A8aOVRIQkaQmgg+A3c2sg5nVB4YDz5VewczaESa4Oc3dP0tiLNWWE01Dzz8PXbuWFIkbOBAOOiitIYlI5khaInD3ImAkMAVYAPzb3eeZ2Tlmdk602pVAc+BuMyMUXDUAABDJSURBVJtlZjOSFU91ZfUVwfLlcNJJcOyx0Lw5/OpX6Y5IRDJQUmsNufskYFKZ18aWen4WcFYyY6ipNWugbl2oVy/dkVTRlClwyilh3oBrr4VRo8L8ASIiZajoXCWydnayNm3CnAH33BOahUREKqBho5XImtnJNm0Knb/nnhuWu3WDN99UEhCRSikRVCIrZif773/DDGHnnhueF98dJCKSACWCSmR001BREdx6K/ToEUpD3HdfmEReReJEpArUR1CJjE4EK1bADTfAEUfA3XdD69bpjkhEspCuCCqRcX0E69bBvfeGPoEddwxXAk8/rSQgItWmRFCJjOojeO896NMHRowoKRK3yy4qEiciNaJEUImMaBpavRouvhj23RdWroRJk1QkTkRqjfoIKpERTUODB4crgPPOgxtvhCZN0hyQiOQSXRFUIm1NQz/+WFLf4qqrwpiAMWOUBESk1ikRVCItTUPPPBMGgl1zTVg+4IBQKE5EJAmUCCqR0kTw7bcwbBgMHRruCBo2LEVfLCL5TIkgDvcU9hFMnhyuAp59Fv785zCNZJ8+KfhiEcl36iyOY9268DMlfQQ77xxGCN9zD3TpkoIvFBEJdEUQR1LnIti0KYwGPvvssNytW5g1TElARFJMiSCOpCWCTz8NM4Sdfz588YWKxIlIWikRxFHr01QWFcFNN0HPnjBnDvzzn2ECGRWJE5E0Uh9BHMVXBLXWR7BiBdx8Mxx1VBgT0KpVLX2wiEj16YogjlppGlq7NkwYU1wkbvZseOopJQERyRhKBHHUuGno7behV68wYczrr4fXdt65VmITEaktSgRxVLtpaNUquPDCMCJ47drQD3DoobUen4hIbVAfQRzVbhoaPDjcCjpyZJg4Ztttaz02EZHaokQQR5USwfffhxUbNYJrrw2v7bdf0mITEaktahqKI+E+giefDOUhrr46LO+3n5KAiGQNJYI4Ku0jWLoUjj8+PFq3huHDUxabiEhtUSKII27T0KRJ4SrghRfCZDHTp0Pv3imNT0SkNqiPII7iRBBz4G/79qE66Jgx0LlzKsMSyRobNmygoKCAtSqjkjINGzakbdu21KtXL+FtlAjiKCyEBg1gq60IA8LGjAkDwu69N1wNvPpqukMUyWgFBQU0btyY9u3bY2bpDifnuTsrVqygoKCADh06JLxdUpuGzGyQmX1qZgvNbHSM983M7ozen21mGVWAf/M0lQsWhDEBF14IBQUqEieSoLVr19K8eXMlgRQxM5o3b17lK7CkJQIzqwOMAY4EugInmVnXMqsdCewePUYA9yQrnupYv3oD/1d0Qxgd/Mkn8NBDoW9AReJEEqYkkFrV+X0n84qgP7DQ3T939/XARGBwmXUGAw958B7QzMwypgjPVj9+zzmrb4MhQ2D+fDjtNNAftYjkmGQmgjbAolLLBdFrVV0HMxthZjPMbMby5ctrPdCKDB6xI6/8ZQ7861+hYJyIZKWnn34aM+OTTz7Z/NrUqVM55phjtljvjDPO4IknngBCR/fo0aPZfffd6datG/3792fy5Mk1juXGG2+kY8eOdO7cmSlTpsRc5+OPP2bAgAF0796dX/7yl/z8888AvPLKK/Tt25fu3bvTt29fXi+uYVZDyUwEsU6dvRrr4O7j3L2fu/dr2bJlrQSXiEGD4ISLyuUlEckyEyZMYP/992fixIkJb3PFFVewZMkS5s6dy9y5c3n++edZuXJljeKYP38+EydOZN68ebz00kucd955bNy4sdx6Z511FjfddBNz5sxh6NCh3HrrrQC0aNGC559/njlz5vDggw9y2mmn1SieYsm8a6gAKF1qsy2wuBrriEgOuOgimDWrdj+zVy+4447466xatYq3336bN954g2OPPZariysAxFFYWMi9997LF198QYMGDQDYcccdGTZsWI3iffbZZxk+fDgNGjSgQ4cOdOzYkffff58BAwZssd6nn37KwIEDATjssMM44ogjuO666+hdaqzSnnvuydq1a1m3bt3mGKsrmVcEHwC7m1kHM6sPDAeeK7POc8Dp0d1D+wA/ufuSJMYkInnmmWeeYdCgQXTq1Intt9+eDz/8sNJtFi5cSLt27WjSpEml61588cX06tWr3OOmm24qt+4333zDzqVK0bdt25Zvvvmm3HrdunXjuefC4fLxxx9n0aJF5dZ58skn6d27d42TACTxisDdi8xsJDAFqAPc7+7zzOyc6P2xwCTgKGAhUAicmax4RCS9KjtzT5YJEyZw0UUXATB8+HAmTJhAnz59Kry7pqp33dx+++0Jr+teruU75vfdf//9XHjhhVx77bUce+yx1K9ff4v3582bx6hRo3j55ZerFGtFkjqgzN0nEQ72pV8bW+q5A+cnMwYRyV8rVqzg9ddfZ+7cuZgZGzduxMy45ZZbaN68OT/88MMW63///fe0aNGCjh078vXXX7Ny5UoaN24c9zsuvvhi3njjjXKvDx8+nNGjtxw+1bZt2y3O7gsKCmjdunW5bbt06bL5IP/ZZ5/x4osvbrHN0KFDeeihh9htt90q/yUkwt2z6tG3b18Xkewwf/78tH7/2LFjfcSIEVu8NnDgQJ82bZqvXbvW27dvvznGL7/80tu1a+c//viju7v/4Q9/8DPOOMPXrVvn7u6LFy/2hx9+uEbxzJ0713v06OFr1671zz//3Dt06OBFRUXl1vv222/d3X3jxo1+2mmn+X333efu7j/88IP36NHDn3jiibjfE+v3DszwCo6rKjonIjlrwoQJDB06dIvXjjvuOB577DEaNGjAI488wplnnkmvXr04/vjjGT9+PE2bNgXg+uuvp2XLlnTt2pVu3boxZMgQanrX4p577smwYcPo2rUrgwYNYsyYMdSpUwcIdwrNmDFjc9ydOnWiS5cutG7dmjPPDK3md911FwsXLuS6667b3BexbNmyGsUEYB6jzSqT9evXz4t/WSKS2RYsWMAee+yR7jDyTqzfu5nNdPd+sdbXFYGISJ5TIhARyXNKBCKSVNnW/JztqvP7ViIQkaRp2LAhK1asUDJIEY/mI2hYxQrJmphGRJKmbdu2FBQUkMpikfmueIayqlAiEJGkqVevXpVmypL0UNOQiEieUyIQEclzSgQiInku60YWm9ly4KsUfmUL4LsUfl+qaf+yWy7vXy7vG6R+/3Zx95g1MrIuEaSamc2oaFh2LtD+Zbdc3r9c3jfIrP1T05CISJ5TIhARyXNKBJUbl+4Akkz7l91yef9yed8gg/ZPfQQiInlOVwQiInlOiUBEJM8pEUTMbJCZfWpmC81sdIz3zczujN6fbWZ90hFndSWwf6dE+zXbzN4xs57piLM6Ktu3UuvtZWYbzez4VMZXU4nsn5kdZGazzGyemb2Z6hhrIoG/zaZm9ryZfRzt35npiLM6zOx+M1tmZnMreD8zjisVTWacTw+gDvA/YFegPvAx0LXMOkcBkwED9gGmpzvuWt6/fYHtoudHZsv+JbJvpdZ7HZgEHJ/uuGv5364ZMB9oFy3vkO64a3n/LgNujp63BL4H6qc79gT3byDQB5hbwfsZcVzRFUHQH1jo7p+7+3pgIjC4zDqDgYc8eA9oZmatUh1oNVW6f+7+jrv/EC2+B1Stjm36JPJvB3AB8CRQ85m+UyuR/TsZeMrdvwZw92zax0T2z4HGZmbAtoREUJTaMKvH3acR4q1IRhxXlAiCNsCiUssF0WtVXSdTVTX23xLOUrJBpftmZm2AocDYFMZVWxL5t+sEbGdmU81sppmdnrLoai6R/bsL2ANYDMwBfu/um1ITXtJlxHFF8xEEFuO1svfVJrJOpko4djM7mJAI9k9qRLUnkX27Axjl7hvDSWVWSWT/6gJ9gUOARsC7Zvaeu3+W7OBqQSL7dwQwC/gFsBvwipm95e4/Jzu4FMiI44oSQVAA7FxquS3h7KOq62SqhGI3sx7AeOBId1+RothqKpF96wdMjJJAC+AoMyty92dSE2KNJPq3+Z27rwZWm9k0oCeQDYkgkf07E7jJQ6P6QjP7AugCvJ+aEJMqI44rahoKPgB2N7MOZlYfGA48V2ad54DTo17+fYCf3H1JqgOtpkr3z8zaAU8Bp2XJmWSxSvfN3Tu4e3t3bw88AZyXJUkAEvvbfBY4wMzqmtnWwN7AghTHWV2J7N/XhKsdzGxHoDPweUqjTJ6MOK7oigBw9yIzGwlMIdzFcL+7zzOzc6L3xxLuNjkKWAgUEs5SskKC+3cl0By4OzpzLvIMqYwYT4L7lrUS2T93X2BmLwGzgU3AeHePebtipknw3+864AEzm0NoShnl7llRntrMJgAHAS3MrAC4CqgHmXVcUYkJEZE8p6YhEZE8p0QgIpLnlAhERPKcEoGISJ5TIhARyXNKBJKRoiqhs0o92sdZd1UtfN8DZvZF9F0fmtmAanzGeDPrGj2/rMx779Q0xuhzin8vc6OKnM0qWb+XmR1VG98tuUu3j0pGMrNV7r5tba8b5zMeAF5w9yfM7HDgNnfvUYPPq3FMlX2umT0IfObuf46z/hlAP3cfWduxSO7QFYFkBTPb1sxei87W55hZuQqjZtbKzKaVOmM+IHr9cDN7N9r2cTOr7AA9DegYbXtJ9Flzzeyi6LVtzOzFqD7+XDM7MXp9qpn1M7ObgEZRHI9G762Kfv6r9Bl6dCVynJnVMbNbzewDC3Xpz07g1/IuUYEyM+tvYR6Jj6KfnaORutcCJ0axnBjFfn/0PR/F+j1KHkpH7Ws99KjsAWwkFBqbBTxNGAXfJHqvBWEkZvEV7aro5/8DLo+e1wEaR+tOA7aJXh8FXBnj+x4gmqcAOAGYTijkNgfYhlD+eB7QGzgOuLfUtk2jn1MJZ9+bYyq1TnGMQ4EHo+f1CZUnGwEjgD9FrzcAZgAdYsS5qtT+PQ4MipabAHWj54cCT0bPzwDuKrX9DcCp0fNmhHpE26T731uP9D5UYkIy1Rp371W8YGb1gBvMbCChjEIbYEdgaaltPgDuj9Z9xt1nmdmBQFfg7ah0Rn3CmXQst5rZn4DlhAqshwBPeyjmhpk9BRwAvATcZmY3E5qT3qrCfk0G7jSzBsAgYJq7r4mao3pYyexpTYHdgS/KbN/IzGYB7YGZwCul1n/QzHYnVK+sV8H3Hw4ca2aXRssNgXZkT20iSQIlAskWpxBmp+rr7hvM7EvCQWwzd58WJYqjgYfN7FbgB+AVdz8pge/4g7s/UbxgZofGWsndPzOzvoQaMTea2cvufm0iO+Hua81sKqG08onAhOKvAy5w9ymVfMQad+9lZk2BF4DzgTsJ9XjecPehUcf61Aq2N+A4d/80kXglP6iPQLJFU2BZlAQOBnYpu4KZ7RKtcy9wH2GKwPeA/cysuM1/azPrlOB3TgOGRNtsQ2jWecvMWgOF7v4IcFv0PWVtiK5MYplIKC52AKHYGtHPc4u3MbNO0XfG5O4/ARcCl0bbNAW+id4+o9SqKwlNZMWmABdYdHlkZr0r+g7JH0oEki0eBfqZ2QzC1cEnMdY5CJhlZh8R2vH/5u7LCQfGCWY2m5AYuiTyhe7+IaHv4H1Cn8F4d/8I6A68HzXRXA5cH2PzccDs4s7iMl4mzGX7qofpGSHMAzEf+NDCROf/oJIr9iiWjwmlm28hXJ28Teg/KPYG0LW4s5hw5VAvim1utCx5TrePiojkOV0RiIjkOSUCEZE8p0QgIpLnlAhERPKcEoGISJ5TIhARyXNKBCIiee7/A0AS6FwmJ89wAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Calculate accuracy\n",
    "accuracy2 = sum(preds2 == labels) / len(labels)\n",
    "print(f'Accuracy: {accuracy2}')\n",
    "\n",
    "# Calculate precision\n",
    "precision2 = sum(preds2[preds2 == 1] == labels[preds2 == 1]) / len(preds2[preds2 == 1])\n",
    "print(f'Precision: {precision2}')\n",
    "\n",
    "# Calculate recall\n",
    "recall2 = sum(preds2[preds2 == 1] == labels[preds2 == 1]) / len(labels[labels == 1])\n",
    "print(f'Recall: {recall2}')\n",
    "\n",
    "# Calculate F1 score\n",
    "f1_score2 = 2 * (precision2 * recall2) / (precision2 + recall2)\n",
    "print(f'F1 Score: {f1_score2}')\n",
    "\n",
    "auc2 = round(roc_auc_score(labels, preds2), 4)\n",
    "print('AUC is ' + repr(auc2))\n",
    "\n",
    "fpr2, tpr2, _ = metrics.roc_curve(labels, preds2)\n",
    "\n",
    "plt.title('ROC Curve')\n",
    "plt.plot(fpr2, tpr2, 'b',\n",
    "label='AUC = %0.2f'% auc2)\n",
    "plt.legend(loc='lower right')\n",
    "plt.plot([0,1],[0,1],'r--')\n",
    "plt.xlim([-0.1,1.1])\n",
    "plt.ylim([-0.1,1.1])\n",
    "plt.ylabel('True Positive Rate')\n",
    "plt.xlabel('False Positive Rate')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### We see that Variant2 is performing better for most of our defined metrics, so this is the one we’re likely to choose to dial up in production."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Shifting more traffic to your chosen model in production"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can continue to use TargetVariant to continue invoking a chosen variant. A simpler approach is to update the weights assigned to each variant using UpdateEndpointWeightsAndCapacities. This changes the traffic distribution to your production variants without requiring updates to your endpoint.\n",
    "\n",
    "Recall our variant weights are as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Variant1': 1.0, 'Variant2': 1.0}"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "{\n",
    "    variant[\"VariantName\"]: variant[\"CurrentWeight\"]\n",
    "    for variant in sm.describe_endpoint(EndpointName=endpoint_name)[\"ProductionVariants\"]\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### We'll first write a method to easily invoke our endpoint (a copy of what we had been previously doing)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "def invoke_endpoint_for_two_minutes():\n",
    "    with open('./data/abtest/test-dataset-input-cols.csv', 'r') as f:\n",
    "        for row in f:\n",
    "            print(\".\", end=\"\", flush=True)\n",
    "            payload = row.rstrip('\\n')\n",
    "            response = sm_runtime.invoke_endpoint(EndpointName=endpoint_name,\n",
    "                                                  ContentType='text/csv', \n",
    "                                                  Body=payload)\n",
    "            response['Body'].read()\n",
    "            time.sleep(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ".............................................................................................................................................................................................................................................................................................................................................."
     ]
    }
   ],
   "source": [
    "# We invoke our endpoint for a bit, to show the even split in invocations:\n",
    "invocation_start_time = datetime.now()\n",
    "invoke_endpoint_for_two_minutes()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "time.sleep(20) # give metrics time to catch up\n",
    "plot_endpoint_metrics(invocation_start_time)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Now let us shift 75% of the traffic to Variant2 by assigning new weights to each variant using UpdateEndpointWeightsAndCapacities. Amazon SageMaker will now send 75% of the inference requests to Variant2 and remaining 25% of requests to Variant1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'EndpointArn': 'arn:aws:sagemaker:us-west-2:570447867175:endpoint/demo-xgb-churn-pred-2020-07-29-15-19-23',\n",
       " 'ResponseMetadata': {'RequestId': '078512cd-f3f7-4058-9f9a-871c6f7fd84a',\n",
       "  'HTTPStatusCode': 200,\n",
       "  'HTTPHeaders': {'x-amzn-requestid': '078512cd-f3f7-4058-9f9a-871c6f7fd84a',\n",
       "   'content-type': 'application/x-amz-json-1.1',\n",
       "   'content-length': '107',\n",
       "   'date': 'Wed, 29 Jul 2020 23:05:49 GMT'},\n",
       "  'RetryAttempts': 0}}"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sm.update_endpoint_weights_and_capacities(\n",
    "    EndpointName=endpoint_name,\n",
    "    DesiredWeightsAndCapacities=[\n",
    "        {\n",
    "            \"DesiredWeight\": 25,\n",
    "            \"VariantName\": variant1[\"VariantName\"]\n",
    "        },\n",
    "        {\n",
    "            \"DesiredWeight\": 75,\n",
    "            \"VariantName\": variant2[\"VariantName\"]\n",
    "        }\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Waiting for update to complete\n",
      "....................................................Done\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'Variant1': 25.0, 'Variant2': 75.0}"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(\"Waiting for update to complete\")\n",
    "while True:\n",
    "    status = sm.describe_endpoint(EndpointName=endpoint_name)[\"EndpointStatus\"]\n",
    "    if status in [\"InService\", \"Failed\"]:\n",
    "        print(\"Done\")\n",
    "        break\n",
    "    print(\".\", end=\"\", flush=True)\n",
    "    time.sleep(1)\n",
    "\n",
    "{\n",
    "    variant[\"VariantName\"]: variant[\"CurrentWeight\"]\n",
    "    for variant in sm.describe_endpoint(EndpointName=endpoint_name)[\"ProductionVariants\"]\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Now let's check how that has impacted invocation metrics:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ".............................................................................................................................................................................................................................................................................................................................................."
     ]
    }
   ],
   "source": [
    "invoke_endpoint_for_two_minutes()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "time.sleep(20) # give metrics time to catch up\n",
    "plot_endpoint_metrics(invocation_start_time)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### We can continue to monitor our metrics and when we're satisfied with a variant's performance, we can route 100% of the traffic over the variant. We used UpdateEndpointWeightsAndCapacities to update the traffic assignments for the variants. The weight for Variant1 is set to 0 and the weight for Variant2 is set to 1. Therefore, Amazon SageMaker will send 100% of all inference requests to Variant2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Waiting for update to complete\n",
      "........................................................Done\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'Variant1': 0.0, 'Variant2': 1.0}"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sm.update_endpoint_weights_and_capacities(\n",
    "    EndpointName=endpoint_name,\n",
    "    DesiredWeightsAndCapacities=[\n",
    "        {\n",
    "            \"DesiredWeight\": 0,\n",
    "            \"VariantName\": variant1[\"VariantName\"]\n",
    "        },\n",
    "        {\n",
    "            \"DesiredWeight\": 1,\n",
    "            \"VariantName\": variant2[\"VariantName\"]\n",
    "        }\n",
    "    ]\n",
    ")\n",
    "print(\"Waiting for update to complete\")\n",
    "while True:\n",
    "    status = sm.describe_endpoint(EndpointName=endpoint_name)[\"EndpointStatus\"]\n",
    "    if status in [\"InService\", \"Failed\"]:\n",
    "        print(\"Done\")\n",
    "        break\n",
    "    print(\".\", end=\"\", flush=True)\n",
    "    time.sleep(1)\n",
    "\n",
    "{\n",
    "    variant[\"VariantName\"]: variant[\"CurrentWeight\"]\n",
    "    for variant in sm.describe_endpoint(EndpointName=endpoint_name)[\"ProductionVariants\"]\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "invoke_endpoint_for_two_minutes()\n",
    "time.sleep(20) # give metrics time to catch up\n",
    "plot_endpoint_metrics(invocation_start_time)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Amazon CloudWatch metrics for the total invocations for each variant below shows us that all inference requests are being processed by Variant2 and there are no inference requests processed by Variant1.\n",
    "\n",
    "You can now safely update your endpoint and delete Variant1 from your endpoint. You can also continue testing new models in production by adding new variants to your endpoint and following steps 2 - 4."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6. Close AWS Instance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "sm_session.delete_endpoint(endpoint_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "Amazon SageMaker enables us to easily A/B test ML models in production by running multiple production variants on an endpoint. You can use SageMaker’s capabilities to test models that have been trained using different training datasets, hyperparameters, algorithms, or ML frameworks; test how they perform on different instance types; or a combination of all of the above. \n",
    "\n",
    "You can provide the traffic distribution between the variants on an endpoint and Amazon SageMaker splits the inference traffic to the variants based on the specified distribution. Alternately, if you want to test models for specific customer segments, you can specify the variant that should process an inference request by providing the TargetVariant header, and Amazon SageMaker will route the request to the variant that you specified."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Referenence:\n",
    "https://aws.amazon.com/ko/blogs/machine-learning/a-b-testing-ml-models-in-production-using-amazon-sagemaker/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}