04_Service_Canary_Rollout.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "4392476c-6bcc-4caa-85b8-16e87bedf99d",
   "metadata": {},
   "source": [
    "# Anyscale Services + Canary Rollout Features\n",
    "\n",
    "Anyscale Services is the part of the Anyscale platform which provides web endpoints to Ray Serve applications. Anyscale Services provides key production features including\n",
    "* High availability (HA)\n",
    "* Canary rollouts for new service versions\n",
    "* Extensive monitoring/management\n",
    "* Support for the entire Ray platform, FastAPI, and applications which go beyond Ray"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e544086-94ac-4ff9-8cbe-98db34d207df",
   "metadata": {},
   "source": [
    "## Setup\n",
    "\n",
    "The service versions are implemented in Python using standard Ray Serve APIs\n",
    "* `1-hello-chat.py` - skeleton for a chat service, it generates a response in a trivial static manner\n",
    "* `2-llm-chat.py` - our real LLM chat service\n",
    "\n",
    "Each service version has a corresponding YAML file used to deploy that version -- in this case, `1-service.yaml` and `2-service.yaml`.\n",
    "\n",
    "Extended configuration is possible via these YAML files as well as via CLI parameters, but the present examples are minimal starting points for clarity."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "73c2b0f3-ebb7-42d3-8884-35e1caa872b8",
   "metadata": {},
   "source": [
    "Currently, the service is not deployed\n",
    "\n",
    "## Initial service rollout"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8ca11884-1261-45eb-a0fe-73a478b0a04f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "! anyscale service rollout -f 1-service.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa2db29c-9c5c-4fc9-bd38-b406e0294d0c",
   "metadata": {},
   "source": [
    "In the Anyscale UI (or via logs) we can monitor the initial rollout\n",
    "\n",
    "> We can launch `load_test.py` in a console to generate a steady stream of requests to our service"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e10a357-b244-4786-85a3-1b8511b1c000",
   "metadata": {},
   "source": [
    "Get a web authentication token from the Anyscale UI and place it in token.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "19ad6f29-eeff-451e-a4b8-ca7aa1e01880",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "with open(\"token.txt\", \"r\") as f:\n",
    "    token = f.read()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e23d03b6-f95e-468a-8cf5-0748dd556805",
   "metadata": {},
   "source": [
    "Each Anyscale service has a unique URL -- calls to this URL will be routed automatically during the version changeover\n",
    "\n",
    "> Get your service URL from the Anyscale UI"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3b381127-c27a-499d-bf5b-0782329ba61b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "base_url = \"https://llms-in-prod-gppbq.cld-kvedzwag2qa8i5bj.s.anyscaleuserdata.com/\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e577feaf-a32d-4eaa-8d3e-8f5734c31d8a",
   "metadata": {},
   "source": [
    "We'll set up minimal code to make a request to our service"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c75996c5-bc66-4562-884b-b0cefdcd9e29",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import requests\n",
    "\n",
    "path = \"/\"\n",
    "full_url = f\"{base_url}{path}\"\n",
    "headers = {\"Authorization\": f\"Bearer {token}\"}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e7a324ea-d85b-4724-9665-7e2c9f172cc7",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "sample_json = '{ \"user_input\" : \"hello\" }'\n",
    "\n",
    "requests.post(\n",
    "    base_url, headers={\"Authorization\": \"Bearer \" + token}, json=sample_json\n",
    ").json()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8dc41227-3bd1-4d21-a15b-6b3890dd74e1",
   "metadata": {},
   "source": [
    "## Upgrading the service to real LLM chat\n",
    "\n",
    "To change over to a real LLM-backed chat service, we run another similar CLI command\n",
    "\n",
    "Note that, although this demo uses the same configurations for the initial and \"real\" services, we can upgrade a service to code requiring different hardware and/or different software. The only thing that has to stay the same is the service name.\n",
    "\n",
    "### Demo of canary rollouts\n",
    "\n",
    "We'll demonstrate rolling out multiple service versions while monitoring both externally and via Anyscale\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17e6f35b-6e25-42d3-9332-30a0291911bb",
   "metadata": {},
   "source": [
    "The canary rollout feature allows zero-downtime upgrades as a live service transitions from one implementation to a new one\n",
    "* Additional clusters are automatically provisioned by Anyscale for new service versions\n",
    "    * service versions do *not* need to share config, dependencies, or even hardware requirements\n",
    "    * the only thing that stays the same is the (internal) name and external endpoint\n",
    "* Load is gradually shifted from the old service to the new one by Anyscale load balancers\n",
    "    * rollout (changeover) schedule can be automatic, customized, or manually controlled\n",
    "* Status of old and new versions are visible and accessible simultaneously in the Anyscale UI\n",
    "    * Grafana integration shows realtime statistics on service transition\n",
    "    * __Rollback__ feature is available if it is necessary to abort the transition and return all traffic to original service"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "48724c7b-2362-40f2-9b37-ddeeefc255c4",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "! anyscale service rollout -f 2-service.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e8151c88-424a-4bc8-ae91-d08f2aa8cf04",
   "metadata": {},
   "source": [
    "At this point, we may want to observe the canary rollout service changeover in\n",
    "* the Anyscale service UI\n",
    "* Grafana timeseries chart of all-version traffic"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "79f6f4f7-48cd-45d0-ac36-665d2428be02",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "sample_json = '{ \"user_input\" : \"When did Taylor Swift\\'s Eras Tour Start?\" }'\n",
    "\n",
    "requests.post(\n",
    "    base_url, headers={\"Authorization\": \"Bearer \" + token}, json=sample_json\n",
    ").json()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d2c96d2-7d22-436a-8fa5-ca97ded1abe9",
   "metadata": {},
   "source": [
    "We can further manage the service via Anyscale UI, Python SDK or CLI"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}