Skip to content

Latest commit

 

History

History
107 lines (68 loc) · 3.9 KB

README.md

File metadata and controls

107 lines (68 loc) · 3.9 KB

Monitor

Monitor is an AI-driven observability interface designed to help humans observe, understand, and steer the internal computations of language models. This README explains how to deploy a local version of Monitor; the deployed version is available at https://monitor.transluce.org.

We recommend developing on a GPU node, since the backend server runs language model inference.

Table of Contents

Developing Locally

First install and activate the monitor venv, and cd to the monitor directory:

luce install monitor  # omit if you already have the venv installed
luce activate monitor  # activates venv and cds automatically

We then need to setup three key components: the NeuronDB database, the backend ASGI server, and the frontend web server.

1. NeuronDB

Monitor requires a PostgreSQL database to read neuron descriptions from. We offer Docker images for starting a local Postgres instance. (Don't worry, you don't have to know how Docker works!)

First, if you don't already have Docker installed, see the Docker installation guide.

Once that's done, your options are to:

  1. [RECOMMENDED] Use our prebuilt Postgres + pgvector Docker image that contains pre-loaded neuron descriptions for llama-3.1-8b-instruct. To use this option, run:

    ./scripts/neurondb.sh  # May need to use `sudo` on Linux

    to pull the image and start a Postgres container. Note that this image is large (~36GB), so pulling it may take a few dozen minutes on a typical home WiFi connection, or <10m on a datacenter-grade ethernet connection.

  2. Use a fresh unpopulated image with no pre-loaded neuron descriptions. To use this option, run:

    ./scripts/neurondb.sh --fresh

    This image is fast to pull and start, but you won't see any neuron descriptions, linter outputs, or steering capabilities in the UI.

The database automatically starts when you run the script; you can check that it's running with docker ps.

For the DBManager to connect to the database, you'll also need to set the following environment variables in the root .env (see the env section of the main README if you haven't configured a .env yet).

PG_USER=clarity
PG_PASSWORD=sysadmin
PG_HOST=localhost
PG_PORT=5432
PG_DATABASE=neurons

If you want to learn more about the NeuronDB and/or Postgres setup, see the NeuronDB README.

2. Backend ASGI server

First download the Llama-3.1 8B Instruct model to your local cache (HF_TOKEN must be set in your root .env):

python3 scripts/download_llm.py

Then run the API server (-d for dev mode):

./scripts/api.sh  -d

3. Frontend web server

First make sure Node.js is installed; if not, run:

luce node install

Then, run the web server; this script also installs dependencies in web/ if web/node_modules/ doesn't exist:

./scripts/web.sh

You can then view the web app at http://localhost:3000.

Deploying to Modal

If you don't already have a Modal account, first create one. Then run the following command to authenticate:

modal token new

Next, open monitor/server_modal.py and see if you'd like to change any of the default settings, e.g., the GPU count/type, concurrency limit, or warm pool.

Finally, deploy by running:

modal deploy monitor/server_modal.py