diff --git a/docs/developer/develop-local.md b/docs/developer/develop-local.md index 62ddc2444..a56e74ed5 100644 --- a/docs/developer/develop-local.md +++ b/docs/developer/develop-local.md @@ -1,6 +1,6 @@ # Developing on a local Turbinia setup (no cloud required) -See [here](../user/turbinia-local-stack.md) on how to setup the local Turbinia stack with Docker. +See [here](../user/install.md) on how to setup the local Turbinia stack with Docker. After you have the local stack up and running a usual development cycle would look like below. diff --git a/docs/user/api-server.md b/docs/user/api-server.md index eae37a5c8..f00550ec0 100644 --- a/docs/user/api-server.md +++ b/docs/user/api-server.md @@ -9,7 +9,7 @@ The following sections describe how to get the Turbinia API server up and runnin ### Installation To use the Turbinia API server you will need to deploy Turbinia in your environment with a configuration that uses Redis and Celery. -Please follow the instructions for deploying a [Turbinia GKE Celery cluster](https://github.com/google/osdfir-infrastructure/tree/main/charts/turbinia) or [local stack using Celery](turbinia-local-stack.md) +Please follow the [instructions](install.md) for deploying Turbinia to Kubernetes or Docker. Note that the Turbinia API server requires access to the Turbinia output directory (```OUTPUT_DIR```) @@ -26,4 +26,4 @@ We also provide a [command-line tool](https://github.com/google/turbinia/tree/ma ### Authentication Turbinia API Server uses OAuth2-proxy to provide OpenID Connect and OAuth2 authentication support. If you deployed Turbinia using GCP and GKE cluster instructions, follow the guide for [external access and authentication](https://github.com/google/osdfir-infrastructure/tree/main/charts/turbinia) to complete the authentication configuration. -For Turbinia deployments using the [local stack](turbinia-local-stack.md), or a non-Google identity provider, make sure to edit the ```oauth2_proxy.cfg``` configuration file in ```docker/oauth2_proxy``` with the appropriate identity provider information such as ```client_id``` and ```client_secret``` prior to deploying the Docker containers in the local stack. If your deployment will use an identity provider other than Google, you will also need to change the ```provider``` and related settings. For more information and how to configure OAuth2-proxy for different providers, refer to the [OAuth2-Proxy Documentation](https://oauth2-proxy.github.io/oauth2-proxy/docs/configuration/oauth_provider). \ No newline at end of file +For Turbinia deployments using the [Docker Installation method](install.md), or a non-Google identity provider, make sure to edit the ```oauth2_proxy.cfg``` configuration file in ```docker/oauth2_proxy``` with the appropriate identity provider information such as ```client_id``` and ```client_secret``` prior to deploying the Docker containers in the local stack. If your deployment will use an identity provider other than Google, you will also need to change the ```provider``` and related settings. For more information and how to configure OAuth2-proxy for different providers, refer to the [OAuth2-Proxy Documentation](https://oauth2-proxy.github.io/oauth2-proxy/docs/configuration/oauth_provider). \ No newline at end of file diff --git a/docs/user/index.rst b/docs/user/index.rst index 7ba3a715d..52324d675 100644 --- a/docs/user/index.rst +++ b/docs/user/index.rst @@ -6,7 +6,6 @@ User documentation :maxdepth: 3 install - turbinia-local-stack how-it-works api-server turbinia-web-ui diff --git a/docs/user/install.md b/docs/user/install.md index 631112c00..34f46b707 100644 --- a/docs/user/install.md +++ b/docs/user/install.md @@ -1,167 +1,98 @@ -**Note**: **_This installation method will be deprecated by the end of 2022. -The current recommended method for installing Turbinia is -[here](https://github.com/google/osdfir-infrastructure)._** - -# **Turbinia Quick Installation Instructions** +# Install Turbinia ## Overview -Turbinia can be run on the [Google Cloud Platform](https://cloud.google.com), on -local machines, or in a hybrid mode. See the -"[how it works](how-it-works.md)" -documentation for more details on what the architecture looks like for each of -these installation types. This doc covers the recommended quick installation -instructions for Cloud installations. This uses -[terraform configs](https://github.com/forseti-security/osdfir-infrastructure) -that are part of the -[Forseti Security repository](https://github.com/forseti-security) -to automate deployment of Turbinia into an existing GCP Project. -for details. - -## Installation - -The following steps can be performed on any Linux machine (Ubuntu 18.0.4 -recommended), and [Cloud Shell](https://cloud.google.com/shell/) is one easy way -to get a shell with access to your GCP resources. - -### GCP Project Setup - -- Create or select a Google Cloud Platform project in the - [Google Cloud Console](https://console.cloud.google.com). -- Determine which GCP zone and region that you wish to deploy Turbinia into. - Note that one of the GCP dependencies is Cloud Functions, and that only - works in certain regions, so you will need to deploy in one of - [the supported regions](https://cloud.google.com/functions/docs/locations). -- Install - [google-cloud-sdk](https://cloud.google.com/sdk/docs/quickstart-linux). - - Note: If you are doing this from cloud shell you shouldn't need this - step. -- Run `gcloud auth login` to authenticate. This may require you to copy/paste - url to browser. -- Run `gcloud auth application-default login` - -### Deploy Turbinia - -- Download the - [Terraform CLI from here](https://www.terraform.io/downloads.html). -- Clone the Forseti Security repository and change to the path containing the - configs - - `git clone https://github.com/forseti-security/osdfir-infrastructure/` - - `cd osdfir-infrastructure` -- Configuration - - - By default this will create one Turbinia server instance and one worker - instance. If you want to change the number of workers, edit the - `modules/turbinia/variables.tf` file and set the `turbinia_worker_count` - variable to the number of workers you want to deploy. - - To adjust the GCP zone and region you want to run Turbinia in, edit the - `modules/turbinia/variables.tf` file and change the `gcp_zone` and - `gcp_region` variables as appropriate to reflect your GCP project's - zone and region. - - If you want to use docker to run Turbinia tasks, please follow the - instructions [here](using-docker.md) to enable docker. - - Running the following commands will leave some state information under - the current directory, so if you wish to continue to manage the number - of workers via Terraform you should keep this directory for later use. - Alternatively, if you wish to store this information in GCS instead, you - can edit `main.tf` and change the `bucket` parameter to the GCS bucket - you wish to keep this state information in. See the - [Terraform documentation](https://developer.hashicorp.com/terraform/cli/commands) - for more information. - - The current configuration does not enable alert notifications by default. - Please see [here](#grafana-smtp-setup) for the instructions - - If you are running multiple workers on a given host and within containers, ensure - that you are mapping the host `OUTPUT_DIR` path specified in the configuration file - `.turbiniarc` to the containers so that they can properly update the `RESOURCE_STATE_FILE`. - -- Initialize terraform and apply the configuration - - `./deploy.sh --no-timesketch` - - If the `--no-timesketch` parameter is not supplied, Terraform will also - create a [Timesketch](http://timesketch.org) instance in the same - project, and this can be configured to ingest Turbinia timeline - output and report data. See the - [Documentation on this](https://github.com/forseti-security/osdfir-infrastructure) - for more details. - - When prompted for the project name, enter the project you selected - during setup. - -This should result in the appropriate cloud services being enabled and -configured and GCE instances for the server and the worker(s) being started and -configured. The Turbinia configuration file will be deployed on these instances -as `etc/turbinia/turbinia.conf`. If you later want to increase the number of -workers, you can edit the `turbinia/variables.tf` file mentioned above and -re-run `terraform apply` -To use Turbinia you can use the virtual environment that was setup by -the `deploy.sh` script.To activate the virtual environment, run the following -command `source ~/turbinia/bin/activate` and then use `turbiniactl`. For more -information on how to use Turbinia please visit [the user manual](https://github.com/google/turbinia). - -### Client configuration (optional) - -If you want to use the command line tool, you can SSH into the server and run -`turbiniactl` from there. The `turbiniactl` command can be used to submit -Evidence for processing or see the status of existing and previous processing -requests. If you'd prefer to use turbiniactl on a different machine, follow the -following instructions to configure the client. The instructions are based on -using Ubuntu 18.04, though other versions of Linux should be compatible. - -- Follow the steps from GCP Project setup above to install the SDK and - authenticate with gcloud. -- Install some python tooling: - - `apt-get install python3-pip python3-wheel` -- Install the Turbinia client. - - Note: You may want to install this into a virtual-environment with - [venv](https://docs.python.org/3.7/library/venv.html) or - [pipenv](https://pipenv.pypa.io/en/latest/)) to reduce potential - dependency conflicts and isolate these packages into their own - environment. - - `pip3 --user install turbinia` -- If running on the same machine you deployed Turbinia from, you can generate - the config with terraform - - `terraform output turbinia-config > ~/.turbiniarc` -- Otherwise, if you are running from a different machine you'll need to copy - the Turbinia config from the original machine, or from the Turbinia server - from `/etc/turbinia/turbinia.conf`. - -### Grafana SMTP Setup - -If you want to receive alert notifications from Grafana, you'll need to setup a SMTP server for Grafana. To configure a SMTP server, you need to add the following environment variables to `Grafana` `env` section in `osdfir-infrastructure/modules/monitoring/main.tf` +Turbinia can be deployed on either Google Cloud Platform or local machines using +two primary installation methods: Kubernetes or Docker, which will be covered in this guide. + +Once Turbinia is up and running using either Kubernetes or Docker, install and +configure the `turbinia-client` CLI using the provided [documentation](https://github.com/google/turbinia/tree/master/turbinia/api/cli) to kick off your first processing request. + +## K8s Installation + +To get started quickly, ensure you have [Helm](https://helm.sh/docs/intro/install/) +and [Kubectl](https://kubernetes.io/docs/tasks/tools/) installed and are authenticated +to your Kubernetes cluster. +> **Note**: To simplify the process of initializing a Google Kubernetes Engine Cluster (GKE), +use the [init-gke.sh](https://github.com/google/osdfir-infrastructure/blob/main/tools/init-gke.sh) +script. Alternatively, for local installations, consider using [KIND](https://kind.sigs.k8s.io/docs/user/quick-start/) or [Minikube](https://minikube.sigs.k8s.io/docs/start/). +Once complete, add the repo containing the Helm charts as follows: + +```console +helm repo add osdfir-charts https://google.github.io/osdfir-infrastructure ``` - { - name = "GF_SMTP_ENABLED" - value = "true" - }, { - name = "GF_SMTP_HOST" - value = "smtp.gmail.com:465" # Replace this if you're not using gmail - }, { - name = "GF_SMTP_USER" - value = "" - }, { - name = "GF_SMTP_PASSWORD" - value = "" - }, { - name = "GF_SMTP_SKIP_VERIFY" - value = "true" - }, { - name = "GF_SMTP_FROM_ADDRESS" - value = "" - } +If you had already added this repo earlier, run `helm repo update` to retrieve +the latest versions of the packages. You can then run `helm search repo osdfir-charts` +to see the available charts. + +To install the Turbinia chart using a release name of my-release: + +```console +helm install my-release osdfir-charts/turbinia ``` ---- +To uninstall the chart: + +```console +helm uninstall my-release +``` + +For instructions on installing Turbinia along with our other integrated DFIR tools, +refer to the main [OSDFIR Infrastructure](https://github.com/google/osdfir-infrastructure) +repository. Additionally, refer to the Turbinia Helm chart +[README](https://github.com/google/osdfir-infrastructure/tree/main/charts/turbinia) +for a comprehensive list of configuration options. + +## Docker Installation + +The second way to run Turbinia is through the provided Docker containers. + +### Caveats -> **NOTE** +rawdisk: As Turbinia uses the loop device to mount different types of evidence +(eg raw disks) the host operating system should support the loop device. Linux is +currently the only OS that supports the processing of raw disks. -> By default Gmail does not allow [less secure apps](https://support.google.com/accounts/answer/6010255) to authenticate and send emails. For that reason, you'll need to allow less secure apps to access the provided Gmail account. +googleclouddisk: Turbinia running in Docker cannot currently process Google Cloud +disks. ---- +DOCKER_ENABLED: If you plan to enable running dependencies in containers make +sure you have Docker installed. -Once completed: +### Steps + +#### Step 1 + +Checkout the [Turbinia source code](https://github.com/google/turbinia). If you +intend to start developing please [fork](https://docs.github.com/en/github/getting-started-with-github/fork-a-repo) +the repository on github first and check out your own forked instance. + +```console +git clone https://github.com/google/turbinia.git +cd turbinia +``` + +#### Step 2 + +Generate configuration file using sed with default local stack values to the ```./conf``` folder. +This folder (and supporting folders) will be mapped by docker compose into the containers. + +```console +mkdir -p ./conf && mkdir -p ./tmp && mkdir -p ./evidence && mkdir -p ./certs && chmod 777 ./conf ./tmp ./evidence ./certs +sed -f docker/local/local-config.sed turbinia/config/turbinia_config_tmpl.py > conf/turbinia.conf +``` + +#### Step 3 + +Let's bring up the local Turbinia stack + +```console +docker-compose -f ./docker/local/docker-compose.yml up +``` -- login to the Grafana Dashboard. -- Select Alerting and choose "Notification channels". -- Fill the required fields and add the email addresses that will receive notification. -- Click "Test" to test your SMTP setup. -- Once everything is working, click "Save" to save the notification channel. +A Turbinia server, worker, api and Redis should now be running on your local system +and a local persistent 'evidence' folder will have been created containing the +Turbinia log file and processing output. +> **Note**: Redis will store it's data in a volume that is mapped to ```./redis-data/```. You can adjust this in the docker-compose.yml configuration. diff --git a/docs/user/turbinia-local-stack.md b/docs/user/turbinia-local-stack.md deleted file mode 100644 index 6f90aba5b..000000000 --- a/docs/user/turbinia-local-stack.md +++ /dev/null @@ -1,46 +0,0 @@ -## Turbinia local stack using Docker -Turbinia can be run locally without any Cloud components using Docker. It will use Redis, Celery and local disk to store data and perform message broker functionality. - -### Caveats -rawdisk: As Turbinia uses the loop device to mount different types of evidence (eg raw disks) the host operating system should support the loop device. Linux is currently the only OS that supports the processing of raw disks. - -googleclouddisk: Turbinia as local stack cannot currently process Google Cloud disks. - -DOCKER_ENABLED: If you plan to enable running dependencies in containers make sure you have docker installed. - -### Running - -#### Step 1 -Checkout the [Turbinia source code](https://github.com/google/turbinia). If you intend to start developing please [fork](https://docs.github.com/en/github/getting-started-with-github/fork-a-repo) the repository on github first and check out your own forked instance. -``` -$ git clone https://github.com/google/turbinia.git -$ cd turbinia -``` -#### Step 2 -Generate configuration file using sed with default local stack values to the ```./conf``` folder. This folder (and supporting folders) will be mapped by docker compose into the containers. -``` -$ mkdir -p ./conf && mkdir -p ./tmp && mkdir -p ./evidence && mkdir -p ./certs && chmod 777 ./conf ./tmp ./evidence ./certs -$ sed -f docker/local/local-config.sed turbinia/config/turbinia_config_tmpl.py > conf/turbinia.conf -``` -#### Step 3 -Let's bring up the local Turbinia stack -``` -$ docker-compose -f ./docker/local/docker-compose.yml up -``` -Redis, a Turbinia server and worker should now be running on your local system and a local persistent 'evidence' folder will have been created containing the Turbinia log file and processing output. -Note: Redis will store it's data in a volume that is mapped to ```./redis-data/```. You can adjust this in the docker-compose.yml configuration. -#### Step 4 -Let's process evidence to test your setup, eg a Chrome Browser history file. -``` -$ curl https://raw.githubusercontent.com/obsidianforensics/hindsight/master/tests/fixtures/profiles/60/History > History -$ tar -vzcf ./evidence/history.tgz History -``` -This command runs the turbinia client, turbiniactl, within the turbinia server docker container and generates a processing request. -``` -$ docker exec -ti turbinia-server turbiniactl compresseddirectory -l /evidence/history.tgz -``` -This will create a task in Turbinia to process the evidence file. A request ID will be returned and we can query the status with below command. -``` -$ docker exec -ti turbinia-server turbiniactl -a status -r {REQUEST_ID} -``` -There will be server and worker output displayed both on the docker-compose terminal as well as in the ```./evidence``` folder.