Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Delete outdated k8s configuration and documentation #1356

Merged
merged 6 commits into from
Oct 19, 2023
Merged

Delete outdated k8s configuration and documentation #1356

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
6c7ed85
delete k8s files as moved to OSDFIR infra
hacktobeer Oct 5, 2023
3622147
rewrite/remove doc old gke references
hacktobeer Oct 5, 2023
faee2ec
fix references
hacktobeer Oct 5, 2023
615b6c2
last refs fixes
hacktobeer Oct 5, 2023
b23681e
install ref
hacktobeer Oct 5, 2023
47164e8
Update docs/user/install.md
hacktobeer Oct 17, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
60 changes: 5 additions & 55 deletions docs/developer/develop-vscode.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

This procedure will get you up and running with a Visual Studio Code environment for Turbinia development. The provided configuration files will create a development container containing all dependencies, pylint and yapf correctly setup and launch configurations for both client, server and workers. With this setup it is possible to initiate full Turbinia debug sessions including breakpoints, watches and stepping.

You can set Visual Studio Code up to run a local stack (using redis and celery) or use a hybrid GCP stack (using pubsub, datastore and cloud functions). We advice you to run a local stack if you don't need to debug or develop Turbinia GCP functionality.
You can set Visual Studio Code up to run a local stack (using redis and celery).

## Before you start

Expand Down Expand Up @@ -39,8 +39,6 @@ _Note_: If vscode does not ask you to reopen in a container you need to verify y

_Note_: The instructions contain shell commands to execute, please execute those commands in the vscode terminal (which runs in the development container) and not in a terminal on your host!

Continue with Step 4 for a local Turbinia setup or Step 5 for a GCP hybrid setup.

#### Step 4 - Local Turbinia setup

The local turbinia setup will use redis and celery. Let's create the configuration file for this setup.
Expand All @@ -51,53 +49,11 @@ _Note_: This command needs to be executed in the vscode terminal!
$ sed -f ./docker/vscode/redis-config.sed ./turbinia/config/turbinia_config_tmpl.py > ~/.turbiniarc
```

Let's verify the installation in Step 6.

#### Step 5 - GCP hybrid Turbinia setup

Follow the ‘GCP Setup’ section [here](../user/install-gcp-pubsub.md) and setup Cloud Functions, a GCE bucket, Datastore and PubSub.

- Create a pubsub topic, eg ‘turbinia-dev’
- Create a GCE storage bucket with a unique name

Create the Turbinia hybrid configuration file.

_Note_: This command needs to be executed in the vscode terminal!

```
$ sed -f ./docker/vscode/psq-config.sed ./turbinia/config/turbinia_config_tmpl.py > ~/.turbiniarc
```

Edit the configuration file `~/.turbiniarc` and set below variables according to the GCP project you are using. Make sure all values are between quotes!

```
TURBINIA_PROJECT = '[your_gcp_project_name]'
TURBINIA_REGION = '[your_preferred_region]' eg 'us-central1'
TURBINIA_ZONE = '[your_preferred_zone]' eg 'us-central1-f'
PUBSUB_TOPIC = '[your_gcp_pubsub_topic_name]' eg 'turbinia-dev'
BUCKET_NAME = '[your_gcp_bucket_name]'
```

Setup authentication for the GCP project.
Let's verify the installation in Step 5.

_Note_: These commands need to be executed in the vscode terminal!

```
$ gcloud auth login
$ gcloud auth application-default login
```
#### Step 5 - Turbinia installation verification

Deploy the Google Cloud Functions

_Note_: This command needs to be executed in the vscode terminal!

```
$ PYTHONPATH=. python3 tools/gcf_init/deploy_gcf.py
```

#### Step 6 - Turbinia installation verification

Let's verify that the GCP hybrid setup is working before we start developing and debugging. We are going to start a server and worker in separate vscode terminals and create a Turbinia request in a third. Open up 3 vscode terminals and execute below commands.
Let's verify that the local setup is working before we start developing and debugging. We are going to start a server and worker in separate vscode terminals and create a Turbinia request in a third. Open up 3 vscode terminals and execute below commands.

_Note_: These commands need to be executed in the vscode terminal!

Expand All @@ -114,12 +70,6 @@ For a local setup
$ python3 turbinia/turbiniactl.py celeryworker
```

For a GCP hybrid setup

```
$ python3 turbinia/turbiniactl.py psqworker
```

Terminal 3 - Fetch and process some evidence

```
Expand All @@ -129,7 +79,7 @@ $ python3 turbinia/turbiniactl.py compresseddirectory -l /evidence/history.tgz
$ python3 turbinia/turbiniactl.py -a status -r [request_id]
```

This should process the evidence and show output in each terminal for server and worker. Results will be stored in `/evidence` and in the GCS bucket.
This should process the evidence and show output in each terminal for server and worker. Results will be stored in `/evidence`.

#### Step 8 - Debugging example

Expand Down
10 changes: 5 additions & 5 deletions docs/user/api-server.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,17 +4,17 @@
Turbinia's API server provides a RESTful interface to Turbinia's functionality. It allows users to create and manage logical jobs, which are used to schedule forensic processing tasks. The API server also provides a way for users to monitor the status of their jobs and view the results of their processing tasks.

## Getting started
The following sections describe how to get the Turbinia API server up and running. Please note that The API server is only compatible with Turbinia deployments that use Redis as a datastore and Celery workers. If your deployment uses GCP PubSub and/or GCP PSQ workers you will not be able to use the API server. GCP PubSub/PSQ dependencies will be deprecated in the near future so it is recommended to redeploy Turbinia and use Redis and Celery.
The following sections describe how to get the Turbinia API server up and running. Please note that The API server is only compatible with Turbinia deployments that use Redis as a datastore and Celery workers. If your deployment uses the old GCP PubSub and/or GCP PSQ workers you will not be able to use the API server. It is recommended to redeploy Turbinia and use Redis and Celery.

### Installation
To use the Turbinia API server you will need to deploy Turbinia in your environment with a configuration that uses Redis and Celery instead of GCP PubSub and PSQ.
To use the Turbinia API server you will need to deploy Turbinia in your environment with a configuration that uses Redis and Celery.

Please follow the instructions for deploying a [Turbinia GKE Celery cluster](install-gke-celery.md) or [local stack using Celery](turbinia-local-stack.md)
Please follow the instructions for deploying a [Turbinia GKE Celery cluster](https://github.com/google/osdfir-infrastructure/tree/main/charts/turbinia) or [local stack using Celery](turbinia-local-stack.md)

Note that the Turbinia API server requires access to the Turbinia output directory (```OUTPUT_DIR```)

### Configuration and UI
If you plan on making the Turbinia API Server and Web UI externally accessible (e.g. internet access), follow the instructions for [external access and authentication](install-gke-external.md)
If you plan on making the Turbinia API Server and Web UI externally accessible (e.g. internet access), follow the instructions for [external access and authentication](https://github.com/google/osdfir-infrastructure/tree/main/charts/turbinia)

### Usage
You may access the API server at ```http://<API_SERVER_ADDRESS>:<API_SERVER_PORT>```, or via https if you deployed Turbinia for external access using a domain and HTTPS certificate.
Expand All @@ -24,6 +24,6 @@ Because the Turbinia API Server is built using the FastAPI framework, it provide
We also provide a [command-line tool](https://github.com/google/turbinia/tree/master/turbinia/api/cli) and a [Python library](https://github.com/google/turbinia/tree/master/turbinia/api/client) to interact with the API server.

### Authentication
Turbinia API Server uses OAuth2-proxy to provide OpenID Connect and OAuth2 authentication support. If you deployed Turbinia using GCP and GKE cluster instructions, follow the guide for [external access and authentication](install-gke-external.md) to complete the authentication configuration.
Turbinia API Server uses OAuth2-proxy to provide OpenID Connect and OAuth2 authentication support. If you deployed Turbinia using GCP and GKE cluster instructions, follow the guide for [external access and authentication](https://github.com/google/osdfir-infrastructure/tree/main/charts/turbinia) to complete the authentication configuration.

For Turbinia deployments using the [local stack](turbinia-local-stack.md), or a non-Google identity provider, make sure to edit the ```oauth2_proxy.cfg``` configuration file in ```docker/oauth2_proxy``` with the appropriate identity provider information such as ```client_id``` and ```client_secret``` prior to deploying the Docker containers in the local stack. If your deployment will use an identity provider other than Google, you will also need to change the ```provider``` and related settings. For more information and how to configure OAuth2-proxy for different providers, refer to the [OAuth2-Proxy Documentation](https://oauth2-proxy.github.io/oauth2-proxy/docs/configuration/oauth_provider).
Loading