Skip to content

Latest commit

 

History

History
81 lines (54 loc) · 5.08 KB

dbt_cloud_setup.md

File metadata and controls

81 lines (54 loc) · 5.08 KB
Raw

How to setup dbt cloud with bigquery

Official documentation

Create a BigQuery service account

In order to connect we need the service account JSON file generated from bigquery:

  1. Open the BigQuery credential wizard to create a service account in your taxi project
  1. You can either grant the specific roles the account will need or simply use bq admin, as you'll be the sole user of both accounts and data.

Note: if you decide to use specific roles instead of BQ Admin, some users reported that they needed to add also viewer role to avoid encountering denied access errors
  1. Now that the service account has been created we need to add and download a JSON key, go to the keys section, select "create new key". Select key type JSON and once you click on create it will get inmediately downloaded for you to use.

Create a dbt cloud project

  1. Create a dbt cloud account from their website (free for solo developers)
  2. Once you have logged in into dbt cloud you will be prompt to create a new project

You are going to need:

  • access to your data warehouse (bigquery - set up in weeks 2 and 3)
  • admin access to your repo, where you will have the dbt project.

Note: For the sake of showing the creation of a project from scratch I've created a new empty repository just for this week project.

image

  1. Name your project
  2. Choose Bigquery as your data warehouse: image
  3. Upload the key you downloaded from BQ on the create from file option. This will fill out most fields related to the production credentials. Scroll down to the end of the page and set up your development credentials.

Note: The dataset you'll see under the development credentials is the one you'll use to run and build your models during development. Since BigQuery's default location may not match the one you sued for your source data, it's recommended to create this schema manually to avoid multiregion errors.
  1. Click on Test and after that you can continue with the setup

Add GitHub repository

Note: This step could be skipped by using a managed repository if you don't have your own GitHub repo for the course.

  1. Select git clone and paste the SSH key from your repo.
  1. You will get a deploy key, head to your GH repo and go to the settings tab. Under security you'll find the menu deploy keys. Click on add key and paste the deploy key provided by dbt cloud. Make sure to tikce on "write access"

Review your project settings

At the end, if you go to your projects it should look some like this: image

[Optional] Link to your github account

You could simplify the process of adding and creating repositories by linking your GH account. Official documentation