Skip to content

Copy table from one dataset to another in google big query using bash script

License

Notifications You must be signed in to change notification settings

anilkhichar/bq-table-copy-automation

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

7 Commits
 
 
 
 
 
 
 
 
 
 

Repository files navigation

bq-table-copy-automation

Bash script to read a mapping Yaml file & copy table from one dataset to another dataset for Google BigQuery

Working on

  • Linux
  • OSX

Getting Started

Prerequisites

  1. Google Cloud SDK
  2. Make sure that Python 2.7 is installed on your system:

    python -V

  3. Initialize the GCloud SDK.

Usage

  1. Copy the yaml_parser.sh, mapping.yaml & bq_cp_util.sh files locally/on the server where gcloud is installed & authenticated.
wget https://github.com/khichar-anil/bq-table-copy-automation/blob/master/yaml_parser.sh
wget https://github.com/khichar-anil/bq-table-copy-automation/blob/master/mapping.yaml
wget https://github.com/khichar-anil/bq-table-copy-automation/blob/master/bq_cp_util.sh
  1. Modify the mapping.yaml as per your source & target datasets. mapping.yaml

    Yaml Attribute Sample Value Description
    sourceProjectId source-project Source table gcp project id.
    targetProjectId target-project Target table gcp project id. Source & target project Ids can be same, if we want to copy table withing same project.
    sourceDataset source_ds Source table BQ dataset name.
    targetDataset target_ds Target BQ dataset name. Source & target dataset can be same, if we want to copy table withing same dataset.
    tables > source source_tbl_1 Source table name.
    tables > target target_tbl_1 Target table name.
  2. ./bq_cp_util.sh

Credits

About

Copy table from one dataset to another in google big query using bash script

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages