Skip to content

Latest commit

 

History

History
108 lines (92 loc) · 3.19 KB

Perform Foundational Data, ML, and AI Tasks in Google Cloud Challenge Lab.md

File metadata and controls

108 lines (92 loc) · 3.19 KB
Raw

Defining some variables given by Cloud Skill Boosts

export BIGQUERY_DATASET_NAME=

export CLOUD_STORAGE_BUCKET_NAME=

export REGION=

Task 1: Run a simple Dataflow job

  • Set the zone by using command
gcloud config set compute/zone $REGION
  • create Bigquery Dataset using command
bq mk $BIGQUERY_DATASET_NAME

gsutil mb gs://$CLOUD_STORAGE_BUCKET_NAME

Task 2: Run a simple Dataproc job

  • Set Region (if in case you get region us-east1 instead of -a replace -b)
gcloud config set compute/zone ${REGION}-a

gcloud dataproc clusters create sample-cluster --region ${REGION}
  • SSH into Dataproc Cluster
gcloud compute ssh sample-cluster-m --zone=${REGION}-a

hdfs dfs -cp gs://cloud-training/gsp323/data.txt /data.txt
  • exit the SSH terminal
  • Submit the Spark Dataproc Job
gcloud dataproc jobs submit spark --cluster sample-cluster \
  --class org.apache.spark.examples.SparkPageRank \
  --region $REGION \
  --jars file:///usr/lib/spark/examples/jars/spark-examples.jar -- /data.txt

Task 3 Dataprep Job

  • Initialize Dataprep

  • Create flow

  • Import to GCS gs://cloud-training/gsp323/runs.csv

  • Edit a Recipe

  • Add the following Steps image

  • Follow this image given below
    image

Task 4: AI

  • Execute The Following Commands
gcloud iam service-accounts create my-natlang-sa \
  --display-name "my natural language service account"

gcloud iam service-accounts keys create ~/key.json \
  --iam-account my-natlang-sa@$DEVSHELL_PROJECT_ID.iam.gserviceaccount.com

4.1 Google Cloud Speech API

wget https://raw.githubusercontent.com/guys-in-the-cloud/cloud-skill-boosts/main/Challenge-labs/Perform%20Foundational%20Data%2C%20ML%2C%20and%20AI%20Tasks%20in%20Google%20Cloud%3A%20Challenge%20Lab/speech-request.json

curl -s -X POST -H "Content-Type: application/json" --data-binary @speech-request.json \ 
"https://speech.googleapis.com/v1/speech:recognize?key=${API_KEY}" > speech.json

gsutil cp speech.json gs://$DEVSHELL_PROJECT_ID-marking/<changefilename>

4.2 Cloud Natural Language API

gcloud ml language analyze-entities --content="Old Norse texts portray Odin as one-eyed and long-bearded, frequently wielding a spear named Gungnir and wearing a cloak and a broad hat." > language.json

gsutil cp language.json gs://$DEVSHELL_PROJECT_ID-marking/<changefilename>

4.3 Google Video Intelligence

  • Copy the Video Intelligence configuration file
wget https://github.com/guys-in-the-cloud/cloud-skill-boosts/blob/main/Challenge-labs/Perform%20Foundational%20Data%2C%20ML%2C%20and%20AI%20Tasks%20in%20Google%20Cloud:%20Challenge%20Lab/video-intelligence-request.json

curl -s -H 'Content-Type: application/json' \
    -H 'Authorization: Bearer '$(gcloud auth print-access-token)'' \
    'https://videointelligence.googleapis.com/v1/videos:annotate' \
    -d @video-intelligence-request.json  > video.json

gsutil cp video.json gs://$DEVSHELL_PROJECT_ID-marking/<changefilename>