Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow S3-Compatible Cloud Object Storage in CloudETL Example #1264

Open
wants to merge 4 commits into
base: 7.7.1-post
Choose a base branch
from
Open

Allow S3-Compatible Cloud Object Storage in CloudETL Example #1264

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
be110b6
Add S3_ENDPOINT_URL to allow use of S3-compatible cloud object storage.
metadaddy Sep 27, 2024
6619c19
Include S3 profile in list-objects call
metadaddy Sep 27, 2024
5959035
Empty commit for confluent-cla-assistant
metadaddy Oct 8, 2024
7d3bf40
Set S3_ENDPOINT_URL correctly when it is not in the AWS config
metadaddy Oct 8, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions cloud-etl/connectors/s3_avro.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
"kafka.api.secret": "$CLOUD_SECRET",
"aws.access.key.id": "$AWS_ACCESS_KEY_ID",
"aws.secret.access.key": "$AWS_SECRET_ACCESS_KEY",
"store.url": "$S3_ENDPOINT_URL",
"s3.bucket.name": "$S3_BUCKET",
"data.format": "AVRO",
"time.interval" : "HOURLY",
Expand Down
1 change: 1 addition & 0 deletions cloud-etl/connectors/s3_no_avro.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
"kafka.api.secret": "$CLOUD_SECRET",
"aws.access.key.id": "$AWS_ACCESS_KEY_ID",
"aws.secret.access.key": "$AWS_SECRET_ACCESS_KEY",
"store.url": "$S3_ENDPOINT_URL",
"s3.bucket.name": "$S3_BUCKET",
"time.interval" : "HOURLY",
"data.format": "BYTES",
Expand Down
4 changes: 2 additions & 2 deletions cloud-etl/read-data.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,7 @@ else
--schema-registry-endpoint $SCHEMA_REGISTRY_URL\
--from-beginning\
--print-key\
--value-format avro
--value-format avro
fi


Expand All @@ -92,7 +92,7 @@ AVRO_VERSION=1.9.1
#fi
if [[ "$DESTINATION_STORAGE" == "s3" ]]; then

for key in $(aws s3api list-objects --bucket $S3_BUCKET | jq -r '.Contents[].Key'); do
for key in $(aws s3api list-objects --bucket $S3_BUCKET --profile $S3_PROFILE | jq -r '.Contents[].Key'); do
echo "S3 key: $key"
#aws s3 cp s3://$S3_BUCKET/$key data.avro
#echo "java -Dlog4j.configuration="file:log4j.properties" -jar avro-tools-${AVRO_VERSION}.jar tojson data.avro"
Expand Down
5 changes: 4 additions & 1 deletion cloud-etl/setup_storage_s3.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
#!/bin/bash

# Source library
source ../utils/helper.sh
source ../utils/ccloud_library.sh
Expand All @@ -25,6 +25,9 @@ fi

export AWS_ACCESS_KEY_ID=$(aws configure get aws_access_key_id --profile $S3_PROFILE)
export AWS_SECRET_ACCESS_KEY=$(aws configure get aws_secret_access_key --profile $S3_PROFILE)
export S3_ENDPOINT_URL=$(aws configure get endpoint_url --profile $S3_PROFILE)
export S3_ENDPOINT_URL="${S3_ENDPOINT_URL:-https://s3.$STORAGE_REGION.amazonaws.com}"

ccloud::create_connector connectors/s3_no_avro.json || exit 1
ccloud::wait_for_connector_up connectors/s3_no_avro.json 300 || exit 1
ccloud::create_connector connectors/s3_avro.json || exit 1
Expand Down