Replicate Meltano Cloud Cloudwatch Logs

[pnadolny13]

Meltano Cloud API logs are stored in Cloudwatch, we should replicate them into Snowflake. The tap is already set up to extract the hub api logs so this would likely only require us to:

• grant the Squared role access to the logs
• inherit the tap-cloudwatch plugin and configure to the Meltano Cloud log group or groups
• replicate data and add dbt staging models

Considerations:

• if each API has its own log group it will require an inherited plugin per group, I wonder if refactoring the tap to accept multiple log groups and queries as different streams would be better.
• how should this data be staged? All of these cloudwatch streams will likely have the same schema so it might be easy to combine them with a log group field that can be used for filtering. Or if we want them separate we can use a macro.

[pnadolny13]

This is what I'm seeing:

• API-Gateway-Execution-Logs_o0at298os6/prod - /jobs
• API-Gateway-Execution-Logs_3ssgdis8jk/prod - /decrypt
• API-Gateway-Execution-Logs_v985hw9g43/auth-github-oidc-wrapper - /userinfo