Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bug: Cloudwatch parsing tests #536

Open
pnadolny13 opened this issue Jan 31, 2023 · 1 comment
Open

bug: Cloudwatch parsing tests #536

pnadolny13 opened this issue Jan 31, 2023 · 1 comment
Assignees
Labels
data/Modeling kind/Bug Something isn't working

Comments

@pnadolny13
Copy link
Contributor

pnadolny13 commented Jan 31, 2023

Related to #108. There are some minor data quality issues that require us to comment out some schema tests.

  • IP address hashes are not parsed for some logs, I suspect due to truncation, but we get it in another message body so we should attempt to parse using the X-Forwarded-For list. Need to figure out what the other one is.
  • status codes, request_path, and latency are missing for ~600 requests. Its an extremely small number relatively but it looks like those logs exist in data dog so it must be a bug with replication that they dont exist in Snowflake.
@pnadolny13 pnadolny13 self-assigned this Jan 31, 2023
@pnadolny13 pnadolny13 changed the title Cloudwatch parsing bugs bug: Cloudwatch parsing Feb 1, 2023
@pnadolny13 pnadolny13 changed the title bug: Cloudwatch parsing bug: Cloudwatch parsing tests Feb 1, 2023
@pnadolny13
Copy link
Contributor Author

        ARRAY_SLICE(
    		split(
    			split_part(
    				split_part(payload, 'X-Forwarded-For=', 2),
    				'=', 1
    			), ', '
    		),
    		0, -1
    	) AS ip_address_array

@pnadolny13 pnadolny13 added data/Modeling kind/Bug Something isn't working labels Apr 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
data/Modeling kind/Bug Something isn't working
Projects
Status: No status
Development

No branches or pull requests

1 participant