Better-integrate our logging with Google Cloud logging

[grahamalama]

It appears that all of our logs are being written as Errors in the Google Cloud Logging dashboard:

I believe this is due to the fact that we're using a StreamHandler to handle logging, which by default writes to stderr (Python docs).

We could modifying our logging config to write to stdout instead. This would turn all of the errors into Info messages.

However, doing so would then have the opposite effect -- we'd, for example, bury exceptions as Info messages.

One option to improve the usefulness of the logging dashboard would be to write our logs in a format that would integrate well with Google Cloud logging. There are a few options outlined here.

I think moving in this direction might also mean not using Mozlog. It seems like the two structured log formats are incompatible.

[mostlygeek]

Logging to stdout is the recommended practice.
What is the reason for logging to stderr instead?

On switching from mozlog, can't we use the mozlog format with google cloud logging?

[grahamalama]

What is the reason for logging to stderr instead?

I don't think there's a good reason -- this is just the default for Python's StreamHandler, which I'm guessing we just forgot to change.

On switching from mozlog, can't we use the mozlog format with google cloud logging?

We might be able to. I'm still trying to understand this part. Looking into this more, I found:

Some fields in the JSON object are recognized as special by the Logging agent and extracted into the LogEntry structure. These special JSON fields can be used to set the following fields in the LogEntry :

• severity

And then looking here:

If you are writing log entries, you should map other severity encodings to one of these standard levels.

So it would seem that the one deviation from mozlog we might want to make is write severity levels using strings (e.g. WARNING) rather than the syslog number (e.g. 6). That would categorize log messages appropriately by level in the Google Cloud Logging dashboard.

[grahamalama]

It would also appear that we could write request information to an httpRequest key following Google's schema, and that might also give us something?

[grahamalama]

We might also be able to write some configuration that converts mozlog records to records that are compatible with Google Cloud Logging:

https://cloud.google.com/logging/docs/structured-logging#structured_logging_operations
https://cloud.google.com/logging/docs/agent/ops-agent/configuration#logging-processors