add option to send debug logs to a unix socket#48300
Conversation
jkarneges
changed the title
jkarneges
force-pushed
the
jkarneges/debug-socket
branch
2 times, most recently
from
65a3fe2 to
d5b6ea1
Compare
src/core/log.rs
Outdated
| // Send immediately if the client's queue isn't full, else drop | ||
| match c.sender.try_send(Rc::clone(&msg)) { | ||
| Err(mpsc::TrySendError::Disconnected(_)) => false, | ||
| _ => true, // Success or full |
There was a problem hiding this comment.
maybe push a log to systemd to know if the queue is full?
There was a problem hiding this comment.
Since drops can only happen when there is at least one debug log connection open, I came up with a way to log this information to the debug clients rather than the main logs. Basically there's a drop counter for the main queue and one for each client, and they stack.
src/core/log.rs
Outdated
| LOGGER.get().expect("logger should be initialized") | ||
| } | ||
|
|
||
| const QUEUE_MAX: usize = 1000; |
There was a problem hiding this comment.
would we ever want to tune this?
There was a problem hiding this comment.
Probably. For I've now made it dynamic based on the number of worker threads. We can tweak this depending on how things go.
src/core/log.rs
Outdated
| fn send(&self, message: String) { | ||
| if let Some(sender) = &self.msgs { | ||
| // Send immediately if the queue isn't full, else drop | ||
| let _ = sender.try_send(LogMessage { inner: message }); |
There was a problem hiding this comment.
systemd log to know we're hitting full queues?
| return; | ||
| } | ||
|
|
||
| let now = OffsetDateTime::now_utc().to_offset(local_offset().unwrap_or(UtcOffset::UTC)); |
There was a problem hiding this comment.
I'd propose default to logging in UTC offset with maybe a flag for local timezone.
There was a problem hiding this comment.
This logic is the same as our global logger which I think is the right way to go. Local time is less surprising behavior out-of-the-box. Folks that want UTC (e.g. on server nodes) can configure timezones accordingly.
jkarneges
force-pushed
the
jkarneges/debug-socket
branch
from
07bef01 to
060271a
Compare
2 participants
This adds an option to connmgr for sending debug logs to a unix socket. The
--debug-socketarg can be used to specify a unix socket path to serve logs from. Any clients connected to that path receive all logs regardless of the configured log level limit.The debug logger is invoked from the global logger so that all logs can be sent through it. However, the debug logger also implements the
Logtrait, in case we ever want to log to it explicitly.To ensure the mechanism doesn't introduce too much overhead when it is unused, the
Log::enabledtrait method skips logging when no clients are connected. This means expensive logging code guarded bylog_enabled!should get skipped appropriately.Broadcasting to clients is done from a separate thread to avoid tying up callers. When a logging call is made, the logger simply sends the log message to a channel and returns. A separate thread then reads from the channel, handling the rest of the pipeline. The channel is bounded and the send is best-effort using non-blocking
try_send()calls, so if the channel is full then logs are dropped.