Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Fix docker logging on WSL #529

Open
BenElgar opened this issue Nov 18, 2021 · 1 comment
Open

[BUG] Fix docker logging on WSL #529

BenElgar opened this issue Nov 18, 2021 · 1 comment
Labels
bug Something isn't working

Comments

@BenElgar
Copy link
Contributor

BenElgar commented Nov 18, 2021

The problem is that docker logs don't appear (and probably never have) when using exo in WSL.

I spent some time debugging this with John and my understanding of the problem is as follows:
for logging we have three components that need to communicate: the docker daemon, the exo daemon and the syslog daemon. In practice, the exo daemon and the syslog daemon always reside on the same network interface so we can think of them together as "exo".

On Linux they both run natively on the host. Simple. On MacOS exo runs natively and the docker daemon runs in a VM. In WSL the situation is reversed and the docker daemon runs on the host and exo runs in the WSL VM.

The problem appears to be that UDP packets aren't forwarded from the Windows host, where the docker daemon is running, to the WSL VM in which exo is running. I can't find any documentation about this but I can find multiple people on the internet complaining that this is a limitation of WSL[0].

I can't really see a solution that doesn't involve either swapping away from UDP or giving up on WSL support.

[0] Complaining people:
https://old.reddit.com/r/bashonubuntuonwindows/comments/klxbzv/wsl_win10_udp_data_transfer/ghdvj7z/?context=3
https://old.reddit.com/r/bashonubuntuonwindows/comments/fzauf2/win_ubuntu_wsl2_udp_recieve_issue/
MicrosoftDocs/WSL#950
microsoft/WSL#4825
https://stackoverflow.com/questions/66296434/pod-udp-hostport-not-working-in-docker-desktop

@BenElgar BenElgar added bug Something isn't working needs triage Incoming request that needs to be triaged and prioritized labels Nov 18, 2021
@brandonbloom
Copy link
Member

Switching from UDP to TCP is problematic because the Docker daemon fails if connections can't be established, and then does not reconnect if the connection is severed. This means that upgrades or other interruptions of the syslog server will cause all logging to stop working. Though, upgrades already run the risk of losing messages with UDP, it would be nice to use something more reliable.

One option is that we can do our own log forwarding process, just like we do for unix processes. Let Docker talk TCP or some other protocol to that, then we forward it along UDP to our syslog server. If we basically just do the simplest possible Syslog forwarder service, we can probably get away with it, but I don't love the idea of extra processes to manage/consider.

There is also some kind of log buffering "dual logging" system in the more recent docker engine. I think we can query/replay recently buffered logs, so we could also create a forwarding system that works that way, instead of overriding the logging driver.

Alternatively, we can explore proper Windows support, abandoning WSL, though I think that might actually be a step backwards for most Windows users, as far as I can understand.

This thread does seem to suggest UDP should work: https://old.reddit.com/r/bashonubuntuonwindows/comments/fzauf2/win_ubuntu_wsl2_udp_recieve_issue/

@BenElgar What do you think?

@brandonbloom brandonbloom removed the needs triage Incoming request that needs to be triaged and prioritized label Feb 28, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants