Port Forwarding - curl timeout for PF Token

[Olanzer]

Asustor Nimbustor (5404T). Has a kernel update for wireguard, but it's not really natively supported (no wg-quick I don't think?)

Running Portainer CE using a stack. Can connect to wg fine with Port Forwarding turned off (at least I think this is working?)

Sat Aug  3 10:31:16 UTC 2024: Falling back to iptables-legacy
Fetching next-gen PIA server list
Verified OK
Verified server list
Registering public key with PIA endpoint; id: aus_perth, cn: perth403, ip: 179.61.228.50
Generating /etc/wireguard/wg0.conf
Using PIA DNS servers: 10.0.0.243,10.0.0.242
Port forwarding is available at this location
Successfully generated /etc/wireguard/wg0.conf
Sat Aug  3 10:31:19 UTC 2024: Bringing up WireGuard interface wg0
[#] ip link add wg0 type wireguard
[#] wg setconf wg0 /dev/fd/63
[#] ip -4 address add 10.49.254.68 dev wg0
[#] ip link set mtu 1420 up dev wg0
[#] resolvconf -a wg0 -m 0 -x
could not detect a useable init system
[#] wg set wg0 fwmark 51820
[#] ip -4 route add 0.0.0.0/0 dev wg0 table 51820
[#] ip -4 rule add not fwmark 51820 table 51820
[#] ip -4 rule add table main suppress_prefixlength 0
[#] iptables-restore -n
iptables-restore v1.8.10 (legacy): iptables-restore: unable to initialize table 'raw'
Error occurred at line: 1
Try `iptables-restore -h' or 'iptables-restore --help' for more information.
interface: wg0
  public key: aYbigtIzBrVfyCRKCXyCKHweQNgUJglDHiKbOnIEbBc=
  private key: (hidden)
  listening port: 33960
  fwmark: 0xca6c
peer: AiKBWW2oJahzloBQe1vqFOoxD17HLVjsdEfzgoi9ESM=
  endpoint: 179.61.228.50:1337
  allowed ips: 0.0.0.0/0
Sat Aug  3 10:31:19 UTC 2024: WireGuard successfully started
Sat Aug  3 10:31:19 UTC 2024: Allowing network access on eth0
Warning: Extension mark revision 0 not supported, missing kernel module?
iptables: No chain/target/match by that name.
Sat Aug  3 10:31:19 UTC 2024: Firewall enabled: Blocking non-WireGuard traffic
Sat Aug  3 10:31:19 UTC 2024: Allowing network access on eth0

But when I turn Port Forwarding to 1, this happens:

Sat Aug  3 10:24:11 UTC 2024: Falling back to iptables-legacy
Fetching next-gen PIA server list
Verified OK
Verified server list
Registering public key with PIA endpoint; id: aus_perth, cn: perth405, ip: 179.61.228.183
Generating /etc/wireguard/wg0.conf
Using PIA DNS servers: 10.0.0.243,10.0.0.242
Port forwarding is available at this location
Successfully generated /etc/wireguard/wg0.conf
Sat Aug  3 10:24:13 UTC 2024: Bringing up WireGuard interface wg0
[#] ip link add wg0 type wireguard
[#] wg setconf wg0 /dev/fd/63
[#] ip -4 address add 10.54.188.151 dev wg0
[#] ip link set mtu 1420 up dev wg0
[#] resolvconf -a wg0 -m 0 -x
could not detect a useable init system
[#] wg set wg0 fwmark 51820
[#] ip -4 route add 0.0.0.0/0 dev wg0 table 51820
[#] ip -4 rule add not fwmark 51820 table 51820
[#] ip -4 rule add table main suppress_prefixlength 0
[#] iptables-restore -n
iptables-restore v1.8.10 (legacy): iptables-restore: unable to initialize table 'raw'
Error occurred at line: 1
Try `iptables-restore -h' or 'iptables-restore --help' for more information.
interface: wg0
  public key: HWsS/MRoE+GMBaylAgc6A1NJ9Ei5foXfQbhqFcJI+Gw=
  private key: (hidden)
  listening port: 59649
  fwmark: 0xca6c
peer: k/6P7abuf0zIGAHj3Sk/0TjBL78CLt1zdcI/dI+TGDE=
  endpoint: 179.61.228.183:1337
  allowed ips: 0.0.0.0/0
Sat Aug  3 10:24:13 UTC 2024: WireGuard successfully started
Sat Aug  3 10:24:13 UTC 2024: Allowing network access
Warning: Extension mark revision 0 not supported, missing kernel module?
iptables: No chain/target/match by that name.
Sat Aug  3 10:24:13 UTC 2024: Firewall enabled: Blocking non-WireGuard traffic
Sat Aug  3 10:24:13 UTC 2024: Starting port forward script
Sat Aug  3 10:24:13 UTC 2024: Verifying API requests. CN: perth405
Sat Aug  3 10:24:13 UTC 2024: Getting PF token
curl: (28) Failed to connect to perth405 port 19999 after 30002 ms: Timeout was reached
curl: (28) Failed to connect to perth405 port 19999 after 30002 ms: Timeout was reached
curl: (28) Failed to connect to perth405 port 19999 after 30002 ms: Timeout was reached
curl: (28) Failed to connect to perth405 port 19999 after 30002 ms: Timeout was reached
curl: (28) Failed to connect to perth405 port 19999 after 30002 ms: Timeout was reached

Docker-compose (tried running the testing version to see if that helps but no dice)

services:

  pia:
      image: thrnz/docker-wireguard-pia:testing
      cap_add:
          - NET_ADMIN
          # SYS_MODULE might not be needed with a 5.6+ kernel?
          - SYS_MODULE
      # If the kernel module isn't available, mounting the tun device may be necessary for userspace implementations
      #devices:
      #  - /dev/net/tun:/dev/net/tun
      environment:
          # The following env vars are required:
          - LOC=aus_perth
          - USER=(Username)
          - PASS=(Password)
          - PORT_FORWARDING=1
          - PORT_PERSIST=1       
      sysctls:
          # The wg-quick script tries to set this when setting up routing, however this requires running the container
          # with the --privileged flag set. Setting it here instead if needed means the container can be run with lower
          # privileges. This only needs setting if strict reverse path filtering (rp_filter=1) is used.
          - net.ipv4.conf.all.src_valid_mark=1
          # May as well disable ipv6. Should be blocked anyway.
          - net.ipv6.conf.default.disable_ipv6=1
          - net.ipv6.conf.all.disable_ipv6=1
          - net.ipv6.conf.lo.disable_ipv6=1
      # The container has no recovery logic. Use a healthcheck to catch disconnects.
      healthcheck:
          test: ping -c 1 www.google.com || exit 1
          interval: 30s
          timeout: 10s
          retries: 3
      volumes:
          # Auth token is stored here
          - /volume1/Docker/pia:/pia
          # If enabled, the forwarded port is dumped to /pia-shared/port.dat for potential use in other containers
          - /volume1/Docker/pia/pia-shared:/pia-shared

I also tried other locations like aus_melbourne and overseas but AFAIK wireguard works on almost all locations with PF turned on anyway.

[thrnz]

If the Wireguard connection itself is up and working as expected, then I'm not sure why port forwarding would be unable to start. It looks like their Perth based server works fine on a Debian host:

vpn-1  | 2024-08-04T00:46:04.259679954Z Sun Aug  4 00:46:04 UTC 2024: WireGuard successfully started
vpn-1  | 2024-08-04T00:46:04.315170769Z Sun Aug  4 00:46:04 UTC 2024: Allowing network access to 172.16.11.2/24 on eth0
vpn-1  | 2024-08-04T00:46:04.322277522Z Sun Aug  4 00:46:04 UTC 2024: Firewall enabled: Blocking non-WireGuard traffic
vpn-1  | 2024-08-04T00:46:04.327330660Z Sun Aug  4 00:46:04 UTC 2024: Starting port forward script
vpn-1  | 2024-08-04T00:46:04.341727667Z Sun Aug  4 00:46:04 UTC 2024: Verifying API requests. CN: perth404
vpn-1  | 2024-08-04T00:46:04.343326879Z Sun Aug  4 00:46:04 UTC 2024: Getting PF token
vpn-1  | 2024-08-04T00:46:05.360529341Z Sun Aug  4 00:46:05 UTC 2024: Obtained PF token. Expires at 2024-10-05T12:46:04.682771315Z
vpn-1  | 2024-08-04T00:46:05.361323446Z Sun Aug  4 00:46:05 UTC 2024: Server accepted PF bind
vpn-1  | 2024-08-04T00:46:05.362124852Z Sun Aug  4 00:46:05 UTC 2024: Forwarding on port 37144
vpn-1  | 2024-08-04T00:46:05.362965459Z Sun Aug  4 00:46:05 UTC 2024: Rebind interval: 900 seconds
vpn-1  | 2024-08-04T00:46:05.363929166Z Sun Aug  4 00:46:05 UTC 2024: Port dumped to /pia-shared/port.dat
vpn-1  | 2024-08-04T00:46:05.365032874Z Sun Aug  4 00:46:05 UTC 2024: This script should remain running to keep the forwarded port alive
vpn-1  | 2024-08-04T00:46:05.365844680Z Sun Aug  4 00:46:05 UTC 2024: Press Ctrl+C to exit
vpn-1  | 2024-08-04T00:46:05.366774386Z Sun Aug  4 00:46:05 UTC 2024: Running /scripts/pf_success.sh
vpn-1  | 2024-08-04T00:46:05.372548629Z Sun Aug  4 00:46:05 UTC 2024: Allowing incoming traffic on port 37144

It might be worth double checking that the vpn itself is otherwise working. Running something like curl https://icanhazip.com/ inside the container might be helpful to confirm that traffic is being tunnelled correctly - it should show the vpn's external ip address.

I wonder if those iptables errors might be relevant:

Error occurred at line: 1
Try `iptables-restore -h' or 'iptables-restore --help' for more information.
...
Sat Aug  3 10:31:19 UTC 2024: Allowing network access on eth0
Warning: Extension mark revision 0 not supported, missing kernel module?
iptables: No chain/target/match by that name.

The first is from wg-quick bringing up the Wireguard interface, though doesn't seem to be fatal. I'm not sure what issues (if any) that one would cause.
The second is from when the container adds firewall rules to only allow tunnelled traffic. If something goes wrong there then it might end up with things getting blocked. Testing with the FIREWALL=0 env var set might show whether that's the culprit.

Running the container with the DEBUG=1 env var set might be useful to find the command that's causing the error, though as the output isn't sanitized in any way, it might need manually pruning before posting.

[Olanzer]

Looks like vpn isn't working at all when I disable Port Forwarding and it "connects":

interface: wg0
  public key: pDfzokR6vUHAPkBy9eI+P4vRG3W9Gx948lU3sesCgWE=
  private key: (hidden)
  listening port: 41219
  fwmark: 0xca6c
peer: em2QlFkY0cF6t6bptXWWivz1H6rUJC9dm9c7sXH5Fxs=
  endpoint: 179.61.228.151:1337
  allowed ips: 0.0.0.0/0
Sun Aug  4 10:50:14 UTC 2024: WireGuard successfully started
Sun Aug  4 10:50:14 UTC 2024: Allowing network access to 172.... on eth0
Warning: Extension mark revision 0 not supported, missing kernel module?
iptables: No chain/target/match by that name.
Sun Aug  4 10:50:14 UTC 2024: Firewall enabled: Blocking non-WireGuard traffic

:/scripts# curl https://icanhazip.com/
curl: (6) Could not resolve host: icanhazip.com

:/scripts# ping -w 5 104.16.185.241
PING 104.16.185.241 (104.16.185.241): 56 data bytes

--- 104.16.185.241 ping statistics ---
5 packets transmitted, 0 packets received, 100% packet loss

It's the Warning: Extension mark revision 0 not supported, missing kernel module? -- line that also has me wondering.

I think the NAS doesn't fully support wg - I know it doesn't have wg-quick installed, the wg update was through a kernel update through the NAS web UI. Asustor uses custom BusyBox with their own ADM O/S so you're limited to what you can do console side.

OpenVPN works, so will use that for now until I can get wireguard up and running.

[Olanzer]

UPDATE: So it seems that if I use the WireGuard feature in ASUSTOR web-UI, I can connect to PIA if I use the .conf file generated by the PIA Manual Connections script (just ran it on my Raspberry Pi). So wg is definitely working, might just be since wg-quick isn't installed the container runs into errors when it tries to call wg-quick routines. Have to work out a way to either install it or find out how the Asustor connects to WireGuard.

[thrnz]

wg-quick is just a bash script that brings up the Wireguard interface and sets up routing. The container uses it to setup networking, though it shouldn't need to be installed on the host itself.

https://www.man7.org/linux/man-pages/man8/wg-quick.8.html
https://git.zx2c4.com/wireguard-tools/tree/src/wg-quick/linux.bash

This suggests that the issue might be related to an iptables module that might not be available. I guess their software sets things up differently to the way wg-quick does.

[Olanzer]

So ASUSTOR does support Entware and opkg applications. I've been able to install wg, wg-quick, iptables and many others to the NAS and can access it from console. Packages are stored in /opt/bin and /opt/sbin. However when I set these as PATH variables in the container, even tired IPTABLES and IPTABLES-RESTORE to /opt/sbin, I'm still getting the same error. Anyway I can get your container to use the opkg versions instead for iptables restore, etc? This might help. I'm pretty new to linux and docker, been a bit of a learning curve, but at least the NAS can run Entware stuff so I can use bash now instead of clunky sh!

[thrnz]

The container doesn't have access to the host's filesystem. It is possible to mount and run the host's binaries inside the container, but it can be a bit fiddly and might not work the same as if they were ran on the host.

It might be worth testing to see if running wg-quick on the host itself using the generated wireguard config file works, or whether it throws the same error as inside the container. To test I assume the generated wg0.conf would need to be placed at /etc/wireguard/wg0.conf, then brought up with wg-quick up wg0. It can then be brought down again with wg-quick down wg0 if needed. If they haven't made any changes to wg-quick then you might get the same error as inside the container anyway.

I don't know if it's relevant, but on OpenWrt theres a kmod-ipt-raw package available that might provide support for the missing module. I wonder if a comparable package for your nas might be available somehow.