iOS app connects to lighthouse (sort of) but cannot see other hosts or resolve their FQDNs

[rob3c]

Hi, I have a small nebula network that connects some linux hosts distributed between a few US states along with some cloud instances. All host firewalls are completely permissive for testing (port, proto, host all set to any for inbound and outbound). The network uses the 100.64.0.0/10 CGNAT range, all hosts can ping each other by IP and ".nebula" FQDN cert name, and SSH works well. The single lighthouse in this test has a nebula IP of 100.64.0.254 and only serves DNS over the nebula1 interface.

Unfortunately, I'm having trouble adding an iOS device (iphone 13 pro) to this setup that successfully allows ping by IP or FQDN on par with the other hosts. (I'm using the Net Analyzer app for ping tests.) I can see handshake log entries for the lighthouse and the phone, and the VPN on the phone thinks that it's successful and remains connected. However, some log entries suggest the handshake may be incomplete, and running list-hostmap in the lighthouse SSH interface doesn't show the phone.

Here's what I've tried so far:

I'm using the latest 0.9.0-127 app version. I created a config choosing the "from scratch" method. I exported the public key and had it signed by the CA. I used generated QR codes to import the signed cert as well as the CA cert. The phone's signed cert -networks entry was set to 100.64.0.13/10, similarly using the /10 prefix length that all other signed host certs used. In the app hosts section, there's a single lighthouse entry with its public IP and nebula IP:4242 values set. The lighthouse toggle enabled. Connection seems to work once it is given iOS permissions, and log entries appear in the lighthouse and phone logs.

Clue: the lighthouse shows both "Handshake message sent" and "Handshake message received" messages for the iphone's certName multiple times, but the iphone log only shows a single "Handshake message sent".

Regardless of that suspicious lack of a "...received" message in the phone log, since it stayed connected (which made me think the app thought everything was ok), I tried (unsuccessfully) ping via IP and FQDN using Net Analyzer, as mentioned above.

I originally thought I might not have the iOS firewall configured correctly for inbound traffic. The app only shows a rendered config that is read-only, and it shows an empty array for the inbound firewall that I assume means everything is blocked. I tried exporting the rendered config, modifying it to have fully permissive inbound entries, but it forced it back to the empty array when I imported it. I then saw the still open issue about editing inbound rules and realized that's the wrong approach.

After looking through the PacketTunnelProvider.swift source and related files, I decided to check for logs using the Console app on my mac. Here are the 2 JSON entries that seem relelvant (with some name and UUID redactions):

NESMVPNSession[Primary Tunnel:<nebula-connection-name>:<uuid-1-redacted>:(null)] starting with configuration: {
    name = <nebula-connection-name-redacted>
    identifier = <uuid-1-redacted>
    applicationName = Nebula
    application = net.defined.mobileNebula
    grade = 1
    VPN = {
        enabled = YES
        onDemandEnabled = NO
        disconnectOnDemandEnabled = NO
        onDemandUserOverrideDisabled = NO
        onDemandRules = (
            {
                action = connect
                interfaceTypeMatch = any
            },
        )
        protocol = {
            type = plugin
            identifier = <uuid-2-redacted>
            serverAddress = Nebula
            identityDataImported = NO
            disconnectOnSleep = NO
            disconnectOnIdle = NO
            disconnectOnIdleTimeout = 0
            disconnectOnWake = NO
            disconnectOnWakeTimeout = 0
            includeAllNetworks = NO
            excludeLocalNetworks = YES
            excludeCellularServices = YES
            excludeAPNs = YES
            excludeDeviceCommunication = YES
            enforceRoutes = NO
            pluginType = net.defined.mobileNebula
            authenticationMethod = 0
            providerConfiguration = {
                id = <uuid-3-redacted>,
            }
            providerBundleIdentifier = net.defined.mobileNebula.NebulaNetworkExtension
        }
        tunnelType = packet
    }
}

and

NESMVPNSession[Primary Tunnel:bigscore-test-cgnat:<guid-1-redacted>:(null)]: plugin set tunnel network settings: {
    tunnelRemoteAddress = 127.0.0.1
    DNSSettings = {
        protocol = cleartext
        server = (
            100.64.0.254,
        )
        matchDomainsNoSearch = NO
        allowFailover = NO
    }
    IPv4Settings = {
        configMethod = manual
        addresses = (
            100.64.0.13,
        )
        subnetMasks = (
            255.192.0.0,
        )
        includedRoutes = (
            {
                destinationAddress = 100.64.0.0
                destinationSubnetMask = 255.192.0.0
            },
        )
        overridePrimary = NO
    }
    IPv6Settings = {
        configMethod = manual
        addresses = ()
        networkPrefixLengths = ()
        includedRoutes = ()
    }
    MTU = 1300
}

I'm no mobile TUN or DNS expert, so I don't know if I caught all relevant config messages or if these look correct as-is or have obvious errors. For example, I'm surprised to not see anything filtering for ".nebula" domains (or even as a search domain, tho I don't want it used for that).

Also, I'm a bit suspicious of the subnet mask being used for both the interface and the destination addresses. I can't easily build the app locally, but if I could I'd try to see if changing v4Netmasks.append(network!.subnetMask) at line 138 in PacketTunnelProvider.swift to v4Netmasks.append("255.255.255.255") would make any difference:

if network!.ipLen == 4 {
        v4Addresses.append(network!.address)
        v4Netmasks.append(network!.subnetMask) // <-- is this right for iOS interfaces for a P2P network like Nebula?
        v4Routes.append(
          NEIPv4Route(
            destinationAddress: network!.maskedAddress,
            subnetMask: network!.subnetMask
          )
        )
      } else {
        v6Addresses.append(network!.address)
        v6PrefixLengths.append(network!.prefixLength as NSNumber)
        v6Routes.append(
          NEIPv6Route(
            destinationAddress: network!.maskedAddress,
            networkPrefixLength: network!.prefixLength as NSNumber
          )
        )
      }

But then I'd wonder how the app is working for anyone on iOS, and it seems like many people use it successfully despite not having to - or even being able to - configure the inbound firewall.

Any thoughts on what I'm doing wrong? What other info could I provide to help get this working if it's not obvious already?

Thanks in advance!
Robert

[nbrownus]

Logs and configs from the lighthouse and phone will be the most useful in helping to determine the cause of your connectivity issues.

You did note that the lighthouse is seeing and responding to handshakes though it's unclear what state the phone is in. Is the phone running on wifi and/or cell during these tests? Do handshakes complete over one and not the other? If things are working while on wifi but fail to work while on cell then it's possible your cell carrier is using the same CGNAT space and causing problems. If CGNAT is your problem, moving to ipv6 on your overlay would be quite helpful since your other options are rfc1918.

To quickly address the other items:

since it stayed connected (which made me think the app thought everything was ok)

The status of the vpn is from the perspective of the OS which is monitoring the process, not mesh connectivity. If the process is alive then the status is connected.

The app only shows a rendered config that is read-only, and it shows an empty array for the inbound firewall

That's correct, currently there is no way to manipulate the inbound firewall outside of compiling mobile nebula on your own. This will hopefully be addressed in the next week or two via #354. Nebulas firewall is stateful, which means that return traffic from allowed outbound flows will be allowed back in. The only time you need to accept inbound traffic in nebulas firewall is if that traffic is intended to be unsolicited, like a web server or database waiting for connections instead of a web browser that initiates a connection. Most phones aren't running a service meant to be reachable from the network.

I decided to check for logs using the Console app on my mac. Here are the 2 JSON entries that seem relelvant (with some name and UUID redactions):

The first log item is showing what iOS knows about the vpn, since iOS does not implement nebula directly, most of this is vanity text to display in the Settings.app to help folks identify which site is which.

The second log item is showing how nebula has configured the iOS vpn stack. Your nebula network, encoded in the certificate, is 100.64.0.13/10. This is showing that the overlay side (tun device) is getting an address of 100.64.0.13 assigned to it and it's network is 100.64.0.0 which is applying the netmask of 255.192.0.0 to the address 100.64.0.13. 255.192.0.0 is the netmask representation of the /10 prefix length. If you were to change the netmask to 255.255.255.255 (prefix len of /32) then you would be saying that this vpn network is a single node network. It would likely still work since we also declare the route but it's atypical device configuration.

For example, I'm surprised to not see anything filtering for ".nebula" domains

We don't own the .nebula tld so we can't make assumptions about how to resolve domains within it. The dns settings in mobile also do not implement split horizon, all dns lookups will go through the configured dns servers.

[rob3c]

Hi, thanks for the reply!

The phone in all of these tests is on the same wifi that one of the desktop linux hosts is using. Cellular doesn't work at all, even with IP. It's good to know that the app remaining in the connected state provides zero info for this problem.

To rule out CGNAT, I've reconfigured the test network to use only ipv6 as you suggested. The linux hosts all work well as before including DNS, but now the iphone app can use nebula by IP over wifi - yay! It still doesn't work over cellular, unfortunately.

I saw the demo video from 5 years ago by Ryan Huber where he shows his iphone connecting via wifi, then switching to cellular and continuing to work (after recovering after a few bad pings), and then switching back to wifi with no problems. He was using the CGNAT range and appeared to use ATT cellular service. I have Verizon via the StraightTalk MVNO on my test phone, so maybe that's an issue. Any insight here would be appreciated!

Anyway, even with ipv6, DNS still isn't working on mobile of course, since the app doesn't support split DNS that the lighthouse would require due to its limited implementation. I thought I could remedy this by creating a custom ".mobileconfig" that doesn't use up a VPN slot and configure the matchDomains field myself to send private domain queries only to the lighthouse DNS server.

Unfortunately, the "com.apple.dnsSettings.managed" payload type only allows configuring DoH or DoT. I could proxy those and forward to the lighthouse, but that requires a public endpoint for the required SSL cert that apple expects, and that requires a much more robust DNS implementation security-wise than simply fielding nebula only queries using the lighthouse over tun0 due to a device-local config.

The alternative split DNS config using unencrypted port 53 seems to require using a VPN slot, which obviously won't work since that would prevent Mobile Nebula from connecting. The real way to do split DNS is in the Mobile Nebula app itself, as apple expects it to provide the authoritative DNS for its interface that way.

We don't own the .nebula tld so we can't make assumptions about how to resolve domains within it. The dns settings in mobile also do not implement split horizon, all dns lookups will go through the configured dns servers.

This doesn't have anything to do with you owning the .nebula TLD. And of course, there is no .nebula TLD anyway officially. Mobile Nebula choosing not to expose e.g. match domains to route private DNS queries over the nebula interface as a convenience feature is what surprises me. It seems very easy to do and otherwise impossible for us to do ourselves (without running a public DNS endpoint), so what's the thinking behind that design decision? Is it just that you don't have the resources to make it a priority and would accept a PR, or are you explicitly avoiding that support for some reason?

[nbrownus]

re: connectivity issues - bummer that ipv6 didn't get around the issue. Based on the information you've provided, it really seems like it's something specific to the phones cellular network. Logs from the lighthouse and the phone would be helpful in determining that.

re dns: I opened #374 to track the issue. We are willing to accept PRs if the author is happy to sign our CLA and there is time to review and merge.