Cannot connect/mount to a new NFS Server where the logs say it's all ok

[Danilo-Araujo-Silva]

Unfortunately, I'm not being able to connect/mount the server and I'd like to request some help.
I'm currently using Windows 11 with Ubuntu 22.04.3 LTS on WSL.

I've created the container with this command:

clear

docker rm -f nfs

# -e NFS_DISABLE_VERSION_3=true \

docker run \
	--name nfs \
	--network host \
	-v /media/shared:/media/shared \
	-v /lib/modules:/lib/modules:ro \
	-e NFS_EXPORT_0='/media/shared *(rw,sync,no_subtree_check)' \
	-e NFS_LOG_LEVEL='DEBUG' \
	--privileged \
	--cap-add SYS_ADMIN \
	--cap-add SYS_MODULE \
	erichough/nfs-server

#docker run \
#	--name nfs \
#	--network bridge \
#	-v /media/shared:/media/shared \
#	-v /lib/modules:/lib/modules:ro \
#	-e NFS_EXPORT_0='/media/shared *(rw,sync,no_subtree_check)' \
#	-e NFS_LOG_LEVEL='DEBUG' \
#	--privileged \
#	--cap-add SYS_ADMIN \
#	--cap-add SYS_MODULE \
#	-p 50111:111/tcp \
#	-p 50111:111/udp \
#	-p 50049:2049/tcp \
#	-p 50049:2049/udp \
#	-p 52765:32765/tcp \
#	-p 52765:32765/udp \
#	erichough/nfs-server

where I have also tried to create it in bridge mode as well as we can see above.

The logs say everything should be ok

==================================================================
      SETTING UP ...
==================================================================
----> log level set to DEBUG
----> will use 32 rpc.nfsd server thread(s) (1 thread per CPU)
----> building /etc/exports from environment variables
----> collected 1 valid export(s) from NFS_EXPORT_* environment variables
----> kernel module nfs is loaded
----> kernel module nfsd is loaded
----> setup complete

==================================================================
      STARTING SERVICES ...
==================================================================
----> mounting rpc_pipefs filesystem onto /var/lib/nfs/rpc_pipefs
mount: mount('rpc_pipefs','/var/lib/nfs/rpc_pipefs','rpc_pipefs',0x00008000,'(null)'):0
----> mounting nfsd filesystem onto /proc/fs/nfsd
mount: mount('nfsd','/proc/fs/nfsd','nfsd',0x00008000,'(null)'):0
----> starting rpcbind
----> starting exportfs
exporting *:/media/shared
----> starting rpc.mountd on port 32767
----> starting rpc.statd on port 32765 (outgoing from port 32766)
----> starting rpc.nfsd on port 2049 with 32 server thread(s)
rpc.nfsd: knfsd is currently down
rpc.nfsd: Writing version string to kernel: -2 +3 +4 +4.1 +4.2
rpc.nfsd: Created AF_INET TCP socket.
rpc.nfsd: Created AF_INET UDP socket.
rpc.nfsd: Created AF_INET6 TCP socket.
rpc.nfsd: Created AF_INET6 UDP socket.
Statd service already running!
----> all services started normally

==================================================================
      SERVER STARTUP COMPLETE
==================================================================
----> list of enabled NFS protocol versions: 4.2, 4.1, 4, 3
----> list of container exports:
---->   /media/shared   *(rw,sync,wdelay,hide,nocrossmnt,secure,root_squash,no_all_squash,no_subtree_check,secure_locks,acl,no_pnfs,anonuid=65534,anongid=65534,sec=sys,rw,secure,root_squash,no_all_squash)
----> list of container ports that should be exposed:
---->   111 (TCP and UDP)
---->   2049 (TCP and UDP)
---->   32765 (TCP and UDP)
---->   32767 (TCP and UDP)

From inside the container I have:

bash-5.0# clear && netstat -tulnp | grep LISTEN && exportfs && cat /media/shared/test.txt && ps aux | grep rpc && showmount -e localhost

bash-5.0#
tcp        0      0 0.0.0.0:32767           0.0.0.0:*               LISTEN      44/rpc.mountd
tcp        0      0 0.0.0.0:2049            0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:39319           0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:53317           0.0.0.0:*               LISTEN      -
tcp        0      0 :::60727                :::*                    LISTEN      -
tcp        0      0 :::36109                :::*                    LISTEN      -
tcp        0      0 :::32767                :::*                    LISTEN      44/rpc.mountd
tcp        0      0 :::2049                 :::*                    LISTEN      -
tcp        0      0 :::111                  :::*                    LISTEN      -
/media/shared   <world>
This text is inside the file.

   40 daemon    0:00 /sbin/rpcbind -s
   44 root      0:00 /usr/sbin/rpc.mountd --port 32767 --nfs-version 4.2 --no-nfs-version 2 --debug all
  103 root      0:00 grep rpc
clnt_create: RPC: Program not registered

But from outside the container I'm not being able to do anything useful yet (none of the commands below work, they are stuck, waiting for something, but cannot perform the task)

sudo mount -t nfs -o vers=4 localhost:/media/shared /mnt/nfs

showmount -e localhost

nc -zv localhost 2049

What can I do?

[icicimov]

@Danilo-Araujo-Silva you just need to add fsid=0 to your NFS_EXPORT_0 and then for nfs v4 the source mount point is /

sudo mount -t nfs -o vers=4 localhost:/ /mnt/nfs

[Danilo-Araujo-Silva]

@Danilo-Araujo-Silva you just need to add fsid=0 to your NFS_EXPORT_0 and then for nfs v4 the source mount point is /

sudo mount -t nfs -o vers=4 localhost:/ /mnt/nfs

Many thanks for the reply, now I've been able to move forward o/

But it seems it's almost there..

I haven't been able to advance with the host network, but with bridge some more things are working now.

I've created the container with

clear

docker rm -f nfs1
#docker rm -f nfs2

# -e NFS_DISABLE_VERSION_3=true \

docker run \
	--name nfs1 \
	--network bridge \
	-v /media/shared:/media/shared \
	-v /lib/modules:/lib/modules:ro \
	-e NFS_EXPORT_0='/media/shared *(rw,sync,no_subtree_check,fsid=0)' \
	-e NFS_LOG_LEVEL='DEBUG' \
	--privileged \
	--cap-add SYS_ADMIN \
	--cap-add SYS_MODULE \
	-p 50111:111/tcp \
	-p 50111:111/udp \
	-p 50049:2049/tcp \
	-p 50049:2049/udp \
	-p 52765:32765/tcp \
	-p 52765:32765/udp \
	erichough/nfs-server

#docker run \
#	--name nfs2 \
#	--network host \
#	-v /media/shared:/media/shared \
#	-v /lib/modules:/lib/modules:ro \
#	-e NFS_EXPORT_0='/media/shared *(rw,sync,no_subtree_check,fsid=0)' \
#	-e NFS_LOG_LEVEL='DEBUG' \
#	--privileged \
#	--cap-add SYS_ADMIN \
#	--cap-add SYS_MODULE \
#	erichough/nfs-server

where the logs where

==================================================================
      SETTING UP ...
==================================================================
----> log level set to DEBUG
----> will use 32 rpc.nfsd server thread(s) (1 thread per CPU)
----> building /etc/exports from environment variables
----> collected 1 valid export(s) from NFS_EXPORT_* environment variables
----> kernel module nfs is loaded
----> kernel module nfsd is loaded
----> setup complete

==================================================================
      STARTING SERVICES ...
==================================================================
----> mounting rpc_pipefs filesystem onto /var/lib/nfs/rpc_pipefs
mount: mount('rpc_pipefs','/var/lib/nfs/rpc_pipefs','rpc_pipefs',0x00008000,'(null)'):0
----> mounting nfsd filesystem onto /proc/fs/nfsd
mount: mount('nfsd','/proc/fs/nfsd','nfsd',0x00008000,'(null)'):0
----> starting rpcbind
----> starting exportfs
exporting *:/media/shared
----> starting rpc.mountd on port 32767
----> starting rpc.statd on port 32765 (outgoing from port 32766)
----> starting rpc.nfsd on port 2049 with 32 server thread(s)
rpc.nfsd: knfsd is currently down
rpc.nfsd: Writing version string to kernel: -2 +3 +4 +4.1 +4.2
rpc.nfsd: Created AF_INET TCP socket.
rpc.nfsd: Created AF_INET UDP socket.
rpc.nfsd: Created AF_INET6 TCP socket.
rpc.nfsd: Created AF_INET6 UDP socket.
rpc.statd: Version 2.3.4 starting
rpc.statd: Flags: No-Daemon Log-STDERR TI-RPC
rpc.statd: Failed to read /var/lib/nfs/state: Address in use
rpc.statd: Initializing NSM state
rpc.statd: Local NSM state number: 3
rpc.statd: Running as root.  chown /var/lib/nfs to choose different user
rpc.statd: Waiting for client connections
----> all services started normally

==================================================================
      SERVER STARTUP COMPLETE
==================================================================
----> list of enabled NFS protocol versions: 4.2, 4.1, 4, 3
----> list of container exports:
---->   /media/shared   *(rw,sync,wdelay,hide,nocrossmnt,secure,root_squash,no_all_squash,no_subtree_check,secure_locks,acl,no_pnfs,fsid=0,anonuid=65534,anongid=65534,sec=sys,rw,secure,root_squash,no_all_squash)
----> list of container ports that should be exposed:
---->   111 (TCP and UDP)
---->   2049 (TCP and UDP)
---->   32765 (TCP and UDP)
---->   32767 (TCP and UDP)

==================================================================
      READY AND WAITING FOR NFS CLIENT CONNECTIONS
==================================================================
rpc.statd: Waiting for client connections
rpc.statd: Waiting for client connections

from inside the container I got o/
(when running clear && netstat -tulnp | grep LISTEN && exportfs && cat /media/shared/test.txt && ps aux | grep rpc && showmount -e localhost)

tcp        0      0 0.0.0.0:2049            0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:39619           0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:32765           0.0.0.0:*               LISTEN      45/rpc.statd
tcp        0      0 0.0.0.0:32767           0.0.0.0:*               LISTEN      44/rpc.mountd
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      40/rpcbind
tcp        0      0 :::2049                 :::*                    LISTEN      -
tcp        0      0 :::32765                :::*                    LISTEN      45/rpc.statd
tcp        0      0 :::32767                :::*                    LISTEN      44/rpc.mountd
tcp        0      0 :::111                  :::*                    LISTEN      40/rpcbind
tcp        0      0 :::46741                :::*                    LISTEN      -
/media/shared   <world>
This text is inside the file.

   40 daemon    0:00 /sbin/rpcbind -s
   44 root      0:00 /usr/sbin/rpc.mountd --port 32767 --nfs-version 4.2 --no-nfs-version 2 --debug all
   45 root      0:00 /sbin/rpc.statd --no-notify --port 32765 --outgoing-port 32766 --no-syslog --foreground
   71 root      0:00 grep rpc
Export list for localhost:
/media/shared *

from outside I got

(base) ➜  ~ docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' nfs1
172.17.0.2
(base) ➜  ~ nc -zv localhost 50049
Connection to localhost (127.0.0.1) 50049 port [tcp/*] succeeded!
(base) ➜  ~ nc -zv localhost 50111
Connection to localhost (127.0.0.1) 50111 port [tcp/*] succeeded!
(base) ➜  ~ nc -zv localhost 52765
Connection to localhost (127.0.0.1) 52765 port [tcp/*] succeeded!
(base) ➜  ~ showmount -e 172.17.0.2
clnt_create: RPC: Timed out
(base) ➜  ~  sudo rm -rf /mnt/nfs && sudo mkdir -p /mnt/nfs && sudo chmod -R 777 /mnt/nfs && sudo chown -R user:user /mnt/nfs && sudo mount -t nfs -o vers=4,port=50049 172.17.0.2:/ /mnt/nfs

But the showmount and mount commands are still not working (where I also tried to use localhost).

I also tried this sudo mount -t nfs -o vers=3,port=50049,mountport=52765 172.17.0.2:/ /mnt/nfs

Would you have suggestions I could try? :)

[icicimov]

This is what I have setup to reproduce your case, in first console I start the NFS docker container with host network enabled:

$ sudo docker run --rm --name nfs --network host --privileged  \
-v /media/shared:/media/nfs -v /lib/modules:/lib/modules:ro \
-e NFS_EXPORT_0='/media *(ro,no_subtree_check,fsid=0,crossmnt)' \
-e NFS_EXPORT_1='/media/nfs *(rw,sync,no_subtree_check)' \
-e NFS_LOG_LEVEL='DEBUG' erichough/nfs-server

then in the second console:

$ grep nfs /proc/mounts
$ sudo mount -v -t nfs -o rw,vers=4.2,nolock localhost:/nfs /mnt/nfs
mount.nfs: timeout set for Fri Aug 23 00:57:28 2024
mount.nfs: trying text-based options 'vers=4.2,nolock,addr=127.0.0.1,clientaddr=127.0.0.1'

$ grep nfs /proc/mounts
localhost:/nfs /mnt/nfs nfs4 rw,relatime,vers=4.2,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=127.0.0.1,local_lock=none,addr=127.0.0.1 0 0

$ ls -l /mnt/nfs/
total 4
-rw-r--r-- 1 root root 26 Aug 22 01:28 test.txt

$ cat /mnt/nfs/test.txt 
This is a NFS shared file
$

$ sudo nfsstat -rs4
Server rpc stats:
calls      badcalls   badfmt     badauth    badclnt
4359       0          0          0          0       

$ sudo nfsstat -rc4
Client rpc stats:
calls      retrans    authrefrsh
4361       0          4361    

As you can see it's working fine here on:

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 20.04.6 LTS
Release:	20.04
Codename:	focal

$ sudo docker -v
Docker version 20.10.7, build f0df350

Do note however that the showmount both on the host and in the container does not work for me as well although the mount was successful:

$ showmount -e
clnt_create: RPC: Program not registered

I don't know how to fix this and have no time to check the /usr/local/bin/entrypoint.sh script, sorry. Probably something to do with the fact that there isn't any real init/service manager inside the container.

bash-5.0# rpcinfo -p localhost
   program vers proto   port  service
    100000    4   tcp    111
    100000    3   tcp    111
    100000    2   tcp    111
    100000    4   udp    111
    100000    3   udp    111
    100000    2   udp    111
    100024    1   udp   2686
    100024    1   tcp  22519
bash-5.0# 

Maybe the maintainer can help with this.

UPDATE: I thought it does not work because showmount is v3 command and the nfsd is started with v4.2 hence the error. But that is not the case:

bash-5.0# cat /proc/fs/nfsd/versions
-2 +3 +4 +4.1 +4.2

So, to make showmount work we just need to start the server in v3 mode by adding -e NFS_VERSION='3' to the docker server startup command and:

$ showmount -e localhost
Export list for localhost:
/media/nfs *
/media     *

END UPDATE

This is what's running inside the container for reference:

bash-5.0# ps aux
PID   USER     TIME  COMMAND
    1 root      0:00 bash /usr/local/bin/entrypoint.sh
   41 daemon    0:00 /sbin/rpcbind -s
   44 root      0:00 /usr/sbin/rpc.mountd --port 32767 --nfs-version 4.2 --no-nfs-version 2 --debug all
   61 root      0:00 sleep 2073600
   62 root      0:00 bash
   75 root      0:00 ps aux
bash-5.0#

and the host:

$ ps aux | grep -E "nfs|rpc"
root         308  0.0  0.0      0     0 ?        I<   Aug02   0:00 [rpciod]
root     3234483  0.0  0.0      0     0 ?        I<   Aug22   0:00 [nfsiod]
statd    3243432  0.0  0.7  58112 29136 ?        Ss   Aug22   0:00 /sbin/rpc.statd --no-notify
root     3403816  0.0  0.1   9376  4692 pts/0    S+   01:38   0:00 sudo docker run --rm --name nfs --network host -v /media/shared:/media/nfs -v /lib/modules:/lib/modules:ro -e NFS_EXPORT_0=/media *(ro,no_subtree_check,fsid=0,crossmnt) -e NFS_EXPORT_1=/media/nfs *(rw,sync,no_subtree_check) -e NFS_LOG_LEVEL=DEBUG --privileged -P erichough/nfs-server
root     3403817  0.0  1.4 829896 57488 pts/0    Sl+  01:38   0:00 docker run --rm --name nfs --network host -v /media/shared:/media/nfs -v /lib/modules:/lib/modules:ro -e NFS_EXPORT_0=/media *(ro,no_subtree_check,fsid=0,crossmnt) -e NFS_EXPORT_1=/media/nfs *(rw,sync,no_subtree_check) -e NFS_LOG_LEVEL=DEBUG --privileged -P erichough/nfs-server
bin      3403921  0.0  0.0  10604   176 ?        S    01:38   0:00 /sbin/rpcbind -s
root     3403924  0.0  1.3  57184 55364 ?        Ss   01:38   0:00 /usr/sbin/rpc.mountd --port 32767 --nfs-version 4.2 --no-nfs-version 2 --debug all
_rpc     3403930  0.0  0.0   7100  3696 ?        Ss   01:38   0:00 /sbin/rpcbind -f -w
root     3403931  0.0  0.0      0     0 ?        S    01:38   0:00 [nfsd]
root     3403932  0.0  0.0      0     0 ?        S    01:38   0:00 [nfsd]
root     3403952  0.0  0.1   9380  4672 pts/2    S+   01:38   0:00 sudo docker exec -it nfs bash
root     3403953  0.0  1.3 757572 56132 pts/2    Sl+  01:38   0:00 docker exec -it nfs bash