Once you have pushed configuration JSON to etcd
, you can start flanneld
. If you published your config at the default location, you can start flanneld
with no arguments.
Flannel will acquire a subnet lease, configure its routes based on other leases in the overlay network and start routing packets.
It will also monitor etcd
for new members of the network and adjust the routes accordingly.
After flannel has acquired the subnet and configured backend, it will write out an environment variable file (/run/flannel/subnet.env
by default) with subnet address and MTU that it supports.
For more information on checking the IP range for a specific host, see Leases and Reservations.
Flanneld does not support running multiple networks from a single daemon (it did previously as an experimental feature).
However, it does support running multiple daemons on the same host with different configurations. The -subnet-file
and -etcd-prefix
options should be used to "namespace" the different daemons.
For example
flanneld -subnet-file /vxlan.env -etcd-prefix=/vxlan/network
- Download a
flannel
binary.
wget https://github.com/flannel-io/flannel/releases/download/v0.19.0/flanneld-amd64 && chmod +x flanneld-amd64
- Run the binary.
sudo ./flanneld-amd64 # it will hang waiting to talk to etcd
- Run
etcd
. Follow the instructions on the etcd page, or, if you have docker just do
docker run --rm --net=host quay.io/coreos/etcd
- Observe that
flannel
can now talk toetcd
, but can't find any config. So write some config. Either getetcdctl
from the etcd page, or usedocker
again.
docker run --rm --net=host quay.io/coreos/etcd etcdctl set /coreos.com/network/config '{ "Network": "10.5.0.0/16", "Backend": {"Type": "vxlan"}}'
Now flannel
is running, it has created a VXLAN tunnel device on the host and written a subnet config file
cat /run/flannel/subnet.env
FLANNEL_NETWORK=10.5.0.0/16
FLANNEL_SUBNET=10.5.72.1/24
FLANNEL_MTU=1450
FLANNEL_IPMASQ=false
Each time flannel is restarted, it will attempt to access the FLANNEL_SUBNET
value written in this subnet config file. This prevents each host from needing to update its network information in case a host is unable to renew its lease before it expires (e.g. a host was restarting during the time flannel would normally renew its lease).
The FLANNEL_SUBNET
value is also only used if it is valid for the etcd network config. For instance, a FLANNEL_SUBNET
value of 10.5.72.1/24
will not be used if the etcd network value is set to 10.6.0.0/16
since it is not within that network range.
Subnet config value is 10.5.72.1/24
cat /run/flannel/subnet.env
FLANNEL_NETWORK=10.5.0.0/16
FLANNEL_SUBNET=10.5.72.1/24
FLANNEL_MTU=1450
FLANNEL_IPMASQ=false
etcd network value is 10.6.0.0/16
. Since 10.5.72.1/24
is outside of this network, a new lease will be allocated.
etcdctl get /coreos.com/network/config
{ "Network": "10.6.0.0/16", "Backend": {"Type": "vxlan"}}
Flannel uses the interface selected to register itself in the datastore.
The important options are:
-iface string
: Interface to use (IP or name) for inter-host communication.-public-ip string
: IP accessible by other nodes for inter-host communication.
The combination of the defaults, the autodetection and these two flags ultimately result in the following being determined:
- An interface (used for MTU detection and selecting the VTEP MAC in VXLAN).
- An IP address for that interface.
- A public IP that can be used for reaching this node. In
host-gw
it should match the interface address.
Please be aware of the following flannel runtime limitations.
- The datastore type cannot be changed.
- The backend type cannot be changed. (It can be changed if you stop all workloads and restart all flannel daemons.)
- You can change the subnetlen/subnetmin/subnetmax with a daemon restart. (Subnets can be changed with caution. If pods are already using IP addresses outside the new range they will stop working.)
- The clusterwide network range cannot be changed (without downtime).
Docker daemon accepts --bip
argument to configure the subnet of the docker0 bridge.
It also accepts --mtu
to set the MTU for docker0 and veth devices that it will be creating.
Because flannel writes out the acquired subnet and MTU values into a file, the script starting Docker can source in the values and pass them to Docker daemon:
source /run/flannel/subnet.env
docker daemon --bip=${FLANNEL_SUBNET} --mtu=${FLANNEL_MTU} &
Systemd users can use EnvironmentFile
directive in the .service
file to pull in /run/flannel/subnet.env
If you want to leave default docker0 network as it is and instead create a new network that will be using flannel you do so like this:
source /run/flannel/subnet.env
docker network create --attachable=true --subnet=${FLANNEL_SUBNET} -o "com.docker.network.driver.mtu"=${FLANNEL_MTU} flannel
Vagrant has a tendency to give the default interface (one with the default route) a non-unique IP (often 10.0.2.15).
This causes flannel to register multiple nodes with the same IP.
To work around this issue, use --iface
option to specify the interface that has a unique IP.
When running with a backend other than udp
, the kernel is providing the data path with flanneld
acting as the control plane.
As such, flanneld
can be restarted (even to do an upgrade) without disturbing existing flows.
However in the case of vxlan
backend, this needs to be done within a few seconds as ARP entries can start to timeout requiring the flannel daemon to refresh them.
Also, to avoid interruptions during restart, the configuration must not be changed (e.g. VNI, --iface values).