Windows Server 2022 failover or switchover with duplicate IP address

[thaala]

Failovers and switchovers using patronis etcdctl ends up with 80% failure because of duplicate IP in the network.
Windows (our case: 3nodesWindows Server 2022, 2nodesPostgres) doesnt reactivate or test the additional IP again after such conflict has been detected.

with an ipconfig /all the standard IP4 status is a (Preferred) state. In case of such error we got a (Duplicate) status instead

The failover stucks until removing this address by powershell command. After removing address vip-manager add it again and failover succeeds.

We found a workaround for the moment. Its forever running task on both postgres servers which looks every 5 seconds for a (duplicate) state of the desired interface and if occurs remove the address....

Powershell:
Remove-NetIPAddress -Confirm:$false -InterfaceAlias yourinterfacename -AddressState Duplicate

A better way could be to add such verfiy command short time after adding the IP to the interface instead doing this ever 5 seconds inside the vip-service. If duplicate state happens the ip can be removed and added again until state is (Preferred) or amount of try ends up with a permanent fail...

Thank you for this software.
BR Thilo

[MarkIITech]

We have been running into issues ever since we updated python from 3.9 to 3.11+. The vip-manager seems to move too quickly and the server sense duplicate IP Addresses on a administrative switchover. We've found that by restarting the vip-manager service on the destination leader, the contention resolves. It is somewhat intermittent, but completely destroys the High-Availability of the ETCD/PGSQL Cluster since we don't know if it will be successful during a true host failure. We've disabled Windows built in Duplicate IP Detection on both Cluster Nodes to see if that would resolve the issue. It became slightly better, Windows stopped popping up regarding duplicate IP Addresses, but we still find that the switchover is unsuccessful 80+% of the time. Any thoughts on how to improve this system or to replace vip-manager with another solution?

[TakashiArakane]

Would disabling Gratuitous ARP in the registry settings help? (Although the pros and cons of disabling it may need to be considered separately.)

Registry key: HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters
Value name: ArpRetryCount
Value type: REG_DWORD
Data: 0x0

Under Windows 2022 environment, the same duplicate detection has been improved as described above.

[MarkIITech]

We did disable the ArpRetryCount on our two nodes and it eliminated the Windows pop-up notification regarding duplicate IP address. We still have a seemingly random failure of vip-manager to switchover. A restart of the vip-manager on the destination leader corrects the problem. We monitor by pinging the virtual IP address and when it drops more than one packet, we know that the switchover has failed. I'd be happy to provide more information if it helps resolve this bug. Regards-

[thaala]

Hello,

removing the Alarm is not the right way to solve this problem because i think that there are really duplicate IP's in the network. Next Eth-Switch must learn that the ip is no more active on this specific port. This may take some time.

I think it would be best to change to a three state switchover instead of a hard on/off procedure. The trigger key in the parameter database
(etc3) should also have an "intermediate" trigger-value which should be shown for a short while. This will give every vip-node a chance to remove the virtual IP. After changing this trigger-key after a little timeout to a real node id - the network should be ready to accept the ip without problems.

BR Thilo

[MarkIITech]

Just to confirm - It looks like last week's update should have corrected the timing issues we were facing?
Thank you in advance-

[pashagolub]

We hope so, yes

[thaala]

@pashagolub: Do you plan to update the vip-manager code for checking of duplicate ip addresses? The windows powershell-workaround seems to work well but i think to do same procedure inside your code would be an enhancement. I see that you're using the iphlpapi.lib. But i dont know how to query the status of an interface figuring out whether the ip is duplicated or not.
If you can implement this check and remove the address again if this problem happened - followed by a next try after short time delay can have a successful IP address add. I am unsure whether this is a "windows only" problem but i didnt think so....

BTW. have a lot of thanks for this software. After long investigation i found that patroni solution is "the only" postgres standby database solution which is functioning well....
BR Thilo

[pashagolub]

@thaala PRs are welcome! We're, of course, interested in improving our software! :)

[thaala]

Hi,

in meanwhile other problem occurres. The Interface where the virtual ip added was identified by the name of the interface. But after some months of 24/7 working the name becomes inaccessible - this may related to the x million calls of the Powershell:
Remove-NetIPAddress - command. This problem can be solved by change the name of the interface and then change it back. Strange....

However this was a workaround for the workaround - its one workaround too much....
i have edited the code by adding two features:

1. Adding a parameter which superseeds the "Interfacename" parameter!
   If Parameter "InterfaceIndex" was set to a interface number it will be used to get the right interface with net.InterfaceByIndex()
   instead of net.InterfaceByName().

2. The ipmanager is expanded by a function "isdup()" which calls window function to evaluate the member
   "DadState" == "Duplicated (2)
   if state "Duplicated" is happened after the virtual ip was added, the virtual ip will be removed again (in 1 second slices)!
   The Test after adding the address happened every second - for max 6 seconds. If not seen "DadState" == "Duplicated"
   this adding is ended successfully. This adding/removing procedure is limited to maximum of 7 retries.....
   if the ip remains "Duplicated" after 7'th try the virtual ip is leaved alone...

Next week we will going to test this software on our prduction system. Unfortunately the function "isdup()" is a dummy for linux systems because my skills on linux programming are nonexistent... This nonexistent skills are also valid for go programming language. (only c#, c++, c)
if you want the code in the mainstream you may advice me how to give it to you - maybe you can make is a bit "smoother".

[thaala]

Dear Pashagolub,

since nearly a year ago i have tried to find a workaround for windows server "duplicate ip" messages.
My solution seems working well. But you maybe need do adjust some things by yourself.
The idea was to detect such situation and remove the IP again and retry it litte bit later again some times.
I will post my "little" changes here - i hope you'll find a way to integrate it onto your mainstream.

ip manager.go:

type ipConfigurer interface {
	queryAddress() bool
	configureAddress() bool
	deconfigureAddress() bool
	getCIDR() string
	cleanupArp()
	isdup() (bool, error)
}

// IPManager implements the main functionality of the VIP manager
type IPManager struct {
	configurer ipConfigurer

	states       <-chan bool
	currentState bool
	dupchecktry  int
	dupcnt       int
	stateLock    sync.Mutex
	recheck      *sync.Cond
}

// NewIPManager returns a new instance of IPManager
func NewIPManager(hostingType string, config *IPConfiguration, states <-chan bool, verbose bool) (m *IPManager, err error) {
	m = &IPManager{
		states:       states,
		currentState: false,
		dupchecktry:  0,
		dupcnt:       0,
	}
	m.recheck = sync.NewCond(&m.stateLock)
	switch hostingType {
	case "hetzner":
		m.configurer, err = newHetznerConfigurer(config, verbose)
		if err != nil {
			return nil, err
		}
	case "basic":
		fallthrough
	default:
		m.configurer, err = newBasicConfigurer(config)
	}
	if err != nil {
		m = nil
	}
	return
}

func (m *IPManager) applyLoop(ctx context.Context) {
	timeout := 0
	for {
		// Check if we should exit
		select {
		case <-ctx.Done():
			m.configurer.deconfigureAddress()
			return
		case <-time.After(time.Duration(timeout) * time.Second):
			actualState := m.configurer.queryAddress()
			m.stateLock.Lock()
			desiredState := m.currentState
			log.Printf("IP address %s state is %t, desired %t", m.configurer.getCIDR(), actualState, desiredState)
			if actualState != desiredState {
				m.stateLock.Unlock()
				var configureState bool
				if desiredState {
					configureState = m.configurer.configureAddress()
					m.dupcnt = retry_seconds + 1
				} else {
					configureState = m.configurer.deconfigureAddress()
					m.dupcnt = 0
					m.dupchecktry = 0
				}

				if !configureState {
					log.Printf("Error while acquiring virtual ip for this machine")
					//Sleep a little bit to avoid busy waiting due to the for loop.
					timeout = 10
					m.dupcnt = 0
				} else {
					if m.dupcnt > 0 {
						timeout = 1
					} else {
						timeout = 0
					}
				}
			} else {
				timeout = 10
				if m.dupcnt > 0 {
					m.dupcnt -= 1
					isb, _ := m.configurer.isdup()
					if isb {
						m.dupchecktry += 1
						if m.dupchecktry <= maxtry_dupcheck {
							log.Printf("Duplicatecheck of ip address %s positve - remove it again, %d retrys left", m.configurer.getCIDR(), maxtry_dupcheck-m.dupchecktry)
							if !m.configurer.deconfigureAddress() {
								log.Printf("Error while removing virtual ip for this machine")
								m.dupcnt = 0
							}
							timeout = 1
						} else {
							log.Printf("Fatal: giving up... duplicate state of ip %s cannot be removed - no more retrys left", m.configurer.getCIDR())
						}
					} else {
						//						log.Printf("Duplicatecheck of ip address %s negative, %d tests will follow", m.configurer.getCIDR(), m.dupcnt)
						timeout = 1
					}
				} else {
					timeout = 0
					// Wait for notification
					m.recheck.Wait()
				}

				// Want to query actual state anyway, so unlock
				m.stateLock.Unlock()
			}
		}
	}
}

// SyncStates implements states synchronization
func (m *IPManager) SyncStates(ctx context.Context, states <-chan bool) {
	ticker := time.NewTicker(10 * time.Second)

	var wg sync.WaitGroup
	wg.Add(1)
	go func() {
		m.applyLoop(ctx)
		wg.Done()
	}()

	for {
		select {
		case newState := <-states:
			m.stateLock.Lock()
			if m.currentState != newState {
				m.currentState = newState
				m.recheck.Broadcast()
			}
			m.stateLock.Unlock()
		case <-ticker.C:
			m.recheck.Broadcast()
		case <-ctx.Done():
			m.recheck.Broadcast()
			wg.Wait()
			m.configurer.cleanupArp()
			return
		}
	}
}

basic_configurere_windows.go:

func (c *BasicConfigurer) isdup() (bool, error) {
	var b []byte
	l := uint32(15000) // recommended initial size
	for {
		b = make([]byte, l)
		err := windows.GetAdaptersAddresses(syscall.AF_INET, windows.GAA_FLAG_INCLUDE_PREFIX, 0, (*windows.IpAdapterAddresses)(unsafe.Pointer(&b[0])), &l)
		if err == nil {
			if l == 0 {
				return false, nil
			}
			break
		}
		if err.(syscall.Errno) != syscall.ERROR_BUFFER_OVERFLOW {
			return false, os.NewSyscallError("getadaptersaddresses", err)
		}
		if l <= uint32(len(b)) {
			return false, os.NewSyscallError("getadaptersaddresses", err)
		}
	}
	ipgiven := c.VIP.AsSlice()
	for aa := (*windows.IpAdapterAddresses)(unsafe.Pointer(&b[0])); aa != nil; aa = aa.Next {

		for unicast := aa.FirstUnicastAddress; unicast != nil; unicast = unicast.Next {
			sa := *unicast.Address.Sockaddr
			if sa.Addr.Family == syscall.AF_INET {
				ipcomp := []byte{(byte)(sa.Addr.Data[2]), (byte)(sa.Addr.Data[3]), (byte)(sa.Addr.Data[4]), (byte)(sa.Addr.Data[5])}

				if bytes.Equal(ipcomp, ipgiven) {
					//					log.Printf("Found IPV4 Address: %d.%d.%d.%d, Dadstate = %d", ipcomp[0], ipcomp[1], ipcomp[2], ipcomp[3], unicast.DadState)

					if unicast.DadState == IpDadStateDuplicate {
						return true, nil
					}
					break
				}
			}
		}
	}
	return false, nil
}

// configureAddress assigns virtual IP address
func (c *BasicConfigurer) configureAddress() bool {
	var (
		ip     = binary.LittleEndian.Uint32(c.VIP.AsSlice())
		mask   = binary.LittleEndian.Uint32(c.Netmask)
		dummy1 uint32
		dummy2 uint32
	)
	log.Printf("Configuring address %s on %s", c.getCIDR(), c.Iface.Name)
	err := iphlpapi.AddIPAddress(ip, mask, uint32(c.hwIdx), &dummy1, &dummy2)
	if err != nil {
		log.Printf("AddIpAddress() Got error: %v", err)
		return false
	}
	// For now it is save to say that also working even if a
	// gratuitous arp message could not be send but logging an
	// errror should be enough.
	//_ = c.ARPSendGratuitous()
	return true
}

// deconfigureAddress drops virtual IP address
func (c *BasicConfigurer) deconfigureAddress() bool {
	var (
		ip = binary.LittleEndian.Uint32(c.VIP.AsSlice())
	)

	log.Printf("Removing address %s on %s", c.getCIDR(), c.Iface.Name)

	err := iphlpapi.DeleteIPAddress(ip)
	if err != nil {
		log.Printf("Remove Address - Got error: %v", err)
		return false
	}
	return true
}

Because my skills with GO-Language are limited - there is one thing i need to correct but i can't find it:
Every second there is a logging (also if erverything went well)
i think you are much better in such things than me...

I have also placed same extra conf - Parameter into "config.go" but i am sure you can add this by yourself also better than me..
....
RetryAfter int mapstructure:"retry-after" //milliseconds
RetryNum int mapstructure:"retry-num"
.....

P.S. i found that things related to "ntecontext" was unneccessary. The binary information inside this data was an ip - address representation only and wasn't needed.

BR
Thilo

[thaala]

Some corrections here...

This was a mistake.
....
RetryAfter int mapstructure:"retry-after" //milliseconds
RetryNum int mapstructure:"retry-num"
.....

instead i have added the ability to define the interface by index isntead of name.
gonfig.go:

type Config struct {
	IP    string `mapstructure:"ip"`
	Mask  int    `mapstructure:"netmask"`
	Iface string `mapstructure:"interface"`
-->	IfIdx string `mapstructure:"interfaceindex"`

func defineFlags() {
	// When adding new flags here, consider adding them to the Config struct above
	// and then make sure to insert them into the conf instance in NewConfig down below.
	pflag.String("config", "", "Location of the configuration file.")
	pflag.Bool("version", false, "Show the version number.")

	pflag.String("ip", "", "Virtual IP address to configure.")
	pflag.String("netmask", "", "The netmask used for the IP address. Defaults to -1 which assigns ipv4 default mask.")
	pflag.String("interface", "", "Network interface to configure on .")
-->	pflag.String("interfaceindex", "", "Network interface-Index to configure on. Takes preceedence over interface .")

basic_configurer.go:

package ipmanager

import (
	"errors"
	"net"
	"strings"

	arp "github.com/mdlayher/arp"
)

// BasicConfigurer can be used to enable vip-management on nodes
// that handle their own network connection, in setups where it is
// sufficient to add the virtual ip using `ip addr add ...` .
// After adding the virtual ip to the specified interface,
// a gratuitous ARP package is sent out to update the tables of
// nearby routers and other devices.
type BasicConfigurer struct {
	*IPConfiguration
	arpClient *arp.Client
	hwIdx     int //used by Windows to delete IP address
}

func newBasicConfigurer(config *IPConfiguration) (*BasicConfigurer, error) {
	c := &BasicConfigurer{IPConfiguration: config, hwIdx: config.Iface.Index}
	if c.Iface.HardwareAddr == nil || c.Iface.HardwareAddr.String() == "00:00:00:00:00:00" {
		return nil, errors.New(`Cannot run vip-manager on the loopback device
as its hardware address is the local address (00:00:00:00:00:00),
which prohibits sending of gratuitous ARP messages`)
	}
	return c, nil
}

// queryAddress returns if the address is assigned
func (c *BasicConfigurer) queryAddress() bool {
	//	iface, err := net.InterfaceByName(c.Iface.Name)
-->	iface, err := net.InterfaceByIndex(c.Iface.Index)
	if err != nil {
		return false
	}
	addresses, err := iface.Addrs()
	if err != nil {
		return false
	}
	for _, address := range addresses {
		if strings.Contains(address.String(), c.getCIDR()) {
			return true
		}
	}
	return false
}

func (c *BasicConfigurer) cleanupArp() {
	if c.arpClient != nil {
		c.arpClient.Close()
	}
}

it avoids relation to names in windows network-conf. Index is more reliable and changes only if that interface was removed.

[thaala]

Hello all,

in meanwhile i have merged my changes for functioning windows switchover here...
modified_files.zip

There are some points where i found problems:

1. Sending gratuitous arp throws erorrs when trying it on a windows teamed connection (aka link aggregated connection) in hyper-v environment. To avoid this errors i have added a boolean variable in configuration to switch it off by default. Windows os handle the gratuitous arp sending by itself after changes on ip interface.
2. Added abilitiy to give an interface index (type "route print" to get it) additional to interfacename. Using index is more reliable under windows, using interface names aren't.
3. Adding parameters and code to check whether the floating ip address is duplictaed in the network - if yes - (ipconfig /all tells: interface has dadstate = 2) on that interface - the ip will be removed and re-added after a wait timeout again up to parameter "max retries" amount. Duplicated addresses cannot be avoided due to switchover design of patroni, but typically the problem disappears after first or second retry. if this correction is not done a reliable switchover cannot be guaranteed because the dadstate = 2 marking stays static and the address becomes not valid.
4. colorful logging to console is fine per default but same control chars inside the log - files are messy. (i havent solved it)

Thank you and your company for that software.

i hope this can help the sparely windows system users using patroni.

BR
Thilo

[pashagolub]

Thanks for your work. We are working with issues and pull requests here. We don't have possibility to use any other kind of submissions, e.g. .zip, .patch, etc.

Thanks for understanding!

[pashagolub]

#290 provides new cross-platform implementation of Gratuitous ARP. So I mark this issue as fixed.

[thaala]

No Problem.
typical the license forces me to make changes public.
You have moved this issue to discussions by yourself. I have tired a pull request but this seems not to work for me..

BR
Thilo