Introduces option etcd/reduce_listing_quorum helpful for maintenance …

…during Disaster Recovery

README.md

@@ -6,6 +6,32 @@ It can be used with or without ETCD.
 
 Only ETCD APIv2 now supported.
 
+- [Config](#config)
+  - [Status](#status)
+  - [Installation](#installation)
+  - [Configuration](#configuration)
+    - [Example of `conf.lua`](#example-of-conflua)
+    - [Usage in `init.lua`](#usage-in-initlua)
+  - [Usage](#usage)
+  - [Topologies](#topologies)
+    - [Single-shard topology](#single-shard-topology)
+      - [Example of `init.lua`](#example-of-initlua)
+      - [Example of `/etc/userdb/conf.lua`](#example-of-etcuserdbconflua)
+      - [Example of ETCD configuration (`etcd.cluster.master`)](#example-of-etcd-configuration-etcdclustermaster)
+        - [Configuration precedence](#configuration-precedence)
+      - [Fencing configuration](#fencing-configuration)
+      - [Fencing algorithm](#fencing-algorithm)
+    - [Operations during disaster](#operations-during-disaster)
+    - [Multi-proxy topology (etcd.instance.single)](#multi-proxy-topology-etcdinstancesingle)
+      - [Example of proxy `init.lua`](#example-of-proxy-initlua)
+      - [Example of `/etc/proxy/conf.lua`](#example-of-etcproxyconflua)
+      - [Example of ETCD configuration (`etcd.instance.single`)](#example-of-etcd-configuration-etcdinstancesingle)
+    - [Multi-shard topology for custom sharding (`etcd.cluster.master`)](#multi-shard-topology-for-custom-sharding-etcdclustermaster)
+    - [Multi-shard topology for vshard-based applications (`etcd.cluster.vshard`)](#multi-shard-topology-for-vshard-based-applications-etcdclustervshard)
+      - [Example of ETCD configuration for vshard-based applications (`etcd.cluster.vshard`)](#example-of-etcd-configuration-for-vshard-based-applications-etcdclustervshard)
+      - [Example of vshard-based init.lua (`etcd.cluster.vshard`)](#example-of-vshard-based-initlua-etcdclustervshard)
+      - [VShard Maintenance](#vshard-maintenance)
+
 ## Status
 
 Ready for production use.
@@ -273,6 +299,28 @@ Fencing algorithm is the following:
 
 **Note:** to request ETCD Quorum Reads are used. So it is safe to use it in split brain.
 
+### Operations during disaster
+
+When ETCD failed to determine leader cluster can't reload it's configuration (or restart).
+
+During disaster, you must recovery ETCD elections mechanism first.
+
+If it is not possible, and you need to reload configuration or restart instance you may switch to manual configuration mode.
+
+To do that set `reduce_listing_quorum` to `true` in `conf.lua` config.
+
+After ETCD becames operation you **must** remove this option from local configuration.
+
+```lua
+etcd = {
+  reduce_listing_quorum = true,
+}
+```
+
+Enabling this option automatically disables `fencing`.
+
+Be extra careful to not allow Tarantool Master-Master setup. (Since ETCD may have different configuration on it's replicas).
+
 ### Multi-proxy topology (etcd.instance.single)
 
 `moonlibs/config` supports multi proxy topology. This topology is usefull when you need to have many stateless tarantool proxies or totally independent masters.

config.lua

@@ -928,6 +928,7 @@ local M
 				and config.get('etcd.fencing_enabled')
 				and msp == 'etcd.cluster.master'
 				and type(cfg.cluster) == 'string' and cfg.cluster ~= ''
+				and config.get('etcd.reduce_listing_quorum') ~= true
 			then
 				M._fencing_f = fiber.create(function()
 					fiber.name('config/fencing')

config/etcd.lua

@@ -37,6 +37,9 @@ function M.new(mod,options)
 	self.boolean_auto = options.boolean_auto
 	self.print_config = options.print_config
 	self.discover_endpoints = options.discover_endpoints == nil and true or options.discover_endpoints
+	if options.reduce_listing_quorum == true then
+		self.reduce_listing_quorum = true
+	end
 	if options.login then
 		self.authorization = "Basic "..digest.base64_encode(options.login..":"..(options.password or ""))
 		self.headers = { authorization = self.authorization }
@@ -179,7 +182,7 @@ function M:recursive_extract(cut, node, storage)
 end
 
 function M:list(keyspath)
-	local res, response = self:request("GET","keys"..keyspath, { recursive = true, quorum = true })
+	local res, response = self:request("GET","keys"..keyspath, { recursive = true, quorum = not self.reduce_listing_quorum })
 	-- print(yaml.encode(res))
 	if res.node then
 		local result = self:recursive_extract(keyspath,res.node)