From 32528da4f09d26cfc6e19ffe78025d4038c69d51 Mon Sep 17 00:00:00 2001 From: Guilherme Salazar Date: Tue, 25 Jul 2023 18:20:57 -0300 Subject: [PATCH] fix(pluginservers): restart all instances of a plugin When the plugin -- which acts as a plugin server -- dies for some reason, all entities associated with it must be invalidated, not only the current. If two instances A and B of an external plugin P exist (e.g., if applied to different route objects), they receive a unique ID [1] defined by the the plugin server when the instance is started. If A and B are started sequentially and are the first instances, they get IDs 0 and 1, respectively. This ID is cached on the Kong side by the plugin server manager [2]. If plugin P dies, Kong plugin server manager will detect this condition and restart it. As it was, the code would remove whichever instance first detected the condition and leave the other in `running_instances`; once the plugin server was up, the removed instance would be restarted, in some cases with the same Id already in use by the stale instance that should have been removed from `running_instances`. Subsequent requests may then run the wrong instance of the external plugin. This change fixes the issue by ensuring that both instances A and B are removed from the `running_instances` table, [1] https://github.com/Kong/go-pdk/blob/4deec45a5bfe421a0ceebcbf7931d395250949b0/server/instance.go#L47 [2] `kong/runloop/plugin_servers/process.lua` --- CHANGELOG.md | 5 +++++ kong/runloop/plugin_servers/init.lua | 15 ++++++++++++++- 2 files changed, 19 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index df532a0c7031..47baba9e02e2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -103,6 +103,11 @@ - Make `kong vault get` CLI command work in dbless mode by injecting the necessary directives into the kong cli nginx.conf. [#11127](https://github.com/Kong/kong/pull/11127) [#11291](https://github.com/Kong/kong/pull/11291) +- Fix an issue where a crashing Go plugin server process would cause subsequent + requests proxied through Kong to execute Go plugins with inconsistent configurations. + The issue only affects scenarios where the same Go plugin is applied to different Route + or Service entities. + [#11306](https://github.com/Kong/kong/pull/11306) #### Admin API diff --git a/kong/runloop/plugin_servers/init.lua b/kong/runloop/plugin_servers/init.lua index c20fb389f30b..89418eb31547 100644 --- a/kong/runloop/plugin_servers/init.lua +++ b/kong/runloop/plugin_servers/init.lua @@ -270,7 +270,20 @@ end --- reset_instance: removes an instance from the table. function reset_instance(plugin_name, conf) local key = type(conf) == "table" and kong.plugin.get_id() or plugin_name - running_instances[key] = nil + local current_instance = running_instances[key] + + -- + -- the same plugin (which acts as a plugin server) is shared among + -- instances of the plugin; for example, the same plugin can be applied + -- to many routes + -- `reset_instance` is called when (but not only) the plugin server died; + -- in such case, all associated instances must be removed, not only the current + -- + for k, instance in pairs(running_instances) do + if instance.rpc == current_instance.rpc then + running_instances[k] = nil + end + end end