Move Lua scripting engine into a Valkey module #2858

rjd15372 · 2025-11-20T10:06:18Z

This PR restructures the Lua scripting functionality by extracting
it from the core Valkey server into a separate Valkey module. This change
enables the possibility of a backwards compatible Lua engine upgrade, as well
as, the flexibility in building Valkey without the Lua engine.

Important: from a user's point of view, there's no difference in using
the EVAL of FUNCTION/FCALL scripts. This PR is fully backward compatible
with respect to the public API.

The main code change is the move and adaptation of the Lua engine source
files from src/lua to src/modules/lua. The original Lua engine code is
adapted to use the module API to compile and execute scripts.

The main difference between the original code and the new, is the
serialization and deserialization of Valkey RESP values into, and from,
Lua values. While in the original implementation the parsing of RESP values
was done directly from the client buffer, in the new implementation the
parsing is done from the ValkeyModuleCallReply object and respective API.

The Makefile and CMake build systems were also updated to build and
integrate the new Lua engine module, within the Valkey server build
workflow.
When the Valkey server is built, the Lua engine module is also built,
and, the Lua module is loaded automatically by the server upon startup.
When running make install the Lua engine module is installed in the
default system library directory.
There's a new build option, called BUILD_LUA, that if set to no allows to
build Valkey server without building the Lua engine.

This modular architecture enables future development of additional Lua
engine modules with newer Lua versions that can be loaded alongside the
current engine, facilitating gradual migration paths for users.

rjd15372 · 2025-11-20T10:07:03Z

This PR is rebased on top of #2836 . When #2836 is merged, I'll update this PR.

This commit restructures the Lua scripting functionality by extracting it from the core Valkey server into a separate Valkey module. This change enables the possibility of a backwards compatible Lua engine upgrade, as well as, the flexibility in building Valkey without the Lua engine. **Important**: from a user's point of view, there's no difference in using the `EVAL` of `FUNCTION/FCALL` scripts. This PR is fully backward compatible with respect to the public API. The main code change is the move and adaptation of the Lua engine source files from `src/lua` to `src/modules/lua`. The original Lua engine code is adapted to use the module API to compile and execute scripts. The main difference between the original code and the new, is the serialization and deserialization of Valkey RESP values into, and from, Lua values. While in the original implementation the parsing of RESP values was done directly from the client buffer, in the new implementation the parsing is done from the `ValkeyModuleCallReply` object and respective API. The Makefile and CMake build systems were also updated to build and integrate the new Lua engine module, within the Valkey server build workflow. When the Valkey server is built, the Lua engine module is also built, and, the Lua module is loaded automatically by the server upon startup. When running `make install` the Lua engine module is installed in the default system library directory. There's a new build option, called `WITHOUT_LUA`, that if set allows to build Valkey server without building the Lua engine. This modular architecture enables future development of additional Lua engine modules with newer Lua versions that can be loaded alongside the current engine, facilitating gradual migration paths for users. Signed-off-by: Ricardo Dias <ricardo.dias@percona.com>

zuiderkwast · 2025-11-20T10:42:39Z

Wow!

Some initial comments:

For the build option, why not use the name BUILD_LUA (default yes), to align with the build options BUILD_TLS and BUILD_RDMA?
For backward compatibility, the lua module needs to load by default. To disable this auto-loading, we need a new config, don't we? I can't see it mentioned here.
Can the lua module be unloaded using MODULE UNLOAD?

rjd15372 · 2025-11-20T10:46:50Z

For the build option, why not use the name BUILD_LUA (default yes), to align with the build options BUILD_TLS and BUILD_RDMA?

Sure, that makes more sense. I'll update the PR.

For backward compatibility, the lua module needs to load by default. To disable this auto-loading, we need a new config, don't we? I can't see it mentioned here.

I was thinking that if the lua module is built then it would always be loaded by default. But we can add a new config to disable auto-loading even if the lua module is built.

Can the lua module be unloaded using MODULE UNLOAD?

Yes

zuiderkwast · 2025-11-20T11:29:47Z

I was thinking that if the lua module is built then it would always be loaded by default. But we can add a new config to disable auto-loading even if the lua module is built.

Yeah, we can discuss it with the core team.

I believe in many contexts, such as in pre-built containers, the module is already built. Many users don't build their own binaries.

rjd15372 · 2025-11-20T11:32:05Z

I think it makes sense to have that config option to disable auto-load.

Signed-off-by: Ricardo Dias <ricardo.dias@percona.com>

rjd15372 · 2025-11-20T14:39:15Z

I think it makes sense to have that config option to disable auto-load.

@zuiderkwast I vote for implementing such option in a follow up PR. This PR is already huge to review.

Signed-off-by: Ricardo Dias <ricardo.dias@percona.com>

codecov · 2025-11-21T11:58:38Z

Codecov Report

❌ Patch coverage is 92.59259% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 73.98%. Comparing base (05540af) to head (f36121f).
⚠️ Report is 1 commits behind head on unstable.

Files with missing lines	Patch %	Lines
src/server.c	83.33%	2 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##           unstable    #2858      +/-   ##
============================================
+ Coverage     72.41%   73.98%   +1.57%     
============================================
  Files           128      124       -4     
  Lines         70414    68643    -1771     
============================================
- Hits          50987    50784     -203     
+ Misses        19427    17859    -1568

Files with missing lines	Coverage Δ
src/config.c	`80.75% <100.00%> (+2.31%)`	⬆️
src/eval.c	`87.46% <100.00%> (ø)`
src/module.c	`26.57% <100.00%> (+16.79%)`	⬆️
src/replication.c	`86.04% <ø> (+0.10%)`	⬆️
src/script.c	`89.03% <ø> (+8.89%)`	⬆️
src/scripting_engine.c	`56.88% <100.00%> (+3.51%)`	⬆️
src/server.c	`88.65% <83.33%> (+0.19%)`	⬆️

... and 26 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Signed-off-by: Ricardo Dias <ricardo.dias@percona.com>

zuiderkwast

Just some initial comments. I'll do another pass later. The change is not as huge as the +/- numbers indicate. Many lines are moved to other files.

The build options should be mentioned in the README.md.

zuiderkwast · 2025-11-20T15:27:47Z

src/modules/lua/engine_lua.c

@@ -0,0 +1,542 @@
+#include "../../valkeymodule.h"


We should add this copyright header to new files:

Suggested change

#include "../../valkeymodule.h"

/*

* Copyright (c) Valkey Contributors

* All rights reserved.

* SPDX-License-Identifier: BSD-3-Clause

*/

#include "../../valkeymodule.h"

zuiderkwast · 2025-11-20T15:30:07Z

src/modules/lua/list.c

@@ -0,0 +1,72 @@
+#include "list.h"


Suggested change

#include "list.h"

/*

* Copyright (c) Valkey Contributors

* All rights reserved.

* SPDX-License-Identifier: BSD-3-Clause

*/

#include "list.h"

zuiderkwast · 2025-11-20T15:32:36Z

src/modules/lua/list.c

+} ListIter;
+
+List *list_create(void) {
+    List *list = ValkeyModule_Calloc(1, sizeof(List));


Not that it matters much, but calloc doesn't seem to be needed. We could use ValkeyModule_Alloc.

Suggested change

List *list = ValkeyModule_Calloc(1, sizeof(List));

List *list = ValkeyModule_Alloc(sizeof(List));

zuiderkwast · 2025-11-20T15:50:19Z

src/modules/lua/script_lua.c

+            if (b) {
+                ValkeyModule_ReplyWithLongLong(ctx, 1);
+            } else {
+                ValkeyModule_ReplyWithCString(ctx, NULL);


This logic is weird. The change is preserving the logic though. It looks correct.

I believe this is equivalent though:

Suggested change

ValkeyModule_ReplyWithCString(ctx, NULL);

ValkeyModule_ReplyWithNull(ctx);

zuiderkwast · 2025-11-20T15:59:00Z

src/modules/lua/script_lua.c

+            error_code = lm_strcpy("ERR");
+            final_msg = lm_asprintf("%s %s", error_code, msg);
+            ValkeyModule_Free(error_code);


Why allocate a temporary "ERR" string? Can't we simply:

Suggested change

error_code = lm_strcpy("ERR");

final_msg = lm_asprintf("%s %s", error_code, msg);

ValkeyModule_Free(error_code);

final_msg = lm_asprintf("ERR %s", msg);

zuiderkwast · 2025-11-20T16:01:52Z

src/modules/lua/script_lua.c

+char *lm_strcpy(const char *str) {
+    size_t len = strlen(str);
+    char *res = ValkeyModule_Alloc(len + 1);
+    strncpy(res, str, len + 1);


We already did strlen above so no need to search for the \0 again. We can use memcpy.

Suggested change

strncpy(res, str, len + 1);

memcpy(res, str, len + 1);

zuiderkwast · 2025-11-20T16:05:00Z

src/modules/lua/script_lua.c

+            error_code = lm_strcpy(err_buffer + 1);
+            final_msg = lm_asprintf("%s %s", error_code, msg);
+            ValkeyModule_Free(error_code);


We can avoid a temp allocation.

Suggested change

error_code = lm_strcpy(err_buffer + 1);

final_msg = lm_asprintf("%s %s", error_code, msg);

ValkeyModule_Free(error_code);

final_msg = lm_asprintf("%s %s", err_buffer + 1, msg);

rjd15372 requested review from madolson and zuiderkwast November 20, 2025 10:06

rjd15372 self-assigned this Nov 20, 2025

rjd15372 added the enhancement New feature or request label Nov 20, 2025

rjd15372 added this to Valkey 9.1 Nov 20, 2025

rjd15372 force-pushed the lua-module branch from 58f217a to 8f0ba6d Compare November 20, 2025 10:28

rjd15372 added 2 commits November 20, 2025 14:28

Fix test client flags

884fa12

Signed-off-by: Ricardo Dias <ricardo.dias@percona.com>

fix build issues and use BUILD_LUA option

18b8500

Signed-off-by: Ricardo Dias <ricardo.dias@percona.com>

rjd15372 force-pushed the lua-module branch from f40d718 to 948a554 Compare November 21, 2025 11:12

fix macos build

afbdbf5

Signed-off-by: Ricardo Dias <ricardo.dias@percona.com>

rjd15372 force-pushed the lua-module branch from 948a554 to afbdbf5 Compare November 21, 2025 11:39

fix 32bit build

f36121f

Signed-off-by: Ricardo Dias <ricardo.dias@percona.com>

zuiderkwast reviewed Nov 21, 2025

View reviewed changes

-#include "../../valkeymodule.h"
+/*
+ * Copyright (c) Valkey Contributors
+ * All rights reserved.
+ * SPDX-License-Identifier: BSD-3-Clause
+ */
+#include "../../valkeymodule.h"

	List *list = ValkeyModule_Calloc(1, sizeof(List));
	List *list = ValkeyModule_Alloc(sizeof(List));

	ValkeyModule_ReplyWithCString(ctx, NULL);
	ValkeyModule_ReplyWithNull(ctx);

Move Lua scripting engine into a Valkey module #2858

Are you sure you want to change the base?

Move Lua scripting engine into a Valkey module #2858

Conversation

rjd15372 commented Nov 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rjd15372 commented Nov 20, 2025

Uh oh!

zuiderkwast commented Nov 20, 2025

Uh oh!

rjd15372 commented Nov 20, 2025

Uh oh!

zuiderkwast commented Nov 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rjd15372 commented Nov 20, 2025

Uh oh!

rjd15372 commented Nov 20, 2025

Uh oh!

codecov bot commented Nov 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

zuiderkwast left a comment

Choose a reason for hiding this comment

Uh oh!

zuiderkwast Nov 20, 2025

Choose a reason for hiding this comment

Uh oh!

zuiderkwast Nov 20, 2025

Choose a reason for hiding this comment

Uh oh!

zuiderkwast Nov 20, 2025

Choose a reason for hiding this comment

Uh oh!

zuiderkwast Nov 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zuiderkwast Nov 20, 2025

Choose a reason for hiding this comment

Uh oh!

zuiderkwast Nov 20, 2025

Choose a reason for hiding this comment

Uh oh!

zuiderkwast Nov 20, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

rjd15372 commented Nov 20, 2025 •

edited

Loading

zuiderkwast commented Nov 20, 2025 •

edited

Loading

codecov bot commented Nov 21, 2025 •

edited

Loading

zuiderkwast Nov 20, 2025 •

edited

Loading