Add support for testing against multiple FDB clusters #3575

ScottDugas · 2025-09-08T19:31:22Z

This changes the way we specify cluster files in tests. It replaces the fdb-environment.properties with fdb-environment.yaml. The new format allows for specifying multiple cluster files.

The majority of tests now randomly chose a file at the start of the test from the list of the available ones.
The following sub-projects set the environment variable in the gradle task because getting the project under test to support custom cluster files seemed like a bigger task:

fdb-relational-cli
fdb-relational-jdbc

Our CI builds also, now, setup two FDB clusters, and create an fdb-environment.properties.

I added one new "test" to validate this all works. It connects to two different clusters. It assumes that there are at least two clusters. At some point in the future we will want to make that a stronger contract to ensure that a bug in our CI setup doesn't cause us to not run all the multi-cluster tests.

I created Draft PR: #3623 to ensure that everything still works if you don't specify fdb-environment.yaml
I also created this PR off the same version of main to confirm the test counts didn't change: #3621 Apparently we do merge main to do the tests so PR #3621 saw fewer tests than this branch when it was last run. I re-ran and they may be the same depending on whether new tests were merged. I validated some of the differences, and they look to be expected.

Resolves: #3574

This changes the way cluster files are specified for the tests, introducing a new yaml file. This makes it more amenable to having more than one cluster file. The fdb-environment.properties is still supported so developers don't immediately have to update their configuration.

This took a lot of trial and error in CI, and adding and removing a lot of logging and investigative code. The key thing that tripped me up was the fact that you have to do `configure new ssd` if it is a new cluster, which we were not doing previously, because the installation created the cluster.

This might be a bit of overkill, if they aren't using FDBDatabaseExtension, but it probably avoids confusion later on.

it was trying to connect to the default, which doesn't exist

…stCopyInSameDatabase

The reason for the issue is that it was reassigning `fdb = dbExtension.getDatabase()`, but it loaded data into `fdb` in a `beforeEach`, so if it picked different clusters it wouldn't have any data, and execute wouldn't fail. In debugging I also removed teh second `openRecordStore`, because the state should respond to calling `markIndexWriteOnly`, you shouldn't have to re-read, and we actually say you should never open the same store with different objects in the same transaction. I also isolated the exception check, and switched to use assertJ for the assertion, because it ends up being quite a bit cleaner than with junit's assertions.

I tried putting it in testFixtures, but that doesn't play well with other sub-projects already depending on the tests configuration

Doing so requires moving everything to testFixtures, so best to do in a separate PR

… it uses in setup

I think in the long term we don't want the driver to work like this, but I'm not sure how we want to have it work, and creating an api for that seems like more work than it is worth at this time.

…tions The external yaml connections do not support some operations that aren't really needed in mixed-mode, so that needs the cluster file from the connection.

…onally

I think people could be confused by getting different clusters each call to `getDatabase`, so this persists that. If you want to test against multiple clusters, use the indexed methods.

ScottDugas · 2025-09-22T21:15:43Z

...t/java/com/apple/foundationdb/record/provider/foundationdb/BlockingInAsyncDetectionTest.java

        factory.clear();

-        FDBDatabase database = factory.getDatabase();
+        FDBDatabase database = factory.getDatabase(FDBTestEnvironment.randomClusterFile());


This test could probably just use dbExtension.getDatabase(), but I didn't want to change too much in this PR. And you have to take care to make sure you're not reusing a database created in the setup.

ScottDugas · 2025-09-22T21:20:37Z

...src/test/java/com/apple/foundationdb/record/provider/foundationdb/indexes/RankIndexTest.java

-                throw e.getCause();
-            }
-        });
+    void writeOnlyRankQuery() throws Exception {


This test was failing, and reworking it to isolate the assertThrows helped me debug what was wrong, and I felt like it would be good to keep it.

ScottDugas · 2025-09-22T21:27:15Z

fdb-relational-core/fdb-relational-core.gradle

    testImplementation(libs.bundles.test.runtime)
    testCompileOnly(libs.bundles.test.compileOnly)

+    testImplementation project(path: ':fdb-extensions', configuration: 'tests')


I thought about putting the new class in a testFixtures, but I couldn't get that to work without putting all the classes that anyone already depends on via this approach in the testFixture, and thought that would be better as a followup.

Actually, I think it would be better to move everything to a testFixtures. I'll do that in a separate PR

ScottDugas · 2025-09-22T21:31:51Z

...lational-server/src/main/java/com/apple/foundationdb/relational/server/RelationalServer.java

        // TODO: Make this multi-query/-tenant/-database!
        try {
-            frl = new FRL();
+            frl = new FRL(com.apple.foundationdb.relational.api.Options.NONE, clusterFile);


We also have Options from apache cli, so this needs to be fully qualified (or statically imported)

ScottDugas · 2025-09-22T21:33:10Z

yaml-tests/src/main/java/com/apple/foundationdb/relational/yamltests/YamlConnection.java

    <T> T executeTransactionally(SQLFunction<YamlConnection, T> transactionalWork) throws SQLException, RelationalException;
+
+    /**
+     * The Cluster File that this connection is connected to, so that other connections can point to the same


I'm not super committed to this API, but at some point we'll want multi-cluster testing in yaml-tests, and I think that's a better time to really think about it.

ScottDugas · 2025-09-24T12:13:18Z

fdb-extensions/src/testFixtures/java/com/apple/foundationdb/test/FDBTestEnvironment.java

+import java.util.Map;
+import java.util.concurrent.ThreadLocalRandom;
+
+/**


I thought about putting this in testFixtures, but since other projects already depend on the tests configuration, doing so would require moving all the other classes into a testFixtures, and I figured that would be better as a separate PR.

I made a change in main to move them into a test fixture, and then went to test publishing of testFixtures so that yaml-tests could be used by clients for their tests, but it looks like we can't publish testFixtures: e9db5c0#diff-9515853bb3a0c957c759e9c453a9e17941f4333751715e75d74efed681fadac0R60 I'm going to look into creating a test-utils library....

fdb-extensions now has a testFixtures. Put the new FDBTestEnvironment there

- the fixtures need snakeyaml - I had a typo in testFixtures

ScottDugas added the testing improvement Change that improves our testing label Sep 8, 2025

ScottDugas added 26 commits September 22, 2025 14:38

Add support for testing specifically against multiple clusters

43aa388

Ensure FDB gets opened even if there are no cluster files

c26e200

Have fdb-extensions get cluster info from yaml too

672b305

Fix handling behavior when there is no cluster file specified

f4aea84

Throw RecordCoreException in FDBDatabaseExtepsion setup

11c900a

Fix up exception conversion

d15b81e

Add snakeyaml dependency to projects depending on core tests

e058377

This might be a bit of overkill, if they aren't using FDBDatabaseExtension, but it probably avoids confusion later on.

Fix sqlline test

8e84a0e

Use dbExtension.getDatabase in FDBRecordContextTest

71db28c

it was trying to connect to the default, which doesn't exist

Create database with cluster file in ResolverMappingReplicatorTest.te…

68fa77a

…stCopyInSameDatabase

Try creating an FDBTestClusterConfig in testFixtures

f418350

I tried putting it in testFixtures, but that doesn't play well with other sub-projects already depending on the tests configuration

Don't use fixtures yet

2b9af6d

Doing so requires moving everything to testFixtures, so best to do in a separate PR

Update some tests to get random cluster files from FDBTestEnvironment

66c5f5f

Update remaining tests to not use null clusterFile

9cb9f2c

Fix checkstyle/javadoc checks

134938e

Add option to start RelationalServer with a clusterFile & use that

64deb2b

LocatableResolverTest needs to use the same clusterFile in tests that…

e48f0f3

… it uses in setup

Fix last couple tests that wipe FDB

74b886e

Fix JDBC tests that depend on creating in process server

6a953b0

I think in the long term we don't want the driver to work like this, but I'm not sure how we want to have it work, and creating an api for that seems like more work than it is worth at this time.

Fixup InProcessRelationalServer to use FDBTestEnvironment

d1c1ee1

Update yaml-tests to use FDBTestEnvironment

fa20608

Have YamlConnection provide cluster file so we can apply direct opera…

44037f2

…tions The external yaml connections do not support some operations that aren't really needed in mixed-mode, so that needs the cluster file from the connection.

ExternalServerTest needs to provide a clusterFile

dae9a82

ScottDugas force-pushed the multiple-fdb-clusters branch from 8611ab5 to dae9a82 Compare September 22, 2025 18:44

ScottDugas changed the title ~~Multiple fdb clusters~~ Add support for testing against multiple FDB clusters Sep 22, 2025

Fixup comments in FDBTestEnvironment

ffec130

ScottDugas added 3 commits September 22, 2025 17:28

Change TestDatabaseExtension to pick a random database

4617fc7

Cleanup some cases where tests were using multiple clusters unintenti…

83f2657

…onally

Have FDBDatabaseExtension use a single cluster for the test

c9eeaa4

I think people could be confused by getting different clusters each call to `getDatabase`, so this persists that. If you want to test against multiple clusters, use the indexed methods.

ScottDugas commented Sep 22, 2025

View reviewed changes

ScottDugas added 2 commits September 23, 2025 15:10

Always provide a cluster file, and reset in afterEach

1e5552f

Properly handle null clusterFile

244c5f0

ScottDugas commented Sep 24, 2025

View reviewed changes

ScottDugas marked this pull request as ready for review September 24, 2025 12:13

ScottDugas requested a review from alecgrieser September 24, 2025 12:13

ScottDugas added 3 commits September 24, 2025 13:12

Merge branch 'main' into multiple-fdb-clusters

0b63c36

fdb-extensions now has a testFixtures. Put the new FDBTestEnvironment there

Fixup post merge

e4d4631

- the fixtures need snakeyaml - I had a typo in testFixtures

Fixup dependencies to use testFixtures.

252f879

ScottDugas mentioned this pull request Sep 24, 2025

Create a new fdb-test-utils sub-project for some test utilities #3631

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add support for testing against multiple FDB clusters #3575

Add support for testing against multiple FDB clusters #3575

Uh oh!

ScottDugas commented Sep 8, 2025 •

edited

Loading

Uh oh!

ScottDugas Sep 22, 2025

Uh oh!

ScottDugas Sep 22, 2025

Uh oh!

ScottDugas Sep 22, 2025

Uh oh!

ScottDugas Sep 24, 2025

Uh oh!

ScottDugas Sep 22, 2025

Uh oh!

ScottDugas Sep 22, 2025

Uh oh!

ScottDugas Sep 24, 2025

Uh oh!

ScottDugas Sep 24, 2025

Uh oh!

Uh oh!

Add support for testing against multiple FDB clusters #3575

Are you sure you want to change the base?

Add support for testing against multiple FDB clusters #3575

Uh oh!

Conversation

ScottDugas commented Sep 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ScottDugas commented Sep 8, 2025 •

edited

Loading