Add `collectionName/databaseName` attributes to MongoDB provider #3322

jesmith17 · 2024-12-21T23:53:15Z

Added support for defining the target collectionName and databaseName for the logs to the config file. This allows users to define the values via config instead of only configuring them via the connectionString.

…g file instead of just the connectionString

ppkarwasz · 2024-12-25T09:11:45Z

@jesmith17,

Thank you for your contribution.

While I am not a MongoDB user, the current behavior of using the connection string to infer the name of the database and collection does not make sense to me. Your proposal looks like a valid improvement. According to the MongoDB connection string documentation the connection string has the form:

mongodb://[username:password@]host1[:port1][,...hostN[:portN]][/[defaultauthdb][?options]]

where defaultauthdb is documented as:

The authentication database to use if the connection string includes username:password@ authentication credentials but the authSource option is unspecified.

Therefore:

defaultauthdb does not seem to be the right value to be used as database name to store the logs: I imagine that some MongoDB installation might use a different database for authentication and to store the data, right?
the collection name does not even appear in the official connection string. It only appears in the Java-specific ConnectionString class. I guess that many users don't know that syntax.

@garydgregory, what do you think?

ppkarwasz

@jesmith17,

This looks like a very valid improvement! 💯

The changes to the MongoDB provider should be documented. Can you:

Modify database.adoc and add a description of the databaseName and collectionName configuration attributes. IMHO these attributes should be marked as "required" in the documentation and the description should mention their fallback values (which are kept only for backward compatibility). The fallback values can be marked as deprecated and we might remove them in a future Log4j Core version.
Since IMHO providing the database and collection name explicitly seems like the right thing to do, all the examples and integration tests should be rewritten to use the databaseName and collectionName attributes.

log4j-mongodb/src/main/java/org/apache/logging/log4j/mongodb/MongoDbProvider.java

log4j-mongodb/src/test/java/org/apache/logging/log4j/mongodb/MongoDbCollectionNameIT.java

ppkarwasz · 2024-12-25T10:06:17Z

@jesmith17,

The integration tests in the log4j-mongodb module use Docker. If you want to run the tests locally, you need to have Docker available on your system and run:

./mvnw verify -pl log4j-mongodb -Pdocker

garydgregory · 2024-12-25T12:06:29Z

I'm not a fan because this creates confusion with the connection string. IMO: It should be that EITHER you provide a connection string OR you provide all the bits and pieces to build a connection string. I don't like the idea of doing it half one way and half the other.

IMO, we should follow the KISS principle and only provide connection string support.

Just like with JDBC, you just need to know a driver's connection string syntax, which invariably changes from time to time, and I certainly don't want to have to change this code because there is some Mongo driver change in some version that breaks our code.

ppkarwasz · 2024-12-25T12:31:07Z

@garydgregory,

I'm not a fan because this creates confusion with the connection string. IMO: It should be that EITHER you provide a connection string OR you provide all the bits and pieces to build a connection string. I don't like the idea of doing it half one way and half the other.

These were my initial thoughts too, until I dug deeper into the connection string.
From what I have understood from the connection string (mainly this documentation), it does not provide neither the database nor the collection name:

it does not contain the database name, because the defaultauthdb part is the name of the database, where the user is defined, not the database that will be used by the MongoDB client. The MongoDB client can use any database available on the server. Did I misinterpret this part?
the official connections string does not even have a "collectionName" part. The collection name is only defined by the Java-specific ConnectionString. So Java developers might know this alternative syntax, but MongoDB administrators might not know it.

As I mentioned before, I have almost no experience with MongoDB, so these interpretations might not make sense.

Just like with JDBC, you just need to know a driver's connection string syntax, which invariably changes from time to time, and I certainly don't want to have to change this code because there is some Mongo driver change in some version that breaks our code.

For JDBC you also need to provide the name of the table.

garydgregory · 2024-12-25T14:26:13Z

So the real problem is that the Mongo documentation could be better, like in any software.

I am concerned this opens the door for more and more options to be added to "override" the connection string components, making maintenance harder.

We should only, IMO, offer configuration hooks for behavior that can't be achieved otherwise. Let's review:

The database as part of the connection string is documented here (where Google took me):
https://www.mongodb.com/docs/manual/reference/connection-string-examples/#std-label-connections-connection-examples
For example. 2 examples I picked from the page:
Self-Hosted records Database Running Locally:

mongodb://myDatabaseUser:D1fficultP%40ssw0rd@localhost/records

and fancier:
MongoDB Atlas Cluster that Authenticates with AWS IAM credentials:

mongodb+srv://<aws access key id>:<aws secret access key>@cluster0.example.com/testdb?authSource=$external&authMechanism=MONGODB-AWS

The collection is specified as "databaseName.collectionName", see our XML configuration files.

The classic problem is that we do not want to create support tickets like "My appender connects to A instead of B, why?" where we then have to say "Oh, that's because you specified the foo in both the connection string and in the other thing and this takes precedence over that, blah blah".

For these use cases of double configuration, we should throw an exception.

Who would port this to log4j-mongodb4? to master?

vy · 2024-12-25T18:56:02Z

Agreeing with @garydgregory's sentiment that we should not add more configuration knobs, if the shared use case can be served with just the connection string. I can spare time for this further after the vacation (i.e., 2025-01-02), if not earlier.

jesmith17 · 2024-12-26T00:31:41Z

I agree that there is a slippery slope here, but specifying a table name for a db connection in the connection string is very very odd. Allowing the collection name via config instead of connection string puts this in line how the JDBC appender works which required a table name as part of the config. I would actually prefer to make this a config only part and have it override the connection string value. With the way the code currently functions you have to extract the db and collection name from the connection string to pass them to other code blocks anyway. But to preserve backwards compatibility I left it in.

ppkarwasz · 2024-12-27T06:16:10Z

I looked into the Spring Boot configuration properties for MongoDB:

Spring Boot offers a set of properties (username, password, host, port, additional-hosts) that are mutually exclusive with uri. We should certainly not introduce those.
The database (and authentication-database) property can however be used **together** with uri`.

It seems that Spring Boot interprets the /defaultauthdb part as "default database to use for authentication and data", while it allows users to specify a different database for each one of these functionalities. We should probably offer a similar configuration option. As documented in Authentication Database, you can create a user in database foo and allow the user to access database bar, so we need at least (and at most 😉) an additional configuration knob for the database to use (or authentication database to use).

jesmith17 · 2024-12-27T11:56:25Z

I would also point out that spring doesn't have a configuration property for the collection, and that's really the key item here. They assume, appropriately, that you will have various objects being written to various collections inside the app. So you configure the collection for items differently than you do the DB.

That's really the pattern I think is needed here.

jesmith17 · 2024-12-29T02:10:25Z

So I am working on making the unit test changes but I am finding 2 key challenges.

The MongoDbProvider and the MongoDbConnection classes aren't easily unitTested. Essentially they don't expose anyways to ask them about the database or collection (or even the connection for that matter) based on how they currently sit. So Im finding that I have to make some more changes to make them really testable.
I am getting blocked by the io.fabric5:docker-maven-plugin and it's unhappiness that I am running on a M1 Mac. It can't seem to resolve any of the docker server connections so it fails when trying to start docker. I have verified that it's all running, but seem to be stuck. Has anyone recently hit this issue with the Mac M1 process and docker? According to the docker-maven-plugin this was supposed to be resolved in 0.38 and the version mine is finding is 0.45.1.

ppkarwasz

@jesmith17,

This looks almost done to me, thanks! It is still missing some details:

can you modify the config files in log4j-mongodb/src/test/resources to use the databaseName and collectionName attributes?
can you add a changelog entry to src/changelog/.2.x.x? Just copy/paste an existing one and modify the data.

log4j-mongodb/pom.xml

log4j-mongodb/src/main/java/org/apache/logging/log4j/mongodb/MongoDbProvider.java

src/site/antora/modules/ROOT/pages/manual/appenders/database.adoc

log4j-mongodb/src/test/java/org/apache/logging/log4j/mongodb/MongoDbProviderTest.java

…neous dependencies.

jesmith17 · 2024-12-29T16:01:55Z

Unable to do the changelog entry as this work is done on the main (3.x) branch and that file doesn't exist. I can work to back-port (or recreate these in the 2.x structure and can make those changes to that version as needed.

ppkarwasz · 2024-12-29T17:18:35Z

Unable to do the changelog entry as this work is done on the main (3.x) branch and that file doesn't exist. I can work to back-port (or recreate these in the 2.x structure and can make those changes to that version as needed.

Sorry, I meant src/changelog/.3.x.x 😉

src/changelog/.3.x.x/update_org_log4j_mongodb.xml

jesmith17

Look good to me

ppkarwasz · 2025-02-12T08:42:42Z

@garydgregory, @vy,

Could you review this PR?

github-actions · 2025-02-12T08:58:01Z

Job	Requested goals	Build Tool Version	Build Outcome
build-macos-latest	clean install	3.9.8	✅
build-ubuntu-latest	clean install	3.9.8	✅
build-windows-latest	clean install	3.9.8	✅

Generated by gradle/develocity-actions

vy · 2025-02-12T19:12:38Z

I will take care of this tomorrow afternoon.

…

Message ID: ***@***.***>

vy

@jesmith17, thanks so much for the changes! I have requested some changes. But I think we have a more fundamental problem: Your PR targets main, i.e., Log4j 3, that is yet to be released. Would you mind editing your PR to target the 2.x branch, please? I will share further remarks once the target is fixed.

vy · 2025-02-14T07:44:54Z

src/site/antora/modules/ROOT/pages/manual/appenders/database.adoc

@@ -817,6 +817,29 @@ for its format.

 **Required**

+| [[MongoDbProvider-attr-databaseName]]databaseName
+| 'string'


Suggested change

| 'string'

| `string`

vy · 2025-02-14T07:45:06Z

src/site/antora/modules/ROOT/pages/manual/appenders/database.adoc

+**Required**
+
+| [[MongoDbProvider-attr-collectionName]]collectionName
+| 'string'


Suggested change

| 'string'

| `string`

vy · 2025-02-14T07:45:51Z

src/site/antora/modules/ROOT/pages/manual/appenders/database.adoc

+|
+It specifies the name of the collection for the appender to use.
+For backward compatibility, the collection name can also be specified in the
+https://mongodb.github.io/mongo-java-driver/5.0/apidocs/mongodb-driver-core/com/mongodb/ConnectionString.html[Java-specific connection string]


Suggested change

https://mongodb.github.io/mongo-java-driver/5.0/apidocs/mongodb-driver-core/com/mongodb/ConnectionString.html[Java-specific connection string]

https://mongodb.github.io/mongo-java-driver/5.0/apidocs/mongodb-driver-core/com/mongodb/ConnectionString.html[Java-specific connection string].

src/site/antora/modules/ROOT/examples/manual/appenders/database/nosql-mongo-keys.json

vy · 2025-02-14T07:51:59Z

src/changelog/.3.x.x/update_org_log4j_mongodb.xml

+  <issue id="3321" link="https://github.com/apache/logging-log4j2/discussions/3321"/>
+  <description format="asciidoc">Update log4j-mongodb to allow collectionName and databaseName to be defined via the config file</description>


Suggested change

<issue id="3321" link="https://github.com/apache/logging-log4j2/discussions/3321"/>

<description format="asciidoc">Update log4j-mongodb to allow collectionName and databaseName to be defined via the config file</description>

<issue id="3322" link="https://github.com/apache/logging-log4j2/pull/3322"/>

<description format="asciidoc">Add `collectionName` and `databaseName` arguments to the MongoDB appender</description>

vy · 2025-02-14T07:57:15Z

log4j-mongodb/src/test/resources/MongoDbAdditionalFields.xml

I see you switched all IT configuration to use the new {database,collection}Name attributes, don't. This effectively removes the checks verifying that the old style is still working. Could you revert all your resources/*.xml changes, please? (Keep the new resource files you added for new tests.)

vy · 2025-02-14T08:00:31Z

log4j-mongodb/src/test/java/org/apache/logging/log4j/mongodb/MongoDbProviderTest.java

+    private String validConnectionStringWithoutDatabase = "mongodb://localhost:27017";
+    private String validConnectionStringWithDatabase = "mongodb://localhost:27017/logging";
+    private String validConnectionStringWithDatabaseAndCollection = "mongodb://localhost:27017/logging.logs";
+
+    private String collectionName = "logsTest";
+    private String databaseName = "loggingTest";


Nit: Could you make all these private static final, please? (I think you can also rename validConnectionStringFoo to CON_STR_FOO to save some visual real estate.)

vy · 2025-02-14T08:01:35Z

log4j-mongodb/src/test/java/org/apache/logging/log4j/mongodb/MongoDbProviderTest.java

+        assertNotNull(provider, "Returned provider is null");
+        assertEquals(
+                this.collectionName,
+                provider.getConnection().getCollection().getNamespace().getCollectionName(),
+                "Collection names do not match");
+        assertEquals(
+                this.databaseName,
+                provider.getConnection().getCollection().getNamespace().getDatabaseName(),
+                "Database names do not match");


All checks are pretty self-explanatory. Could you remove all messages (Returned provider is null, Collection names do not match, etc.), please?

vy · 2025-02-14T08:17:26Z

log4j-mongodb/pom.xml

@@ -151,7 +150,7 @@
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <configuration>
-          <skip>true</skip>
+          <skip>false</skip>


You can remove the entire <configuration>, since <skip>false is the default.

jesmith17 · 2025-02-15T01:23:09Z

@vy Because the filestrucutres between main and 2.x are different I don't think there is a way to just retarget the code changes. In the 2.x branch the MongoDbProvider class is just a shell that wraps calls to MongoDb4Provider (which is in a different project). I can create a new PR with these changes (and the feedback above) and do that if that makes sense. but trying to port these changes to the 2.x branch ends up being a much bigger change to the core of the appender itself.

To your point on the changes to the IT tests to use the new format in all cases, I agree that a test needs to exist for using the old format, but I made those changes at the request of @ppkarwasz on Dec 29th. I don't personally have a strong opinion either way, so if you can both get connected and let me know which way you prefer it, I can make the changes.

jesmith17 · 2025-02-15T01:31:16Z

Actually for the IT tests, I just added an additional one that explicitly tests the old format. That's more descriptive and clear anyway.

jesmith17 added 2 commits December 21, 2024 17:40

Added ability to configure database and collection name via the confi…

91a0db4

…g file instead of just the connectionString

Added changelog file

1836809

ppkarwasz changed the title ~~Main~~ Add collectionName/databaseName attributes to MongoDB provider Dec 25, 2024

Formatting

9d9608d

ppkarwasz requested changes Dec 25, 2024

View reviewed changes

Solved merge conflict in MongoDbProvider class

aa4c0bc

Added unit tests for MongoDbProvider

b746300

ppkarwasz reviewed Dec 29, 2024

View reviewed changes

Cleanup on documentation and updates to JUnit 5 from 4. Removed extra…

3bd99b7

…neous dependencies.

Updated example config files to use new configuration options

73b96f2

ppkarwasz approved these changes Dec 29, 2024

View reviewed changes

src/changelog/.3.x.x/update_org_log4j_mongodb.xml Outdated Show resolved Hide resolved

ppkarwasz and others added 2 commits December 29, 2024 19:53

Update src/changelog/.3.x.x/update_org_log4j_mongodb.xml

c1dd909

Formatting

a6dcb4f

jesmith17 commented Feb 11, 2025

View reviewed changes

Fixed type in one of the IT tests that was causing it to fail

bd66bc7

ppkarwasz requested review from garydgregory and vy February 12, 2025 08:42

vy requested changes Feb 14, 2025

View reviewed changes

vy reviewed Feb 14, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add `collectionName/databaseName` attributes to MongoDB provider #3322

Add `collectionName/databaseName` attributes to MongoDB provider #3322

jesmith17 commented Dec 21, 2024

ppkarwasz commented Dec 25, 2024

ppkarwasz left a comment

ppkarwasz commented Dec 25, 2024

garydgregory commented Dec 25, 2024

ppkarwasz commented Dec 25, 2024

garydgregory commented Dec 25, 2024

vy commented Dec 25, 2024

jesmith17 commented Dec 26, 2024

ppkarwasz commented Dec 27, 2024

jesmith17 commented Dec 27, 2024

jesmith17 commented Dec 29, 2024

ppkarwasz left a comment

jesmith17 commented Dec 29, 2024

ppkarwasz commented Dec 29, 2024

jesmith17 left a comment

ppkarwasz commented Feb 12, 2025

github-actions bot commented Feb 12, 2025

vy commented Feb 12, 2025 via email

vy left a comment

vy Feb 14, 2025

vy Feb 14, 2025

vy Feb 14, 2025

vy Feb 14, 2025

vy Feb 14, 2025

vy Feb 14, 2025

vy Feb 14, 2025

vy Feb 14, 2025

jesmith17 commented Feb 15, 2025

jesmith17 commented Feb 15, 2025

	https://mongodb.github.io/mongo-java-driver/5.0/apidocs/mongodb-driver-core/com/mongodb/ConnectionString.html[Java-specific connection string]
	https://mongodb.github.io/mongo-java-driver/5.0/apidocs/mongodb-driver-core/com/mongodb/ConnectionString.html[Java-specific connection string].

		<issue id="3321" link="https://github.com/apache/logging-log4j2/discussions/3321"/>
		<description format="asciidoc">Update log4j-mongodb to allow collectionName and databaseName to be defined via the config file</description>

Add collectionName/databaseName attributes to MongoDB provider #3322

Are you sure you want to change the base?

Add collectionName/databaseName attributes to MongoDB provider #3322

Conversation

jesmith17 commented Dec 21, 2024

ppkarwasz commented Dec 25, 2024

ppkarwasz left a comment

Choose a reason for hiding this comment

ppkarwasz commented Dec 25, 2024

garydgregory commented Dec 25, 2024

ppkarwasz commented Dec 25, 2024

garydgregory commented Dec 25, 2024

vy commented Dec 25, 2024

jesmith17 commented Dec 26, 2024

ppkarwasz commented Dec 27, 2024

jesmith17 commented Dec 27, 2024

jesmith17 commented Dec 29, 2024

ppkarwasz left a comment

Choose a reason for hiding this comment

jesmith17 commented Dec 29, 2024

ppkarwasz commented Dec 29, 2024

jesmith17 left a comment

Choose a reason for hiding this comment

ppkarwasz commented Feb 12, 2025

github-actions bot commented Feb 12, 2025

Generated by gradle/develocity-actions

vy commented Feb 12, 2025 via email

vy left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jesmith17 commented Feb 15, 2025

jesmith17 commented Feb 15, 2025

Add `collectionName/databaseName` attributes to MongoDB provider #3322

Add `collectionName/databaseName` attributes to MongoDB provider #3322