Skip to content

Conversation

@FGasper
Copy link
Collaborator

@FGasper FGasper commented Nov 12, 2025

This adds an opt-in mode to tail the oplog directly instead of reading a change stream. Oplog tailing is less resource-intensive on the server than a change stream, which yields better performance and allows verification of more migrations.

Oplog tailing carries a few caveats (or else it would be the default when possible):

  • The oplog’s format is undocumented, by design. A future MongoDB version may not work with the present logic. Change streams, by contrast, are fully documented and intended for use in applications like Migration Verifier.
  • To capture transactions that are in progress when verification starts, oplog tailing actually starts from the start optime of the oldest transaction that’s in progress when verification starts. This means a few extra rechecks may be enqueued at the start of a verification.

Additional notes:

  • Since migration-verifier always reads with majority read concern, we can ignore the term and hash that normally, with the timestamp, comprise an oplog entry’s “optime”.
  • ChangeStreamReader.start() is now part of ChangeReaderCommon so that its retry bits can be shared with the new OplogReader type.
  • This includes a nascent agg library, which simplifies aggregation usage in Go by providing types for various expressions.
  • The retryer now wraps errors with the function description message.
  • To ensure that OplogReader doesn’t read too far “back in time”, the integration test suite now kills all transactions between test methods.
  • Some of the tests are reordered a bit so that document insertion happens before verifier creation. This was originally done to troubleshoot test failures but seems worth retaining for general clarity.

@FGasper FGasper requested a review from tdq45gj November 21, 2025 00:03
@FGasper FGasper marked this pull request as draft November 21, 2025 00:07
@FGasper FGasper marked this pull request as ready for review November 21, 2025 00:40
@FGasper FGasper marked this pull request as draft November 21, 2025 01:19
@FGasper
Copy link
Collaborator Author

FGasper commented Nov 21, 2025

(Still not review-ready because it has to store the selected change reader option then fail if it changes between invocations.)

@FGasper FGasper marked this pull request as ready for review November 24, 2025 14:49
@FGasper FGasper marked this pull request as draft November 24, 2025 15:54
@FGasper FGasper marked this pull request as ready for review November 24, 2025 17:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant