Skip to content

Conversation

urvisavla
Copy link
Contributor

@urvisavla urvisavla commented Oct 17, 2025

PR Checklist

PR Structure

  • This PR has reasonably narrow scope (if not, break it down into smaller PRs).
  • This PR avoids mixing refactoring changes with feature changes (split into two PRs
    otherwise).
  • This PR's title starts with name of package that is most changed in the PR, ex.
    services/friendbot, or all or doc if the changes are broad or impact many
    packages.

Thoroughness

  • This PR adds tests for the most critical parts of the new functionality or fixes.
  • I've updated any docs (developer docs, .md
    files, etc... affected by this change). Take a look in the docs folder for a given service,
    like this one.

Release planning

  • I've reviewed the changes in this PR and if I consider them worthwhile for being mentioned on release notes then I have updated the relevant CHANGELOG.md within the component folder structure. For example, if I changed horizon, then I updated (services/horizon/CHANGELOG.md. I add a new line item describing the change and reference to this PR. If I don't update a CHANGELOG, I acknowledge this PR's change may not be mentioned in future release notes.
  • I've decided if this PR requires a new major/minor version according to
    semver, or if it's mainly a patch change. The PR is targeted at the next
    release branch if it's not a patch change.

What

Add a new command scan-and-replace to re-export of all ledgers within a specified range, overwriting any existing data in the data lake. It is restricted to bounded mode.

Why

#5514

Known limitations

The command uses a single PutFile operation and does not check for the existence. So, the uploader cannot distinguish between a new file and an overwritten one. Therefore, for metrics logging label already_exists will always be recorded as false.

@urvisavla urvisavla requested review from sreuland and tamirms and removed request for sreuland October 17, 2025 20:11
Copy link
Contributor

@sreuland sreuland left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice work and test coverage! left a few comments for consideration, nothing blocking.

_ Mode = iota
ScanFill Mode = iota
Append
ScanReplace
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not a big deal for me but I would prefer the command to be called replace, overwrite, or rewrite instead of scan-and-replace

scanAndFillCmd.PersistentFlags().AddFlagSet(commonFlags)
viper.BindPFlags(scanAndFillCmd.PersistentFlags())

scanAndReplaceCmd.PersistentFlags().AddFlagSet(commonFlags)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: I think we can use the common flags in the other commands as well

Copy link
Contributor

@tamirms tamirms left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

left a few comments for minor issues but, overall, this looks good!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants