feat: support update/merge-into using RewriteColumns mode by fangbo · Pull Request #199 · lance-format/lance-spark

fangbo · 2026-02-03T09:45:16Z

This is for #166

add a new parameter rewrite_columns to specify update/merge-into to use RewriteColumns mode. If the parameter value is false RewriteRows mode is used which means that the rows are deleted and new updated rows are inserted.
UpdateColumnsExtractor rule is used to extract updated columns from spark sql: update or merge into. The specific updated columns will be injected to LancePositionDeltaOperation.
if rewrite_columns is true, LancePositionDeltaOperation.representUpdateAsDeleteAndInsert return false. It means that LanceDeltaWriter.update will be invoked for columns updating.
In LanceDeltaWriter.update , the new updated columns values are collected for a Fragment. Then Fragment.updateColumns is invoked to update the specific columns.

fangbo · 2026-02-04T03:49:59Z

@jackye1995 @jiaoew1991 @hamersaw Do you think this approach is reasonable ?

jackye1995 · 2026-02-06T04:30:13Z

...3.5_2.12/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UpdateColumnsExtractor.scala

+        }
+
+      } catch {
+        case _: Exception =>


this should not just fail silently

Sorry it's my mistake. I have made some modification and throw a RuntimeException to inform users to set spark.sql.lance.rewrite_columns to false to disable this feature.

jackye1995 · 2026-02-06T04:32:23Z

lance-spark-3.5_2.12/src/test/java/org/lance/spark/update/MergeIntoRewriteRowsTest.java

+                "spark.sql.extensions", "org.lance.spark.extensions.LanceSparkSessionExtensions")
+            .config("spark.sql.catalog." + catalogName + ".impl", "dir")
+            .config("spark.sql.catalog." + catalogName + ".root", tempDir.toString())
+            .config("spark.sql.catalog." + catalogName + ".storage.rewrite_columns", "false")


seems strange that we are making this a storage option. I think this should be more like a Spark SQL conf, so we can do something like SET to enable/disable it if necessary.

Good suggestion ! I have defined a key spark.sql.lance.rewrite_columns in SparkUtil.REWRITE_COLUMNS. The method SparkUtil#rewriteColumns check the rewrite mode from spark session configuration.

jackye1995 · 2026-02-11T04:38:44Z

lance-spark-3.5_2.12/src/main/java/org/lance/spark/write/SparkPositionDeltaWrite.java

+        writer.writeBatch();
+        writer.end();
+      } catch (IOException e) {
+        throw new RuntimeException("Cannot write schema root", e);


I don't think you need to rethrow as RuntimeException? The method already does throws IOException

jackye1995 · 2026-02-11T04:39:43Z

lance-spark-base_2.12/src/main/java/org/lance/spark/LanceSparkWriteOptions.java

  private final boolean useQueuedWriteBuffer;
  private final int queueDepth;
  private final int batchSize;
+  private final boolean rewriteColumns;


looks like this is added but never used?

jackye1995 · 2026-02-11T04:40:39Z

lance-spark-3.5_2.12/src/main/java/org/lance/spark/LancePositionDeltaOperation.java

+  }
+
+  public void setUpdatedColumns(List<String> updatedColumns) {
+    LOG.info("Set updated columns: {}", updatedColumns);


this feels like a debug message, or could just be removed?

jackye1995 · 2026-02-11T04:41:07Z

lance-spark-3.5_2.12/src/main/java/org/lance/spark/write/SparkPositionDeltaWrite.java

+    private final Set<Long> fieldsModified = new HashSet<>();
+    private final Map<Integer, FragmentMetadata> updatedFragments = new HashMap<>();
+
+    private int currentUpdateFragmentId = -1;


can we use Optional instead to be clear?

jackye1995

mostly looks good to me, could you also update the related documentation?

fangbo · 2026-02-11T12:12:48Z

mostly looks good to me, could you also update the related documentation?

@jackye1995 Thanks for your review. I have fixed the comments and added documentation in update.md

jackye1995 · 2026-02-11T18:07:41Z

docs/src/operations/dml/update.md

+
+## Column Rewrite Mode
+
+`lance-spark` introduces a column rewrite mode for `UPDATE` and `MERGE INTO` operations, which can significantly improve performance for narrow updates that only affect a few columns.


I think you should also update the MERGE doc? Currently the update doc talks about both update and merge.

I think you should also update the MERGE doc? Currently the update doc talks about both update and merge.

Thanks for your advice. I have added a new documentation merge-into.md

fangbo · 2026-02-25T02:14:45Z

@jackye1995 I have rebased this PR and fixed the issues you commented. Could you please review it again? Thanks a lot.

github-actions bot added the enhancement New feature or request label Feb 3, 2026

fangbo mentioned this pull request Feb 3, 2026

feat: support UPDATE COLUMNS FROM syntax for column-level updates #194

Merged

fangbo changed the title ~~feat: support update using RewriteColumns mode~~ feat: support update/merge into using RewriteColumns mode Feb 4, 2026

fangbo changed the title ~~feat: support update/merge into using RewriteColumns mode~~ feat: support update/merge-into using RewriteColumns mode Feb 4, 2026

fangbo force-pushed the update-rewrite-column branch 2 times, most recently from 6799b3a to 9e01904 Compare February 6, 2026 02:21

jackye1995 reviewed Feb 6, 2026

View reviewed changes

jackye1995 reviewed Feb 11, 2026

View reviewed changes

fangbo force-pushed the update-rewrite-column branch from 4d62dc0 to 4ceb03b Compare February 24, 2026 02:28

fangbo force-pushed the update-rewrite-column branch 3 times, most recently from 6f9677d to e4fe695 Compare February 26, 2026 02:22

fangbo.0511 added 2 commits March 18, 2026 11:14

feat: support update using RewriteColumns mode

3b21965

rebase main

ececb75

fangbo force-pushed the update-rewrite-column branch from a963cae to ececb75 Compare March 18, 2026 03:21

fix codestyle

7039168


		## Column Rewrite Mode

		`lance-spark` introduces a column rewrite mode for `UPDATE` and `MERGE INTO` operations, which can significantly improve performance for narrow updates that only affect a few columns.

Conversation

fangbo commented Feb 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fangbo commented Feb 4, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jackye1995 left a comment

Choose a reason for hiding this comment

Uh oh!

fangbo commented Feb 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fangbo commented Feb 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

fangbo commented Feb 3, 2026 •

edited

Loading

fangbo commented Feb 11, 2026 •

edited

Loading

fangbo commented Feb 25, 2026 •

edited

Loading