feat: Add strict parameter to pl.concat(how='horizontal') #25452

cBournhonesque · 2025-11-21T18:30:07Z

PR that closes #19133
Adopted from #20019

…d added a corresponding unit test

…dataframes when how=horizontal

…ola-rs#20084)

… single element and ddof=1 and there are nulls elsewhere in the Series (pola-rs#20077)

…0103)

…s#20107)

…dding a 'strict' keyword argument to concat_df(how='horizontal')

…lementation and added a more robust set of tests on concat

Change-Id: Icd5fb4d566af250b49f25834c931f8d7377e7515

Change-Id: Ie5f9bd91b671ae5f7e31e228f7b5a246d9168ab3

Change-Id: Iddb25d6798666abc0b9374b071b369f3e92cde39

Change-Id: I32f869c406cc0b34386c7c22aceea12495655eb5

Change-Id: I377b4efd92e0fd1a25639bdc723ceaeaea86113e

Change-Id: I9298a909b7f6eaac6893cecdbdd3be3286fd26f1

Change-Id: I74bdd5d39ec41fa5b4fe3371d62cbeac7b34ee62

Change-Id: If50f99a0b18a3a604620a40623a8323cb52d4e78

Change-Id: I5206f843ef735230ef1c2d37f0627a4e74b54b48

cBournhonesque · 2025-11-21T22:09:19Z

@orlp I followed the suggestion in #20019 (comment)
please let me know if that works!

codecov · 2025-11-21T22:38:54Z

Codecov Report

❌ Patch coverage is 84.78261% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.16%. Comparing base (76758c7) to head (9a99383).
⚠️ Report is 10 commits behind head on main.

Files with missing lines	Patch %	Lines
crates/polars-stream/src/nodes/zip.rs	75.00%	5 Missing ⚠️
...ates/polars-plan/src/plans/ir/visualization/mod.rs	0.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #25452      +/-   ##
==========================================
- Coverage   82.16%   82.16%   -0.01%     
==========================================
  Files        1728     1727       -1     
  Lines      240097   240195      +98     
  Branches     3028     3031       +3     
==========================================
+ Hits       197277   197353      +76     
- Misses      42040    42061      +21     
- Partials      780      781       +1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

crates/polars-core/src/frame/horizontal.rs

crates/polars-stream/src/nodes/zip.rs

orlp · 2025-11-24T13:51:11Z

crates/polars-stream/src/nodes/zip.rs

                }
-                let out_df = concat_df_horizontal(&out, false)?;
+                let strict_concat = matches!(self.extend_behavior, ExtendBehavior::Raise);
+                let out_df = concat_df_horizontal(&out, false, strict_concat)?;


I don't understand why that is true.
If we have ExtendBehavior::FillNulls, couldn't we have one input head that has data with a different height than others, in which we case we would want to concat without strict?
I don't see where that happens in the code either

If we are in the branch all input heads are broadcast inputs, meaning they have length 1.

crates/polars-stream/src/physical_plan/fmt.rs

crates/polars-stream/src/physical_plan/mod.rs

orlp · 2025-11-24T13:56:01Z

py-polars/src/polars/functions/eager.py

        Only relevant for LazyFrames. This determines if the concatenated
        lazy computations may be executed in parallel.
+    strict
+        When how=`horizontal`, require all DataFrames to be the same height


Missing period. Can you also emphasize that this is only to disable broadcasting? Different heights are never allowed anyway, only for broadcasting:

When how=`horizontal`, require all DataFrames to be the same height, raising an error instead of broadcasting unit height DataFrames.

Are different heights never allowed?
In this example I have two non-unit dataframes being concatenated together: #25263 (comment)

~~No, they're never allowed.~~

EDIT: my bad, you're right. Please disregard this. I'm too deep in the implementation of things where this is never allowed that I forgot this is the API by which users can request the ExtendWithNulls behavior.

Change-Id: I0934aa1fdabe12b6a333f0c97ec0958d0a545175

Change-Id: Id71f67f073bcf3d1dd3d26f62654b50ec40ae673

orlp · 2025-11-25T10:15:36Z

Can you address the final comment and then do a rebase on main? Either a squash or cherry-pick is fine whatever you prefer, but right now there's a lot of extra irrelevant commits in this branch.

nimit and others added 30 commits November 27, 2024 02:47

added 'strict' as a keyword argument (default: False) to pl.concat an…

6038130

…d added a corresponding unit test

bug fix with keyword argument change in pl.concat

d14049c

Merge branch 'pola-rs:main' into strict-concat-19133

ae2a335

rust changes

6a0df66

fixed python errors with concat_df_horizontal, concat_lf_horizontal

ea847bf

fixed exception type in python unit test for strict concatenation of …

8f2883d

…dataframes when how=horizontal

build: Bump chrono-tz to 0.10 (pola-rs#20094)

21d110a

chore(rust): Update AWS doc dependencies (pola-rs#20095)

d85fc9c

docs(rust): Fix inconsistency between code and comment (pola-rs#20070)

f00e6fd

fix: Only slice after sort when slice is smaller than frame length (p…

a413c4a

…ola-rs#20084)

fix: Return null instead of 0. for rolling_std when window contains a…

5a8cd16

… single element and ddof=1 and there are nulls elsewhere in the Series (pola-rs#20077)

build: Bump atoi_simd to version 0.16 (pola-rs#20098)

755ee47

build: Bump thiserror to version 2 (pola-rs#20097)

af4d5a5

build(rust): Fix path to polars-dylib crate in workspace (pola-rs#2…

06b2ab6

…0103)

build: Bump fs4 to version 0.12 (pola-rs#20101)

511c219

build: Bump object_store to version 0.11 (pola-rs#20102)

b276485

build: Bump memmap2 to version 0.9 (pola-rs#20105)

bc3d320

refactor(rust): Replace custom PushNode trait with Extend (pola-r…

8f72e20

…s#20107)

build: Upgrade sqlparser-rs from version 0.49 to 0.52 (pola-rs#20110)

527af9c

fixed LazyFrame test, Python & Rust documentation changes regarding a…

0d4029a

…dding a 'strict' keyword argument to concat_df(how='horizontal')

Merge branch 'pola-rs:main' into strict-concat-19133

90e65e8

doc fix, removed unnecessary generic

6078964

pass additional parameter to Zip from IR::HConcat for strict concat

5f3e8a6

made broadcasting congruent with InputHead constructor in ZipNode imp…

d5b373b

…lementation and added a more robust set of tests on concat

Merge branch 'pola-rs:main' into strict-concat-19133

80edf1a

concat tests

353a563

linting

68dd6bf

rename test vars for better readability

65cb6b1

rename test vars for better readability

d418bb5

linting change

57e00ec

chore: add version

98e9496

Change-Id: Icd5fb4d566af250b49f25834c931f8d7377e7515

cBournhonesque requested review from MarcoGorelli, alexander-beedie, c-peters, orlp, reswqa and ritchie46 as code owners November 21, 2025 18:30

github-actions bot added the title needs formatting label Nov 21, 2025

Charles Bournhonesque added 4 commits November 21, 2025 13:53

Merge branch 'main' into strict-concat-19133

0b4adce

Change-Id: Ie5f9bd91b671ae5f7e31e228f7b5a246d9168ab3

add ExtendBehavior to zip

fc897dc

Change-Id: Iddb25d6798666abc0b9374b071b369f3e92cde39

chore: cargo fmt

e61845e

Change-Id: I32f869c406cc0b34386c7c22aceea12495655eb5

all tests pass

c0d83e5

Change-Id: I377b4efd92e0fd1a25639bdc723ceaeaea86113e

cBournhonesque changed the title ~~Strict concat 19133~~ feat: Add strict parameter to pl.concat(how='horizontal') Nov 21, 2025

github-actions bot added enhancement New feature or an improvement of an existing feature python Related to Python Polars rust Related to Rust Polars and removed title needs formatting labels Nov 21, 2025

Charles Bournhonesque added 4 commits November 21, 2025 16:39

make pre-commit

b463cb2

Change-Id: I9298a909b7f6eaac6893cecdbdd3be3286fd26f1

nit: visualization fixes

b04af17

Change-Id: I74bdd5d39ec41fa5b4fe3371d62cbeac7b34ee62

nit: fix serde

e0ba843

Change-Id: If50f99a0b18a3a604620a40623a8323cb52d4e78

udpdate schema hashes

dfe1c31

Change-Id: I5206f843ef735230ef1c2d37f0627a4e74b54b48

github-actions bot added the changes-dsl Do not merge if this label is present and red label Nov 21, 2025

orlp requested changes Nov 24, 2025

View reviewed changes

Charles Bournhonesque added 2 commits November 24, 2025 12:21

address comments

6deec30

Change-Id: I0934aa1fdabe12b6a333f0c97ec0958d0a545175

fix ci

9a99383

Change-Id: Id71f67f073bcf3d1dd3d26f62654b50ec40ae673

feat: Add strict parameter to pl.concat(how='horizontal') #25452

Are you sure you want to change the base?

feat: Add strict parameter to pl.concat(how='horizontal') #25452

Conversation

cBournhonesque commented Nov 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cBournhonesque commented Nov 21, 2025

Uh oh!

codecov bot commented Nov 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

orlp Nov 24, 2025

Choose a reason for hiding this comment

Uh oh!

cBournhonesque Nov 24, 2025

Choose a reason for hiding this comment

Uh oh!

orlp Nov 25, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

orlp Nov 24, 2025

Choose a reason for hiding this comment

Uh oh!

cBournhonesque Nov 24, 2025

Choose a reason for hiding this comment

Uh oh!

orlp Nov 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

orlp commented Nov 25, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

cBournhonesque commented Nov 21, 2025 •

edited

Loading

codecov bot commented Nov 21, 2025 •

edited

Loading

orlp Nov 25, 2025 •

edited

Loading