Skip to content

ENH: Optional preservation of Sparse columns in Parquet/Feather via Arrow metadata [DRAFT] #63051

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 12 commits into
base: main
Choose a base branch
from
Draft

ENH: Optional preservation of Sparse columns in Parquet/Feather via Arrow metadata [DRAFT] #63051

wants to merge 12 commits into from
+282 −0

Conversation

@antznette1
Copy link

@antznette1 antznette1 commented Nov 9, 2025

Description:

  • Adds xfail tests defining expected behavior for preserving Sparse columns on Parquet/Feather roundtrip.
  • No engine code changes yet; this is to gather API feedback.
  • Proposed API:
    • to_parquet(..., preserve_sparse=False)
    • to_feather(..., preserve_sparse=False)
    • read_parquet(..., preserve_sparse=False)
    • read_feather(..., preserve_sparse=False)
  • Request feedback on:
    • Flag name: preserve_sparse vs preserve_extension_arrays
    • Arrow schema metadata namespacing (e.g., pandas.sparse.*)
    • Parameter placement in IO functions
  • Links:

antznette1 and others added 12 commits November 6, 2025 01:04
@antznette1
ENH: to_excel(autofilter=...) apply Excel autofilter over written ran…
e5975ac 
…ge for xlsxwriter/openpyxl; keep engine_kwargs semantics intact
@antznette1
TST/DOC: add tests for to_excel(autofilter=True) for openpyxl/xlsxwri…
e227aa7 
…ter and a user guide snippet
@antznette1
Address reviewer feedback on autofilter PR
9f37767 
- Remove duplicate to_excel function code in generic.py
- Add NotImplementedError for odfpy engine when autofilter=True
- Remove broad exception handling from autofilter implementations
- Add comprehensive tests for nonzero startrow/startcol
- Add tests for MultiIndex columns with merge_cells=True and False
- Improve tests to verify each column has autofilter
- Remove redundant test_to_excel test
- Remove redundant pytest.importorskip from test functions
@antznette1
FIX: Fix docstring formatting, missing imports, and undefined variables
d81840c
@antznette1
FIX: Move ExcelWriter and ExcelFormatter imports to avoid circular im…
44010db 
…port

- Keep ExcelWriter and ExcelFormatter in TYPE_CHECKING for type hints
- Import both at runtime inside to_excel method to avoid circular import
@antznette1
Merge branch 'main' into autofilter-feature
ea914ef
@antznette1
Merge branch 'main' into autofilter-feature
3124ef0
@jbrockmendel @antznette1
REF: inline array_to_datetime64 cases, update tests (pandas-dev#63015)
ea941f6
@antznette1
TST: narrow PR to tests-only; revert engine and unrelated test changes
dfba035
@antznette1
TST: xfail sparse roundtrip tests for Parquet and Feather (preserve_s…
21b7cf1 
…parse flag)
@antznette1
Merge branch 'main' into enh/io-sparse-preserve
c2212aa
@antznette1
ENH: plumb no-op preserve_sparse flag in Parquet/Feather IO; TST: use…
20a06a6 
… tm.assert_series_equal in sparse IO tests
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

ENH: Optional preservation of Sparse columns in Parquet/Feather via Arrow metadata

2 participants

@antznette1 @jbrockmendel