Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SNOW-753219: Add support for "to_arrow" and "from_arrow" methods #704

Open
tmespe opened this issue Mar 3, 2023 · 1 comment
Open

SNOW-753219: Add support for "to_arrow" and "from_arrow" methods #704

tmespe opened this issue Mar 3, 2023 · 1 comment
Labels
feature New feature or request

Comments

@tmespe
Copy link

tmespe commented Mar 3, 2023

What is the current behavior?

Currently the only way to "export" a snowpark dataframe is by converting it to a pandas dataframe. While this is great, newer alternatives are starting to gain traction, so a way to export to them to other libraries than pandas would be good.

Currently you can convert to a pandas dataframe, and then into other types, but a more direct way would be better.

There is already support for fetch_arrow_all in snowflake-connector, which I believe snowpark already uses, and I've tested it to work with Polars.from_arrow().

What is the desired behavior?

Expose an arrow representation of the dataframe so that it can be picked up by f.ex Polars.from_arrow().

Inversely it would be great to be able to do the reverse operation using polars.to_arrow().

How would this improve snowflake-snowpark-python?

This would allow snowpark to be more flexible as Arrow is becoming widely supported in the data space. It would also allow basic support for other libraries without having adding direct support for certain libraries.

References, Other Background

@tmespe tmespe added the feature New feature or request label Mar 3, 2023
@github-actions github-actions bot changed the title Add support for "to_arrow" and "from_arrow" methods SNOW-753219: Add support for "to_arrow" and "from_arrow" methods Mar 3, 2023
@mjclarke94
Copy link

mjclarke94 commented Aug 16, 2023

This one is pretty important for us. Loading data in to pandas just to turn it in to polars, and back. Pretty long winded way of saying "grab this data".

Plus, with pandas 2.0 supporting arrow data types there's potentially some easy wins here with regards to weird conversion issues to/from date time types.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants