Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error while inserting a list/array object existing in a Dataframe Column to a Postgresql table column #29

Open
vaibhav90 opened this issue Nov 1, 2021 · 1 comment

Comments

@vaibhav90
Copy link

vaibhav90 commented Nov 1, 2021

Proving an example to replicate this issue:
Consider the following Pandas Dataframe called my_data:
| id | coordinates |
| 01| [50.1, 68.2] |
|02| [52.2, 67.9] |

and consider the following postgresql table definition

CREATE TABLE coordinates (
id int2,
coordinates int8range
);

Now, even if you cast this dataframe column to the type object
my_data['coordinates'] = my_data['coordinates'].astype(object)

the following insert operation throws an error:
d6tstack.utils.pd_to_psql(my_table, db, table, schema if_exists='replace')

The Error log is:
BadCopyFileFormat: extra data after last expected column

The same insert operation works with dataframe.to_sql functionality provided by psycopg2

@marksparrish
Copy link

You need to add sep='\t' to the call.

d6tstack.utils.pd_to_psql(df, str(engine.url), table_name, if_exists='append',sep='\t')

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants