Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Send data to database using plain SQL and psycopg2 #7

Open
wants to merge 12 commits into
base: master
Choose a base branch
from
Open

Send data to database using plain SQL and psycopg2 #7

wants to merge 12 commits into from

Commits on Apr 19, 2023

  1. sort imports

    @emiedmee
    emiedmee committed Apr 19, 2023
    Configuration menu
    Copy the full SHA
    674075b View commit details
    Browse the repository at this point in the history

  2. format code

    @emiedmee
    emiedmee committed Apr 19, 2023
    Configuration menu
    Copy the full SHA
    cd29500 View commit details
    Browse the repository at this point in the history

  3. Add types everywhere

    @emiedmee
    emiedmee committed Apr 19, 2023
    Configuration menu
    Copy the full SHA
    3a7776a View commit details
    Browse the repository at this point in the history

  4. Just specify datatypes as strings and create tables before insert 

    - Got errors with sqlalchemy
- It simple, so no need to use a library for that anyway
- Also tables are nowhere create yet
    @emiedmee
    emiedmee committed Apr 19, 2023
    Configuration menu
    Copy the full SHA
    cc0e9b3 View commit details
    Browse the repository at this point in the history

Commits on Apr 20, 2023

  1. Create own function to send data to db (just the skeleton) 

    Because pandas.DataFrame.to_sql() gives some "multiple schemas" error that I can't solve
    @emiedmee
    emiedmee committed Apr 20, 2023
    Configuration menu
    Copy the full SHA
    bf7e246 View commit details
    Browse the repository at this point in the history

  2. Create table if not exist and beginning for insert command

    @emiedmee
    emiedmee committed Apr 20, 2023
    Configuration menu
    Copy the full SHA
    86e198a View commit details
    Browse the repository at this point in the history

  3. Only import required things

    @emiedmee
    emiedmee committed Apr 20, 2023
    Configuration menu
    Copy the full SHA
    9d91cd2 View commit details
    Browse the repository at this point in the history

  4. Some test data for merged GSA and GSV

    @emiedmee
    emiedmee committed Apr 20, 2023
    Configuration menu
    Copy the full SHA
    1ac7fe9 View commit details
    Browse the repository at this point in the history

  5. Fix some column_casting data types

    @emiedmee
    emiedmee committed Apr 20, 2023
    Configuration menu
    Copy the full SHA
    bb88dd3 View commit details
    Browse the repository at this point in the history

  6. Insert dataframes in database using psycopg2 and template SQL statement

    @emiedmee
    emiedmee committed Apr 20, 2023
    Configuration menu
    Copy the full SHA
    755e235 View commit details
    Browse the repository at this point in the history

  7. Replace pd.NaN and pd.NaT with None, because they are incompatible wi… 

    …th psycopg2
    @emiedmee
    emiedmee committed Apr 20, 2023
    Configuration menu
    Copy the full SHA
    06378fb View commit details
    Browse the repository at this point in the history

  8. Remove sqlalchemy, only psycopg2 is needed for SQL now and comment ou… 

    …t original send_data_to_db()

Signed-off-by: Emiel Demeestere <emiel.demeestere@ugent.be>
    @emiedmee
    emiedmee committed Apr 20, 2023
    Configuration menu
    Copy the full SHA
    97f3a12 View commit details
    Browse the repository at this point in the history