Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Better error handling #26

Open
Arpan-Kreeti opened this issue Oct 28, 2022 · 1 comment
Open

Better error handling #26

Arpan-Kreeti opened this issue Oct 28, 2022 · 1 comment

Comments

@Arpan-Kreeti
Copy link
Collaborator

Ideas on error handling

If an error is encountered during the insertion of data we should not fail the entire process.

Ideally, we should localize the error only for the file, or even better we can continue with that file after logging the error.
If a file logs more than X errors we can choose to stop processing the file.

Errors can be written to some error log file which we can save in the user's CSV directory.
The CSV directory can have a results folder with the schema SQL file and any error logs.
On the processing end, we can inform the users that errors were encountered in the following files and point then to the log file for details.

Changes required here:

repo.insert_all(name, encoded_data_chunk)


In case of errors during schema inference that particular file can be skipped.

@Arpan-Kreeti
Copy link
Collaborator Author

Running iconv is goo idea to get rid of incorrect bytes of data

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant