Database storage and updating

[AlexS12]

Currently the database is not hosted in the repository due to a storage restriction in GitHub. Users are encouraged to download it using the script flight_safety/get_data.py which will download it from dropbox and place it in the data folder, or to do the process themselves.

Issues with this approach:

• Original database is a Microsoft Access (mdb) file.
• The database we are using is that one, converted to sqlite3, which lets users working on Linux use it comfortably.
• The sqlite3 database is stored in a dropbox account. If the original file is updated, we need to convert it and upload it again. As the dropbox account is personal, only @AlexS12 can do it.
• The database is hosted in a personal Dropbox account.
• This workflow could lead to different users having different versions of the database.
• We are forced to maintain the script to download the data and place it in the convenient place.

We should radically change this approach. It could be great if we could find somewhere to place the data in sqlite format and some automatic process could convert the original mdb files each month (they are supposed to be updated every month). When importing flight_safety, it could check if the user is working with the last version of the database and warn him otherwise.

Available options? other approaches?

[astrojuanlu]

Have you considered Git LFS?

https://git-lfs.github.com/

According to https://help.github.com/articles/about-storage-and-bandwidth-usage/,

All personal and organization accounts using Git LFS receive 1 GB of free storage and 1 GB a month of free bandwidth.

[astrojuanlu]

What others do is to upload the data to Google Big Query, which provides an SQL API to query it online. But they have a freemium model and the documentation is quite obscure, so every time I try to use it I give up (see for instance https://github.com/ofek/pypinfo)

[astrojuanlu]

And lastly, to provide some automated conversion, I can think of two options:

• Heroku. I use it to host some Telegram bots and has plenty of free hours. https://www.heroku.com/
• An automated Travis CI job with write permissions https://docs.travis-ci.com/user/cron-jobs/