Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use postgres for backend #44

Closed
wants to merge 14 commits into from
Closed

Use postgres for backend #44

wants to merge 14 commits into from

Conversation

YaxelPerez
Copy link
Contributor

(in progress)

This PR replaces filesystem persistence with a Postgres backend using the Django ORM. More discussion in issue #26.

@YaxelPerez YaxelPerez linked an issue Sep 7, 2020 that may be closed by this pull request
Move spreadsheet storage to postgres #26
Open
@YaxelPerez YaxelPerez force-pushed the 26-postgres-backend branch 2 times, most recently from 1cc68d0 to 1fb282e Compare September 14, 2020 06:11
@YaxelPerez
Copy link
Contributor Author

Where to go from here?

As it stands, this PR is not ready to merge. Here is what needs to be done:

Actually make torquedata use Postgres

Currently torquedata is using an Sqlite3 database for development purposes. Changing the database engine to postgres should be trivial (documented here).

(settings.py can be moved to settings.py.tmpl, keeping in line with how our ansible setup currently handles torquedata configuration)

Configure file handling

MEDIA_ROOT is not currently configured in settings.py (see documentation on managing files).

Issue #20 mentions using Backblaze to store attachments. pyutil/django-b2 would probably work well for this purpose. It seems relatively well-maintained but documentation is sparse. It exposes a custom storage backend called B2Storage which can replace Django's default FileSystemStorage (see documentation on the FileField storage parameter).

Add Edit model

Discussed in DESIGN.md

As per issue #32, users can edit data fields (e.g., to fix a typo in a city's name in the "City" field) if they have proper permissions.

Add PostgreSQL Text Search

Discussed in issue #45. Currently spreadsheet/proposal data is stored in a JSONField (see documentation on querying JSONField) which might not be amenable to this solution.

More django-admin commands

Currently the only way to configure a template is by making a call to the torquedata API. It might be convenient to expand on the library of convenience functions provided by ./manage.py.

@frankduncan frankduncan changed the base branch from master to main September 23, 2020 14:28
@Nolski Nolski force-pushed the 26-postgres-backend branch 23 times, most recently from 32e44c4 to bb60339 Compare October 7, 2020 15:10
@Nolski Nolski force-pushed the 26-postgres-backend branch 22 times, most recently from aac4c1b to 26ded06 Compare October 30, 2020 16:06
@Nolski
Finish django compatibility on core api endpoints
1c18b66 
This is related to issue #26

This is finishing up the existing work started by YaxelPerez. With this
commit torque now uses postgres and django for all API endpoints except
search and editing.
@Nolski
Re-implemnt search API using django/postgres
bd0390c 
This is related to issue #26

With this commit torque's search API endpoint now uses postgres' Full
Text Search feature to perform search queries.
@Nolski
Reimplement edit API using postgres
7c4c02d 
This resolves issue #26

This is a re-implementation of all edit functionality but now using
django and postgres
@Nolski
Add edit buttons to main example template
17230d0
@Nolski
Add black formatting to deploy
cee453d
@frankduncan frankduncan deleted the 26-postgres-backend branch March 21, 2021 13:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@frankduncan frankduncan Awaiting requested review from frankduncan

@Nolski Nolski Awaiting requested review from Nolski

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Move spreadsheet storage to postgres
3 participants
@YaxelPerez @Nolski @frankduncan