-
Notifications
You must be signed in to change notification settings - Fork 31
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add Elastic alias for cleaner ingestion #1071
Conversation
fc4e4f0
to
e54a484
Compare
django_app/redbox_app/redbox_core/management/commands/change_es_aliased_index.py
Outdated
Show resolved
Hide resolved
9eccc2b
to
93ad77b
Compare
3abbed9
to
73d68dc
Compare
django_app/redbox_app/worker.py
Outdated
|
||
env = Settings() | ||
alias = f"{env.elastic_root_index}-chunk-current" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
should/could be part of Settings
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this is working me for me in dev. would be good to merge some of these commands into one as you suggest (but in a seperate PR)
73d68dc
to
a9232e7
Compare
Context
When ingesting our files, we should be making sure we remove old artifacts. The standard ElasticSearch approach is to use an 'alias' rather than 'index' as a reference, which allows us to update the index 'behind the scenes' without affecting users. I've started updating the file ingestion to reflect this.
Changes proposed in this pull request
notebooks
for reference and remove the jupyter poetry dependencyGuidance to review
To try this out locally, try out the commands as a Scheduled Command from the local django admin in the order listed in the jupyter notebook:
You can check the current list of indices via http://localhost:9200/_cat/indices and the aliased index via http://localhost:9200/_alias/redbox-data-{environment}-chunk-current
Alternatively, you can run the notebook from the
/redbox/django_app
directory after runningpoetry install
andpoetry run jupyter notebook
Relevant links
Things to check