Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: Configmap not updated when exclusion added/removed dynamically. #2408

Closed
ikreymer opened this issue Feb 18, 2025 · 0 comments · Fixed by #2409
Closed

[Bug]: Configmap not updated when exclusion added/removed dynamically. #2408

ikreymer opened this issue Feb 18, 2025 · 0 comments · Fixed by #2409
Assignees
@ikreymer
Labels
bug Something isn't working

Comments

@ikreymer
Copy link
Member

Browsertrix Version

v1.13.0

What did you expect to happen? What happened instead?

Although exclusions are applied dynamically, the configmap used to start a new crawler instance is not updated. This means that if crawler pod is restarted, it will not pick up new exclusions.

Reproduction instructions

  1. Add/Remove exclusions while crawl is running.
  2. Delete pod/wait for it to restart due to time limit/size limit etc (easier to delete pod)
  3. Observe restarted crawler does not pick updated exclusions.

Screenshots / Video

No response

Environment

No response

Additional details

No response
@ikreymer ikreymer added the bug Something isn't working label Feb 18, 2025
@ikreymer ikreymer self-assigned this Feb 18, 2025
@ikreymer ikreymer moved this from Triage to Todo in Webrecorder Projects Feb 18, 2025
ikreymer added a commit that referenced this issue Feb 18, 2025
@ikreymer
exclusion update:
04fafa9 
- ensure newly added / remove exclusions are updated in the configmap that is mounted into the crawler volume
- exclusions are updated dynamically if crawler pod is running, but when crawler pod is restarted, this ensures new exclusions are also picked up
- adds a lastConfigUpdate timestamp to CrawlJob: if lastConfigUpdate in spec is different from current, the configmap is recreated
- only updates for exclusion add/remove so far, can later be expanded to other crawler settings (see: #2355 for broader running crawl config updates)
- fixes #2408
@ikreymer ikreymer mentioned this issue Feb 18, 2025
Merged
@github-project-automation github-project-automation bot moved this from Todo to Done! in Webrecorder Projects Feb 19, 2025
emma-sg pushed a commit that referenced this issue Feb 19, 2025
@ikreymer @emma-sg
ensure running crawl configmap is updated when exclusions are added/r…
ac4467f 
…emoved (#2409)

exclusions are already updated dynamically if crawler pod is running,
but when crawler pod is restarted, this ensures new exclusions are also
picked up:
- mount configmap in separate path, avoiding subPath, to allow dynamic
updates of mounted volume
- adds a lastConfigUpdate timestamp to CrawlJob - if lastConfigUpdate in
spec is different from current, the configmap is recreated by operator
- operator: also update image from channel avoid any issues with
updating crawler in channel
- only updates for exclusion add/remove so far, can later be expanded to
other crawler settings (see: #2355 for broader running crawl config
updates)
- fixes #2408
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ikreymer ikreymer

Labels
bug Something isn't working
Projects
Webrecorder Projects
Archived in project
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

ensure running crawl configmap is updated when exclusions are added/removed webrecorder/browsertrix
1 participant
@ikreymer