Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix(scrapers.DupChecker): ensure raising SingleDuplicateError #4375

Merged
merged 2 commits into from
Aug 29, 2024
Merged

fix(scrapers.DupChecker): ensure raising SingleDuplicateError #4375

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
6f00430
fix(scrapers.DupChecker): ensure raising SingleDuplicateError
grossir Aug 29, 2024
881b857
Merge branch 'main' into fix_dup_checker
grossir Aug 29, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 8 additions & 7 deletions cl/scrapers/DupChecker.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -135,19 +135,20 @@ def press_on(
else:
already_scraped_next_date = True

# When in a full crawl, we do not raise a loop breaking
# `ConsecutiveDuplicatesError`
if not self.full_crawl:
if already_scraped_next_date:
if self.court.pk == "mich":
# Michigan sometimes has multiple occurrences of the
# same case with different dates on a page.
raise SingleDuplicateError(logger=logger)
else:
message = "Next case occurs prior to when we found a duplicate. Court is up to date."
raise ConsecutiveDuplicatesError(message, logger=logger)

message = "Next case occurs prior to when we found a duplicate. Court is up to date."
raise ConsecutiveDuplicatesError(message, logger=logger)
elif self.dup_count >= self.dup_threshold:
message = f"Found {self.dup_count} duplicates in a row. Court is up to date."
raise ConsecutiveDuplicatesError(message, logger=logger)
else:
# This is a full crawl. Do not raise a loop breaking `ConsecutiveDuplicatesError`,
# but say that we shouldn't press on, since the item already exists.
raise SingleDuplicateError(logger=logger)

# Full crawl or not, this is a duplicate and we shouldn't store it
raise SingleDuplicateError(logger=logger)
Loading