feat(cl_back_scrape_citations): command to scrape citations #4303
Conversation
- Remove "method" argument - Log errors instead of returning the error message - Return the cleaned up content using site.cleanup_content - Update tests - Update opinions and oral arguments scraper caller to reflect changes
for more information, see https://pre-commit.ci
grossir
force-pushed
the
scrape_citations_command
branch
from
0255163
to
d0ea84b
Compare
grossir
force-pushed
the
scrape_citations_command
branch
from
d0ea84b
to
fac3f13
Compare
|
We can test this by cloning a few
Citations to pre-populate
Commands to run the testQuery to check everything went OKAssumes you are running the query inmediately after the scrapers, otherwise change the interval hours to a bigger value select * from search_citation where cluster_id in (
select id from search_opinioncluster where docket_id in (
select id from search_docket where court_id = 'md' and date_created > (now() - interval '6 hours')
)
);and you should see something like this, the cluster_id should be small for new documents, big for "updated" citations. In this case we have a single big cluster_id since we only downloaded one cluster where the hash will match |
|
@mlissner can you review this? When writing the tests I realized that we don't have a backscraper for |
Reword docstrings, catch exceptions and refactor code following code review
- create cl.scrapers.exceptions file to hold exceptions raised when ingesting a single case - use the exceptions to bubble errors to the main loop, to avoid returning break / continue flags - refactor DupChecker to raise errors - refactor get_binary_content to raise errors - refactor cl_scrape_oral_arguments to new paradigm - cl_back_scrape_citations can now re-scrape a single case without re-downloading the binary content or manipulating the site object - adapted DupChecker and ContentType tests to changes - refactores logger.calls to use lazy formatting
|
@mlissner can you review this again? I updated the PR with the requested changes |
|
Very nice. I've updated to HEAD and set for auto-merge. Thank you! |
grossir
dismissed
mlissner’s stale review
I am using the "dismiss review" button, since I understand this PR has been approved, but @mlissner didn't clear the "changes requested" status and merging is still blocked
Suspect IssuesThis pull request was deployed and Sentry observed the following issues:
Did you find this useful? React with a 👍 or 👎 |
Related to preventing further duplicates as seen on freelawproject#4376, due to changes introduced in freelawproject#4303 - Refactor tests for DupChecker.press_on method: replaces fixtures, loops and if clauses by explicit test objects and explicit press_on calls for each scenario
get_binary_contentto be re-used by the new command and to reduce copy pasted codeApplies to a specific family of court's sites described here:
freelawproject/juriscraper#858 (comment)