Generate crosswalk and import cap pdf

[jtmst]

Import Harvard Case Law Access Project (CAP) PDFs to CourtListener

Issues Addressed:

• Import CAP PDFs to CL
• Generate crosswalk between CAP and CL

Changes Implemented:

1. Created a management command to generate crosswalk files between CAP and CL data.
2. Developed a management command to import CAP PDFs to CL using the generated crosswalk.
3. Implemented S3 storage integration for PDF storage (configurable for local storage in development).
4. Added error handling and logging for better debugging and monitoring.
5. Added settings for CAP R2 env variables
6. Added tests for both commands

Testing Instructions:

Prerequisites:

• Ensure you have the necessary environment variables set for R2 and S3 access.
• For local testing, configure the storage to use local file system instead of S3.

Steps:

1. Generate the crosswalk:

    docker exec -it cl-django python /opt/courtlistener/manage.py generate_cap_crosswalk
   

   This command will create crosswalk files in cl/search/crosswalks/.

2. Import CAP PDFs:

    docker exec -it cl-django python /opt/courtlistener/manage.py import_harvard_pdfs
   

   This command will use the generated crosswalk to fetch and store PDFs.

3. To test with specific parameters:

    docker exec -it cl-django python /opt/courtlistener/manage.py generate_cap_crosswalk --reporter "A.2d" --volume 100
   
   

Verification:

• Check the cl/search/crosswalks/ directory for generated crosswalk files.
• Verify that PDFs are stored either in S3 or local storage (based on configuration).
• Examine the logs for any errors or warnings during the process.

Screenshots:

CAP Crosswalk File (Sample Data):

Imported PDFs (Local Storage):

[FRodriguez18]

Looks good! Didn't approve it because I think John proposed some changes, but this looks good to me 👍

[mlissner]

@flooie I just put this on your backlog to prioritize. The background is that we want to have the Harvard PDFs imported into CL and we want to do so regularly. Grab me when you have a sec, and I can give the details before you review or assign to somebody else to review.

[quevon24]

In general, it is necessary to add typing to the functions in addition to updating the docstrings of the functions so that they are in accordance with the format used in courtlistener.

[quevon24]

It would be great if you could check the citation with get_citations() from eyecite and use the data from the returned object to filter the OpinionCluster objects, here is an example: https://github.com/freelawproject/courtlistener/blob/main/cl/corpus_importer/management/commands/harvard_opinions.py#L636

[jtmst]

I've added this, but I'll note this could affect performance when run against all the data. Run speed doesn't seem to be a huge concern at the moment, but something to note for later if the run time gets to get out of control in testing.

[jtmst]

Agreed that this will help accuracy though, thanks for the suggestion

[quevon24]

Maybe this could be in cl/lib/storage.py, what do you think @flooie?

[flooie]

Yeah - I think we should move anything storage related together