Expand web crawl unit tests

[ppinchuk]

Slowly working toward the goal of 80%+ coverage

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 43.32%. Comparing base (47acad4) to head (de15329).

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #350      +/-   ##
==========================================
+ Coverage   40.02%   43.32%   +3.30%     
==========================================
  Files          45       45              
  Lines        4240     4240              
  Branches      380      380              
==========================================
+ Hits         1697     1837     +140     
+ Misses       2523     2378     -145     
- Partials       20       25       +5     

Flag	Coverage Δ
unittests	43.32% <ø> (+3.30%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.