Skip to content

v2.6.2
Compare
Choose a tag to compare
@sakan811 sakan811 released this 11 Jun 09:31
· 555 commits to master since this release
v2.6.2
ed00d44

Performance & Multithreading:

  • Increased ThreadPool workers from 5 to 9.
  • Added threading.Lock and fine-tuned thread execution with sleep intervals for stability.

Error Handling & Logging:

  • Improved exception handling, especially for database deletion and DataFrame creation errors.
  • Refined log message levels for better clarity and debugging.

Driver & Wait Time Adjustments:

  • Adjusted web driver wait times and removed unnecessary wait times from certain scrapers.

Scraping Enhancements:

  • Introduced logic to check if all dates were scraped, with additional safeguards to prevent scraping past dates.
  • Added more default values to the scrape_missing_dates function for robustness.

New Features:

  • Introduced the to_sqlite flag for flexibility in data handling.
  • Added multiple parsers to control scraper usage, and refactored key scripts (scrape.py, thread_scrape.py, etc.).
Assets 2
Loading