[2023-10-21] Introducing Generalization Tests for Improved Parser Accuracy

[Elijas]

Hello sec-parser Community,

We're excited to introduce a key update to the sec-parser project: a feature we call "Generalization Tests."

What are Generalization Tests?

Simply put, Generalization Tests evaluate the parser's performance across a variety of financial reports. These tests scrutinize a broad array of documents to ensure consistent and accurate functioning of the parser.

Why is this Important?

Up to this point, our focus has been on unit tests and end-to-end tests to maintain the parser's quality. Generalization Tests provide an extra layer of confidence. They help us quickly spot any issues the parser may have with specific types of documents.

How Does it Work?

We utilize a tool named pytest to execute these Generalization Tests. The tests run in parallel, allowing multiple reports to be checked simultaneously. As of now, we have 73 unique 10-Q reports in our test database. These tests can be completed in less than 10 seconds on a laptop. For those interested in extending the database, we offer a script that makes it easy to add more reports by simply entering the stock tickers you wish to include.

How to Run Generalization Tests?

Here's how to run these tests:

1. Open Your Terminal: Navigate to the sec-parser project directory.

2. Execute the Test Command: Enter the following command to initiate the tests:

   task generalization-tests

Note
To run a specific test, open Taskfile.yml and locate the generalization-tests command. Copy this command and paste it into your terminal, making any adjustments you need. For example, you could replace tests/generalization/ with a specific test file like tests/generalization/processing_steps/test_top_level_section_title_classifier.py.

Note
These tests are primarily for development and are not part of the regular CI/CD pipeline, nor are they included in the end-to-end tests or unit tests.

Updated Test Files

The project's file hierarchy now includes a new folder dedicated to Generalization Tests. You'll find it at tests/generalization/ in the project tree.

tests/
├── generalization/
│   ├── __init__.py
│   ├── conftest.py
│   ├── processing_steps/
│   │   └── test_top_level_section_title_classifier.py
...

How to Contribute and Stay Updated

For the most accurate and current guidelines on how to contribute, test, and modify the sec-parser project, please always consult our sec-parser Contributor Guide. Your feedback and contributions are always welcome to make sec-parser even better. Should you encounter any issues or have suggestions, we invite you to share them with the community.

Warm regards,
Elijas