[2023-10-14] Introducing e2e Testing in sec-parser: Validating Against Real-world Data

[Elijas]

Hello Alphanome.AI Community,

We are pleased to bring you another significant advancement in the sec-parser project: the launch of End-to-End (e2e) Testing. This comes on the heels of achieving 100% unit test coverage, and it adds an extra dimension of reliability to the project by subjecting it to real-world data scenarios.

The Importance of End-to-End Testing

While unit tests have been instrumental in maintaining code quality by focusing on individual components, e2e testing takes it a step further by validating the parser against complex, real-world data. This ensures that sec-parser not only works in theory but also performs reliably when faced with actual SEC EDGAR HTML documents.

Introducing the sec-parser-test-data Repository

To facilitate this real-world testing, we've established a separate sec-parser-test-data repository. This repository contains a curated collection of real-world input documents and their corresponding expected outputs. This enables contributors to run in-depth tests without adding unnecessary bulk to the main codebase.

How to Execute e2e Tests for sec-parser

When you make any changes to the sec-parser codebase, it's critical to ensure that your modifications don't negatively impact the tool's reliability or efficiency. Follow these steps to run the e2e tests and validate your changes:

1. Navigate to the Project Directory

Open your terminal and navigate to the sec-parser directory.

Note
The e2e data will be automatically cloned from GitHub when you run the verification command; no need for a separate download step.

2. Execute Verification Command

Run the following command to execute e2e tests:

task e2e-verify

Expected Outcomes: Success

If the dataset matches, the output will look like this:

Expected Outcomes: Failure

If discrepancies are found, the output will indicate the issues:

Things to note

• Note that certain performance metrics, such as parsing time, are also tracked to ensure efficiency as the project grows.
• These e2e tests are part of our CI/CD pipeline and run automatically after each commit to prevent regressions.
• For detailed instructions and guidelines, please refer to our Contribution Guide.

Updating the e2e Dataset

To align the e2e dataset with improvements to sec-parser, execute:

task e2e-generate

Please include the sec-parser commit hash in your commit message for version tracking. For detailed instructions, refer to the Contribution Guide.

Final Thoughts

We're eager to hear your feedback, answer any questions you may have, and encourage your contributions. Our commitment to excellence drives us, and your involvement will help make sec-parser an even more robust and reliable tool for the community.

Warm regards,
Elijas