[Feature] Bulk Transaction Import via CSV and JSON with Validation Engine #578
Description
Description
Currently, users must enter transactions manually, which is time-consuming and acts as a barrier for new users migrating from other platforms or attempting to digitize bank statements. To address this, we need to develop a bulk import feature that accepts CSV and JSON files. This feature will include a backend processing engine to parse files, validate data integrity, and handle duplicate entries before persisting data to the database.
Proposed Solution
The solution involves creating a file upload interface on the client side and a robust processing controller on the server side. The system should handle the following:
- File Parsing: Parse incoming CSV (using a library like
csv-parser) and JSON files. - Validation Engine: Validate data types (e.g., ensuring "amount" is a number, "date" is valid) and ensure required fields are present.
- Duplicate Handling: Check existing records to prevent double-entry of transactions.
- Batch Insertion: Efficiently insert valid records into the MongoDB database.
Technical Implementation Details
- Frontend: Create
ImportModal.jsto handle file selection and FormData submission. - Backend Routes: Add a new route in
transaction.js(e.g.,POST /api/v1/transactions/import) to handle the file upload. - Controller Logic: Create or update
importController.js. This file should contain the logic to iterate through the parsed data, sanitize inputs, and perform bulk write operations.
Likely Affected Files
server/routes/transaction.jsserver/controllers/importController.jsclient/src/components/Transactions/ImportModal.js
Verification Steps
To verify the implementation of this feature, please follow these steps:
- Environment Setup: Ensure the backend server and frontend client are running locally.
- Prepare Data: Create a sample
test_data.csvandtest_data.jsoncontaining valid transaction fields (date, amount, category, description). Create a second set of files with invalid data types to test validation. - UI Testing: Navigate to the Transactions page, open the Import Modal, and upload the valid CSV file.
- Database Verification: Check the MongoDB collection to confirm that all rows from the CSV were inserted correctly.
- Validation Testing: Attempt to upload the file with invalid data. Verify that the server returns an appropriate error message and does not crash.
- Duplicate Testing: Upload the same valid CSV file a second time. Verify that the system identifies duplicates and either skips them or prompts the user (depending on implementation logic), ensuring no duplicate records are created in the database.