SQS Validation Flow

---
title: SQS Validation Flow
---
flowchart LR
    filing-api["filing-api"]
    s3["`S3 Bucket`"]
    cfpb-regtech-dev-s3-queue-test["cfpb-regtech-dev-s3-queue-test queue"]
    sqs-csv-poller["sqs-csv-poller
    sqs_csv_to_parquet/sqs_listener.py
    sbl_validation_processor/csv_to_parquet.py"]
    cfpb-regtech-dev-pqs-validate["cfpb-regtech-dev-pqs-validate queue"]
    sqs-pqs-poller["sqs-pqs-poller
    sqs_parquet_validation/sqs_listener.py"]
    validator-job["validator-job
    sqs_parquet_validation/validator_job.py
    sbl_validation_processor/parquet_validator.py"]
    cfpb-regtech-dev-res-aggregate["cfpb-regtech-dev-res-aggregate"]
    sqs-res-poller["sqs-res-poller
    sqs_validation_aggregator/sqs_listener.py
    sbl_validation_processor/results_aggregator.py"]
    validator-job["validator-job
    sqs_parquet_validation/validator_job.py
    sbl_validation_processor/parquet_validator.py"]    
    filing-api---|write csv|s3
    s3---|sqs message|cfpb-regtech-dev-s3-queue-test
    sqs-csv-poller---|pull message|cfpb-regtech-dev-s3-queue-test
    sqs-csv-poller---|write pqs and .done_pqs|s3
    s3---|sqs message|cfpb-regtech-dev-pqs-validate
    sqs-pqs-poller---|pull message|cfpb-regtech-dev-pqs-validate
    sqs-pqs-poller---|create k8s job|validator-job
    validator-job---|scan pqs|s3
    validator-job---|write result pqs and .done_res|s3
    s3---|sqs message|cfpb-regtech-dev-res-aggregate
    sqs-res-poller---|pull message|cfpb-regtech-dev-res-aggregate
    sqs-res-poller---|write report csv|s3

Loading