Add parallelized-embedding-pipeline #403
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 2 commits
August 28, 2025 12:41
- Enhanced README.md with workflow diagram and detailed architecture - Added complete example-workflow.json with all required metadata - Created resources folder with workflow diagram and author photos - Added professional author information for Solomon Ojo and Dave Horne - Included comprehensive deployment guides and resource links - Ready for AWS Step Functions workflows collection contribution
- Complete document vectorization pipeline implementation - Enhanced README.md with workflow diagram and comprehensive documentation - Added example-workflow.json with all required metadata for AWS samples - Included resources folder with workflow diagram and author photos - Added professional author information for Solomon Ojo and Dave Horne - All Lambda functions, deployment scripts, and configuration files - Ready for production use and AWS Step Functions workflows collection
bfreiberg
suggested changes
Sep 18, 2025
|
|
||
| ## Architecture | ||
|
|
||
|  |
There was a problem hiding this comment.
It would be great to add a sentence to the overview above how the different "layers (Raw, Cleaned, and Curated) fit into this.
|
|
||
| ### Workflow Diagram | ||
|
|
||
| The following diagram illustrates the complete Step Functions workflow for the document vectorization pipeline: |
There was a problem hiding this comment.
Suggested change
| The following diagram illustrates the complete Step Functions workflow for the document vectorization pipeline: | |
| The following diagram illustrates the complete AWS Step Functions workflow for the document vectorization pipeline: |
There was a problem hiding this comment.
Please use the full service name on each first mention
Comment on lines
+55
to
+59
| - **S3 Bucket**: Stores documents in different stages (raw, cleaned, curated) | ||
| - **SQS Queue**: Handles document processing events | ||
| - **Step Functions**: Orchestrates the document processing workflow | ||
| - **Lambda Functions**: Process documents and generate embeddings | ||
| - **Aurora PostgreSQL**: Database with pgvector extension for storing vectors |
There was a problem hiding this comment.
Add correct service name prefixes
| 4. **To test database connectivity:** Use `./test-connection.sh` | ||
| 5. **To test full pipeline functionality:** Use `./test-functionality.sh` | ||
|
|
||
| ### Deployment Scripts |
There was a problem hiding this comment.
Is this table really necessary given the section above?
| 1. VPC with public and private subnets | ||
| 2. Aurora PostgreSQL database with pgvector support | ||
| 3. Lambda functions for document processing | ||
| 4. Step Functions for workflow orchestration |
There was a problem hiding this comment.
Suggested change
| 4. Step Functions for workflow orchestration | |
| 4. Step Functions state machine for workflow orchestration |
| Properties: | ||
| CodeUri: functions/GetByteRangesFunction | ||
| Handler: app.lambda_handler | ||
| Runtime: python3.9 |
There was a problem hiding this comment.
see above and applies to other functions too
| Properties: | ||
| LayerName: shared-libraries | ||
| Description: Common libraries and secure utilities for document processing | ||
| ContentUri: functions/shared |
| ManagedPolicyArns: | ||
| - arn:aws:iam::aws:policy/service-role/AWSVPCFlowLogsRole | ||
|
|
||
| VPCFlowLogs: |
There was a problem hiding this comment.
Consider removing this for a demo
| Value: !Sub ${AWS::StackName}-vpc-flow-logs | ||
|
|
||
| # Enhance RDS Cluster Configuration | ||
| VectorDBCluster: |
There was a problem hiding this comment.
didn't you already define this above?
| PreferredBackupWindow: "03:00-04:00" | ||
| PreferredMaintenanceWindow: "mon:04:00-mon:05:00" | ||
|
|
||
| VectorDBInstance: |
There was a problem hiding this comment.
didn't you already define this above?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This project contains a document vectorization pipeline using AWS services, specifically designed to process text, PDF, and Word documents, extract their content, generate vector embeddings in parallel and store them in a PostgreSQL database optimized for vector searches.
Thank you
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.