Feat/url content generation #11
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit introduces the capability to generate viral content based on the textual content of a user-provided URL. Key changes include: - Modified `services/groq.js` to adapt its AI content strategy prompting. It can now generate strategies based on provided text content from a URL, in addition to the existing topic-based generation. - Added a new `services/webExtractor.js` service. This service uses Playwright to navigate to a given URL and extract its main text content. - Integrated the `WebExtractorService` into `server.js`. The `ViralContentSystem` now has a new method `createViralContentFromUrl` that orchestrates fetching content from a URL, generating an AI strategy based on that content, and then processing it through the existing media creation and social media distribution pipeline. - Updated the `serviceRegistry` in `server.js` to include and manage the new `WebExtractorService`. - Modified the existing `/mcp/viral-content` API endpoint to support a new method, `create_viral_content_from_url`. This allows clients to trigger the new URL-based content creation flow. Conceptual data structures for future user and link management have also been outlined, paving the way for further development of user-specific features like toggling auto-upload and managing link lists.
… done so far and provide feedback for Jules to continue.
This commit incorporates significant improvements towards production readiness, focusing on comprehensive error handling, system resilience, and enhancing scalability and performance through asynchronous processing. **1. Comprehensive Error Handling & Resilience:** - **Service-Level Enhancements:** - `services/webExtractor.js`: I've implemented retry logic for Playwright operations, with contextual error logging. - `services/groq.js`: I've added retry logic for Groq API calls, specific handling for API error codes (401, 403, 429), validation of API responses, safe JSON parsing, and API key presence check. - **Pipeline Orchestration (`ViralContentSystem` in `server.js`):** - I've added granular try-catch blocks and contextual error logging around each service call within the content creation pipelines. This improves observability and helps pinpoint failures. - **Standardized API Error Responses (`server.js`):** - I've refined the `/mcp/viral-content` endpoint for robust validation of methods and parameters. - I've ensured consistent JSON-RPC error responses and appropriate HTTP status codes (4xx for your client errors, 5xx for server errors). - **General Recommendations & Testing:** - I've outlined error handling guidelines for other services. - I've documented testing considerations for the new error handling mechanisms. **2. Scalability & Performance:** - **Job Queue System (BullMQ & Redis):** - I've introduced BullMQ with Redis for background job processing. - I've created `config/redisConfig.js` for Redis connection settings. - I've developed `lib/queue.js` to initialize and export a configured BullMQ `Queue` instance with default job options (retries, backoff, etc.). - **Asynchronous API Endpoint (`server.js`):** - I've refactored the `/mcp/viral-content` endpoint to submit jobs to the BullMQ queue. - The API now returns a `202 Accepted` response with a `jobId` immediately, instead of processing tasks synchronously. - **Background Processing (`worker.js`):** - I've created `worker.js` to process jobs from the BullMQ queue. - I now handle the actual content creation tasks asynchronously. - **`ViralContentSystem` Refactoring (`core/viralSystem.js`):** - I've moved `ViralContentSystem`, `serviceRegistry`, and service initialization logic to a new `core/viralSystem.js` module. - This allows both `server.js` (API) and my background processing to use independent, fully functional instances of `ViralContentSystem` with all dependent services (AI, media, etc.) correctly initialized. - **Background Processing Integration & Logging:** - I've ensured my background processing correctly uses the refactored `ViralContentSystem`. - I've implemented comprehensive logging in my background processing for job lifecycle events (start, completion, failure) with job IDs and error details. - **Deployment & Testing (Conceptual):** - I've outlined conceptual changes for deployment configurations (e.g., Procfile, Docker) to manage new web and background processes. - I've documented testing strategies for the asynchronous, queue-based architecture. These changes significantly improve the application's robustness, maintainability, and its ability to handle tasks efficiently, paving the way for further production hardening.
… done so far and provide feedback for Jules to continue.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.