fix: add csv-writer dependency for market-trends-scraper #4
+887
−2,258
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The market-trends-scraper.js script requires the csv-writer package which was missing from package.json dependencies.
- Add push-to-notion.js script to push market trends to Notion database - Update weekly-market-trends.yml workflow to push to both Google Sheets and Notion - Add @notionhq/client and other dependencies to package.json - Update .env.example with Notion configuration variables - Remove push-to-notion.js from .gitignore (was listed as old helper script) Requires NOTION_TOKEN and NOTION_TRENDS_DB_ID secrets in GitHub.
Remove redundant code to consolidate repository: - Remove intelligence-hub/ subfolder (duplicate of root files) - Remove scrapers/ folder (legacy Puppeteer scrapers, replaced by simplified mock-data versions in root) Active scripts remain in root directory: - market-trends-scraper.js - icp-monitor.js - concurrent-tracker.js - All other trackers and utilities This cleanup reduces confusion and maintenance overhead.
Script to remove duplicate scrapers from recruitin-content-intelligence-system that are now maintained in this repo (intelligence-hub).
Code reviewNo issues found. Checked for bugs and CLAUDE.md compliance. |
- Remove mandatory Google credentials decode step - Make Google Sheets push conditional (skip if no credentials) - Add continue-on-error to Sheets push so Notion still runs - Support both base64 and raw JSON in GOOGLE_CREDENTIALS
- Remove intelligence-hub.yml (referenced deleted scrapers/ folder) - Fix weekly-icp-activity.yml: remove mandatory credential decode step - Fix weekly-concurrent-activity.yml: remove mandatory credential decode step - Make Google Sheets push conditional and continue-on-error The scrapers now run from individual weekly-*.yml workflows.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The market-trends-scraper.js script requires the csv-writer package
which was missing from package.json dependencies.