Fix broken profile scrapers due to upstream site changes#90
Open
robmoss344 wants to merge 1 commit intoLucksi:masterfrom
Open
Fix broken profile scrapers due to upstream site changes#90robmoss344 wants to merge 1 commit intoLucksi:masterfrom
robmoss344 wants to merge 1 commit intoLucksi:masterfrom
Conversation
- Instagram: Picuki.com redirected to Cloudflare-protected TikTok viewer; replaced broken HTML scraping with clear unavailable message - Twitter: nitter.net returns empty responses; replaced with unavailable message - TikTok: updated URLebird selectors (h2.text-dark, img.u-image, div.stats-header) to match redesigned layout; now correctly extracts username, followers, and stats - GitHub: fixed silent error handler to expose actual exception message - GitLab: added empty-list guard so missing users exit cleanly instead of IndexError - Chess.com: added HTTP status check; exits gracefully on non-200 responses - Gravatar: added 404 handling and switched optional keys to .get() to prevent KeyError on minimal profiles - Tellonym: Cloudflare 403 blocks all requests; replaced with unavailable message - Joinroll: public API requires auth (HTTP 500); replaced with unavailable message All scrapers now either return data or fail with an informative message. Tested against username 'torvalds' — TikTok, GitHub, NGL, and Chess.com confirmed working. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
h2.text-dark,img.u-image,div.stats-header) to match redesigned layout; correctly extracts username, followers, likes, following, and profile picIndexError.get()to preventKeyErroron minimal profilesTest plan
torvaldsNoneTypecrashespython3 -m py_compile Core/Support/Username/Scraper.py🤖 Generated with Claude Code