X (Twitter) User Scraper

This scraper zeroes in on public profiles to gather user details and high-engagement tweets from X (formerly Twitter). It cuts through the platform’s dynamic interface, automates the heavy lifting, and delivers structured data ready for analysis. If you need dependable Twitter data extraction, this tool keeps things simple and efficient.

     

Created by Bitbash, built to showcase our approach to Scraping and Automation!
If you are looking for X (Twitter) User Scraper you've just found your team — Let’s Chat. 👆👆

Introduction

This project automates the process of collecting user information and tweets from any public X profile. It solves the hassle of navigating dynamic, script-heavy pages by handling browser automation behind the scenes. It’s built for analysts, researchers, engineers, and anyone who needs reliable social data at scale.

Why This Scraper Matters

• Navigates live Twitter pages and interacts with dynamic UI elements.
• Pulls structured user profile data such as IDs, names, counts, and verification details.
• Captures the most-liked recent tweets with detailed engagement metrics.
• Handles lazy-loaded elements and infinite scroll automatically.
• Supports scalable crawling with adjustable limits.

Features

Feature	Description
Browser automation with Playwright	Ensures stable rendering and interaction with dynamic Twitter elements.
Crawlee-based crawling	Efficient request handling and scaling for multiple user profiles.
User data extraction	Gathers IDs, profile images, verification info, counts, and metadata.
Tweet extraction	Collects popular tweets with full engagement metrics.
Flexible configuration	Adjust starting URLs, crawl depth, and request limits.

---

What Data This Scraper Extracts

Field Name	Field Description
user.id	Unique identifier for the X profile.
user.screen_name	Public username of the account.
user.name	Display name on the profile.
user.followers_count	Number of followers.
user.friends_count	Number of following accounts.
user.profile_image_url	URL of the profile photo.
tweet.id	Unique ID of the tweet.
tweet.full_text	Complete tweet text.
tweet.favorite_count	Total likes on the tweet.
tweet.retweet_count	Retweets received.
tweet.created_at	Timestamp of when the tweet was posted.

---

Example Output

Example:

{
  "user": {
    "__typename": "User",
    "id": "VXNlcjo0NDE5NjM5Nw==",
    "rest_id": "44196397",
    "is_blue_verified": true,
    "legacy": {
      "created_at": "Tue Jun 02 20:12:29 +0000 2009",
      "favourites_count": 60807,
      "followers_count": 189827332,
      "friends_count": 662,
      "listed_count": 152087,
      "name": "Elon Musk",
      "screen_name": "elonmusk",
      "statuses_count": 47242
    }
  },
  "tweet": {
    "__typename": "Tweet",
    "rest_id": "1519480761749016577",
    "legacy": {
      "created_at": "Thu Apr 28 00:56:58 +0000 2022",
      "full_text": "Next I’m buying Coca-Cola to put the cocaine back in",
      "favorite_count": 4468299,
      "retweet_count": 625073,
      "reply_count": 182762
    }
  }
}

---

Directory Structure Tree

X (Twitter) User Scraper/
├── src/
│   ├── main.js
│   ├── crawler/
│   │   ├── twitterCrawler.js
│   │   └── playwrightClient.js
│   ├── extractors/
│   │   ├── userExtractor.js
│   │   └── tweetExtractor.js
│   ├── utils/
│   │   ├── logger.js
│   │   └── helpers.js
│   └── config/
│       └── settings.example.json
├── data/
│   ├── sample-user.json
│   └── sample-tweets.json
├── package.json
├── README.md
└── .gitignore

---

Use Cases

• Analysts use it to gather public profile metrics so they can study user influence and growth trends.
• Researchers use it to collect tweet datasets so they can perform sentiment or behavioral analysis.
• Journalists use it to reference verified statements quickly so they can support reporting workflows.
• Developers use it to integrate Twitter user data into apps so they can enrich features with social insights.
• Marketers use it to track competitor activity so they can refine content and engagement strategies.

---

FAQs

Does this scraper bypass login requirements? It works on publicly accessible data. If a page requires authentication, the scraper won’t extract those elements.

How many tweets can it collect at once? By default it targets the 100 most-liked recent tweets, but you can adjust limits in the configuration file.

Is the scraper affected by UI changes? Since it relies on live page structure, major layout changes may require updates to selectors.

Can I run multiple profiles in one job? Yes — add more profile URLs to the input list to run them consecutively.

---

Performance Benchmarks and Results

Primary Metric: Handles an average of 30–40 tweets extracted per second once the page is fully loaded. Reliability Metric: Maintains a 95%+ success rate across diverse public profiles with stable network conditions. Efficiency Metric: Optimizes browser sessions to keep memory usage moderate during long crawls. Quality Metric: Produces highly complete records with consistent field accuracy, even on profiles with heavy media content.

"Bitbash is a top-tier automation partner, innovative, reliable, and dedicated to delivering real results every time." Nathan Pennington Marketer ★★★★★

"Bitbash delivers outstanding quality, speed, and professionalism, truly a team you can rely on." Eliza SEO Affiliate Expert ★★★★★

"Exceptional results, clear communication, and flawless delivery. Bitbash nailed it." Syed Digital Strategist ★★★★★