Skip to content

Tools for scraping Ethereum events directly from an Ethereum Node in Python

Notifications You must be signed in to change notification settings

bhemen/ethereum-scraping

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

20 Commits
 
 
 
 
 
 
 
 

Repository files navigation

Scraping blockchain data

General resources

General tutorials on how blockchains work can be found in the wiki

Getting started

Most of these scripts connect to a blockchain node (validator) and request information through their API. There are a number of services that provide (limited) free access to nodes.

Before you get started, sign up for free accounts at

When you sign up, they'll give you an endpoint that you can copy/paste into the scripts.

Alternatively, you can run your own Ethereum node. I recommend Erigon as your execution client, and I've used Lighthouse as a consensus client. Lighthouse works well, but I haven't tried any other consensus clients. Here's a good tutorial on installing Erigon.

web3.py

The web3 library is a great tool for scraping data from Ethereum.

EatTheBlocks has a good tutorial on the Javascript Web3 library (although we use the Python Web3 library, the concepts are very similar).

Fungible and non-fungible tokens

The ERC20 standard governs how most "fungible" tokens behave. Read the documentation for ERC20 tokens

The ERC721 standard governs how most "non-fungible" tokens behave. Read the documentation for ERC721 tokens

The 'Tokens' Tutorial on EatTheBlocks is another good place to learn about interacting with Ethereum Tokens

eventscanner

The web3 library provides a tool "eventscanner" that can scan the blockchain for events from a specific contract. The script get_contract_events.py provides a modified version of the web3 event scanner that can scan the Ethereum blockchain for all events emitted by a specific contract This performs alright for contracts that don't emit too many events (e.g. Compound governance) but performs badly for contracts that emit a lot of events (e.g. all USDC transfers). If you're interested in the web3 eventscanner more generally, here's a walkthrough of how it works.

event-scraper

The script get_contract_logs.py allows you to scrape the Ethereum blockchain for events emitted by a specific contract and write the results to a csv. The functionality is very similar to eventscanner, but get_contract_logs.py is much simpler Ethereum nodes are very finicky, and the different scripts

About

Tools for scraping Ethereum events directly from an Ethereum Node in Python

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages