KLV Crawler

This crawler can crawl tens of thousands of apps information from the Xiaomi App Store, and save them into the MongoDB. It is based on Python + Scrapy.

Getting Started

Prerequisties:

You'll need to install:

• Python (2.7.10)
• scrapy(python lib)
• pymongo(python lib)
• MongoDB (3.0.0+)

Installing

Python(Mac OSX)

sudo pip install scrapy
sudo pip install pymongo

MongoDB(CentOS7)

yum -y install mongodb-org mongodb-org-server
systemctl start mongod

Usage

Download

git clone https://github.com/BitTigerInst/KLV_Crawler.git

Run

cd KLV_Crawler
scrapy crawl cnproxy
scrapy crawl vkea_xiaomi

Team Members

Vkea95

License

This project is licensed under the MIT License - see the LICENSE.md file for details

Project Information

• category: big data
• team: KLV Team
• description: Crawling information from Xiaomi APP store.
• stack: python, scrapy, mongodb