conference collector

collect conference data from http://www.guide2research.com/ mongodb & mysql support

install

1. install crontab(archlinux)

   sudo pacman -S --needed cronie
   sudo systemctl start cronie

   install crontab(centos7)

   systemctl start crond

2. install dependency(centos7)

   sudo yum -y install https://centos7.iuscommunity.org/ius-release.rpm
   sudo yum -y install python36u python36u-pip python36u-devel
   sudo pip3.6 install --upgrade mysqlclient scrapy

usage

1. change usr, passwd and pipline in settings.py

2. if using mysql, create database and table refer to create.sql

3. collect one time

   python main.py
   #centos7
   python3.6 main.py

4. update every day

   01 01 * * * python /home/root/top/main.py

reference

crontab

archwiki