Osiris

Requirements

• Python >= 3.6
• MongoDB >= 3.8
• Virtualenv >= 15.0.0

Create Virtual Environment

virtualenv -p python3 venv

-p #Set the python version virtualenv need to use

venv #The name of virtual  environment

Install Python Requirements

For this step you need instaled all items in Requirements and create a virtual environment using virtualenv

1. Active virtual environment

• For this we gonna to use the source command on linux

   source venv/bin/activate

  After this step the name of virtual environment will appear first on the terminal linux path like this.

   (venv)[root@localhost ~]#

2. Install Python requirements from the requirements.txt using the following command

   pip install -r requirements.txt

   Now we have installed all requirements to run the tool

If you need exit of virtualenv environment use this command.

deactivate

How to use

Start Tool

With all requirements installed is time to prepare tool for run

1. The first step is create and initialize database

   python configInit.py

After this step the tool is able to interact by command line or web interface

Command Line

The simplest interaction form to the tool with the tool, The command line allow 8 operations

Add projects to mining

python config.py --projects projects.json

where need a json formatted

Quantity of Finished Projects

python config.py --projects

Add GitHub Keys

python config.py --keys keys.json

where need a json formatted

Quantity of key in Database

python config.py --keys

Quantity of finished projects

python config.py --finished

Crawler Status

python config.py --status

status: ON/OFF projects in database completed projects GitHub keys on in database

Interface Web

To activate the Web Interface use this command.

python configWeb.py

The interface run in

http://localhost:5000/

projects CRUD can be found here

http://localhost:5000/projects

GitHub keys CRUD can be found here

http://localhost:5000/keys

Database metrics to .CSV

Can you use command

python extractDataset.py

to extract all repositories metrics of database each to .csv file in ./results in project directory for compatibility was used # instead of / in file name for each repository file then owner/repo now owner#repo

JSON Valid Format

projects.json

The field visited is True only if the project is completed Example json for one project

{
	"visited":false,
	"name":"owner/repo"
}

Example json for many project

[
	{"visited":false,"name":"owner/repo"},
	{"visited":false,"name":"owner/repo"},
	{"visited":false,"name":"owner/repo"},
	{"visited":false,"name":"owner/repo"}
]

keys.json (GitHub api key)

Example json for one project

{
	"client_id":"xxxxxxxxxxxxxxxxxxx",
	"client_secret":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
}

Example json for many project

[
	{
		"client_id":"xxxxxxxxxxxxxxxxxxx",
		"client_secret":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
	},
	{
		"client_id":"xxxxxxxxxxxxxxxxxxx",
		"client_secret":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
	},
	{
		"client_id":"xxxxxxxxxxxxxxxxxxx",
		"client_secret":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
	},
	{
		"client_id":"xxxxxxxxxxxxxxxxxxx",
		"client_secret":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
	}
]