Minellius

A experimental github data analysis solution
Group project of COMP-3002, 2018 fall, HITsz.

👀 Overview

Minellius, a experimental github data analysis solution, which designed to combine data collection, persistence, integration, analysis, visualization, real-time monitoring, trend tracking, originating from a group project of COMP-3002, 2018 fall, HITsz.

Minellius aims to help users understand the history, current, trend, ups and downs of github and even the open source world, enhance the sense of the times and the cultural aesthetics of the community; adjust the path of developers' learning and self-improvement to better integrate and participate in the development of community resources and culture.

📑 Table of Contents

• 👀 Overview
• 📑 Table of Contents
• 🔨 How It Works
• ⭐️ Features
• 👓 Example Snapshot
• 🔧 Installation & Usage
• 💻 Platform Support
• 🌏 Language Support
• 👍 Acknowledgement
• 👉 Source Code Reference (Main)
• ❔️ FAQ
• 📙 Docs
• 🏷️ Resource & Reference
• 😆 Contributors
• 📜 License

🔨 How It Works

Crawl data from Gharchive and Github Api, then Aggregate, store, and visualize them.

⭐️ Features

1. Data Visualization with Charts and Tables
2. Streaming Crawling and Analysis, Minimal Processing Space Requirement
3. Basic User System with Bearer JWT Auth and Github Oauth Auth
4. Secure Store and Transmission
5. Continuous Integration
6. Auto Recovery and Email Alert
7. Autogen Documents
8. I18n (Multi-language)
9. Multi-platform
10. Antd Design

👓 Example Snapshot

Current Snapshot

Period Snapshot

🔧 Installation & Usage

Client

From Binary (Recommended)

Please see release page, and download the latest version for your platform. By default, we provide windows version (portable exe) and linux version (AppImage). And because of project deadline limitation, we only provide 1280x720px client.

From Source Code (Not Recommended)

1. First, enter into client directory and install the dependencies.Sure, you need node.js environment and avaiable network (for china users).

cd client
npm install

2. Then build client for your platform.

# auto select suitable platform (recommended)
npm run electron:local
# or manually select platform
npm run electron:windows
npm run electron:linux
npm run electron:mac # no test

3. You can find output in client/release directory, then copy and use it.

From Source Code To Dev (Only to developers)

Again, install then start hot-reload dev server on http://localhost:4200 with electron. If you need chrome devtools, edit client/src/main.ts, set devTools to be true and restart it.

cd client
npm install
npm run start

Server

Important: Target server of our client has been set as our server, if you need deploy your server, you should edit it in the client source and rebuild the client as well.

With Docker-compose (Recommended)

0. Sure, you need docker and docker-compose environment.

1. First, copy docker-compose.sample.yaml to docker-compose.yaml and edit environment variables in it or export them to your current shell.

2. Copy monitor/.docker-alertd.sample.yaml to monitor/.docker-alertd.yaml. Edit alert email information in it.

3. Optional: We recommend postgresql database to deploy our service, if you use it, you can create a new database and import init tables from deploy/database/postgresql/minellius.sql. Then you can skip later database import.

4. Optional: Because of huge amount of data, our 100GB database has no more space, so we cancel its auto start function in docker mode. To crawl period data, you should manually edit crawler/Dockfile and add crawler/src/go.py to the cmd field. For new fishs to docker or needing more information, see Dockerfile Reference.

5. Then easily build and start.

docker-compose build
docker-compose up

6. Waiting minellius-server container init the database tables, you need import init insert data from deploy/database/common. If you use postgresql, you can easily import all (see step 3) .sql files, else you need manually import one by one now.

7. If you need https and http2.0, install and start a nginx web server, copy our configuration file to its vhost directory, editting to fit yourself. And if you need free https cert, visit Let's encrypt.

Manually Deploy (Not Recommended)

Important: if you insist on manually deployment, you will lose the automatic email alert, and the automatic restart of the crawler will depends all on the third-party monitoring tool you use.

0. export need enviroment variables, see in docker-compose.sample.yaml environments fields.

1. Optional: We recommend postgresql database to deploy our service, if you use it, you can create a new database and import init tables from deploy/database/postgresql/minellius.sql. Then you can skip later database import.

2. minellius-server, ensuring you have node.js installed.

cd server
npm install
npm run start:prod
# or for developement use
npm run start:dev

4. database import - waiting minellius-server container init the database tables, you need import init insert data from deploy/database/common. If you use postgresql, you can easily import all (see step 3) .sql files, else you need manually import one by one now.

5. minellius-crawler, ensuring you have basic python 3.6+ tools installed. And you need daemon tools to help you run them as services. Without docker, we recommend pm2.

cd crawler/
pip install -r requirements.txt
cd src
python current.py
# if you need period
python go.py

6. If you need https and http2.0, install and start a nginx web server, copy our configuration file to its vhost directory, editting to fit yourself. And if you need free https cert, visit Let's encrypt.

💻 Platform Support

Windows	Linux	Mac
10+ (7+ possible)	3.20+ With AppImage	Need manually build, no test now

🌏 Language Support

1. English
2. 简体中文

   More languages are coming soon...

👍 Acknowledgement

Electron	Angular	NG-Zorro	Nestjs	Docker

👉 Source Code Reference (Main)

1. client
   • infra - angular + electron + ant zorro
   • components - client/src/app/**/*.component.ts
   • providers & services - client/src/app/providers/*.service.ts
   • dtos - client/src/app/dtos/*.dto.ts
   • functools - client/src/app/functools/*.functool.ts & innerCode
   • directives - client/src/app/directives/*.directive.ts
   • interfaces - client/src/app/interfaces/*.interface.ts & innerCode
   • animations - client/src/app/animations/*.animations.ts
   • style - client/src/app/**/*.scss & innerCode
   • module - client/src/app/**/*.module.ts
   • router - client/src/app/app-routing.module.ts & routeService & routerService
   • guards - client/src/app/guards/auth.guard.ts
   • current - client/src/app/components/current*.* & client/src/app/providers/current.service.ts
   • period - client/src/app/components/period*.* & client/src/app/providers/period.service.ts
   • user - client/src/app/componets/log*.* & client/src/app/providers/user.service.ts & client/src/app/components/control.component.ts
   • helper - client/src/app/components/helper
   • i18n - TranslateService & src/app/providers/custom-translate.service.ts & src/assets/i18n/*.json
   • introduction - src/app/components/{home,video}.components.*
   • img & video & geo - src/assets
   • test - **/*.spec.ts
   • bootstrap - main{.ts,.js} | index.html
   • config
   • ...
2. server
   • nodejs + nest + native python + postgresql
   • auth - server/src/auth/**/* & server/src/role/**/*
   • user & group & permission ... - server/src/role/**/*
   • data - server/src/data/**/*
   • database - server/src/database/**/* & deploy/database/* & **/*.entities.ts & TypeOrmModule & **/*.service.ts
   • config - **/*-config.interface.ts && **/*-config{.*} && environments variables & **/*{.json,.yaml,.yml} ...
   • https & http2.0 - minellius.nginx.sample.conf + nginx proxy pass
   • monitor - monitor/.docker-alertd.yaml + docker-alertd + docker-compose.yaml + docker-compose + docker + pm2.json + pm2
   • cralwer - crawler/src/*.py
   • document - server/src/**/*.dto.ts & server/src/**/*.controller.ts & swagger
   • controller - server/src/**/*.controller.ts
   • service - server/src/**/*.service.ts
   • module - server/src/**/*.module.ts
   • ...

❔️ FAQ

1. The range of the period event data you provide?

We provide data from 2015.01.01 to 2018.12.26. But since we don't have a low-cost way to store such a large amount of data, after the end of the semester, we will only provide the last month of data as the basic demo of our application service. And we will provide download link for that data, you can find them in resource section.

📙 Docs

• Server Api Document
• Initiation Document (Chinese Assignment)
• Software_Requirements_Specification.pdf (Chinese Assignment)

🏷️ Resource & Reference

1. Course Information
2. Zipped Sql File from 2015.01.01 to 2018.01.26 Download
3. Promotional Video (Short Version)
4. Promotional Video (Full Version)
5. Presentation Slides (Chinese Assignment)

😆 Contributors

Zhou Yeheng	Cui Tianyu	Cao Weihan	Lin jintao	Fang An

📜 License

This software is licensed under the MIT

MIT License

Copyright (c) 2018 Zhou Yeheng, Cui Tianyu, Cao Weihan, Lin Jintao, Fang An

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.