The project is based on Go and Vue to build a management system for sensitive information detection. For the full introduction, please refer to articles and videos. For now, all the scans are only targeted to the public environments, not local environments.
For the usage of GShark, please refer to the wiki.
- Support multiple platforms, such as GitLab, GitHub, Searchcode, and Postman
- Flexible menu and API permission settings
- Flexible rules and filtering rules
- Utilize gobuster for subdomain brute force
- Easy-to-use management system
- Support for Docker deployment
git clone https://github.com/madneal/gshark
cd gshark
docker-compose build && docker-compose up
- Nginx
- MySQL(version above 8.0)
It is recommended to deploy the Front-End project using nginx. Place the dist folder in /var/www/html, and adjust the nginx.conf file (/etc/nginx/nginx.conf for Linux) to set up reverse proxy for the backend service. For detailed deployment tutorials, you can watch videos on bilibili or youtube. For deployment on Windows, refer to this link.
Can use nginx -t to locate the nginx.conf file, then modify the nginx.conf:
// config the user accoring to your need
user www www;
worker_processes 1;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
sendfile on;
keepalive_timeout 65;
server {
listen 8080;
server_name localhost;
location / {
autoindex on;
root html;
index index.html index.htm;
}
location /api/ {
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
rewrite ^/api/(.*)$ /$1 break;
proxy_pass http://127.0.0.1:8888;
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root html;
}
}
include servers/*;
}
The deployment work is straightforward. Find the corresponding version zip file from releases.
Unzip and copy the files inside dist to /var/www/html folder of Nginx.
unzip gshark*.zip
cd gshark*
mv dist /var/www/html/
# for Mac
mv dist /usr/local/www/html/
Start the Nginx and the Front-End is deployed successfully.
Tip
If you installed Nginx by Homebrew, you need to stop Nginx by:
brew services stop nginxFor the first time, you need to rename config-temp.yaml to config.yaml. Then you can run the binary file gshark directly. Then access to localhost:8080 for local deployment.
If you have not initial the database in the before, you will be redirec to the database initial page at first.
For the scan service, it's necessary to config the corresponding rules. For example, Github or Gitlab rules.
For the incremental deployment, sql.md should be executed for the corresponding database operations.
git clone https://github.com/madneal/gshark.git
cd server
go mod tidy
mv config-temp.yaml config.yaml
go build
./gshark
or
go run main.go
cd ../web
npm install
npm run serve
To execute the scan task for GitHub, you need to add a GitHub token for crawl information in GitHub. You can generate a token in tokens. Most access scopes are enough. For the GitLab search, remember to add a token too.
Obtain the postman.sid cookie:
For the Github or Gitlab rule, the rule will be matched by the syntax in the corresponding platforms. Directly, you config what you search at GitHub. You can download the rule import template CSV file, then batch import rules.
Filter is only addressed to GitHub search now. There are three classes of filters, including extension, keyword, sec_keyword. For extension and keyword, they can used for blacklist or whitelist.
For more information, you can refer to this video.
You are supposed to rename config-temp.yaml to config.yaml and config the database information and other information according to your environment.
- 2021-11-12 发布文章《GShark:多平台的敏感信息监测工具》
修复
- 组件版本升级
- 将后端服务与扫描服务分离
- 改进 Docker 镜像
修复
- 组件依赖升级
- 修复部分安全问题
- 丰富使用说明 README
- 恢复本地 build 脚本
- 简化项目，移除 autocode
修复
- 组件依赖升级
- 修复 API 列表排序问题
- 移除任务列表
修复
- 修复数据库初始化的问题
- 修复 Postman 搜索结果格式
- 前端组件安全升级
修复
- 组件版本的安全升级
- 修复 golang User-Agent 被 Postman 禁用的问题
- 修复前端缺少 postman 规则选项问题
新增
- 增加 Docker 版本 MySql 的数据持久化
- GitHub client 可以忽略 TLS 错误