Skip to content

Commit 387d57e

Browse files
megvii-mgemegvii-mge
committed
docs(build): add Dockerfile and prebuilt doc
1 parent bcfbc53 commit 387d57e

10 files changed

+353
-17
lines changed

‎Dockerfile

+31
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
FROM ubuntu:18.04
2+
LABEL maintainer=megengine@megvii.com
3+
4+
RUN apt update \
5+
&& apt install -y curl \
6+
&& apt install -y ffmpeg \
7+
&& apt install -y yasm \
8+
&& apt install -y clang \
9+
&& apt install -y redis-server \
10+
&& apt install -y python3 \
11+
&& apt install -y python3-pip \
12+
&& apt install -y git \
13+
&& apt install -y vim \
14+
&& apt install -y build-essential
15+
16+
RUN curl --proto '=https' --tlsv1.2 https://sh.rustup.rs -sSf -o run.sh \
17+
&& chmod a+x run.sh \
18+
&& ./run.sh -y \
19+
&& export PATH=$HOME/.cargo/bin:${PATH} \
20+
&& cargo --version
21+
22+
RUN mkdir -p $HOME/megflow-runspace
23+
WORKDIR $HOME/megflow-runspace
24+
COPY . $HOME/megflow-runspace/
25+
26+
RUN PATH=$HOME/.cargo/bin:${PATH} \
27+
&& cargo build \
28+
&& cd flow-python \
29+
&& python3 setup.py install --user \
30+
&& cd examples \
31+
&& cargo run --example run_with_plugins -- -p logical_test

‎README.md

+21-3
Original file line numberDiff line numberDiff line change
@@ -11,20 +11,38 @@ MegFlow 是一个面向视觉应用的流式计算框架, 目标是简单、高
1111
- 支持 demux/reorder/transform 等通用函数式组件
1212
- Python 插件内置有栈协程，不依赖 asyncio
1313
- 基础测试工具，支持插件沙盒，用于单测插件
14-
14+
1515
## HowTo
16-
* [how to build and run in 15 minutes](docs/how-to-build.zh.md)
16+
* [how to run in 15 minutes](docs/how-to-run-in-15-minutes.zh.md)
17+
* [how to build with docker](docs/how-to-build-with-docker.zh.md)
18+
* [how to build from source](docs/how-to-build-from-source.zh.md)
1719
* [how to add my service](docs/how-to-add-graph.zh.md)
1820
* [how to add plugins](docs/how-to-add-plugins.zh.md)
1921
* [how to optimize and debug](docs/how-to-debug.zh.md)
2022
* [how to contribute](docs/how-to-contribute.zh.md)
2123
* [FAQ](docs/FAQ.zh.md)
2224

25+
## Current Support List
26+
27+
| 系统环境 | 支持情况 |
28+
| - | - |
29+
| Windows 10 WSL ubuntu18.04 | ✔️ |
30+
| x86 Ubuntu16.04 有 GPU | ✔️ |
31+
| x86 Ubuntu18.04 无 GPU | ✔️ |
32+
| ARM | / |
33+
34+
| Python 版本 | 支持情况 |
35+
| - | - |
36+
| 3.6 | ✔️ |
37+
| 3.7 | ✔️ |
38+
| 3.8 | ✔️ |
39+
| 3.9 | / |
40+
2341
## Built-in Applications
2442
* 猫猫围栏
2543
* 电梯电动车报警
2644

27-
## Coming soon
45+
## Coming Soon
2846
- 进程级别的节点、子图支持
2947
- 插件自动化测试部署
3048
- 调试工具，建图实时预览 profile 工具

‎docs/how-to-build.zh.md renamed to ‎docs/how-to-build-from-source.zh.md

+6-13
Original file line numberDiff line numberDiff line change
@@ -1,19 +1,12 @@
11
# Building from Source
22

3-
## Prerequisites
4-
5-
### 软硬件环境
6-
7-
| 测试通过的环境 | 备注 |
8-
| - | - |
9-
| win10 WSL ubuntu18.04 | - |
10-
| x86 Ubuntu16.04 服务器有 GPU | - |
11-
| x86 Ubuntu18.04 无 GPU | 运行时选 CPU config |
3+
如果仅仅是想运行应用，推荐使用 [docker](how-to-build-with-docker.zh.md) 或 [prebuilt .whl 包](how-to-run-in-15-minutes.zh.md) ，源码安装需要 Rust/ffmpeg 环境。
124

13-
支持主流 x86 Linux 版本，ARM 还在开发。
5+
## Prerequisites
146

157
### 安装 Rust
168
```bash
9+
$ sudo apt install yasm git build-essential ffmpeg curl
1710
$ curl --proto '=https' --tlsv1.2 https://sh.rustup.rs -sSf | sh
1811
```
1912

@@ -41,18 +34,18 @@ $ conda --version
4134
conda 4.10.3
4235
```
4336

44-
创建一个 Python3.8（已测试 3.6.13/3.7.11/3.8.11 可用。**3.9 暂不可用**，这里以 3.8 为例）的环境，激活。
37+
创建一个 Python3.x（这里以 3.8 为例） 的环境，激活。
4538
```bash
4639
$ conda create --name py38 python=3.8
4740
$ conda activate py38
4841
```
4942

43+
5044
## Build
5145

5246
MegFlow 需要编译 ffmpeg。考虑到 ffmpeg 依赖较多、本身又是常用工具，最简单的办法就是直接装 ffmpeg 把编译依赖装上
5347

5448
```bash
55-
$ sudo apt install yasm # ffmpeg 编译依赖
5649
$ sudo apt install ffmpeg
5750
$ ffmpeg
5851
ffmpeg version 3.4.8...
@@ -74,7 +67,7 @@ $ python3 setup.py install --user
7467

7568
P.S. 默认 ffmpeg 依赖自动从 github 上拉取源码构建，这会使得首次构建的时间较长。若希望缩短首次构建时间，或者希望依赖一个指定版本的 ffmpeg，可以启用环境变量`CARGO_FEATURE_PREBUILD`并参考[rust-ffmpeg](https://github.com/zmwangx/rust-ffmpeg/wiki/Notes-on-building)自行构建
7669

77-
## Python “开机自检”用例
70+
## Python“开机自检”
7871
```bash
7972
$ cd examples
8073
$ cargo run --example run_with_plugins -- -p logical_test

‎docs/how-to-build-with-docker.zh.md

+33
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
# Building with docker
2+
3+
## Build Docker Image
4+
5+
MegFlow 提供了 [Dockerfile](../Dockerfile)，能够“可复现地”生成运行环境、减少依赖缺失的痛苦
6+
7+
```bash
8+
$ cd MegFlow
9+
$ docker build -t megflow .
10+
```
11+
稍等一段时间（取决于网络和 CPU）镜像构建完成并做了基础自测
12+
```bash
13+
$ docker images
14+
REPOSITORY TAG IMAGE ID CREATED SIZE
15+
megflow latest c65e37e1df6c 18 hours ago 5.05GB
16+
```
17+
直接用 ${IMAGE ID} 进入开始跑应用
18+
```bash
19+
$ docker run -p 18081:8081 -p 18082:8082 -i -t c65e37e1df6c /bin/bash
20+
```
21+
22+
## Python Built-in Applications
23+
24+
接下来开始运行好玩的 Python 应用
25+
26+
* [猫猫围栏运行手册](../flow-python/examples/cat_finder/README.md)
27+
* 图片注册猫猫
28+
* 部署视频围栏，注册的猫离开围栏时会发通知
29+
* 未注册的不会提示
30+
* [电梯电瓶车告警](../flow-python/examples/electric_bicycle/README.md)
31+
* 电梯里看到电瓶车立即报警
32+
* Comming Soon
33+
* OCR： 通用字符识别

‎docs/how-to-run-in-15-minutes.zh.md

+79
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,79 @@
1+
# Run in 15 minutes
2+
3+
4+
5+
## 安装 python3.x （推荐 conda）
6+
7+
打开 [miniconda 官网](https://docs.conda.io/en/latest/miniconda.html) 下载 miniconda 安装包，修改权限并安装。
8+
9+
```bash
10+
$ wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
11+
$ chmod a+x Miniconda3-latest-Linux-x86_64.sh
12+
$ ./Miniconda3-latest-Linux-x86_64.sh
13+
```
14+
15+
安装时接受 conda 修改默认 .bashrc 环境变量（zsh 用户还需自行修改 .zshrc 中的 conda initialize 配置）。成功后 `conda` 可正常运行
16+
```bash
17+
$ conda --version
18+
conda 4.10.3
19+
```
20+
21+
创建一个 Python3.x（这里以 3.8 为例） 的环境，激活。
22+
```bash
23+
$ conda create --name py38 python=3.8
24+
$ conda activate py38
25+
```
26+
27+
## 安装 Prebuilt 包
28+
29+
从 [MegFlow release](https://github.com/MegEngine/MegFlow/releases) 下载对应 python 版本的 .whl 包，安装
30+
```bash
31+
$ python3 -m pip install pyflow-0.1.0-py38-none-linux_x86_64.whl --force-reinstall
32+
```
33+
完成后应该可以 `import pyflow`
34+
```bash
35+
$ python3
36+
Python 3.8.3 (default, May 19 2020, 18:47:26)
37+
[GCC 7.3.0] :: Anaconda, Inc. on linux
38+
Type "help", "copyright", "credits" or "license" for more information.
39+
>>> import pyflow
40+
```
41+
.whl 打包了可执行文件 `run_with_plugins`，如果使用 conda 位置应该在
42+
```bash
43+
$ cd ${HOME}/miniconda3/envs/py38/lib/python3.8/site-packages/pyflow/
44+
$ sudo apt install build-essential -y
45+
$ ldd run_with_plugins # 可看到仅依赖常见库
46+
$ ./run_with_plugins --help
47+
run_with_plugins 1.0
48+
megvii
49+
...
50+
```
51+
52+
## Python“开机自检”
53+
54+
```bash
55+
$ cd ${MegFlow_PATH}/flow-python/examples # 这行必须
56+
$ run_with_plugins -p logical_test
57+
```
58+
59+
`logical_test` 是 examples 下最基础的计算图测试用例，运行能正常结束表示 MegFlow 编译成功、基本语义无问题。
60+
61+
此处常见问题：`error while loading shared libraries: libpython3.8.xxx`。如果使用 conda 只需要
62+
```bash
63+
$ export LD_LIBRARY_PATH=/home/`whoami`/miniconda3/pkgs/python-3.8.11-h12debd9_0_cpython/lib:${LD_LIBRARY_PATH}
64+
```
65+
66+
> 工作原理：[pyflow](../flow-python/pyflow/__init__.py) 仅是一层接口，由 run_with_plugins “注入”建图/调度/优化等实现。
67+
68+
## Python Built-in Applications
69+
70+
接下来开始运行好玩的 Python 应用
71+
72+
* [猫猫围栏运行手册](../flow-python/examples/cat_finder/README.md)
73+
* 图片注册猫猫
74+
* 部署视频围栏，注册的猫离开围栏时会发通知
75+
* 未注册的不会提示
76+
* [电梯电瓶车告警](../flow-python/examples/electric_bicycle/README.md)
77+
* 电梯里看到电瓶车立即报警
78+
* Comming Soon
79+
* OCR： 通用字符识别

‎flow-python/gen_whl.sh

+28
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
# build py36~39 version
2+
conda activate py36
3+
cargo build --example run_with_plugins --release
4+
cp ../target/release/examples/run_with_plugins ./pyflow/
5+
ldd pyflow/run_with_plugins
6+
rm -rf ./build
7+
python3 whl-py36-setup.py bdist_wheel -p linux-x86_64 -d py36_dist --python-tag py36
8+
9+
conda activate py37
10+
cargo build --example run_with_plugins --release
11+
ldd pyflow/run_with_plugins
12+
cp ../target/release/examples/run_with_plugins ./pyflow/
13+
rm -rf ./build
14+
python3 whl-py37-setup.py bdist_wheel -p linux-x86_64 -d py37_dist --python-tag py37
15+
16+
conda activate py38
17+
cargo build --example run_with_plugins --release
18+
ldd pyflow/run_with_plugins
19+
cp ../target/release/examples/run_with_plugins ./pyflow/
20+
rm -rf ./build
21+
python3 whl-py38-setup.py bdist_wheel -p linux-x86_64 -d py38_dist --python-tag py38
22+
23+
24+
rm -rf dist
25+
mkdir dist
26+
cp py36_dist/pyflow-0.1.0-py36-none-linux_x86_64.whl dist/
27+
cp py37_dist/pyflow-0.1.0-py37-none-linux_x86_64.whl dist/
28+
cp py38_dist/pyflow-0.1.0-py38-none-linux_x86_64.whl dist/

‎flow-python/setup.py

+23-1
Original file line numberDiff line numberDiff line change
@@ -9,11 +9,33 @@
99

1010
#!/usr/bin/env python
1111
# coding=utf-8
12-
from setuptools import setup
12+
from setuptools import setup, find_packages
1313

1414
setup(
1515
name="pyflow",
1616
version="0.1.0",
1717
packages=["pyflow"],
18+
author="Megvii IPU-SDK Team",
19+
author_email="megengine@megvii.com",
20+
url="https://github.com/MegEngine/MegFlow",
21+
include_package_data=True,
22+
classifiers=[
23+
'Development Status :: 3 - Alpha',
24+
'Intended Audience :: Developers',
25+
'License :: OSI Approved :: Apache Software License',
26+
'Natural Language :: English',
27+
'Operating System :: POSIX :: Linux',
28+
'Programming Language :: C++',
29+
'Programming Language :: Rust',
30+
'Programming Language :: Python :: 3.8',
31+
'Topic :: Software Development :: Libraries :: Application Frameworks',
32+
'Topic :: Scientific/Engineering',
33+
'Topic :: Scientific/Engineering :: Mathematics',
34+
'Topic :: Scientific/Engineering :: Artificial Intelligence',
35+
'Topic :: Software Development',
36+
'Topic :: Software Development :: Libraries',
37+
'Topic :: Software Development :: Libraries :: Python Modules',
38+
],
39+
python_requires='>=3.7,<3.8',
1840
)
1941

‎flow-python/whl-py36-setup.py

+44
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
# -*- coding: utf-8 -*-
2+
# MegFlow is Licensed under the Apache License, Version 2.0 (the "License")
3+
#
4+
# Copyright (c) 2019-2021 Megvii Inc. All rights reserved.
5+
#
6+
# Unless required by applicable law or agreed to in writing,
7+
# software distributed under the License is distributed on an
8+
# "AS IS" BASIS, WITHOUT ARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
9+
10+
#!/usr/bin/env python
11+
# coding=utf-8
12+
from setuptools import setup, find_packages
13+
14+
setup(
15+
name="pyflow",
16+
version="0.1.0",
17+
packages=["pyflow"],
18+
author="Megvii IPU-SDK Team",
19+
author_email="megengine@megvii.com",
20+
url="https://github.com/MegEngine/MegFlow",
21+
include_package_data=True,
22+
classifiers=[
23+
'Development Status :: 3 - Alpha',
24+
'Intended Audience :: Developers',
25+
'License :: OSI Approved :: Apache Software License',
26+
'Natural Language :: English',
27+
'Operating System :: POSIX :: Linux',
28+
'Programming Language :: C++',
29+
'Programming Language :: Rust',
30+
'Programming Language :: Python :: 3.6',
31+
'Topic :: Software Development :: Libraries :: Application Frameworks',
32+
'Topic :: Scientific/Engineering',
33+
'Topic :: Scientific/Engineering :: Mathematics',
34+
'Topic :: Scientific/Engineering :: Artificial Intelligence',
35+
'Topic :: Software Development',
36+
'Topic :: Software Development :: Libraries',
37+
'Topic :: Software Development :: Libraries :: Python Modules',
38+
],
39+
python_requires='>=3.6,<3.7',
40+
package_data={
41+
"":['run_with_plugins']
42+
}
43+
)
44+

‎flow-python/whl-py37-setup.py

+44
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
# -*- coding: utf-8 -*-
2+
# MegFlow is Licensed under the Apache License, Version 2.0 (the "License")
3+
#
4+
# Copyright (c) 2019-2021 Megvii Inc. All rights reserved.
5+
#
6+
# Unless required by applicable law or agreed to in writing,
7+
# software distributed under the License is distributed on an
8+
# "AS IS" BASIS, WITHOUT ARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
9+
10+
#!/usr/bin/env python
11+
# coding=utf-8
12+
from setuptools import setup, find_packages
13+
14+
setup(
15+
name="pyflow",
16+
version="0.1.0",
17+
packages=["pyflow"],
18+
author="Megvii IPU-SDK Team",
19+
author_email="megengine@megvii.com",
20+
url="https://github.com/MegEngine/MegFlow",
21+
include_package_data=True,
22+
classifiers=[
23+
'Development Status :: 3 - Alpha',
24+
'Intended Audience :: Developers',
25+
'License :: OSI Approved :: Apache Software License',
26+
'Natural Language :: English',
27+
'Operating System :: POSIX :: Linux',
28+
'Programming Language :: C++',
29+
'Programming Language :: Rust',
30+
'Programming Language :: Python :: 3.7',
31+
'Topic :: Software Development :: Libraries :: Application Frameworks',
32+
'Topic :: Scientific/Engineering',
33+
'Topic :: Scientific/Engineering :: Mathematics',
34+
'Topic :: Scientific/Engineering :: Artificial Intelligence',
35+
'Topic :: Software Development',
36+
'Topic :: Software Development :: Libraries',
37+
'Topic :: Software Development :: Libraries :: Python Modules',
38+
],
39+
python_requires='>=3.7,<3.8',
40+
package_data={
41+
"":['run_with_plugins']
42+
}
43+
)
44+

0 commit comments

Comments
 (0)