OneFlow

OneFlow 是一个深度学习框架，旨在易用，可扩展且高效。使用 OneFlow，很容易做到:

• 模型编程使用与 pytorch 类似的 API

• 使用 global API 将模型扩展到 n 维并行以便于分布式执行

• 使用静态图编译器加速/部署模型

Latest News

• Version 1.0.0 is out!
  • Full changelog

安装 OneFlow-DCU

System Requirements

• Linux.

• Python 3.7, 3.8, 3.9

• (推荐) Upgrade pip

  python3 -m pip install --upgrade pip #--user
  

Pip 安装

可以再光合光合开发者社区 AI 生态包中获取最新的 Oneflow-DCU Release 版本（需对应 DCU Toolkit 版本与 python 版本）

python3 -m pip install oneflow-0.9+dtk22101.git.5be579-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

使用镜像

python3 -m pip install --pre oneflow -f https://oneflow-staging.oss-cn-beijing.aliyuncs.com/branch/master/cu118

• To install other available builds for different variants:

  • Stable

    python3 -m pip install --find-links https://release.oneflow.info oneflow==0.9.0+cu118

  • Nightly

    python3 -m pip install --pre oneflow -f https://staging.oneflow.info/branch/master/[PLATFORM]
    

  • All available [PLATFORM]:

    Platform	CUDA Driver Version	Supported GPUs
    cu118	>= 450.80.02	GTX 10xx, RTX 20xx, A100, RTX 30xx
    cpu	N/A	N/A

• If you are in China, you could run this to have pip download packages from domestic mirror of pypi:

  python3 -m pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple
  

  For more information on this, please refer to pypi 镜像使用帮助

Use docker image

docker pull image.sourcefind.cn:5000/dcu/admin/base/oneflow:0.9.1-centos7.6-dtk-22.10.1-py39-latest

在 DCU 平台上源码编译（DTK-22.10.1，Python3.9）

• 拉取官方 CPU 镜像

  docker pull oneflowinc/manylinux2014_x86_64_cpu:latest
  

• 使用官网镜像建立 docker

  docker run -it --network=host --name=oneflow_compile --privileged --device=/dev/kfd --device=/dev/dri --ipc=host --shm-size=16G  --group-add video --cap-add=SYS_PTRACE --security-opt seccomp=unconfined -u root --ulimit stack=-1:-1 --ulimit memlock=-1:-1 -v /public/home/xxx:/home oneflowinc/manylinux2014_x86_64_cpu:latest /bin/bash
  
  docker exec -it oneflow_compile /bin/bash
  

• 拉取 oneflow 代码

  git clone -b 0.9.1-rocm http://developer.hpccube.com/codes/aicomponent/oneflow.git
  

• 在开发者社区 DCU Toolkit 中下载 DTK-22.10.1 解压至 /opt/ 路径下，并建立软链接

  cd /opt && ln -s dtk-22.10.1 rocm
  

• 导入环境变量以及安装必要依赖库

  export ROCM_PATH=/opt/rocm
  export HIP_PATH=${ROCM_PATH}/hip
  export CPACK_INSTLL_PREFIX=$ROCM_PATH
  export AMDGPU_TARGETS="gfx900;gfx906"
  export PATH=${ROCM_PATH}/bin:${ROCM_PATH}/llvm/bin:${ROCM_PATH}/hcc/bin:${ROCM_PATH}/hip/bin:$PATH
  export LD_LIBRARY_PATH=${ROCM_PATH}/lib:${ROCM_PATH}/lib64:$LD_LIBRARY_PATH
  export LD_LIBRARY_PATH=${ROCM_PATH}/hip/lib:${ROCM_PATH}/llvm/lib:${ROCM_PATH}/opencl/lib/x86_64:$LD_LIBRARY_PATH
  export C_INCLUDE_PATH=${ROCM_PATH}/include:${ROCM_PATH}/hip/include/hip:${ROCM_PATH}/llvm/include:/opencl/include:${C_INCLUDE_PATH}
  export CPLUS_INCLUDE_PATH=${ROCM_PATH}/include:${ROCM_PATH}/hip/include/hip:${ROCM_PATH}/llvm/include:/opencl/include:${CPLUS_INCLUDE_PATH}
  export PATH=${ROCM_PATH}/miopen/bin:${ROCM_PATH}/rocblas/bin:${ROCM_PATH}/hipsparse/bin:$PATH
  export LD_LIBRARY_PATH=${ROCM_PATH}/miopen/lib:${ROCM_PATH}/rocblas/lib:$LD_LIBRARY_PATH
  export MIOPEN_SYSTEM_DB_PATH=${ROCM_PATH}/miopen/share/miopen/db/
  export LD_LIBRARY_PATH=/usr/lib64:$LD_LIBRARY_PATH
  export LIBRARY_PATH=/usr/lib64:$LIBRARY_PATH                     
  export RCCL_PATH=$ROCM_PATH/rccl
  export NCCL_PATH=$ROCM_PATH/rccl
  export LD_LIBRARY_PATH=$RCCL_PATH/lib:$LD_LIBRARY_PATH
  
  export MIOPEN_FIND_MODE=3
  export HSA_FORCE_FINE_GRAIN_PCIE=1
  export MIOPEN_COMPILE_PARALLEL_LEVEL=1
  
  source /opt/rh/devtoolset-7/enable
  
  export PV=39
  ln -s /opt/python/cp${PV}-cp${PV}/bin/python3 /usr/bin/python3
  ln -s /opt/python/cp${PV}-cp${PV}/bin/pip3 /usr/bin/pip3
  
  yum install -y numactl libffi* openblas openblas-devel libibverbs-devel
  cd oneflow && pip3 install -r dev-requirements.txt -i http://mirrors.aliyun.com/pypi/simple/ --trusted-host mirrors.aliyun.com
  

• cmake && make

  cd oneflow && mkdir build && cmake .. -DBUILD_CUDA=OFF -DBUILD_ROCM=ON -DONEFLOW=ON -DUSE_CLANG_FORMAT=OFF -DCMAKE_BUILD_TYPE=Release -DTHIRD_PARTY=ON -DTREAT_WARNINGS_AS_ERRORS=OFF -DTHIRD_PARTY_MIRROR=aliyun -DBUILD_HWLOC=OFF -DCMAKE_C_COMPILER=${ROCM_PATH}/llvm/bin/clang -DCMAKE_CXX_COMPILER=${ROCM_PATH}/llvm/bin/clang++ -DBUILD_TESTING=ON -DBUILD_RDMA=ON -DBUILD_PROFILER=ON
  
  make -j32
  

• 验证安装

  cd build && source source.sh    # 将oneflow导入PYTHONPATH
  python3 -c “import oneflow”
  

Advanced features

• OneFlow-XRT: An extension for OneFlow to target third-party compiler, such as XLA, TensorRT and OpenVINO etc.

Getting Started

• Please refer to QUICKSTART
• 中文版请参见 快速上手

Documentation

• API Reference
• Usage & Design Docs
• System Design

Model Zoo and Benchmark

• Libai(Toolbox for Parallel Training Large-Scale Transformer Models)

  • BERT-large

  • GPT

    使用LiBai的GPT2，DCU与A800的精度对比曲线如下：

    

  • T5

  • VisionTransformer

  • SwinTransformer

• FlowVision(Toolbox for Computer Vision Datasets, SOTA Models and Utils)

• OneFlow-Models(Examples of How to Implement Models in Various Fields with OneFlow)

  • ResNet-50
  • Wide&Deep

• OneFlow-Benchmark(Outdated)

Communication

• GitHub issues: any install, bug, feature issues.

• www.oneflow.org: brand related information.

• 中文

  • QQ 群: 331883
  • 微信号（加好友入交流群）: OneFlowXZS
  • 知乎

• International

  • Discord
  • Twitter
  • LinkedIn
  • Medium

The Team

OneFlow was originally developed by OneFlow Inc and Zhejiang Lab.

License

Apache License 2.0