|
1 | | -# AMT |
2 | | -Official code for "AMT: All-Pairs Multi-Field Transforms for Efficient Frame Interpolation" (CVPR2023) |
| 1 | +# AMT: All-Pairs Multi-Field Transforms for Efficient Frame Interpolation |
| 2 | + |
| 3 | + |
| 4 | +This repository contains the official implementation of the following paper: |
| 5 | +> **AMT: All-Pairs Multi-Field Transforms for Efficient Frame Interpolation**<br> |
| 6 | +> [Zhen Li](https://paper99.github.io/)<sup>\*</sup>, [Zuo-Liang Zhu](https://nk-cs-zzl.github.io/)<sup>\*</sup>, [Ling-Hao Han](https://scholar.google.com/citations?user=0ooNdgUAAAAJ&hl=en), [Qibin Hou](https://scholar.google.com/citations?hl=en&user=fF8OFV8AAAAJ&view_op=list_works), [Chun-Le Guo](https://scholar.google.com/citations?hl=en&user=RZLYwR0AAAAJ), [Ming-Ming Cheng](https://mmcheng.net/)<br> |
| 7 | +> (\* denotes equal contribution) <br> |
| 8 | +> Nankai University <br> |
| 9 | +> In CVPR 2023<br> |
| 10 | +
|
| 11 | +[[Paper](https://github.com/MCG-NKU/E2FGVI)] |
| 12 | +[[Project Page](https://nk-cs-zzl.github.io/projects/amt/index.html)] |
| 13 | +[[Web demos](#web-demos)] |
| 14 | +[Video] |
| 15 | + |
| 16 | +AMT is a **lightweight, fast, and accurate** algorithm for Frame Interpolation. |
| 17 | +It aims to provide practical solutions for **video generation** from **a few given frames (at least two frames)**. |
| 18 | + |
| 19 | + |
| 20 | +* More examples can be found in our [project page](https://nk-cs-zzl.github.io/projects/amt/index.html). |
| 21 | + |
| 22 | +## Web demos |
| 23 | +Integrated into [Hugging Face Spaces 🤗](https://huggingface.co/spaces) using [Gradio](https://github.com/gradio-app/gradio). Try out the Web Demo: [](https://huggingface.co/spaces/johngoad/frame-interpolation) |
| 24 | + |
| 25 | +Try AMT to interpolate between two or more images at [](https://colab.research.google.com/drive/1IeVO5BmLouhRh6fL2z_y18kgubotoaBq?usp=sharing) |
| 26 | + |
| 27 | + |
| 28 | +## Change Log |
| 29 | +- **Apr 20, 2023**: Our code is publicly available. |
| 30 | + |
| 31 | + |
| 32 | +## Method Overview |
| 33 | + |
| 34 | + |
| 35 | +For technical details, please refer to the [method.md](docs/method.md) file, or read the full report on [arXiv](). |
| 36 | + |
| 37 | +## Dependencies and Installation |
| 38 | +1. Clone Repo |
| 39 | + |
| 40 | + ```bash |
| 41 | + git clone https://github.com/MCG-NKU/AMT.git |
| 42 | + ``` |
| 43 | + |
| 44 | +2. Create Conda Environment and Install Dependencies |
| 45 | + |
| 46 | + ```bash |
| 47 | + conda env create -f environment.yaml |
| 48 | + conda activate amt |
| 49 | + ``` |
| 50 | +3. Download pretrained models for demos from [Pretrained Models](#pretrained-models) and place them to the `pretrained` folder |
| 51 | + |
| 52 | +## Quick Demo |
| 53 | + |
| 54 | +**Note that the selected pretrained model (`[CKPT_PATH]`) needs to match the config file (`[CFG]`).** |
| 55 | + |
| 56 | + > Creating a video demo, increasing $n$ will slow down the motion in the video. (With $m$ input frames, `[N_ITER]` $=n$ corresponds to $2^n\times (m-1)+1$ output frames.) |
| 57 | +
|
| 58 | + |
| 59 | + ```bash |
| 60 | + python demos/demo_2x.py -c [CFG] -p [CKPT] -n [N_ITER] -i [INPUT] -o [OUT_PATH] -r [FRAME_RATE] |
| 61 | + # e.g. [INPUT] |
| 62 | + # -i could be a video / a regular expression / a folder contains multiple images |
| 63 | + # -i demo.mp4 (video)/img_*.png (regular expression)/img0.png img1.png (images)/demo_input (folder) |
| 64 | + |
| 65 | + # e.g. a simple usage |
| 66 | + python demos/demo_2x.py -c cfgs/AMT-S.yaml -p pretrained/amt-s.pth -n 6 -i assets/quick_demo/img0.png assets/quick_demo/img1.png |
| 67 | + |
| 68 | + ``` |
| 69 | + |
| 70 | + + Note: Please enable `--save_images` for saving the output images (Save speed will be slowed down if there are too many output images) |
| 71 | + + Input type supported: `a video` / `a regular expression` / `multiple images` / `a folder containing input frames`. |
| 72 | + + Results are in the `[OUT_PATH]` (default is `results/2x`) folder. |
| 73 | + |
| 74 | +## Pretrained Models |
| 75 | + |
| 76 | +<p id="Pretrained"></p> |
| 77 | + |
| 78 | +<table> |
| 79 | +<thead> |
| 80 | + <tr> |
| 81 | + <th> Dataset </th> |
| 82 | + <th> :link: Download Links </th> |
| 83 | + <th> Config file </th> |
| 84 | + <th> Trained on </th> |
| 85 | + <th> Arbitrary/Fixed </th> |
| 86 | + </tr> |
| 87 | +</thead> |
| 88 | +<tbody> |
| 89 | + <tr> |
| 90 | + <td>AMT-S</td> |
| 91 | + <th> [<a href="https://drive.google.com/file/d/1WmOKmQmd6pnLpID8EpUe-TddFpJuavrL/view?usp=share_link">Google Driver</a>][<a href="https://pan.baidu.com/s/1yGaNLeb9TG5-81t0skrOUA?pwd=f66n">Baidu Cloud</a>]</th> |
| 92 | + <th> [<a href="cfgs/AMT-S.yaml">cfgs/AMT-S</a>] </th> |
| 93 | + <th>Vimeo90k</th> |
| 94 | + <th>Fixed</th> |
| 95 | + </tr> |
| 96 | + <tr> |
| 97 | + <td>AMT-L</td> |
| 98 | + <th>[<a href="https://drive.google.com/file/d/1UyhYpAQLXMjFA55rlFZ0kdiSVTL7oU-z/view?usp=share_link">Google Driver</a>][<a href="https://pan.baidu.com/s/1qI4fBgS405Bd4Wn1R3Gbeg?pwd=nbne">Baidu Cloud</a>]</th> |
| 99 | + <th> [<a href="cfgs/AMT-L.yaml">cfgs/AMT-L</a>] </th> |
| 100 | + <th>Vimeo90k</th> |
| 101 | + <th>Fixed</th> |
| 102 | + </tr> |
| 103 | + <tr> |
| 104 | + <td>AMT-G</td> |
| 105 | + <th>[<a href="https://drive.google.com/file/d/1yieLtKh4ei3gOrLN1LhKSP_9157Q-mtP/view?usp=share_link">Google Driver</a>][<a href="https://pan.baidu.com/s/1AjmQVziQut1bXgQnDcDKvA?pwd=caf6">Baidu Cloud</a>]</th> |
| 106 | + <th> [<a href="cfgs/AMT-G.yaml">cfgs/AMT-G</a>] </th> |
| 107 | + <th>Vimeo90k</th> |
| 108 | + <th>Fixed</th> |
| 109 | + </tr> |
| 110 | + <tr> |
| 111 | + <td>AMT-S</td> |
| 112 | + <th>[<a href="https://drive.google.com/file/d/1f1xAF0EDm-rjDdny8_aLyeedfM0QL4-C/view?usp=share_link">Google Driver</a>][<a href="https://pan.baidu.com/s/1eZtoULyduQM8AkXeYEBOEw?pwd=8hy3">Baidu Cloud</a>]</th> |
| 113 | + <th> [<a href="cfgs/AMT-S_gopro.yaml">cfgs/AMT-S_gopro</a>] </th> |
| 114 | + <th>GoPro</th> |
| 115 | + <th>Arbitrary</th> |
| 116 | + </tr> |
| 117 | +</tbody> |
| 118 | +</table> |
| 119 | + |
| 120 | +## Training and Evaluation |
| 121 | + |
| 122 | +Please refer to [develop.md](docs/develop.md) to learn how to benchmark the AMT and how to train a new AMT model from scratch. |
| 123 | + |
| 124 | + |
| 125 | +## Citation |
| 126 | + If you find our repo useful for your research, please consider citing our paper: |
| 127 | + |
| 128 | + ```bibtex |
| 129 | + @inproceedings{licvpr23amt, |
| 130 | + title={AMT: All-Pairs Multi-Field Transforms for Efficient Frame Interpolation}, |
| 131 | + author={Li, Zhen and Zhu, Zuo-Liang and Han, Ling-Hao and Hou, Qibin and Guo, Chun-Le and Cheng, Ming-Ming}, |
| 132 | + booktitle={IEEE Conference on Computer Vision and Pattern Recognition (CVPR)}, |
| 133 | + year={2023} |
| 134 | + } |
| 135 | + ``` |
| 136 | + |
| 137 | + |
| 138 | +## License |
| 139 | +This code is licensed under the [Creative Commons Attribution-NonCommercial 4.0 International](https://creativecommons.org/licenses/by-nc/4.0/) for non-commercial use only. |
| 140 | +Please note that any commercial use of this code requires formal permission prior to use. |
| 141 | + |
| 142 | +## Contact |
| 143 | + |
| 144 | +For technical questions, please contact `zhenli1031[AT]gmail.com` and `nkuzhuzl[AT]gmail.com`. |
| 145 | + |
| 146 | +For commercial licensing, please contact `cmm[AT]nankai.edu.cn` |
| 147 | + |
| 148 | +## Acknowledgement |
| 149 | + |
| 150 | +We thank Jia-Wen Xiao, Zheng-Peng Duan, Rui-Qi Wu, and Xin Jin for proof reading. |
| 151 | +We thank [Zhewei Huang](https://github.com/hzwer) for his suggestions. |
| 152 | + |
| 153 | +Here are some great resources we benefit from: |
| 154 | + |
| 155 | +- [IFRNet](https://github.com/ltkong218/IFRNet) and [RIFE](https://github.com/megvii-research/ECCV2022-RIFE) for data processing, benchmarking, and loss designs. |
| 156 | +- [RAFT](https://github.com/princeton-vl/RAFT), [M2M-VFI](https://github.com/feinanshan/M2M_VFI), and [GMFlow](https://github.com/haofeixu/gmflow) for inspirations. |
| 157 | +- [FILM](https://github.com/google-research/frame-interpolation) for Web demo reference. |
| 158 | + |
| 159 | + |
| 160 | +**If you develop/use AMT in your projects, welcome to let us know. We will list your projects in this repository.** |
| 161 | + |
| 162 | +We also thank potential future contributors. |
0 commit comments