diff --git a/index.html b/index.html index 61bb43d..373119f 100644 --- a/index.html +++ b/index.html @@ -1,63 +1,490 @@ - - + - - - - X-Why - + + + + + Nerfies: Deformable Neural Radiance Fields + + + + + + + + + + + + + + + + + + + + -
- License: MIT - Standard - Python Style Guide -
- -

X-Why

-

XWhy: eXplain Why with SMILE -- Statistical Model-agnostic Interpretability with Local Explanations

- -
- XWhy, SMILE, Explainability, Interpretability, XAI, machine learning explainability, responsible ai -
- -

Abstract

-

Machine learning is currently undergoing an explosion in capability, popularity, - and sophistication. However, one of the major barriers to widespread acceptance of machine learning (ML) is - trustworthiness: most ML models operate as black boxes, their inner workings opaque and mysterious, and it can - be difficult to trust their conclusions without understanding how those conclusions are reached. Explainability - is therefore a key aspect of improving trustworthiness: the ability to better understand, interpret, and - anticipate the behaviour of ML models. To this end, we propose a SMILE, a new method that builds on previous - approaches by making use of statistical distance measures to improve explainability while remaining applicable - to a wide range of input data domains.

- -

Installation

- 
pip install xwhy
- -

Citations

-

It would be appreciated a citation to our paper as follows if you use X-Why for your research:

- 
@article{Aslansefat2021Xwhy,
-   author  = {{Aslansefat}, Koorosh and {Hashemian}, Mojgan and {Martin}, Walker, {Akram} Mohammed Naveed, {Sorokos} Ioannis and {Papadopoulos}, Yiannis},
-   title   = "{SMILE: Statistical Model-agnostic Interpretability with Local Explanations}",
-   journal = {arXiv e-prints},
-   year    = {2021},
-   url     = {https://arxiv.org/abs/...},
-   eprint  = {},
-}
+ -

Contribution

-

If you are interested in contributing to this project, please check the contribution guidelines.

- +
+
+
+
+
+

Nerfies: Deformable Neural Radiance Fields

+
+ + Keunhong Park1, + + Utkarsh Sinha2, + + Jonathan T. Barron2, + + + Sofien Bouaziz2, + + + Dan B Goldman2, + + + Steven M. Seitz1,2, + + + Ricardo Martin-Brualla2 + +
+ +
+ 1University of Washington, + 2Google Research +
+ +
+
+ + + + + + + Paper + + + + + + + + arXiv + + + + + + + + + Video + + + + + + + + + Code + + + + + + + + + Data + +
+ +
+
+
+
+
+
+ +
+
+
+ +

+ Nerfies turns selfie videos from your phone into + free-viewpoint + portraits. +

+
+
+
+ + +
+
+
+ +
+
+
+ + +
+
+ +
+
+

Abstract

+
+

+ We present the first method capable of photorealistically reconstructing a non-rigidly + deforming scene using photos/videos captured casually from mobile phones. +

+

+ Our approach augments neural radiance fields + (NeRF) by optimizing an + additional continuous volumetric deformation field that warps each observed point into a + canonical 5D NeRF. + We observe that these NeRF-like deformation fields are prone to local minima, and + propose a coarse-to-fine optimization method for coordinate-based models that allows for + more robust optimization. + By adapting principles from geometry processing and physical simulation to NeRF-like + models, we propose an elastic regularization of the deformation field that further + improves robustness. +

+

+ We show that Nerfies can turn casually captured selfie + photos/videos into deformable NeRF + models that allow for photorealistic renderings of the subject from arbitrary + viewpoints, which we dub "nerfies". We evaluate our method by collecting data + using a + rig with two mobile phones that take time-synchronized photos, yielding train/validation + images of the same pose at different viewpoints. We show that our method faithfully + reconstructs non-rigidly deforming scenes and reproduces unseen views with high + fidelity. +

+
+
+
+ + + +
+
+

Video

+
+ +
+
+
+ +
+
+ +
+
+ +
+ + +
+
+

Visual Effects

+

+ Using nerfies you can create fun visual effects. This Dolly zoom effect + would be impossible without nerfies since it would require going through a wall. +

+ +
+
+ + + +
+

Matting

+
+
+

+ As a byproduct of our method, we can also solve the matting problem by ignoring + samples that fall outside of a bounding box during rendering. +

+ +
+ +
+
+
+ + + +
+
+

Animation

+ + +

Interpolating states

+
+

+ We can also animate the scene by interpolating the deformation latent codes of two input + frames. Use the slider here to linearly interpolate between the left frame and the right + frame. +

+
+
+
+ Interpolate start reference image. +

Start Frame

+
+
+
+ Loading... +
+ +
+
+ Interpolation end reference image. +

End Frame

+
+
+
+ + + +

Re-rendering the input video

+
+

+ Using Nerfies, you can re-render a video from a novel + viewpoint such as a stabilized camera by playing back the training deformations. +

+
+
+ +
+ + +
+
+ + + + +
+
+

Related Links

+ +
+

+ There's a lot of excellent work that was introduced around the same time as ours. +

+

+ Progressive Encoding for Neural Optimization introduces an idea similar to our windowed position encoding for coarse-to-fine optimization. +

+

+ D-NeRF and NR-NeRF + both use deformation fields to model non-rigid scenes. +

+

+ Some works model videos with a NeRF by directly modulating the density, such as Video-NeRF, NSFF, and DyNeRF +

+

+ There are probably many more by the time you are reading this. Check out Frank Dellart's survey on recent NeRF papers, and Yen-Chen Lin's curated list of NeRF papers. +

+
+
+
+ + +
+
+ + +
+
+

BibTeX

+ 
@article{park2021nerfies,
+  author    = {Park, Keunhong and Sinha, Utkarsh and Barron, Jonathan T. and Bouaziz, Sofien and Goldman, Dan B and Seitz, Steven M. and Martin-Brualla, Ricardo},
+  title     = {Nerfies: Deformable Neural Radiance Fields},
+  journal   = {ICCV},
+  year      = {2021},
+}
+
+
+ + + + +