Neural Radiance Fields (NeRF)

Introduction

Neural Radiance Fields (NeRF) represent a scene as a continuous volumetric function using a neural network. Given a sparse set of input images, NeRF can synthesize photorealistic novel views of complex scenes.

Scene Representation

NeRF models a scene as a function $F_\Theta$ that maps a 3D position $\mathbf{x} = (x, y, z)$ and viewing direction $\mathbf{d} = (\theta, \phi)$ to a color $\mathbf{c} = (r, g, b)$ and volume density $\sigma$:

$F_\Theta: (\mathbf{x}, \mathbf{d}) \rightarrow (\mathbf{c}, \sigma)$

Volume Rendering

To render a pixel, we cast a ray $\mathbf{r}(t) = \mathbf{o} + t\mathbf{d}$ and integrate:

$C(\mathbf{r}) = \int_{t_n}^{t_f} T(t) \cdot \sigma(\mathbf{r}(t)) \cdot \mathbf{c}(\mathbf{r}(t), \mathbf{d}) \, dt$

where the transmittance is:

$T(t) = \exp\left(-\int_{t_n}^{t} \sigma(\mathbf{r}(s)) \, ds\right)$

In practice, this is approximated with quadrature using stratified sampling.

Positional Encoding

Raw coordinates are mapped to higher dimensions using sinusoidal functions:

$\gamma(p) = \left(\sin(2^0 \pi p), \cos(2^0 \pi p), ..., \sin(2^{L-1} \pi p), \cos(2^{L-1} \pi p)\right)$

This allows the MLP to represent high-frequency details.

Hierarchical Sampling

NeRF uses two networks:

1. Coarse network: Sampled uniformly along the ray
2. Fine network: Samples concentrated where the coarse network predicts high density

Training

The loss is simply the MSE between rendered and ground truth pixel colors:

$\mathcal{L} = \sum_{\mathbf{r} \in \mathcal{R}} \left\| \hat{C}(\mathbf{r}) - C_{gt}(\mathbf{r}) \right\|^2$

Limitations

• Slow training: Hours per scene
• Slow rendering: Seconds per frame (ray marching)
• Static scenes only: Original NeRF cannot handle dynamic content
• Requires accurate camera poses

Extensions

• Instant-NGP: Hash-based encoding for fast training (minutes)
• Mip-NeRF: Anti-aliased rendering with cone tracing
• D-NeRF: Dynamic scenes with deformation fields
• NeRF in the Wild: Handles varying lighting and transient objects

Conclusion

NeRF pioneered neural scene representations and novel view synthesis. While largely superseded by 3D Gaussian Splatting for real-time applications, NeRF's mathematical framework remains foundational.

References

1. Mildenhall et al., "NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis," ECCV 2020.
2. Barron et al., "Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields," ICCV 2021.