3D Gaussian Splatting

Introduction

3D Gaussian Splatting is a recent breakthrough in real-time radiance field rendering. Unlike Neural Radiance Fields (NeRF), which rely on expensive volumetric ray marching, Gaussian Splatting represents scenes as collections of 3D Gaussians that can be efficiently rasterized.

Key Idea

The core idea is to represent a 3D scene using a set of 3D Gaussian primitives. Each Gaussian is defined by:

• Position $\mu \in \mathbb{R}^3$: the center of the Gaussian
• Covariance $\Sigma \in \mathbb{R}^{3 \times 3}$: defines the shape and orientation
• Opacity $\alpha \in [0, 1]$: controls transparency
• Color (represented via spherical harmonics for view-dependent effects)

The 3D Gaussian function is:

$G(x) = e^{-\frac{1}{2}(x - \mu)^T \Sigma^{-1} (x - \mu)}$

Rendering Pipeline

1. Project 3D Gaussians to 2D screen space
2. Sort by depth (front-to-back)
3. Rasterize using alpha blending:

$C = \sum_{i=1}^{N} c_i \alpha_i \prod_{j=1}^{i-1}(1 - \alpha_j)$

where $c_i$ is the color and $\alpha_i$ is the opacity of the $i$-th Gaussian.

Optimization

The Gaussians are optimized from multi-view images using gradient descent:

• Photometric loss between rendered and ground truth images
• Adaptive density control: splitting large Gaussians, cloning small ones, pruning transparent ones
• Covariance regularization to prevent degenerate shapes

Advantages over NeRF

| Feature | NeRF | 3D Gaussian Splatting | |---------|------|----------------------| | Rendering | Ray marching (slow) | Rasterization (fast) | | Speed | ~30s per frame | Real-time (100+ FPS) | | Representation | Implicit (MLP) | Explicit (Gaussians) | | Editing | Difficult | Straightforward | | Training | Hours | Minutes |

Applications

• Virtual Reality: Real-time rendering of photorealistic scenes
• Digital Twins: Rapid reconstruction of physical environments
• Film & VFX: Quick scene capture and novel view synthesis
• Robotics: Scene understanding and navigation

Conclusion

3D Gaussian Splatting has fundamentally changed real-time neural rendering by replacing implicit neural representations with explicit Gaussian primitives, achieving real-time performance while maintaining high visual quality.

References

1. Kerbl et al., "3D Gaussian Splatting for Real-Time Radiance Field Rendering," SIGGRAPH 2023.
2. Zwicker et al., "EWA Splatting," IEEE TVCG 2002.