Painting with Gaussians

3D Gaussian splatting barely existed three years ago. It was a single graphics paper at SIGGRAPH 2023. By 2026 it is the thing quietly rendering a photoreal Apple Maps Flyover, unveiled at WWDC 2026, which Apple described only as “aerial imagery combined with vision intelligence models” and pointedly never named on stage, even though the graphics world recognized it on sight. It is also on your phone: apps like Niantic’s Scaniverse, Luma, and Polycam turn a slow walk around a room into a splat in about a minute. It is how NVIDIA renders scenes to train robots and self-driving cars, and how world-model startups like Fei-Fei Li’s World Labs generate entire 3D scenes. This post is about what that phrase actually means. The place to start is the problem it was built to solve: a photograph that does not exist.

Take a few dozen photos of a statue, walking around it with your phone. Now imagine asking for a photo you never took: the statue seen from a spot you never stood in, the light glinting off the marble exactly as it would have. No new photograph exists, yet the answer is completely determined. Every viewpoint you did capture constrains what the missing one must look like. The small scene on the right is the kind of thing that fills that gap: not a mesh, not a point cloud, but a haze of overlapping colored blobs that, seen from any angle, composites back into a photograph.

The problem: photos in, new views out

You have a set of photographs of a scene, each taken from a known position and pointing in a known direction. You want to produce a photograph from a new position and direction, one that was never captured, and you want it to look real. That is the whole problem, and it has a name: novel view synthesis. Everything in this post is machinery for solving it.

The obvious first move is to rebuild the shape of the scene. Feed the photos to a reconstruction pipeline, recover a cloud of 3D points where the surfaces are, stitch them into a mesh of triangles, and render the mesh from the new viewpoint. This works, and for a long time it was the only game in town. But it fights the world at every step. A mesh is a hard commitment: this triangle is surface, that empty space is not, with a crisp boundary in between. Real scenes refuse to cooperate. Where exactly is the surface of a head of hair, or a plume of smoke seen through fog? These are not surfaces with clean edges; they are partly-there, semi-transparent, wispy. Force a mesh onto them and you get holes, jagged silhouettes, and a plastic sheen where there should be softness.

There is a deeper problem than geometry, and it is the one that motivates everything here: appearance. A mesh tells you where a surface is, but a photograph is about how a surface looks, and how it looks depends on where you look from. This is view dependence: the color arriving at your eye from a single point is not one fixed value but a function of the direction you view it from, a unit vector $\mathbf{d}$ we will call the view direction. A bare mesh, one flat color painted per point, throws that information away. It is the whole reason we are about to reach past geometry, and we will pay it off in full at the end.

This is the reframing that unlocks everything that follows. Do not model the scene as surfaces. Model it as a radiance field: a function defined at every point $\mathbf{x}$ in space and every view direction $\mathbf{d}$, returning a color $c$ and a density $\sigma$. In symbols,

$F(\mathbf{x}, \mathbf{d}) = \big(\, c(\mathbf{x}, \mathbf{d}),\ \sigma(\mathbf{x}) \,\big)$

where $c$ is the color the point emits toward you (allowed to change with the direction $\mathbf{d}$, which is exactly what captures the moving highlight) and $\sigma$ is the density, how solidly the point blocks light (a property of the point alone, the same from every angle: a wall is a wall no matter where you stand). We will define both precisely in the next section. The move from “surfaces” to “a field of color and density” costs us the clean geometry, but it buys us everything meshes could never represent: soft edges, semi-transparency, and view-dependent shine, all in one continuous object.

But a field is not yet a photograph. A photograph is taken by a camera, and it pays to be precise about what a camera even is. A view is just a place to stand and a way to look: where the camera sits, the direction it faces, and how wide a cone it takes in, its field of view. That cone, the wedge of space actually in sight, is the camera’s view frustum, and anything outside it never lands in the picture. Stretch a grid of pixels across the cone and each pixel singles out one ray: a straight line from the camera, out through that pixel, into the scene. The pixel’s color is whatever that ray gathers on its way through the fog. Move the camera and you get a fresh frustum, a fresh fan of rays, and a different photograph of the same scene. That is novel view synthesis made concrete: choose a camera nobody used, and work out what each of its rays would have seen.

So the whole problem funnels down to a single question about a single ray: as it crosses the glowing, light-blocking fog, what color does it come back with? Answering that is volume rendering, and it is the heart of the post.

Volume rendering: from a field to a pixel

Take that one ray and follow it. Walking it from the camera outward, step by step, accumulating what the fog does to the light traveling back toward you, is called ray marching, and doing that accumulation correctly is volume rendering.

Let us make that precise. Parameterize the ray by a distance $t$ from the camera: the point at distance $t$ along the ray is $\mathbf{r}(t) = \mathbf{o} + t\mathbf{d}$, where $\mathbf{o}$ is the camera position and $\mathbf{d}$ is the view direction. As $t$ grows we move away from the camera, deeper into the scene. At each point on the ray the field hands us two numbers:

• The density $\sigma(t)$, shorthand for $\sigma(\mathbf{r}(t))$. It is the amount of light-blocking per unit length at that point: the chance, per tiny step $dt$, that a photon traveling through is absorbed or scattered away. Clear air has $\sigma = 0$; the surface of a solid object has $\sigma$ shooting up very large. Density has no color and no direction; it only says how solid the fog is here.
• The emission, or emitted color, $c(t)$, shorthand for $c(\mathbf{r}(t), \mathbf{d})$. It is the color the fog at this point sends back toward the camera. This is where view dependence lives: the same point can emit one color when viewed head-on and another at a glancing angle, which is how the field represents a shifting highlight.

Now follow the light. Rather than trace it front-to-back, ask how much of the light from a given point survives the trip forward to the camera. Define the transmittance $T(t)$: the fraction of light emitted at distance $t$ that makes it all the way back to the camera without being blocked by the fog in front of it. Right at the camera nothing stands in the way, so $T(0) = 1$. As the ray pushes deeper, more fog accumulates in front, so $T$ can only fall.

How fast does it fall? Zoom in on one tiny step $dt$ at distance $t$. The fog there has density $\sigma(t)$, and across that sliver it removes a fraction $\sigma(t)\,dt$ of the light currently passing through, not a fixed amount, but a fixed fraction of whatever has survived this far. That one word, fraction, is the whole story. Because every step skims off a slice of what is left, the surviving light shrinks in proportion to itself:

$dT = -\,\sigma(t)\, T(t)\, dt.$

Anything that loses a fixed fraction of itself at every step decays exponentially, the same compounding, run in reverse, that grows a bank balance or dims a beam passing through one tinted pane after another. Accumulate the losses over the whole trip from the camera out to $t$ and that sentence becomes a formula:

$T(t) = \exp\!\left(-\int_0^t \sigma(s)\, ds\right).$

Unpack it from the inside out. The integral $\int_0^t \sigma(s)\,ds$ just adds up the density all along the path: the total fog piled between the camera and the point. Wrapping it in $\exp(-\,\cdot\,)$ squeezes that pile into a survival fraction between 0 and 1. With no fog in the way the integral is 0 and $T = 1$, so all the light gets through; as fog accumulates the integral climbs and $T$ slides toward 0, until the point is buried and its light never reaches you.

We now have every piece to write the color of the pixel. The point at distance $t$ emits color $c(t)$. It emits it at a rate set by how much fog is right there, $\sigma(t)$, because clear air emits nothing and dense fog emits a lot. And whatever it emits is dimmed by the factor $T(t)$ on its way back to the camera. Multiply the three together for the contribution of the slab at $t$, then add up every slab along the ray from the near edge $t_n$ to the far edge $t_f$:

$C = \int_{t_n}^{t_f} T(t)\, \sigma(t)\, c(t)\, dt, \qquad T(t) = \exp\!\left(-\int_{t_n}^{t} \sigma(s)\, ds\right).$

This is the volume rendering integral. Every term earns its place. Drop $c(t)$ and you have no color; drop $\sigma(t)$ and clear air would contribute as loudly as a solid wall; drop $T(t)$ and objects hidden behind others would show through as if the fog in front were not there. Together they say: a pixel is the sum of the colors along its ray, each weighted by how much fog emits it and discounted by how much fog hides it.

An integral is a statement, not a recipe a computer can run. To evaluate it, chop the ray into $N$ short segments and assume $\sigma$ and $c$ are roughly constant across each one. Index the segments by $\nu$, and let $\delta_\nu$ be the length of segment $\nu$. Two quantities fall out. First, the fraction of light a single segment removes. Over a segment of length $\delta_\nu$ with density $\sigma_\nu$, the surviving light is $e^{-\sigma_\nu \delta_\nu}$, so the fraction absorbed, the segment’s opacity or alpha, is

$\alpha_\nu = 1 - e^{-\sigma_\nu \delta_\nu}.$

Alpha runs from 0 (a perfectly clear segment) to 1 (a fully opaque one), and it is just the discrete stand-in for density: high density over the segment means alpha near 1. Second, the transmittance up to segment $\nu$ becomes a running product of what every earlier segment let through:

$T_\nu = \prod_{\mu < \nu} (1 - \alpha_\mu).$

Each factor $(1 - \alpha_\mu)$ is the fraction of light that survived segment $\mu$; multiply them all and you have the fraction that survives every segment in front of $\nu$. The continuous exponential of an integral has become a plain product, one factor per segment. Substituting both into the integral turns it into a sum:

$C = \sum_{\nu} T_\nu\, \alpha_\nu\, c_\nu.$

This is alpha compositing: walk the segments front to back, and each one lays down its color $c_\nu$, painted at strength $\alpha_\nu$ and faded by everything already in front of it, $T_\nu$. It is the exact same “over” operation that stacks semi-transparent layers in any image editor, here derived from the physics of light through fog rather than assumed. The interactive figure above makes this real, on the same captured scene from the top of the post, now with a slider on every splat’s opacity. Turn it down and each splat blocks almost nothing: light passes straight through and the scene is a translucent haze, transmittance near 1 everywhere. Turn it up and the front splats begin to hide the ones behind them, transmittance falls off with depth, and the solid photoreal image accumulates, one $\alpha_\nu\, c_\nu$ at a time.

That one equation, $C = \sum_{\nu} T_\nu\, \alpha_\nu\, c_\nu$, is the spine of this entire post. Fix it in your mind, because from here on nothing about the goal changes: every method we meet is producing exactly this weighted sum of colors along a ray. What changes is only how each method stores the field and finds the segments. NeRF, in the next section, will answer by asking a neural network for $\sigma$ and $c$ at hundreds of sampled points per ray. 3D Gaussian splatting will answer by skipping the ray march entirely and letting explicit blobs deposit their own $\alpha_\nu\, c_\nu$ onto the pixels they cover. Two machines, wildly different in speed, both evaluating this same sum.

NeRF: store the field in a network

The spine equation is hungry: to evaluate it we need a color and a density at every point we choose to sample along a ray. Where do those numbers come from? NeRF’s answer is disarmingly simple, which is to memorize them. Train a small function to recall the glowing fog of the last section, so that when you hand it a point and a viewing angle it hands back the fog’s color and density right there.

NeRF stores that rule in an MLP: a small neural network, a stack of matrix multiplies interleaved with simple nonlinear functions, that maps an input vector to an output vector. Here the input is a point together with a view direction, and the output is the pair the spine equation was asking for:

$(x, y, z, \theta, \phi) \to (c, \sigma).$

The three numbers $(x, y, z)$ locate a point in space; the two angles $(\theta, \phi)$ name the direction you are looking from. Out come precisely the two quantities from section 2: the emitted color $c$ and the density $\sigma$. Nothing about volume rendering changes. The MLP is just one way to supply the $(c, \sigma)$ that the compositing sum consumes at each sample, a stand-in for the abstract field $F$ we drew earlier. Notice that density is fed only the position: a wall is a wall no matter where you stand, so $\sigma$ ignores the angles. Color is fed both position and direction, and that is exactly where the view-dependent highlight lives.

There is a catch that nearly sinks the idea. Feed the raw coordinates straight into the network and the rendered scene comes out soft and blurred, stripped of fine detail. The reason is a known bias of these networks: left to themselves they prefer smooth, slowly varying outputs, the low frequencies, and strongly resist representing anything that changes sharply from point to point. But a crisp edge or a fine texture is a sharp, high-frequency change, so the raw-coordinate network smears exactly the detail we most want.

The fix is to stop feeding the network the bare coordinate and instead feed it a spray of sines and cosines of that coordinate taken at a range of frequencies. This is positional encoding, written $\gamma(p)$ for a single coordinate $p$:

$\gamma(p) = \big(\sin(2^0\pi p),\, \cos(2^0\pi p),\, \dots,\, \sin(2^{L-1}\pi p),\, \cos(2^{L-1}\pi p)\big).$

Each sine and cosine pair is a wave, and the multipliers $2^0, 2^1, \dots, 2^{L-1}$ make successive waves oscillate faster and faster. The slow waves let the network place broad features; the fast ones hand it a vocabulary for sharp edges and fine texture that the bare coordinate could never coax out of it. Apply $\gamma$ to each of $x$, $y$, $z$ (and to the view direction) before the first layer, and the same network that produced mush now resolves crisp detail. And there is the rub: NeRF is photoreal, and NeRF is slow. Look again at the spine equation and count the work. Every pixel is one ray; every ray needs many samples to approximate its integral; and every single sample is a full query of the network, a complete run through all those matrix multiplies. Multiply pixels by samples by the cost of one query and a single image can demand millions of network evaluations. The field is beautifully stored but agonizingly expensive to read back. That expense is the problem the next idea was built to solve.

The turn: rasterize, do not sample

Notice what actually makes NeRF slow. The scene is never stored anywhere you can point to; it lives implicitly inside the network’s weights, and the only way to learn what sits at a location is to ask, point by point, millions of times per image. So here is a different thought: instead of interrogating a network about mostly empty space, put the scene out in the open where you can see it.

Almost nothing about the goal changes, and that is the whole point. The pixel’s color is still the very same compositing sum from section 2, $C = \sum_{\nu} T_\nu\, \alpha_\nu\, c_\nu$, with each primitive contributing a color $c_\nu$ at strength $\alpha_\nu$, faded by everything already stacked in front of it, $T_\nu$. What changes is where those $\alpha_\nu$ and $c_\nu$ come from. NeRF manufactured them on demand, one network query per sample. Now the primitives themselves carry their color and opacity as stored properties, so there is no per-sample network to run at all. The expensive middleman is gone, and the same sum is left standing. Crucially, every number describing a blob (its position, its shape, its color, its opacity) is differentiable, which means the whole bag of blobs can be fit to your photos by gradient descent: render the current blobs, compare against a real photo, and nudge every parameter a little in the direction that shrinks the difference, repeating until the render and the photograph agree.

The word for throwing a primitive forward onto the image plane, rather than shooting a ray out to hunt for it, is splatting, and it is what makes this fast enough to render in real time. The one question left is what the blobs should be. The answer, and the reason the whole method is named for it, is Gaussians.

3D Gaussian splatting

We finally get to name the blob. Each of those soft clouds is a 3D Gaussian, and once you see what a single one is, the rest of the method is just arranging millions of them and reusing the sum we already built.

The primitive. A 3D Gaussian is the smooth, bell-shaped falloff you know from the “bell curve,” lifted into three dimensions. It is a function that is 1 at a center point and decays smoothly toward 0 as you move away in any direction. Written out,

$G(\mathbf{x}) = \exp\!\left(-\tfrac12\, (\mathbf{x}-\boldsymbol{\mu})^\top\, \Sigma^{-1}\, (\mathbf{x}-\boldsymbol{\mu})\right).$

Two objects define its shape. The mean $\boldsymbol{\mu}$ is a point in 3D space: it is simply where the blob sits, the location of its bright center. The covariance $\Sigma$ is a $3\times 3$ matrix that encodes the blob’s shape, size, and orientation all at once: how far it reaches, and in which directions it reaches farthest. A blob that reaches equally in every direction is a round ball, or isotropic. A blob that stretches by different amounts along different axes, wide in one direction and thin in another, is anisotropic, and that is exactly what you want: a thin flake to sit flat against a wall, a long sliver to trace an edge, a fat lump to fill a volume.

Walk the exponent to see why it falls off. The vector $\mathbf{x}-\boldsymbol{\mu}$ is the offset from the center to the point you are asking about. At the center that offset is zero, so the exponent is zero, and $\exp(0) = 1$: the blob is at full strength. Move away and the offset grows, the exponent goes increasingly negative, and $\exp$ of a negative number slides toward 0: the blob fades.

The matrix $\Sigma^{-1}$ sitting between the two offset vectors is what makes the falloff directional. It measures the offset not in plain distance but in units of the blob’s own spread: a step along a direction the blob is stretched costs little (you are still well inside the bell), while the same step along a thin direction costs a lot (you fall off the edge fast). So $\Sigma^{-1}$ turns raw distance from the center into “how many blob-widths away am I,” and $\exp(-\tfrac12\,\cdot\,)$ turns that into a smooth brightness that is 1 at the core and trails gently to nothing.

Why factor the covariance. Here is a snag that comes straight from section 4. We fit these blobs to the photos by gradient descent: render the current heap, compare it to a real photograph, and nudge every parameter a little to shrink the difference. That means $\Sigma$ has to be nudged too. But not every $3\times 3$ matrix is a legal blob shape. To describe a real ellipsoid, a covariance must be symmetric positive semidefinite: symmetric, and never assigning a negative spread to any direction. Freely nudging the six independent numbers inside $\Sigma$ will, sooner or later, push it out of that legal set, and you get a “shape” that curves the wrong way or collapses to nonsense, which the renderer cannot draw and the gradient cannot recover from.

The fix is to never store $\Sigma$ directly. Store instead the two ingredients that build it: a rotation and a set of scales.

$\Sigma = R\, S\, S^\top\, R^\top.$

Here $S$ is a diagonal matrix of three positive scales, the blob’s radius along each of its own three axes (this is the stretching), and $R$ is a rotation that turns those axes to any orientation (this is the tilting). Read left to right, the product takes a unit ball, stretches it by $S$ into an ellipsoid, and spins it by $R$. Any $\Sigma$ built this way is automatically symmetric positive semidefinite, a valid shape by construction, no matter how the optimizer pushes $R$ and $S$ around. We have moved the gradient descent onto three scales and a rotation, quantities that cannot represent an illegal blob, and recover the covariance from them only when we need it. Build a scene below: drop a few Gaussians, set their scales and orientations, then orbit the camera and watch each 3D blob project down to a 2D footprint, sort by depth, and blend.

Projecting a blob to the screen. A Gaussian lives in 3D, but a pixel lives on a flat screen, so before we can composite anything we have to ask what a 3D blob looks like from the camera: what 2D footprint does it cast?

Making that precise means carrying the covariance through the projection. The screen-space covariance $\Sigma'$, the $2\times 2$ matrix describing the flat elliptical splat, comes from the 3D covariance by

$\Sigma' = J\, W\, \Sigma\, W^\top\, J^\top.$

Read it as two steps applied to the blob, each also applied to its mirror on the other side, which is why every factor shows up twice. First $W$, the world-to-camera rotation: it re-expresses the blob in the camera’s own frame, turning the whole scene so the camera looks straight down its own axis. That handles orientation but not yet the flattening. Then $J$, the Jacobian of the perspective projection: perspective (far things look smaller, the projection that gives depth) is not a simple linear map, so we replace it, right at the blob’s location, with the linear map that best matches it there. That is what a Jacobian is: the local linear approximation of a curved transformation at one point. We do not need every entry of $J$; what matters is that it captures how the projection locally stretches and shears space around the blob, so applying it to $\Sigma$ squashes the ellipsoid into exactly the 2D ellipse the camera would see. This is the EWA (elliptical weighted average) splat, and it is precisely what you were watching in the lab: the thin sliver a tilted splat becomes when you orbit is this $J\,W$ at work, flattening the 3D ellipsoid into its edge-on footprint.

Color that changes with the view. A blob so far has one color, but section 1 warned us that real surfaces do not: the whole reason we chose a radiance field was to capture appearance that shifts with viewpoint.

Storing an arbitrary function of direction sounds expensive, but there is a compact way to do it. The set of all viewing directions forms a sphere, and there is a standard family of gentle patterns painted on that sphere, from a flat constant, through a few broad lobes, to finer ripples, called spherical harmonics. They are to the sphere of directions what the sines and cosines of section 3 were to a coordinate: a small basis you can mix to build up any smooth pattern. A blob then stores just a handful of mixing weights, and reconstructs its color for a given direction $\mathbf{d}$ as a weighted sum,

$c(\mathbf{d}) = \sum_{\ell,\, m} c_{\ell m}\, Y_{\ell m}(\mathbf{d}).$

Each $Y_{\ell m}$ is one of those fixed patterns on the sphere (the index $\ell$ sets how fine the pattern is, $m$ which variant at that fineness), and each $c_{\ell m}$ is the blob’s own learned weight for it. A few weights buy a soft, believable glint: keep only the first, constant term and the blob is flatly colored from every side; add the higher terms and it can brighten on one flank and dim on another, exactly the moving highlight. Drag the direction below and watch a fixed set of weights repaint the sphere as the viewpoint moves.

The abstract sphere above shows what the bands do. Here they are on the real capture: toggle the harmonics off and the sunflowers and the glazed jug go matte, one flat color per splat; toggle them on and the highlights come alive, sliding across the petals and the glaze as you orbit.

Back to the integral. Every piece is now on the table, and the last step is the one we have been promising since section 2: it is the same sum. To render a frame, project all the blobs to the screen with the EWA map, sort them front to back by depth, and composite them with the exact spine equation,

$C = \sum_{\nu} T_\nu\, \alpha_\nu\, c_\nu.$

Nothing here is new. The index $\nu$ now runs over the depth-sorted blobs covering a pixel instead of the sample slabs along a ray. Each blob’s color $c_\nu$ is the spherical-harmonic color read off in the current view direction. Each blob’s $\alpha_\nu$ is its stored opacity multiplied by the value of its projected 2D Gaussian at that pixel: full strength at the splat’s center, fading toward its elliptical edge, exactly the falloff the primitive was built to have. And $T_\nu$ is the same running product of $(1-\alpha_\mu)$ over the nearer blobs, the fraction of light that survives everything stacked in front. Alpha-composite them in order and the pixel is done.

This is why it runs in real time. There is no ray to crawl along and, decisively, no network to query per pixel: the color and opacity are read straight off each blob’s stored parameters, and the expensive per-sample MLP evaluation that made NeRF slow is simply absent. Sorting and rasterizing a heap of splats is work a graphics card does by the millions every frame, so the same weighted sum that took NeRF seconds per image takes splatting milliseconds.

Where this leaves us

Both methods evaluate the identical spine equation; they differ entirely in how they store the field and how much it costs to read back. The trade is compact:

The honest caveats matter. Strongly view-dependent effects remain hard: a few spherical-harmonic weights capture a soft glint well but struggle with sharp mirror reflections, refraction through glass, or a highlight that snaps across a surface, and both methods can smear these into a blur. Very large scenes strain the approach, since a city block or a whole building may need so many blobs that they no longer fit comfortably in memory. And that is the flip side of splatting’s speed: storing millions of explicit Gaussians, each with its position, covariance factors, opacity, and a stack of color weights, costs far more disk and memory than a compact network, which makes streaming and sharing these scenes its own open problem.

The direction of travel is toward keeping splatting’s speed while trimming its cost and widening what it can represent: fewer blobs for the same quality, compressed representations, and better handling of the reflective, glassy surfaces that still give it trouble. The spine equation, though, is not going anywhere. Whatever stores the field next will still, in the end, be evaluating that one weighted sum of colors along a ray.