Radiance Fields and Gaussian Splatting

Every coordinate you have handled in the preceding nineteen chapters came with a named sensor, a time of measurement, and a record of how it was derived. You can filter it, register it, classify it, mesh it and put it into a city model, and at each of those stages you can say where the number came from. What none of that machinery gives you is a photorealistic view of the scene from a camera position nobody occupied. The methods of this chapter give you exactly that, and they do it by fitting a scene model to photographs instead of measuring the scene. So they recover geometry only as far as geometry is needed to explain appearance. What they minimise is the difference between a rendered image and a photograph, and a photograph records how the scene appeared from one viewpoint rather than where its surfaces lie. So the break with the rest of the book is a break in the evidence and not in the mathematics. You'll find the algorithms themselves familiar.

You'll be offered one of the two families that have dominated the visual computing literature since 2020. Neural Radiance Fields (NeRF) encodes the scene in the weights of a coordinate network, from dozens of calibrated photographs and one to two GPU-days of training, and renders photorealistic novel views. 3D Gaussian Splatting, three years later, encodes it as a collection of learned primitives and reaches a comparable or better result in 27 to 42 minutes across the three benchmark suites on a single GPU. How fast does it render? Here is the number: over a hundred frames per second on those suites, and above thirty at 1080p. But the scene then occupies a model file of several hundred megabytes before compression, so what limits your delivery is the size of that file rather than the time it took to produce. And neither method replaces the point cloud for surveying or heritage measurement, and the reason lies in the evidence behind their coordinates rather than in the quality of the images they render. I use both families as an appearance layer over a measured record. The heritage façade is where that decision is taken most often. You can see why: a photorealistic façade is convincing enough that the temptation to read it as a measurement-grade record is strongest there.

What does the smallest complete instance of that definition look like? Figure 20.5 holds it. The entire input is 37 photographs of a bronze bell and the camera poses supplied with them, and what comes out of the optimisation is a set of network weights. Note what is absent here: no instrument measured the bell. So what changes at this point in the book is the record of the scene itself rather than the processing applied to it. The preceding chapters worked on the noise, the density, the coordinates and the classes of a point cloud, and never on the choice of the point cloud as the record.