Introduction to 3D Point Clouds

Open a point cloud from a recent project and look at what is actually in the file. You will find a list of coordinates, one per echo, in the order of a fixed angular sweep and in no other. Nothing in it indicates which return came from the structure the survey was commissioned for, which from the scaffolding in front of it, and which from the tree between the two. Everything in the line of sight entered on equal terms, because the ranging measurement distinguishes no surface from another. That is what timing pulses of light gives you. This form of data persisted from the airborne laser altimeters of the 1970s through the commercial laser scanners of the 1990s. As the cost has fallen, a single surveyor can now carry such an instrument to a site.

So how much of such a file is measurement, and how much is something computed afterwards? The 60 m by 60 m airborne window of figure 1.1 holds 90 632 returns, and every one of them was measured. The classification drawn in panel (c) was computed afterwards rather than measured. Over the next nineteen chapters we turn a record of this kind into a digital terrain model (DTM), a building reconstruction, or a restoration plan. Every step of that route is a computation of the second kind.

Let's read that definition against one measured point. Figure 1.3 opens a record from the same survey, a return from inside a tree crown. Every field in it is either a coordinate or a property of the pulse that produced it. So the whole content of a point cloud fits into thirty bytes repeated once per echo. That is all you are given. Records of that form are the primary output of LiDAR sensors and photogrammetric pipelines, and geomatics, remote sensing, robotics and computer vision all work on them.

So what do you need to know before the rest of the book will make sense? You need to know what the data holds and what its attributes mean. You need to know what other representations it converts into and what each conversion costs. You also need to know which sensors produce it and the vocabulary in which accuracy is stated and defended. Appendix A collects the vectors, rigid transformations, principal component analysis (PCA) and least squares the chapters draw on. Let's start with the data itself, since everything else in the chapter is a statement about it.

1.1 What a Point Cloud Is

Start with the coordinates, since none of them was measured directly. A laser scanner emits hundreds of thousands to a few million pulses per second towards the scene, times the echo of each, and converts the round-trip time to a range. It records the beam direction at the same moment. So your coordinates are computed from a distance along a known ray rather than from three measured lengths, and section 1.4 explains how that affects the accuracy you obtain. The set you obtain this way is the point cloud of definition 1.1, and for $n$ samples in three-dimensional Euclidean space we write it as:

$$\mathcal{P} = \left\{ \boldsymbol{p}_i = (x_i, y_i, z_i)^\top \in \mathbb{R}^3 \;\middle|\; i = 1, 2, \ldots, n \right\} \tag{1.1}$$

Now look at the braces, which matter as much as the coordinates inside them. A set carries no order and no adjacency. The index $i$ therefore records where a point is stored in the file and nothing else. Two points with neighbouring indices need not lie anywhere near each other. Think of an image, a regular grid of pixels, or a mesh, whose vertices carry explicit connectivity. A point cloud has neither, which makes it an unstructured, unordered collection of samples. That lack of structure gives you both the flexibility of the format and its computational cost.

• Advantage: you can represent any geometry at all, with no connectivity and no topological constraint, since a sample specifies nothing beyond the position it records. A tree with thousands of leaves, a cliff face with overhangs, the interior of a cave: you capture each of them by adding points wherever you need finer resolution.
• Challenge: nothing records neighbourhood relationships, surface normals or connectivity, so you compute all three yourself. How expensive is that? Answer the query “which points are close to this point?” by comparing every pair and it costs $O(n^2)$. Now put a real number in. A terrestrial scan of a building façade holds some 50 million points, which puts that sweep at an order of $10^{15}$ distances. An airborne survey of a city runs to several billion. You cannot compute that many distances, so every stage runs on a spatial index instead, a k-d tree or an octree. These partition space recursively so that a query visits only the cells near its point. Chapter 4 builds them before Part II uses them.

Now ask the opposite question. What can you infer from an empty region in the file? On its own, nothing, since three situations produce the same absence of points. The volume may be genuinely empty, hidden behind a nearer surface, or never sampled by the instrument. A point cloud records only where a surface returned a measurement, and it holds no evidence about anywhere else. So where free space has to be certified rather than assumed, use the occupancy grids and signed-distance fields of sections 1.2.2 and 1.2.4, which record the state of every location rather than the position of every return.

What turns a list of coordinates into a shape you can recognise? Density, and figure 1.1 shows it happening on one airborne window in three steps. Look at panel (a) first. You have 400 points over 60 m by 60 m, roughly a tenth of a point per square metre. Nothing in it reads as ground or as a building, since at that spacing no surface is sampled often enough to appear as one. Now take panel (b), the same window with all 90 632 points, where the ground, the buildings and the trees stand apart in the geometry itself with no label on any point. Panel (c) adds the survey's classification to those identical points, the one thing in the figure nobody measured. So geometry comes from the sensor, structure from density, and meaning only from a computation.

Density is only one axis of scale, and the other is size. Figure 1.2 puts five datasets on one logarithmic axis, and the gap to look at is the thousand-fold one between a single campaign and a national programme such as France LiDAR HD. That is why the storage and tiling of chapter 4 are a separate problem from the algorithms of Part II. Once a dataset outgrows the memory of your machine you process it tile by tile, whatever the algorithm.

1.1.1 Point Attributes

The record you have just read in figure 1.3 is stored in the LAS format, published by the American Society for Photogrammetry and Remote Sensing (ASPRS). Table 1.1 lists the ten per-point fields you will meet most often, the coordinates and nine attributes whose presence depends on the sensor and on the processing pipeline. Read the table by its middle column rather than its first. Why? A measured attribute, a computed one and an assigned one fail in entirely different ways, and only the measured one is evidence about the world on its own. Chapter 4 covers how they are stored and what each costs you in bytes.

Intensity is the attribute you will find most often. It measures the power of the returned pulse. That power rises with the reflectivity of the surface, falls as the incidence angle moves off perpendicular, and falls again with range. So three quantities are mixed into the one number you read. That mixture helps where geometry provides no distinction, as on asphalt and the markings painted on it. Both lie in the same plane, and they separate on intensity because the paint returns far more of the incident power. The same mixture makes raw intensity unusable as a measurement of material. Two returns from one material carry different values, and two from different materials can carry the same one. So calibrate it radiometrically before you discriminate materials with it (chapter 2), and until that is done I treat an intensity channel as an aid to interpretation and nothing more.

RGB colour is projected onto each point from a camera co-registered with the scanner, so the point cloud carries appearance as well as geometry. Remember that the ranging instrument never measured it, and section 1.3.1 works through the errors you can get from the projection at a depth discontinuity.

For airborne LiDAR, the return number and the number of returns record how the pulse interacted with the canopy on its way down. Why does a single pulse produce several returns? Because part of the beam passes between the leaves and carries on below. A semi-permeable surface therefore gives you a first return from the top of the canopy, intermediate ones from branches, and a last one from the ground. That multi-return capability is why you take LiDAR rather than photogrammetry to forested terrain, since a photogrammetric surface is reconstructed from what the cameras recorded and the cameras recorded the canopy. Now note the limit. Two returns resolve as separate only when the surfaces lie more than about half a pulse length apart, of order a metre. So low vegetation just above the ground merges with it into one return that no later processing can split. Full-waveform recording (chapter 2) digitises the whole echo instead, and overlapping returns can be separated afterwards.

Attribute	Origin	Description
$(x, y, z)$	computed	Cartesian coordinates in a local or global reference frame, computed from the range, the beam direction and, on a moving platform, the platform trajectory and calibration
Intensity ($I$)	measured	Amplitude of the returned laser signal, which relates to surface reflectivity
RGB $(r, g, b)$	measured	Colour from a co-registered camera
Return number	measured	Position of this return within the sequence its pulse produced
Number of returns	measured	Total returns generated by that single pulse
Classification	assigned	Semantic label such as ground, vegetation, or building
GPS time	measured	Timestamp of acquisition
Scan angle	measured	Angle of the laser beam relative to nadir
Normal $\boldsymbol{n}$	computed	Surface normal estimated from neighbouring points (chapter 7)
NIR	measured	Near-infrared reflectance

1.1.2 Density and Spacing

Before you choose any algorithm, ask yourself one question about the data. Can it resolve the thing you were commissioned to measure? Two numbers determine the answer. Point density is the number of points per unit area, written $\rho$ and quoted in points per square metre for airborne and mobile surveys. The point spacing is the typical distance between a point and its nearest neighbours, written $s$. For a roughly uniform point cloud the two are tied by $s \approx 1/\sqrt{\rho}$. Points spread evenly over an area give each point a cell of $1/\rho$ square metres, and the spacing is the side of that cell. That is a nominal grid conversion and not a measured nearest-neighbour distance. Scatter the same points at random over the same area and the mean distance to the nearest one falls to $1/(2\sqrt{\rho})$, half the nominal figure. Now let's work one case through. A survey at 4 pts/m$^2$ gives each point a quarter of a square metre, so you get a spacing near 50 cm. Want 25 cm instead? That takes four times the density and, in the air, four times the flying. So the square root is what makes resolution expensive.

That ceiling is hard, and no later processing lifts it. A feature narrower than the spacing may still intercept a sample, as a power line does, and its shape and its extent are still not in the data. So nothing narrower than 50 cm is reconstructable from that 4 pts/m$^2$ survey, however carefully you filter the points. Every choice of neighbourhood size in the chapters ahead has this limit as its floor. Figure 1.4 shows you the ceiling on real data, the same window at a quarter of a point per square metre, at two and a half, and at the 25.2 the tile delivers.

Specifications use a more careful vocabulary than we have, and a delivery is accepted or rejected on the difference. The nominal pulse spacing (NPS) and nominal pulse density (NPD) count pulses rather than points. They use only the first return of each pulse within a single swath, so multiple returns from one pulse do not inflate them (Heidemann, 2018). Their aggregate forms (ANPS, ANPD) count the overlap of adjacent swaths as well. You can see what follows. A delivery can show a high point density and still fail its pulse-density requirement. So when I check one against its contract I compute the pulse density myself, from the first returns of one swath. For a terrain product I also compute the ground-return density. In a dense canopy this can be an order of magnitude below the pulse density above it. The United States Geological Survey quality levels of chapter 3 are stated in these terms.

Within one survey the density is not a constant, and every part of the variation has a cause you can predict. Adjacent flight strips double it in their overlap. Along-track spacing follows from the ground speed. Across-track spacing follows from the angular step. A terrestrial scanner spreads a fixed angular step over a growing arc, so its spacing grows linearly with range. What a specification gives you, then, is an average over a delivery, and the local density around any one point can differ from it several-fold. So put the stated figure in the contract and the local figure in your parameters. You will meet that heterogeneity at every later stage, since chapter 5 resamples it away when uniformity matters, chapter 7 sizes neighbourhoods around it, and chapter 10 shows how it biases features computed at a fixed radius.

1.2 Point Clouds and Other 3D Representations

At this point you know what a point cloud holds, coordinates, a few attributes, and a density that fixes what can be resolved. What it does not hold is any statement about the surface between two samples, and that absence is why most downstream tasks require your data in another form. Four representations are in common use beside the point cloud. Only point-based and sparse-convolution networks work on irregular point sets or sparsely occupied voxel coordinates as they stand (chapter 19). Figure 1.5 renders one mobile laser scan of a pruned street tree through a single camera as all four, and table 1.2 compares the five families on the properties that separate them. Let's ask of each what it adds and what it costs.

1.2.1 Meshes

A mesh consists of vertices connected by edges to form triangular or polygonal faces. What does that connectivity give you? The topology of the surface, which vertices are connected and which faces are adjacent, and with it the right to say something about the surface between the samples. Three jobs need exactly that. Volume computation needs it, since a volume is an integral over a closed boundary and unconnected samples bound nothing. Finite element analysis needs it, since the elements are the faces. Shaded visualisation needs it, since a renderer shades a facet rather than a sample. A triangular mesh of a terrain surface, one height per planimetric position, is a triangulated irregular network (TIN). No instrument delivers a mesh. So you build one by surface reconstruction (chapter 15), and what you get is always an approximation, since the reconstruction interpolates across every gap between samples whether or not a surface was there. Chapter 15 builds one.

1.2.2 Voxel Grids

A voxel grid discretises 3D space into a regular grid of small cubes (volume elements, or voxels), the 3D analogue of pixels. Each is either occupied or empty or stores a value such as density or colour. What the grid gives you is addressability. Any location maps to its cell by a division and a rounding, so free space is explicit. An algorithm that requires regular spatial sampling, a 3D convolutional network above all, applies without modification. What it costs you is set by the resolution rather than by the amount of data, since memory grows as $O(r^3)$ for $r$ cells along each side. Now halve the cell size and you multiply the storage eightfold, whether or not the new cells contain anything. Octrees store only the occupied regions, which saves a great deal, since most of the volume of a scanned scene holds no surface. So choose the cell size against the smallest feature you have to keep. Anything narrower arrives as a single occupied cell, and you cannot recover it from that.

1.2.3 Depth Maps and 2.5D Representations

A depth map stores a single height or distance value for each cell of a regular 2D grid, indexing two dimensions and storing the third. This “2.5D” form suits a scene where each $(x, y)$ position has at most one $z$ value, as in many DTMs and rooftops viewed from above. That single value per cell is also the limitation. An overhang, a tunnel or a canopy places two or more surfaces above the same $(x, y)$, and the grid keeps one. You lose the others at the moment of conversion rather than degrading them, and no later step recovers them. So check a scene for stacked surfaces before you convert it.

1.2.4 Implicit and Neural Representations

A further family encodes the surface as a continuous function, which replaces the question of resolution with the question of what the function was fitted to. A signed-distance field stores at each location the distance to the nearest surface, negative inside and positive outside. The surface is therefore its zero level set and comes out at whatever resolution the deliverable requires. Radiance-field representations instead fit a model of scene appearance to a set of photographs. A neural radiance field (NeRF) encodes it in the weights of a network. 3D Gaussian Splatting encodes it in explicit Gaussian primitives optimised against the same photographs. Both suit photorealistic view synthesis, and neither suits certifiable measurement. Why the difference? You can trace a retained survey point back to the observation that produced it, with an instrument, a time and an error behind it. The geometry of a learned reconstruction is whatever reduced a rendering error against the input images. So a surface can be placed correctly for the wrong reason, and no per-point provenance tells you which. That cuts inside a point cloud too, since the interpolated points of chapter 5 were measured by nothing, and chapter 20 takes it up.

All four panels of figure 1.5 derive from the measured points alone, so every difference between them is a difference of representation and not of data. Let's read them in order. The point cloud in (a) is drawn with eye-dome lighting, a screen-space shading that darkens each point against the depth of its neighbourhood and so recovers the silhouettes a flat projection loses. For the mesh in (b) we bin the points into a 0.16 m occupancy grid. We smooth it with a narrow Gaussian. We extract the iso-surface with Marching Cubes (Lorensen and Cline, 1987). We then simplify by vertex clustering (Rossignac and Borrel, 1993), which merges every vertex sharing a 0.42 m cell into its centroid. Here that reduces the number of triangles from 21 362 to 2 710. That eightfold reduction follows from the cell size, two and a half times the grid the triangles were built on. For the voxel grid in (c) we assign every point to the fixed 0.30 m cell that contains it and draw a cube for each of the 2 098 occupied cells. The depth map in (d) is a range image on 0.08 m cells, 157 by 157 of them, each keeping the depth of the nearest point that falls in it. Now look at the white cells, the ones no point reaches. The notch they cut in the outline of the scanned patch is the region the tree occluded from the scanner, the free-space ambiguity of section 1.1 made visible.

Panel (b) places surface where no pulse measured any, which a renderer or a volume computation requires and a deformation measurement must not have (section 1.2.1). So I archive the points as the record of a project and treat every mesh, grid and raster as a derived product rebuilt from them. Which of them you can build at all depends on where the points came from, so let's look at the sources.

Property	Point Cloud	Mesh	Voxel	Depth Map	Implicit / neural
Structure	Unordered	Graph	Regular grid	2D grid	Continuous function
Topology	No	Yes	Implicit	Implicit	Implicit
Memory	$O(n)$	$O(n)$	$O(r^3)$	$O(r^2)$	$O(\text{params})$
Geometry	As measured	Approximated	Discretised	Projected	Fitted
Per-return attributes	Yes	No	Aggregated per cell	No	No
Overhangs	Yes	Yes	Yes	No	Yes
Typical use case	Raw survey, classification, change detection	Visualisation, FEM simulation, volume computation	Volumetric analysis, 3D CNNs, occupancy mapping	DTM generation, orthoimage production	View synthesis, learned reconstruction

1.3 Where Point Clouds Come From

The choice of source is the first engineering decision in a project, and the one no later processing revises. Six technologies produce point clouds in routine practice. All six deliver coordinates, so they share the algorithms downstream. Their density, accuracy, range and noise differ enough that the same algorithm needs different parameters on each (chapters 2 and 3). Here is what each is and what you would use it for.

1. LiDAR (Light Detection and Ranging): active sensors that range to a surface from the light they emit, which is why they work in darkness and need no texture. Pulsed time-of-flight systems time the echo of a short pulse. Concentrating the energy into a short burst takes them to long range and lets them record several returns from one pulse. Continuous-wave phase-shift systems recover the range from the phase of a modulated beam instead, which gives you a high rate and high precision at short range from a single return (chapter 2). The family depends on the platform. There is airborne laser scanning (ALS) and UAV-borne scanning (carried by an unmanned aerial vehicle). There is also terrestrial laser scanning (TLS) and mobile laser scanning (MLS, vehicle-mounted, handheld or backpack). The wavelength divides them into topographic and bathymetric. A typical ALS system reaches 10–50 pts/m$^2$ and TLS hundreds of points per square centimetre at close range. That gap follows from range and not from the ranging principle, since a fixed angular step spreads over a wider arc the further the beam travels. The pulse rate behind those densities is bounded in turn, since each echo has to be matched to its pulse, and chapter 2 explains how high-rate systems resolve that ambiguity.
2. Photogrammetry / Structure from Motion (SfM): SfM recovers the camera positions and a sparse set of points from features matched across overlapping photographs. Dense multi-view stereo (MVS) then gives nearly every pixel a depth. Both steps rest on matching image texture, which determines where they work. On a well-textured surface the point clouds match LiDAR in density, with higher noise and no vegetation penetration. On a blank wall and on water they thin or fail, since nothing there distinguishes one pixel from its neighbours. They carry colour at no additional cost to you, since the colour was the measurement.
3. Structured light scanners: project known patterns of light onto a surface and infer geometry from the deformation of the pattern, which supplies the texture photogrammetry has to find. The projector fixes a working volume, so range is limited to a few metres, and the accuracy inside it is the finest of the six. That is why you meet them in industrial metrology, quality control and heritage documentation.
4. Time-of-Flight (ToF) cameras: compact depth sensors in consumer cameras and some mobile devices. They apply the time-of-flight or phase-shift principle at every pixel of an array at once rather than sweeping one beam. They therefore give you real-time depth maps that convert directly to point clouds. Their accuracy is on the order of a centimetre, an order below a short-range terrestrial scanner. They suit indoor mapping and robotics where speed matters more than precision, and I would not put one on work where a tolerance has to be certified.
5. Radar and SAR: Synthetic Aperture Radar produces 3D surface models through interferometry (InSAR). This compares the phase of two acquisitions rather than their amplitudes and so resolves a change in range far finer than either image resolves detail. Its spatial resolution is lower than LiDAR's, and it works in all weather and at night, since the sensor supplies its own illumination at a wavelength atmospheric water droplets do not scatter. Deformation monitoring over wide areas is where that combination is useful.
6. Sonar and multibeam echosounders: the underwater equivalent of LiDAR, timing acoustic pulses to map the seabed and submerged structures. Sound travels in water where light at most wavelengths is absorbed within a few metres. Bathymetric LiDAR works in both media, using green laser wavelengths (532 nm) that penetrate shallow water.

So which one do you choose? No single source dominates (figure 1.6), and the choice follows the task rather than the merits of the instrument. Look first at the four capability columns, which mark what the physics of acquisition fixes and no later computation supplies. Colour comes only from a camera that observed the scene (section 1.3.1). Canopy penetration comes only from pulses that record several returns. All-weather work comes only from microwaves and sound, and work in water only from green light and sound. Satisfy those on the day, since a missing capability is data you never collected rather than a quality you can improve later. Now the rest is ordinary engineering. TLS is your default for millimetre work on a fixed structure. Photogrammetry suits a job where colour or low cost matters most and the surface is opaque and well textured. Airborne LiDAR is what you need to penetrate vegetation or to cover hundreds of square kilometres. And InSAR reports a displacement rate rather than a position, so it answers a different question altogether.

1.3.1 Point Cloud Colourisation

LiDAR scanners measure geometry and not colour, and a good many deliverables need both. Colourisation gives each point an RGB colour taken from a co-registered camera. What do you need for it? Known camera intrinsics (focal length, distortion) and extrinsics (position and orientation relative to the scanner), since without both nothing fixes which pixel a point falls on. You either calibrate the two during acquisition, with an integrated camera on MLS and ALS systems, or estimate them through co-registration, with an external camera aligned to a TLS scan. Perspective projection then finds the pixel for each point and transfers its colour. Colour-based classification needs that information, since colour separates surfaces geometry leaves indistinguishable.

Now, the projection has one failure mode, and you should expect it on every job. Wherever the camera and the scanner occupy different viewpoints, parallax and unshared occlusions place a nearer surface on the camera ray. A background point therefore takes the colour of a foreground surface. A colourised point cloud can therefore look photorealistic and still carry wrong colours along every depth discontinuity, a defect that survives inspection because it is thin and everywhere rather than large and local. A depth test from the camera viewpoint discards a point when a nearer point projects onto the same pixel. It removes these errors wherever the nearer surface is sampled densely enough to cover the pixel. Where the foreground is too thin the leak remains, which is why I check a colourised point cloud along its silhouettes first.

1.4 Accuracy and the Error Budget

Remember that no coordinate was measured directly (section 1.1). Each one combines a measured range, a measured beam direction and, on a moving platform, a measured position and orientation. Every one of those carries an error that propagates into your result. That is why the accuracy classes of section 1.3 run from the millimetre level of a tripod-mounted scanner to the centimetre-to-decimetre level of an airborne survey although the ranging principle is the same. Two words in common use describe two different problems, and you will be asked for both in any report you sign. Accuracy is the closeness of a measurement to the true value. The precision is the closeness of repeated measurements to each other. The two are set against each other in figure 1.7, precision across and accuracy down. An instrument precise but inaccurate is dominated by systematic error, a bias with a cause such as a calibration offset or an atmospheric delay. An instrument accurate on average but imprecise is dominated by random error, noise that changes from measurement to measurement.

Random error averages down. The mean of $n$ independent measurements has a standard deviation of $\sigma/\sqrt{n}$, so a hundred points reduce the noise on their mean by a factor of ten. Everything in that sentence hangs on the word independent. Write the variance of a mean error as $\operatorname{Var}(\bar e) = n^{-2}\sum_i\sum_j \operatorname{Cov}(e_i,e_j)$ and the cross terms become visible. An error every point shares, from the trajectory or from a calibration, enters every pair of the sum and survives the average undiminished. So what a larger neighbourhood has to give you is independent information and not simply more points, which is one half of the scale-of-support trade-off that chapter 5 states in full at its first pipeline instance. Systematic error does not average down at all. Every point in a poorly georeferenced scan is shifted by the same amount, and only calibration and independent control remove a bias like that. And when a result looks wrong, the first thing I check is which of the two is responsible, because a remedy aimed at the wrong one costs a day and changes nothing.

The sources combine into an error budget, the accounting of how much each contributes to the total. Independent random errors combine in quadrature, $\sigma_{\text{total}} = \sqrt{\sigma_1^2 + \sigma_2^2 + \cdots}$, so each enters through its square rather than through its size. What does that do to your idea of which term to reduce first? Let's work the three cases on the right of figure 1.7 through, in millimetres. Two equal terms of 5 give $\sqrt{25+25}$, that is 7.1, so the second adds 2.1 to the first. A 10 beside a 2 gives $\sqrt{100+4}$, that is 10.2, so the smaller adds two tenths of a millimetre. A 10 beside a 5 gives $\sqrt{100+25}$, that is 11.2, so a term at half the largest still adds more than a millimetre. So terms of comparable size all matter, and I ignore one below about a third of the largest, since at that ratio it moves the total by some five per cent. That rule holds only for independent random errors, since biases add linearly, with their signs, and errors that share a cause do not cancel.

You will meet this budget again wherever it becomes decisive. Section 2.10 builds the full LiDAR budget from the instrument outward. In chapter 3, we attach typical accuracies to each platform. In chapter 8, we add the residual of alignment. In chapter 14, the budget becomes the level of detection (section 14.3.5), the smallest change you can distinguish from noise. Every derived product inherits the budget of the points beneath it, because no computation removes an error the measurements already contain.

1.5 The Processing Pipeline

No single computation turns raw points into a classified map, a 3D building model, or a terrain surface. Why is a whole sequence forced on you? Because the raw record carries several unrelated defects at once, outliers among the returns, scans not yet registered into one coordinate system, and no semantic label on any point. So the conversion runs through stages that together form a processing pipeline, each resolving a defect or computing an input a later stage needs. That order is not arbitrary, since normals cannot be fitted before outliers are removed and segmentation cannot use normals that do not yet exist. Nor is it rigid, and you will skip, repeat and reorder stages as your application requires. Figure 1.8 lays out one common route, and the dashed frame marks the stages chapters 19 and 20 revisit with learned methods.

Now read it as a chain of dependencies. Spatial indexing (chapter 4) builds the structure every later stage uses to find the points near a given one. Sampling (chapter 5) thins a point cloud denser or less evenly spaced than the task needs, which is already a decision about the scale of support. Filtering (chapter 6) removes the outliers left by sensor noise, multipath reflections and moving objects, since an outlier left in place enters every neighbourhood that contains it. Normal estimation (chapter 7) fits a local surface to each neighbourhood and returns the perpendicular direction, which segmentation, reconstruction and most descriptors require. Registration (chapter 8) brings scans from different viewpoints, or different times, into one coordinate system. It draws on those normals, since its point-to-plane form measures the distance from a point to a surface rather than to another point.

Analysis then adds meaning to the geometry. Ground extraction (chapter 9) separates the terrain from everything standing on it, which fixes the reference height every object above it is measured against. Local features (chapter 10) are computed from the local covariance and indicate whether a neighbourhood is linear, planar or volumetric, standing in for wire, surface and vegetation. Descriptors (chapter 13) extend the same computation into rotation-invariant signatures for matching. Segmentation (chapter 11) groups the points and clusters the groups into object instances. Classification (chapter 12) labels every point. Change detection (chapter 14) compares two epochs and reports only differences exceeding the measurement uncertainty, for the reason section 1.4 gave. Surface reconstruction (chapter 15) closes the samples into a continuous surface. Object reconstruction (chapter 16) fits parametric models to roofs, tree crowns, road corridors and power lines. Notice how little of that you could reorder.

1.5.1 Derived Products

You will rarely hand over a processed point cloud itself. The people who commissioned the survey work in rasters, drawings and models rather than in points. They expect terrain models and orthophotos, 3D city models and building information models (BIM), or the plans, sections and elevation drawings many commissions contract for. The last family is treated on its own in section 16.9.1. Figure 1.9 orders them into three tiers by how much processing stands between your points and the product. A terrain raster needs the ground extraction of chapter 9 and little else, while a semantic city model needs classification and reconstruction on top. The taxonomy follows canonical references. For the terrain, vegetation and building products, it follows Vosselman and Maas (2010). For the scene-understanding family, it follows the ASPRS LAS 1.4 specification (ASPRS, 2019) (chapter 4). For the derived rasters, it follows the USGS Lidar Base Specification (Heidemann, 2018), since maintained as a versioned online standard (USGS, 2025). For the level-of-detail definitions, it follows the OGC CityGML 3.0 standard (OGC, 2021).

The progression has four processing levels (figure 1.10), each abstracting further from the raw measurements. You might expect the data volume to fall at every level. It does not. Preprocessing thins the point cloud, so the volume drops there. Analysis then attaches normals, features and labels to every surviving point, which can carry the enriched point cloud past the raw scan in size. Only the final products greatly reduce the volume. So size your storage plan on the labelled point cloud rather than on the delivery. What none of it tells you is how to choose a method inside a stage, which is the last thing this chapter explains to you.

1.6 How to Read This Book

The chapters develop the pipeline stages in order, so the book reads front to back as one continuous course. If this is your first course, take that route. Read this chapter in full, then the ranging principle (section 2.1) and the error budget (section 2.10) from chapter 2, then the algorithms of Part II. But few professionals read that way, and you should enter wherever a problem points you, provided you bring what that part assumes. Part II assumes the definitions and the accuracy vocabulary of this chapter. Part III assumes the neighbourhoods and normals Part II computes. Part IV assumes the labels Part III produces. Part V assumes all three, since a learned method is trained on whatever the earlier stages delivered.

Are you a surveying practice extending into mobile or UAV scanning? Start at chapter 3 and chapter 8, which settle what the new platform can deliver and how its scans are brought together. An architecture, engineering, and construction (AEC) professional starts at the scan-to-BIM chain of chapters 11, 16 and 17, where measured points become the objects a model is built from. A machine learning researcher can invert the order, starting from chapters 12, 19 and 20 and reading backwards into the geometric chapters. These say what the data measures, why it is irregular and which invariances a network must respect.

Three threads connect every chapter to every other, whatever your entry point. The error budget of chapter 2 sets what you can claim from any result, appendix A carries the common mathematics, and section 1.7 the implementations.

1.7 Software and Reproducibility

This book shows no screenshots and prints no code. I still owe you a map of the software you will actually run, because the algorithms of the following chapters reach a survey only through it. So here is the ecosystem as it stands, arranged by the role each tool plays rather than by vendor. The roles have been stable for fifteen years. The names inside them turn over, so read the names as the current occupants of each role. Before you build a production stack on any of them, check the companion website, where I keep this list current. Tables 1.3 and 1.4 collect them, each marked with the role it plays.

Open-source libraries and processing engines. The reproducible form of a project is the code here, which runs identically on every dataset it is given. PDAL (the Point Data Abstraction Library) is the engine most pipelines are built on. It reads and writes every common format and chains filters described in a JSON pipeline file. It exposes the same pipeline to Python, so a classification or a tiling run is a text file you can archive and rerun. PCL (the Point Cloud Library) is the C++ library behind much of the robotics and registration literature, with filters, normals, features, ICP and its variants, segmentation and surface reconstruction under one API. Open3D covers the same ground with a Python-first design, registration, Poisson reconstruction and a viewer included, and carries a deep-learning extension. CGAL supplies the computational geometry the others use, from point set processing to the reconstruction algorithms of chapter 15, with the exactness guarantees those algorithms need. lidR is the R package that forestry runs on, from ground classification and canopy height models to individual tree detection, over a catalogue of tiles processed in parallel. LAStools is the long-standing command-line toolbox for LAS and LAZ, partly open and partly licensed, still the fastest route through many routine ALS tasks. The surrounding tools include laspy and PDAL's Python bindings for reading in Python. They include GDAL for every raster a point cloud produces. They also include Entwine and Untwine for the tiled indexes of chapter 4, and the LASzip library that makes LAZ and COPC possible.

Desktop software. Quality control belongs in a workbench with a viewport, where you open a point cloud, turn it, measure it and judge it by eye. You see a misaligned scan or a stripe of outliers in seconds, before any statistic flags it and sometimes where none would. CloudCompare is the open-source workbench, and it is more than a viewer. It includes registration, cloud-to-cloud and M3C2 distances, subsampling, segmentation and a scriptable command line, which is why it appears in so many published workflows. MeshLab does the same for meshes, and QGIS renders point clouds through PDAL beside the vector and raster layers a GIS project already holds. The commercial workbenches are organised by the industries they were developed for. Terrasolid's TerraScan is the reference for airborne classification and strip adjustment and runs inside MicroStation. Leica Cyclone, Trimble RealWorks, FARO SCENE and Autodesk ReCap serve terrestrial and BIM-bound work, registration and scan-to-BIM handover above all. Global Mapper with its LiDAR module and Esri ArcGIS Pro bring point clouds into general GIS practice. The photogrammetry packages, Agisoft Metashape and Pix4D, produce point clouds from images and then process them with much the same toolset. The instrument makers ship their own software for the first steps, RIEGL's RiPROCESS and the Leica and Trimble field suites among them. That is where trajectory processing and calibration are usually done before the data reach anything above.

Web delivery and streaming. A point cloud of a billion points is delivered today through a browser rather than a file transfer. Potree is the open-source web viewer that made that ordinary, streaming an octree of tiles to a WebGL client. The same idea extends to the OGC 3D Tiles standard through Cesium (chapter 17), so that point clouds, city models and terrain stream through one client. The formats behind both, EPT, COPC and 3D Tiles, are treated in chapter 4, and they matter more than the viewers, since a client can be replaced and an archive cannot.

Cloud platforms and software as a service. The processing you once ran on a workstation is increasingly sold as a hosted service, and you should know the kinds before you meet the brands. Hosting and tiling services, Cesium ion the best known, take your point cloud and return a streamable tileset. Reality-capture platforms include Leica Reality Cloud Studio on HxDR, Trimble Connect, Autodesk Construction Cloud, Bentley iTwin, NavVis IVION and Pointerra. They host the scans of a project beside its design models and let a client walk through them without any software of their own. Processing services classify or reconstruct on your behalf. These include Flai and Pointly for automated and assisted classification, Pix4Dcloud for photogrammetric processing, and ArcGIS Online for scene layers in the I3S encoding. Every one of these changes where your data reside and who holds the parameters. The algorithm remains one of the methods this book describes. So before you contract for a hosted service, ask the two questions chapter 4 will make precise. In what format do the results come back to you? Can you rerun the same processing with the same parameters a year later?

Deep-learning frameworks. The learned methods of chapter 19 run on PyTorch, and on the point cloud libraries built over it. Pointcept carries the current transformer backbones. The segmentation and detection baselines are provided by Open3D-ML and MMDetection3D. The sparse convolution engines, MinkowskiEngine, spconv and TorchSparse, supply the lattice route of section 19.4.1. Whether you run a model somebody else trained or fit one yourself, check its behaviour on the acquisition conditions you intend to work in, for the reason section 19.12 gives.

	Tool	What it is used for here	Origin	Terms
Library	PDAL	pipelines of readers, filters and writers, in JSON and Python	Hobu and contributors	open source, BSD
Library	PCL	C++ filters, features, registration, segmentation, reconstruction	Open Perception	open source, BSD
Library	Open3D	Python and C++ registration, reconstruction, visualisation	Intel Labs	open source, MIT
Library	CGAL	exact computational geometry, point set processing, reconstruction	CGAL Project, GeometryFactory	open source, GPL and LGPL, commercial licence available
Library	lidR	forestry processing of ALS catalogues in R	J.-R. Roussel and contributors	open source, GPL
Library	LAStools	command-line LAS and LAZ processing	rapidlasso	partly open source, partly licensed
Library	laspy	LAS and LAZ reading and writing in Python	community	open source, BSD
Library	GDAL	the rasters a point cloud produces	OSGeo	open source, MIT
Library	Entwine, Untwine	tiled octree indexes (EPT, COPC)	Hobu	open source, LGPL and GPL
Library	LASzip	the compression behind LAZ and COPC	rapidlasso	open source
Desktop	CloudCompare	workbench: viewing, registration, distances, M3C2, segmentation	D. Girardeau-Montaut and contributors	open source, GPL
Desktop	MeshLab	mesh editing and repair	ISTI-CNR	open source, GPL
Desktop	QGIS	point clouds beside vector and raster layers, through PDAL	QGIS.org, OSGeo	open source, GPL
Desktop	TerraScan	airborne classification, strip adjustment, on MicroStation	Terrasolid	commercial
Desktop	Leica Cyclone	terrestrial registration and scan-to-BIM	Leica Geosystems, Hexagon	commercial
Desktop	Trimble RealWorks	terrestrial registration and deliverables	Trimble	commercial
Desktop	FARO SCENE	terrestrial registration	FARO	commercial
Desktop	Autodesk ReCap	scans into the Autodesk design tools	Autodesk	commercial
Desktop	Global Mapper	GIS with a LiDAR module	Blue Marble Geographics	commercial
Desktop	ArcGIS Pro	LAS datasets inside a GIS	Esri	commercial
Desktop	Metashape, Pix4D	point clouds from photographs	Agisoft, Pix4D	commercial

	Tool	What it is used for here	Origin	Terms
Web	Potree	streaming octree viewer in the browser	M. Schütz, TU Wien	open source, BSD
Web	Cesium	3D Tiles client for point clouds, city models and terrain	Cesium GS	open source, Apache
Cloud	Cesium ion	tiling and hosting as a service	Cesium GS	hosted service
Cloud	Reality Cloud Studio	scan hosting and processing on HxDR	Leica Geosystems, Hexagon	hosted service
Cloud	Trimble Connect	project data beside design models	Trimble	hosted service
Cloud	Autodesk Construction Cloud	scans beside the construction model	Autodesk	hosted service
Cloud	Bentley iTwin	reality data inside a digital twin platform	Bentley Systems	hosted service
Cloud	NavVis IVION	indoor and enterprise scan navigation	NavVis	hosted service
Cloud	Pointerra	hosting and analysis of very large clouds	Pointerra	hosted service
Cloud	Flai, Pointly	automated and assisted classification	Flai, Pointly	hosted service
Cloud	Pix4Dcloud	photogrammetric processing	Pix4D	hosted service
Cloud	ArcGIS Online	scene layers in the I3S encoding	Esri	hosted service
Deep learning	PyTorch	the framework the point networks are written in	Linux Foundation, Meta	open source, BSD
Deep learning	Pointcept	the current transformer backbones	Pointcept contributors	open source, MIT
Deep learning	Open3D-ML, MMDetection3D	segmentation and detection baselines	Intel Labs, OpenMMLab	open source
Deep learning	MinkowskiEngine, spconv, TorchSparse	sparse convolution on the lattice route	NVIDIA, community, MIT HAN Lab	open source

Practical work moves between these roles rather than settling in one. I open an unfamiliar dataset in CloudCompare before writing anything. I process it with PDAL or LAStools while the sequence is still changing. Once it has stopped changing, I move it into a library pipeline so that the project can be reproduced without me. That last step is what reproducibility means in this book. It requires five lines in a project record: the tool and its version, the parameters, a hash of the input files, the coordinate reference system, and the environment the run was made in. Record that environment as a container image or an environment file. A result that arrives without those five cannot be repeated, and a result that cannot be repeated cannot be checked, which is the standard every deliverable in chapter 18 is held to.

You can now say what a point cloud is and what its attributes record. You know which sources produce it and which capabilities the physics of acquisition fixes. You also know how an error that averages down differs from one that does not. What you cannot yet do is put a number on it. Chapter 2 follows a single laser pulse from emission to a coordinate and builds the error budget every chapter after it inherits.

1.8 Notes and Comments

The modern era of ALS began in the late 1970s and 1980s with the airborne laser profiling instruments developed at NASA's Wallops Flight Facility (Krabill et al., 1984). The computational thread came a decade later, when Hoppe et al. (1992) demonstrated that geometric information could be extracted from raw, unstructured 3D points through surface reconstruction. A common mistake among newcomers, including those arriving from computer vision, is to treat a point cloud as a 3D image with an ordering the point cloud does not have. That mistake is costly. Every convolution, stride and neighbour lookup available to you directly on an image has to be built first from the coordinates.

Commercial airborne LiDAR systems became available in the mid-1990s, a history chapter 2 recounts in full, and the technology then spread from aircraft to tripods, vehicles and drones. The sensors have changed every few years while the geometric problems have not. That is why this book is organised around the processing pipeline rather than the instruments, and why each chapter states its problem before it names a method. Understanding the difference between filtering, segmentation and classification therefore lets you follow the literature as the sensors change.

Two threads in particular have moved fast. Guo et al. (2020) review deep learning for 3D point clouds and record segmentation accuracies that hand-crafted feature pipelines had not approached. Those comparisons hold for the datasets, splits and evaluation dates they name. A benchmark rank transfers to other data only where the sensor, the density and the landscape resemble the benchmark's. National mapping programmes have changed in step. As a snapshot in 2026, the French LiDAR HD programme was surveying metropolitan France at a specified minimum of 10 pulses per square metre and approaching its first complete coverage. Both come after the geometric material they rest on, the learned classifiers in chapters 12 and 19 and the national programmes in sections 3.11 and 17.10. Jump into a deep-learning tutorial without knowing what a covariance matrix reveals about a neighbourhood and you obtain a working classifier with no way to anticipate where it fails.

The field has not settled whether neural representations should replace point clouds outright. For survey delivery I would compare the traceability and the independent geometric checks each supports before replacing the measured point layer with a NeRF or with 3D Gaussian Splatting, for the reason section 1.2.4 gives. Chapter 20 treats the two as complementary.

Wehr and Lohr (1999) give an early overview of ALS physics. The geometric relationships underlying ALS acquisition are derived by Baltsavias (1999). The book by Vosselman and Maas (2010) remains the standard reference on airborne and terrestrial scanning together, which is where I would send you first. Shan and Toth (2018) cover topographic laser ranging in depth. For the built environment beyond point clouds, the open-access textbook by Arroyo Ohori et al. (2026) treats meshes, voxels, and the CityGML and CityJSON data models taken up in chapter 17. The historical milestones are surveyed in section 2.11 of chapter 2.

The software named in section 1.7 describes 2026 and will date, so the companion website keeps that list current, with the licence and the maintainer of each entry.

The term is also written “LIDAR” (an acronym for Light Detection And Ranging), “Lidar” (a word from “light” and “radar”), or “lidar”. There is no agreed convention, and this book writes “LiDAR”, which keeps the acronym visible and stays readable in running text.

References

• Arroyo Ohori, K., Ledoux, H. and Peters, R. (2026). 3D Modelling of the Built Environment. Delft University of Technology. https://github.com/tudelft3d/3dbook.
• American Society for Photogrammetry and Remote Sensing (2019). LAS Specification, Version 1.4, Revision 15. American Society for Photogrammetry and Remote Sensing, Bethesda, MD.
• Baltsavias, E. P. (1999). Airborne laser scanning: basic relations and formulas. ISPRS Journal of Photogrammetry and Remote Sensing, 54(2–3), 199–214. doi:10.1016/S0924-2716(99)00015-5.
• Guo, Y., Wang, H., Hu, Q., Liu, H., Liu, L. and Bennamoun, M. (2020). Deep learning for 3D point clouds: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(12), 4338–4364. doi:10.1109/TPAMI.2020.3005434.
• Heidemann, H. K. (2018). Lidar Base Specification (ver. 1.3, February 2018). U.S. Geological Survey (report). doi:10.3133/tm11b4.
• Hoppe, H., DeRose, T., Duchamp, T., McDonald, J. and Stuetzle, W. (1992). Surface reconstruction from unorganized points. In Proceedings of the 19th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH), pp. 71–78. doi:10.1145/133994.134011.
• Krabill, W. B., Collins, J. G., Link, L. E., Swift, R. N. and Butler, M. L. (1984). Airborne laser topographic mapping results. Photogrammetric Engineering and Remote Sensing, 50, 685–694.
• Lorensen, W. E. and Cline, H. E. (1987). Marching cubes: A high resolution 3D surface construction algorithm. ACM SIGGRAPH Computer Graphics, 21(4), 163–169. doi:10.1145/37402.37422.
• Open Geospatial Consortium (2021). OGC City Geography Markup Language (CityGML) Part 1: Conceptual Model Standard, version 3.0.0. OGC 20-010, Open Geospatial Consortium.
• Rossignac, J. and Borrel, P. (1993). Multi-resolution 3D approximations for rendering complex scenes. In Modeling in Computer Graphics: Methods and Applications, pp. 455–465. doi:10.1007/978-3-642-78114-8_29.
• Shan, J. and Toth, C. K. (2018). Topographic Laser Ranging and Scanning: Principles and Processing. CRC Press, 2 ed.
• U.S. Geological Survey (2025). Lidar Base Specification Online, 2025 rev. A. U.S. Geological Survey, National Geospatial Program. https://www.usgs.gov/ngp-standards-and-specifications/lidar-base-specification-online.
• Vosselman, G. and Maas, H. G. (2010). Airborne and Terrestrial Laser Scanning. Whittles Publishing.
• Wehr, A. and Lohr, U. (1999). Airborne laser scanning–-an introduction and overview. ISPRS Journal of Photogrammetry and Remote Sensing, 54(2–3), 68–82. doi:10.1016/S0924-2716(99)00011-8.