Deep Learning on Point Clouds

A team of three I advised once spent six GPU-weeks training a Point Transformer V2 on heritage data. A pretrained MinkUNet, fine-tuned for a single afternoon on the same labels, then beat it. That outcome is not an accident of one project. Between PointNet in 2017 and Point Transformer in 2021, nearly every architecture published for point clouds reported gains on Semantic3D, S3DIS, or ScanNet, and yet the operational picture stayed more sober: the mid-70s mIoU that the strongest backbones report on Semantic3D drops below 60% when the same architecture is transferred to forested terrain it has never seen, and on a sparsely labelled airborne tile a tuned KPConv outperforms a fashionable transformer by a margin smaller than the standard deviation across training seeds.

Deep learning on point clouds names the family of neural architectures that consume an unordered set $\{(\mathbf{p}_i, \mathbf{f}_i)\}_{i=1}^{n}$ of coordinates $\mathbf{p}_i \in \mathbb{R}^3$ with optional per-point attributes $\mathbf{f}_i$ directly, and learn task-specific feature representations end-to-end from data, in place of hand-crafted ones. Such an architecture must be invariant to the $n!$ orderings of its input and must accommodate the irregular structure and variable size that distinguish point sets from the regular grids assumed by standard convolutional networks. The chapter picks up the pipeline where the feature and classification chapters (Chapters 10 and 12) left it. The eigenvalue features, FPFH descriptors, and tuned classifiers of those chapters demand expert knowledge and turn brittle when conditions change, and the architectures below replace that manual feature engineering with end-to-end optimisation.

The architectures of this chapter still lean on earlier machinery. The farthest point sampling of Chapter 5 drives PointNet++, the supervoxels of Chapter 11 underpin Superpoint Graphs, and the learned representations built here carry forward into the neural scene representations of Chapter 19. The chapter walks through the three architecture families with the benchmarks they win and the regimes where they lose, then turns to 3D object detection, instance segmentation, and the pre-training strategies that matter when labels are scarce.

Challenges of point cloud data for neural networks

Standard CNNs assume that input data lies on a regular grid: pixels in rows and columns, voxels in a 3D lattice. Point clouds violate this assumption in several ways:

1. Permutation invariance: the output must be identical regardless of point ordering. For $n$ points there are $n!$ possible orderings, and a classification network must produce the same result for all of them.
2. Irregular structure: points are scattered in continuous 3D space with varying local density and no grid topology.
3. Variable size: different point clouds have different numbers of points.
4. Transformation invariance: the representation should be invariant (or equivariant) to rigid transformations (a rotated chair is still a chair).

Three families of approaches

Three families of architectures address the irregular structure of point clouds. Point-based methods use per-point MLPs with symmetric aggregation, convolution-based methods define continuous convolution kernels in 3D space, and graph or attention-based methods build neighbourhood graphs and propagate features along their edges. The figure below maps the landscape that the rest of the chapter explores.