SparseOcc++: Geometry-Aware Sparse Latent Representation for Semantic Occupancy Prediction

Abstract

Vision-based 3D semantic occupancy prediction is crucial for autonomous driving but faces a significant challenge: dense representations are often computationally inefficient due to the sparsity of 3D voxels, while Bird's-Eye View (BEV) and Tri-Perspective View (TPV) projections compromise fine-grained 3D structure. Our preliminary study shows that fully sparse representations can offer a promising balance between efficiency and structural fidelity.

However, existing sparse approaches, including our preliminary method, SparseOcc, rely on an entangled geometric-semantic representation in which scene completion is implicitly achieved by indiscriminately propagating high-dimensional semantic features into surrounding empty areas and performing voxel-wise classification. To ensure structural completeness, this entanglement yields excessive activations in empty regions, leading to a significant computational burden and ambiguity. To address this issue, we introduce SparseOcc++, which features a novel geometry-aware sparse representation that explicitly disentangles scene completion from semantic classification. Instead of indiscriminate propagation and voxel-wise classification, SparseOcc++ reformulates scene completion as a regression task. Specifically, we define a scene completion field (SCF) on sparse anchor voxels to predict signed distances to scene boundaries. To robustly model complex outdoor scenes, we propose an orthogonal decomposition strategy and a discretized learning scheme. Furthermore, we propose a geometry-guided propagation mechanism that converts the learned SCF into a complete volumetric scene, ensuring that subsequent semantic segmentation is confined to geometrically verified regions.

Extensive experiments demonstrate that SparseOcc++ establishes a new state-of-the-art: it improves IoU by 2.3% and runs 3.9 times faster than SparseOcc on nuScenes, and achieves a 5.9 times speedup over the dense counterpart, OccFormer, on SemanticKITTI.

Method

Overview of the proposed SparseOcc++. It first generates a set of sparse anchor voxels, which are employed as queries to gather features from images via deformable cross attention. To achieve compact scene completion, a scene completion field (SCF) is learned on these anchor voxels for efficient geometry-guided propagation. Finally, it employs a lightweight sparse U-Net to predict the final fine-grained semantic occupancy within the completed scene.

Experiments

Semantic occpancy prediction results on nuScenes-Occupancy validation set. For accuracy evaluation, We report the geometric metric IoU, semantic metric mIoU, and the IoU for each semantic class. For efficiency evaluation, we report the FLOPs, training GPU memory, and 3D/overall inference latency. The C denotes camera and the bold numbers indicate the best results.

Semantic scene completion results on SemanticKITTI validation set. For accuracy evaluation, We report the geometric metric IoU, semantic metric mIoU, and the IoU for each semantic class. For efficiency evaluation, we report the FLOPs. The C denotes camera and the bold numbers indicate the best results. The methods with “*” are RGB-input variants reported by for fair comparison.

Qualitative results of 3D semantic occupancy on the nuScenes-Occupancy validation set. The input multi-view images are shown on the leftmost side, and the occupancy predictions of the dense baseline C-CONet, SparseOcc, SparseOcc++, and the ground-truth are then visualized sequentially. Compared to the 3D dense representation based C-CONet, our sparse representation achieves better completion and segmentation. The regions highlighted by rectangles show areas with noticeable differences. Best viewed in color and zoomed in.

Qualitative results on the SemanticKITTI validation set. The input monocular image is shown on the leftmost side, and the 3D occupancy predictions of the dense baseline OccFormer, SparseOcc, SparseOcc++, and the ground truth are then visualized sequentially. The regions highlighted by rectangles show areas with noticeable differences. Best viewed in color and zoomed in.

BibTeX


      @inproceedings{tang2024sparseocc,
        title     = {SparseOcc: Rethinking Sparse Latent Representation for Vision-Based Semantic Occupancy Prediction},
        author    = {Tang, Pin and Wang, Zhongdao and Wang, Guoqing and Zheng, Jilai and Ren, Xiangxuan and Feng, Bailan and Ma, Chao},
        booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
        year      = {2024}
      }