3DRef: 3D Dataset and Benchmark for Reflection Detection

Abstract

We present the 3DRef dataset, the first large-scale 3D reflection detection dataset containing over 50,000 aligned samples of multi-return Lidar, RGB images, and semantic labels across diverse indoor environments. Textured 3D ground truth meshes enable automatic point cloud labeling for precise annotations. We benchmark Lidar point cloud segmentation and image segmentation methods on glass, mirror, and other reflective object detection. 3DRef provides a comprehensive multi-modal testbed to drive future research towards reliable reflection detection for autonomous systems.

Dataset Overview

The 3DRef dataset contains three sequences captured in diverse indoor environments with various reflective surfaces like glass, mirrors, whiteboards, and monitors. It provides:

48,024 labeled Lidar point clouds from 3 sensors (Ouster, Livox, Hesai)
3,799 labeled RGB images
Precise global alignment of all data
Labeled ground truth 3D meshes enabling automatic point cloud annotation

Benchmark Results

We benchmark Lidar-based and RGB-based reflection detection methods on 3DRef, evaluating factors like multi-return analysis and retraining on the new data.

Key results show:

Explicit multi-return information improves Lidar reflection detection
Retraining image segmentation networks on 3DRef boosts performance
3DRef's diversity challenges current methods but enables developing robust models

Lidar benchmark results — Lidar-based reflection detection results

RGB benchmark results — RGB-based reflection detection results

Dataset Format

In the dataset link, we provide the dataset in some folders. We describe the dataset format in the following. For the convinence of download, We separate the folder into 4 zip files.

raw: Contains the raw sensor data for each sequence, including pose files, images, meshes, raycast point clouds, and more. Each sequence is in a separate subfolder.
- seq1
  - hesai_pose.txt
  - images
  - livox_pose.txt
  - mesh
  - ouster_pose.txt
  - raycast
    - hesai
    - livox
    - ouster
  - vo_kf.txt
- seq2, seq3, ...
rgb: Folder for RGB images and masks split into train/test folders for each label type (glass, mirror, other reflective, all reflective). Images and masks are paired in separate subfolders.
- alllabel
  - test
    - image
    - mask
  - train
    - image
    - mask
- glass, mirror, otherref, ...
script: Helper scripts for dataset processing.
semantickitti: Labeled Lidar point clouds in SemanticKitti format, with separate folders for XYZI and XYZIR channels. Point clouds for each sequence are under sequences/00/velodyne. The data can be generate using the provided script pcd2kitti.py
network: Pretrained weights for reflection detection networks like EBLNet, PCSeg, and SATNet.

The raw folder contains the core data needed to recreate the annotations and formatted dataset. The rgb and semantickitti folders provide the formatted data split into train/test sets ready for benchmarking. The network folder enables out-of-the-box evaluation using provided models. Refer to the readme for additional details.

BibTeX

@article{zhao20233dref,
  title={3DRef: 3D Dataset and Benchmark for Reflection Detection in RGB and Lidar Data},
  author={Zhao, Xiting and Schwertfeger, S{\"o}ren},
  journal   = {3DV},
  year      = {2024},
 }