Characterizing Visual Localization and Mapping Datasets

Sajad Saeedi   Eduardo D C Carvalho   Wenbin Li   Dimos Tzoumanikas
Stefan Leutenegger   Paul H. J. Kelly   Andrew J. Davison

Imperial College London

ICRA 2019

Overview

Benchmarking mapping and motion estimation algorithms is established practice in robotics and computer vision. As the diversity of datasets increases, in terms of the trajectories, models and scenes, it becomes a challenge to select datasets for a given benchmarking purpose. This paper addresses this concern by developing novel metrics to evaluate trajectories and the environments without relying on any SLAM or motion estimation algorithm. The metrics, which so far have been missing in the research community, can be applied to the plethora of datasets that exist. The proposed metric is a general metric and can be used for other purposes. To demonstrate the effectiveness of the metric in robotic applications, the metric has also been applied to run-time SLAM adaptation. Additionally, to improve the robotics SLAM benchmarking, the paper presents a new dataset for visual localization and mapping algorithms. A broad range of real-world trajectories is used in very high-quality scenes and a rendering framework to create a set of synthetic datasets with ground-truth trajectory and dense map which are representative of key SLAM applications such as virtual reality (VR), micro aerial vehicle (MAV) flight, and ground robotics.

Interior Scenes

Diamond and Deer indoor scenes

In this work, we render ground-truth data on two high resolution scenes provided by kujiale.com. Scene Diamond (left, 3,539,795 vertices and 6,951,971 faces) particularly represents a wide interior space with low light condition whilst Deer (right, 6,863,205 vertices and 13,617,380 faces) shows a crowded living room with multiple small objects occluded from each other.

Real-world Trajectories

Camera trajectories used in LMData

Downloads

Counts, frame rates and durations below reproduce the original release description. Timestamp units and pose conventions have not been independently verified. Do not infer an image-to-pose alignment from these descriptions.

Rendered RGB-D GT: Deer Scene

Rendered RGB-D GT: Diamond Scene

Evaluation

The following results are reproduced from the original release, not a new experiment.

No. Model Trajectory ATE [m] RPE [m] Tracked [%]
1 Deer Ground Robot 0.01452 0.01104 59
2 Deer Walk-Head 0.2074 0.05496 99
3 Deer Walk 0.08640 0.1454 100
4 Deer Run 0.04650 0.0672 64
5 Deer VR slow 0.6152 0.06583 98
6 Deer VR fast 0.9051 0.3117 40
7 Deer MAV slow 0.0102 0.0929 100
8 Deer MAV fast 0.9525 0.0681 96
9 Diamond Ground Robot 0.2112 0.0181 100
10 Diamond Walk-Head 0.0187 0.0482 100
11 Diamond Walk 0.0479 0.0530 100
12 Diamond Run 0.1408 0.0750 100
13 Diamond VR slow 0.0192 0.0667 100
14 Diamond VR fast 1.354 0.3113 57
15 Diamond MAV slow 0.0086 0.0758 100
16 Diamond MAV fast 0.0090 0.1162 100

Absolute trajectory error (ATE), relative pose error (RPE), and tracked frames for ORBSLAM2.0 in RGB-D mode.

Sample MAV Slow and MAV Fast trajectories

License

The scenes, images, video and related ground truth data presented within this page is intended for research use ONLY.

Acknowledgements

This research is supported by Engineering and Physical Sciences Research Council (EPSRC), grant references EP/K008730/1 and EP/N018494/1. We also thank Dr. Rui Tang from kujiale.com for providing us the scenes Diamond and Deer.

Citation

Download BibTeX

@inproceedings { Characterizing19,
      author = {Saeedi, Sajad and Carvalho, Eduardo D C and Li, Wenbin and Tzoumanikas, Dimos and Leutenegger, Stefan and Kelly, Paul H J and Davison, Andrew J},
   booktitle = { International Conference on Robotics and Automation (ICRA) },
       title = { Characterizing Visual Localization and Mapping Datasets },
       pages = { 6699--6705 },
        year = { 2019 },
organization = { IEEE }
}