The process by which robots use vision, laser and/or sonar sensors to map an environment and, at the same time, determine their location in it is known as Simultaneous Localization and Mapping (SLAM), a field of robotics that has been the focus of intense research for three decades.

Over the years different research teams have taken different approaches to the problem. They use different types of sensors, different makes and models of sensing devices, conduct tests in different environments and employ different algorithms to perform mapping and localization. This has led to a multitude of SLAM algorithms, some of which are undoubtedly more suited to certain applications and environments than others, but with no effective way to compare them.

“SLAM is an essential building block of autonomous robots because robots, such as planetary rovers and undersea research craft, cannot be provided with an accurate map beforehand. In such situations, the only solution is for them to create a representation of the environment as they go and determine their location in it by themselves,” says Matteo Matteucci, a roboticist at the Politecnico di Milano University in Italy.

Benchmarking different SLAMs: Until now researchers developing a new robot have had to develop SLAM algorithms from scratch or at best make informed guesses as to which existing algorithms would be best suited to the tasks the robot is to perform. That is now set to change thanks to the work of a group of researchers led by Matteucci in the EU-funded Rawseeds project.

The team, from four universities in Italy, Spain, and Germany, has created a unique set of free benchmarking tools, enabling fellow roboticists to effectively compare SLAM approaches and algorithms against each other for the first time.

To kick off the benchmarking process they developed their own robotic test platform, incorporating six different vision, laser and sonar sensor types, which they used to gather synchronized sensor data for SLAM. They then operated the platform in different indoor and outdoor environments, varying factors such as the lighting or the presence of people or moving objects.

“Our goal was to establish a common, predefined way of measuring the performance of SLAM algorithms that differ by approach and sensors used – benchmarks that other algorithms could then be compared against,” Matteucci explains.

The results from the platform, developed by Politecnico di Milano and University of Milano-Bicocca, were peer reviewed and validated by the other two academic partners, the University of Zaragoza in Spain and University of Freiburg in Germany. They then put the benchmarks to the test by using them to see whether they could effectively develop better SLAM algorithms and hence smarter robots.

“We succeeded in what we set out to do… The team at the University of Zaragoza, for example, actually improved their results with their visual odometry algorithm thanks to having such a big SLAM dataset available to them,” Matteucci says.

Disseminating Rawseeds: And big it certainly is. The Rawseeds benchmarking toolkit, which is available for free from the project website, weighs in at more than half a gigabyte of data. That forced the researchers to turn to an unusual distribution channel to make their work available to the wider robotics research community over the internet.

“Because we couldn’t afford high bandwidth hosting we developed a flexible distribution system based on the Bit Torrent peer-to-peer file-sharing protocol,” the project coordinator says. “It provides a practical way to distribute the toolkit by having users act as peers, or servers, for others who want to download it.”

In Matteucci’s view, getting the toolkit and benchmarking data out to the robotics community is essential to the overall success of the Rawseeds team’s work. Several other research groups have already downloaded it and the partners are actively promoting it. In the future, similar benchmarks could be used to compare algorithms for other key features of autonomous robotic systems, such as navigation, limb movement and gripping, he notes.

“We hope that by providing these benchmarks and an effective way to compare SLAM approaches, sensors and algorithms we will help researchers develop better algorithms, save time in the development of new robotic platforms and ultimately contribute to the development of smarter future generations of robots,” Matteucci says.

Rawseeds received funding from the ICT strand of the EU’s Sixth Framework Program for research.