|PIAZZA ENRICO||Cycle: XXXIII |
Section: Computer Science and Engineering
Tutor: AMIGONI FRANCESCO
Advisor: MATTEUCCI MATTEO Major Research topic
:Probabilistic Benchmarking of Robot Tasks and SystemsAbstract:
Benchmarking can be defined as a standardised procedure to measure the performance of a software/hardware solution against a reference performance. Several areas of research require benchmarking to assess the value of new results against some reference performance. An example of wide-spread use of benchmarking in robotics research are publicly available datasets against which algorithms are compared and ranked. Nonetheless, benchmarking and experimental methodology in robotics research is not considered by the scientific community to be as proficient as in other areas, such as physics and medicine.
While important developments have happened in the last decade concerning the availability of development frameworks that enable prototyping and provide access to a database of algorithms to implement several robot subsystems (e.g., navigation, perception, mapping), none of those developments provides information about the quality of the provided algorithms, which can only be inferred from their popularity in the robotics community.
The performance of a robotic system is a non-linear and complex function of the performance of its components. This complexity is due to the interaction between components. Benchmarking of single components can provide a general measure of the performance, but in order to apply benchmarking to a complex system, it is necessary to consider the structure of the system.
Classical complex system design requires to break down the system into subsystems, and these subsystems into individual components that can be developed and tested individually. Each subsystem implements part of the functionalities required by the robotic system to accomplish its task. The final product is the result of the integration of these subsystems, that often is itself an iterative process where the performance of the system is verified to assess the quality of the integration and reduce performance bottlenecks. Additionally, different functionalities often need to be developed and tested in different environments, that may also be different from the one in which the final product is deployed.
Because of this, assessing the performance of the components during their development requires to have an understanding of their contribution to the performance of the system.
In the early phases of development, the ability of composing the components performance - obtaining an estimation of the performance of the system - allows to identify the performance requirement of the components leading to a more efficient development process. In the integration phases, understanding of the contribution of the components performance to the system performance provides the information necessary to identify how the system can be improved.
A more thorough understanding of the dependency of the performance between the functionalities of a robotic system also allows to better understand which properties of the performance are important, allowing to develop more significant benchmarks.
The objective of this thesis is to introduce a framework for benchmarking of autonomous robot systems which enables characterising the impact of the performance of the functionalities on the performance of the system, the performance dependency between functionalities, and supports good experimental methodology.