Several areas of research require benchmarking to assess the value of new results against some reference performance. However, benchmarking is almost not used in current Robotics research, and its relevance is frequently underestimated. While important developments have happened in the last few years concerning the availability of development systems that enable prototyping robot systems supported by software architectures that isolate the developer from the operating system details, simultaneously providing access to a database of algorithms to implement several robot subsystems (e.g., navigation, perception, mapping), none of those developments provides information about the quality of the provided algorithms, which can only be inferred from their “popularity” in the robotics community.
Among other features, benchmarking the primitive actions used to plan robot tasks enables designing better task plans by composing the primitives whose joint performance improves the task performance against some reference performance. Dependable robot systems require their sub-systems to self-monitor continuously their performance, so as to detect and even predict failures or severe performance degradation and act to avoid or at least mitigate the consequences. Thus, to tackle the benchmarking problem for robot systems, one must first find a performance metric that is common to the different sub-systems of very distinctive nature that compose them.
One can define the elements of a task for a robot system as (Amigoni, 2013):
- Functionality: one of the basic abilities that a robot system is required to possess in order to be subjected to a given experiment, e.g., self-localization, grasping, mapping or human-robot speech dialogue.
- Functional Module: (hardware and software) component of a robot system that is involved in providing the robot system with a specific Functionality.
- Task: a detailed description of the operation or set of operations that a robot system is required to perform in order to accomplish its goals. These operations are typically based on functionalities.
In this thesis, the starting plan is to extend past work (Lima and Saridis, 1999) on joint consistent definition of reliability and cost of robot primitive actions (here called functionalities) and tasks, using a probabilistic approach that ensures a common metric for functionalities of different nature. Those performance metrics will be extended to provide and test methods and consistent metrics for task performance prediction from self-monitored functional modules performance. This requires estimating statistically significant scores for the functionalities composing a task plan, based on repeated experiments with them, and studying the impact of their performance in the task performance (also a statistically significant score resulting from repeated experiments with the task execution). Such studies will enable a more consistent propagation of functionality cost + reliability to task cost + reliability (former work implicitly assumed independent functionalities with equal impact on task performance). Due to the nature of the consistent joint definition of cost and reliability, the robot system can, during task execution, self-monitor the performance of the composing functional modules implementing the functionalities and predict its impact in the task performance (because it consists of comparing the current output of a functional module with the desired one), possibly re-planning if a performance failure is anticipated.
(Amigoni et al, 2013) F. Amigoni, A. Bonarini, G. Fontana, M. Matteucci, V. Schiaffonati (2013), “Benchmarking Through Competitions: The RoCKIn Approach”, ERF 2013 Workshop on Competitions (Eds.: A. Winfield, P. U. Lima)
(Lima and Saridis, 1999) Lima, Pedro U., and George N. Saridis (1999). "Intelligent controllers as hierarchical stochastic automata." Systems, Man, and Cybernetics, Part B: Cybernetics, IEEE Transactions on, 29.2: 151-163.