Section: Systems and Control
Tutor: BASCETTA LUCA Major Research topic
:Beyond full rationality: inverse reinforcement learning and agent base modelling of tradeoff selection and adaptation in multipurpose water resources systems
Advisor: CASTELLETTI ANDREA FRANCESCOAbstract:
Beyond full rationality: inverse reinforcement learning and agent base modelling of tradeoff selection and adaptation in multipurpose water resources systems
Growing population and changing climate are challenging world water resources by increasing demand of fresh water and modifying water availability in space and time. There is now awareness that capacity expansion alone is not enough to sustainably quench the globe’s thirst, as it should be combined with a more adaptive and flexible management of the available resources. While scientific literature is recognizing the issue and offering theoretical methodologies, system operators are already implementing practical solutions. The study of their system operations reveals that they are constantly adapting the tradeoff among the operational targets in response to unexpected events, such as climatic changes. The additional degree of freedom helps them to mitigate the costs of operating the system under unexpected stress. In this research, we analyze and model the behavior of a water resources system operator in the management of multi-purpose water systems.
We will first frame a single decision-maker, multi-objective problem as a multi-agent cooperative setting where a single control is the outcome of a negotiation among multiple conflicting agents. Each agent represents one of the different targets or objectives that the control aims to satisfy; the agents periodically negotiate the operational policy on the basis of their past perceived performance. Modeling the drivers of this negotiation completes our understanding of the complex behaviors of the system manager as well as their non-linear reactions to changes in the external conditions. The second step will be to perform the analysis of the same multi-objective problem within the framework of Inverse Reinforcement Learning, a machine learning technique that uses a timeserie of human-made control decisions to select a set of plausible goals the decisions were aiming at. A preparatory analysis will aim to identify and categorize the different mathematical representations used in the literature for these goals. A tentative activities plan for this research is the following: during the first year (2014-2015) we will design the multi-agent negotiation framework and we will develop a synthetic case study to test framework on, while deepening the knowledge of relevant literature; the second year (2015-2016) will be employed to finalize the multi- agent negotiation framework as well as to identify suitable applications to the real-world problems which the research group has already knowledge of, such as lake Como (IT); in the last year (2016-2017) we will apply the Inverse Reinforcement Learning framework on similar case studies, starting with Cancano- San Giacomo system (IT) that is (supposedly) operated only for hydropower production. These applications will help to cross-check weaknesses and strength of the methods in order to devise a set of recommendations to perform studies of tradeoff selection and empower the theoretical formalization of the custom practices the system operators are already applying in their daily activities.
Advisor: Andrea Castelletti
Co-Advisor: Matteo Giuliani