Policy Search for Path Integral ControlVicenç Gómez; Hilbert J. Kappen; Jan Peters; Gerhard Neumann
In: Toon Calders; Floriana Esposito; Eyke Hüllermeier; Rosa Meo (Hrsg.). Machine Learning and Knowledge Discovery in Databases - European Conference, ECML PKDD 2014, Proceedings, Part I. European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD-2014), September 15-19, Nancy, France, Pages 482-497, Lecture Notes in Artificial Intelligence (LNAI), Vol. 8724, Springer, 2014.
Path integral (PI) control defines a general class of control problems for which the optimal control computation is equivalent to an inference problem that can be solved by evaluation of a path integral over state trajectories. However, this potential is mostly unused in real-world problems because of two main limitations: first, current approaches can typically only be applied to learn open-loop controllers and second, current sampling procedures are inefficient and not scalable to high dimensional systems. We introduce the efficient Path Integral Relative-Entropy Policy Search (PI-REPS) algorithm for learning feedback policies with PI control. Our algorithm is inspired by information theoretic policy updates that are often used in policy search. We use these updates to approximate the state trajectory distribution that is known to be optimal from the PI control theory. Our approach allows for a principled treatment of different sampling distributions and can be used to estimate many types of parametric or non-parametric feedback controllers. We show that PI-REPS significantly outperforms current methods and is able to solve tasks that are out of reach for current methods.