Actor-critic algorithms
Vijay R. Konda, John N. Tsitsiklis · 2002
Abstract. In this article, we propose and analyze a class of actor-critic algorithms. These are two-time-scale algorithms in which the critic uses temporal difference learning with a linearly parameterized approximation architecture, and the actor is updated in an approximate gradient direction, based on information provided by the critic. We show that the features for the critic should ideally span a subspace prescribed by the choice of parameterization of the actor. We study actor-critic algorithms for Markov decision processes with Polish state and action spaces. We state and prove two results regarding their convergence.
Cited by 1,666 works.
Authors
References
- Reinforcement Learning: An Introduction
- Dynamic Programming and Optimal Control
- Simple statistical gradient-following algorithms for connectionist reinforcement learning
- Neuro-dynamic Programming
- Adaptive Algorithms and Stochastic Approximations
- Markov Chains and Stochastic Stability
- Policy Gradient Methods for Reinforcement Learning with Function Approximation
- An analysis of temporal-difference learning with function approximation
- Stochastic approximation for Monte Carlo optimization
- A New Approach to the Limit Theory of Recurrent Markov Chains