Convergence of a Q-Learning Variant for Continuous States and Actions

Document Type

Article

Publication Date

2014

Publication Title

Journal of Artificial Intelligence Research

DOI

10.1613/jair.4271

ISSN

1076-9757

Abstract

This paper presents a reinforcement learning algorithm for solving infinite horizon Markov Decision Processes under the expected total discounted reward criterion when both the state and action spaces are continuous. This algorithm is based on Watkins' Q-learning, but uses Nadaraya-Watson kernel smoothing to generalize knowledge to unvisited states. As expected, continuity conditions must be imposed on the mean rewards and transition probabilities. Using results from kernel regression theory, this algorithm is proven capable of producing a Q-value function estimate that is uniformly within an arbitrary tolerance of the true Q-value function with probability one. The algorithm is then applied to an example problem to empirically show convergence as well.

Comments

JAIR is an open access journal and articles are published for free distribution on the internet by the AI Access Foundation.

This document is currently not available here.

Share

COinS