In this paper, we propose a policy gradient method for confounded partially observable Markov decision processes (POMDPs) with continuous state and observation spaces in the offline setting. We first establish a novel identification result to non-parametrically estimate any parameterized history-dependent policy gradient under POMDPs using the offline data. The identification boils down to solving a sequence of conditional moment restrictions and we adopt the min-max learning procedure with general function approximation for estimating the policy gradient.
We then provide a finite-sample non-asymptotic bound for estimating the gradient uniformly over a pre-specified policy class in terms of the sample size, length of horizon and concentratability coefficient in solving the conditional moment restrictions. Lastly, by deploying the proposed gradient estimation in the gradient ascent algorithm, we show the last-iterate global convergence of the proposed algorithm in finding the history-dependent optimal policy under some technical conditions. To the best of our knowledge, this is the first work studying the policy gradient method for POMDPs under the offline setting.
嘉宾介绍
Zhengling Qi is an assistant professor at School of Business, the George Washington University. He got his PhD degree from Department of Statistics and Operations Research at the University of North Carolina, Chapel Hill. His research has been focused on statistical machine Learning and related non-convex optimization. He is now mainly working on reinforcement learning and causal inference problems.
