Mathematical Sciences: Faculty Presentations (1991-2022)

Exploration Using Without-Replacement Sampling of Actions is Sometimes Inferior

Document Type

Presentation

Presentation Date

3-2021

Copyright

This work is archived and distributed under the repository's Standard Copyright and Reuse License (opens in new tab). End users may copy, store, and distribute this work without restriction. For all other uses, permission must be obtained from the copyright owners or their authorized agents.

Abstract or Description

Presentation given at the 100th Meeting of the Southeastern Section of the Mathematical Association of America.

Abstract

In many statistical and machine learning applications, without-replacement sampling is considered superior to with-replacement sampling. In some cases, this has been proven, and in others the heuristic is so intuitively attractive that it is taken for granted. In reinforcement learning, many count-based exploration strategies are justified by reliance on the aforementioned heuristic. This paper will detail the non-intuitive discovery that when measuring the goodness of an exploration strategy by the stochastic shortest path to a goal state, there is a class of processes for which an action selection strategy based on without-replacement sampling of actions can be worse than with replacement sampling. Specifically, the expected time until a specified goal state is first reached can be provably larger under without-replacement sampling. Numerical experiments describe the frequency and severity of this inferiority

Sponsorship/Conference/Institution

100th Meeting of the Southeastern Section of the Mathematical Association of America

Location

Virtual

Source

https://maasoutheastern.org/2021-conference/

This document is currently not available here.

Share

COinS