Reward Shaping

Introduction

Often, a very simple pattern of extra rewards suffices to render straightforward an otherwise completely intractable problem.

Current Problems

Consider the following example of bugs that can arise:

In a system that learns to ride a simulated bicycle to a particular location. To speed up learning, they provided positive rewards whenever the agent made progress towards the goal. Because no penalty was incurred for riding away from the goal.



Hence it’s now better for the bicycle to try to go in a cycle than to go to the goal.

Preliminaries

A finite-state Markov decision process(MDP) is a tuple $M=(S,A,T,\gamma ,R)$, where:

• $S$ is a set of states;
• $$\begin{gathered} A= \\ {a}_1,\dots ,a_k \end{gathered}$$ is a set of actions;
• $$\begin{gathered} T= \\ {P}_{sa}(\cdot |s\in S,a\in A) \end{gathered}$$ is the next state transition probability;
• $\gamma$ is the discount factor;
• $R:S\times A\times S\to R$ is a bounded real function called the reward function.

A policy over a set of states S is a function $\pi :S\to A$.

Thus,

• Value function $V_M^{\pi }(s)=\mathbb{E}[R_1+\gamma R_2+\cdots ]$
• Action function $Q_M^{\pi }(s,a)={\mathbb{E}}_{s^{\prime \sim P_{sa}}}[R(s,a,s^{\prime })+\gamma V_M^{\pi }(s^{\prime })]$

Hence, the optimal value function is

$$V_M^{\text{*}}(s)=\underset{\pi }{\sup }{V}_M^{\pi }(s)$$

The optimal Q-function is

$$Q_M^{\text{*}}(s,a)=\underset{\pi }{\sup }{Q}_M^{\pi }(s,a)$$

The optimal policy is

$${\pi }_M^{\text{*}}(s)=\arg \underset{a\in A}{\max }{Q}_M^{\text{*}}(s,a)$$

Views

We will run on a transformed MDP $M^{\prime }=(S,A,T,\gamma ,R^{\prime })$ where $R^{\prime }=R+F$, and $F:S\times A\times S\to \mathbb{R}$ is also a bounded real-value function called the shaping reward function.

We are trying to learn a policy for some MDP M, and we wish to help our learning algorithm by giving it additional shaping rewards which will hopefully guide it towards learning a good policy faster.

But for what forms of shaping-rewards $F$ can we guarantee that ${\pi }_{M^{\prime }}^{\text{*}}$, the optimal policy in $M^{\prime }$ will also be optimal in $M$?

Definition A shaping reward function $F$ is potential-based if there exists $\Phi :S\to \mathbb{R}$ s.t.

$$F(s,a,s^{\prime })=\gamma \Phi (s^{\prime })-\Phi (s)$$

If $F$ is a potential-based shaping function, then every optimal policy in $M^{\prime }$ will also be an optimal policy in $M$.

Proof omission.

Conclusion

This suggests that a way to define a good potential function might be to try to approximate $V_M^{\text{*}}(s)$.