Actor Critic

Algorithm

In REINFORCE, we know that

The variance of the gradient estimator scales unfavorably with the time horizon, since the effect of an action is confounded with the effects of past and future actions.

It make a lot of sense to learn the value function in addition to the policy, since knowing the value function can assist the policy update, such as by reducing gradient variance in REINFORCE, and that is what the Actor-Critic dose.

Actor Critic methods consists of two components, which may optionally share parameters:

1. Critic update the value function parameters and depending on the algorithm it could be action-value $Q(s,a)$ or state-value $V(s)$.

2. Actor update the policy function parameters, in the direction suggested by the critic.

Implements

1. Initialize initial state $s$, actor parameters $\theta$ and critic parameters $\mu$;

2. For $t=1\dots T$:

   1. Sample reward $r_t\sim R(s,a)$ and next state $s^{\prime }\sim P(s^{\prime }|s,a)$;

   2. Sample next action $a^{\prime }\sim {\pi }_{\theta }(a^{\prime }|s^{\prime })$;

   3. Update the policy parameters by $\theta \leftarrow \theta +{\alpha }_{\theta }{Q}_{\mu }(s,a){\nabla }_{\theta }\log {\pi }_{\pi }(a|s)$;

   4. Compute the TD residual ${\sigma }_t=r_t+\gamma Q_{\mu }(s^{\prime },a^{\prime })-Q_{\mu }(s,a)$

   5. Update the critic parameters by $\mu \leftarrow \mu +{\alpha }_{\mu }{\sigma }_t{\nabla }_{\mu }{Q}_{\mu }(s,a)$

   6. Update $a\leftarrow a^{\prime }$ and $s\leftarrow s^{\prime }$

Reference

• https://lilianweng.github.io/lil-log/2018/04/08/policy-gradient-algorithms.html#actor-critic