Goal-Conditioned Reinforcement Learning with Imagined Subgoals

This paper

1. tries to find subgoal $s_g$ from initial state $s_i$ to final state $s_j$ which have the maximum distance to $s_i$ and $s_j$
2. train policy $\pi(a|s,g)$ by minimize the # KL Divergence between $\pi(a|s,g)$ and $\pi(a|s,s_g)$, and use $\pi(a|s,g)$ in test phase

The second part increase the training time and difficulty cause it need to fit two policy. But it reduce the subgoal inference phase in test cases. This is a compromise, and I don't think it deserves it.

Inferring subgoals and conducting subgoals are unsplitable in a complete task.

Besides, a lot of subgoal searching algorithms are specifying the threshold between two subgoals. Is there some way to automatic do this job?

Update @<2023-08-26 Sat>

1. What is the subgoal here? Middle state between $s$ and $g$.
2. How to generate subgoals? By using $\pi^{H}(s_g|s,g)$.

3. How to make sure subgoal is valid?

   1. Log probability of real subgoal.
   2. Weighted by advantage.
4. How to use subgoal? Minimize the $D_{KL}(\pi(s,g) || \pi(s,s_g))$. Transfer the knowledge in subgoal to final goal.