Latent Landmark Graph for Efficient Exploration-exploitation Balance in Hierarchical Reinforcement Learning

Goal-conditioned hierarchical reinforcement learning (GCHRL) decomposes the desired goal into subgoals and conducts exploration and exploitation in the subgoal space. Its effectiveness heavily relies on subgoal representation and selection. However, existing works do not consider distinct information across hierarchical time scales when learning subgoal representations and lack a subgoal selection strategy that balances exploration and exploitation. In this paper, we propose a novel method for efficient exploration-exploitation balance in HIerarchical reinforcement learning by dynamically constructing Latent Landmark graphs (HILL). HILL transforms the reward maximization problem of GCHRL into the shortest path planning on graphs. To effectively consider the hierarchical time-scale information, HILL adopts a contrastive representation learning objective to learn informative latent representations. Based on these representations, HILL dynamically constructs latent landmark graphs and selects subgoals using two measures to balance exploration and exploitation. We implement two variants: HILL-hf generates graphs periodically, while HILL-lf generates graphs adaptively. Empirical results on continuous control tasks with sparse rewards demonstrate that both variants outperform state-of-the-art baselines in sample efficiency and asymptotic performance, with HILL-lf further reducing training time by 40% compared to HILL-hf.