Prioritization Hindsight Experience Based on Spatial Position Attention for Robots

Sparse rewards pose significant challenges in deep reinforcement learning as agents struggle to learn from experiences with limited reward signals. Hindsight experience replay (HER) addresses this problem by creating “small goals” within a hierarchical decision model. However, HER does not consider the value of different episodes for agent learning. In this paper, we propose SPAHER, a framework for prioritizing hindsight experiences based on spatial position attention. SPAHER allows the agent to prioritize more valuable experiences in a manipulation task. It achieves this by calculating transition and trajectory spatial position functions to determine the value of each episode for experience replays. We evaluate SPAHER on eight robot manipulation tasks in the Fetch and Hand environments provided by OpenAI Gym. Simulation results show that our method improves the final mean success rate by an average of 3.63% compared to HER, especially in challenging Hand environments. Notably, these improvements are achieved without any increase in computation time.