Feudal Latent Space Exploration for Coordinated Multi-agent Reinforcement Learning

Abstract: We investigate how multiple agents learn to coordinate to form efficient exploration in reinforcement learning. Though straightforward, independent exploration of the joint action space of multiple agents will become exponentially more difficult as the number of agents increases. To tackle this problem, we propose Feudal Latentspace Exploration (FLE) for Multi-agent Reinforcement Learning. FLE introduces a feudal commander to learn a low-dimensional global latent structure that instructs multiple agents to explore coordinately. Under this framework, the multi-agent policy gradient is adopted to optimize both the agent policy and latent structure end-to-end. We demonstrate the effectiveness of this method in two multi-agent environments which need explicit coordination. Experimental results validate that FLE outperforms baseline MARL approaches which use independent exploration strategy in terms of mean rewards, efficiency, as well as the expressiveness of coordination policies.

Download: [PDF]