A multi-region neural model of learning in large action spaces

Motor skill learning involves navigating vast action spaces, making it impractical to learn the value of every possible movement. Computational frameworks have proposed reinforcement learning (RL) models that reduce action space complexity by learning policies over low-dimensional action embeddings. These embeddings, learned in a supervised manner, capture the structure of state transitions, enabling value generalization across similar actions. Here, we link this framework to neuroscience, proposing that motor cortex learns the action embedding space while the basal ganglia learn policies over it using RL.This perspective aligns with recent neural data showing that, contrary to traditional models of basal ganglia function, similar reaching targets are similarly encoded by this region. We show this data could be explained by the basal ganglia learning policies over an action embedding space, where similar actions have similar embeddings. Indeed, simulations of reaching tasks with close or distant targets confirmed that agents trained with actor-critic RL replicated the neural overlap observed for similar actions. Through this framework, we offer a computational view of how these two key motor regions of the brain collaborate to efficiently learn a new skill, identifying a potential mechanism for generalization in the high dimensional space of potential actions afforded by the body.