SHARP: Generating Synthesizable Molecules via Fragment-Based Hierarchical Action-Space Reinforcement Learning for Pareto Optimization

Designing drug-like molecules that satisfy multiple objectives–such
as high binding affinity, synthesizability, and drug-likeness–poses
a complex global optimization problem over an astronomically large
chemical space. Existing deep learning-based molecular generative
models often treat this task as distribution modeling, relying on
atom-level autoregressive actions with less consideration of explicit
optimization feedback. Consequently, they frequently generate invalid
structures, converge to local optima, or produce synthetically infeasible
candidates. Here, we introduce Synthesizable Hierarchical Action-space
Reinforcement learning for Pareto optimization (SHARP), a molecular
generator that addresses these limitations via a fragment-based hierarchical
action space and reinforcement learning. SHARP ensures synthetic accessibility
by applying action masks guided by a pretrained Synthesizability Estimation
Model (SEM). The reinforcement learning (RL) policy is trained using
a composite reward function integrating docking scores, pharmacophore
matching, and solvent accessibility to generate functionally relevant
and experimentally tractable molecules. Furthermore, across four lead
optimization tasks–fragment growing, linker design, scaffold
hopping, and side chain decoration–on a diverse receptor set,
SHARP consistently outperforms prior methods in producing molecules
at high affinity with reasonable synthesizability. These results demonstrate
that reinforcement learning with a chemically intuitive action space
design can be an effective solution to the optimization challenges
in AI-driven drug discovery, offering a robust framework for rational
molecular design in structure-based applications.

开发同时满足高结合亲和力、可合成性与成药性等多重目标的类药分子，是在天文数字级规模的化学空间中求解复杂全局优化问题的核心任务。现有的基于深度学习的分子生成模型常将该任务视作分布建模，依赖原子级自回归生成操作，却较少显式考虑优化反馈。因此，这类模型常生成无效分子结构、收敛至局部最优解，或产出合成路径不可行的候选分子。
在此研究中，我们提出面向帕累托优化的可合成分层动作空间强化学习模型（Synthesizable Hierarchical Action-space Reinforcement learning for Pareto optimization，SHARP）——一款基于片段式分层动作空间与强化学习框架构建的分子生成器，可有效克服上述局限。SHARP通过由预训练可合成性评估模型（Synthesizability Estimation Model，SEM）指导的动作掩码机制，保障生成分子的可合成性。该强化学习（RL）策略采用融合对接评分、药效团匹配与溶剂可及性的复合奖励函数进行训练，以生成具备功能相关性且实验可行的分子。
此外，在多样化的受体数据集上，SHARP在四项先导化合物优化任务（片段生长、连接子设计、骨架跃迁及侧链修饰）中均持续优于现有方法，所生成分子兼具高结合亲和力与合理的可合成性。上述实验结果表明，采用符合化学直觉的动作空间设计的强化学习方法，可有效解决人工智能驱动药物研发中的优化难题，为基于结构的理性分子设计提供了一套鲁棒性极强的研究框架。