小柯机器人

美国加州大学Takaki Komiyama和佛罗里达大学Ryoma Hattori团队合作取得一项新突破。他们的研究表明可通过眶额皮层（OFC）进行元强化学习（meta-RL）。这一研究成果于2023年11月13日发表在国际学术期刊《自然-神经科学》上。

研究人员证明了眶额皮层介导元RL。研究人员对小鼠和深度RL模型进行了跨程序的概率反转学习任务训练，在此期间，它们通过元强化学习改进了逐次试验的RL策略。OFC中依赖于Ca²⁺/钙调蛋白依赖性蛋白激酶II的突触可塑性对该元强化学习是必要的，但对特定任务中逐次试验RL非必须。

在元强化学习后，OFC的活动会加强编码价值信号，而OFC失活则会损害RL行为。对OFC活动的纵向追踪显示，元强化学习逐渐塑造了群体的价值编码，从而调控持续的行为策略。该研究结果表明，两种不同的RL算法具有不同的神经机制和时间尺度，它们在OFC中共存以产生适应性决策。

据悉，包涵多个时间尺度强化学习的元强化学习框架在训练由新环境产生的深度强化学习模型中取得了成功。有人猜想前额叶皮层可能介导大脑中的元强化学习，但相关证据并不多。

附：英文原文

Title: Meta-reinforcement learning via orbitofrontal cortex

Author: Hattori, Ryoma, Hedrick, Nathan G., Jain, Anant, Chen, Shuqi, You, Hanjia, Hattori, Mariko, Choi, Jun-Hyeok, Lim, Byung Kook, Yasuda, Ryohei, Komiyama, Takaki

Issue&Volume: 2023-11-13

Abstract: The meta-reinforcement learning (meta-RL) framework, which involves RL over multiple timescales, has been successful in training deep RL models that generalize to new environments. It has been hypothesized that the prefrontal cortex may mediate meta-RL in the brain, but the evidence is scarce. Here we show that the orbitofrontal cortex (OFC) mediates meta-RL. We trained mice and deep RL models on a probabilistic reversal learning task across sessions during which they improved their trial-by-trial RL policy through meta-learning. Ca2+/calmodulin-dependent protein kinase II-dependent synaptic plasticity in OFC was necessary for this meta-learning but not for the within-session trial-by-trial RL in experts. After meta-learning, OFC activity robustly encoded value signals, and OFC inactivation impaired the RL behaviors. Longitudinal tracking of OFC activity revealed that meta-learning gradually shapes population value coding to guide the ongoing behavioral policy. Our results indicate that two distinct RL algorithms with distinct neural mechanisms and timescales coexist in OFC to support adaptive decision-making.

DOI: 10.1038/s41593-023-01485-3

Source: https://www.nature.com/articles/s41593-023-01485-3

Nature Neuroscience：《自然—神经科学》，创刊于1998年。隶属于施普林格·自然出版集团，最新IF：28.771
官方网址：https://www.nature.com/neuro/
投稿链接：https://mts-nn.nature.com/cgi-bin/main.plex