Actor-critic models of the basal ganglia New anatomical and computational perspectives
➤ Gửi thông báo lỗi ⚠️ Báo cáo tài liệu vi phạmNội dung chi tiết: Actor-critic models of the basal ganglia New anatomical and computational perspectives
Actor-critic models of the basal ganglia New anatomical and computational perspectives
Actor-critic models of the basal ganglia: New anatomical and computational perspectivesDaphna Joel*, Yael Niv* and Eytan Ruppint♦Department of Psychol Actor-critic models of the basal ganglia New anatomical and computational perspectiveslogy, Tel-Aviv University, Tel Aviv 69978, IsraeltSchools of Medicine and Mathematical Sciences, Tel-Aviv University, Tel Aviv 69978, IsraelReprint requests are to be sent to:Dr. Daphna JoelDepartment of Psychology, Tel-Aviv University, Tel Aviv 69978, IsraelTel: 972-3-6408996Fax: 972-3-6407391Email Actor-critic models of the basal ganglia New anatomical and computational perspectives: djoel(Ịị)post.tau.ac.ilRunning title: New perspectives on actor-critic models1Actor-critic models of the basal ganglia: New anatomical and computatiActor-critic models of the basal ganglia New anatomical and computational perspectives
onal perspectivesAbstractA large number of computational models of information processing in the basal ganglia have been developed in recent years. PrActor-critic models of the basal ganglia: New anatomical and computational perspectivesDaphna Joel*, Yael Niv* and Eytan Ruppint♦Department of Psychol Actor-critic models of the basal ganglia New anatomical and computational perspectivesemporal difference prediction error signal in the critic, and between dopamine-dependent long-term synaptic plasticity in the striatum and learning guided by a prediction error signal in the actor. We selectively review several actor-critic models of the basal ganglia with an emphasis on two importa Actor-critic models of the basal ganglia New anatomical and computational perspectivesnt aspects: the way in which models of the critic reproduce the temporal dynamics of dopamine firing, and the extent to which models of the actor takeActor-critic models of the basal ganglia New anatomical and computational perspectives
into account known basal ganglia anatomy and physiology. To complement the efforts to relate basal ganglia mechanisms to reinforcement learning, we iActor-critic models of the basal ganglia: New anatomical and computational perspectivesDaphna Joel*, Yael Niv* and Eytan Ruppint♦Department of Psychol Actor-critic models of the basal ganglia New anatomical and computational perspectivesrning mechanism, and relate the evolved mechanism to the basic model of the critic. We conclude our discussion of models of the critic by a critical discussion of the anatomical plausibility of implementations of a critic in basal ganglia circuitry, and conclude that such implementations build on as Actor-critic models of the basal ganglia New anatomical and computational perspectivessumptions that are inconsistent with the known anatomy of the basal ganglia. We return to the actor component of the actorcritic model, which is usualActor-critic models of the basal ganglia New anatomical and computational perspectives
ly modeled at the striatal level with very little detail. We describe an alternative model of the basal ganglia which takes into account several imporActor-critic models of the basal ganglia: New anatomical and computational perspectivesDaphna Joel*, Yael Niv* and Eytan Ruppint♦Department of Psychol Actor-critic models of the basal ganglia New anatomical and computational perspectives ganglia performs reinforcement-biased dimensionality reduction of cortical inputs. We further suggest that since such selective encoding may bias the representation at the level of the frontal cortex towards the selection OÍ rewarded plans and actions, the reinforcement-driven dimensionality reduct Actor-critic models of the basal ganglia New anatomical and computational perspectivesion framework may serve as a basis for basal ganglia actor mcxlels. We conc lude with a short discussion of the dual role of die dopamine signal in reActor-critic models of the basal ganglia New anatomical and computational perspectives
inforcement loaming and in behavioral switching.Key words: Basal ganglia: Dopamine: Reinforcement learning: Actor-Critic: Dimensionality reduction: EvActor-critic models of the basal ganglia: New anatomical and computational perspectivesDaphna Joel*, Yael Niv* and Eytan Ruppint♦Department of Psychol Actor-critic models of the basal ganglia New anatomical and computational perspectivesrmation processing in the basal ganglia have been developed in recent years (Houk et al., 1995; see Figure 1 for a general scheme of basal ganglia connections). A recent review groups these models into three main (not mutually exclusive) categories: models of serial processing, models of action sele Actor-critic models of the basal ganglia New anatomical and computational perspectivesction, and models of reinforcement learning (Gillies & Arbuthnott, 2000). The first category'- includes models that assign a central role to the basalActor-critic models of the basal ganglia New anatomical and computational perspectives
ganglia loop structure in generating sequences of activity patterns (e.g., Berns & Sejnowski, 1998). The second class focuses on the tonic inhibitoryActor-critic models of the basal ganglia: New anatomical and computational perspectivesDaphna Joel*, Yael Niv* and Eytan Ruppint♦Department of Psychol Actor-critic models of the basal ganglia New anatomical and computational perspectivesn (e.g.. Gurney et al., 2001). In this paper, we focus on the third class of models, which assign a major role for the basal ganglia in reinforcement learning (RL).The interest in RL models of the basal ganglia has been initiated by the seminal studies of Wolfram Schultz, which provided experimental Actor-critic models of the basal ganglia New anatomical and computational perspectives evidence suggesting that RL plays an important role in basal ganglia processing (Schultz et al., 2000; Schultz & Dickinson, 2000). Recording the actiActor-critic models of the basal ganglia New anatomical and computational perspectives
vity of dopaminergic (DA) neurons in monkeys during the acquisition and performance of behavioral tasks, Schultz and colleagues found that DA neurons Actor-critic models of the basal ganglia: New anatomical and computational perspectivesDaphna Joel*, Yael Niv* and Eytan Ruppint♦Department of Psychol Actor-critic models of the basal ganglia New anatomical and computational perspectives reward to reward-predicting stimuli. The firing pattern of DA neurons was also found to reflect information regarding the timing of delayed rewards (relative to the reward-predicting stimulus), as could be seen by the precisely timed depression of DA firing when an expected reward was omitted. This Actor-critic models of the basal ganglia New anatomical and computational perspectives pattern4https://khothuvien.cori!of activity is very similar to that generated by computational algorithms of RL, in particular Temporal Difference (TActor-critic models of the basal ganglia New anatomical and computational perspectives
D) models (Sutton, 1988), as described in detail in another paper in this volume (See article by Suri and Schultz).In the context of basal ganglia modActor-critic models of the basal ganglia: New anatomical and computational perspectivesDaphna Joel*, Yael Niv* and Eytan Ruppint♦Department of Psychol Actor-critic models of the basal ganglia New anatomical and computational perspectivess to perform actions so as to maximize the weighted sum of future rewards, which is computed at every timestep by a critic sub-network (Barto, 1995). The critic is adaptive, in that it learns to predict the weighted sum of future rewards based on the current sensory input and the actor's policy, by Actor-critic models of the basal ganglia New anatomical and computational perspectivesmeans of an iterative process in which it compares its own predictions to the actual rewards obtained by the acting agent. The learning rule used by tActor-critic models of the basal ganglia New anatomical and computational perspectives
he adaptive critic is the TD learning rule (Sutton, 1988) in which the error between two adjacent predictions (the TD error) is used to update the criActor-critic models of the basal ganglia: New anatomical and computational perspectivesDaphna Joel*, Yael Niv* and Eytan Ruppint♦Department of Psychol Actor-critic models of the basal ganglia New anatomical and computational perspectivesKaelbling et al., 1996: Tesauro, 1995; Zhang & Dietteric, 1996).The analogy between the basal ganglia and actor-critic models builds on the strong resemblance between DA neuron activity and the TD prediction error signal, and between DA-dependent long-term synaptic plasticity in the striatum (Charpi Actor-critic models of the basal ganglia New anatomical and computational perspectiveser & Deniau, 1997; Wickens et al., 1996) and learning guided by a prediction error signal in the actor. Actorcritic models of basal ganglia functioninActor-critic models of the basal ganglia New anatomical and computational perspectives
g have gained popularity in recent years, and several models have been proposed. A comparison between these models shows that they mainly differ in twActor-critic models of the basal ganglia: New anatomical and computational perspectivesDaphna Joel*, Yael Niv* and Eytan Ruppint♦Department of Psychol Actor-critic models of the basal ganglia New anatomical and computational perspectivesecture responsible for5producing the short phasic response of DA neurons to unpredicted rewards and rewardpredicting stimuli, and the depression induced by reward omission. Models of the actor differ in the extent to which they take into account known basal ganglia anatomy and physiology.In the foll Actor-critic models of the basal ganglia New anatomical and computational perspectivesowing section we briefly review several actor-critic models of the basal ganglia with an emphasis on the mechanism responsible for reproducing the temActor-critic models of the basal ganglia New anatomical and computational perspectives
poral dynamics of DA firing and on the architecture of the actor. Section 3 introduces an alternative approach to modeling a critic network, which useActor-critic models of the basal ganglia: New anatomical and computational perspectivesDaphna Joel*, Yael Niv* and Eytan Ruppint♦Department of Psychol Actor-critic models of the basal ganglia New anatomical and computational perspectivested in Section 2. Section 4 provides a critical discussion of the anatomical plausibility of the implementation of an adaptive critic in basal ganglia circuitry. In section 5 we return to the actor component of the actor-critic model and describe an alternative model of the basal ganglia which takes Actor-critic models of the basal ganglia New anatomical and computational perspectives into account several important, and previously neglected, anatomical and physiological characteristics of basal ganglia-thalamocortical connectivity.Actor-critic models of the basal ganglia New anatomical and computational perspectives
This model sees the main computational role of the basal ganglia as being a key station in a dimension reduction coding-decoding cortico-striato-pallActor-critic models of the basal ganglia: New anatomical and computational perspectivesDaphna Joel*, Yael Niv* and Eytan Ruppint♦Department of Psychol Actor-critic models of the basal ganglia New anatomical and computational perspectives Actor-Critic Models of Reinforcement Learning in the Basal Ganglia2.1 Houk, Adams and Barto (1995)One of the first actor-critic models of the basal ganglia was presented by Houk et al. (1995). This model suggests that striosomal modules fulfill the main functions of the adaptive critic,6 Actor-critic models of the basal ganglia New anatomical and computational perspectivesGọi ngay
Chat zalo
Facebook