BG / OFC / ACC / dlPFC and maintenance / output gating in the Rubicon model (pcore, agate, pbwm)

[rcoreilly]

In developing the new examples/ofcacc model, I am now re-confronting the long history of issues about active maintenance, output gating and relationship between the BG and PFC.

The broadest issues concern the role of BG in driving active maintenance of goals / task / other info in PFC, which was the core idea in the original PBWM model: BG gating toggles active maintenance in PFC.

The BG learns from phasic DA in a way that no other area can, and thus is in a position to provide a learned value-based selection process, via opposing Go vs. NoGo pathways. In the classical BG action selection model, the BG drives a "Go" gating of "good" actions, and a "NoGo" inhibition of "bad" actions (i.e., Thorndike's Law of Effect), and this gating signal functions directly to immediately initiate the action by disinhibiting the corresponding cortical area.

However, this classical model does not capture the well-established early, goal-driven action plan selection dynamic that is actually evident in most BG / PFC recordings in rodents to primates (in well-learned tasks): The action selection takes place at the very start of a trial (or as soon as relevant information is available), and it is somehow then maintained over the subsequent discrete motor actions (e.g., running down a Y maze), until a desired outcome is achieved, or the goal is abandoned.

This dynamic is consistent with the general goal-selection / goal-engaged framework O'Reilly, 2020 (aka the Rubicon model of Heckhausen & Gollwitzer, 1987 -- might as well use that name going forward -- it is cool), where behavior is organized into alternating periods of goal selection, followed by execution of the goal to the point of either success or failure / goal abandonment. This is the guiding framework for the ofcacc model. Its advantages are:

• Goal selection involves a coordinated evaluation of outcome (US, in OFC), action plan (dlPFC / SMA), and net utility (gains vs. effort, time, uncertainty and other cost factors, in ACC), all of which are evaluated by the BG. By re-using common OFC and ACC representations, the BG's job is made simpler and more plausible: it can learn over time about the relative values and costs of these codes, and make consistent decisions based on those evaluations. It can perhaps integrate various contextual and circumstantial factors, but largely it can rely on the "auditor" evaluations provided by OFC, ACC.

• During goal-engaged execution, the system isn't constantly second-guessing and micro-managing everything: a "best effort" is made to achieve the goal. This leads to the strong asymmetries in value functions highlighted in the above article. Nevertheless, the system is continuously tracking progress toward the outcome, and insufficient progress can result in goal abandonment, leading to negative dopamine dips and associated "disappointment".

• In either success or failure / goal abandonment, the "final reckoning" occurs, where the accumulated time and effort is fully registered by the ACC, and the exact nature of the OFC outcome is also learned, along with the associated phasic DA as a function of prior expectations (RPE). A key, motivating property of this goal-driven framework is that US outcomes are relatively rare, and thus represent special moments where learning should be concentrated, to update the estimates for subsequent goal-selection events. This reflects a basic assumption that, generally speaking, a temporally-extended sequence of actions is required to achieve most desired outcomes (often with little immediate prospect of reward along the way), and that the best point at which to evaluate these action sequences is at the point when an outcome has been achieved (or foresworn).

• This Rubicon model contrasts with a more uniform, continuous mode of action selection in standard RL models, which just involve a continuous sequence of state / action decisions and evaluations. The classical BG model is more consistent with this model, as the "actor" system. A high-level claim and test of this goal-driven framework is that, in real-world naturalistic tasks (not necc Atari games!), the goal-oriented parsing of time and effort in the Rubicon framework results in better outcomes, more reliably and quickly, compared to the continuous mode RL models.

Thus, the primary challenge here is to get the various brain parts to do their jobs properly in the context of the overall Rubicon model.

The ofcacc model uses the approach environment to operationalize the most essential aspects of this process:

• At the start of a behavioral trial, the model sees a CS stimulus by virtue of looking at a given location, at some initial distance, and it has an internal "drive" body state that requires a specific US to satisfy it.

• If the CS is known to be associated with the desired US, then the model should gate in a plan to approach and consume the US. This should be driven by BG gating evaluating the match of US and drive (strength of relevant OFC rep?), while the ACC should reflect the time-cost associated with the distance to achieve the goal. In more complex versions, it would be good to have different US's at different distances, with a range of internal drive strengths, such that ACC utility signals are really needed to select the best overall choice.

• If the current "view" (CS, distance) does not produce a BG Go signal, then a left or right exploratory looking action should be taken, to find another option. There is some question as to whether this exploration behavior is itself a kind of goal-driven action, but for now, we can treat it as a default-level behavior produced in the absence of an engaged approach goal.

• Once a goal is engaged, and approach & consume behavior is properly executed (which can be initially coached by subcortical instincts, or random exploration of some sort if we really want to go there), then the special outcome-driven learning occurs, based on an accumulated time counter and the DA signal from actually getting the US in the context of the drive, subtracting the time cost.

In this context, the simplistic, original PBWM-style "BG gates in active maintenance of OFC, ACC, and SMA reps" seems like all we need (modulo a few details about what happens at the point of US / reckoning).

The challenge at hand is thus to reconcile this with all the new info about BG / PFC summarized in Hazy Frank O'Reilly 2021 and captured in the nascent agate framework, and figure out how this all might generalize and work in broader contexts, etc.

Key figure for PFC / BG / thalamus spiral loops from O'Reilly, 2020:

[rcoreilly]

Primate PFC and agate model

Here's a quick summary of the new Hazy Frank O'Reilly 2021 primate-oriented synthesis of a range of data on BG / PFC:

• Primate dlPFC layer 3 pyramidal neurons are specialized for NMDA-enhanced recurrent excitatory active maintenance of the "classical" sort -- this is the "Wang / Arnsten / Goldman-Rakic" classical model.

• PFC receives two distinct, overlapping projections from thalamus: MD vs. VA/VL, where VA/VL is subject to BG disinhibitory gating, and MD is largely not.

• MD is similar overall to Pulvinar, and thus seems good for predictive learning according to the deep predictive learning theory. Critically, it projects preferentially to the middle cortical lamina (4, 3 primarily), with more focal, organized connectivity (similar overall to the "patch" type of thalamic pathway, vs. matrix) and thus seems ideally situated for gating this layer 3 active maintenance. This is the origin of the "attentional gating" (agate) theory of PFC working memory: instead of being BG-gated, active maintenance is "attentionally" gated via MD predictive learning. A functional advantage is that maintenance of random bits of sensory information may not have reliable long-term learnable value functions that the BG can recognize, and so depending on the BG for whether to maintain this stuff is probably not a great idea. Prior PBWM models were consistent with this: it was better in general to allow some non-gated "prospective" maintenance, and requiring full BG gating before anything could be maintained just doesn't work..

• VA/VL projects more to deep cortical layer 5 (and layer 1, where the tufts of these neurons go), and is broad and diffuse, consistent with a "matrix" style pathway. Thus, putting this together with the BG gating of this VA/VL thalamus, it seems like BG actually drives output gating of layer 5 output neurons, e.g., to directly initiate actions consistent with the classical model.

This is a nice story with clear divisions of labor, and it makes sense according to the classical BG gating model, but as discussed above, this classical output gating model doesn't really fit with the Rubicon goal-directed model, which requires BG gating of active maintenance of goal / plan states.

Also, the primate-specific nature of the layer 3 active maintenance mechanism calls into question the generality of this framework for the more foundational Rubicon goal-management model, which we think is very much present in rodents and is the functional origin of OFC / ACC / SMA like brain areas, which carries forward into primates too.

Thus, in this context, it may make more sense to think of the layer 3 agate thing as an evolutionarily late adaptation for holding onto lots of bits of information during fluid online information processing of complex internal mental models. This is not needed in the core rodent-level goal-driven dynamics, where the BG plays a central role.

[rcoreilly]

Updated synthesis across rodent, primate, and MD / VThal

Note: Updated based on lit review below: #56 (comment)

• BG gating is always about motivationally-relevant, dopamine-trained decisions -- these are always colored by OFC / ACC inputs, and are tied to the overall goal-driven processing dynamics as envisioned in the Rubicon model. These primarily gate the output layer 5 neurons, who thus reflect the "engaged goal / plan" level signal throughout PFC (OFC gates in the target US, ACC gates in the expected net utility, and dlPFC gates in the overall action plan).

• Independent of any layer 3 active maintenance in primates, this BG disinhibitory gating signal can toggle a subset of layer 5 neurons into active maintenance of engaged goal / plan states, while also providing a transient signal in other layer 5 neurons reflecting the decision action as such. This is consistent with Sommer Wurtz (2000) data showing heterogenous delay (maintenance) and output / response firing in identified layer 5 output pathway neurons.

• Synthesizing the agate and rodent literature from below (e.g., RikhyeGilraHalassa18, SchmittWimmerNakajimaEtAl17 ChernyshevaSychFominsEtAl21 and GuoYamawakiSvobodaEtAl18 et al), it is clear that there are two parallel thalamocortical circuits via these distinct thalamic nuclei, both of which are capable of extended active maintenance out to 1-2 sec at a time in rodent:

  • VM/VA which is BG gated and projects to everything but the layer 6 CT neurons according to Guo et al. This circuit is dominated by PT (5IB pyramidal tract) <-> VM connectivity, with some more minor CT -> VM excitation and VM -> 2/3 -> PT net inhibition. Thus, it is mainly the "output gating" pathway, with some other relatively minor tweaks, but it is also definitely capable of active maintenance within the PT cells, under the influence of this BG gating.
  • MD which is generally not BG gated and likely is of the classic pulvinar form: CT -> MD -> 4 / 2/3 / 6CT (everyone gets the word from MD, CT provides the predictive input). As in the pulvinar case, we can think of the CT as an extended context maintenance, but it clearly has a much longer update cycle, of roughly 1-2+ seconds, compared to the posterior cortical pulvinar theta / alpha cycle. Thus, it represents a long-sought bridge to longer timescale prediction. Somehow we had previously overlooked this simple SRN-style active maintenance mechanism. And like the posterior case, we would predict that it exhibits regular updating (at this longer time scale), which is indeed the case.

• In the rodent, there is good evidence for N2B longer-term NMDA-mediated maintenance dynamics in the deep layer PT and CT neurons, and there is no reason to believe this does not also persist in primate (true?). Thus, the layer 3 NMDA maintenance represents a third additional bonus form of maintenance on top of the other two.

• Consistent with the SRN / CT model, in rodent, there is consistent evidence of sequential bumps of PFC / MD / VM activity across the longer delay periods, with relatively few if any neurons providing a reliable bridge across the full delay. It is likely that this reflects some kind of accommodation dynamic on the part of the PFC (e.g., CT) neurons, which would make sense in terms of keeping the context reps changing and not just getting stuck. Note that the long period of sustained maintenance (2+ sec) is incompatible with a reservoir-style asymmetric connectivity mechanism, which would update at much faster timescales on the order of excitatory synaptic integration.

• TODO: add an accumulating decay with appropriate timescale (in theta cycle increments) to CT, to drive these reps to always keep updating. This is likely to be very important for posterior models too, where CT can just get stuck -- it needs to have both context maintenance and intrinsic updating dynamics, to keep things moving. For standard one-to-one super -> CT this won't work, so we either need to switch back to full or figure out some other topo-based connectivity to allow adaptation to work.

• CT also enters one-way into PT maintenance dynamic through VM -- can keep that updating as well. It is unclear that BG actually needed for sustained maint -- striatum cells reflect it but may just be reflecting CT / PFC stuff. Do PT neurons also have auto-shutoff or not? In primate, lots of evidence of final action triggering clearing of prior delay activity -- so BG maint is a toggle, but CT can keep the continuity in a more automatic way.

[rcoreilly]

TODO: major unknown is source of drivers for MD into various areas (OFC, ACC, mPFC, etc)

[rcoreilly]

To emphasize a few things:

• "agate" was always a bit vague on the exact nature of the attention, as Tom pointed out. It was still based on a phasic, irregular style of updating, like we had always pictured for the BG. By contrast, the new MD <-> PFC SRN-based framework is in some sensethe more obvious extension of the posterior cortical one: continuous, regular gating via context updates. The two things that I didn't appreciate before are: 1. The timescale of updating can be much longer (1-2 sec vs. 200 msec); 2. A turn-off mechanism is absolutely necessary, to prevent the thing from just freezing in one active state forever. The lack of this in the posterior model is a significant oversight, and in general it seems we always neglect the need to turn things off, rather than just on.

• In the context of this continuous updating SRN-like mechanism, the BG adds the critical element of phasic, discrete gating. Also, consistent with updated agate era story, it seems like the BG is a very broad, nonselective gating signal, potentially coordinated across the entire PFC for a rodent. This is a strong departure from the PBWM, and having the more selective, SRN-like MD maint pathway makes such a thing possible -- the ongoing WM context is always there, chugging away independent of BG gating, and BG just needs to jump in and say "go" when the right convergence of state happens.

• By fully embracing the Rubicon framework, we establish a reliable toggle mode for BG gating: once at the start to engage the goals, and once at the end to clear them. For simple externally-cued delayed action tasks in the lab, this has the simple form of gating at cue onset, followed by gating to make the response and get the reward. It remains a bit murkier for more naturalistic contexts -- still an important TODO

• This toggle-mode BG model also says that you don't have an actual hierarchy of goal engagement -- there is only one engaged goal state at a time. However, the MD / SRN maintenance again allows for a continuity and updating of state as you go along, so every such goal engagement state is contextualized by the emergent context state. This can effectively nest and coordinate goals and subgoals over time.

  • Maybe in the human we have one extra anatomical outer loop in the frontal polar region?

• In the context of above, the special layer 3 primate maintenance loops could represent an even longer scale form of maintenance that escapes the automatic inhibition and updating of the CT maintenance, thereby giving extra-long-scale continuity and temporal abstraction for the primate. These neurons would be subject to MD inputs, and presumably buffeted around by that, but it is not clear exactly what determines their maintenance duration etc.

[rcoreilly]

The CT dynamics are a bit tricky actually:

• The update interval is fine-grained even if the maintenance window per neuron is long.
• So we effectively need a thresholding per neuron for turning them off -- in general this requires an "offline" accumulation dynamic before inhibition takes effect. Need to look into that.

[rcoreilly]

Turning off active maint at the "Reckoning"

This perennial question of how the active maintenance gets turned off re-emerges here: when we get our US (or give up on it), how does that then reset the maintenance?

[rcoreilly]

Per updated synthesis, probably CT has auto-shutoff and just keeps shuffling through reps over time -- it is a never-ending stream -- and then BG gating of PT neurons is a toggle, so it turns on with initial goal selection, and turns off with BG gating at end. The toggle was always a problem before because we didn't have the "backup generator" of the CT maintenance.

[rgobbel]

I've always assumed that BG gating requires something to gate in order for it to be maintained. In the absence of that, it should just set things to some default state. This is essentially what you're saying in your reply above, yes?

[rcoreilly]

Mechanisms of maintenance in layer 5, rodent

The emerging story from top references below is that maintenance occurs via a SRN-like chain of MD <-> mPFC (also termed prelimbic cortex, PrL, a putative rodent analog of dlPFC in primates) interactions, over relatively long time scales (e.g., each mPFC neuron being active for roughly 2-5 secs), with a dynamically updating population of neurons chaining along "relay race" style. Computationally, this is like the reservoir computing dynamic of a randomly shifting pattern, but with much longer update windows, presumably due to the time dynamics of the NMDA and bursting etc in the MD <-> PFC circuits. It is good for giving a unique timestamp to each point in time as the pattern of activity present at that moment, and thus for providing a good timing signal.

This rat-level dynamic differs from the classical primate dlPFC delay period activity, where a more "canonical" sustained activity pattern is maintained over time, via the layer 3 specialized circuits presumably. However, the delay windows are relatively long -- 5 sec or more. At shorter intervals, it is possible that individual neurons could span the gap..

The Halassa lab studies show that MD -> PFC has a contextual gating-like effect, helping to translate cues into task rule behavior. MD receives cue information from a subset of cue-selective PFC cells, and then projects that back to task-selective PFC cells that are cue-independent, but mediate actual task performance. They talk a lot about MD -> inhibitory effects but my sense is that this is similar to how all thalamic projections are likely to drive both excitation and inhibition in ways that cancel out for task-relevant cells but inhibit task irrelevant ones.

A key missing part of this story is about the potential role of the drivers into the MD vs. the CT top-down "predictive" inputs -- for the cue-selective parts, it would seem that the drivers are task relevant cues (e.g., from OFC?). Others could be having motor output drivers. Need to go back and find the papers on that!

The MD <-> PFC nature of this maintenance circuit suggests a natural way in which the stronger layer 3 recurrent maintenance could evolve.

Overall, this literature provides an important additional ingredient to the "agate" model above -- sequences of SRN updates instead of some kind of more static sustained activity. Also, it shows how in PFC the time scale of these CT / SRN updates is much longer than posterior cortex (2+ sec vs 200 msec), providing a clue for the long-sought bridge to longer behaviorally-relevant timescales.

Note that in well-practiced tasks, OFC is not important (but mPFC / PrL is reliably important), and it has a separate MD <-> OFC pathway -- we can plausibly assume similar dynamics across all areas.

RikhyeGilraHalassa18

Taken together, these results indicate that transient PFC responses were due to the learned cue, with cue-selective cells representing a physical cue and cue-invariant cells representing its meaning as a task rule.

At the population level, the cueing context was much more decodable from both persistent and transient MD neurons than from PFC RS neurons (Fig. 2e, f). Therefore, across the PFC–MD network, MD and PFC FS neurons were the most informative of the cueing context, whereas PFC transient neurons were most informative of the rule (Fig. 2f).

Furthermore, when we presented all four cues in a randomized manner, we found MD neurons that responded to all four cues, suggesting that their combination was encoded as a single context (Supplementary Fig. 6e–h). Altogether, MD activity reflected the statistical regularity of cue presentation over a multitrial timescale, which we refer to as the cueing context (Supplementary Fig. 6i,j).

In contrast, more than 75% of the variance in the delay- period activity of both transient and persistent MD neurons could be explained by inputs from cue-selective PFC neurons, with MD neurons more likely to receive inputs from the context-congruent cortical cue set (Fig. 2h).

Therefore, these findings suggest that MD neurons gained their contextual selectivity, at least partly, by pooling context-specific cue inputs from the PFC. MD transient cells pooled from PFC cells in a temporally precise manner, while MD persistent cells pooled from PFC cells over a broader temporal window (Supplementary Fig. 7g,h).

We noticed that mice performed much better on sessions in which the four cues were separated into two cueing contexts compared to sessions in which the cues were equiprobable (Fig. 3a). This performance advantage occurred regardless of how cues in the block were grouped with respect to modality. Critically, although performance within either of the two blocks was higher than when cues were fully randomized across the session, there was a clear and consistent behavioral detriment for 4–8 trials upon switching from one block to another (Fig. 3b). This decline in behavioral performance at the switch suggested that, despite previous learning, mice had to readjust to using cues in the new cueing set.

In contrast, although the coupling between cue-selective PFC neurons and context-selective MD neurons followed broadly similar dynamics, they were too fast to correlate with behavioral performance (Supplementary Fig. 10c–g). Therefore, the output of the PFC cue-invariant cells, and not the MD or PFC cue-selective cells, were most likely used for controlling sensory selection and successful task performance.

Notably, these functional inputs originated from the two distinct MD functional subgroups; persistent MD neurons were more likely to provide inhibitory functional inputs, while transient MD neurons predominantly provided excitatory ones (Fig. 4d).

SchmittWimmerNakajimaEtAl17

We found that certain PFC neurons signalled the task rule by an increase in spiking for a brief moment during the delay (Fig. 1b)... The majority of such neurons signalled only one rule, but a minority (17%) signalled both rules at different temporal offsets (Extended Data Fig. 1e, f)... Tuned peaks tiled the entire delay (Fig. 1c), and tuning was independent of delay length (Extended Data Fig. 1h).

The largest source of 4AFC errors was executive (Extended Data Fig. 2f), consistent with our interpretation that similar misat- tribution of task rules explains most of the errors in the 2AFC task (Supplementary Discussion 2).

When multiple cells encoding the same rule were simultaneously recorded (Extended Data Fig. 3a), their consistent temporal order indicated that a neural sequence maintained the relevant categorical representation over time 11–15. Consistent with this, we found that many PFC neuronal pairs exhibited robust short-latency cross-correlations, indicating their co-modulation at synaptic timescales 16,17. This co- modulation was related to similarity in both categorical and temporal tuning (Extended Data Fig. 3b–f).

Temporally limited manipulations revealed that early PFC manipulation diminished late task rule representation, even when the rule presentation period itself was spared (Fig. 1k, l and Extended Data Fig. 4b-d)

MD were non-selective to rule as the vast majority showed identical activity for both the attend to vision and attend to audition trials.. Notably, peaks were only encountered in the lateral mediodorsal thalamus (Extended Data Fig. 7f)

Overall, these data indicate that mediodorsal engagement in the delay is not to relay categorical rule information, but rather to sustain existing cortical representations. Consistent with this, neither overall spike rates nor tuning peaks were changed in error trials (Extended Data Fig. 7i, j), confirming the conclusions that most errors in this task are due to rule misattribution and that this information is generated cortically.

We found that mediodorsal neurons increased spiking upon task engagement, but that only inhibitory cortical (fast-spiking) neurons and not excitatory (regular-spiking) neurons showed a similar increase (Fig. 3a). This was also true for changes seen in the task delay; mediodorsal and cortical fast-spiking neurons showed additional spiking enhancement, whereas regular-spiking neuronal spike rates remained relatively unaltered (Fig. 3a).

This modulatory mediodorsal-to-PFC input is very different from the driving PFC to mediodorsal thalamus input we observe after SSFO- mediated activation of the PFC (Fig. 3d and Extended Data Fig. 8g–i), and is consistent with the theoretical prediction that strong (driver– driver) thalamocortical loops are not implemented in the brain 28

Progressively increasing mediodorsal excitability to mimic task engagement enhanced corre- lated spiking among model PFC neurons, but resulted in rule-specific neural sequences only when co-tuned ‘starter’ neurons were synchro- nized by a common input (Extended Data Fig. 9g, h). The theoretical prediction that a combinination of mediodorsal amplification of cortical connectivity with a specific input that drives initial synchrony is sufficient to generate a sequential categorical representation, was experimentally validated (Extended Data Fig. 9i, j).

Because other thalamic circuits are thought to enhance connectivity between cortical areas, the precise engagement of the thalamus in a cognitive process may determine which cortical regions process information locally and which are engaged when processing spans multiple cortical nodes.

NakajimaSchmittHalassa19

Another paper in the above series, showing PFC -> BG -> visTRN control over visual processing -- another mechanism of top-down control. Need to look back at this again later.

ChernyshevaSychFominsEtAl21

Summary paper: ParnaudeauBolkanKellendonk18

In addition, analysis of mPFC interactions with MD in the thalamus revealed that activity of the MD->mPFC pathway is essential for sustaining prefrontal activity during WM maintenance 7–9. The reciprocal mPFC-MD pathway, on the other hand, was found to mainly guide successful WM retrieval and to support subsequent choice 8. This efferent pathway appears less important for WM maintenance during the delay period, when MD activity actually leads activity in the mPFC 8, and it is not essential for encoding.

Moreover, lesions of dorsomedial, but not dorsolateral, striatum result in WM impairments 12,13. Additionally, electrophysiological recordings from either mPFC or dmStr revealed neuronal populations that display activity patterns spanning WM maintenance periods in a sequential manner 8,14–16.

Taken together, our results show that mPFC -> dmStr projection neurons display orderable activity patterns in all task phases, including the delay period, and thus indicate the presence of neuronal sequences in this specific population not only during maze runs but also during WM maintenance.

Across task periods, we found a relatively high fraction of neurons predictive for left vs. right turning in the encoding and retrieval periods but not in the maintenance period.

Because neural sequences also have been found in striatum for tasks not involving spatial WM44 and because sequential delay activity in dmStr was dissociated from stimulus encoding activity 14, delay-period spanning neural sequences in dmStr may have a more general role in preparatory activity for future choices rather than serving solely working memory.

BolkanStujenskeParnaudeauEtAl17

This is ref 8 from above. Same group as 7: SpellmanRigottiAhmariEtAl15

Note: this and the above paper used a much simpler deterministic alternation paradigm, and they readily admit that this prevents anything interesting from being seen during the delay period -- thus the usual task state maintenance is definitely happening during the delay period (as in the Halassa lab papers), but in these papers it is missing.

Delay-elevated activity may instead reflect a general attentive or task-engaged state. Although our findings do not exclude this possibility, we did not observe overt behavioral differences between correct and incorrect trials, nor was the latency for mice to make a spatial choice altered by MD terminal inhibition (data not shown). A final possibility is that explicit spatial representations are unnecessary for two-choice spatial working memory tasks and that an abstract task rule, such as ‘go to the opposite location’, is sufficient to guide behavioral performance.

Notably, in a two-choice version of this task analogous to ours, seminal work demonstrating sustained delay period activity in primate PFC also reported that spatial preferences in delay-tuned neurons were either absent38 or minimal39,40 (6–13% of sustained delay units). mPFC delay activity did also not appear to encode timing of the delay interval, as delay-elevated mPFC neurons did not scale their activity according to distinct delay durations (Supplementary Fig. 8), a feature observed in neural ensembles explicitly linked to interval timing 32

Nevertheless, task-rule encoding in PFC neurons has been frequently reported in both rodents41,42 and primates43,44. Moreover, findings from a recent working-memory-guided, top-down attention task, in which task rules are varied on a trial-by-trial basis, explicitly demon- strate task-rule encoding in mouse mPFC neurons during the delay period45. Like the delay-elevated mPFC activity in our task, this study observes temporally sparse and sequential mPFC spiking coding for one of two task rules during the delay period. Notably, and further strengthening our inference, delay-elevated activity in this task requires MD activity for its sustained maintenance across the delay. Our results make clear that this effect is due to activity in MD-to-mPFC projections and not to alterations in general excitability.

Using pathway-specific inhibition, we found directional interactions between mouse MD and medial PFC (mPFC), with MD-to-mPFC supporting working memory maintenance and mPFC-to-MD supporting subsequent choice. We further identify mPFC neurons that display elevated spiking during the delay, a feature that was absent on error trials and required MD inputs for sustained maintenance. Strikingly, delay-tuned neurons had minimal overlap with spatially tuned neurons, and each mPFC population exhibited mutually exclusive dependence on MD and hippocampal inputs. These findings indicate a role for MD in sustaining prefrontal activity during working memory maintenance. Consistent with this idea, we found that enhancing MD excitability was sufficient to enhance task performance

That is, while mPFC neurons primarily projected to and received input from the lateral and medial MD, OFC neurons predominately projected to and received input from the central MD.

we generally observed denser connectivity with lateral MD when dorsal mPFC was densely labeled and denser connectivity with medial MD when ventral mPFC was densely labeled. As recently reported in the mouse 22, this reflects the presence of two distinct and topographically organized MD–mPFC circuits. This topographic and reciprocal MD–PFC organization is consistent with anatomical findings in primates 19 and rats 20,21.

We found a robust impairment of task performance when MD terminals were inhibited within the mPFC for the duration of a trial (Fig. 1e). No effect was observed when MD terminals within OFC were inhibited in an identical manner (Fig. 1e).

These findings confirm previous observations that in rodents, mPFC neurons do not represent goal arm locations in sustained firing during the delay phase of T-maze tasks 18,28 and further demonstrate that spatial representations of arm locations in mPFC neurons are not dependent on MD-to-mPFC activity.

we found a subset of mPFC neurons (266 of 891) that exhibited significant elevations in spik- ing during the delay relative to the intertrial interval (ITI; z >2, P < 0.05 for 2 or more consecutive 1-s bins), in which behavioral conditions were equivalent but mice were not required to maintain a working memory trace. Elevations in spiking for each neuron were not sustained; rather, each neuron exhibited a preferred temporal offset during the delay phase (Fig. 5a).

AkhlaghpourWiskerkeChoiEtAl16

Ref 14 from above, recording in dmStr

To some extent, shows sequence of delay-period activity like mPFC, but actually most information encoding was at the start:

In fact, sample information most often peaked towards the onset of the delay period, with more than 80% of the stimulus encoding neurons (45/53) displaying peak information within the first 2 s of the delay period. On average, peak sample information was significantly earlier than peak activity (p<0.001; Wilcoxon signed rank test. Only neurons with significant information peaks were considered for this analysis).

Thus, it seems likely that BG is doing early gating and then likely reflecting delay-period activity?

some neurons preferentially encoded lever identity during the sample press while others encoded the lever identity during the choice press (Figure 7C). There was significantly more dispersion in the difference between sample and choice information than expected by chance (inset in Figure 7C, p<10–5; two sample F-test for equality of variance for (sample information – choice information) in shuffled versus non-shuffled data). This analysis supports the conclusion that DMS firing patterns are not merely reflecting position or motor actions.m

YangShiWangEtAl14

This is a "direct hit" for large-N electrophysiology in the rat, doing delayed alternation Y-maze task. They do not show sustained delay period activity, but rather a sequence of firing over time that looks more like a "relay race" -- a chain of more short-term windows of activity. This would give better timing information, and should be something that the SRN-style CT layers can support.

Firstly, most of the task-related cells are multi-event related cells, that is, an individual cell could be involved in different task events (Table 1). The multi-event involvement may facilitate information transfer across task periods. Secondly, all of the delay cells, either differential or non-differential ones, showed transient but not persistent activity, which is different from dlPFC cells in monkeys exhibiting sus- tained firing during delay [1]. Thirdly, differential and non-differential firing existed in all of the task periods. It was possible that the differential activities reflect trial-type effects on cells, such as spatial information or working memory, while the non-differential activities reflect general task components, such as motor running, reward expectation, and reward consuming. Fourthly, the transient and differential delay activities in individual cells appeared at different temporal points during the delay. The sequential firing pattern during the delay may reflect relay-race transfer of working memory information [24]. The firing difference between the rodent mPFC (transient activity) and the monkey dlPFC (persistent activity) may reflect the difference in evolutional hierarchy of the two species.

WangStradtmanWangEtAl08

Tom Hazy pointed out this paper a while ago -- they find "classical" NMDAR-based sustained maintenance mechanisms in layer 5 PFC in the rat. These are the same NR2B subunit type as in the layer 3 primate circuits, enriched in mPFC in rat layer 5 relative to other cortical areas.

These NMDA have longer decay time (81 ms vs 43 in V1) -- TODO: could try faster tau in posterior models.

Since the duration of NMDA current is largely controlled by the receptor subunit compositions and the decay time constant of synaptic responses generated by NR1-NR2B receptors is much slower compared to NR1-NR2A receptors (Cull-Candy et al, 2001)

DembrowChitwoodJohnston10

In several neocortical regions, including the rat dorsal medial PFC (mPFC), layer V pyramidal neurons have distinct interconnectivity, morphology, and firing patterns depending on their long-range projection targets (Molna ́r and Cheung, 2006; Morishima and Kawaguchi, 2006; Otsuka and Kawaguchi, 2008; Brown and Hestrin, 2009)

The distinct responses of neurons that project to the pons [corticopontine (CPn)] and neurons that project to the contralateral cortex [commissural (COM)] to dynamic subthreshold stimuli can be abolished by blocking the hyperpolarization-activated cyclic nucleotide-gated cation h-current.

Consequently, alpha2-adrenergic and cholinergic modulation, which modify h-currents (Carr et al., 2007; Wang et al., 2007; Barth et al., 2008; Heys et al., 2010), alter the dynamic and integrative properties of CPn but not COM neurons. Additionally, intrinsic properties not accounted for by the h-current were also unique between COM and CPn neurons. CPn, but not COM, neurons reliably display activity-dependent persistent firing in the presence of cholinergic agonists. With such disparate integrative properties and responses to neuromodulation, CPn and COM neurons may contribute in different ways to mnemonic persistent activity.

Increases in acetylcholine occur in the PFC during cue detection and working memory-like tasks (Parikh et al., 2007).

The findings of this study expand this pattern to the rostroventral rat mPFC, which is known to contribute to working memory-like tasks, pattern recognition, motor preparation, and fear acquisition/ extinction in these animals (Dalley et al., 2004; Burgos-Robles et al., 2007; Fujisawa et al., 2008).

The ability of cholinergic modulation to make neurons fire persistently was also dependent on their projection target. CPn neurons were more likely to fire persistently than COM neurons. This persistent activity was maintained in the presence of synaptic blockers, indicating that it functions independently of any network reverberations. Thus, it appears that neurons projecting subcortically to the pons can be more readily switched into a “persistent” mode, whereas intracortically projecting neurons have a lower propensity to do so.

DembrowJohnston14

Review paper of above..

GilmartinBalderstonHelmstetter14

Lots of interesting data on PL / mPFC for trace conditioning, and ACh modulation thereof.

SreenivasanDEsposito19

broad review paper..

GuoYamawakiSvobodaEtAl18

The anterolateral motor cortex (ALM) and ventral medial (VM) thalamus are functionally linked to support persistent activity during motor planning.

In the mouse, the anterolateral motor (ALM) cortex has premotor-like properties, particularly in behavioral tasks involv- ing motor planning during a temporal delay (Guo et al., 2014; Inagaki et al., 2018). ALM neurons show persistent activity anticipating specific movements, seconds before movement onset. These studies have implicated strong bidirectional interactions between ALM and the thalamus, including the VM nucleus, as being crucial for these behavioral functions (Guo et al., 2017). The ALM-recipient subdivision of VM receives input from substantia nigra pars reticulata as well as the fastigial nucleus of the cerebellum (Gao et al., 2018).

An important implication of this result (addressed in experiments presented in a later section) is that if the potential ALM 7 VM T-C-T loop is tightly “closed” by monosynaptic connections, such connections must involve the PT neurons, rather than the CT neurons, given the prevalence of monosynaptic VM ¡ PT and the paucity of VM ¡ CT connections.

Thus, layer 2/3 pyramidal neurons also receive strong VM excitation.

Consistent with the electrophysiological results, anatom- ically the M1 PT axons were found to ramify in the laterally located M1-projecting subregion of VM, rather than in the ALM- projecting subregion (Fig. 5F–I ). These findings provide further evidence that the T-C-T loops in the motor areas are relatively segregated.

Thus, these results indicate that ALM-projecting VM TC neurons, rather than MD TC neu- rons, are the primary thalamic targets of ALM PT axons

[rcoreilly]

Synapse-level NMDA active maintenance: fast weights binding

Todo: some evidence for this in above papers -- makes a lot of sense computationally -- explore in model!

[rgobbel]

If I'm understanding it correctly, all this is right in line with the way I was thinking about the role of the BG way back when I was writing my dissertation: phasic dopamine acts as a signal to instantiate a previously-learned "set" (action-plan selection).
There was a short but intriguing paper I found years ago (Ragsdale & Graybiel PNAS 1990) that seemed to hint at possibilities for hierarchical control—some sort of hierarchy, at least. Maybe relevant to this topic? It describes a very regular pattern of differences in corticostriatal projections to patch and matrix. Never followed up in any way by anybody, as far as I can tell, but I've wondered about it for years.

[rcoreilly]

Scope of BG Gating: Spirals and Loops

To make the BOA model, we need to know how to connect the xFC -> BG -> thal <-> xFC circuits. The simplest conceptual version of the Rubicon model is one global BG gating event that engages goal representations distributed across OFC, ACC, and dlPFC. However, there is a big literature on all the different pathways, and things are very different between primates and rodents, so we need to sort through this literature to determine a good starting point and what to anticipate in the design for future expansion into the primate level.

Primate loops and spirals

The original parallel loops paper is Alexander, DeLong & Strick (1986), identifying the following loops:

Note that there are both open and closed (looping back to itself) aspects to the loops. This figure from Bosch-BoujuHylandParr-Brownlie13 shows more clearly how there is a "spiraling" structure to the open parts of the loop: "higher" (dlPFC) and more "motivational" areas (OFC, ACC) drive "lower" areas:

This spiral is well documented by Haber and colleagues across the BG and dopaminergic pathways -- here's HaberMcFarland02 which has a good summary of relevant data:

And Tom Hazy's rendition of the thalamic connectivity from OReillyHazy22 (Thalamus chapter) usefully comparing the primate and rodent pathways:

And here's a detailed map of the thalamus from RovoUlbertAcsady12

The green areas in VL / VA / VM and parts of MD indicate areas where BG inhibition is the primary driver, while red areas have cortical drivers, with the Pulvinar (Pul) areas being the prototype -- these are the predictive learning parts of the thalamus. Parts of the MD also have this cortical-driver property.

Rodent BG thalamocortical circuits

CollinsAnastasiadesMarlinEtAl18 and AnastasiadesCollinsCarter21 provide good updated accounting of at least the prelimbic (PL) based circuit, which is roughly analogous to dlPFC in rodents.

In sensory cortex, first-order inputs densely innervate layer 4 (L4) to drive activity in the local network (Agmon and Con- nors, 1991; LeVay and Gilbert, 1976), whereas higher-order in- puts target superficial layer 1 (L1) (Castro-Alamancos and Con- nors, 1997; Jones, 2001). In contrast, the PFC lacks a traditional thalamo-recipient layer (Van De Werd et al., 2010), with both MD and VM axons targeting superficial layers (Deniau et al., 1994; Groenewegen, 1988).

Most PFC CT neurons were located in layer (L6), with a second, smaller population also present in layer 5 (L5) (Figure 2B; L5 = 26% ± 0.1%, L6 = 74% ± 0.1%).

These findings demonstrate that both L5 and L6 CT neurons can send diverging axonal projections to both MD and VM.

To isolate L5 CT inputs, we took advantage of the fact that they overlap with pyramidal tract (PT) neurons, whereas L6 CT neurons do not (Harris and Shepherd, 2015). ... We observed strong overlap of PT and CT neurons in superficial L5, which was absent from L6 (Figure 3B; CT overlap: superficial L5 = 80% ± 3%, deep L5 = 8% ± 1%, L6 = 0.6% ± 0.8%).

NOTE: 20% of L5 are "CT" not "PT" -- could potentially lump these in with L6 pure CT

Consistent with established anatomy (Krettek and Price, 1977), we found MD axons were densest in deep L1 and L3, whereas VM axons were concentrated in more superficial L1.

These results determine that both MD and VM make monosynaptic connections onto L5 CT neurons in the PFC but largely avoid L6 CT neurons, which constitute the main population that provides cortical output to thalamus.

NOTE: this is consistent with Guo -- and with idea that CT is not directly gated by BG (or anything else).

These findings indicate that MD input is biased onto CC neurons over CT neurons in L5, whereas VM input is equivalent at both populations, suggesting different functions for these inputs.

Also, MD targets 2/3 much more strongly -- VM is fairly uniform.

These findings support a role for MD in driving L2/3 CC neurons, with VM providing broader, more modulatory influence across a variety of cell types and cortical layers.

PFC provides input across the rostral-caudal and ventral-medial axes of VM, with similar distributions observed after injections in motor cortex (Guo et al., 2017),

In sensory cortex, thalamic inputs can also target neurons in deep layers, with prominent bands of axons in L5 and L6 (Constantinople and Bruno, 2013; Crandall et al., 2017; Gilbert and Wiesel, 1983). In contrast, thalamic axons in deeper layers of PFC are sparse, which may explain the relatively weak thalamic input to L6 neurons.

VP -> MD: Root et al, 2015

RootMelendezZaborszkyEtAl15 is a major review of ventral pallidum (VP) which is a major output of the ventral striatum (nucleus accumbens) -- it is the equivalent of GPe for the pcore model (by hypothesis).

Most neurons were categorized as Type I (46%), which belong to a group of motivational salience-encoding neurons observed throughout all basal forebrain regions (Lin and Nicolelis, 2008; Avila and Lin, 2014b). The Type I neurons of Avila and Lin (2014a)
are highly sensitive to cues predicting the start of a reward trial as well as conditioned stimuli and the reward itself, especially when predicted rewards are comparatively robust. The motivational salience signaling of Type I neurons is correlated with faster decision speed (Avila and Lin, 2014b) and a short latency frontal cortex potential (Nguyen and Lin, 2014). Together, Type I neurons were interpreted as non-cholinergic corticopetal basal forebrain neurons (Avila and Lin, 2014a).

A major target of VPvm is the MD (Haber et al., 1985; Zahm and Heimer, 1990; Groenewegen et al., 1993; Kalivas et al., 1993; Zahm
et al., 1996; Churchill et al., 1996; Heimer et al., 1997; O’Donnell et al., 1997; Tripathi et al., 2013), and ultrastructural visualization of this projection has identified large terminals that synapse primarily on dendritic shafts (Kuroda and Price, 1991). VPr and VPvl also project to the MD (Young et al., 1984; Zahm and Heimer, 1987; Groenewegen et al., 1993; Tripathi et al., 2013) but the VPdl projection to MD is significantly less than that from other VP subregions (Zahm et al., 1996; O’Donnell et al., 1997). The VP projection to MD is predominantly GABAergic and partly cholinergic (Haber et al., 1985; Young et al., 1984; Kuroda and Price, 1991; Ray et al., 1992; Mariotti et al., 2001).

VPvm efferents terminate onto MD neurons that project to the prefrontal cortex (Vives and Mogenson, 1985; Lavı ́n and Grace, 1996; O’Donnell et al., 1997). The part of MD that receives VP inputs projects strongly to cortical areas that in turn project to AcbC (Zahm and Brog, 1992; Zahm et al., 1996; Zahm, 1999). Therefore, VPvm may be capable of altering AcbC-VPdl processing through a serial, laterally spiraling circuit.

The MD response to electrical stimulation of the VP is predominantly inhibitory

The VP has weak projections to several thalamic nuclei, including ventromedial nucleus, nucleus reuniens, paraventricu- lar, intralaminar central medial and paracentral nuclei (Groene- wegen et al., 1999).

The VP projects to the basolateral amygdala (BLA) (Conrad and Pfaff, 1976b; Troiano and Siegel, 1978b; Haber et al., 1985; Carlsen et al., 1985; Mascagni and McDonald, 2009). Roughly 75% of VP neurons that project to the BLA are from cholinergic neurons (Carlsen et al., 1985; Zaborszky et al., 1986a,b). Tracer injections in BLA and cortical regions demonstrate scant double-labeling (Zaborszky et al., 1986a,b)

Ventral striatopallidal processing has been hypothesized to involve the ‘‘initiation’’ of actions (Mogenson et al., 1980) and if so, firing patterns might be expected to precede the onset of approaching to obtain cocaine. However, it was identified that VP firing patterns were coincident with approach, response, and retreat behaviors (Fig. 8A, B), and did not occur preceding the onset of approach. Moreover, VP behavioral firing patterns during cocaine self-administration differed according to the particular subregion in which the VP neuron was recorded from.

Fig8: VPvm showed phasic inhibition (hence disinhibition of MD) at onset of approach.

A significant population of VPdl neurons continues to change its firing rate through the entire sequence of acquiring cocaine, approach- response-retreat (Fig. 8G). In other words, VPdl sends a signal related to pursuing cocaine and the reaction to obtain cocaine.
The pursue and react signals of VPdl neurons are similar to proposed function of AcbC neurons in biasing an organism to ‘‘go to it’’ (the reward) (Floresco, 2014).

Firing patterns involving the retreat may reflect processing of the outcome of responding, which likely is a function of the part of VP that receives inputs from AcbSh (e.g., the VPvm; Leung and Balleine, 2013).

This is supported by the finding that approach or response-related firing patterns within the mPFC typically occur after approach or response-related firing patterns occur within the Acb (Chang et al., 2002). Our results together with those of Chang and colleagues suggest that mPFC behavioral firing patterns are facilitated by VPvm-MD-mPFC or VPvm-VTA-mPFC circuits (Fig. 7). This may facilitate prefrontal processing of response-outcome relationships in order to guide future behavior

[rcoreilly]

Further details on MD, VM projections: AnastasiadesCollinsCarter21

AnastasiadesCollinsCarter21 provide more info on PFC terminals and effects of MD, VM projections.

Overall, the simplest summary is that MD is net disinhibitory of NMDA Ca+ channel maintenance currents, while VM is net inhibitory, with also direct transient excitation of PT cells. In other words:

• MD turns on active maintenance and VM turns it off, while transiently activating PT neurons. See below for the Rubicon-based implications of this.

By targeting either VIP+ or NDNF+ cells, MD and VM could either inhibit or disinhibit the PFC, with important implications for network activity (Letzkus et al., 2011; Palmer et al., 2012; Pi et al., 2013).

We first show that L1 consists of two sublayers, with VM selectively driving NDNF+ cells in L1a and MD engaging VIP+ cells in L1b. We then show that these interneurons participate in distinct inhibitory networks, with NDNF+ cells inhibiting PV+ cells and VIP+ cells inhibiting SOM+ cells in L2/3. We also show that NDNF+ cells inhibit the apical dendrites of pyramidal cells, suppressing action potential (AP)-evoked Ca2+ signals in the distal dendrites of PT cells in L5. Lastly, we show that NDNF+ cells mediate thalamus-evoked inhibition, blocking VM-evoked Ca2+ signals in the apical dendrites of PT cells.

These results indicate that thalamic inputs differentially engage L1 interneurons, with VM stronger onto VIP- (NDNF+) cells and MD showing no bias. They also show how differences in synapse strength depend on the source of thalamic input, whereas short-term dynamics depend on the postsynaptic cell type.

These experiments indicate that NDNF+ synapses are enriched at the api- cal dendrites of PT cells, where they strongly inhibit electrogenesis and Ca2+ signals.

Interestingly, MD targeting of VIP+ cells in PFC resembles activation of somatosensory VIP+ cells by inputs from the motor cortex and higher order thal- amus (Lee et al., 2013; Williams and Holtmaat, 2019). VIP+ cells in turn inhibit SOM+ cells, a motif observed across cortex (Lee et al., 2013; Williams and Holtmaat, 2019), including the PFC (Pi et al., 2013). Thus, our results suggest that MD activates a conserved disinhibitory circuit, with the inhibition of SOM+ cells suppressing dendritic inhibition of pyramidal cells, potentially enabling dendritic electrogenesis and synaptic plasticity (Wil- liams and Holtmaat, 2019).

MD input to L1b VIP+ cells may combine with input to L3 pyramidal cells (Collins et al., 2018) to maintain local excitation and support delay period activity (Bolkan et al., 2017; Schmitt et al., 2017). Similarly, MD input may also amplify cortical or basolateral amygdala inputs to superficial layers (Little and Carter, 2013; Schmitt et al., 2017), as suggested for higher order thalamic in- puts to sensory cortex (Williams and Holtmaat, 2019; Zhang and Bruno, 2019)

[rcoreilly]

Primate <-> rodent homologies: IL -> 25, Cg -> 24, PL -> 32?

• VogtPaxinos14 make a strong mapping between PL (rodents) = 32 pgACC in primates, IL (rodents) = 25 sgACC in primates -- this paper also has a lot of rich additional connectivity with meaningful subdivisions between striatal, spinal and other subcortical projections.

• HeilbronnerRodriguez-RomagueraQuirkEtAl16 -- key paper suggesting that IL -> 25 and Cg (rodent cingulate) -> 24 are solid mappings, and PL -> 32 has some interpretational issues despite clear connectivity mappings.

• WallisCardinalAlexanderEtAl17 -- reinterpretation of some primate mappings

Summary of Ongur & Price data on subcortical <-> vmPFC connectivity, with theoretical overlay:

Vogt & Paxinos (2014) mapping:

Functional Interpretation

• 24 / Cg is more motor-specific cost -- effort, conflict, time, uncertainty, etc, corresponding to associated motor cortex hierarchy: more posterior 24 maps to more posterior frontal areas (SMA, M1) while more anterior are higher level (dlPFC in primate, ALM in rats).

• 32 / PL is the highest-level ACC representing the integration of action cost from 24 and outcome value in OFC into an overall utility -- this is the "ACC" typically discussed in the BOA model. It is an important clarification to distinguish cost from utility in 24 vs 32, even if the biology might not be as clean as all that.

• 25 / IL is the most unambiguous mapping but also perhaps the most confusing functionally. In humans it is associated with depression, and it might be associated with aversive outcomes and avoidance behavior. But could it also be the thing that gets activated when you give up a goal? Monitoring for bad outcomes et?

• Although PL is often characterized as the "dlPFC of rats", the above suggests that it is instead the "pgACC" of rats, and that ALM (anterior-lateral motor cortex) in rodents is the better mapping onto primate dlPFC.

• The different regions of OFC can be mapped onto different corresponding "OFC" areas in rodent, per HeilbronnerRodriguez-RomagueraQuirkEtAl16 -- VOLO, MO

[rcoreilly]

A Strong Rubicon MD / VThal Hypothesis

The Collins et al (2018), Anastasiades et al (2021), and Root et al (2015) papers in rodents (quoted above) together support the general idea from agate that MD and VM (Vthal) thalamic pathways have distinct cortical laminar targets, with MD targeting layer 3 "active maintenance" cells with more focal connectivity, while VM drives very broad modulatory impacts across the entire layer, including layer 5. Furthermore, MD activation of VIP+ is disinhibitory of NMDA Ca++ maintenance, while VM is net inhibitory, while providing transient PT activation. Notably, and in distinction with posterior cortical areas, layer 6 CT neurons do not receive direct input from either MD or VM.

Putting all of this together with the basic Rubicon framework:

• Goal selection is completed by a phasic gating dynamic through these pathways: OFC, ACC -> VS -> VP -> MD -> OFC, ACC, dlPFC -- this engages robust active maintenance in the target PFC layers via layer 3 recurrent connections, with direct excitation and VIP+ disinhibition.

• Goal completion involves a final goal-completing action that triggers phasic gating through dlPFC -> VM -> dlPFC that transiently activates PT neurons associated with the action while globally inhibiting Ca++ maintenance currents.

• Intermediate actions along the way toward the goal may involve VL / VA gating of PT that has a more phasic character, affecting SMA and similar lower-level motor areas, assuming that these projections are similar to VM in having direct PT excitation with broad NDNF+ inhibition of maintenance.

• The CT -> MD, VThal projections provide the ongoing slowly-updating "context" reps, independent of any gating signals, and potentially an open-loop CT -> MD -> layer 3 predictive error-driven learning circuit, which is enlarged in primates where more of MD is pulvinar-like instead of being BG gated.

Keeping MD gating separated and uniquely capable of driving layer 3 active maintenance in dlPFC, OFC, ACC provides a nice extra level of robustness for the goal-engaged state, vs. the "hustle and bustle" of Vthal gating during the task.

Having a special VM -> turn off maintenance pathway is also critical for this long-sought mechanism.

However, one missing piece is a way of turning off maintenance in OFC, ACC -- the MD does have some balanced effects NDNF+ cells and could certainly serve as a toggle. Also Tom points out in other papers some indication of diversity in the MD projections, with some more focal and some broader -- so here's a potential resolution:

• US's preferentially activate a matrix-style MD pathway that preferentially targets NDND+ cells and serves to turn off the prior goal maintenance states, while CS's preferentially activate the core-style MD pathway targeting VIP+ and engaging active maintenance.

[rcoreilly]

Credit assignment for MD gating pathway across temporal delay

If we postulate MD gating driving active maintenance of goal reps in PFC areas, then we have to confront the same temporal credit assignment problem as always: how does DA at the final outcome drive learning that will sensibly affect gating at the start of goal selection next time around? There is a circular dependency: you have to already gate in a PFC rep and maintain it in some relatively consistent form over time, in order to associate that PFC rep with BG Go pathway at the time of phasic DA at the outcome. The promise of avoiding these kinds of circular dependencies was one attraction of the non-BG gated agate model.

This is the mother of all learning problems -- the crux of all hard bootstrapping problems in action learning: how do you know what to do at the start (early in time) based on what happens at the end (later in time). We have discussed this in many different contexts over the years, e.g., in terms of inverting forward (predictive) models, and PBWM, etc. Some of these options are discussed in #63 as well:

• BPTT -- literally backpropagate later outcomes to earlier states -- problems with this are articulated at start of CT Learning: The SRN model #63

• SRN / temporally evolving representations -- these work well for tracking temporally evolving sequences with fairly regular support from predictive error signals to shape the representations as they evolve -- this is exemplified by the deep_music model, and is good for unfolding a sequence of actions. But the key principle, and strong empirical data, behind the Rubicon model is that there is a punctate front-loaded goal-selection process, that then drives action toward achieving that goal. This seems to require a stable maintained activity state between goal selection and outcome, which thus is not suitable for the SRN.

• Stable bridging activity -- If you can maintain an identical representation over time, then learning at the end automatically transfers to the start. This is the key idea behind BG gating, and also the LSTM which maintains any gated representation in stable form over time. It is also possible to encode and recall these representations (e.g., from the hippocampus) without maintaining through the duration, but in general neonatal hippocampal damage does not prevent successful intelligent action learning, so we can set that aside. The basic problem with stable bridging activity is the catch-22 enumerated above: you need to prospectively decide what to keep stable in order to bridge it, so this requires a lot of trial-and-error in the general case.

• Cheat: get rid of time entirely -- this is what the transformer does, using full backprop across a temporal window to arbitrarily connect anything across that window. But this only works for a fixed window, and is obviously not what the biology does.

The CS-US bootstrap

By front-loading everything on a small fixed set of USs (unconditioned stimuli: motivationally privileged outcomes), we can potentially short-circuit the catch-22. Here's the general scenario:

• Have instinct-driven simple generic exploratory primitives, so you explore things by default.

• When a US occurs, encode associated sensory cues (CSs) present at the same time, automatically. This is the job of the BLA (basolateral amygdala).

• Next time, when a CS is encountered, automatically do BG gating, to maintain the associated US (and CS probably), at least.

• The key bootstrapping step is that this CS-triggered BG gating automatically gates in everything else in OFC, ACC, dlPFC via the MD thalamus (and BLA has direct input to ventral pallidum -> MD gating pathways). The activity in these areas is initially just a product of the wandering SRN-like representation interacting with whatever is happening at the moment. However, by virtue of the punctate gating, all this stuff gets maintained as a stable bridging representation, which serves to contextualize the subsequent behavioral epoch until US or giving up (somehow -- tbd) toggles it back off. Over the course of learning, these initially random-ish representations take on predictive agency for the actions that ensue, and thus become goal and action-plan representations. The key thing is that the CS-triggered gating defines the behaviorally-relevant time window for this stable bridging activity, greatly reducing the search space of when you might otherwise gate. This is ultimately anchored by the very limited set of USs.

• Likely all the ongoing CT predictive learning stuff is key for shaping the stably maintained predictive reps, and there are issues about making sure that a single hog-like rep doesn't take over everything, but hopefully this can be mitigated.

• Over time, the BG gating learns to recognize the OFC / ACC / dlPFC activity states as good or bad, and gating can be shaped by them. There is a major bootstrapping issue of how these reps get activated in different situations (CS + everything else at start of goal-selection state) so that BG has something useful to evaluate in these reps to begin with.. This is a major emergent aspect of learning that is far from clear to me at this point.

There is some ambiguity in the recording data above about whether stable task-relevant bridging activity is actually present in rodent PL etc, but with the right paradigms, I think they did actually find it -- some paradigms were too degenerate to find something distinctive. Logically, this is necessary.

Ok, I think this is enough of a direction to proceed at this point..

[rcoreilly]

BG Matrix Match Computation

At a very general ACT-R production-system level, the BG Matrix should detect "match" or "conditional" situations: fire when the object is A and position is B etc.

In the BOA rubicon case, VS needs to detect when the CS matches a currently relevant drive state (e.g., food when hungry), and not just respond to any CS.

A general solution to this problem, consistent with striatum physiology and anatomy, and existing theories (Bogacz? Graybiel?), is that there are sparse and systematic patterns of conjunctive connectivity, e.g., a distinct subset of neurons receives from each different drive input, and, orthogonally, from different CS inputs, so distinct neurons can compute the conjunction (match) of drive * CS.

This strategy requires high-threshold AND-like behavior, so the neurons don't fire for either drive or CS independently, but only to the conjunction. This is consistent with the quiet, inhibited behavior of striatal neurons.

Thus, BG is uniquely situated for doing this AND "match" like computation. Note that this is distinct from the broadly integrative conjunctive nature of hippocampal representations. Although maybe EC2 has some similar characteristics?

[rcoreilly]

General US vs. CS functionality / timing issues

For BLA:

• Learning outside of the time of the US introduces a lot of noise. This was true in PVLV and is true here in BOA. There are always small fluctuations in DA and during the "search" process looking for the right CS to approach, BLA gets a lot of transient CS-driven activity reflecting the CS currently being viewed. Per the satiety effect results above, it is likely that BLA continues to activate for these CSs even when they don't match active drives. Thus, learning just based on DA + activity will be noisy. It would be much better to have something that further restricts to actual US. Note that this cannot be a modulatory projection as in the VS because US is not present at time of CS. One possibility is ACh?

• However, the CS activity also produces PPTg deltas based on BLA activations, driving ACh. Also note that PPTg needs to be US-specific in order for each US-distinct CS to actually register a separate ACh signal to enable VS gating to that specific CS, and the same is true for DA, which should reflect the actual learned incentive value of each specific CS.

• The PVLV logic is that BLA drives its own DA (and now ACh) but that learning at time t is based on sending input at time t-1, so that this self-generated DA cannot self-reinforce the same CS -- can only drive second-order conditioning to a neutral CS that precedes it.

• To minimize non-US learning, basically we need something actually US specific -- ACh by itself is not. From VS matrix case below, we can say that US has a double-ACh, but that is still not US specific -- happens when you see two CS in a row. I don't see any way other than to mark the US projections as modulatory, and give them a special role in modulating the learning rate -- with params avail to turn off non-US learning, preventing second-order conditioning. We can start with no 2nd order and then later we can see if we can tolerate a slower lrate for non-US -- that would be consistent with the data on second-order being slower!

For VS Matrix (and trace learning issue raised above):

• There are two modes of learning and gating: initial US driven gating to learn to associate BLA-US + drive -> gate, and later CS->BLA->VS gating at CS onset (at which point we want to make sure later US-driven gating is not interfering with trace learning!)

• Here we need to deal with timing. We can assume that US-driven ACh comes earlier than CS-driven, based on US activation that drives BLA and ACh in parallel. Then, say that DA and ACh from previous trial drives learning and clearing of trace on current trial. Actually need ACh on 2 trials in a row to clear: If t-1 and t ACh, then first apply and clear existing trace with the current dopamine, then add a new trace based on pre * post activity. Table below shows that if we don't have this rule, 2nd trial with CS+ would have prior ACh and would clear trace -- need prior and current -- that ONLY happens at US.

• What happens with the US-driven trace: does it add to CS-trace? It would have to somehow be prevented -- otherwise is just going to happen... start with this and revisit. Also, IF there is actual gating on 2 successive CS-driven ACh-enabled trials (shouldn't happen with drive modulation?) then 2nd trial will drive learning of first.. different kind of second-order effect I guess. Could be interesting.

• Meanwhile, we need the ACh modulation of gating to be all about the current trial ACh -- need to gate when CS and US are present. So, logic of using current + previous ACh for learning and only current for gating seems reasonable.

Summary diagram:

	t: 0	1	2	3	4	notes
CS	CS-	CS+	CS+	CS+	-	CS- = wrong drive, + = right
US	-	-	-	...+	US	US effects at end, US itself next trial
BLA dw	w	w	-	-	+	US mods dwt so weak at t:1
Vs Clr	-	+	-	-	+	Clear then Tr, would clear on t:2 but no cur ACh
Vs Tr	-	+	-	-	+	Assuming no prior ACh -- need to clear prior ACh on US
ACh	+	+	-	...+	+	...+ = plus phase ACh onset -- enough to drive 2 in a row but no gating
DA	+	+	-	...+	+

[rcoreilly]

Just need a USActive signal

The above logic is insufficient for really handling the differences between CS and US. Need to learn VS gating on US on same trial, and that just doesn't happen with above logic. And too many potential false-alarms on double-ACh etc.

The actual US is a powerful signal coming from hypothalamus etc, with strong driving input to everything -- it is entirely plausible that it has different effects, and it just makes sense to assume that it is available in BLA and VS to modulate learning.

• Add a USLayer emer.LayerNames to BLA and Matrix, use this to directly get USActive status. Get rid of double-ACh logic.

• Also, SpkMax needs different logic: the time integration for CaSpkP is such that it carries over activity from prior trial long enough past the min cycles to cause false alarms -- need a separate integrator for SpkMax that only increments when past the min cycles.

[rcoreilly]

NoGo learning needs activity

This is a familiar situation from the PBWM model and general BG learning: if there is a negative DA outcome associated with a gating action, the Tr trace of Go pathway activity will capture appropriate negative dwts, but often there was no corresponding No pathway activity to generate a trace and get reinforced.

Thus, we need to do something at the time of Go gating to always engage some corresponding No units -- presumably those closest to firing. This was done in earlier models by having a separate competitive selection of NoGo units, with something always being selected..

Ideally, we have a rebound activation of No after Go has fired. Need to work on pcore dynamics for this.

[rcoreilly]

Gating often comes too late

To know if you should approach or not. This is likely a major source of residual errors. The action is generated within the first 100 msec often and the gating takes more than 100 to show up in PT..

Could have 2x theta cycles per action, and only take the action after the 2nd theta cycle. Could also speed up Forward if it selects that within the first theta cycle and it is gated, then just do that.

Confirmed with the TwoThetas flag that even with an explicit gating layer signal, it doesn't get the action 100% correct -- needs the two thetas..

[rcoreilly]

OFC cannot represent drive "desire" -- must reflect CS "reality"

For OFC to provide a useful VS gating signal, it has to reflect information about the current CS context, and NOT directly reflect the current Drive state. The Drive state represents what we want, but if that is presented to the VS as actually being present, then it will happily gate at the wrong time, thinking a matching CS is present for its current Drive input modulator. This was the source of erroneous gating when Drives -> OFC were present initially in BOA.

As a more general principle of organization, these kinds of strict "firewall" and "accounting" kinds of principles are key to identify and maintain throughout the model. The motivational system cannot be subject to (too much) delusion, without resulting in dysfunctional behavior. It needs just the right amount of delusion to get past the pain and pick itself off the floor every day, and cheating on the accounting only leads to bad decision making in the future..

[rcoreilly]

PT thalamic gating dynamics

Finally need to confront the long-postponed challenge of how to properly simulate the effect of BG -> thalamus gating signal on the PT neurons.

There is a bit of a catch-22 situation here: you need the PT -> thalamus activity to allow thalamus to get active when the BG inhibition is removed, but you also want thalamus -> PT to have a special effect on the PT neurons (uniquely turning on maintenance currents in the basic Rubicon maintenance case).

Biologically, the MD thalamic inputs are likely to have more of an effect on apical dendrites in layer 3 where NMDA maintenance channels are concentrated. In the axon neuron, we can drive NMDA channels via the MD input selectively -- this works!

Also, there is a diversity of PT neuron firing profiles, with some showing active maintenance and others only phasic firing at putative BG gating time -- need to sort that out eventually but for now we can have OFC and ACC just do the maintenance and not get to the other Vthal gating yet, which is likely to have more phasic effects + maintenance.

[rcoreilly]

Actually the diversity of PT dynamics is a pressing issue in OFC & ACC: there is no distinctive "I've decided to approach this CS" signal if the very same PT neurons need to be active in advance of gating.

A simple strategy is to route the super neurons directly into MD, as a proxy for the more transient PT population, and then have the actual PT layer only get active post-gating, representing the subset of active maintenance PT's.

CT can also provide some general MD activation but the CT reps in general are more variable and contextualized, so the main super neurons should provide a cleaner representation for what to maintain.

[rcoreilly]

Discrete high-threshold behavior is inherently brittle

The Rubicon model, which posits a discrete gating event of goal engagement (triggered by VS -> VP -> MD gating of OFC, ACC, dlPFC) and relies upon the stable bridging activity strategy, creates a significant modeling challenge because any such discrete, rare event in a neural system is inherently brittle. Most existing RL and other AI models avoid this problem by choosing actions stochastically on a moment-by-moment basis, driven by current state representations, with no longer term goals or plans. However, it is critical to not continue to avoid this challenge, but rather to tackle it head-on, with full awareness of the nature of the problems and ways to make it as robust as possible.

The source of the brittleness is fairly obvious at a general level: if the gating event doesn't happen or happens at the wrong time, then you either miss the opportunity or lock the system into a bad goal state that drives behavior over some period of time. Because there can only be one goal active at a time (in terms of shaping current proactive behavior in a coordinated way), an ill-timed gating event can trigger the loss of an important prior goal -- if you just keep updating all the time, nothing proactive and coordinated will ever emerge. Likewise, never engaging (characteristic of major depressive disorder), results in lack of proactive behavior entirely. The gambler by Kenny Rogers summarizes the situation well :)

You've got to know when to hold 'em, know when to fold 'em, Know when to walk away, know when to run.

(where 'em are goals, not cards)

The extensive experience with the PBWM BG-gating model shows that as a general-purpose working memory maintenance mechanism, gating is not very robust. At an abstract computational level, it is equivalent to a POMDP (partially-observable markov decision process) ToddNivCohen08, which is a notoriously badly-behaved beast, due to extreme combinatorial explosion. In PBWM, we had to have many "stripes" of parallel maintenance to amortize the brittleness in any one. Likewise, typical LSTMs operate with gates at a per unit level, not one gate controlling many units, achieving a very high level of parallelism and thus robustness.

Indeed, it is this brittleness of discrete gating that makes the CT / SRN / reservoir computing model more appealing as a basic working memory system: it is continuously updating and doesn't depend on discrete gating. But it also cannot bridge longer time gaps. Thus, the current framework that combines this continuous model with the discrete gating updates can give the best of both.

Here are some key properties of the biological system that help mitigate the challenge:

• Restricted learning that focuses on times of a small subset of known important outcomes (USs). This provides a "ground truth" to the learning of what is important to maintain, greatly limiting the search space and possibilities of discrete gating for "false alarms". The BOA model has wrestled with this, needing to introduce ACh-mediated modulation of learning at US outcomes in the VS and BLA to prevent ongoing learning from "corrupting" the "purity" of the US-focused learning. Once learning in the gating system starts over-generalizing, it is hard to recover.

• Strong contextualization: the internal drive state (hypothalamus, insula) strongly contextualizes what is considered relevant at the current time, further focusing on specifically relevant cues for the current internal state.

• Engaged protection: a maintained goal state is protected against random replacement, by actively suppressing the gating mechanism. This is not yet implemented in BOA but was in earlier PBWM models, and needs to be there, also for the following point.

• Strict accounting: the Rubicon model holds that every goal engaged episode has a clear moment of accounting when the goal state becomes disengaged, where all the ensuing costs are properly attributed, even as they were relatively downplayed during the episode itself. This ensures that subsequent goal engagement is properly valued, and explains our tendency toward procrastination. It also requires the engaged protection mechanism, so that goals are only disengaged when successful (the US itself drives the disengagement and accounting), or abandoned (accompanied by a strong cost factor which we experience as disappointment and sadness).

Note that all of these factors only work with a restricted small subset of a priori known important things. This suggests that discrete BG gating and the stable bridging strategy may only work well in this one case of goal-based gating. Thus, all further expanded, flexible use of such a strategy in humans must somehow ride on top of this core goal-driven foundation. Thus, getting this goal-based system right, with all its special case logic, may be critical for doing anything coherent in a neural system over time.

[rcoreilly]

Foregoing goals is not a good option

One plausible reaction to the above, and this whole effort, is to continue the current RL strategy and ignore the idea of actively selecting and maintaining goals. Here's some arguments against that idea, starting with an analysis of how RL models work in the first place without some kind of goal-like construct.

The standard TD Q-learning RL model bases action choices on a lookup table (or function approximator nnet) that maps state + action -> value. The values are accumulated over experience, and action selection is a softmax over action values at each state. This avoids any need for goal maintenance, by assuming a single fixed set of values over states and actions. The limitations of such a memorization-based "model free" approach are severe, and it is surprising that the field has proceeded so long without addressing them (indicating the difficulty of doing so; note: I am a bit behind the times on the very latest SOA, so there may be new stuff I don't know about). There is much discussion in the cog-neuro oriented literature of how the PFC implements a "model based" alternative, but there aren't any general successful models thereof (again, caveat above).

Even if you allow for a highly abstracted notion of state (as opposed to the simplistic grid worlds often used), the idea of a fixed value function doesn't make any sense in the real world. Everything depends on your internal context: a thirsty animal has a very different value function than a hungry, scared, or horny one, etc. It is not even a good "baseline" model -- states derived from some kind of perceptual input of the world are just not the right level of representation for values-driving-actions. Value is signaled by a very small subset of key perceptual inputs (CSs), and most other perceptual inputs are only useful for shaping action in a "pragmatic" value-independent manner -- one patch of ground or another doesn't have any specific value per se -- it is just a means to different ends depending on what you're trying to do.

Thus, the entire framework of standard RL models is inflexible and leads to overly rigid behavior because the models have no idea what they are actually trying to do -- they are just memorizing values associated with states. This can be done very easily by existing learning techniques, without having to tackle the difficult issues listed above, but it is a one-way street to nowhere.

The model-based framework in RL supposes that we have an internal model of the world and simulate that to determine actions. Aside from the extreme difficulty in actually doing that sufficiently well over a behaviorally-relevant timescale, in any kind of reasonably complex world, it is also typically applied within a monolithic non-goal-driven framework. Instead of proactively engaging a goal state that then drives coordinated actions over time, the model is invoked on the fly at each point of decision. One example of such a model is a successor representation (predictive map), which tells you which states are likely to follow from the current ones -- instead of baking value into the states, you can project states forward in time and try to anticipate value from those future states (which still need to map to value somehow).

The Rubicon goal-driven framework instead posits that action selection is goal driven "from the start", not just reactively at points of indecision. To be clear, all manner of actions can be taken "opportunistically" depending on local context, and in an exploratory manner, but the value-driven aspect of instrumental action -- action directed toward an outcome -- is informed and coordinated proactively by a maintained goal state.

The basic premise here is that proactive, coordinated action over time is going to be sufficiently more effective than reactive or state-value memorized action, to justify all the additional "hardware" and difficulty required. It should be pretty obvious that proactive, coordinated action is going to be more effective than reactive, state-based action -- it probably doesn't require much further justification.

Nevertheless, some salient examples include: squirrels burying nuts; birds building nests; humans investing in all manner of future-oriented outcomes, some with incredibly long payback times: farming, education (especially grad school!), retirement savings, etc. People growing up in challenging socioeconomic situations typically don't engage in as much proactive goal-driven behavior as those in wealthier circumstances, furthering the existing disparities. These examples are at much longer timescales than the actively-maintained Rubicon goal-engagement scope (and thus speak to the need for additional longer timescale mechanisms), but the same principles apply.

[rcoreilly]

Rubicon gating logic diagram

[rcoreilly]

Novelty driven default approach behavior

In the initial BOA model, the scaffolding for initial exploration of CSs was driven by the instinct actions, but these did not result in a VS BG "Go" gating signal, and thus there was nothing for the dopamine to respond to.

It is more plausible that initial exploration is driven by a bias to approach novel stimuli, and novelty-driven dopamine etc is widely documented. Here's some of the motivation and background for the "Novelty Value" component added to the 2010 version of the PVLV model:

For visual stimuli, the NV component is interpreted biologically in terms of known direct projections from the superior colliculus (SC) to midbrain dopaminergic cells (VTA/ SNc) that have been shown to trigger DA cell firing that attenuates with repeated exposure (Comoli et al., 2003; Dommett et al., 2005; Humphries et al., 2006; Redgrave et al., 1999), or for hippocampal detection of novel stimulus features (Lisman and Grace, 2005).

This pattern is consistent with empirical data showing that DA cells respond phasically to the occurrence of novel stimuli (Horvitz, 2000; Kakade and Dayan, 2002). A recent study showed that preferences for novelty-based choice are accompanied by striatal activations modeled by adding a novelty term to the value of each action, such that reward value and novelty contribute additively to action selection (Wittmann et al., 2008).

The functional role of NV-derived burst firing in PVLV is similar to that proposed by Kakade and Dayan (2002), encouraging a kind of computational exploration with regard to the maintenance of new stimuli which might turn out to be rewarding. In the context of working memory this implies that potential CSs are more likely to be updated and maintained in working memory when they are novel, thereby speeding the discovery of appropriate working memory representations that can contribute to favorable performance. In addition to this simple ‘novelty bonus,’ other mechanisms, likely dependent on more elaborate prefrontal-dependent processes, may contribute to exploration based on uncertainty about whether a given action might lead to better outcomes than have been experienced in the past (Dayan and Sejnowski, 1996; Frank et al., 2009).

Across a range of papers summarized below, BG (VS) and BLA areas consistently show up as having strong novelty coding, and, interestingly, the hippocampus does not. These novelty signals end up in medial frontal areas ultimately, but likely this is reflecting the input from VS and BLA.

Computationally, given that the CS -> US coding that drives VS gating arrives from the BLA, it would be convenient if BLA itself provided the novelty signal. A simple version of this would be that "US" associated with the CS is "novelty" itself, and by default, all novel CSs end up activating that US. Then, as an actual CS -> US association is learned in the BLA, it ends up out-competing the initial novelty-driven CS -> US association.

It is also important, as well-documented by the latent inhibition phenomenon, that CSs not associated with any other US end up producing a decreasing novelty response over time, which has the effect of making it more difficult to then associate them with something positive.

[rcoreilly]

GhazizadehFakharianAminiEtAl20

• Cortex: max novelty (NV) coding in TEav (anterior ventral), 36r (perirhinal) relative to Good-Bad value coding (GB).
• Subcortical: Amyg, VS show strong NV and GB coding.

ZhangBromberg-MartinSogukpinarEtAl22

Systematically compares novelty vs. surprise vs. recency. Novelty shows strongest response, then surprise, then recency, and they are all correlated. The strongest responding areas are:

• anterior ventral medial temporal cortex (AVMTC)
• 9 / 46 v -- shows weak surprise
• amygdala
• basal forebrain (Ach)

WilsonRolls93

All about amygdala novelty / familiarity responding in primates:

• The responses of three groups of neurons reflected different forms of memory. One group (n = 10) responded maximally to novel stimuli and signif- icantly less so to the same stimuli when they were famil- iar.
• Two other groups responded to the sight of particular categories of familiar stimuli: to foods (n = 6) or to faces (n = 10).
• all located in BLA

MisslinRopartz81

• old study found amygdala lesions deminished species-typical responses to novel stimuli. But was interpreted in terms of defensive responding -- key fact that rats show initial fear response to novelty, then some curiosity -- so our approach model may not be ethologically correct. but this is just a difference in instinctual response patterns, not about the basic logic overall.

WeierichWrightNegreiraEtAl10

fMRI study showing strong novelty responding in amygdala..

MormannKornblithQuirogaEtAl08

One of many papers from Fried lab on human recordings, showing mainly latency and selectivity properties of amygdala vs. other areas..

ChandranYiannakasKayyalEtAl22

Recent arxiv paper on conditioned taste aversion pathways between insula and amygdala, showing changes in excitatbility -- decreasing with familiarity...

[rcoreilly]

ACh acetylcholine cholinergic lit review (Extinction especially)

It is clear from various sources that ACh signals reward salience: non-RPE (non-prediction discounted), positively-rectified (fires for both positive and negative valences) signal for both US and CS onset. As such, it is a valuable modulator of both BG gating (via the cholinergic interneurons (CIN) in the BG) and learning in the BOA system. There is extensive cholinergic modulation of the BLA and medial PFC areas.

One critical outstanding question: does ACh fire for the absence of an expected US? If so, it can continue to be a useful learning modulator during extinction learning as well as acquisition. If not, things get more complicated.

SturgillHegedusLiEtAl20

Sturgill, J. F., Hegedus, P., Li, S. J., Chevy, Q., Siebels, A., Jing, M., Li, Y., Hangya, B., & Kepecs, A. (2020). Basal forebrain-derived acetylcholine encodes valence-free reinforcement prediction error (p. 2020.02.17.953141). bioRxiv. https://doi.org/10.1101/2020.02.17.953141

Summary: Key review of reward salience function of ACh and distinction between different types of neurons within the basal forebrain, which have different response properties.

CrimminsLingawiChiengEtAl23

Crimmins, B. E., Lingawi, N. W., Chieng, B. C., Leung, B. K., Maren, S., & Laurent, V. (2023). Basal forebrain cholinergic signaling in the basolateral amygdala promotes strength and durability of fear memories. Neuropsychopharmacology, 48(4), Article 4. https://doi.org/10.1038/s41386-022-01427-w

Summary: NBM (nucleus basalis of Meynert) facilitates acquisition and, possibly extinction learning. HDB (horizontal limb of diagonal band of broca) appears to only facilitate acquisition -- everything is weaker without it.

These results agree with the literature [29–31] and indicate that although NBM and HDB cholinergic neurons differ in their anatomical targets, both innervate the BLA.

The behavioral protocol (Fig. 2A) started with fear conditioning during which an auditory conditioned stimulus (CS) was paired with a foot shock unconditioned stimulus (US) in context A. Fear to the CS was then extinguished in a distinct context B and was later tested in that context. Next, two retrieval tests were given: one in which the CS was presented in context A and the other where it was presented in context B. These tests allowed assessing the capacity of the fear memory to renew after extinction. Indeed, an extinguished fear memory typically fails to elicit fear in its extinction context (e.g., context B), but it does so outside of that context (e.g., context A) [10–13].

AchNBM→BLA silencing reduced freezing during the post-extinction test relative to the control rats (F1,31 = 9.76, p < 0.01), which displayed equivalent freezing (smallest p = 0.58). The silencing also abolished fear renewal (F1,31 = 10.44, p < 0.01).

By contrast, rats with AchNBM→BLA silencing froze very little in both contexts (p = 0.60).

Together, these data show that AchNBM→BLA silencing during fear conditioning enhanced the loss of fear produced by extinction, as fear was reduced in the post-extinction test. They also show that this silencing prevented the renewal of fear typically observed after extinction. By contrast, AchHDB→BLA silencing during fear conditioning had no effect across extinction, the post-extinction test, and the retrieval tests.

These results therefore indicate that AchNBM→BLA silencing during fear conditioning weakens the acquisition of a fear memory and that the capacity to observe this weakening depends on the strength of the original memory.

Together, these results indicate that AchNBM→BLA and AchHDB→BLA silencing during extinction enhanced the loss of fear produced by extinction, as fear was reduced in the post-extinction test. They also dissociate the long- term effects of the two silencing, as only AchHDB→BLA silencing was found to abolish renewal of the extinguished fear memory.

We propose that these mechanisms involve cholinergic regulation of BLA activity by the basal forebrain and that in their absence, a fear memory is forgotten instead of being inhibited across extinction. That is, we take the lack of renewal in our experiments as evidence of forgetting. This proposal is more suitable to the results obtained following AchNBM→BLA silencing during fear conditioning, as we demonstrated that this silencing directly interferes with fear memory formation. By contrast, we cannot exclude the possibility that the absence of renewal produced by AchHDB→BLA silencing during fear extinction reflected an enhancement of extinction.

WilsonFadel17

Wilson, M. A., & Fadel, J. R. (2017). Cholinergic regulation of fear learning and extinction. Journal of Neuroscience Research, 95(3), 836–852. https://doi.org/10.1002/jnr.23840

Summary: Has a lot of review info on basic anatomy etc, and shows clear evidence of ACh involvement in extinction learning.

Mammalian forebrain cholinergic neurons have been broadly grouped into four clusters with the nomenclature of Mesulam and colleagues (1983b). The first three sub- groups (Ch1–3) consist of neurons in the medial septum (MS) and vertical and horizontal limbs of the diagonal band of Broca (DBB) and provide cholinergic innervation of the hippocampus and olfactory bulb. The fourth group (Ch4) of basal forebrain cholinergic neurons includes a loosely clustered arrangement of cholinergic neurons located in the nucleus basalis magnocellularis (nBM) and rostrally contiguous ventral pallidum/substantia innominata (Mesulam et al., 1983a,b). These neurons project diffusely to all layers and areas of the neocortical mantle (Bigl et al., 1982; Struble et al., 1986), in which the primary physiological effect of ACh is to modulate the response of pyramidal cells to other, particularly glutamatergic, cortical input (McCormick, 1993; Metherate and Ashe, 1993)

As the rostralmost component of the cortical mantle, the PFC derives its major cholinergic innervation from the Ch4 group of basal forebrain neurons. Some reports in rodents have suggested that a smaller portion of cholinergic innervation of the medial PFC additionally arises from the DBB and MS as well as from the pedunculopontine and laterodorsal tegmental areas of the brainstem (Lamour et al., 1984; Eckenstein et al., 1988).

Studies using immunotoxic lesions of cholinergic neurons in these areas similarly have suggested that cholinergic projections into the prelimbic (32) and infralimbic cortex might be segregated throughout this continuum. These studies using immunotoxic lesions demonstrated significant loss of acetylcholinesterase staining in both hippocampus and PFC following injections into the basal forebrain or MS/ vertical DBB. In contrast, targeting the nBM/horizontal DBB affected predominantly cholinergic projections to the prelimbic cortex but spared hippocampal projections (Knox and Keller, 2015).

BFCS is also a primary source of cholinergic innervation of subcortical limbic structures, such as the hippocampus and the amygdala. In fact, cholinergic projections from the Ch4 subgroup form the densest source of neuromodulatory inputs to the amygdala, especially the BLA complex (Muller et al., 2011)

In addition, cholinergic inputs from the BFCS into the BLA express only M2 receptors, suggesting a role in autoreceptor control of ACh release in this region, whereas GABAergic projections from the BFCS express both M1 and M2 mAChR subtypes (Muller et al., 2013, 2016). This has similarly been demonstrated in hippocam- pus and PFC, in which cholinergic inputs from the BFCS and the MS have presynaptic M2 receptors (Rouse et al., 1999; Volpicelli and Levey, 2004). The functional role of mAChRs on different cell types in the amygdala was sim- ilarly suggested by using optogenetic stimulation of cho- linergic fibers in the BLA, which influenced both interneurons and pyramidal cells. Although the physiolog- ical effects differed according to cell type and stimulation parameters, the net effect of optogenetic activation was enhancement of the “signal-to-noise ratio,” similar to cholinergic modulation of neocortical pyramidal cells (Unal et al., 2015).

Although cholinergic innervation of the cortical mantle is diffuse, reciprocal excitatory inputs from cortex back to the basal forebrain derive disproportionately from limbic- and paralimbic-associated areas, including pre- frontal and agranular insular regions (Carnes et al., 1990; Zaborszky et al., 1997; Mesulam, 2013). Additional major sources of inputs to the basal forebrain derive from the nucleus accumbens (Mogenson et al., 1983; Zaborszky and Cullinan, 1992) and parts of the amygdala (Grove, 1988), although inputs from the central nucleus may be restricted to a caudal part of the basal forebrain choliner- gic system (Gastard et al., 2002). The Ch4 neurons also receive input from brainstem monoamines, including constituents of the ascending reticular activating system and its rostral extension in the ventral tegmental area (Jones and Cuello, 1989; Zaborszky, 1989; Smiley et al., 1999) and the hypothalamus (Zaborszky and Cullinan, 1989).

In support of this notion, several microdialysis stud- ies have demonstrated an association between ACh release in the hippocampus, PFC, and amygdala and associative learning responses, although only a few studies have spe- cifically examined ACh efflux during Pavlovian fear con- ditioning or extinction. Increases in ACh efflux are seen during fear conditioning procedures, and, in hippocam- pus, unpaired shocks and tones led to higher efflux than tones paired with shocks (Nail-Boucherie et al., 2000; Calandreau et al., 2006). Exposure to a conditioned cue or context or objects in a novel object task also increases ACh in the hippocampus and/or PFC (Acquas et al., 1996; Nail-Boucherie et al., 2000; Stanley et al., 2012), and ACh released in the PFC is increased during extinc- tion of an operant task (Izaki et al., 2001).

Recent studies by Jiang and coworkers (2016) with pharmacological and optogenetic techniques to activate or inhibit cholinergic inputs into the BLA during training support this notion. Coadministration of nAChR and mAChR antagonists into the BLA reduced freezing to the CS and enhanced extinction (Jiang et al., 2016). Optogenetic activation of the BFCS failed to change cue- induced freezing during the retention trial but attenuated extinction training, whereas optogenetic inhibition decreased freezing during training and cue presentation 24 hr later (Jiang et al., 2016).

In addition, immunotoxic lesions of the BFCS either preconditioning or postconditioning did not significantly diminish either acquisition or expression of contextual freezing, although pretesting lesions did significantly reduce ultrasonic vocalizations after return to the context (Frick et al., 2004). Immunotoxic lesions of the MS that reduced cholinergic inputs to the hippocampus failed to shift acquisition of contextual fear responses or modify cFos activation of CA1 neurons associated with that response. In contrast, these MS lesions specifically attenuated extinction of contextual fear responses and the induction of ERK-positive neurons in the dorsal hippo- campus associated with extinction processes (Tronson et al., 2009).

The authors speculate that cholinergic regulation of pyramidal neuron outputs via disinhibition is seen in other areas with microcircuits similar to those in the cortex involving parvalbumin interneurons, such as BLA (Letzkus et al., 2015).

In the PFC, activation of mAChRs increases excit- ability of infralimbic neurons via decreased M-type potas- sium currents and potassium channel AHPs, and these changes are linked to modulating extinction recall (Santini and Porter, 2010; Santini et al., 2012). Activation of mAChRs in prelimbic cortex by carbachol causes both acute and long-term depression (LTD) of synaptic efficacy from hippocampal but not cortical inputs. The effects of carbachol required activation of the hippocampal inputs for induction of LTD and appeared to be mediated via M2 mAChRs (both presynaptic and postsynaptic) in this region (Wang and Yuan, 2009).

[rcoreilly]

Lateral Habenula

ArakiMcGeerKimura88

Araki, M., McGeer, P. L., & Kimura, H. (1988). The efferent projections of the rat lateral habenular nucleus revealed by the PHA-L anterograde tracing method. Brain Research, 441(1), 319–330. https://doi.org/10.1016/0006-8993(88)91410-2

Summary: clearly shows projections to Substantia innominata (SI -- part of NBM), HDB cholinergic nuclei, in addition to RMTg and MD. Also main inputs are from VP.

Labeled fibers emerging from the LHb followed a ventral direction through either the rostral part of the mediodorsal thalamic nucleus or the fasciculus retroflexus. Those going through the mediodorsal thalamic nucleus seemed to extend ros- trally, providing ramifying axonal collaterals to the central medial thalamic nucleus. These fibers con- tinued to extend ventrally through the ventromedial thalamic nucleus and the zona incerta and finally merged with the medial forebrain bundle (Fig. 2F). The major portion of LHb efferents travelled through the peripheral portions of the fasciculus retroflexus and in the boundary regions of the para- fascicular thalamic nucleus (Fig. 3A). Many labeled axons branched out from this fiber bundle to caudal parts of the mediodorsal thalamic nucleus (Fig. 2G). These axonal fibers passed through the fields of Forel from Forel's Hi to H_, (Fig. 2G. H) and then also entered into the medial forebrain bundle. This bun- dle appeared to carry the greater part of rostrally di- rected fibers.

Rostraily oriented fibers provided ramifying col- laterals to the lateral hypothalamic area (Fig. 3B), and extended to basal forebrain structures (Fig. 2D-F). Here labeled axons were distributed to the ventral pallidum and substantia innominata, and to nuclei of the horizontal and vertical limbs of the diag- onal band (Fig. 2A-C). Onlv a few labeled axons were found in the lateral preoptic area (Fig. 2C).

Fibers passing in the fasciculus retroflexus extended toward the ventral tegmental area (VTA) (Fig. 2H, I). These caudally oriented fi- ber bundles seemed to be the major efferents of the LHb. A few labeled axons turned rostrally to join in the medial forebrain bundle. In addition, a small number of fibers extended laterally to the pars com- pacta of the substantia nigra (SNC) and also to the zone between the SNC and the medial lemniscus (Fig. 2J).

The descending fiber bundles around the VTA spread into various midbrain regions such as the ros- tral linear nucleus raphe, interfascicular nucleus and retrorubral region (Fig. 2J, K). Some of these fibeis turned dorsally along the midline and entered into the central grey to terminate in the ventrolateral por- tion (Fig. 2J). The rest of the fibers were scattered in such areas as the dorsal raphe, deep mesencephalic nucleus, median raphe, oral pontine reticular forma- tion and pedunculopontine tegmental nucleus (Figs.
2L-N and 3C,D). The most caudal fibers were situ- ated in the laterodorsal tegmental nucleus and raphe pontis nucleus (Fig. 2N).

Sutherland82

Sutherland, R. J. (1982). The dorsal diencephalic conduction system: A review of the anatomy and functions of the habenular complex. Neuroscience & Biobehavioral Reviews, 6(1), 1–13. https://doi.org/10.1016/0149-7634(82)90003-3

Might LHb itself have ACh neurons?? Generally consistent with Araki et al.

The entopeduncular nucleus, which is the non-primate homologue of the internal segment of the globus pallidus, has a massive projection to the lateral habenular nucleus. This pallido-habenular pathway has been demonstrated using an autoradiographic tracing method in cats [76], using HRP tracing in rats [32] and monkeys [85] and electrophysiologi- cally in cats [47]. In rats, virtually every entopeduncular neuron projects to the lateral habenula, raising the possibility that these fibers are collaterals of the pallido-thalamic path- way. The entopeduncular projection appears to be topo- graphically organized within the lateral habenular nucleus. Neurons of the lateral part of the entopeduncular nucleus project to the lateral part of the lateral habenula and medial entopeduncular neurons project to medial neurons in the lat- eral habenula.

Compared to the rat, there are fewer pallidal neurons in squirrel monkey that project to the habenula [85]. The largest number of neurons that do contribute to the pallido- habenular connection in monkeys is primarily located within the rostral pole of the internal segment of globus pallidus.

There are a few lateral habenular afferents from the septal area, originating in the nucleus of the diagonal band, lateral septal nucleus, and nucleus accumbens [20,32]. These septal regions are probably the only forebrain areas that have con- nections with both medial and lateral habenular nuclei. Au- toradiographic and HRP tracing have demonstrated a projection to the lateral habenula from the nucleus basalis [32,119] A portion of the substantia innominata, located lat- eral to the nucleus of the diagonal band and ventrolateral to the nucleus accumbens, projects to the lateral habenula. This area in the substantia innominata has been interpreted as being "ventral pallidum", since it appears to be a rostral extension of the globus pallidus [32,119].

In terms of the origin of afferent pathways, the medial and lateral nuclei are very different. Only two substantial projec- tions provide input to both nuclei: the fibers from the nucleus of the diagonal band and the median raphe.

A decline of choline acetyltransferase and acetylcholinesterase in the medial habenular nucleus has been reported following placement of a knife cut between medial and lateral habenula [18]. This suggests the presence of a projection from the cholinergic cell bodies of the lateral nucleus [86] to the medial nucleus.

HikosakaSesackLecourtierEtAl08

Hikosaka, O., Sesack, S. R., Lecourtier, L., & Shepard, P. D. (2008). Habenula: Crossroad between the basal ganglia and the limbic system. Journal of Neuroscience, 28(46), 11825–11829. https://doi.org/10.1523/JNEUROSCI.3463-08.2008

Good overall review, citing Araki et al, confirming ACh projections.

MathisLecourtier17

Mathis, V., & Lecourtier, L. (2017). Role of the lateral habenula in memory through online processing of information. Pharmacology Biochemistry and Behavior, 162, 69–78. https://doi.org/10.1016/j.pbb.2017.07.004

• LHb -> MD exists, is key in PVLV / BOA for driving reset of goal reps after giving up.
• also LHb -> Septum

The LHb receives direct projections from the mPFC (Chiba et al., 2001; Kim and Lee, 2012; Vertes, 2006), the medial septal area (Meibach and Siegel, 1977), and is directly or indirectly connected with the entire basal ganglia (Hong and Hikosaka, 2013). While inputs from the basal ganglia, and more precisely from the monkey's internal portion of the globus pallidus (the rodents' entopeduncular nucleus), are more parti- cularly involved in the signalling of error-prediction (Hikosaka, 2010; Matsumoto and Hikosaka, 2007, 2009), Bromberg-Martin and Hikosaka (2011) further proposed that the information transmitted by the basal ganglia to the LHb serves the latter to participate in the evaluation of the immediate consequences of an action and to anticipate the occur- rence of a reward or a punishment; this appears particularly interesting because it positions the LHb as a link between our past experiences (memories) and the ongoing situation.

Finally, the LHb projects to the medio- dorsal nucleus of the thalamus (Araki et al., 1988), particularly in- volved in memory processes (Wolff et al., 2015).

Moreover, the LHb has been shown to interact with the key neu- rotransmitter systems involved in memory processes. It participates to the modulation of the activity of midbrain dopamine neurons, and of dopamine release in the mPFC and the basal ganglia (Brown and Shepard, 2016; Lecourtier et al., 2008); it modulates the activity of raphe serotonergic neurons (Wang and Aghajanian, 1977), as well as serotonin transmission (Kalén et al., 1990). Montalbano et al. (1991) and Zagami et al. (1995) further proposed that the LHb influences hippocampal cellular activity through an action on the raphe. Finally, the LHb modulates the septo-hippocampal cholinergic pathway (Nilsson et al., 1990).

Also, Stamatakis and Stuber (2012) have shown that acute optogenetic activation of LHb inputs projecting to the rostromedial tegmentum (RMTg) induces conditional avoidance; the fact that place avoidance lasted at least 7 days after the optogenetic manipulation further suggests that the LHb-RMTg pathway is involved in the expression of an emotional memory.

LavezziZahm11

Lavezzi, H. N., & Zahm, D. S. (2011). The mesopontine rostromedial tegmental nucleus: An integrative modulator of the reward system. Basal Ganglia, 1. http://www.ncbi.nlm.nih.gov/pubmed/22163100

• RMTg can inhibit PPTg -- direct blocking of PPTg from dips -- included in new equations!

This GABAergic midbrain structure is now thought to convert excitation of the LHb into an inhibitory influence on the VTA [36,43–45] through glutamatergic LHb neu- rons projecting to GABAergic RMTg neurons which, in turn, inhibit dopaminergic VTA/SNc neurons. These findings at the least suggest that the RMTg has a role as a modulator of reward and arousal, and this has been a sufficient stimulus to lead to further neuroanatomi- cal and functional investigation of the structure.

Other robust outputs from the RMTg occupy the medial part (pars dissipata) of the pedunculopontine tegmental nucleus (PPTg), periaqueductal gray, including the dorsal raphe nucleus (DR), and pontine and medullary reticular formation (Fig. 3C). Varyingly moderate to sparse labeling was observed in numerous other structures, including in the ventral pallidum, diagonal band com- plex, lateral preoptic region, thalamic midline and intralaminar cell groups, lateral hypothalamus, entopeduncular nucleus, parafasci- cular thalamic nucleus, deep mesencephalic nucleus, laterodorsal tegmental nucleus, locus coeruleus and subcoeruleus complex, and deep cerebellar nuclei.

The LHb increases neuronal firing in response to aversive stim- uli or omission of reward [27–31] and because LHb projections to the ventral tegmental area are mainly glutamatergic [32,33] they can inhibit dopamine neurons only via an inhibitory relay. The neuroanatomical evidence that the RMTg is such a mediator has been described. Functional data relevent to this idea follow.

[rcoreilly]

ACh gating of PFC for gating nonlinearity

Functionally, the PFC dynamic during gating is highly nonlinear and thus parameter sensitive. Given that we reliably have ACh at gating, and that it is known to modulate PFC excitability, in a pathway-specific fashion, it is likely that we can make gating more reliable.

The existing evidence shows strong ACh modulation of layer 3 NMDA active maintenance circuits.

A simple mechanism is that ACh is needed to enable activation of a new WM active state from thalamic gating -- without the ACh, existing NMDA currents can maintain, but nothing new can be activated. Mechanistically, ACh modulates K channels -- try that.

GalvinYangPaspalasEtAl20 (Arsten, Wang senior authors)

Galvin, V. C., Yang, S. T., Paspalas, C. D., Yang, Y., Jin, L. E., Datta, D., Morozov, Y. M., Lightbourne, T. C., Lowet, A. S., Rakic, P., Arnsten, A. F. T., & Wang, M. (2020). Muscarinic M1 receptors modulate working memory performance and activity via KCNQ potassium channels in the primate prefrontal cortex. Neuron, 106(4), 649-661.e4. https://doi.org/10.1016/j.neuron.2020.02.030

In classic excitatory glutama- tergic synapses, AMPA receptor (AMPAR) activation provides the critical membrane depolarization needed to eject magnesium and allow NMDAR activation. However, in dlPFC delay cells, AMPAR play a surprisingly minimal role, and ACh provides this critical de- polarization step via activation of nicotinic a7 receptors expressed within the postsynaptic density (PSD) on dendritic spines (Yang et al., 2013).

KCNQ2, KCNQ3, and KCNQ5 are found in the brain, where they can form heteromeric (KCNQ2/3, KCNQ3/5) as well as homomeric channels (Delmas and Brown, 2005). M1Rs can excite neural membranes by closing KCNQ channels via PIP2 signaling (Suh and Hille, 2002); thus, KCNQ channels are often referred to as ‘‘M channels.’’ This change in channel conductance occurs fast, with complete closure within 5–16 s of M1R stimula- tion (Suh and Hille, 2002; Suh et al., 2004). However, M1Rs can also hyperpolarize the membrane through downstream protein ki- nase C (PKC)-cyclic AMP (cAMP)-protein kinase A (PKA) signaling mechanisms that increase the open state of KCNQ2 channels (Iyengar, 1993). Thus, M1R-Gq activation can potentially influence WM circuits through two parallel but opposing signaling pathways (summarized in Figures 1E and 1F).

We found that all three isoforms were localized on spines and dendrites in layer III of the dlPFC. KCNQ5 was seen in spines, often near the PSD (Figures 6A–6C), as well as on dendrites (e.g., Figure 6C). KCNQ3 and KCNQ2 were seen at peri- and extra-synaptic locations on spines (Figures 6D, 6E, 6G, and 6H) and had prominent localiza- tion on dendrites (Figures 6F and 6I). As with M1Rs, there were also examples of KCNQ3 on dendrites with interneuron-like characteristics (Figure S8). Thus, KCNQ channels were in similar locations as those of M1Rs in dendritic trees and spines.

Here we show a mechanism for M1R modulation of PFC neuronal activity via control of the KCNQ potassium channel open state. We report that M1R and KCNQ channels are local- ized on dendrites and spines in layer III of the monkey dlPFC, the layer that contains the recurrent excitatory microcircuits needed for persistent firing. We found that local application of compounds that block M1R reduce delay cell firing and that application of compounds that stimulate M1R produce a quite narrow, inverted-U dose response in aging macaques, where low-dose M1R agonists or PAMs enhanced delay-related firing, but higher doses reduced delay-related firing. This inverted-U dose response is consistent with the effects of other neuromo- dulatory systems on delay cell activity in the PFC, such as dopa- mine (Vijayraghavan et al., 2007), norepinephrine (Wang et al., 2007), and cholinergic actions at nicotinic receptors (Yang et al., 2013). A variety of mechanisms can erode responses at high doses; e.g., by activating negative feedback mechanisms or producing nonspecific excitation or by engaging additional cell types, such as GABAergic interneurons (Arnsten and Wang, 2016). These data highlight the sensitivity of dlPFC cir- cuits compared with V1; for example, where nicotinic agonists show a linear dose response within individual neurons (Disney et al., 2007).

YangPaspalasJinEtAl13

Yang, Y., Paspalas, C. D., Jin, L. E., Picciotto, M. R., Arnsten, A. F. T., & Wang, M. (2013). Nicotinic α7 receptors enhance NMDA cognitive circuits in dorsolateral prefrontal cortex. Proceedings of the National Academy of Sciences, 110(29), 12078–12083. https://doi.org/10.1073/pnas.1307849110

The current study reveals that α7-nAChRs are positioned in the synapses of layer III dlPFC pyramidal cell circuits in primates, and that endogenous stimulation of these receptors has an es- sential role in the WM-related firing in NMDAR-mediated cir- cuits. In particular, the data establish that endogenous ACh stimulation of α7-nAChRs plays a remarkably specific role in enhancing persistent neuronal firing in the absence of sensory stimulation, and explain why depletion of ACh from primate dlPFC would have such devastating effects on spatial WM, producing deficits comparable to ablation of the cortex itself (8).

[rcoreilly]

The axon NeuroMod params already have AChDisInhib -- easy to get that working. May have some benefits.

However, ACh is released at each CS -- the critical gating signal is just the disinhibition from VSMatrix -- so just ACh disinhib is not actually as discriminative.

Probably the more important factor is having a bigger SNR on BG gating -- it is often only disinhibited for brief windows -- may need some longer time constants in the dynamics there.

Also, the existing ModGain on MD -> PT likely encompasses ACh effects already.

[rcoreilly]

STN and SKCa channels for pausing (sAHP)

The STNp pausing behavior looks like the following, as documented physiologically, and required for providing a BG gating window:

• (After inactivity), STNp fires a few (3 or so) spikes, via reciprocal GPeTA interactions, triggered by PFC excitatory input and ACh disinhibition.
• A long-lasting (~200msec) AHP ensues, during which spiking is not possible -- this is when BG gating can happen. This is putatively SKCa dependent, consistent with a number of other similar neuron types.
• After recovery from this AHP, there is regular spiking without further pausing -- until a reset, no further BG gating happens.
• Offset of ACh and PFC causes inactivity, which allows the SKCa pausing channels to recover.

The following mechanism is consistent with available data:

• The SKCa conductance happens relatively deterministically via Ca binding, with activation time constant of about 5-15 msec, and deactivation of around 50 msec. It has a sharp threshold: above a critical Ca level, it is essentially full on, and below this, it is essentially off, due to the power of 4 (4 independent binding sites). By itself, it doesn't have many of the necessary properties above: long AHP time, activation and recovery dynamics.

• SKCa is activated in many cases by intracellular Ca sources, which have slow dynamics, and allows for the necessary additional inactivation dynamics: There is a slow recharge process rebuilding intracellular Ca sources, which are released by spiking activity and prevented from recovering until spiking stops. When released, Ca has a slow decay / buffering time constant, keeping the SKCa channels open during the long 200 msec AHP. The intracellular store recovery time constant is slower than this 200 msec AHP pause, so it does not recover during that window, effectively inactivating the SKCa channels after the first pause. Only once there is a longer window of inactivation do the intracellular stores recover.

To model this, we have the following variables and dynamics:

• SKCaIn = intracellular store available for release. This increases in inverse proportion to spiking activity (indexed by CaSpkP -- rate is effectively 0 above some low threshold), up to a maximum level (rate proportional to Max - SKCaIn).
• SKCaR = released Ca from SKCaIn -- each spike drives transfer from SKCaIn -> SKCaR with a large rate constant, quickly emptying available stores.
• SKCaM= "m" Ca gating factor driven by SKCaR.

AdelmanMaylieSah12

Adelman, J. P., Maylie, J., & Sah, P. (2012). Small-conductance Ca2+-activated K+ channels: Form and function. Annual Review of Physiology, 74, 245–269. https://doi.org/10.1146/annurev-physiol-020911-153336

The SK channel family members show very similar Ca2+ gating profiles. They respond rapidly to Ca2+ , and the dose-response relationships show a Hill coefficient of 3–5, indicating cooperative, Ca2+-dependent gating. Fast application of saturating Ca2+ (10 μM) to inside-out patches shows that SK channels have activation time constants of 5–15 ms and deactivation time constants of ∼50 ms (25). Consistent with the macroscopic currents, single-channel analysis shows that SK channels are voltage independent and that their gating is well described by a model with four closed states and two open states, with Ca2+-dependent transitions between the closed states (24).

SK channels are activated by elevations in cytosolic Ca2+ from several different sources: Ca2+ influx via Cav channels (Figure 2a); Ca2+ influx via Ca2+-permeable agonist-gated ion channels, such as NMDARs and nicotinic acetylcholine receptors (nAChRs); and Ca2+ released from intracellular Ca2+ stores by generation of inositol trisphosphate (IP3) via G protein–coupled receptors, Ca2+- induced Ca2+ release (CICR), or a combination of the two. Cytosolic Ca2+ then returns to resting levels due to a combination of diffusion and buffering, uptake into organelles, and extrusion by exchangers in the plasma membrane (56). These mechanisms have different cellular distributions and kinetic properties, and as SK channel activity largely follows that of free Ca2+ near the channel, the kinetics of the macroscopic current thus depends on the source and location of Ca2+.

Ca2+ is also stored in the smooth endoplasmic reticulum and can be released by activation of IP3 receptors subsequent to IP3 generation by metabotropic receptor activation. Ca2+ ions act cooperatively to activate IP3 receptors, and IP3 generation can cooperatively interact with AP-induced Ca2+ rises to amplify Ca2+ release from intracellular stores (Figure 2b). This form of Ca2+ release activates SK channels (63, 64). Activation of IP3 receptors and the subsequent release of Ca2+ from intracellular stores are intrinsically slow, and the Ca2+ transients evoked by such release are in general an order of magnitude slower than those due to influx from extracellular sources. Thus, in midbrain dopamine neurons, both synaptically released glutamate (65) and ACh (66) evoke a slow inhibitory postsynaptic potential (IPSP) that results from mobilization of Ca2+ from IP3-sensitive intracellular stores and subsequent activation of SK channels. A similar cholinergic-mediated hyperpolarization is also present in cortical (63) and amygdala (48) pyramidal neurons. Interestingly, when coupled to Ca2+ influx via APs, the actions of IP3 on Ca2+ release can be amplified by CICR from IP3 receptors, and store-released Ca2+ can trigger an AHP that greatly inhibits AP generation (64).

[rcoreilly]

HallworthWilsonBevan03

Hallworth, N. E., Wilson, C. J., & Bevan, M. D. (2003). Apamin-Sensitive Small Conductance Calcium-Activated Potassium Channels, through their Selective Coupling to Voltage-Gated Calcium Channels, Are Critical Determinants of the Precision, Pace, and Pattern of Action Potential Generation in Rat Subthalamic Nucleus Neurons In Vitro. Journal of Neuroscience, 23(20), 7525–7542. https://doi.org/10.1523/JNEUROSCI.23-20-07525.2003

This looks like a definitive early reference for SKCa in STN -- huge amount of data but nothing obviously about pausing. Looking for updates in papers that cite this one:

ChuAthertonWokosinEtAl15

Chu, H.-Y., Atherton, J. F., Wokosin, D., Surmeier, D. J., & Bevan, M. D. (2015). Heterosynaptic regulation of external globus pallidus inputs to the subthalamic nucleus by the motor cortex. Neuron, 85(2), 364–376. https://doi.org/10.1016/j.neuron.2014.12.022

Looks important for slow homeostatic regulation of BG activity!

TachibanaKitaChikenEtAl08

Tachibana, Y., Kita, H., Chiken, S., Takada, M., & Nambu, A. (2008). Motor cortical control of internal pallidal activity through glutamatergic and GABAergic inputs in awake monkeys. European Journal of Neuroscience, 27(1), 238–253. https://doi.org/10.1111/j.1460-9568.2007.05990.x

Nice awake-behaving data consistent with overall pcore model.

BogaczMoraudAbdiEtAl16

Bogacz, R., Moraud, E. M., Abdi, A., Magill, P. J., & Baufreton, J. (2016). Properties of Neurons in External Globus Pallidus Can Support Optimal Action Selection. PLOS Computational Biology, 12(7), e1005004. https://doi.org/10.1371/journal.pcbi.1005004

Bayesian model of GPe / STN -- definitely need to get on top of this framework!

[rcoreilly]

PPTg temporal derivative mechanism

The PPTg temporal derivative could be modulated such that it is engaged when goal-engaged, but not during exploration. Need to know more about the connectivity and computations in this area.

Key points:

• "PPTg" is really 2 nuclei: PPN and LDT that are very similar anatomically, and are primarily ACh nuclei. LDT is "limbic" and PPN is "motor". We care mostly about LDT for BOA / goal engaged dynamics.
• LDT primary inputs are OFC, ACC, superior colliculus, and SN(r / c). It hardly gets any amygdala input -- very minor.
• Thus, it is clearly wrong to have "PPTg" be the direct output funnel for Amygdala! Instead, amygdala projects directly to VTA / SNc etc, and LDT provides a largely independent modulation of VTA based on its own separate inputs, that are all about goal state and sensory novelty via SC.
• In sum, LDT is not doing a temporal derivative, but rather a novelty filter on top of Amygdala inputs, which can look like a temporal derivative at CS onset, but is fundamentally more powerful and sensitive to goal state and direct sensory input via SC.
• Computationally, LDT is effectively the "explore" signal, up-regulating DA, BG, etc when something salient should be considered. TODO: add ACh modulation of thalamus too -- that is a major output and another key knob for modulating gating.
• LDT and PPN are likely the primary source drivers of ACh in the brain, providing driving input to BG CINs and likely other brainstem ACh -- they have the broadest input and directly or indirectly drive the other ACh areas.

Huerta-OcampoDautanGutEtAl21

Huerta-Ocampo, I., Dautan, D., Gut, N. K., Khan, B., & Mena-Segovia, J. (2021). Whole-brain mapping of monosynaptic inputs to midbrain cholinergic neurons. Scientific Reports, 11, 9055. https://doi.org/10.1038/s41598-021-88374-6

This looks like the current definitive study of inputs to PPTg (PPN, LDT). An absolute goldmine!

A recent study, for example, has shown that midbrain cholinergic neurons innervate the striatal complex and make direct connec- tions with striatal cholinergic interneurons (CINs); manipulations of either midbrain cholinergic neurons (i.e. PPN/LDT) or CINs have a similar influence on action strategy encoding, suggesting convergent functional roles between midbrain and striatal cholinergic systems (13)

Cholinergic neurons in the PPN and LDT form a continuum that extends from the caudal end of the substan- tia nigra (SN) to the central gray matter near the fourth ventricle. The efferent connectivity of these two structures is characterized by a topographical arrangement where PPN neurons preferentially innervate motor neuronal systems, whereas LDT neurons preferentially innervate limbic neuronal systems18. For example, cholinergic neurons of the PPN innervate the dopaminergic neurons of the substantia nigra pars compacta (SNc), the dorsal striatum and thalamic relay nuclei. In contrast, cholinergic neurons of the LDT innervate the ventral striatum and limbic thalamic nuclei. While not entirely segregated, both structures provide converging innervation to a certain subset of structures (i.e. the intralaminar thalamic nuclei, VTA), although detailed examination of the postsynaptic targets at the level of the VTA has revealed a divergent modulation over dopamine efferent circuits9, thus suggesting a largely unexplored level of functional selectivity between PPN and LDT cellular targets. Thus, the identification of the afferent systems to PPN and LDT cholinergic neurons is critical to integrate an input/ output connectivity map and to understand how their activity is regulated.

LDT is our primary region of interest for "limbic" control i.e., goal activation system.

Cortical inputs: single largest source to LDT, followed by superior colliculus (strong novelty input!), Raphe, and Gi (gigantocellular reticular nucleus -- some kind of major motor coordination brainstem area)

Our data show that the SC constitutes the main input structure to both the cholinergic PPN and LDT, in line with initial studies describing the connectivity between these structures21. Input neurons in the SC were distributed in superficial, intermediate and deep layers, where there is a convergence of signals from visual, auditory and somatosensory modalities. Superficial layers receive input primarily from the retina and visual cortex. Deep layers, in contrast, receive inputs from several sensory modali- ties, inputs from motor areas, and projections from areas that are not purely sensory or motor (22).

No inputs to the LDT were observed to arise in the dorsal striatum or the STN. These results suggest that basal ganglia inputs to the cholinergic midbrain are functionally segregated, where dorsal basal ganglia structures innervate PPN cholinergic neurons, whereas ventral basal ganglia structures innervate LDT cholinergic neurons, in line with the functional segregation observed in cortical and midbrain areas. Notably, the SN seems to be a point of con- vergence of afferents to both cholinergic regions.

Given the anatomical distribution of input neurons in the SN and DR, however, it is likely that both dopaminergic and serotonergic neurons directly innervate cholinergic neurons of the PPN and the LDT.

Nigral projections arising from the pars reticulata have been described in detail (27,28) and are known to be the major input to the PPN. These synapses have been electrophysiologically characterized in vitro and their stimulation induces inhibition of PPN neurons (29,30), which suggests that this projection is largely GABAergic in nature (31). A similar role although less prominent could be underlying the connectivity between the pars reticulata and the LDT likely in the context of behavioral activa- tion, as reported before (9).

Using conditional anterograde tracing of cholinergic neurons in ChAT::Cre rats, we have recently revealed the pattern of axonal innervation of PPN and LDT cholinergic neurons3,7,8. These previous studies together with our current dataset allow us to correlate the connectivity maps of midbrain cholinergic neurons. PPN cholinergic neurons maintain reciprocal connections with the basal ganglia, in particular the SN, GPe and striatum, the SC and motor nuclei of the brainstem, such as PnO, PnC and Gi. Interestingly, the connectivity with the cortex is also reciprocal, particularly with the motor and anterior cingulate cortices. In contrast, largely unidirectional efferent connectivity was observed with most thalamic nuclei, the ventral pallidum and amygdala. LDT cholinergic neurons maintain reciprocal connections with the inferior colliculus, DR, Gi, SN and ventral striatum. No LDT cholinergic axons were detected in the cor- tex, suggesting that this pathway is unidirectional.

Cholinergic neurons are distributed across the brain in anatomically defined cell clusters50. Among them, cholinergic neurons of the basal forebrain and CINs share important functional properties with the cholinergic neurons of the PPN and LDT. For example, cho- linergic neurons of the basal forebrain and the PPN/LDT increase their discharge during behavioral arousal51. Moreover, CINs and cholinergic PPN/LDT neurons were found to be critical for the modulation of striatal output neurons during action selection13. These studies raise the question of whether cholinergic signaling from these systems is functionally overlapping. Recent studies have provided the identification of presynaptic inputs to the cholinergic basal forebrain14,15 and CINs16,17. The comparison of the distribution of input neurons in these studies with the one reported here reveals that PPN/LDT cholinergic inputs are far more widely distributed across different brain regions than the inputs to any of these two other cholinergic groups.

DautanSouzaHuerta-OcampoEtAl16

Dautan, D., Souza, A. S., Huerta-Ocampo, I., Valencia, M., Assous, M., Witten, I. B., Deisseroth, K., Tepper, J. M., Bolam, J. P., Gerdjikov, T. V., & Mena-Segovia, J. (2016). Segregated cholinergic transmission modulates dopamine neurons integrated in distinct functional circuits. Nature Neuroscience, 19(8), Article 8. https://doi.org/10.1038/nn.4335

This is another key ref -- Dautan is the man..

TODO: fill in later.

OmelchenkoSesack05

Omelchenko, N., & Sesack, S. R. (2005). Laterodorsal tegmental projections to identified cell populations in the rat ventral tegmental area. The Journal of Comparative Neurology, 483(2), 217–235. https://doi.org/10.1002/cne.20417

This is one of the key refs from Mollick et al PVLV paper.

One major source of both excitatory and inhibitory inputs to VTA neurons arises from the mesopontine tegmentum (Hallanger and Wainer, 1988; Oakman et al., 1995; Charara et al., 1996). Historically, this region has been divided into the rostral pedunculopontine tegmentum (PPT) and the more caudal laterodorsal tegmentum (LDT), although these two areas are considerably homogeneous, according to morphological and physiological studies (Satoh et al., 1983; el Mansari et al., 1989; Steriade et al., 1990b; Clements et al., 1991; Kayama et al., 1992; Semba and Fibiger, 1992; Ford et al., 1995; Datta and Siwek, 2002). These regions correspond to the two main clusters of brainstem cholinergic neurons with ascending projections (Woolf and Butcher, 1986). Other cells in these areas contain glutamate or GABA alone or colocalized with acetylcholine (Clements et al., 1991; Lavoie and Parent, 1994b; Jia et al., 2003). Together, projections from the mesopontine tegmentum to the VTA appear to communicate information regarding environmental salience and behavioral state (Steriade et al., 1990a; Jones, 1991; Winn et al., 1997; Maloney et al., 1999; Miller et al., 2002).

In the primate, the PPT has been shown to provide both glutamate and GABA inputs to midbrain DA neurons (Lavoie and Parent, 1994a; Charara et al., 1996). In the rat, the midbrain projection from the mesopontine teg- mentum is organized topographically, such that the LDT and caudal PPT primarily innervate the VTA, whereas more rostral portions of the PPT innervate the more me- dial substantia nigra (SN) (Oakman et al., 1995).

This study presents the first evidence that the brain- stem LDT provides a prominent source of asymmetric synapses to mesoaccumbens DA neurons in the rat VTA, consistent with a major excitatory influence of this path- way. This innervation appears to be selective for DA cells and not GABA neurons that project to the NAc, and in this regard exhibits specificity that is the inverse of afferents from the PFC to these cell populations (Carr and Sesack, 2000b) (Fig. 9). Presumed excitatory synapses from the LDT also target VTA neurons that contribute to the me- soprefrontal projection. In this case, LDT afferents lack specificity and synapse onto both DA and GABA cell pop- ulations. The latter neurons were shown not to be inner- vated by the PFC (Carr and Sesack, 2000b), whereas the mesoprefrontal DA cell population appears to receive con- vergent input from both major sources of presumed exci- tatory drive to the VTA (Fig. 9).

This study also demonstrates that symmetric synaptic inputs from the LDT have a unique pattern of cellular targets in the VTA that distinguish them from asymmet- ric, presumed excitatory afferents coming from the LDT (Fig. 9) or PFC (Carr and Sesack, 2000b). With regard to DA neurons, symmetric, presumed inhibitory LDT inputs appear to be selective for mesoprefrontal cells. The fact that mesoaccumbens DA neurons were not found to re- ceive a similar input is consistent with neurochemical studies showing a dominant excitatory influence of the LDT on mesoaccumbens DA transmission (Forster and Blaha, 2000) and suggests that the putative inhibitory influence coming from the LDT is relatively selective for cortically projecting DA cells. At the same time, symmet- ric synapses from the LDT appear to target GABA neu- rons in relative preference to DA cells in the VTA, and this innervation is fairly equally distributed to both mesoac- cumbens and mesoprefrontal GABA neurons. The latter finding reveals a previously unknown, extrathalamic and potentially disinhibitory pathway communicating be- tween the LDT and the cortex.

The results, taken together with our previous report of PFC afferents to the VTA (Carr and Sesack, 2000b), indi- cate that specific DA and GABA cell populations are dif- ferentially regulated by distinct sources. Whether individ- ual VTA neurons receive convergent input from the PFC and LDT or from asymmetric and symmetric synapses of LDT axons requires further study.

[rcoreilly]

Superior Colliculus and novelty / onset sensory coding

DuttaGutfreund14

Dutta, A., & Gutfreund, Y. (2014). Saliency mapping in the optic tectum and its relationship to habituation. Frontiers in Integrative Neuroscience, 8. https://www.frontiersin.org/articles/10.3389/fnint.2014.00001

Shows stimulus-specific adaptation (SSA) in OT of owls, related to same in SC of mammals.

The emerging hypothesis is that the evolutionary role of the OT/SC is to sort stimuli based on saliency, select the most salient stimulus, and send this information to the appropriate brain regions to direct orient- ing movements, attention and autonomic responses (reviewed in Boehnke et al., 2011; Knudsen, 2011; Krauzlis et al., 2013)

Thus, SC inactivation specifically disrupts the ability to select the behaviorally relevant stimulus among other non-relevant stimuli.

Given its suggested role in habituation of the orienting response, we expect to find neural correlates of habituation in the OT/SC. In other words, we expect that neural representa- tion will be strongly modulated by the history of events in a way that suppresses the representation of background stimuli and rela- tively enhances the representation of odd stimuli.

It was found that most neurons in the OT responded more strongly to a low probability stimulus than to a high prob- ability stimulus (in the oddball protocol) or more strongly to the first stimulus, which is different from its past compared to the last stimulus (in the constant order protocol).

In all cases (ITD, ILD, intensity, and frequency), SSA was evident and developed rapidly after one trial (Figures 5A–C).

BoehnkeBergMarinoEtAl11

Boehnke, S. E., Berg, D. J., Marino, R. A., Baldi, P. F., Itti, L., & Munoz, D. P. (2011). Visual adaptation and novelty responses in the superior colliculus. European Journal of Neuroscience, 34(5), 766–779. https://doi.org/10.1111/j.1460-9568.2011.07805.x

Shows rapid adaptation in all types of SC neurons. Didn't seem to have a novel stimulus comparison condition unfortunately, but at least we know that showing the same stimulus repeatedly results in much lower neural responding in SC, producing a kind of short term visual novelty signal.

Reductions in response magnitude with repetition have been previ- ously observed in cortical areas, including V1 (Muller et al., 1999), V4 (Motter, 2006) and frontal eye fields (Mayo & Sommer, 2008), and also in single neurons of the SC (Goldberg & Wurtz, 1972; Woods & Frost, 1977) and multiunit activity of the superficial layers of the SC (Mayo & Sommer, 2008). In the context of an attentional cueing task, a repetition effect has been described in the SC (Robinson & Kertzman, 1995; Dorris et al., 2002; Bell et al., 2004; Fecteau et al., 2004) and lateral intraparietal area (Robinson et al., 1995). The present study is the first to systematically explore the repetition effect using long stimulus sequences studied across different cell types and layers in the SC, the first to report the increase in response onset latency with repetition in the SC, and the first to explore the mechanism of this response decrement through modeling and experimental manipulations (oddball stimuli).

Visual only cells with transient visual responses and no saccade-related activity (VT) tended to be located more superficially than the other classes of visually- responsive cells (see Fig. 1D). Thus VT cells were typically found in the upper superficial gray layer (e.g. SCS), where retinal Y-type cells terminate directly and indirectly through magnocellular lateral geniculate nucleus and V1 (May, 2006).

Visual-only cells that had sustained visual responses (VS) typically paused during saccadic eye movements (Fig 1A). Previously we have shown VS neurons to be sensitive to color signals whereas VT cells were not (White et al., 2009), suggesting parvocellular input. These features, along with a mean depth of about 900 μm (Fig. 1D), suggest they were located in the lower superficial layers. This area receives visual input from higher occipital and parietal areas (Graham et al., 1979; Tigges and Tigges, 1981). The VS neurons we identified were likely the same as “visual-tonic” neurons described previously (Li and Basso, 2008; McPeek and Keller, 2002). Finally, visuomotor cells with transient or sustained activity - VmT or VmS - were easily characterized because of their bursts of activity time-locked to saccades and their bursts of visual activity time locked to stimulus onset (Fig. 1A). Our sample of visuomotor neurons was by necessity biased to those with robust visual responses and were always found more than 1mm below the dorsal surface (Fig. 1D).

VT (visual, transient) cells don't activate in saccades, show rapid attenuation to repeated stimuli.

VT is dark blue, VS red. most decrease on 2nd repetition.

responses to unexpected brighter / dimmer and absent stimuli were consistent with raw adaptation model.

Alternatively, some portion of the response reduction may be affected locally in the SC by an increase in global inhibition from the basal ganglia. The intermediate layer of the SC projects the transient visual response monosynaptically to the substantia nigra compacta (Redgrave & Gurney, 2006), the response is then processed through the basal ganglia, and the substantia nigra pars reticulata (SNr) projects back to the intermediate layer of the SC (Hikosaka & Wurtz, 1983; Jiang et al., 2003) to modulate neuronal firing via GABAergic synapses (Isa et al., 1998; Kaneda et al., 2008). A visual transient that is not accompanied by a response or reward (as in our simple fixation task) could result in increased SNr inhibition with each repetition (or less disinhibition), and thus reduced subsequent responses.

WolfLintzCostabileEtAl15

Wolf, A. B., Lintz, M. J., Costabile, J. D., Thompson, J. A., Stubblefield, E. A., & Felsen, G. (2015). An integrative role for the superior colliculus in selecting targets for movements. Journal of Neurophysiology, 114(4), 2118–2131. https://doi.org/10.1152/jn.00262.2015

todo

KanedaIsa13

Kaneda, K., & Isa, T. (2013). GABAergic mechanisms for shaping transient visual responses in the mouse superior colliculus. Neuroscience, 235, 129–140. https://doi.org/10.1016/j.neuroscience.2012.12.061

todo

ComoliCoizetBoyesEtAl03

Comoli, E., Coizet, V., Boyes, J., Bolam, J. P., Canteras, N. S., Quirk, R. H., Overton, P. G., & Redgrave, P. (2003). A direct projection from superior colliculus to substantia nigra for detecting salient visual events. Nature Neuroscience, 6, 974–980. http://www.ncbi.nlm.nih.gov/pubmed/12925855

Title pretty much says it all!

However, because response latencies in cortical regions engaged in feature detection and object recognition are typically longer than those of dopaminergic neurons14, extra-foveal cortical processing is unlikely to provide an invariant short-latency route to the ventral midbrain. In contrast, visual response latencies of neurons in the midbrain SC, a region specialized for the detection of unexpected salient events15,16, are consistently less than those seen in dopaminergic neurons17,18.

[rcoreilly]

Novelty Mechanism

I just realized that the Inhib.FFPrv mechanism is too generic to result in SSA = stimulus-specific adaptation -- it is a single global inhib level for the whole layer, whereas you need something operating on the individual neurons.. So, some kind of neuron-specific delayed inhibition -- we have all these slower AHP mechanisms -- basically something like that?

[rcoreilly]

Extinction, BLANovelty, and BLANothing

There are 3 related cases that require 2 additional BLA layers:

• Extinction: When a CS starts out with a reliable CS -> US link, and then (or on some % of trials) it experiences CS -> no US.

  • This is handled by the BLAPosExtD2 layer, which learns from the DA dips, and is already constrained to a specific US by virtue of the maintained OFCPT activity. It inhibits the corresponding BLAPosAcqD1 acquisition coding, representing the balance between the two.

• BLANovelty: a novel CS that has no existing US association, can activate a VS "curiosity" gating event to engage approach. To be able to do this, it needs a matching BLA and Drive that can activate VS gating. BLANovelty layer has default GeBase activity and is inhibited by other US-specific BLA, so if a CS becomes associated with a specific US, it no longer activates the generic novelty.

• BLANothing: but if a novel CS never results in any interesting US outcome, then the BLA needs to learn this and have that learning block activity of BLANovelty -- i.e., effectively an Extinction layer for BLANovelty = BLANothing. The same D2 DA dip learning dynamics should apply.

New plan: just make the first pool in BLAPosAcqD1 the novelty pool, driven by a constant input pool of neurons, and then everything else is just aligned according to number of USs etc.

[rcoreilly]

Extinction and "paying the effort costs"

A key principle of the Rubicon framework is that effort costs must be properly accounted for and "paid" at the time of a US outcome (but otherwise relatively discounted during the goal engaged state).

But in terms of more fine-grained credit assignment, it should be clear that effort costs are associated directly with the dlPFC action plan, not the US outcome per se. You could have 2 different action plans for accomplishing the same US outcome, that vary in effort, and would want to select the lower effort plan, obviously. This is accomplished in part by having the dlPFC and ACCcost layers predict the Effort outcome (while OFC only predicts US and overall US value), and thus they should represent it and drive VS gating accordingly.

There are several related questions here:

1. How are effort costs represented and reflected in dopamine, when the goal US is accomplished? Is there an explicit BLANeg representation of the effort? If so, it would start to dilute (compete with) the BLA rep of the US itself -- but again the effort is attributable to the action plan, not the US outcome, so that wouldn't be appropriate. Likewise, the OFCval prediction should presumably be about the US value, not the effort discounted value. Ultimately, the DA is mostly about training the VS gating, which should be effort discounted, so we probably don't want to mess with that.

2. ... when the US is abandoned (extinction): there is an automatic negative DA from (0 outcome - positive expectation) but should accumulated effort also add additional negative DA? Again, this just affects the VS gating, so probably that makes sense.. But again, it shouldn't affect the BLA side of things, which is only US focused.

3. What about other kinds of negative outcomes encountered along the way while pursuing a goal, and negative US cases in general? How do these differ from effort? We have BLANeg representations for other kinds of negative outcomes (pain, etc) -- what is the difference from experiencing pain while pursuing a goal vs. pain as a discrete US outcome? Effort is multiplicatively discounting on US value -- is pain the same? Is there a sense in which negative USs are a completely separate factorization of the problem space, so they shouldn't "contaminate" the positive US outcomes associated with actual Drives? i.e., in the course of goal pursuit, you may see stimuli that predict negative outcomes along the way, and these are manged as "updates" to the existing plan in some way?

[rcoreilly]

BLA Extinction learning rule

For the novelty / curiosity case, it is not clear how to learn the BLAPosExtD2 weights to oppose the novelty activity. There is no learned expectation of a positive US outcome, so there isn't a corresponding negative DA from VSPatch. The only thing would be the effort cost per above, but this can be relatively small for plausible Pavlovian situations (e.g., the PVLV test model for one..).

For this one case, the most sensible rule would be to learn at the time of the US in proportion to the activity in the corresponding BLA Acq pool, in proportion to the inverse of the raw US outcome itself. I.e., if there is no US, then you strongly learn to extinguish the associated BLA rep.

If somehow we could arrange for the BLA Ext pool to get activated for the absence of the US in the same way that the Acq pool (naturally) gets activated for the presence of the US, that would do the trick. For the novelty case, the absence of any other US would have to activate the novelty Ext pool. Then, we can use the same learning rule as BLA Acq, based on the delta activation! That is a major bonus -- currently we're having to use a different learning rule for Ext vs. Acq.

The other major advantage of this learning dynamic is that it makes it fully independent of the VSPatch side of things: the BLA can learn directly based on US outcomes, instead of relying on the VSPatch to first acquire an expectation, and then continue to generate negative DA long enough for BLA to extinguish -- this was a major source of parameter sensitivity in the Mollick et al PVLV model -- if PV (VSPatch) extinguishes too quickly, LV (BLA) is left with residual CS firing.

As usual, we have the challenge of detecting the absence of something as an event unto itself. So we still need at least the LHb to decide when to give up. The new logic is that LHb drives ACh in addition to DA dips, and also VS / MD gating directly. For the pure latent inhibition novel CS pavlovian case (and also just general robustness), we also needed to add a Max timeout (in the Effort tracking system) that decides to give up after sufficient effort (time) has been expended without any outcome. This can potentially interact with the time integral of VSPatch dipping -- i.e., if there has been even a minimal level of VSPatch unfulfilled expectation, the timeout is significantly shortened -- just long enough to allow for a bit of premature expectations. Also, the Max timeout can be shorter for novelty-triggered "larks" compared to something with an actual US association. In any case, this timeout plus a threshold on the VSPatch extinction learning could preserve a small dip, conditioned on BLA sufficient to drive gating, for BLA to extinguish on its own timescale.

To obtain the required no-US activation of BLA Ext, the maintained OFC PTpred activity triggered by the original BLA Acq activation can excite the corresponding Ext pool, in an ACh-gated manner: the self-ACh triggered by the BLA at CS onset won't activate the Ext because the PTpred is delayed til the next trial, when the ACh burst is gone, so only the LHb-triggered ACh will do the trick. Indeed we already have PTpred -> Ext connectivity for this purpose, to drive negative DA learning. The strong ACh modulation is the key missing piece.

This is consistent with data from Herry et al. (2008), as discussed in Mollick et al. (2020):

For example, Herry et al. (2008) showed that a distinct set of BLA neurons progressively increased in activity in response to CS-onset over multiple US omission trials (extinction training), in contrast with those (acquisition-coding) neurons that had acquired activity in response to CS-onset during fear acquisition.

Direct BLA Acq -> Ext connectivity could also accomplish this if somehow it is significantly delayed, so that initial Acq activity at CS onset with ACh doesn't end up immediately inhibiting Ext. It would be more robust to not strictly require OFC PTpred active maintenance inputs, as long as there is some residual Acq activity around to be extinguished. So Acq -> Ext is some kind of delayed metabotropic thing? Could use CTCtxt mechanism + ACh gating. Once Ext is active, it supports Ext activity later. It is also essential that actual US input does NOT activate these Ext guys -- we need inhibition there to turn them off, as they would otherwise get active from PTp etc.

[rcoreilly]

Challenges in temporally-delayed opponent-process learning

Both the BG and BLA require 2 "challenging" (non-trivial, biologically complicated) learning dynamics:

• They need to learn about the activity state present at the initial point of decision-making, which reflects conditions that are likely to be present next time around, but the direction of learning depends on the subsequent outcome (positive or negative dopamine). This drives the need for a trace-based learning mechanism, where synapses record the activation state as a kind of synaptic tag that is then driven up or down based on the subsequent outcome later in time.

  • In the BLA, only the sending activity should be in the trace, because we don't yet know what US outcome will happen, and that determines the receiver activity state.
  • In the BG, we have recorded both sender and receiver in the trace, under the assumption that the specific subset of BG neurons active then is somehow special (they are after all the ones that got activated in the first place). But in the case of the NoGo pathway, it might make more sense to activate the receivers at the time of the outcome if needed (see next point). And maybe there is some computational advantage to having the Go pathway receivers determined at time of outcome too -- they will have more information about what kind of action plan / goal rep etc was actually needed in the end. This actually seems very important.

• There are opponent Go vs. No or Acq vs. Ext pathways that both potentially need to learn, depending on the outcome. Per above, it now seems clear that using receiver activation state at the time of outcome is likely to be much more robust: if you need to recruit NoGo learning because of a negative outcome, activate the neurons that are associated with that negative outcome and the PFC state that led to it!

  • This resolves a central conundrum in NoGo learning: you need to have a population of opposing neurons active at the time of gating, to tag them for later learning, but if they are too active, then they prevent gating in the first place! Although there is physiological evidence of NoGo neurons getting active during choice behavior, that should be a consequence of learning, not a prerequisite for learning.
  • In the current pcore model, we have a semi-hacky activation scheme in NoGo at time of gating -- would be much better to activate NoGo via D2 at outcome time, in context of state then.
  • In the current BLA Ext impl, we have ACh at US time modulating the PFC maintained activity input, which otherwise is suppressed -- same kind of thing could work for BG: selection of neurons is specifically upregulated at outcome time -- VS neurons already gate at this time, but it is actually unclear if we're distinguishing this from the initial gating event -- that might be a major bug too..

This all suggests that we might want to rethink the cortical trace learning too, which has so far not demonstrated any dramatic improvements in temporally-extended predictive learning cases where it should theoretically be beneficial. Currently, the trace is a sender * receiver Ca co-product, but we could try a longer sender-only trace that is multiplicative on the current-trial Ca co-product instead.

[rcoreilly]

The cortical idea doesn't really work because it adds another sender-only term on top of integrated synaptic Ca. Surprisingly it is actually ok for ra25 but doesn't work well for deep_fsa.

Also, it is important to emphasize that the BG "US-time-receiver" learning logic only works when there is more of a global Go / No signal from the BG, as opposed to the more fine-grained parallel "choose specific action" logic that we originally considered (and most others still think about in the context of BG function). For the BOA VS->MD->OFC, ACC, dlPFC gating, this logic seems sound. It is not clear if some more fine-grained gating might still apply to SMA level motor control.

In the global case, it doesn't really matter which specific striatal neurons might fire -- just the relative overall balance of Go / No. It is more important that the receivers be chosen so their weights make sense in terms of everything else they are representing, so they get good at evaluating specific situations. In this respect, it is important to use some kind of postsynaptic excitation signal at the time of gating, because that reflects the current synaptic weights. However, full spiking rate, especially for NoGo neurons, is too high threshold -- in many cases there may be no significant spiking and yet you want to drive significant learning.

This also means that we don't need to track gating for learning: ACh modulation can select the times to update the trace, regardless of actual gating. We already stopped caring about specific pools, due to the global nature of gating, so this eliminates this further.

We still do have to deal with the pesky all-NoGo problem and the NoGateLRate param: if the BG never gates, it never explores and gets USs to learn about, etc. This does seem more like a larger motivational level issue: if not getting reward regularly, lower the threshold and explore more. That is better than just randomly learning something.

[rcoreilly]

Urgency

In an attempt to get rid of NoGateLRate, added an Urgency layer that projects (strongly) to Go layer, and provides an increasing Go bias that is reset when a positive US is received. This is a better overall logic for ealing with lack of Go firing -- it is sensitive to the longer time window and isn't operating automatically on each trial. It uses Effort increments, and piggy-backs on all that infrastructure so it is essentially automatic.

No Gate learning logic

Even with Urgency, we still need NoGateLRate. And not just for all-NoGo. It is a deeper principle that makes sense in general (as has been rediscovered several times..).

• If Trace is always the same regardless of gating, then negative outcomes reduce Go weights -- so failures due to not Go firing are just punished. In pcore this punishment outweighs increases in Urgency input.

• If Trace is 0 on non-gating, then nothing happens -- no learning..

• If Trace is negative on non-gating, then failures properly upregulate Go firing. Moreover, success (positive DA) properly reinforces non-gating in that context -- both directions make sense (and vice-versa for NoGo).

• It is not clear that removing this actually would affect the overall Go bias (pcore has other sources of this..).

[rcoreilly]

Pavlovian logic doesn't work..

The Pavlovian case in VSMatrix doesn't fit with NoGateLRate logic: we deliver the positive DA value regardless of whether the initial CS gating happened. And if the initial CS gating does NOT happen, then the sign flip decreases the chance of subsequent gating, and everything goes in the wrong direction.

Also, in the previous iteration it learned at the time of the US (via CurTrl* param), which then ended up spreading to the CS time. This was not a good learning dynamic overall given that we want learning to generalize to CS onset.

In general, the Pavlovian case is weird, in that it drives reinforcement regardless of goal engagement.

But it should in general be true that a strong US via BLA attached to an active drive will always drive goal engagement, especially in a pavlovian context where there is no effort.. so for now, it makes sense to just change the params so this always happens, and in general have this as the starting bias.

[rcoreilly]

confirming that the above does work for PVLV: stronger Go bias and turning off / down NoGateLRate -- we can revisit this later at some point but for now it should be sufficient, and we'll see how it all goes in boa context.

[rcoreilly]

BLA Novelty needs to be CS dependent

Right now, the BLA Novelty layer that drives the first pool of BLA is always active, and thus the BLA novelty neurons are always active, which gives them a significant head start and drives corresponding OFC activity all the time -- there is too much novelty activity even after extinction (which only turns it off when a CS comes in).

A good solution is to make the Novelty layer just a coarsely-tuned CS layer that responds to any CS. In biology this would likely just be direct BLA neurons that are broadly tuned, but it is not easy to have differential weights for the first pool in a layer, so we're just hacking it by introducing this extra layer and projection. This works well and is now configured using the new DefParams setup.

[rgobbel]

Considering the magnitude of the architectural change from before, when PPTg was just doing a temporal derivative, how does the current picture square with what's currently known about connectivity, in terms of, e.g., the size of the fiber bundles that traverse from one nucleus to another? Just a random thought that occurred to me.

I think I need to review my neuroanatomy. If you have suggestions for a good current source, I'd greatly appreciate it.

[rcoreilly]

OFC prediction is too static

OFC should predict US and PV values. But over the course of a goal engaged episode, these are relatively static in most cases: new info on this does not come in frequently, and is mainly based on the BLA US learning. This doesn't give much for OFC predictive learning to do, especially on a trial-by-trial basis.

In the pvlv example, we needed to connect ofcPTp and ofcCT to effort and urgency to force it to update on a trial-by-trial basis. But in the BOA framework, these should be predicted by ACC, not OFC.

One general idea is that OFC should "track progress toward the goal" -- this would amount to a kind of distance-like representation that increments steadily upward toward the overall US outcome that is predicted at a given point in time.

Unlike effort, where the hypothalamus very plausibly provides increments (and integrals) of metabolic effort expended, it is unclear what kind of low-level brainstem mechanism would naturally provide a tracking toward the ultimate OFC value. It is kind of like a TD chaining dynamic with discounting that drives a progressive increment toward the final value.

[rcoreilly]

PL as central, defining locus of control

BarattaSeligmanMaier23

Baratta, M. V., Seligman, M. E. P., & Maier, S. F. (2023). From helplessness to controllability: Toward a neuroscience of resilience. Frontiers in Psychiatry, 14. https://www.frontiersin.org/journals/psychiatry/articles/10.3389/fpsyt.2023.1170417

• Nice integrative review of literature showing that PL is the central locus of "control", along with the DMS, MD thalamus circuit that is important for activating PL layer V descending projections to the dorsal raphe, which communicate the state of "control" in stressful situations. They also link this to the goal directed instrumental conditioning literature.

• In the BOA model, PL is the core central integrator of all the other BOA inputs , so this makes sense. We can use PL layer 5 activity as the unique barometer of goal engaged state.

[rcoreilly]

Chiba, T., Kayahara, T., & Nakano, K. (2001). Efferent projections of infralimbic and prelimbic areas of the medial prefrontal cortex in the Japanese monkey, Macaca fuscata. Brain Research, 888(1), 83–101. http://www.ncbi.nlm.nih.gov/pubmed/11146055

Key reference for IL, PL projections in monkey! Probably shorter to list where they do NOT project vs. where they do.

[rcoreilly]

MD connectivity in rodent

KuramotoPanFurutaEtAl17

Kuramoto, E., Pan, S., Furuta, T., Tanaka, Y. R., Iwai, H., Yamanaka, A., Ohno, S., Kaneko, T., Goto, T., & Hioki, H. (2017). Individual mediodorsal thalamic neurons project to multiple areas of the rat prefrontal cortex: A single neuron-tracing study using virus vectors. Journal of Comparative Neurology, 525(1), 166–185. https://doi.org/10.1002/cne.24054

The rat MD is usually subdi- vided into three segments, the medial (MDm), central (MDc), and lateral (MDl) divisions, on the basis of the cytoarchitecture and chemoarchitecture (for reviews, see Groenewegen and Witter, 2004; Jones, 2007; Vertes et al., 2015).

Specifically, the MDm mainly receives input from the accumbens nucleus, ventral pallidum, and amygdaloid nucleus, and projects to the prelimbic, the dorsal part of the agranular insular area, and part of the orbital area; the MDc from the endopiriform nucleus and olfactory tubercle, to the ventral part of the agranular insular and orbital areas; and the MDl from the substantia nigra pars reticulata and brainstem, to the anterior cingulate and prelimbic areas (Krettek and Price, 1977; Groenewegen, 1988; Ray and Price, 1992; Reep et al., 1996; Hoover and Vertes, 2007).

• MDm is core BOA gating area
• MDc is OFC / olfactory-specific
• MDI is motor / ACC specific

However, only one study has been reported that partially demon- strates multiple prefrontal-area projections from single MDl neurons; doubly retrograde-labeled neurons were detected in the MDl after the two kinds of retrograde tracers were separately applied into the anterior cingu- late and prelimbic areas in the rat (Cond�e et al., 1990).

In the cerebral cortex, all 14 single-labeled MD neu- rons sent their axon fibers to multiple areas simultane- ously (Table 2). For example, MDc neuron 7, which formed the most widespread axonal arborization in the cortex, sent fine varicose axon fibers to very wide fields over the anterior cingulate, prelimbic, frontal associa- tion, ventral orbital, medial orbital, secondary motor, retrosplenial, and secondary visual areas (Fig. 7E, Table 2).

The main target areas of MD neurons were the prefron- tal, secondary motor, and frontal association areas, and additionally MD neurons sent a small number of axon fibers, less than 30 mm in axon length, to the primary motor, secondary visual, and retrosplenial areas (Table 2).

All five MDc neurons sent their axons abundantly to the orbital areas, including lateral (n 5 2), ventral, (n 5 2), ventrolateral (n 5 1), and dorsolateral orbital areas (n 5 1), suggesting that the main target area of the MDc neurons was the orbitofrontal cortex. In addition, MDc neurons projected to the agranular insular (n 5 2), anterior cingulate (n 5 1), and secondary motor areas (n 5 1) (Figs. 7, 8, 11, Table 2).

All four MDl neurons intensely projected to the prelim- bic area, which was considered to be a principal target of the MDl. Furthermore, some MDl neurons intensely projected to the dorsal part of the anterior cingulate (n 5 3), as well as the secondary motor (n 5 2), frontal association (n 5 1), and medial orbital areas (n 5 1) (Figs. 9–11, Table 2).

Axonal arborizations of MDm neurons appeared to be more highly heterogeneous in terms of the target corti- cal areas than those of MDc and MDl neurons. MDm neurons sent axon fibers intensely to the orbital (n 5 2), prelimbic (n 5 2), agranular insular (n 5 2), infralim- bic (n 5 1), and frontal association areas (n 5 1).

Thus, the principal target areas of MDc and MDl neurons appeared to be complementary, whereas the target areas of MDm neurons overlapped with those of MDc and MDl neurons.

More interestingly, cortical axon fibers from single MD neurons were not homogeneously spread, but were rather distributed in a patchy fashion to form axon arbors (Figs. 5–10). The patchy axon arbors were observed in all the reconstructed MD neurons. Further- more, 10 of 14 MD neurons appeared to form two or more patchy axon arbors, but the remaining four MD neurons were likely to have only one patchy axon arbor (MDm neuron 1, Fig. 6E–G; MDm neuron 2, Fig. 5E,F; MDc neuron 10, Fig. 8I,J; MDl neuron 12, Fig. 10D,E). The spread of the single patchy axon arbors in the tan- gential direction to the cortical surface was approxi- mately 1 mm in diameter.

As indicated by yellow (layers 2/3) and blue colors (layer 5), all but one MD neuron pre- ferred the middle layers, especially the deep part of layers 2/3 and the upper part of layer 5, to the other layers; the percentage of layers 2–5 axon length in total cortical axon length was 80.2 6 13.7%.

In the present study, 10 of the 14 single-labeled MD neu- rons sent axon collaterals to the neostriatum, indicating that the neostriatum is one of the principal target regions of MD neurons. Furthermore, the striatal projections from the 10 MD neurons tended to be located in the medial part of the neostriatum (Figs. 5A,E, 6A,E, 7E, 8I, 9A,D, 10A,D). This result is in accordance with previous results by anterograde-labeling studies in rats (Krettek and Price, 1977; Groenewegen, 1988; Erro et al., 2002), and retrograde- labeling studies in monkeys (Parent et al., 1983; Smith and Parent, 1986).

Furthermore, the present single-neuron labeling study clearly showed that individual MDl neurons sent more abundant axon fibers to the neostriatum than MDc and MDm neurons. With anterograde tract-tracing techniques, the rat MDl has been indicated to receive input from the substantia nigra pars reticulata (one of the basal ganglia output nuclei) (Groenewegen, 1988; Kuramoto et al., 2011). Thus, MDl–striatal projection appears to be a feedback projection, like the thalamostriatal projection from the intralaminar thalamic nuclei, which are well known to receive basal ganglia afferents (for reviews, see Groenewegen and Witter, 2004; Jones, 2007).

Recently, a further distinction has been made between matrix-type neurons, with widely ramified axons targeting multiple, distant cortical areas (multiar- eal subtype) and those with an axon that arborizes within a single or a few adjacent areas (focal subtype) (Clasc�a et al., 2012). MDm neuron 2 is classified as the focal subtype because it projected mainly to adjacent prelimbic and infralimbic areas. In contrast, almost all MD neurons examined (13 of 14 5 93%) sent axon fibers principally to the middle layers, being classified into core-type neurons. Thus, the rat MD appears to be almost homogeneously composed of core-type neurons.

These previ- ous and present results suggest that the afferents to the prefrontal cortex generally might form “cluster” structures.. Although the precise relationship between the anatomical “axonal clusters” and “physiologically dis- tinct clusters” of the prefrontal cortex awaits further studies, afferents to the prefrontal cortex may selec- tively activate cortical neuron groups within the physio- logically distinct clusters of the prefrontal cortex.

Recently, in mice trained by a spatial working mem- ory task (T maze delayed non-match to sample task), it has been reported that MD neuronal spiking synchro- nizes with local field potentials of the prefrontal cortex in the low beta frequency range (13–20 Hz), with MD spikes leading (Parnaudeau et al., 2013).

AnastasiadesCollinsCarter21

Anastasiades, P. G., Collins, D. P., & Carter, A. G. (2021). Mediodorsal and ventromedial thalamus engage distinct L1 circuits in the prefrontal cortex. Neuron. https://doi.org/10.1016/j.neuron.2020.10.031

MD primarily drives VIP+ cells in L1b, which in turn inhibit L2/3 SOM+ cells, engaging a local disinhibitory circuit. In contrast, VM engages NDNF+ cells in L1a, which then inhibit L2/3 PV+ cells, along with L2/3 and L5 pyramidal cells. By targeting the apical dendrites of PT cells, NDNF+ cells robustly inhibit VM- mediated activation of dendritic Ca2+ signals. NDNF+ cells therefore participate in both a thalamus-evoked inhibitory circuit and a separate, distinct disinhibitory circuit. Together, these re- sults indicate that higher order thalamic inputs engage different inhibitory networks to have distinct functional influences on the PFC.

However, these inputs have different axonal projections, with MD innervating L1b and L3 and VM arborizing in L1a. We previ- ously found MD drives L2/3 pyramidal cells (Collins et al., 2018), and our new data indicate MD also activates L1b VIP+ cells. In contrast, VM engages NDNF+ cells in L1a, which in turn play a critical role in regulating the activity of L5 PT cells. These results highlight how synaptic connectivity provides an important addi- tional metric for understanding the diversity of thalamic nuclei.

Evidence from sensory cortices suggests that this sublaminar organization may be a general principle, with VIP+ cells consistently located further from the cortical surface (Abs et al., 2018; Schuman et al., 2019). Interestingly, MD targeting of VIP+ cells in PFC resembles activation of somatosensory VIP+ cells by inputs from the motor cortex and higher order thal- amus (Lee et al., 2013; Williams and Holtmaat, 2019). VIP+ cells in turn inhibit SOM+ cells, a motif observed across cortex (Lee et al., 2013; Williams and Holtmaat, 2019), including the PFC (Pi et al., 2013). Thus, our results suggest that MD activates a conserved disinhibitory circuit, with the inhibition of SOM+ cells suppressing dendritic inhibition of pyramidal cells, potentially enabling dendritic electrogenesis and synaptic plasticity (Wil- liams and Holtmaat, 2019). In contrast, VM strongly activates L1a NDNF+ cells, which in turn inhibit PV+ cells that synapse at the soma of pyramidal cells. A similar form of disinhibition via PV+ cells has been reported to drive a form of non-VIP+- dependent plasticity during auditory learning (Letzkus et al., 2011). Therefore, our results indicate that thalamic nuclei engage distinct inhibitory micro-circuits that can influence dendritic and somatic activity, with important implications for PFC function.

Our experiments suggest multiple ways in which MD and VM differentially impact specific micro-circuits to shape activity in the PFC (Bolkan et al., 2017; Schmitt et al., 2017). MD input to L1b VIP+ cells may combine with input to L3 pyramidal cells (Collins et al., 2018) to maintain local excitation and support delay period activity (Bolkan et al., 2017; Schmitt et al., 2017). Similarly, MD input may also amplify cortical or basolateral amygdala inputs to superficial layers (Little and Carter, 2013; Schmitt et al., 2017), as suggested for higher order thalamic in- puts to sensory cortex (Williams and Holtmaat, 2019; Zhang and Bruno, 2019). Activation of superficial networks also influ- ences deep layers, including connections onto L5 PT cells that ‘‘close the loop’’ between PFC and thalamus (Collins et al., 2018).

By targeting their dendrites, VM can also directly and selectively engage PT cells, which innervate numerous down- stream targets (Collins et al., 2018; Economo et al., 2018). Consistent with our findings, a recent study from sensory cortex showed how activity within PT dendrites plays an essential role in regulating cortical function via increasing outputs to thalamus and other regions (Takahashi et al., 2020). Therefore, VM-evoked activation of PT cell dendrites has important implications for cognitive behaviors that depend upon communication between PFC, thalamus, and subcortical structures (Bolkan et al., 2017; Guo et al., 2017; Schmitt et al., 2017; Warden et al., 2012).

CollinsAnastasiadesMarlinEtAl18

Collins, D. P., Anastasiades, P. G., Marlin, J. J., & Carter, A. G. (2018). Reciprocal circuits linking the prefrontal cortex with dorsal and ventral thalamic nuclei. Neuron, 98(2), 366-379.e4. https://doi.org/10.1016/j.neuron.2018.03.024

• prior work

PerryLomiMitchell21

Perry, B. A. L., Lomi, E., & Mitchell, A. S. (2021). Thalamocortical interactions in cognition and disease: The mediodorsal and anterior thalamic nuclei. Neuroscience & Biobehavioral Reviews, 130, 162–177. https://doi.org/10.1016/j.neubiorev.2021.05.032

• Nice broad review

Schmitt et al. (2017) found that spiking in the MDl sustained neural
activity in the dmPFC when representing two distinct task related rules,
even though the MDl neurons themselves were not rule tuned. In a
subsequent study, Rikhye et al. (2018) showed that a subset of MDl cells
helped to sustain context relevant representation in the dmPFC, while a
different subset suppressed context irrelevant ones. Consistent with
these findings, Marton et al. (2018) found that selective disruption of
dmPFC outputs to MDl with optogenetics, interrupted behavioral flexibility
and prevented mice from switching from visual to auditory cues in
a decision-making task.

In one
study, Bolkan et al. (2017) used optogenetics to produce temporally
precise pathway specific MD→dmPFC or dmPFC→MD inhibition while
mice performed a spatial alternation task in the T-maze. Bolkan et al.
(2017) found that inhibition of MD→dmPFC projections, but only during
an extended inter-trial interval, resulted in impaired performance.
By contrast, inhibition of dmPFC→MD projections only reduced performance
during the subsequent choice phase of the task. These effects
were specific to MD→dmPFC pathways as inhibition of MD→OFC
pathways did not alter behavior in this task. Interestingly though,
working memory deficits are not typically reported in rats, primates, or
in humans with damage to the MD (e.g., Kopelman, 2015; Mitchell,
2015; Pergola et al., 2018). Thus, this performance deficit observed in
mice is perhaps either linked to difficulties in implementing an optimal
response strategy over time (see discussion below), or instead may be
linked to specific deficits caused by temporary inactivation vs the ability
of the system to adapt after permanent damage.

Using a different approach, Alcaraz et al. (2018) used chemogenetics
to selectively inhibit projection defined neurons in the dmPFC or MDl
during an instrumental learning task. Both pathways were found to
participate in behavioral adaptions to the current goal value, but only
MD→dmPFC neurons were required to integrate current causal relationships
between actions and outcomes. Others have reported inactivation
of the MD contributes to impaired timing of instrumental
actions (Lusk et al., 2020). Taken together all these studies suggest, as
previously observed: 1) interdependence between the MD and mPFC
during spatial and adaptive decision-making tasks, and 2) that the MD
provides a unique influence within the PFC in that task related neural
activity in the MD does not simply replicate that of its efferent or afferent
interconnected cortex with selective manipulations of corticothalamic
and thalamocortical pathways resulting in differential outcomes.

Finally, work in rats from our own lab has reported differential
choice response impairments after excitotoxic lesions to combined
MDm/MDc during an attentional set-shifting task (Ouhaz et al., 2021).

Rats with permanent MD lesions required more trials to criterion to
learn the initial sensory discrimination. However, once acquired, they
were able to transfer this choice response strategy to further
intra-dimensional shifts and reversals linked to the same sensory
discrimination. This result provides a direct contrast to rats with excitotoxic
lesions to the ATN (Wright et al., 2015) or nucleus reuniens
(Linley et al., 2016) – two neighboring thalamic structures to the MD.

The conclusion from all this evidence suggests the MD, particularly the MDmc (MDm in rodents), contributes
to monitoring and feedback to the cortex about the optimal task
relevant choice response strategy. A related proposal has been made for
the lateral MD in primates contributing to the internal monitoring and
feedback of optimal sequences of eye movement (Sommer and Wurtz,
2008).

Prior to continuing our discussion on the contribution
of the ATN to HD processing, it is important to highlight that the
ATN also contributes to other higher cognitive functions in addition to
supporting optimal spatial information integration (e.g., Aggleton et al.,
1996; Henry et al., 2004; Mitchell and Dalrymple-Alford, 2006; Jankowski
et al., 2013; Aggleton and Nelson, 2015). For example, evidence
gathered from humans and rodents suggests the ATN contribute to
nonspatial cognition via interactions with the amygdala, prelimbic
cortex, and the ACC (Shibata and Kato, 1993; Shibata and Naito, 2005;
Wright et al., 2013; Mathiasen et al., 2017). In rodents, lesion studies
indicate a role for the ATN in “relational” processing. For example, ATN
lesions spare recognition memory for single items (Mitchell and Dalrymple-Alford,
2005; Law and Smith, 2012) but impairments emerge
when animals are required to combine memory about multiple items
with their temporal and spatial properties (e.g., Parker and Gaffan,
1997; Wolff et al., 2006; Dumont and Aggleton, 2013). T

[rcoreilly]

Additional papers:

WolffHalassa24

Wolff, M., & Halassa, M. M. (2024). The mediodorsal thalamus in executive control. Neuron. https://doi.org/10.1016/j.neuron.2024.01.002

Rios-FlorezLimaMoraisEtAl21

Ríos-Flórez, J. A., Lima, R. R. M., Morais, P. L. A. G., de Medeiros, H. H. A., Cavalcante, J. S., & Junior, E. S. N. (2021). Medial prefrontal cortex (A32 and A25) projections in the common marmoset: A subcortical anterograde study. Scientific Reports, 11(1), 14565. https://doi.org/10.1038/s41598-021-93819-z