Abstract

It is well established that learning can occur without external feedback, yet normative reinforcement learning theories have difficulties explaining such instances of learning. Here, we propose that human observers are capable of generating their own feedback signals by monitoring internal decision variables. We investigated this hypothesis in a visual perceptual learning task using fMRI and confidence reports as a measure for this monitoring process. Employing a novel computational model in which learning is guided by confidence-based reinforcement signals, we found that mesolimbic brain areas encoded both anticipation and prediction error of confidence—in remarkable similarity to previous findings for external reward-based feedback. We demonstrate that the model accounts for choice and confidence reports and show that the mesolimbic confidence prediction error modulation derived through the model predicts individual learning success. These results provide a mechanistic neurobiological explanation for learning without external feedback by augmenting reinforcement models with confidence-based feedback.

Figure 1 shows this model in a one-dimensional cable, corresponding to a section of neurite. In each unit of time the cargo moves a unit distance forwards or backwards, or remains in the same place, each with different probabilities. In the simplest version of the model, the probabilities of forward and backward jumps are constant for each time step (Figure 1B, top panel). Cargo can also undergo extended unidirectional runs (Klumpp and Lipowsky, 2005; Müller et al., 2008; Hancock, 2014). The model can account for these runs with jump probabilities that depend on the previous movement of the particle (Figure 1B, bottom panel, Materials and methods).

While the movement of individual cargoes is stochastic, the spatial distribution of a population (Figure 1C) changes predictably. This is seen in Figure 1D, which shows the distribution of 1000 molecules over time, without (top panel) and with (bottom panel) unidirectional runs. The bulk distribution of cargo can therefore be modelled as a deterministic process that describes how cargo concentration spreads out in time.

A convenient and flexible formulation of this process is a mass-action model (Voit et al., 2015) that spatially discretizes the neuron into small compartments. In an unbranched neurite with N compartments, the mass-action model is:

(1)u1⇌b1a1u2⇌b2a2u3⇌b3a3...⇌bN−1aN−1uN

where ui is the amount of cargo in each compartment, and ai and bi denote trafficking rate constants of cargo exchange between adjacent compartments. This model maps onto the well-known drift-diffusion equation when the trafficking rates are spatially homogeneous (Figure 1E; Smith and Simmons, 2001). We used this to constrain trafficking rate constants based on single-particle tracking experiments (Dynes and Steward, 2007) or estimates of the mean and variance of particle positions from imaging experiments (Roy et al., 2012, see Materials and methods).

With a compartment length of 1 μm, the simulations in Figure 1D gave mean particle velocities of 15 μm per minute, which is within the range of experimental observations for microtubule transport (Rogers and Gelfand, 1998; Dynes and Steward, 2007; Müller et al., 2008). The variances of the particle distributions depended on whether unidirectional runs are assumed, and respectively grew at a rate of ~0.58 and ~1.33 μm2 per second for the top and bottom panels of Figure 1D. The mass action model provides a good fit to both cases (Figure 1F). In general, the apparent diffusion coefficient of the model increases as run length increases (Figure 1—figure supplement 1A). The accuracy of the mass-action model decreases as the run length increases. However, the model remains a reasonable approximation for many physiological run lengths and particle numbers, even over a relatively short time window of 100 s (Figure 1—figure supplement 1B).

eLife Digest

Each year, up to 15% of the world's population experience symptoms of an influenza infection, also commonly known as flu. The most common culprit is a strain of the virus called influenza type A subtype H3N2. One reason that so many people become infected each year is that this virus evolves rapidly. Within a few years, proteins on the surface of the virus known as antigens become less recognizable to the immune system of a person who has been previously infected. This means that the person can become ill with the virus again because their immune system cannot mount an effective response to the evolved virus strain.

Influenza virus strains evolve rapidly because their genetic material accumulates mutations quickly. Although some of these mutations are beneficial to the virus, other mutations are harmful and reduce the ability of the virus to spread. Sometimes beneficial mutations may occur alongside harmful ones, but it is not known how the harmful mutations affect the evolution of the virus.

Here, Koelle and Rasmussen used computer models of H3N2 influenza to examine the effect of harmful mutations on the evolution of this virus population. The models show that harmful mutations limit how quickly the antigens can evolve. Also, the presence of these harmful mutations effectively acts as a sieve: they allow only large changes in the antigens to establish in the virus population.

The models suggest that there are three routes by which large changes in the antigens on H3N2 viruses may occur. The first is by a single mutation that has a big effect on the antigens in viruses that only carry a few harmful mutations, but these large mutations would not happen very often. Another route may be through more common mutations that have only a small or moderate benefit, which would allow the virus to become more common in the population before it acquires a beneficial mutation with a much greater effect. The third possibility is that a large beneficial mutation may arise in viruses that have many harmful mutations. These harmful mutations may initially limit the ability of the virus to spread, but over time, some of these harmful mutations may then be lost.

Koelle and Rasmussen found that the computer models could recreate the patterns of virus evolution that have been observed in real strains of H3N2. Researchers use predictions of influenza evolution to help them decide which virus strains should be included in flu vaccines each year. Koelle and Rasmussen findings indicate that harmful mutations should be considered when making these predictions.

Introduction

TNF Receptor Associated Factor 2 (TRAF2) is an adaptor protein that transduces signals following ligation of certain cytokine receptors including those binding TNF. It was first identified together with TRAF1 as a component of TNF receptor-2 and then TNF receptor-1 (TNFR1) signalling complexes (Rothe et al., 1994; Shu et al., 1996). TRAF2, like most other TRAFs, contains a RING domain, several zinc fingers, a TRAF-N, and a conserved TRAF-C domain which is responsible for oligomerisation and receptor binding through its MATH region (Takeuchi et al., 1996; Uren and Vaux, 1996).

Activation of JNK and NF-κB by TNF is reduced in cells from Traf2-/- mice while only JNK signalling was affected in lymphocytes from transgenic mice that express a dominant negative (DN) form of TRAF2 that lacks the RING domain (Lee et al., 1997; Yeh et al., 1997). Traf2-/-Traf5-/- mouse embryonic fibroblasts (MEFs) have a pronounced defect in activation of NF-κB by TNF, suggesting that absence of TRAF2 can be compensated by TRAF5 (Tada et al., 2001). Although activation of NF-κB was restored in Traf2-/-Traf5-/- cells by re-expression of wild type TRAF2, it was not restored when the cells were reconstituted with TRAF2 point mutants that could not bind cIAPs (Vince et al., 2009; Zhang et al., 2010). These data, together with a wealth of different lines of evidence showing that cIAPs are critical E3 ligases required for TNF-induced canonical NF-κB (Blackwell et al., 2013; Haas et al., 2009; Silke, 2011), support the idea that the main function of TRAF2 in TNF-induced NF-κB is to recruit cIAPs to the TNFR1-SC. However, it remains possible that the RING of TRAF2 plays another function, such as in activating JNK and protecting cells from TNF-induced cell death (Vince et al., 2009; Zhang et al., 2010). Furthermore it has been shown that TRAF2 can K48-ubiquitylate caspase-8 to set the threshold for TRAIL or Fas induced cell death (Gonzalvez et al., 2012). Moreover, TRAF2 inhibits non-canonical NF-κB signalling (Grech et al., 2004; Zarnegar et al., 2008) and this function requires the RING domain of TRAF2 to induce proteosomal degradation of NIK (Vince et al., 2009). However, structural and in vitro analyses indicate that, unlike TRAF6, the RING domain of TRAF2 is unable to bind E2 conjugating enzymes (Yin et al., 2009), and is therefore unlikely to have intrinsic E3 ligase activity.

Sphingosine-1-phosphate (S1P) is a pleiotropic sphingolipid mediator that regulates proliferation, differentiation, cell trafficking and vascular development (Pitson, 2011). S1P is generated by sphingosine kinase 1 and 2 (SPHK1 and SPHK2) (Kohama et al., 1998; Liu et al., 2000). Extracellular S1P mainly acts by binding to its five G protein-coupled receptors S1P1-5 (Hla and Dannenberg, 2012). However, some intracellular roles have been suggested for S1P, including the blocking of the histone deacetylases, HDAC1/2 (Hait et al., 2009) and the induction of apoptosis through interaction with BAK and BAX (Chipuk et al., 2012).

Recently, it was suggested that the RING domain of TRAF2 requires S1P as a co-factor for its E3 ligase activity (Alvarez et al., 2010). Alvarez and colleagues proposed that SPHK1 but not SPHK2 is activated by TNF and phosphorylates sphingosine to S1P which in turn binds to the RING domain of TRAF2 and serves as an essential co-factor that was missing in the experiments of Yin et al. Alvarez and colleagues, observed that in the absence of SPHK1, TNF-induced NF-κB activation was completely abolished.

Although we know a lot about TRAF2, there are still important gaps particularly with regard to cell type specificity and in vivo function of TRAF2. Moreover, despite the claims that SPHK1 and its product, S1P, are required for TRAF2 to function as a ubiquitin ligase, the responses of Traf2-/- and Sphk1-/- cells to TNF were not compared. Therefore, we undertook an analysis of TRAF2 and SPHK1 function in TNF signalling in a number of different tissues.

Surprisingly, we found that neither TRAF2 nor SPHK1 are required for TNF mediated canonical NF-κB and MAPK signalling in macrophages. However, MEFs, murine dermal fibroblasts (MDFs) and keratinocytes required TRAF2 but not SPHK1 for full strength TNF signalling. In these cell types, absence of TRAF2 caused a delay in TNF-induced activation of NF-κB and MAPK, and sensitivity to killing by TNF was increased. Absence of TRAF2 in keratinocytes in vivo resulted in psoriasis-like epidermal hyperplasia and skin inflammation. Unlike TNF-dependent genetic inflammatory skin conditions, such as IKK2 epidermal knock-out (Pasparakis et al., 2002) and the cpdm mutant (Gerlach et al., 2011), the onset of inflammation was only delayed, and not prevented by deletion of TNF. This early TNF-dependent inflammation is caused by excessive apoptotic but not necroptotic cell death and could be prevented by deletion of Casp8. We observed constitutive activation of NIK and non-canonical NF-κB in Traf2-/- keratinocytes which caused production of inflammatory cytokines and chemokines. We were able to reverse this inflammatory phenotype by simultaneously deleting both Tnf and Nfkb2 genes. Our results highlight the important role TRAF2 plays to protect keratinocytes from cell death and to down-regulate inflammatory responses and support the idea that intrinsic defects in keratinocytes can initiate psoriasis-like skin inflammation.

TNF Receptor Associated Factor 2 (TRAF2) is an adaptor protein that transduces signals following ligation of certain cytokine receptors including those binding TNF. It was first identified together with TRAF1 as a component of TNF receptor-2 and then TNF receptor-1 (TNFR1) signalling complexes (Rothe et al., 1994; Shu et al., 1996). TRAF2, like most other TRAFs, contains a RING domain, several zinc fingers, a TRAF-N, and a conserved TRAF-C domain which is responsible for oligomerisation and receptor binding through its MATH region (Takeuchi et al., 1996; Uren and Vaux, 1996).

Activation of JNK and NF-κB by TNF is reduced in cells from Traf2-/- mice while only JNK signalling was affected in lymphocytes from transgenic mice that express a dominant negative (DN) form of TRAF2 that lacks the RING domain (Lee et al., 1997; Yeh et al., 1997). Traf2-/-Traf5-/- mouse embryonic fibroblasts (MEFs) have a pronounced defect in activation of NF-κB by TNF, suggesting that absence of TRAF2 can be compensated by TRAF5 (Tada et al., 2001). Although activation of NF-κB was restored in Traf2-/-Traf5-/- cells by re-expression of wild type TRAF2, it was not restored when the cells were reconstituted with TRAF2 point mutants that could not bind cIAPs (Vince et al., 2009; Zhang et al., 2010). These data, together with a wealth of different lines of evidence showing that cIAPs are critical E3 ligases required for TNF-induced canonical NF-κB (Blackwell et al., 2013; Haas et al., 2009; Silke, 2011), support the idea that the main function of TRAF2 in TNF-induced NF-κB is to recruit cIAPs to the TNFR1-SC. However, it remains possible that the RING of TRAF2 plays another function, such as in activating JNK and protecting cells from TNF-induced cell death (Vince et al., 2009; Zhang et al., 2010). Furthermore it has been shown that TRAF2 can K48-ubiquitylate caspase-8 to set the threshold for TRAIL or Fas induced cell death (Gonzalvez et al., 2012). Moreover, TRAF2 inhibits non-canonical NF-κB signalling (Grech et al., 2004; Zarnegar et al., 2008) and this function requires the RING domain of TRAF2 to induce proteosomal degradation of NIK (Vince et al., 2009). However, structural and in vitro analyses indicate that, unlike TRAF6, the RING domain of TRAF2 is unable to bind E2 conjugating enzymes (Yin et al., 2009), and is therefore unlikely to have intrinsic E3 ligase activity.

Sphingosine-1-phosphate (S1P) is a pleiotropic sphingolipid mediator that regulates proliferation, differentiation, cell trafficking and vascular development (Pitson, 2011). S1P is generated by sphingosine kinase 1 and 2 (SPHK1 and SPHK2) (Kohama et al., 1998; Liu et al., 2000). Extracellular S1P mainly acts by binding to its five G protein-coupled receptors S1P1-5 (Hla and Dannenberg, 2012). However, some intracellular roles have been suggested for S1P, including the blocking of the histone deacetylases, HDAC1/2 (Hait et al., 2009) and the induction of apoptosis through interaction with BAK and BAX (Chipuk et al., 2012).

Recently, it was suggested that the RING domain of TRAF2 requires S1P as a co-factor for its E3 ligase activity (Alvarez et al., 2010). Alvarez and colleagues proposed that SPHK1 but not SPHK2 is activated by TNF and phosphorylates sphingosine to S1P which in turn binds to the RING domain of TRAF2 and serves as an essential co-factor that was missing in the experiments of Yin et al. Alvarez and colleagues, observed that in the absence of SPHK1, TNF-induced NF-κB activation was completely abolished.

I am a nested frag

Although we know a lot about TRAF2, there are still important gaps particularly with regard to cell type specificity and in vivo function of TRAF2. Moreover, despite the claims that SPHK1 and its product, S1P, are required for TRAF2 to function as a ubiquitin ligase, the responses of Traf2-/- and Sphk1-/- cells to TNF were not compared. Therefore, we undertook an analysis of TRAF2 and SPHK1 function in TNF signalling in a number of different tissues.

Surprisingly, we found that neither TRAF2 nor SPHK1 are required for TNF mediated canonical NF-κB and MAPK signalling in macrophages. However, MEFs, murine dermal fibroblasts (MDFs) and keratinocytes required TRAF2 but not SPHK1 for full strength TNF signalling. In these cell types, absence of TRAF2 caused a delay in TNF-induced activation of NF-κB and MAPK, and sensitivity to killing by TNF was increased. Absence of TRAF2 in keratinocytes in vivo resulted in psoriasis-like epidermal hyperplasia and skin inflammation. Unlike TNF-dependent genetic inflammatory skin conditions, such as IKK2 epidermal knock-out (Pasparakis et al., 2002) and the cpdm mutant (Gerlach et al., 2011), the onset of inflammation was only delayed, and not prevented by deletion of TNF. This early TNF-dependent inflammation is caused by excessive apoptotic but not necroptotic cell death and could be prevented by deletion of Casp8. We observed constitutive activation of NIK and non-canonical NF-κB in Traf2-/- keratinocytes which caused production of inflammatory cytokines and chemokines. We were able to reverse this inflammatory phenotype by simultaneously deleting both Tnf and Nfkb2 genes. Our results highlight the important role TRAF2 plays to protect keratinocytes from cell death and to down-regulate inflammatory responses and support the idea that intrinsic defects in keratinocytes can initiate psoriasis-like skin inflammation.

Materials and methods

Embryos were isolated from a wild B. schlosseri colony from Monterey Bay. Metaphase chromosomes were isolated as previously described (Shoguchi et al., 2004). B. schlosseri metaphase chromosome suspension was partitioned into wells in the microfluidic device as previously described (Fan et al., 2011; Xu et al., 2011). The contents of each microfluidic well were amplified individually and prepared for sequencing. Each well contained between 1–4 metaphase chromosomes (Figure 3, Figure 3—figure supplement 1, Figure 3—figure Supplement 2). 21 wells were made into libraries and sequenced using Illumina Hiseq (2 x 100).

Chromosome assignment

Since a particular chromosome has an equal chance of occupying any of the 21 wells, we can denote the presence of a particular chromosome for example chromosome A across the 21 wells in the form of a vector (1,0,1,0,1,0,1,0,0,0,0,0,1,0,0,0,1,0,0,0,1) where 1 denotes presence in a well and 0 as absent. To deduce this configuration and perform the assignment of contigs, we aligned the reads from each well onto the contigs of the assembly. Contigs that share the same configuration across 21 wells can be considered to be from the same chromosome. To determine which contigs belong together and share the same configuration, we cluster the contigs using the Pearson correlation with each other using the vector across 21 wells. The number of clusters that result from k-means clustering using the Pearson correlation as the distance is then inferred to be the number of chromosomes. To determine this optimal number of clusters K for K-means clustering, we perform the clustering procedure iteratively for each K and record the within sum of squares error for each iteration as shown in Figure 3. Additionally, using the clustered data, we were able to deduce the configuration and deduce the number of chromosomes in each well.

Reads from each of the chromosome preparation were aligned to the 356a draft genome assembly using the BWA (Li and Durbin, 2009) package. Subsequently, SAMtools (Li et al., 2009) were used to filter for high quality mapping of reads with MAPQ score of greater than 30. The filtering was performing by using AWK to filter the fifth column of the SAM file for alignments >25. In addition to filter for a reasonable insert size, AWK is used to filter the corresponding columns in SAM files. Filtered SAM files were then parsed using BEDtools (Quinlan and Hall, 2010) to obtain the number of reads that are associated with each of the scaffolds. Specifically, the coverageBED command is used to calculate the number of reads and coverage associated with each of the scaffolds in the assembled genome. Utilizing the number of reads associated with each scaffold from a particular chromosome preparation were arranged in the following data format:

Partial correlation coefficients are listed only for parameters that are found to improve predictive effect on outcomes using Akaike information criteria models. Episode features are from 500 single episode simulations. Long-term shedding outcomes were measured over 10-years during 500 simulations.

In each cell the number of reads associated with the scaffold and preparation was normalized by the total number of reads across each chromosome preparation. This is done by dividing the number of reads associated with the particular scaffold and preparation by dividing each entry by the total number of reads from a chromosome preparation. This value is then scaled by the fractional coverage of reads for a particular scaffold. These normalization steps ensure that a valid comparison can be made across each preparation. These normalized reads are then used to perform the K-means clustering. The optimal number of clusters is determined by iteratively performing the clustering process with each value of K. Within sum of square errors for each cluster is calculated and plotted in Figure 3A. A knee exists in the curve near K = 13, after which increasing the number of clusters only creates marginal improvement in the error (Figure 3A). To estimate the configuration after the clustering step, 17 out of the 21 wells were deduced to contain information that is used in the clustering process. The average number of normalized reads counts from each well that align to each scaffold in a cluster group is calculated and plotted in Figure 3B, Figure 3—figure supplements 1 and 2. Each peak represented can be inferred to denote the presence of a specific chromosome in the well. This approach yielded 13 well resolved chromosomes (Figure 3, Figure 3—figure supplement 1), close to the ∼16 chromosomes that were predicted by a previous study using metaphase spreads (Colombera, 1963).

356a-chromosomes hybrid assembly

Reads from each of the individual chromosome sample preparation were subsequently assembled using Velvet (Zerbino, 2010; Figure 3—figure supplement 2). Velvet was compiled for our assembly to have a max hash length of 75. This is to allow for the use of larger hash length for the assembly of the reads from the individual well. Since the paired reads from each preparation is ∼86 after filtering for low quality bases, an optimal hash length is selected from the range of 51–75 to obtain the optimal assembly for each preparation. The assembled chromosome level contigs were then merged with 356a draft assembly using Minimus2 http://sourceforge.net/apps/mediawiki/amos/index.php?title=Minimus2 (Figure 3—figure supplement 3). During this stage, minimus2 used a nucmer based overlap detector to detect overlap between sequences and subsequently merging the two sequence sets to generate the final merge assembly of chromosome with the draft. An overall improvement in N50 was achieved, yielding a final 580 Mbp draft 356a-chromosomes hybrid assembly (Supplementary file 2D).

I am an h4

I am an h5

I am an h6

A knee exists in the curve A knee exists in the curve A knee exists in the curve A knee exists in the curve A knee exists in the curve A knee exists in the curve A knee exists in the curve A knee exists in the curve A knee exists in the curve A knee exists in the curve A knee exists in the curve A knee exists in the curve A knee exists in the curve A knee exists in the curve A knee exists in the curve A knee exists in the curve A knee exists in the curve A knee exists in the curve A knee exists in the curve A knee exists in the curve A knee exists in the curve

Alexander von Humboldt-Stiftung (Postdoctoral Fellowship)

Deutsche Forschungsgemeinschaft (SCHE 1575/3-1)

Bundesministerium für Bildung und Forschung (BCCN-II, 01GQ1005C)

European Commission (FP7-611687 (NEBIAS))

Hansjörg Scherberger

The funders had no role in study design, data collection and interpretation, or the decision to submit the work for publication.

Acknowledgements

The authors thank M Sartori, M Dörge, K Menz, R Ahlert, N Nazarenus, and L Burchardt for assistance and MM Fabiszak, M Hepp-Reymond, and W Freiwald for helpful comments on an earlier version of the manuscript. This work was supported by the Federal Ministry of Education and Research (Bernstein Center for Computational Neuroscience II Grant DPZ-01GQ1005C, the German Research Foundation (SCHE 1575/3-1), the European Union Grant FP7– 611687 (NEBIAS), and the Humboldt Foundation.

Ethics

Animal experimentation: All procedures and animal were conducted in accordance with the guidelines for the care and use of mammals in neuroscience and behavioral research (National Research Council, 2003), and were in agreement with German and European laws governing animal care. Authorization for conducting this study has been granted by the regional government office, the Animal Welfare Division of the Office for Consumer Protection and Food Safety of the State of Lower Saxony, Germany (permit no. 032/09). Monkey handling also followed the recommendations of the Weatherall Report of good animal practice. Animals were pairhoused in a spacious cage (well exceeding legal requirements) and were maintained on a 12-hour on/off lighting schedule. Housing procedures included an environmental enrichment program with access to toys, swings, and hidden treats (e.g., seeds in sawdust). Monkeys had visual and auditory contact to other monkeys. They were fed on a diet of enriched biscuits and fruits. Daily access to fluids was controlled during training and experimental periods to promote behavioral motivation. All surgical procedures were performed under anesthesia, and all efforts were made to minimize post-surgical pain or suffering. Institutional veterinarians continually monitored animal health and well-being.

Optimal decision-making requires balancing fast but error-prone and more accurate but slower decisions through adjustments of decision thresholds. Here, we demonstrate two distinct correlates of such speed-accuracy adjustments by recording subthalamic nucleus (STN) activity and electroencephalography in 11 Parkinson’s disease patients during a perceptual decision-making task; STN low-frequency oscillatory (LFO) activity (2–8 Hz), coupled to activity at prefrontal electrode Fz, and STN beta activity (13–30 Hz) coupled to electrodes C3/C4 close to motor cortex. These two correlates differed not only in their cortical topography and spectral characteristics but also in the relative timing of recruitment and in their precise relationship with decision thresholds. Increases of STN LFO power preceding the response predicted increased thresholds only after accuracy instructions, while cue-induced reductions of STN beta power decreased thresholds irrespective of instructions. These findings indicate that distinct neural mechanisms determine whether a decision will be made in haste or with caution.

eLife is a non-profit organisation inspired by research funders and led by scientists. Our mission is to help scientists accelerate discovery by operating a platform for research communication that encourages and recognises the most responsible behaviours in science.eLife Sciences Publications, Ltd is a limited liability non-profit non-stock corporation incorporated in the State of Delaware, USA, with company number 5030732, and is registered in the UK with company number FC030576 and branch number BR015634 at the address:
eLife Sciences Publications, Ltd
Westbrook Centre, Milton Road
Cambridge CB4 1YG
UK