Of course, these basic actions are themselves composed of even more elemental actions reflecting a nested hierarchy of

action complexity. It is has been proposed that the brain MK-1775 nmr implements such a hierarchical scheme, with different levels of a hierarchy tasked with selecting actions at different levels of abstraction [44]. The notion of a hierarchy in RL appeals to a long literature in cognitive neuroscience suggesting the existence of a cognitive hierarchy within prefrontal cortex, with certain brain systems sitting higher up in the hierarchy (possibly located more anteriorly within prefrontal cortex) and thereby exerting control over systems lower down in the hierarchy 45 and 46]. Consistent with hierarchical RL, a recent study reported neural activity in ACC and insula correlating with prediction errors based on ‘pseudo-rewards’ (representing the completion of an elemental action forming part of a rewarding option) in a temporally extended, multi-step decision-making task [47]. Another perspective has been to use Bayesian inference to learn about reward

distributions, or any other task-related decision variable, instead of using prediction errors 9, 48, 49 and 50]. One advantage of the Bayesian approach is that this method provides a natural way to resolve the issue of how to set the rate at which a belief about the world is updated in the face of new information [51]. Among other factors, the GSK J4 in vivo amount of volatility present in the environment (the extent to which reinforcement contingencies are subject to change), should influence the rate at which new information is incorporated into one’s beliefs, and this can be modeled in a very straightforward way in a Bayesian framework [48]. Another advantage of Bayesian inference is that because these models encode representations

of full probability distributions (or approximations Dimethyl sulfoxide thereof), it is straightforward to extract a measure of the degree of uncertainty (or conversely precision) one has in a particular belief. Such uncertainty or precision signals can be used not only to inform setting of learning rates (see [52]), but can also be used to inform decision-strategies such as when to explore or exploit a given decision option (i.e. one might want to explore an option about which one is maximally uncertainty) 53, 54, 55 and 56•]. Supporting the relevance of a Bayesian framework, uncertainty and precision signals have been reported in a number of brain structures including the midbrain, amygdala, prefrontal and parietal cortices 36, 57, 58, 59 and 60].