Free download.
Book file PDF easily for everyone and every device.
You can download and read online Studies in Perception and Action X: Fifteenth International Conference on Perception and Action file PDF Book only if you are registered here.
And also you can download or read online all Book PDF file that related with Studies in Perception and Action X: Fifteenth International Conference on Perception and Action book.
Happy reading Studies in Perception and Action X: Fifteenth International Conference on Perception and Action Bookeveryone.
Download file Free Book PDF Studies in Perception and Action X: Fifteenth International Conference on Perception and Action at Complete PDF Library.
This Book have some digital formats such us :paperbook, ebook, kindle, epub, fb2 and another formats.
Here is The CompletePDF Book Library.
It's free to register here to get Book file PDF Studies in Perception and Action X: Fifteenth International Conference on Perception and Action Pocket Guide.

Perception of shape properties from multiple cues. ISBN link. Percezione di superfici e illuminazione dinamica. Embodied processes and technologies, UoB UK. Parise, C. It suggests biologically plausible explanations for many cognitive phenomena, including consciousness. In active inference, action selection is driven by an objective function that evaluates possible future actions with respect to current, inferred beliefs about the world. Active inference at its core is independent from extrinsic rewards, resulting in a high level of robustness across e.

In the literature, paradigms that share this independence have been summarized under the notion of intrinsic motivations. In general and in contrast to active inference, these models of motivation come without a commitment to particular inference and action selection mechanisms. In this article, we study if the inference and action selection machinery of active inference can also be used by alternatives to the originally included intrinsic motivation. The perception-action loop explicitly relates inference and action selection to the environment and agent memory, and is consequently used as foundation for our analysis.

We reconstruct the active inference approach, locate the original formulation within, and show how alternative intrinsic motivations can be used while keeping many of the original features intact. Furthermore, we illustrate the connection to universal reinforcement learning by means of our formalism.

Active inference research may profit from comparisons of the dynamics induced by alternative intrinsic motivations. Research on intrinsic motivations may profit from an additional way to implement intrinsically motivated agents that also share the biological plausibility of active inference.

Active inference Friston et al. To this end, both the free energy principle and intrinsic motivations aim to bridge the gap between giving a biologically plausible explanation for how real organism deal with the problem and providing a formalism that can be implemented in artificial agents. Additionally, they share a range of properties, such as an independence of a priori semantics and being defined purely on the dynamics of the agent environment interaction, i.

Despite these numerous similarities, as far as we know, there has not been any unified or comparative treatment of those approaches. We believe this is in part due to a lack of an appropriate unifying mathematical framework. To alleviate this, we present a technically complete and comprehensive treatment of active inference, including a decomposition of its perception and action selection modes.

Such a decomposition allows us to relate active inference and the inherent motivational principle to other intrinsic motivation paradigms such as empowerment Klyubin et al. Furthermore, we are able to clarify the relation to universal reinforcement learning Hutter, Our treatment is deliberately comprehensive and complete, aiming to be a reference for readers interested in the mathematical fundament. A considerable number of articles have been published on active inference e.

Active inference defines a procedure for both perception and action of an agent interacting with a partially observable environment. The definition of the method, in contrast to other existing approaches e. Most approaches for perception and action selection are generally formed of three steps: The first step involves a learning or inference mechanism to update the agent's knowledge about the consequences of its actions. In a second step, these consequences are evaluated with respect to an agent-internal objective function.

Finally, the action selection mechanism chooses an action depending on the preceding evaluation. In active inference, these three elements are entangled. On one hand, there is the main feature of active inference: the combination of knowledge updating and action selection into a single mechanism.

Variational inference is a way to turn Bayesian inference into an optimization problem which gives rise to an approximate Bayesian inference method Wainwright and Jordan, Behaviour in active inference is thus the result of a variational inference-like process. On the other hand, the function i. This suggests that expected free energy is the only objective function compatible with active inference.

In summary, perception and action in active inference intertwines four elements: variational approximation, inference, action selection, and an objective function. Besides these formal features, active inference is of particular interest for its claims on biological plausibility and its relationship to the thermodynamics of dissipative systems. According to Friston et al. Therefore, it is claimed, actions must minimize variational free energy to resist the dispersion of states of self-organizing systems see also Friston, b ; Allen and Friston, Active inference has also been used to reproduce a range of neural phenomena in the human brain Friston et al.

Furthermore, the principle has been used in a hierarchical formulation as theoretical underpinning of the predictive processing framework Clark, , p. Of particular interest for the present special issue, the representation of probabilities in the active inference framework is conjectured to be related to aspects of consciousness Friston, a ; Linson et al. These strong connections between active inference and biology, statistical physics, and consciousness research make the method particularly interesting for the design of artificial agents that can interact with- and learn about unknown environments.

However, it is currently not clear to which extent active inference allows for modifications. We ask: how far do we have to commit to the precise combination of elements used in the literature, and what becomes interchangeable? One target for modifications is the objective function. In situations where the environment does not provide a specific reward signal and the goal of the agent is not directly specified, researchers often choose the objective function from a range of intrinsic motivations.

Computational approaches to intrinsic motivations Oudeyer and Kaplan, ; Schmidhuber, ; Santucci et al. Intrinsic motivations have been used to enhance behaviour aimed at extrinsic rewards Sutton and Barto, , but their defining characteristic is that they can serve as a goal-independent motivational core for autonomous behaviour generation. This characteristic makes them good candidates for the role of value functions for the design of intelligent systems Pfeifer et al. We attempt to clarify how to modify active inference to accommodate objective functions based on different intrinsic motivations.

This may allow future studies to investigate whether and how altering the objective function affects the biological plausibility of active inference. Another target for modification, originating more from a theoretical standpoint, is the variational formulation of active inference. As mentioned above, variational inference formulates Bayesian inference as an optimization problem; a family of probability distributions is optimized to approximate the direct, non-variational Bayesian solution.

Active inference is formulated as an optimization problem as well. We consequently ask: is active inference the variational formulation of a direct non-variational Bayesian solution? Such a direct solution would allow a formally simple formulation of active inference without recourse to optimization or approximation methods, at the cost of sacrificing tractability in most scenarios. To explore these questions, we take a step back from the established formalism, gradually extend the active inference framework, and comprehensively reconstruct the version presented in Friston et al.

We disentangle the four components of approximation, inference, action selection, and objective functions that are interwoven in active inference. One of our findings, from a formal point of view, is that expected free energy can be replaced by other intrinsic motivations. Our reconstruction of active inference then yields a unified formal framework that can accommodate:. We believe that our framework can benefit active inference research as a means to compare the dynamics induced by alternative action selection principles.

Furthermore, it equips researchers on intrinsic motivations with additional ways for designing agents that share the biological plausibility of active inference. Finally, this article contributes to the research topic: Consciousness in Humanoid Robots, in several ways. First, there have been numerous claims on how active inference relates to consciousness or related qualities, which we outlined earlier in the introduction. The most recent work by Linson et al.

Furthermore, intrinsic motivations including the free energy principle for this argument have a range of properties that relate to or are useful to a range of classical approaches recently summarized as as Good Old-Fashioned Artificial Consciousness GOFAC, Manzotti and Chella, For example, embodied approaches still need some form of value-function or motivation Pfeifer et al. The enactive AI framework Froese and Ziemke, , another candidate for GOFAC, proposes further requirements on how value underlying motivation should be grounded in constitutive autonomy and adaptivity.

Guckelsberger and Salge present tentative claims on how empowerment maximization relates to these requirements in biological systems, and how it could contribute to realizing them in artificial ones. Finally, the idea of using computational approaches for intrinsic motivation goes back to developmental robotics Oudeyer et al.

Whether these Good Old-Fashioned approaches will ultimately be successful is an open question, and Manzotti and Chella asses them rather critically. However, extending active inference to alternative intrinsic motivations in a unified framework allows to combine features of these two approaches. For example it may bring together the neurobiological plausibility of active inference and the constitutive autonomy afforded by empowerment. Our work is largely based on Friston et al. This means many of our assumptions are due to the original paper.

Recently, Buckley et al. Our work here is in as similar spirit but focuses on the discrete formulation of active inference and how it can be decomposed. As we point out in the text, the case of direct Bayesian inference with separate action selection is strongly related to general reinforcement learning Hutter, ; Leike, ; Aslanides et al. This approach also tackles unknown environments with- and in later versions also without externally specified reward in a Bayesian way.

Other work focusing on unknown environments with rewards are e. We would like to stress that we do not propose agents using Bayesian or variational inference as competitors to any of the existing methods. Instead, our goal is to provide an unbiased investigation of active inference with a particular focus on extending the inference methods, objective functions and action-selection mechanisms.

Furthermore, these agents follow almost completely in a straightforward if quite involved way from the model in Friston et al. A small difference is the extension to parameterizations of environment and sensor dynamics. These parameterizations can be found in Friston et al. We note that work on planning as inference Attias, ; Toussaint, ; Botvinick and Toussaint, is generally related to active inference. In this line of work the probability distribution over actions or action sequences that lead to a given goal specified as a sensor value is inferred.

Since active inference also tries to obtain a probability distribution over actions the approaches are related. The formalization of the goal however differs, at least at first sight. How exactly the two approaches relate is beyond the scope of this publication. Going forward, we will first outline our mathematical notation in Section 4.

We then introduce the perception-action loop, which contains both agent and environment in Section 5. In Section 6 we introduce the model used by Friston et al. We then show how to obtain beliefs about the consequences of actions via both direct Bayesian inference Section 6. These beliefs are represented in the form of a set of complete posteriors. Such a set is a common object but usually does not play a prominent role in Bayesian inference.

Here, it turns out to be a convenient structure for capturing the agent' knowledge and describing intrinsic motivations. Under certain assumptions that we discuss in Section 6. We then discuss in Section 7 how those beliefs i. We present standard i. Then we look at different instances of intrinsic motivation functions. For this we explicitly show how our formalism produces the original expression in Friston et al.

Looking at the formulations of other intrinsic motivations it becomes clear that the expected free energy relies on expressions quite similar or identical to those that occur in other intrinsic motivations. This suggests that, at least in principle, there is no reason why active inference should only work with expected free energy as an intrinsic motivation. Finally, in Section 8 formulate active inference for arbitrary action-value functions which include those induced by intrinsic motivations. Modifying the generative model of Section 6. We explain the additional trick that is needed.

In the Appendix we provide some more detailed calculations as well as notation translation tables Appendix C from our own to those of Friston et al. We will explain our notation in more detail in the text, but for readers that mostly look at equations we give a short summary. Note that, Appendix C comprises a translation between Friston et al. Mostly, we will denote random variables by upper case letters e.

An exception to this are random variables that act as parameters of probability distributions. We distinguish different types of probability distributions with letters p, q, r, and d. Here, p corresponds to probability distributions describing properties of the physical world including the agent and its environment, q identifies model probabilities used by the agent internally, r denotes approximations of such model probabilities which are also internal to the agent, and d denotes a probability distribution that can be replaced by a q or a r distribution.

We write conditional probabilities in the usual way, e. In this section we introduce an agent's perception-action loop PA-loop as a causal Bayesian network. This formalism forms the basis for our treatment of active inference. The PA-loop should be seen as specifying the true dynamics of the underlying physical system that contains agent and environment as well as their interactions. In Friston's formulation, the environment dynamics of the PA-loop are referred to as the generative process.

In general these dynamics are inaccessible to the agent itself. Nonetheless, parts of these true dynamics are often assumed to be known to the agent in order to simplify computation see e. We first formally introduce the PA-loop as causal Bayesian network, and then state specific assumptions for the rest of this article.

The new environment state leads to a new sensor value s 1 which, together with the performed action a 1 and the memory state m 1 , influence the next memory state m 2. The loop then continues in this way until a final time step T. Figure 1. First two time steps of the Bayesian network representing the perception-action loop PA-loop. We assume that all variables are finite and that the PA-loop is time-homogeneous 1. Under the assumption of time-homogeneity and the causal dependencies expressed in Figure 1 , the joint probability distribution over the entire PA-loop is defined by:.

In the following we will refer to a combination of initial environment distribution, environment dynamics, and sensor dynamics simply as an environment. Similarly, an agent is a particular combination of initial memory step, memory dynamics, and action generation. The indexing convention we use here is identical to the one used for the generative model see Section 6. However, we assume an efference-like update of the memory. This additional information can be used in inference about the environment state and fundamentally change the intrinsic perspective of an agent. We do not discuss these changes in more detail here but the reader should be aware of the assumption.

These actuators will usually be noisy and the robot will not have access to the final effect of the signal it sends. Similarly, the sensor value is the signal that the physical sensor of the robot produces as a result of a usually noisy measurement, so just like the actuator, the conversion of a physical sensor configuration to a sensor value is part of the sensor dynamics p s t e t which in turn belongs to the environment.

As we will see later, the actions and sensor values must have well-defined state spaces A and S for inference on an internal model to work. This further justifies this perspective. Since we are interested in intrinsic motivations, our focus is not on specific environment or sensor dynamics but almost exclusively on action generation mechanisms of agents that rely minimally on the specifics of these dynamics. In order to focus on action generation, we assume that all the agents we deal with here have the same memory dynamics.

This type of memory is also used in Friston et al. In this respect, it could be called a perfect memory. A more efficient memory use might store only a sufficient statistic of the past data and keep reusable results of computations in memory. Such improvements are not part of this article see e. Formally, the state space M of the memory is the set of all sequences of sensor values and actions that can occur.

Since we have assumed a time-homogeneous memory state space M we must define it so that it contains all these possible sequences from the start. Formally, we therefore choose the union of the spaces of sequences of a fixed length similar to a Kleene-closure :.

Psychology: Ecological Psychology - Routledge

This perfect memory may seem unrealistic and can cause problems if the sensor state space is large e. However, we are not concerned with this type of problem here. Usually, the computation of actions based on past actions and sensor values becomes a challenge of efficiency long before storage limitations kick in: the necessary storage space for perfect memory only increases linearly with time, while, as we show later, the number of operations for Bayesian inference increases exponentially.

Given a fixed environment and the memory dynamics, we only have to define the action generation mechanism p a t m t to fully specify the perception-action loop. This is the subject of the next two sections. In order to stay as close to Friston et al. These are the variational inference and the action selection. We then show how these two building blocks are combined in the original formulation.

We eventually leverage our separation of components to show how the action selection component can be modified, and thus extend the active inference framework. Ultimately, an agent needs to select actions. Inference based on past sensor values and actions is only needed if it is relevant to the action selection. Friston's active inference approach promises to perform action selection within the same inference step that is used to update the agent's model of the environment. In this section, we look at the inference component only and show how an agent can update a generative model in response to observed sensor values and performed actions.

The natural way of updating such a model is Bayesian inference via Bayes' rule. This type of inference leads to what we call the complete posterior. The complete posterior represents all knowledge that the agent can obtain about the consequences of its actions from its past sensor values and actions. In Section 7 we discuss how the agent can use the complete posterior to decide what is the best action to take. Bayesian inference as straightforward recipe is usually not practical due to computational costs.

The memory requirements of the complete posterior update increases exponentially with time and so does the number of operations needed to select actions. To keep the computational tractable, we have to limit ourselves to only use parts of the complete posterior. Furthermore, since the direct expressions even of parts of complete posteriors are usually intractable, approximations are needed. Friston's active inference is committed to variational inference as an approximation technique.

Therefore, we explain how variational inference can be used as an approximation technique. Our setup for variational inference generative model and approximate posterior is identical to the one in Friston et al. We will look at the extension to action inference in Section 7. In the perception-action loop in Figure 1 , action selection and any inference mechanism used in the course of it depends exclusively on the memory state m t. We then have:. The inference mechanism, internal to the action selection mechanism p a m , takes place on a hierarchical generative model or density, in the continuous case.

The generative model we investigate here is a part of the generative model used in Friston et al. The generative models in Friston et al. Note that we are not inferring the causal structure of the Bayesian network or state space cardinalities, but define the generative model as a fixed Bayesian network with the graph shown in Figure 2. It is possible to infer the causal structure see e. Figure 2. An edge that splits up connecting one node to n nodes e. They are either set to past values or, for those in the future, a probability distribution over them must be assumed.

The variables in the Bayesian network in Figure 2 that model variables occurring outside of p a m in the perception-action loop Figure 1 , are denoted as hatted versions of their counterparts. More precisely:. To clearly distinguish the probabilities defined by the generative model from the true dynamics, we use the symbol q instead of p.

To save space, we combine the parameters and hyperparameters by writing. We will see in Section 6. Such an agent would model a future that goes beyond the externally specified last time step T. The generative model assumes that the actions are not influenced by any other variables, hence we have to specify action probabilities. This means that the agent does not model how its actions come about, i. Instead, the agent is interested in the parameters of the environment and sensor dynamics.

It actively sets the probability distributions over past and future actions according to its needs. In practice, it either fixes the probability distributions to particular values by using Dirac delta distributions or to values that optimize some measure. We look into the optimization options in more detail later. Note that the parameters and hyperparameters are standard random variables in the Bayesian network of the model. Also, the rules for calculating probabilities according to this model are just the rules for calculating probabilities in this Bayesian network.

The following procedures including both Bayesian and variational inference can be generalized to also infer hyperparameters. However, our main reference Friston et al. During action generation [i. This data can be plugged into the generative model to obtain posterior probability distributions over all non-observed random variables. These estimations are done by setting:. The resulting posterior probability distribution over all non-observed random variables is then, according to standard rules of calculating probabilities in a Bayesian network:.

Eventually, the agent needs to evaluate the consequences of its future actions. We call each such distribution a Bayesian complete posterior. The complete posteriors are probability distributions over all random variables in the generative model including parameters, latent variables, and future variables. Figure 3. Predictions for future sensor values can be obtained by marginalising out other random variables e. All intrinsic motivations discussed in this article evaluate future actions based on quantities that can be derived from the corresponding complete posterior.

It is important to note that the complete posterior can be factorized into a term containing the influence of past sensor values and actions data. Using the conditional independence. This equation represents the desired factorization. This formulation separates complete posteriors into a predictive and a posterior factor. The predictive factor is given as part of the generative model Equation 8. This factor contains the dependence of the complete posterior on future actions.