Friday, March 13, 2015

-The theory of reinforcement learning provides a normative account1, deeply rooted in psychological2 and neuroscientific3 perspectives on animal behaviour, of how agents may optimize their control of an environment. To use reinforcement learning successfully in situations approaching real-world complexity, however, agents are confronted with a difficult task: they must derive efficient representations of the environment from high-dimensional sensory inputs, and use these to generalize past experience to new situations. Remarkably, humans and other animals seem to solve this problem through a harmonious combination of reinforcement learning and hierarchical sensory processing systems4, 5, the former evidenced by a wealth of neural data revealing notable parallels between the phasic signals emitted by dopaminergic neurons and temporal difference reinforcement learning algorithms3. While reinforcement learning agents have achieved some successes in a variety of domains6, 7, 8, their applicability has previously been limited to domains in which useful features can be handcrafted, or to domains with fully observed, low-dimensional state spaces. Here we use recent advances in training deep neural networks9, 10, 11 to develop a novel artificial agent, termed a deep Q-network, that can learn successful policies directly from high-dimensional sensory inputs using end-to-end reinforcement learning. We tested this agent on the challenging domain of classic Atari 2600 games12. We demonstrate that the deep Q-network agent, receiving only the pixels and the game score as inputs, was able to surpass the performance of all previous algorithms and achieve a level comparable to that of a professional human games tester across a set of 49 games, using the same algorithm, network architecture and hyperparameters. This work bridges the divide between high-dimensional sensory inputs and actions, resulting in the first artificial agent that is capable of learning to excel at a diverse array of challenging tasks.

***

This demo follows the description of the Deep Q Learning algorithm described in
Playing Atari with Deep Reinforcement Learning,
a paper from NIPS 2013 Deep Learning Workshop from DeepMind. The paper is a nice demo of a fairly
standard (model-free) Reinforcement Learning algorithm (Q Learning) learning to play Atari games.

In this demo, instead of Atari games, we'll start out with something more simple:
a 2D agent that has 9 eyes pointing in different angles ahead and every eye senses 3 values
along its direction (up to a certain maximum visibility distance): distance to a wall, distance to
a green thing, or distance to a red thing. The agent navigates by using one of 5 actions that turn
it different angles. The red things are apples and the agent gets reward for eating them. The green
things are poison and the agent gets negative reward for eating them. The training takes a few tens
of minutes with current parameter settings.

Over time, the agent learns to avoid states that lead to states with low rewards, and picks actions
that lead to better states instead.

Princeton University physicists built a powerful imaging device called a
scanning-tunneling microscope and used it to capture an image of an
elusive particle that behaves simultaneously like matter and antimatter.
To avoid vibration, the microscope is cooled close to absolute zero and
is suspended like a floating island in the floor above. The setup
includes a 40-ton block of concrete, which is visible above the
researchers. The research team includes, from left, graduate student
Ilya Drozdov, postdoctoral researcher Sangjun Jeon, and professors of
physics B. Andrei Bernevig and Ali Yazdani. (Photo by Denise Applewhite, Office of Communications)

Majorana fermions are predicted to
localize at the edge of a topological superconductor, a state of matter
that can form when
a ferromagnetic system is placed in proximity to
a conventional superconductor with strong spin-orbit interaction. With
the
goal of realizing a one-dimensional topological
superconductor, we have fabricated ferromagnetic iron (Fe) atomic chains
on
the surface of superconducting lead (Pb). Using
high-resolution spectroscopic imaging techniques, we show that the onset
of
superconductivity, which gaps the electronic
density of states in the bulk of the Fe chains, is accompanied by the
appearance
of zero energy end states. This spatially
resolved signature provides strong evidence, corroborated by other
observations,
for the formation of a topological phase and
edge-bound Majorana fermions in our atomic chains. Observation of Majorana fermions in ferromagnetic atomic chains on a superconductor

In the early seventies my own Father was a accountant who made use of this system of data storage and while having the equipment for card production and sorting to list his clients, he made use of the rooms in larger office tower that were in hermetically sealed rooms to prevent corruption.

Above left: The row of tape drives for the UNIVAC I computer. Above right: The IBM 3410 Magnetic Tape Subsystem, introduced in 1971.

The progressive feature of computation data development is always an interesting one to me. The age of its users may be of interest in judging how far back certain features of the development mechanical type set is used helps to portray the age of its users. It's evolutionary history.

I mean using Magnetic tapes in terms of data storage although used in type set cards punches as 1's and O's was the precursor to what we see today in the google data centre. Can you imagine how large the data room would be needed in order to compress what you see today in our historical development of the seventies?

How science data storage is effected by what would constrain the amount of information given from experimental procedures? We could not do much with efficiency given that constraint.

It was designed by the RobotCub Consortium, of several European universities and is now supported by other projects such as ITALK.[1] The robot is open-source, with the hardware design, software and documentation all released under the GPL license. The name is a partial acronym, cub standing for Cognitive Universal Body.[2] Initial funding for the project was €8.5 million from Unit E5 – Cognitive Systems and Robotics – of the European Commission's Seventh Framework Programme, and this ran for six years from 1 September 2004 until 1 September 2010.[2]

The motivation behind the strongly humanoid design is the embodied cognition
hypothesis, that human-like manipulation plays a vital role in the
development of human cognition. A baby learns many cognitive skills by
interacting with its environment and other humans using its limbs and
senses, and consequently its internal model of the world is largely
determined by the form of the human body. The robot was designed to test
this hypothesis by allowing cognitive learning scenarios to be acted
out by an accurate reproduction of the perceptual system and
articulation of a small child so that it could interact with the world
in the same way that such a child does.[3]

Embodied cognition is a topic of research in social and cognitive psychology, covering issues such as social interaction and decision-making.[2] Embodied cognition reflects the argument that the motor system
influences our cognition, just as the mind influences bodily actions.
For example, when participants hold a pencil in their teeth engaging the
muscles of a smile, they comprehend pleasant sentences faster than
unpleasant ones.[3]
And it works in reverse: holding a pencil in their teeth to engage the
muscles of a frown increases the time it takes to comprehend pleasant
sentences.[3]

The mind-body problemis a philosophical problem arising in the fields of metaphysics and philosophy of mind.[2] The problem arises because mental phenomena arguably differ, qualitatively or substantially, from the physical body on which they apparently depend. There are a few major theories on the resolution of the problem. Dualism is the theory that the mind and body are two distinct substances,[2] and monism is the theory that they are, in reality, just one substance. Monist materialists (also called physicalists) take the view that they are both matter, and monist idealists take the view that they are both in the mind. Neutral monists take the view that both are reducible to a third, neutral substance.

A dualist view of reality may lead one to consider the corporeal as little valued[3] and trivial. The rejection of the mind–body dichotomy is found in French Structuralism, and is a position that generally characterized post-war French philosophy.[5]
The absence of an empirically identifiable meeting point between the
non-physical mind and its physical extension has proven problematic to
dualism and many modern philosophers of mind maintain that the mind is
not something separate from the body.[6] These approaches have been particularly influential in the sciences, particularly in the fields of sociobiology, computer science, evolutionary psychology and the various neurosciences.[7][8][9][10]

History

A computational model going beyond Turing machines was introduced by Alan Turing in his 1938 PhD dissertation Systems of Logic Based on Ordinals.[2] This paper investigated mathematical systems in which an oracle was available, which could compute a single arbitrary (non-recursive) function from naturals to naturals. He used this device to prove that even in those more powerful systems, undecidability is still present. Turing's oracle machines are strictly mathematical abstractions, and are not physically realizable.[3]

Hypercomputation and the Church–Turing thesis

The Church–Turing thesis
states that any function that is algorithmically computable can be
computed by a Turing machine. Hypercomputers compute functions that a
Turing machine cannot, hence, not computable in the Church-Turing sense.
An example of a problem a Turing machine cannot solve is the halting problem.
A Turing machine cannot decide if an arbitrary program halts or runs
forever. Some proposed hypercomputers can simulate the program for an
infinite number of steps and tell the user whether or not the program
halted.

Hypercomputer proposals

A Turing machine that can complete infinitely many steps.
Simply being able to run for an unbounded number of steps does not
suffice. One mathematical model is the Zeno machine (inspired by Zeno's paradox).
The Zeno machine performs its first computation step in (say) 1 minute,
the second step in ½ minute, the third step in ¼ minute, etc. By
summing 1+½+¼+... (a geometric series) we see that the machine performs infinitely many steps in a total of 2 minutes. However, some[who?] people claim that, following the reasoning from Zeno's paradox, Zeno machines are not just physically impossible, but logically impossible.[4]

Turing's original oracle machines, defined by Turing in 1939.

In mid 1960s, E Mark Gold and Hilary Putnam independently proposed models of inductive inference (the "limiting recursive functionals"[5] and "trial-and-error predicates",[6] respectively). These models enable some nonrecursive sets of numbers or languages (including all recursively enumerable
sets of languages) to be "learned in the limit"; whereas, by
definition, only recursive sets of numbers or languages could be
identified by a Turing machine. While the machine will stabilize to the
correct answer on any learnable set in some finite time, it can only
identify it as correct if it is recursive; otherwise, the correctness is
established only by running the machine forever and noting that it
never revises its answer. Putnam identified this new interpretation as
the class of "empirical" predicates, stating: "if we always 'posit' that
the most recently generated answer is correct, we will make a finite
number of mistakes, but we will eventually get the correct answer.
(Note, however, that even if we have gotten to the correct answer (the
end of the finite sequence) we are never sure that we have the correct answer.)"[6]L. K. Schubert's 1974 paper "Iterated Limiting Recursion and the Program Minimization Problem" [7] studied the effects of iterating the limiting procedure; this allows any arithmetic
predicate to be computed. Schubert wrote, "Intuitively, iterated
limiting identification might be regarded as higher-order inductive
inference performed collectively by an ever-growing community of lower
order inductive inference machines."

A proposed technique known as fair nondeterminism or unbounded nondeterminism may allow the computation of noncomputable functions.[9]
There is dispute in the literature over whether this technique is
coherent, and whether it actually allows noncomputable functions to be
"computed".

It seems natural that the possibility of time travel (existence of closed timelike curves
(CTCs)) makes hypercomputation possible by itself. However, this is not
so since a CTC does not provide (by itself) the unbounded amount of
storage that an infinite computation would require. Nevertheless, there
are spacetimes in which the CTC region can be used for relativistic
hypercomputation.[10] Access to a CTC may allow the rapid solution to PSPACE-complete problems, a complexity class which while Turing-decidable is generally considered computationally intractable.[11][12]

In 1994, Hava Siegelmann
proved that her new (1991) computational model, the Artificial
Recurrent Neural Network (ARNN), could perform hypercomputation (using
infinite precision real weights for the synapses). It is based on
evolving an artificial neural network through a discrete, infinite
succession of states.[17]

The infinite time Turing machine is a generalization of the
Zeno machine, that can perform infinitely long computations whose steps
are enumerated by potentially transfinite ordinal numbers.
It models an otherwise-ordinary Turing machine for which non-halting
computations are completed by entering a special state reserved for
reaching a limit ordinal and to which the results of the preceding infinite computation are available.[18]

Jan van Leeuwen and Jiří Wiedermann wrote a 2000 paper[19] suggesting that the Internet should be modeled as a nonuniform computing system equipped with an advice function representing the ability of computers to be upgraded.

A symbol sequence is computable in the limit if there is a finite, possibly non-halting program on a universal Turing machine that incrementally outputs every symbol of the sequence. This includes the dyadic expansion of π and of every other computable real,
but still excludes all noncomputable reals. Traditional Turing machines
cannot edit their previous outputs; generalized Turing machines, as
defined by Jürgen Schmidhuber,
can. He defines the constructively describable symbol sequences as
those that have a finite, non-halting program running on a generalized
Turing machine, such that any output symbol eventually converges, that
is, it does not change any more after some finite initial time interval.
Due to limitations first exhibited by Kurt Gödel (1931), it may be impossible to predict the convergence time itself by a halting program, otherwise the halting problem could be solved. Schmidhuber ([20][21]) uses this approach to define the set of formally describable or constructively computable universes or constructive theories of everything. Generalized Turing machines can solve the halting problem by evaluating a Specker sequence.

In 1970, E.S. Santos defined a class of fuzzy logic-based "fuzzy algorithms" and "fuzzy Turing machines".[24]
Subsequently, L. Biacino and G. Gerla showed that such a definition
would allow the computation of nonrecursive languages; they suggested an
alternative set of definitions without this difficulty.[25] Jiří Wiedermann analyzed the capabilities of Santos' original proposal in 2004.[26]

Dmytro Taranovsky has proposed a finitistic
model of traditionally non-finitistic branches of analysis, built
around a Turing machine equipped with a rapidly increasing function as
its oracle. By this and more complicated models he was able to give an
interpretation of second-order arithmetic.[27]

Analysis of capabilities

Many hypercomputation proposals amount to alternative ways to read an oracle or advice function embedded into an otherwise classical machine. Others allow access to some higher level of the arithmetic hierarchy. For example, supertasking Turing machines, under the usual assumptions, would be able to compute any predicate in the truth-table degree containing or . Limiting-recursion, by contrast, can compute any predicate or function in the corresponding Turing degree, which is known to be . Gold further showed that limiting partial recursion would allow the computation of precisely the predicates.

Criticism

Martin Davis, in his writings on hypercomputation [39][40]
refers to this subject as "a myth" and offers counter-arguments to the
physical realizability of hypercomputation. As for its theory, he argues
against the claims that this is a new field founded in 1990s. This
point of view relies on the history of computability theory (degrees of
unsolvability, computability over functions, real numbers and ordinals),
as also mentioned above.Andrew Hodges wrote a critical commentary[41] on Copeland and Proudfoot's article[1].

^"Let
us suppose that we are supplied with some unspecified means of solving
number-theoretic problems; a kind of oracle as it were. We shall not go
any further into the nature of this oracle apart from saying that it
cannot be a machine" (Undecidable p. 167, a reprint of Turing's paper Systems of Logic Based On Ordinals)