Subscribe to this blog

Follow by Email

Understanding understanding – could an A.I. cook meth?

What would it take to say that an
artificial system “understands” something? What do we mean when we say humans
understand something? I asked those questions on Twitter recently and it
prompted some very interesting debate, which I will try to summarise and expand
on here.

Several people complained that the
questions were unanswerable until I had defined “understanding”, but that was exactly
the problem – I didn’t have a good understanding of what understanding means.
That’s what I was trying to unpick.

I know, of course, that there is a rich
philosophical literature on this question, but the bits of it I’ve read were
not quite getting at what I was after. I was trying to get to a
cognitive or computational framework defining the parameters that constitute
understanding in a human, such that we could operationalise it to the point
that we could implement it in an artificial intelligence.

So, rather than starting with a definition,
let me start with an illustration and see if we can use it to tease out the
parameters that characterise understanding, especially the difference between
understanding something and just knowing something or being able to do
something.

I
know it when I see it

Here goes (and if you haven’t watched
Breaking Bad, I can only apologise for my choice of example):

Jesse Pinkman knows how to cook crystal
meth. But Walter White understands
the process. Jesse can follow the steps of the protocol. He knows that he
should do step 1, then step 2, then step 3 – he can carry out the algorithm.
Walter knows why they do step 1, then
step 2, then step 3. Whatever constitutes that understanding, it is easily
recognisable to us because he invented the protocol, because he can teach it to
someone else, and because he can modify it as needed to scale it up or if
various ingredients become hard to obtain.

So, what’s the difference? What parameters
differ between Jesse’s brain and Walt’s brain when it comes to their knowledge
of cooking meth? Why does Walt’s state constitute understanding, while Jesse’s
does not?

Jesse’s knowledge is specific, isolated,
fragmentary. He understands that when
you add substance A to substance B you produce substance C. Walt’s knowledge,
by contrast is situated in a much wider and deeper context. He understands how substance A reacts with substance B
to produce substance C. That’s because he knows the properties of these
substances that drive their reactivity. So, he has a level of knowledge that is
more fundamental and more general, to which he can relate more specific facts.

And, beyond that, Walt understands why those substances have those
properties. He has even more fundamental levels of knowledge about types of
substances – not just those particular ones – and about their physical
structures at a deeper level that produce their chemical properties. That’s why
he could come up with the protocol in the first place and that’s why he can
change it as circumstances demand.

So, we have a hierarchy of understanding – that, how, why. At each level
we are situating some facts in the context of a wider body of knowledge, but
also, crucially, a deeper level of
knowledge. But not one that is reductionist – that is bogged down in the
details of a lower ontological level. Instead, it is one that sees the
important principles at that lower level and how they determine the properties
at the higher level.

The simple accumulation of knowledge – the
addition of extra facts – does not confer deeper understanding. What is
required is the abstraction of general principles from all that knowledge, so
that new facts can be situated within a logical, coherent framework – one that
crucially entails causal relationships.

Responses
from the hive-mind

This ties in to a lot of the responses I
got to my questions on Twitter. These mostly converged onto ideas of
abstraction – of categories, principles, causal relationships – and abduction,
or abductive reasoning – forming a hypothesis from a set of observations, or
thinking about why some things were
observed or some facts hold true. It’s basically guessing, but, here, in a way
that is informed by wider knowledge and experience.

But there were a lot of other parameters
suggested, and I have organised those below in a kind of ascending order. They
start with those needed for a very simple kind of understanding, which might be
programmed into an artificial intelligence. (Indeed, A.I. is exceptionally good at some of them).

And they ascend to properties that might be
thought of as necessary for a higher kind of understanding, which may
ultimately depend on internal models and representations, consciousness,
awareness, and our own histories of embodied experience. And, who knows, maybe
those properties could be built into an artificial intelligence too and maybe they
are exactly the type of properties that would be required for what some people
might consider “real” understanding, as opposed to a convincing simulation of
it.

The elements
of understanding

1.Categorisation – building a
kind of internal database of types of
things, with knowledge of the properties that define each type. These
necessarily have a hierarchical nature. So, Rover is a golden retriever, which
is a subtype of dog. Dogs are characterised by A, B, and C – these collective
properties form the schema or concept of “dog”, while properties D, E, and F
characterise the subtype of golden retrievers. And, of course, dogs are
themselves subtypes of mammals, which are subtypes of animals, and there are
schemas for those levels as well.

2.Generalisation – categories
allow you to generalise. When you see a Corgi for the first time you may still
be able to recognise it as a type of dog and make some predictions about its
behaviour, because you understand some general things about dogs.

3.Abstraction – the activity that
enables you to build categories. You have to be able to see which are the
properties that are essential to define a category at each level, and which are
incidental.

4.Compression – a related way to
look at abstraction. How much detailed information can you throw away when
moving from one level to the next, while retaining the important properties? Being
able to see the forest for the trees depends on extracting trends or patterns,
while ignoring a lot of the details, much of which may be noise.

5.Pattern recognition – again,
this is related to abstraction and compression. The ability to detect
statistical regularities in a set of observations, but hopefully without over-fitting noise.

6.Abduction – drawing hypotheses
about the state of a system. Now we’re starting to go beyond just the
properties of objects, to consider the properties of the relations between objects.

7.Causal reasoning – this is
where we really get into the meat of it, in my opinion, as we begin to
understand systems, not just things or types of things. If I understand a
system, I should be aware of the causal relations between its elements and the
causal dependencies of system behaviour on those relations. I should have an
understanding of causality both within levels and between levels.

8.Counterfactual reasoning – one
way to demonstrate such understanding is to be able to consider counterfactuals
and their likely consequences. If such and such were NOT the case, how would
that affect the operation of the system? To perform such reasoning, you have to
know which details are important for the causal dynamics of the system.

9.Prediction – this is an
extension of counterfactual reasoning. If you really understand a system you
should be able to predict what would happen if you changed something about it,
or predict how it would behave in some new scenario.

10.Manipulation – if we can manipulate and control a system with
predictable outcomes, many would say that demonstrates understanding. It
certainly demands knowledge of the causal relations at the level of operation
one is aiming to control, though it is possible to achieve without a deeper
understanding of the underlying principles at play.

11.Invention – this is an even deeper level of understanding. It
requires not just knowing the causal relations in a given system but understanding
the more abstract principles underlying those relations. Those then become
elements that can be reconfigured into new arrangements to carry out novel
functions or operations.

12.Analogy – again, this is a question of abstraction of deeper
principles. First, seeing that some particular causal relation between two
things is a relation between two types
of things. Then seeing that that particular relation is itself a type of relation with more general
properties. This allows you to go beyond knowledge or understanding of a
particular system to understanding of systems in general.

13.Mathematical description – some would argue you haven’t really
understood a system until you can express it in mathematical terms, the most
abstract level of reasoning. I think that goes too far – most of our
understanding is intuitive or logical, without involving formal mathematics.
However, mathematical expression can reveal deep correspondences that might not
otherwise be apparent, such as the correspondence between Shannon information
and thermodynamic entropy, or between predictive inference in perception and
Bayesian statistical reasoning.

14.Awareness? – Do you have to know
you understand something to understand it? I don’t see any reason why you
would, but, again, some people might have this as a criterion.

15.Articulability? – Being able to explain something to someone else or
teach them how to do something can certainly be a good test of understanding,
but I don’t know if it is a necessary criterion – maybe just an additional way
we recognise it.

16.Embodied phenomenological experience? – Do you have to have lived
experience of something to truly understand it? This feels very vague, but
hints at the idea that one thing that may limit artificial intelligence is that
its knowledge is not gained by active exploration in the world. Perhaps
disembodied knowledge will never reach a human level of understanding that
comes with being embodied agents in the world. Like I said, it’s vague but
links to the question of how information comes to have meaning, and whether
meaning is required for true understanding.

17.Perspective? – does understanding necessarily entail some subjective perspective?
This is really just the context of the life history of the organism or agent
that is doing the understanding. Clearly it can be a barrier to shared understanding,
when there are implicit assumptions of wider context, prior positions, or
values that are not themselves shared.

I am sure there are other criteria or
properties that could be considered, but I think those capture most of the
responses I got on Twitter. The key thing, to my mind, seems to be embedding
knowledge at one level in the context of knowledge of another level, in
particular knowledge of the causal relations of a system.

Now, the question is whether that
discussion gets us any closer to knowing what we’d need to build into an A.I.
to make it capable of real understanding.

Artificial
intelligence

Some people would argue that A.I. is
already capable of some kind of understanding. For example, DeepMind’s
incredibly impressive engine AlphaZero knows how to play chess. But does it
understand it? It certainly seems to, if you measure it by its success, and
also by the kinds of moves it makes. Indeed, some people argued that its style
of play – the “beauty” of some of its moves and strategies – shows more
understanding than any human has ever demonstrated.

On the other hand, its knowledge is all at
one level. It doesn’t know chess is a game – it doesn’t know what a game is. It
doesn’t know it’s a metaphor for war. It doesn’t relate its knowledge of chess
to its knowledge of anything else, because it doesn’t know about anything else.

Maybe all it would need to develop this
understanding is to be exposed to lots more information. It clearly has the
computational power to recognise patterns in massive amounts of data and to
make predictions from massive amounts of prior experience. But is that how
humans develop understanding? Or do they do something different? Do they
actively extract rules and principles with less data? Are they wired to make
abstractions and draw general inferences? Is our neural architecture
particularly attuned to causal structure?

Perhaps it is the hierarchical architecture
of the cerebral cortex that fosters the development of understanding, with each
level being able to abstract more general principles by integrating across multiple
units at the level below. This gives more opportunity to see emergent causal
relations, to draw more distant analogies, to derive deeper principles.

Indeed, the expansion of the human cortex
is characterised not by the expansion of individual areas, but by the addition of more areas, particularly of association cortex – the bits that integrate
information from lower levels. We don’t just have more raw brainpower –
evolution has extended the hierarchy of our neural architecture to higher and
higher levels.

Of course, A.I.’s like AlphaZero have a
hierarchical structure, with multiple layers, but it is all devoted to one
thing. Maybe if we hooked a bunch of them up in parallel, each doing different
things (say, playing different games), and then added another layer on top,
that layer could extract more general principles (like aspects of game theory
in general).

I don’t know whether the preceding
discussion is really any use in thinking more precisely about how to
operationalize and implement understanding in artificial systems, but maybe
some people from that field will chime in. (This excellent blog on the interactions between the A.I. field and neuroscience is relevant).

Understanding
in neuroscience

The discussion above also resonates with
another question I asked recently on Twitter. I had been browsing the articles
in the latest issue of Neuron, which span all levels of neuroscience, from the
molecular up to human cognition, and wondered what would it take for a single
person to really understand all of them?

This is one of the flagship journals of our
field, yet most neuroscientists, myself included, can only really understand a
small slice of the papers in each issue. This reflects the history of the
field, which is, in fact, a loose agglomeration of many traditionally distinct
disciplines – molecular and cellular neuroscience, developmental neurobiology,
genetics, animal behaviour, electrophysiology, pharmacology, neuroanatomy,
circuit and systems neuroscience, cognitive science, computational
neuroscience, neuroimaging, psychology, psychiatry, neurology…

The reason the discussion of understanding
is particularly relevant to neuroscience is that this field is by its nature hierarchical.
These different disciplinary approaches are not defined solely by their methods
but by their objects and levels of analysis, from single cells, to
microcircuits, to extended systems and brain regions, to the ultimate emergent
level of mind and behaviour.

The key challenge for any neuroscientist is
to see across these levels. To understand how the dynamics of molecules within
a cell determine its electrophysiological properties, in ways that determine
its role in information processing within a microcircuit, which is itself a
subcomponent of a larger circuit, the activity of which has some mental correlates, and so on. This is a challenge that one would hope could be met by new
generations of students who do not have the baggage of having been educated in one of the historical
silos.

But it is very easy to get bogged down in
the details at each level. What is important is to abstract the general
principles at play, which are required to understand the emergent functions of
the next level up.

In fact, I think that’s not just the
approach that we need to take to understand the nervous system as scientists. I
think it’s the approach that the nervous system itself takes. From neuron to
neuron, from region to region, from level to level, information is abstracted
from all the noisy details, and this information has meaning that is understood in the context of
lots of other information.

Perhaps it’s anthropomorphic to say that
each neuron is trying to understand the neurons talking to it or that each
level is trying to understand the one below it. But, then again, perhaps not –
perhaps there is a deep mathematical correspondence between understanding at
the psychological level and even the simplest elements of information
processing at the neural level.

With thanks to all those who engaged on Twitter - you're the reason it's such a great platform for scientific and philosophical discussions!

Comments

Post a Comment

Popular posts from this blog

Can molecular memories of our ancestors’
experiences affect our own behaviour and physiology? That idea has certainly
grabbed hold of the public imagination, under the banner of the seemingly
ubiquitous buzzword “epigenetics”. Transgenerational epigenetic inheritance is
the idea that a person’s experiences can somehow mark their genomes in ways
that are passed on to their children and grandchildren. Those marks on the
genome are then thought to influence gene expression and affect the behaviour
and physiology of people who inherit them. The way this notion is referred to – both in
popular pieces and in the scientific literature – you’d be forgiven for
thinking it is an established fact in humans, based on mountains of consistent,
compelling evidence. In fact, the opposite is true – it is based on the
flimsiest of evidence from a very small number of studies with very small
sample sizes and serious methodological flaws. [Note that there is, by contrast,
very good evidence for this kind…

I recently wrote a blogpost examining the
supposed evidence for transgenerational epigenetic inheritance (TGEI) in
humans. This focused specifically on a set of studies commonly cited as
convincingly demonstrating the phenomenon whereby the experiences of one
generation can have effects that are transmitted, through non-genetic means, to
their offspring, and, more importantly, even to their grandchildren. Having
examined what I considered to be the most prominent papers making these claims,
I concluded that they do not in fact provide any evidence supporting that idea,
as they are riddled with fatal methodological flaws. While the scope of that piece was limited
to studies in humans, I have also previously considered animal studies making
similar claims, which suffer from similar methodological flaws (here and here).
My overall conclusion is that there is effectively no evidence for TGEI in
humans (contrary to widespread belief) and very little in mammals more
generally (with one very…

GWAS (genome-wide association studies) for
psychiatric illnesses may be about to become a victim of their own success. The
idea behind these studies is that common genetic variation – ancient mutations
that segregate in the population – may partly underlie the high heritability of
common psychiatric and neurological disorders, such as schizophrenia, autism,
epilepsy, ADHD, depression, and so on. The accumulating evidence from over ten
years of GWAS strongly supports that idea, with many hundreds of such risk
variants now having been identified. The problem is it’s not at all clear what
to do with that information. GWAS are a method to carry out a kind of
genetic epidemiology, based on a simple premise – if a particular genetic
variant at some position in the genome (say an “A” base, as opposed to a “T” at position 236,456 on chromosome 9) – is associated with an increased risk
of some condition, then the frequency of the “A” version should be higher in
people with the condition than pe…