1.
Real-time signal recognition in the time domain as a Humanlike
Perception System

For over 15 years Dr. G. Roscher has been developing
a method for the real-time recognition of signals in the time
domain - in contrast to the established frequency domain methods
such as the Discrete Fourier Transform (DFT). This new method is
based on the “old” peak measurement and evaluates each
event in the signal as:

- extreme amplitude,

- extreme slope (optional),

- and as an further option each extreme curvature.

Each event is transformed into a data structure named
Virtual Source (in German: Virtuelle Quelle –
VQ). The result of this transformation is the description of
the signal as a sequence of VQs. Further steps build up a
hierarchical system of chained lists of VQs, named

- SuperPeaks,

- Cycles and

- Classes.

This description of the signal can be easily
manipulated by mathematical methods and can be easily recognized.
Previous researchers have not appreciated the sophisticated
performance in the time domain of the hard-wired parallel processing
human visual or auditory system and brain. All the established
groups have used frequency domain methods! These methods are
approximations which lack the accuracy and performance of the human
recognition system.

This high performance requires the application of
time domain methods for signal recognition of the highest accuracy,
which employs the latest technologies in the fields of Data Base
(DBS) and Knowledge Management Systems (KMS). Each incoming signal
is stored in the DBS. These signals are transformed in the VQs,
segmented and indexed by the time and by the segment number. The
classification is achieved using KMS for the acquisition of personal
knowledge in direct communication between the user and the KMS.

2. Description of the methods and presentation of
significant examples

2.1
Simulation of the Human Visual System

In the Human Visual System, cones and rods are
connected by neurons to the brain. Cones, rods and connecting neurons
are powerful systems; they carry out complex and complicated
information processes for signal analysis and recognition in the time
domain at different levels. These information processes are
modeled by the method of Virtual Sources VQs.

As Albert Einstein said

“We should make things as
simple as possible, but not simpler”.

2.2
The model of the Virtual Sources

The VQs on different levels are characterized in
general by:

Time of occurrence
t,

Duration
d,

Amplitude
a [, aR,
aG, aB]

Amplitude difference
p [, pR,
pG, pB]

Extreme slope
w [, wR,
wG, wB]

Extreme curvature
k [, kR,
kG, kB]

Virtual localization of the channel or the
source [x, y [, z]]

Chains of pointers to VQs of the same and lower
levels.

The recognition is performed by using the following
methods:

- Fuzzy-logic for pattern recognition,

- Neural networks for feature extraction and
classification,

- Evolutionary algorithms for classification,

- Dynamic lists for internal management of the signal
description in real-time,

- Data Base System for external management of the
signal description.

- Knowledge Management System for acquisition of
expert knowledge.

- Special heuristic algorithms for solutions without
algorithms.

The classification of the evolutionary algorithm is
described for this simple example:

The classification at the first level led in a
fuzzy manner to two classes: Class1, named “VQClass black
to white” and Class2, named “VQClass white to
black”. In the second level, all VQs generated at the
same time are collected and used to build the two lines described by
the VQs of level 1. That will be the recognized object named “VQ
White line on the blackboard”. Some times later, the
same object appears to be translated in a defined direction. The
description of this object is in a fuzzy manner the same as the “VQ
White line on the blackboard”.

The evolutionary algorithm for automatic
classification worked in the following way:

The first recognised object is stored in the so
called garbage class. Each following object is compared with the
members of the garbage class. If the new object matches with one
member of the garbage class, to a defined degree they found a new
class. These two objects are the parents of the new class and will
never be eliminated from the class description. Each new object which
matches with the members of this class is added to the class
description, up to tmax
members in the class description. tmax
will be greater than two. If
tmax+1members are in the class description, one
member is eliminated by a heuristic algorithm. If one object does not
match with an established class description, it is checked against
members of the garbage class. If it matches with one member in the
garbage class, the two objects are the parents of a new class, else
the new incoming object is stored in the garbage class.

2.3
The one channel ECG as example for the signal s = f(t)

The user can evaluate the VQs as description of real
objects and can name these objects or classes by using interactive
methods in a graphical user interface. The notion of naming objects
which has a computer aided description by the user builds a new type
of knowledge management system. This is demonstrated by the
application of the classification and recognition of single channel
ECG-signals in Fig. 2 by using the wireless portable PhysioCord
and the high performance stationary ECG-System HeartScope.

Fig.2: The ECG as example for automatic
classification and recognition

The application of the method isperformed for the single channel ECG using a high
sampling accuracy (sampling rate: 512 samples / sec). In
Fig. 2 the classification of single channel ECG is presented (only
the first red marked channel EKG2 is analysed). The red
numbers signify the beat number of
the 24 hour ECG, going from 70021
to 70037
in Fig. 2. The blue
numbers are the Inter Beat Interval
(IBI) and go from 360
in minimum to 493
in this example. This high variation is generated by the pathological
heart beat 70022.
The green numbers are
the automatic generated classes and go from 1
= garbage class to class
37 in this example.

Class 2(Klasse 2, Anzahl = number = 35064 beats) and
Class 3
(Klasse 3, Anzahl = number = 41999 beats) are the normal heartbeats,
differentiated by the slope of R and S. The representation of
these normal heart-beats is quite different from the
representation presented in the literature. The high flexibility of
feature extraction, recognition and evolutionary algorithm for
classification opened this innovative way.

Class 15are pathological heartbeats, 11 in one day. When
Heartbeat 6161
appears first, it doesn’t match with one of the existent
classes and is stored in the garbage class. The Heartbeat 9408
matches first with heartbeat 6161,
and they form class
15. Only 5 heartbeats are used as
Templates for the class description (tmax
= 5). Heartbeat 70022
is classified in class
15, and is a member of the class
description and is presented in the ongoing ECG, in ECG-channel EKG2.

The experienced physician can evaluate the automatic
generated classes, can name the classes, can delete non-significant
classes and can introduce his knowledge to the automatically
generated signal description.

2.4
The EEG as example for the dynamic 3D signal s = f(x, y, z, t)

The EEG system
BrainScopeconsists of a
special amplifier system for high quality signal detection in open
field conditions during communicative situations. A high performance
computer system which is capable of processing the huge amounts of
data produced by a multi channel EEG record to gain information in
real-time has also been developed. Algorithms for recognition of
events in single channels are implemented in the first level of the
computer system. High performance 3D image processing algorithms are
used in the second level, interpreting the sampled values of each
channel as pixels of the image, from 256 to 2000 times per second.
This method describes the EEG activity as sequences of VQs with
parameters of amplitude, time and space.

The network of
two or more Personal Computers (PCs) is co-ordinated through the
computer system for presentation of EEG activity and control.
Multi-media approaches to the application of
psychological tests are possible through the user interface including
tests in media of sound, words, pictures and moving pictures. These
tests can be arranged and carried out in computer controlled
sequences and modified by user interactions. Tools are also provided
to allow the user to create his own tests. These methods are
integrated into the powerful Graphic User Interface and use a Data
Base System. Incorporated into this User Interface are state of the
art EEGSYS algorithms from the NIMH (Washington / USA) for mappings,
FFT, etc.

The BrainScope demonstrates the impacts and
applications of the new strategy for EEG investigation in
communicative situations between:

- patient and physician for subjective evaluation,

- patient and information technology for stimulation
and acquisition of signals and reactions,

- physician and information technology for
quantitative analysis of signals and reactions.

The major advantage of this new
strategy is that the three processes can be carried out

in real-time.

It optimises the capacity of humans to
interpret information with the capability of modern information
technology to manipulate and process data. It therefore requires use
by an experienced and trained operator who can make accurate
observations during the process of an investigation. The user can,
for example, click on a significant EEG pattern (this makes it a
further recognisable phenomenon through fuzzy logic) and correlate it
with his own observations. The computer system recognises this EEG
activity, i.e. it interprets this as a possible description of the
state of the brain, sets a defined stimulus and recognises and
evaluates the Event Related Potential (ERP) immediately.

Fig. 3: The EEG for a complex, 3D signal

ERP are EEG-changes, related to a particular event
(e. g. acoustic or visual stimulus or motor reactions) and give hints
to the underlying information process.

In Fig 3 the recognition of Event Related Potentials
(ERPs) in the EEG is presented as N1, P2, N2 and P3 components in a
single trial without averaging. In the right part the ongoing signal
is presented. The red line marks the start of the stimulus
(Reiz = stimulus) and the blue =
reaction. The generated ERPs are marked by coloured lines:

magenta = N1,
– in the EEG, the negative signal is above!

brown
= P2,

grey = N2,

black =P3

On the left the sequence of coloured maps is
presented using the powerful 3D-Mapping of the NIMH / Washington.
Normally, each sample generates one map. Because of space
restrictions, only significant maps are shown in Figure 3. The white
crosses are the symbols for the VQs and describe the evolution of the
appearance from the start up to the extreme value of the amplitude.
These ERPs can be seen in the many channels EEG in the right side of
in Fig. 3 by the vertical lines. The N2
has two components: one earlier component in the central region and
one in the frontal region, 16 milliseconds later. These VQs are the
compressed description of the sequence of maps on the left side of
Fig. 3. The description of the named objects in the complex 3D-signal
can be easily recognized and manipulated by data base mechanisms and
statistics.

The real localisation of the EEG-activity in the
brain is not possible by mathematical methods (inverse problem). The
VQ is a simple evaluation in the computer model, using the centre of
gravity of all clustered potentials. The basic hypothesis for this
method is as follows:

The same electrical activity in the
brain translates consistently to

the same electrical activity

detected by the electrodes on top of
the head and therefore creates

the same Virtual Sources.

In this way the computer builds up the ongoing EEG as
a sequence of VQs.

The user can select important VQs with the
mouse and define templates for the recognition of the VQ in the
ongoing complex signal. These predefined templates are chosen from
either the EEG display or the ERP display (in the presented example:
N1,P2,
N2 and
P3). The ongoing signal display can be examined stepwise by locating
a line cursor and continuously clicking the mouse, each VQ of the
current click can be figured and displayed in a list box. The user
can name the VQs, can manipulate the proposed parameters of the
description of the VQs and store this description in the Data
Base System. There is a user-friendly way to train the system to
recognise specialised events in the complex signal.

However, the true value of the system becomes evident
when it is trained to detect and estimate latency, virtual
localisation and amplitude of the complex signal. With the taught
high performance computer system, the patterns can be recognised in
milliseconds. The templates have to be selected to best represent the
pattern which is intended to be recognised.

If a priori defined EEG-activity occurred, described
by a stored sequence of VQs during the EEG reading, the stimulus was
given to the patient. What happens is that the taught high
performance computer system recognises the sequence of VQs and then
starts a predefined action with a defined delay. We name this
feature of the system: "stimulus, triggered by the state of the
system".

2.5
System functionality

The system can be used in three modes:

-
Learning mode:the methods of
recognition and classification works with imprinted methods for
feature extraction and classification. New classes are built by the
algorithms.

-
Teaching mode:the qualified user
evaluates the classes or specialised features in the signal. This
description is understandable for the user and the computer.

-
Evaluation mode:the evaluated classes
or specialised features are the basis for recognition and
classification in autonomous applications.

Applications
are possible for noisy signals as

1.
s = f(t)

2.
s = f(x, y, t)

3.
s = f(x, y, z, t)

The advantage of the computer system is the
evaluation of each pattern of the signal over a long duration without
loss of attention and with a short reaction time (BrainScope <= 50
ms; human being>=200ms).