Patent application title: Method of Obtaining a Saliency Map From a Plurality Of Saliency Maps Created From Visual Quantities

Abstract:

The invention relates to a method of obtaining a saliency map from a
plurality of saliency maps created from different visual quantities,
comprising - a step for normalizing said saliency maps based on the
theoretical maximum of each visual quantity, -an intra-map competition
step selecting the main saliency areas in each saliency map, -an
inter-map competition step based on the sum of the intra-map competitions
with an inter-map redundancy term that is a function of the product of
the intra-map competitions and of the probability of a site appearing on
said saliency maps.

Claims:

1. Method of obtaining a saliency map from a plurality of saliency maps
created from different visual quantities, wherein it comprisesa step for
normalizing said saliency maps based on the theoretical maximum of each
visual quantity,an intra-map competition step selecting the main saliency
areas in each saliency map,an inter-map competition step based on the sum
of the intra-map competitions with an inter-map redundancy term that is a
function of the product of the intra-map competitions and of the
probability of a site appearing on said saliency maps.

2. Method according to claim 1, wherein, in the normalization step,the
sites of said saliency maps are normalized in relation to the theoretical
maximum of the visual quantity value,said maps are quantized linearly
using a predetermined number of levels.

3. Method according to claim 1, wherein, in the intra-map competition
step,a preliminary list is defined, containing the sites said to be
salient including sites for which the visual quantity value is high,the
preliminary list of the sites said to be salient is scanned in descending
order of said values, and the other sites said to be salient present in a
circular area centred around the site said to be salient and of a
predetermined radius are inhibited,a list of the salient sites is
established, including the non-inhibited sites for which the ratio with
the value of the next higher value site is greater than a predetermined
threshold.

4. Method according to claim 1, wherein the visual quantities are relative
to the chromatic and achromatic components.

5. Method according to claim 4, wherein the saliency maps relative to the
chromatic components are merged and then the resultant saliency map is
merged with the saliency map relative to the achromatic component.

Description:

[0001]The invention relates to a method of obtaining a saliency map from a
plurality of saliency maps created from different visual quantities.

[0002]Human beings have a selective visual attention, meaning that our
visual system responds primarily to a certain number of signals
originating from the objects and events of our environment.

[0003]The signal that most obviously and most intuitively attracts our
attention is undoubtedly the sudden appearance of an object in a scene.

[0004]Finally, various studies seek to estimate, from real fixed points,
the similarities of the visual characteristics attracting our gaze. As a
general rule, these studies relate to the measurement of various
quantities such as the variance normalized by the average brightness of
the image, the entropy and the correlation between the measured fixed
point and its vicinity. The main conclusions are as follows: [0005]the
contrast measurements of the fixed regions are higher than those of
regions taken at random. In other words, the contrast of an area,
regardless of its nature (luminance, colour, movement, texture, etc.),
attracts our attention even when this area has nothing to do with the
task to be carried out by the observer. [0006]based on the correlation
measurements, these studies also show that the fixed regions differ from
their vicinity.

[0007]The detection of saliency points in an image makes it possible
subsequently to improve encoding and indexing methods. Obtaining saliency
maps as a way of obtaining a list of the salient points of an image is
described in the European patent application published under the number
EP1544792, filed under the name of Thomson Licensing SA on Dec. 18, 2003.

[0008]The creation of saliency maps is relative to different visual
quantities: one saliency map possibly being relative to the chromatic
components, one map for each chromatic component, or even relative to the
achromatic components. However, once the different saliency maps have
been created, merging them can generate undesirable results.

[0009]A conventional merging method consists in normalizing the different
saliency maps so as to obtain the same dynamic range. The normalization
of a map C, denoted N(C), uses the overall maximum determined on the map
C. The final saliency map S is then simply obtained by the following
relation:

CS(s)=N(N(CSA(s))+N(CS.sub.Cr1(s))+N(CS.sub.Cr2(s)))

[0010]with CSA(s) representing the saliency map of the achromatic
component, CS.sub.Cr1(s) representing the saliency map of the first
chromatic component and CS.sub.Cr2(s) representing the saliency map of
the second chromatic component.

[0011]One advantage of this method is its simplicity. However, it does
present various drawbacks: [0012]this method does not distinguish
between a saliency map having a quasi-uniform distribution and a saliency
map having one or more saliency peaks; [0013]when a number of saliency
peaks are present in a saliency map, this type of merging clearly favours
the highest saliency peak; [0014]this method is very sensitive to impulse
noise, [0015]there is no interaction between maps.

[0016]The invention therefore proposes to remedy at least one of the
abovementioned drawbacks. To this end, the invention proposes a method of
obtaining a saliency map from a plurality of saliency maps created from
different visual quantities. According to the invention, the method
comprises [0017]a step for normalizing said saliency maps based on the
theoretical maximum of each visual quantity, [0018]an intra-map
competition step selecting the main saliency areas in each saliency map,
[0019]an inter-map competition step based on the sum of the intra-map
competitions with an inter-map redundancy term that is a function of the
product of the intra-map competitions and of the probability of a site
appearing on said saliency maps.

[0020]Such a method of merging saliency maps involves two competition
methods: [0021]an intra-map competition for identifying the most
relevant areas of the map; [0022]an inter-map competition exploiting the
redundancy and complementarity of the different maps. The use of
inter-map redundancy is a way of reinforcing the saliency of certain
areas when the latter generate saliency in a number of dimensions.
Conversely, when an area generates saliency only in one visual dimension,
it is necessary to use the inter-map complementarity.

[0023]According to a preferred embodiment, in the normalization step,
[0024]the sites of said saliency maps are normalized in relation to the
theoretical maximum of the visual quantity value, [0025]said maps are
quantized linearly using a predetermined number of levels.

[0026]According to a preferred embodiment, in the intra-map competition
step, [0027]a preliminary list is defined, containing the sites said to
be salient including sites for which the visual quantity value is high,
[0028]the preliminary list of the sites said to be salient is scanned in
descending order of said values, and the other sites said to be salient
present in a circular area centred around the site said to be salient and
of a predetermined radius are inhibited, [0029]a list of the salient
sites is established, including the non-inhibited sites for which the
ratio with the value of the next higher value site is greater than a
predetermined threshold.

[0030]According to a preferred embodiment, the visual quantities are
relative to the chromatic and achromatic components.

[0031]According to a preferred embodiment, the saliency maps relative to
the chromatic components are merged and then the resultant saliency map
is merged with the saliency map relative to the achromatic component.

[0032]The invention will be better understood and illustrated by means of
exemplary embodiments and advantageous implementations, by no means
limiting, with reference to the single appended figure representing an
exemplary search for the local maximums on the unmodified component A.

[0033]The embodiment described below proposes a coherent merging for two
maps, denoted CS.sup.Cr1 and CS.sup.Cr2 derived from a component Cr1 and
Cr2. Generalization to n saliency maps is easy to envisage.

[0034]The merging method comprises a preliminary dynamic normalization
step. Unlike the known normalizations that use a normalization based on
the overall maximum of each map, the normalization used in the merging
method is based on the theoretical maximum of each visual dimension.
These maximums are determined experimentally using particular tests. For
example, for the component Cr1, an image with uniform luminance but
having a saturated red pattern generates a dynamic close to the maximum
dynamic of the visual axis Cr1. Repeating this type of experimentation is
a way of defining the theoretical maximums of the components A, Cr1, Cr2.

[0035]The two maps CS.sup.Cr1 and CS.sup.Cr2 are then normalized and
quantized linearly on L levels. After normalization and quantization,
they are respectively denoted CSNQ.sup.Cr1 and CSNQ.sup.Cr2.

[0036]Following the normalization step, the method includes an intra-map
competition step. This intra-map competition modifies the value of each
site s of the maps CSNQ.sup.Cr1 and CSNQ.sup.Cr2 according to
the nearest local maximum. This type of competition is given by the
following relation:

[0037]The function NearestMaxcr1 (respectively NearestMaxcr2)
returns the value of the local maximum of the component Cr1 (respectively
Cr2) nearest to the site s. This value is taken from the list L1
(respectively L2) of size K1 (respectively K2) values. The size of the
lists is determined in such a way as to obtain a ratio between the local
maximum n and the local maximum n+1 greater than a threshold, set
arbitrarily at 1.3. This makes it possible to take into account only the
main saliency areas.

[0038]The local maximum n+1 is determined by inhibiting a circular area
centred around the local maximum n and with a radius of two visual
degrees as represented in FIG. 1. The size of the circle is proportional
to the viewing distance.

[0039]Following the intra-map competition step, an inter-map competition
is applied. This inter-map competition exploits the redundancy and the
complementarity of the different maps. The term Intermap is given by the
following relation:

Intermap(s)=complementarity(s)+redundancy(s)

[0040]The term "complementarity(s)" is obtained by adding together the
results of the intra-map competition:

complementarity(s)=int raMap.sup.Cr1(s)+int raMap.sup.Cr2(s)

[0041]The inter-map redundancy is processed on the basis of a joint
analysis of the distributions of the maps to be merged.

deduced from the combined histogram of the maps CSNQC1 and
CSNQC2 modifies the value of the site s concerned according to
its probability of appearing. The quantity of information conveyed by a
site s is inversely proportional to its probability of appearing.
Consequently, the above factor increases the value of a site s when its
probability of appearing is low. Conversely, the value of the site s is
reduced when its probability of appearing is high.

[0044]The merging of the maps CS.sup.Cr1 and CS.sup.Cr2 is given by the
term intermap(s).

[0045]When the visual quantities Cr1 and Cr2 represent the chromatic
components, a third saliency map relative to an achromatic component can
also be introduced. A hierarchical approach is then introduced for
carrying out the merging of the three saliency maps.

[0046]The saliency map is thus obtained by firstly merging the two
saliency maps relative to the achromatic components and then performing a
merging between this resultant chromatic saliency map and the achromatic
saliency map.

[0047]Such a hierarchical approach can also be applied by merging a
temporal saliency map with the spatial saliency maps. The chromatic and
achromatic saliency maps are then merged according to the abovementioned
hierarchical approach. A hierarchical merging of this spatial saliency
map is then performed with the temporal saliency map.