Easy To Use Patents Search & Patent Lawyer Directory

At Patents you can conduct a Patent Search, File a Patent Application, find a Patent Attorney, or search available technology through our Patent Exchange. Patents are available using simple keyword or date criteria. If you are looking to hire a patent attorney, you've come to the right place. Protect your idea and hire a patent lawyer.

Methods and apparatus for processing video feed metadata for respective
video feeds and controlling an indicator for at least one the video feeds
on a video wall. The video stream indicator can allow a viewer of the
video wall to reduce the likelihood of missing an event of interest in
the video feeds.

1. A method, comprising: receiving video feed metadata for respective
video feeds, the metadata including sensor control parameters; and
processing the metadata and controlling an indicator for at least one the
video feeds on a video wall.

2. The method according to claim 1, wherein the sensor control parameters
include platform position, platform heading, sensor frustum, and zoom
setting.

3. The method according to claim 1, wherein a first one of the video
feeds includes video from a drone.

4. The method according to claim 3, wherein the drone video settings are
controlled by a remote operator.

5. The method according to claim 4, wherein the drone video settings
include zoom setting.

6. The method according to claim 3, wherein a heading of the drone is
controlled by a remote operator.

7. The method according to claim 1, wherein the indicator includes color.

8. The method according to claim 1, wherein the indicator includes
priority information.

9. The method according to claim 8, wherein the priority information is
relative to others of the video feeds.

10. An article, comprising: a non-transitory computer-readable medium
having stored instructions that enable a machine to: receive video feed
metadata for respective video feeds, the metadata including sensor
control parameters; and process the metadata and controlling an indicator
for at least one the video feeds on a video wall.

11. The article according to claim 10, wherein the sensor control
parameters include platform position, platform heading, sensor frustum,
and zoom setting.

12. The article according to claim 10, wherein a first one of the video
feeds includes video from a drone.

13. The article according to claim 12, wherein the drone video settings
are controlled by a remote operator.

14. The article according to claim 13, wherein the drone video settings
include zoom setting.

15. The article according to claim 12, wherein a heading of the drone is
controlled by a remote operator.

16. The article according to claim 10, wherein the indicator includes
color.

17. The article according to claim 10, wherein the indicator includes
priority information.

18. The article according to claim 17, wherein the priority information
is relative to others of the video feeds.

19. A video wall system, comprising: a module configured to receive video
feed metadata for respective video feeds, the metadata including sensor
control parameters; and a signal processor configured to process the
metadata and control an indicator for at least one the video feeds on a
video wall.

Description

BACKGROUND

[0001] As is known in the art, security, public safety, military
personnel, and others, may simultaneously view a number of live video
feeds, such as from Unmanned Arial Vehicles (UAVs), drones, fixed
surveillance cameras, and other surveillance systems in operation
centers. Such multi-feed displays are referred to as video walls.
However, when viewing a video wall having a number of video feeds, it can
be challenging to determine which of the video feeds is of interest. For
example, an operator of a UAV may have zoomed in on a target of interest.
However, the viewers of the video for that UAV may not be in contact with
the operator and have no idea that a significant event may be in progress
on one of the feeds on the wall. A viewer of the video wall may miss
events of interest on a video feed.

SUMMARY

[0002] The present invention provides method and apparatus for processing
metadata or sensor data for multiple video feeds to identify platform or
sensor events that are of particular interest. In embodiments, the
metadata is processed to trigger recognition or emphasis of significant
events associated with one or more of the video feeds. For example, the
recognition of a significant event can result in automated highlighting
of a particular feed on the video wall to facilitate viewers of the wall
perceiving the event of interest.

[0003] In embodiments, an operations center includes a video wall having a
number of video feeds, such as unmanned aerial vehicle feeds, manned
surveillance aircraft feeds, aerostat surveillance feeds, fixed (e.g.
pole mounted sensor) feeds, open source cable news broadcasts, and the
like, to provide situational awareness. Personnel in the operations
center may be working on personal computers performing various tasks
while having a field of view that includes the video wall.

[0004] Conventional video walls do not include any indication of priority
for the various video feeds so that personnel in the vicinity, or
attentive to, the video wall, may not notice significant events captured
in one or more of the video feeds. For example, surveillance video
typically includes long monotonous periods of insignificant activity
punctured by brief occurrences of significant events. Viewers may want to
`keep an eye` on the wall but may not be able to completely focus on the
video wall feeds.

[0005] Embodiments of the invention utilize video feed metadata to provide
some level of priority or activity indication to one or more video feeds.
Illustrative metadata includes platform position, altitude, heading, and
velocity together with sensor frustum (field of view of the sensor), etc.
The metadata is streamed together with video frames from the sensors.
Since the platforms and the onboard sensors may be controlled by one or
more humans or automated systems, a significant change in metadata may
correspond to a significant event being captured, or expected to be
captured, in the video feed. For example, the sensor direction may be
changed and/or the sensor is zoomed in or out (denoting a change to its
frustum) or the platform heading or velocity may be changed. The metadata
can be processed to recognize such changes and provide an indicator that
can be perceived by viewers of the video wall.

[0006] For example, zoom in, zoom out, frustum centroid change, platform
heading change, platform velocity change can be recognized A relatively
minor zooming in of a video sensor suggests an operator wants to take a
closer look but there is possibly nothing of interest happening. On the
other hand, if the sensor is zoomed in substantially and the platform
begins to hover over an area of terrain, this suggests a highly
significant event may be occurring. Alternatively, if the sensor is
zoomed out and the platform begins to move after hovering, this suggests
that the significant event has ended.

[0007] A variety of suitable indications can be used to direct user
attention to a particular one of the video feeds. Illustrative
indications include coloring a border around a video feed, increasing the
size of one video feed relative to other video feeds, flashing the video
feed for a few seconds, flashing a text overlay on the video feed, and
the like. Some changes, such as a zoom out event, may be associated with
the end of notification as they may signify a return to normal.

[0008] Embodiments can also include a priority level indicator. For
example, a red indicator may signify highest priority, a yellow indicator
may signify possible event upcoming, and the like.

[0009] In general, video indicators are controlled by processing video
metadata without reference to the video stream images. The video
indicators increase the ability of users of a video wall to more
efficiently obtain information from the video wall and decrease the
number of missed events of interest.

[0010] In embodiments, a video wall displays a number of full motion video
(FMV) feeds from various sources. When an event occurs, it is likely that
the video sensor is refocused and/or panned/tilted for a better view
either manually or by someone in a remote location or automatically by a
smart sensor. Scene analytic `tags` can be generated based on video
sensor metadata available, for example, in Standardization Agreement
(STANAG) digital motion imagery standards (e.g. 4609 FMV feed).

[0011] In one aspect of the invention, a method comprises: receiving video
feed metadata for respective video feeds, the metadata including sensor
control parameters; and processing the metadata and controlling an
indicator for at least one the video feeds on a video wall.

[0012] The method can further include one or more of the following
features: the sensor control parameters include platform position,
platform heading, sensor frustum, and zoom setting, a first one of the
video feeds includes video from a drone, the drone video settings are
controlled by a remote operator, the drone video settings include zoom
setting, a heading of the drone is controlled by a remote operator, the
indicator includes color, the indicator includes priority information,
and/or the priority information is relative to others of the video feeds.

[0013] In a further aspect of the invention, an article comprises: a
non-transitory computer-readable medium having stored instructions that
enable a machine to: receive video feed metadata for respective video
feeds, the metadata including sensor control parameters; and process the
metadata and controlling an indicator for at least one the video feeds on
a video wall.

[0014] The article can further include instructions for one or more of the
following features: the sensor control parameters include platform
position, platform heading, sensor frustum, and zoom setting, a first one
of the video feeds includes video from a drone, the drone video settings
are controlled by a remote operator, the drone video settings include
zoom setting, a heading of the drone is controlled by a remote operator,
the indicator includes color, the indicator includes priority
information, and/or the priority information is relative to others of the
video feeds.

[0015] In a further aspect of the invention, a video wall system,
comprises: a module configured to receive video feed metadata for
respective video feeds, the metadata including sensor control parameters;
and a signal processor configured to process the metadata and control an
indicator for at least one the video feeds on a video wall.

[0016] The system can further include one or more of the following
features: the sensor control parameters include platform position,
platform heading, sensor frustum, and zoom setting, a first one of the
video feeds includes video from a drone, the drone video settings are
controlled by a remote operator, the drone video settings include zoom
setting, a heading of the drone is controlled by a remote operator, the
indicator includes color, the indicator includes priority information,
and/or the priority information is relative to others of the video feeds.

BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The foregoing features of this invention, as well as the invention
itself, may be more fully understood from the following description of
the drawings in which:

[0018] FIG. 1 is a schematic representation of a system for controlling a
video wall and providing feed indicators;

[0019] FIG. 2 is a schematic representation showing further details of the
system of FIG. 1;

[0020] FIG. 3 is a schematic representation showing further details of a
video wall and feed indicators for the system of FIG. 1;

[0022] FIG. 5 is a schematic representation of an illustrative computer
that can perform at least a portion of the processing described herein.

DETAILED DESCRIPTION

[0023] FIG. 1 shows an illustrative system 100 including a video wall 102
having a number of video screens 104 having respective indicators 106. A
number of video feeds 108a-N, which can comprise remote video sensors,
are provided to a video feed processor 110. The video feed processor 110
processes the video feed and associated metadata to control the
indicators 106 on the video wall 102.

[0024] The metadata can comprise various parameters, such as platform
position and zoom length, that can be analyzed to selectively activate an
indicator 104 for a screen on which a video feed is being shown. The
indicator 104 can be activated to call attention to a particular feed
that may be of interest to viewers of the wall 103 based on the metadata.
For example, a drone being controlled by a remote operator may have a
heading change and an aggressive zoom that may be indicative of some
upcoming event of interest. By processing the metadata, the indicator 106
for that feed 104 can be activated to reduce the likelihood that viewers
of the wall 102 will miss an event of interest.

[0025] It is understood that each feed 104 may have a respective indicator
106. In other embodiments, an indicator system provides an indicator
directed to and activated for each of the feeds. For example, the
indicator system may emit a low power laser light to corner of a given
feed 104 to indicate a particular priority of that feed. Red may indicate
highest priority and green may indicate lowest priority or no expected
significant events upcoming.

[0026] In another embodiment, the system 100 prioritizes the video feeds
104 by processing the metadata and showing the video feeds such that the
highest priority feed is in a first priority position, such as top left,
a second highest priority feed is in a second priority position, such as
top, to right of the highest priority feed, and the like.

[0027] FIG. 2 shows a system 200 receiving a number of illustrative real
time surveillance video feeds 202a-e, including a first drone 202a, a
second drone 202b, a first aerostat 202c, a first aircraft 202d, and
second aircraft 202e. It is understood that a drone refers to a type of
UAV, such as a remotely controlled surveillance aircraft. Each of the
video feeds 204a-e includes sensor metadata, such as platform position,
heading, velocity, and sensor frustum. It is understood that sensor
frustum refers to a field of view that appears on a screen, which can be
defined by taking a frustum (truncation with parallel planes) for a
camera with rectangular viewports, e.g., a frustum of a rectangular
pyramid. The planes that cut the frustum perpendicular to the viewing
direction are called the near plane and the far plane.

[0028] A feed processing module 206 receives the video feeds and metadata
204. A video feed module 208 receives the video stream, a metadata
processing module 210 receives the metadata, and a feed indicator control
module 212, which is coupled to the metadata processing module 210,
controls the indicators 214a-f on a video wall 216 for each of the
screens 216a-f.

[0029] While illustrative metadata having certain parameters is shown, it
is understood that a wide range of parameters can be provided. In
addition, while embodiments are shown in conjunction with video feeds, it
is understood that any type of data stream can be processed, such as
audio only data, which can be provided to a video wall screen with
optional text-to-speech (TTS), for example.

[0030] In the illustrated embodiment, the video feed in the bottom middle
position 216e on the video wall 216 has an active indicator 214e to
enhance the ability of viewers of the video wall to focus on the tagged
frames of that video stream.

[0031] FIG. 3 shows an illustrative video wall 300 having screens 302 for
showing respective video feeds with indicators 304 for each screen. It is
understood that the indicators 304 can be provided in a variety of
formats, configurations, and technologies that are suitable to bring
viewer attention to a given screen. In embodiments, the indicators 304
illuminate in a particular color. The colors can indicate a discrete
attention level by increasing wavelength, e.g., (violet, blue, green,
yellow, orange, red). As is known in the art, blue (470 nm) has a shorter
wavelength than red (700 nm), for example. In other embodiments, each
indicator 304a can comprises a number of bars so that the number of bars
illuminated corresponds to an attention level. In further embodiments,
the bars 304b are located around each edge of the screen. A lower
priority can be one bar under the screen illuminated, and a higher
priority is a bar along each screen edge being illuminated. In
embodiments, a sound, such as a chirp, can be generated for events above
a given priority level. This can enable personnel not presently focused
on the video wall to bring their attention on the video wall including a
particular feed in response to the chirp.

[0032] In embodiments, video feed metadata is processed independently such
that an indicator 304 for a given feed is controlled using the metadata
for that field. Based on the metadata, the feed indicator is controlled
to be off, green, red, flashing, etc.

[0033] In other embodiments, video feed metadata is processed for the
video feeds such that the indicator for one feed is controlled using
metadata from at least one other feed. In embodiments, the
independent/combined metadata processing is selected manually or on some
criteria. For example, during times of intense activity for multiple
feeds, the metadata may be processed as a whole to control the various
feed indicators in such a way to provide some feed priority for viewers
of the video wall. In one embodiment, a video feed from a drone
maneuvering in a manner consistent with known attack protocols can result
in the indicator for that feed being controlled to indicate highest
priority for all the feeds active on the video wall. In another
embodiment, a video feed from a drone within a given range of a
particular geographic location can result in the indicator for that feed
being controlled to indicate highest priority for all the feeds active on
the video wall. A variety of high priority items, such as events,
actions, locations, etc., can be ranked in priority. If multiple ones of
the high priority items occur simultaneously, the ranking can be
translated to the indicators for the respective video feed.

[0043] It is understood that in surveillance missions, the term `stare` is
typically used to refer to controlling a sensor platform and/or sensor in
such a way as to maintain an approximately constant field of view so as
to capture the activity in the field of view over the period of time in
which the stare is being executed.

[0044] FIG. 4 shows an illustrative sequence of steps for video feed
indicator control. In step 400, sensor feed and metadata is received. In
step 402, the video feed metadata is processed to analyze operator
control actions for the sensor, for example. Based on the metadata, in
step 404, indicators for the various feeds are controlled to provide some
level of priority to the video feeds on the video wall. In step 406, the
video feeds and screen indicator information is transmitted to the video
wall. With this arrangement, a viewer of the video wall may be provided
an indication that an event of interest is occurring, or may be
occurring.

[0045] In embodiments, metadata processing and control of indicators is
performed in a `pipeline` in order to minimize latency between metadata
artifact and indicator reaction. In one particular embodiment, a first
step in the pipeline strips metadata from the video. A single metadata
record is typically inserted into line 21, for example, of 3-4
consecutive video frames. A next step normalizes the metadata and
populates an internal data structure by performing appropriate
processing, e.g., determining the sensor frustum. A next step compares
the newly created metadata data structure to the sequence of preceding
metadata structures to compute factors, such as delta zoom, length of
stare, percent of trajectory change, etc. A further step determines
which, if any, indicator(s) should be generated. In general, this
processing is performed on a sensor by sensor basis. Further processing
can be performed to determine any relationships between sensors.

[0046] FIG. 5 shows an exemplary computer 500 that can perform at least
part of the processing described herein. The computer 500 includes a
processor 502, a volatile memory 504, a non-volatile memory 506 (e.g.,
hard disk), an output device 507 and a graphical user interface (GUI) 508
(e.g., a mouse, a keyboard, a display, for example). The non-volatile
memory 506 stores computer instructions 512, an operating system 516 and
data 518. In one example, the computer instructions 512 are executed by
the processor 502 out of volatile memory 504. In one embodiment, an
article 520 comprises non-transitory computer-readable instructions.

[0047] Processing may be implemented in hardware, software, or a
combination of the two. Processing may be implemented in computer
programs executed on programmable computers/machines that each includes a
processor, a storage medium or other article of manufacture that is
readable by the processor (including volatile and non-volatile memory
and/or storage elements), at least one input device, and one or more
output devices. Program code may be applied to data entered using an
input device to perform processing and to generate output information.

[0048] The system can perform processing, at least in part, via a computer
program product, (e.g., in a machine-readable storage device), for
execution by, or to control the operation of, data processing apparatus
(e.g., a programmable processor, a computer, or multiple computers). Each
such program may be implemented in a high level procedural or
object-oriented programming language to communicate with a computer
system. However, the programs may be implemented in assembly or machine
language. The language may be a compiled or an interpreted language and
it may be deployed in any form, including as a stand-alone program or as
a module, component, subroutine, or other unit suitable for use in a
computing environment. A computer program may be deployed to be executed
on one computer or on multiple computers at one site or distributed
across multiple sites and interconnected by a communication network. A
computer program may be stored on a storage medium or device (e.g.,
CD-ROM, hard disk, or magnetic diskette) that is readable by a general or
special purpose programmable computer for configuring and operating the
computer when the storage medium or device is read by the computer.
Processing may also be implemented as a machine-readable storage medium,
configured with a computer program, where upon execution, instructions in
the computer program cause the computer to operate.

[0049] Processing may be performed by one or more programmable processors
executing one or more computer programs to perform the functions of the
system. All or part of the system may be implemented as, special purpose
logic circuitry (e.g., an FPGA (field programmable gate array) and/or an
ASIC (application-specific integrated circuit)).

[0050] Having described exemplary embodiments of the invention, it will
now become apparent to one of ordinary skill in the art that other
embodiments incorporating their concepts may also be used. The
embodiments contained herein should not be limited to disclosed
embodiments but rather should be limited only by the spirit and scope of
the appended claims. All publications and references cited herein are
expressly incorporated herein by reference in their entirety.

[0051] Elements of different embodiments described herein may be combined
to form other embodiments not specifically set forth above. Various
elements, which are described in the context of a single embodiment, may
also be provided separately or in any suitable subcombination. Other
embodiments not specifically described herein are also within the scope
of the following claims.