Sign up to receive free email alerts when patent applications with chosen keywords are publishedSIGN UP

Abstract:

A method includes reading a composite video descriptor data structure and
a plurality of window descriptor data structures. The composite video
descriptor data structure defines a width and height of a composite video
frame and each window descriptor data structure defines the starting X
and Y coordinate, width and height of each constituent video window to be
rendered in the composite video frame. The method further includes
determining top and bottom Y coordinates for each constituent video
window, as well as determining left and right X coordinates for each
constituent video window. The method also includes dividing each
constituent video window using the top and bottom Y coordinates to obtain
Y-divided sub-windows, dividing each Y-divided sub-window using left and
right X coordinates to obtain X and Y divided sub-windows, and storing X,
Y coordinates of opposing corners of each X and Y divided sub-window in
the storage device.

Claims:

1-24. (canceled)

25. A method comprising the steps of: reading, by a processor from a
storage device, a composite video descriptor data structure and a
plurality of window descriptor data structures, the composite video
descriptor data structure defining a width and height of a composite
video frame and each window descriptor data structure defining starting X
and Y coordinates, width and height of each constituent video window to
be rendered in the composite video frame; determining, by the processor,
top and bottom Y coordinates, and left and right X coordinates for each
constituent video window; dividing, by the processor, each constituent
video window using the top and bottom Y coordinates, and left and right X
coordinates of all windows to obtain sub-windows including overlapping
sub-windows from distinct constituent video windows; computing, by the
processor, an ordering key for each sub-window by multiplying a Y
coordinate for sub-window by a constant to produce a product, and adding
an X coordinate for window to the product; and storing, by the processor,
X, Y coordinates of opposing corners of each X and Y divided sub-window
in the storage device.

26. The method of claim 25 wherein: the constant is greater than or equal
to a largest X coordinate of any sub-window.

27. The method of claim 25 further comprising the step of: determining,
by the processor, an order for the sub-windows based on the ordering key
computed for each sub-window.

28. The method of claim 27 wherein: said step of reading further includes
reading a priority value for each constituent video window; and said
method further comprises the steps of: programming, by the processor, a
direct memory access (DMA) engine to write video data from non-contiguous
memory to an encoder as composite video based upon said determined order,
including upon detection of two or more sub-windows having the same
ordering key, programming, by the processor, the direct memory access
(DMA) engine to write video data from non-contiguous memory to the
encoder as composite video for a sub-windows having a highest priority of
the corresponding constituent video window and not writing video data
from non-contiguous memory to the encoder for any sub-window having a
lower priority of the corresponding constituent video window.

29. An apparatus, comprising: a processor; storage coupled to the
processor, said storage containing a composite video descriptor data
structure and a plurality of window descriptor data structures, wherein
the composite video descriptor data structure defines a width and height
of a composite video frame, and each window descriptor data structure
defines starting X and Y coordinates, width and height of each
constituent video window to be rendered in the composite video frame;
wherein the processor uses the composite video descriptor and window
descriptor data structures to: determine top and bottom Y coordinates,
and left and right X coordinates for each constituent video window;
divide each constituent video window using the top and bottom Y
coordinates, and the left and right X coordinates to obtain sub-windows;
compute an ordering key for each sub-window by multiplying a Y coordinate
for the X and Y divided sub-window by a constant to produce a product and
adding an X coordinate for the sub-window to the product; and store X, Y
coordinates of opposing corners of each X and Y divided sub-window in the
storage.

30. The apparatus of claim 29 wherein: the constant is greater than or
equal to largest X coordinate of and sub-window.

31. The apparatus of claim 30 wherein: the processor determines an order
for the sub-windows based on the ordering key computed for each
sub-window.

32. The apparatus of claim 32 further comprising: the storage further
stores a priority for each window descriptor data structure; a direct
memory access (DMA) engine; and wherein the processor programs the direct
memory access (DMA) engine to write video data from non-contiguous memory
of said storage to an encoder as composite video based upon said
determined order, including upon detection of two or more sub-windows
having the same ordering key, said direct memory access (DMA) engine to
write video data from non-contiguous memory to the encoder as composite
video for a sub-windows having a highest priority of the corresponding
constituent video window and not writing video data from non-contiguous
memory to the encoder for any sub-window having a lower priority of the
corresponding constituent video window.

Description:

CROSS-REFERENCE TO RELATED APPLICATION

[0001] None.

BACKGROUND

[0002] It is often desirable to display two or more separate video sources
on one display simultaneously. Picture-In-Picture (PIP) is one such
format in which two video sources are displayed, one appearing to be on
top of the other. Display devices typically function in a raster mode in
which the pixels of the display device are rendered in sequential order
starting with the top left pixel, continuing to the right along the first
line, then dropping down to the left side of the second line, continuing
along the second line, and so on to the bottom line of pixels to complete
one full video frame. The rasterization process repeats typically at a
rate of 30 or 60 frames per second.

[0003] Rendering multiple video sources is inherently incompatible with
rasterization process of a display device, as the display device is
incapable of rendering just a single video source on a portion of the
display. As explained above, the display rasterizes across all pixels of
the display 30 times per second, and not just a subset of the pixels.

SUMMARY

[0004] Construction of composite video is described herein that does not
require construction of the composite video in memory. Instead, the
composite video is constructed on-the-fly as it is written to the video
encoder.

[0005] For example, a method includes reading a composite video descriptor
data structure and a plurality of window descriptor data structures. The
composite video descriptor data structure defines width and height of a
composite video frame and each window descriptor data structure defines
the starting X and Y coordinate, width and height of each constituent
video window to be rendered in the composite video frame. The method
further includes determining top and bottom Y coordinates for each
constituent video window, as well as determining left and right X
coordinates for each constituent video window. The method also includes
dividing each constituent video window using the top and bottom Y
coordinates to obtain Y-divided sub-windows, dividing each Y-divided
sub-window using left and right X coordinates to obtain X and Y divided
sub-windows, and storing X, Y coordinates of opposing corners of each X
and Y divided sub-window in the storage device.

[0006] Another method includes computing an ordering key for each
sub-window to be assembled into a composite video frame. Further, based
on the ordering keys, the method includes determining an order for the
sub-windows to be assembled into the composite video frame, and writing,
by a direct memory access (DMA) engine, video data for each sub-window
from non-contiguous memory to an encoder as composite video based on the
determined order.

[0007] In yet another embodiment, an apparatus includes a processor and
storage coupled to the processor. The storage contains a composite video
descriptor data structure and a plurality of window descriptor data
structures. The composite video descriptor data structure defines a width
and height of a composite video frame, and each window descriptor data
structure defines starting X and Y coordinates, width and height of each
constituent video window to be rendered in the composite video frame. The
processor uses the composite video descriptor and window descriptor data
structures to determine top and bottom Y coordinates for each constituent
video window, determine left and right X coordinates for each constituent
video window, divide each constituent video window using the top and
bottom Y coordinates to obtain Y-divided sub-windows, divide each
Y-divided sub-window using left and right X coordinates to obtain X and Y
divided sub-windows, and store X, Y coordinates of opposing corners of
each X and Y divided sub-window in the storage.

[0008] Another apparatus includes a processor and storage coupled to the
processor and containing information on each of a plurality of
sub-windows to be included in a composite video frame. The processor
computes an ordering key for each sub-window to be included into the
composite video frame. Based on the ordering keys, the processor
determines an order for the sub-windows to be included into the composite
video frame. The apparatus also includes a direct memory access (DMA)
engine coupled to the processor and storage. The DMA engine writes video
data for each sub-window from non-contiguous storage to an encoder as
composite video based on the order determined by the processor.

BRIEF DESCRIPTION OF THE DRAWINGS

[0009] For a detailed description of exemplary embodiments of the
invention, reference will now be made to the accompanying drawings in
which:

[0010] FIG. 1 shows a system for constructing composite video in
accordance with various embodiments of the invention;

[0011] FIG. 2 shows an example of a composite video constructed using the
system of claim 1;

[0012] FIG. 3 shows a method performed by the system of claim 1 for
constructing composite video;

[0013] FIGS. 4-8 show examples of various steps in the process for how the
composite video of FIG. 2 is constructed;

[0014] FIG. 9 shows a method for determining the order for how the various
sub-windows in the composite video are to be written to a video encoder;
and

[0015] FIG. 10 provides a further example usable to illustrate the method
of FIG. 9.

NOTATION AND NOMENCLATURE

[0016] Certain terms are used throughout the following description and
claims to refer to particular system components. As one skilled in the
art will appreciate, companies may refer to a component by different
names. This document does not intend to distinguish between components
that differ in name but not function. In the following discussion and in
the claims, the terms "including" and "comprising" are used in an
open-ended fashion, and thus should be interpreted to mean "including,
but not limited to . . . ." Also, the term "couple" or "couples" is
intended to mean either an indirect or direct electrical connection.
Thus, if a first device couples to a second device, that connection may
be through a direct electrical connection, or through an indirect
electrical connection via other devices and connections.

DETAILED DESCRIPTION

[0017] The following discussion is directed to various embodiments of the
invention. Although one or more of these embodiments may be preferred,
the embodiments disclosed should not be interpreted, or otherwise used,
as limiting the scope of the disclosure, including the claims. In
addition, one skilled in the art will understand that the following
description has broad application, and the discussion of any embodiment
is meant only to be exemplary of that embodiment, and not intended to
intimate that the scope of the disclosure, including the claims, is
limited to that embodiment.

[0018] Some video systems capable of simultaneously displaying multiple
video sources first assemble the composite video frame from the various
video sources in memory and then perform a memory write to a video
encoder to cause the composite video to be rendered on display device.
This so-called "in-memory construction of composite video frames"
unfortunately requires a high degree of processor support and greater
memory bandwidth.

[0019] The embodiments disclosed herein are directed to constructing a
composite video frame "on the fly" meaning the step of assembling the
composite frame in memory before transferring the frame to the encoder is
avoided. Instead, the composite frame is constructed and transferred to
the encoder at the same time.

[0020] FIG. 1 illustrates a system 100 in accordance with various
embodiments. The system 100 includes a processor 20 coupled to storage
104 and a direct memory access (DMA) engine 106. The storage 104 may
comprise volatile storage (e.g., random access memory), non-volatile
storage (e.g., hard disk drive, sold-state storage, read-only memory),
etc.), or combinations of volatile and non-volatile storage. The DMA
engine 106 can be programmed by the processor 102 to write contents of
storage 104 to an encoder 108. The encoder 108 converts input video data
to a format compatible with a display device 112 and provides the encoded
video data to a physical interface (PHY) 110 which provides the
appropriate electrical signaling to the display device 112.

[0021] Video data from one or more video sources 120 can be stored in
storage 104. A video source may be a digital camera, another storage
device on which a video signal has already been stored, etc. Decoder 114
decodes the video signal from the video sources and stores the decoded
video data in storage 104. Reference numerals 122 and 124 illustrate
decoded video data from the video storage devices 120.

[0022] FIG. 2 shows an example of a composite video frame 200 to be
constructed in accordance with the embodiments described herein. The
composite video frame 200 comprises two video sources 202 and 204
simultaneously rendered on a display device. Video source 204 appears to
be rendered "on top of" video source 202 in a Picture-In-Picture (PIP)
effect. Each video source 202, 204 is referred to as a "video window."
The following description describes how the composite video frame 200 is
constructed. The described process constructs a composite video frame
preferably 30 times per second.

[0023] For each composite video to be constructed, storage 104 also stores
a composite video descriptor (CVD) data structure 126 and one or more
window descriptor (WD) data structures 128. The CVD data structure 126
preferably defines the height and width of the composite video window,
the number of constituent video windows that are to be rendered as part
of the composite video, and pointers to WD data structures 128 defining
the various constituent video windows. Each WD data structure 128 defines
the height and width of the video window of the corresponding video
window to be included as part of the composite video, the coordinate (X
and Y) of the starting pixel of the video window, a priority value, and a
pointer to the video data that represents the video window. The starting
pixel preferably is the upper left most pixel in the video window, but
can be other than the upper left most pixel in other embodiments. For a
composite video containing two video windows, the composite video is
represented by one CVD data structure 126 to define the size and shape of
the composite video window as well as two WD windows to define the size,
shape and video contents of each constituent video window.

[0024] The data structures 126, 128 may be created by the processor 102 in
response, for example, to input by a user via an input device (e.g.,
keyboard, mouse, remote control, etc.) as to how the composite video is
to be rendered on display device 112.

[0025] FIG. 3 illustrates a method 130 preferably performed by the
processor 102. The various actions depicted in FIG. 3 can be performed in
the order shown or in a different order, and two or more actions can be
performed in parallel rather than sequentially. As the method 130 of FIG.
3 is explained, reference will be made to an example depicted in FIGS.
4-8.

[0026] At 132, the processor 102 reads the CVD and corresponding WD data
structures 126, 128 from storage 104. At 134, the processor determines
the top and bottom Y coordinates for each constituent video window
defined by the data structures. FIGS. 4 and 5 show an example of this
process. In FIG. 4, the composite video frame defined by the data
structures is placed in the fourth quadrant of an X-Y Cartesian
coordinate system. The Y pixel coordinates are assumed to be positive in
this example. As shown, the starting pixel for the composite video frame
has a coordinate of (0, 0). In this example, the composite video frame
has a width of 60 pixels (60 pixels along the X axis) and has a height of
60 pixels (60 pixels along Y axis). Based on the starting pixel
coordinate, width and height of the composite video frame, the lower
right most pixel has a coordinate of (60, 60).

[0027] Reference numeral 202 identifies a larger video window in which one
video source is rendered, and reference numeral 204 identifies a smaller
video window in which another video is rendered. Smaller video window 204
is itself defined by a starting pixel having a coordinate of (20, 20) and
has a width of 20 pixels and a height of 20 pixels as well. Based on the
starting pixel coordinate, width and height of the video window 204, the
lower right most pixel of window 204 has a coordinate of (40, 40).

[0028] FIG. 5 illustrates the top and bottom Y coordinates for each
constituent video window in the example of FIG. 4. Top and bottom Y
coordinates for video window 202 is 0 and 60, respectively. Top and
bottom Y coordinates for smaller video window 204 is 20 and 40,
respectively.

[0029] Referring again to FIG. 3, at 136 method 130 comprises determining
the left and right X coordinates for each constituent video window. This
determination is made based on the starting X coordinate and width of
each video window as provided in WD data structure 128. The left and
right X coordinate for larger video window 202 is 0 and 60, respectively,
while the left and right X coordinate for smaller video window 204 is 20
and 40, respectively.

[0030] Once the top and bottom Y coordinate and left and right X
coordinate of each constituent video window is determined, then at 138
the method comprises dividing each constituent video window using the top
and bottom Y coordinates determined at 134 to obtain "Y-divided
sub-windows." A Y-divided sub-window is a portion (a sub-window) of a
video window that has been divided based on a Y coordinate. Each window
is divided only for Y values that are greater than the window's starting
Y coordinate and less than the sum of the starting Y coordinate and the
window's height.

[0031] This process is exemplified in FIG. 7. For the larger video window
202, the starting Y coordinate is 0 and the sum of the starting
coordinate (0) and the window's height (60) is 60. Thus, window 202 is
divided for based on all top and bottom Y coordinates determined at 134
that are greater than 0 and less than 60. As shown in FIG. 5, the top and
bottom Y coordinates determined at 134 are 0, 20, 40, and 60. Of those
four coordinates, only Y coordinates 20 and 40 are greater than 0 and
less than 60, and thus coordinates 20 and 40 are used to divide the
larger video window 202. As such, video window 202 is divided into three
Y-divided sub-windows 202a, 202b, and 202c as shown. The space between
the three sub-windows is not actually present as the three sub-windows
abut one another--the space is merely present in FIG. 7 for ease of
understanding the process. The upper sub-window 202a has a starting
coordinate of (0, 0) and, due to the division on Y coordinate 20, a
height of 20 meaning its lower right pixel has a coordinate of (60, 19).
The middle sub-window 202b has a starting coordinate of (0, 20) and, due
to the further division of video window 202 on Y coordinate 40,
sub-window 202b has a height of 20 resulting in its lower right pixel
having a coordinate of (60, 40). Finally, the lower sub-window 202c has a
starting coordinate of (0, 40) and a lower right pixel coordinate of (60,
60).

[0032] The starting Y coordinate of video window 204 is (20, 20) and,
based on its height of 20, the sum of its starting Y coordinate (20) and
its height (20) is 40. From FIG. 5, there are no Y coordinates that are
greater than 20 and less than 40, and thus video window 204 is not
divided at 138. Thus, when action 138 refers to each constituent video
window being divided, this process is only for those constituent windows
that meet the criteria noted above and thus can be sub-divided. Note that
in FIG. 7, video window 204 is shown off to the right for convenience,
but actually overlaps the middle sub-window 202b.

[0033] At 140, the method further comprises dividing each Y-divided
sub-window using the left and right X coordinates determined at 136 to
thereby obtain "X and Y divided sub-windows." These are windows that have
been divided both on X and Y. Each Y-divided sub-window is divided based
on two criteria:

[0034] (1) for X values that are greater than a window's starting X
coordinate and less than the sum of the starting X coordinate and the
window's width, and

[0035] (2) only those Y-divided sub-windows are divided on X that have a
top Y coordinate that equals the starting Y co-ordinate Y (i.e., starting
Y co-ordinate of middle sub-window 202b equals to starting Y co-ordinate
of video window 204. This condition checks that two windows are
overlapping.

[0036] This process is exemplified in FIG. 8. For the upper Y-divided
sub-window 202a, the starting X coordinate is 0 and the sum of the
starting coordinate (0) and the window's width (60) is 60. Thus,
Y-divided sub-window 202a is not further sub-divided on X because both
criteria above are not met. Specifically, 0 (the starting Y co-ordinate
of sub-window 202a) does not equal 20 (the starting Y co-ordinate of
video window 204). A similar analysis applies to bottom Y-divided
sub-window 202c. However, middle Y-divided sub-window 202b is divided on
X because both criteria are met. That is, 20 (the starting Y co-ordinate
of middle Y-divided sub-window 202b) equals 20 (the starting Y
co-ordinate of video window 204). As a result, middle Y-divided
sub-window 202b is divided on X resulting in three X and Y divided
windows 202b1, 202b2, and 202b3 as shown. Note that the
middle X and Y divided sub-window 202b2 but is shown off to the
right for clarity purposes.

[0037] Referring again to FIG. 3, the X, Y coordinates of opposing corners
(e.g., upper left and lower right) of each X and Y divided sub-window
(e.g., 202b1, 202b2, and 202b3) as well as those Y-divided
sub-windows (202a and 202c) that were not further divided on X and those
video windows that were not divided even on Y (e.g., window 204) are
stored in storage 104. These are the various windows comprising the
composite video frame. Some or all of each constituent video source is
rendered into each such sub-window.

[0038] FIG. 9 describes an illustrative method 250 that determines the
order for how the various sub-windows are to be populated with video
data. The processor 102 may perform the actions illustrated in FIG. 9.
The various actions depicted in FIG. 9 can be performed in the order
shown or in a different order, and two or more actions can be performed
in parallel rather than sequentially.

[0039] At 252, the method of FIG. 9 includes computing ordering keys for
the various sub-windows resulting from method 130 of FIG. 3. An
embodiment of a formula to compute each such ordering key includes the
use of a constant. The constant preferably is greater than the largest X
coordinate (e.g., right edge) of each sub-window (which would be the X
co-ordinate of composite video 202). In the above example, the largest X
coordinate is 60 and thus for this example, the constant is chosen to be
100. The ordering key formula is computed as:

Ordering key=(Y)(CONSTANT)+X (1)

where X and Y are the starting coordinate of the sub-window and CONSTANT
is 100 in this example. Referring again to FIG. 8, the ordering keys for
the six windows shown are as follows:

[0040] At 254, the method includes determining whether multiple
sub-windows have the same value for their ordering key. In the present
example, X and Y divided sub-window 202b2 and 204 have the same
ordering key value 202. That two windows have the same value of the
ordering key means that those two windows completely or exactly overlap
one another. Thus, only one of the windows should be rendered and the
other dropped (ignored). The priority value mentioned above that is
stored in the WD data structure 128 for each video window is used in this
regard. In some implementations, a window having a lower priority than
another window is dropped. If multiple windows have the same ordering
key, then at 258, the method comprises selecting one of the windows based
on the priority value and dropping the remaining window(s). In the
present example, video window 204 has a higher priority value than video
window 202 and thus X and Y divided sub-window 202b2 is dropped and
video window 204 is used instead. This is illustrated in FIG. 10.

[0041] At 256, the ordering key values for the remaining windows are used
to determine the order of the various windows and sub-windows to
construct the composite video frame. The various windows are placed in
order from smallest ordering key value to largest. In the present
example, the order determined at 256 is: 202a, 202b1, 204,
202b3, and 203c.

[0042] At 260, processor 102 then programs the DMA engine 106 to write
video data for each sub-window from storage 104 to the encoder 108 as
composite video based on the order of sub-windows determined at 256.
Other video data transfer mechanisms besides DMA can be used as well. In
the present example, the DMA engine 106 is programmed to first transfer
video data for window 202 starting at that window's top row of pixels and
continuing with each subsequent row until the first row of sub-window
202b1 is reached.

[0043] Once 202b1 is reached and because the display renders the
pixels across each entire row of the display device 112, the DMA engine
then assembles the next rows by first pulling window 202 data from the
first row corresponding to sub-window 202b1, and then pulls data
from the first row of window 204 followed by data from window 202
corresponding sub-window 202b2. That process continues until the
first row of bottom sub-window 202c is reached at which time the DMA
writes data from window 202 corresponding to sub-window 202c to the
encoder.

[0044] As such, the composite video frame is generated on-the-fly. The
preferred embodiments described above can support any complex composite
video layout and dramatically reduces memory bandwidth requirements for
video compositions which otherwise require in-memory composition as noted
above. As a result, the load on the processor and memory is reduced.
Further, the number of sub-windows that can be supported is not dependent
on hardware.

[0045] The above discussion is meant to be illustrative of the principles
and various embodiments of the present invention. Numerous variations and
modifications will become apparent to those skilled in the art once the
above disclosure is fully appreciated. It is intended that the following
claims be interpreted to embrace all such variations and modifications.