Saturday, May 11, 2013

Frames Capture and Video Creation Using Xuggler

The concept is to open
the media file, loop through a specific video stream and at specific intervals
capture the corresponding frame, convert it to an image and dump the binary
contents into a file. Here is what the code for all these looks like:

//
if uninitialized, back date mLastPtsWrite to get the very first frame

if(mLastPtsWrite ==
Global.NO_PTS)

mLastPtsWrite
= event.getTimeStamp() - MICRO_SECONDS_BETWEEN_FRAMES;

//
if it's time to write the next frame

if(event.getTimeStamp()
- mLastPtsWrite >=

MICRO_SECONDS_BETWEEN_FRAMES)
{

String
outputFilename = dumpImageToFile(event.getImage());

//
indicate file written

doubleseconds = ((double)
event.getTimeStamp()) /

Global.DEFAULT_PTS_PER_SECOND;

System.out.printf(

"at
elapsed time of %6.3f seconds wrote: %s\n",

seconds,
outputFilename);

//
update last write time

mLastPtsWrite
+= MICRO_SECONDS_BETWEEN_FRAMES;

}

}

privateString
dumpImageToFile(BufferedImage image) {

try{

String
outputFilename = outputFilePrefix +

System.currentTimeMillis()
+ ".png";

ImageIO.write(image, "png", newFile(outputFilename));

returnoutputFilename;

}

catch(IOException e) {

e.printStackTrace();

returnnull;

}

}

}

}

This might seem a bit overwhelming, but it is really quite straightforward. Let
me provide some details for you. We start by creating anIMediaReader from an input file. The media reader is used to read and
decode media. Since we wish to manipulate the captures video frames as images,
we use the setBufferedImageTypeToGenerate method to denote this. The reader opens up
a media container, reads packets from it, decodes the data, and then dispatches
information about the data to any registered IMediaListener objects. Here is where our custom class,
ImageSnapListener, comes into play.

Our listener extends the MediaListenerAdapter, which is an adapter (provides empty methods)
implementing the IMediaListener interface. Objects that implement this
interface are notified about events generated during the video processing. We
only care about handling video events, thus we only implement the IMediaListener.onVideoPicture method. Inside that we use the provided IVideoPictureEvent object to find what stream (video only) we
are dealing with.

Since we wish to capture frames in specific times, we have to mess a little
with timestamps. First, we make sure we handle the very first frame by checking
against the value of the Global.NO_PTS constant, which is a value that means no time stamp is set
for a given object. Then, if the minimum elapsed time has passes, we capture
the frame by invoking the IVideoPictureEvent.getImage method, which returns the underlying BufferedImage. Note that we are talking about elapsed video time and not “real
time”. We then dump the image data to a file in PNG format using the ImageIO.write utility method. Finally, we update the last write time.

Let’s run this application in order to see the results. As input file, I am
using an old Sun commercial proclaiming that “Java
is Everywhere”. I have downloaded
locally the MP4 version provided. Here is what the output console will look
like:

at elapsed time of 0.000 seconds wrote:
c:/snapshots/mysnapshot1298228503292.png
at elapsed time of 10.010 seconds wrote:
c:/snapshots/mysnapshot1298228504014.png
at elapsed time of 20.020 seconds wrote:
c:/snapshots/mysnapshot1298228504463.png
…
at elapsed time of 130.063 seconds wrote:
c:/snapshots/mysnapshot1298228509454.png
at elapsed time of 140.007 seconds wrote:
c:/snapshots/mysnapshot1298228509933.png
at elapsed time of 150.017 seconds wrote: c:/snapshots/mysnapshot1298228510379.png

The total video time is about 151 seconds so we capture 16 frames.

In order to create video, we will have to take a bit more low level approach in
comparison to the MediaTool API that we have seen so far. Don’t worry though, it is not
going to be complicated. The main idea is that we create a media writer, add
some stream information to it, encode our media (the screenshot images), and
close the writer. Let’s see the code used to achieve this:

packagecom.javacodegeeks.xuggler;

importjava.awt.AWTException;

importjava.awt.Dimension;

importjava.awt.Rectangle;

importjava.awt.Robot;

importjava.awt.Toolkit;

importjava.awt.image.BufferedImage;

importjava.util.concurrent.TimeUnit;

importcom.xuggle.mediatool.IMediaWriter;

importcom.xuggle.mediatool.ToolFactory;

importcom.xuggle.xuggler.ICodec;

publicclassScreenRecordingExample {

privatestaticfinaldoubleFRAME_RATE
= 50;

privatestaticfinalintSECONDS_TO_RUN_FOR
= 20;

privatestaticfinalString
outputFilename = "c:/mydesktop.mp4";

privatestaticDimension
screenBounds;

publicstaticvoidmain(String[] args)
{

// let's
make a IMediaWriter to write the file.

finalIMediaWriter writer
= ToolFactory.makeWriter(outputFilename);

screenBounds
= Toolkit.getDefaultToolkit().getScreenSize();

// We tell
it we're going to add one video stream, with id 0,

// at
position 0, and that it will have a fixed frame rate of FRAME_RATE.

// if the
source image is already the target type, return the source image

if(sourceImage.getType()
== targetType) {

image
= sourceImage;

}

//
otherwise create a new image of the target type and draw the new image

else{

image
= newBufferedImage(sourceImage.getWidth(),

sourceImage.getHeight(),
targetType);

image.getGraphics().drawImage(sourceImage, 0, 0, null);

}

returnimage;

}

privatestaticBufferedImage
getDesktopScreenshot() {

try{

Robot
robot = newRobot();

Rectangle
captureSize = newRectangle(screenBounds);

returnrobot.createScreenCapture(captureSize);

}

catch(AWTException e) {

e.printStackTrace();

returnnull;

}

}

}

We start by creating an IMediaWriter from a given
output file. This class encodes and decodes media, handling both audio and
video streams. Xuggler guesses the output format from the file name extension
(in our case MP4) and sets some default values appropriately. We then use the addVideoStream method to add a
new video stream, providing its index, the codec type used (MPEG-4 here) and the video dimensions. The
dimensions are set equal to half of the screen’s dimensions in this example.

Then we execute a loop that runs for a number of times equal to the desired
frame rate multiplied with the desired running time. Inside the loop, we
generate a screen snapshot as described in the Java2D: Screenshots with Java
article. We retrieve the screenshot as aBufferedImage and convert it to
the appropriate type (TYPE_3BYTE_BGR) if it is not already
there.

Next, we encode the image to the video stream using the IMediaWriter.encodeVideo method. We provide
the stream index, the image, the elapsed video time and the time unit. Then, we
sleep for the appropriate number of time, depending on the desired frame rate.
When the loop is over, we close the writer and write the trailer if necessary,
depending on the video format (this is done automatically by Xuggler).

If we execute the application, a video will be created which has recorded your
desktop actions.