Much of this book has assumed you already had media of some kind to play and edit—but where does this media come from in the first place? Digital media has to come from one of two places: either it's completely synthetic or it's captured from a real-world source. Capture, via devices like microphones and video cameras, is far more common.

The problem is that capture doesn't officially work in QuickTime for Java. The problem dates back to Apple's Java 1.4.1 rearchitecture, which broke QTJ and forced massive changes to the API in QTJ 6.1. One of the things that was not updated for QTJ was the ability to get an on-screen component from a SequenceGrabber, which is the QuickTime capture component. Instead, Apple just put a statement in the QTJ 6.1 documentation:

Although sequence grabbing is currently not supported in QuickTime for Java 1.4.1, it may be provided in future releases.

But if you think back to how the QTJ 6.1 situation was described in Chapter 1, you might recall that QTJ classes that didn't require working with AWT—such as the quicktime.std classes that simply wrapped straight-C calls—were unaffected by the Java 1.4.1 changes and still worked. Given that, notice in the Javadoc the package called quicktime.std.sg, which contains the SequenceGrabber class among several others. Besides, capture, per se, doesn't necessarily imply using the screen, so shouldn't it still work?

The good news is that it does. In this chapter, I'll introduce the parts of the capture API that still work in QTJ, even without official support: capturing audio, capturing to disk, and even getting captured video on screen with a little QuickDraw voodoo. QTJ still needs proper support for on-screen, video-capture preview, but there's plenty to do in the meantime.

Capturing and Previewing Audio

Audio capture is a good place to start because that sidesteps the problem of the broken video preview. There's plenty to be learned in just opening up the default microphone and looking at the incoming level data—that is, how loud or soft the incoming sound data is.

How do I do that?

Setting up audio capture requires a number of steps. You start by constructing a SequenceGrabber. This object coordinates all the capture channels (audio, video...even text capture), and allows you to set capture parameters like whether to save the captured data to disk, set a maximum amount of time to let the capture run, etc.

Note

Don't scoff—there really are text-capture devices. For example, you could capture the closed captions off regular TV (also called "line 21" data).

Once you have the SequenceGrabber, you use an optional prepare( ) call to indicate whether you intend to preview the captured media, record it to disk, or both.

To work with sound, you need to create a sound channel, by calling the SGSoundChannel constructor and passing in the SequenceGrabber. This object allows you to configure the audio capture, choose among audio capture devices (see the next lab), and get the device's driver. The driver, exposed by the SPBDevice class, provides methods for checking the input line level.

As an example, compile and run the AudioCapturePreview application as shown in Example 6-1. Note that you need to have at least one audio capture device hooked up to your computer. Most Macs come with a built-in microphone. If you don't have one, you can use a USB capture device (like a headset or external microphone) or a FireWire device (like an iSight).

Note

Compile and run this example from the book's downloadable code with ant run-ch06-audiocapture-preview.

When run, the application brings up a small window with a green bar that indicates the current level on the line, as seen in Figure 6-1. At maximum input volume—if you're speaking loudly and directly into the microphone—it will stretch all the way to the right of the window.

There is also a Preview checkbox that is off initially. Clicking this will play the captured audio over the headset or speakers.

What just happened?

The constructor does some simple AWT business, adding the Preview checkbox and an AudioLevelMeter, which is an inner class that will be explained shortly. Then it calls setUpAudioGrab( ).

setUpAudioGrab( ) is responsible for initializing the audio capture. As described earlier, the first step is to create a new SequenceGrabber object. Next, tell the grabber what you intend to do with it, via the prepare( ) method, which takes two self-explanatory booleans: prepareForPreview and prepareForRecord.

Tip

You don't have to call prepare( ). If you don't, SequenceGrabber will take care of its setup when you start grabbing, possibly making the startup take longer.

You also need to tell the SGSoundChannel what you want to do via setUsage( ), inherited from SGChannel. As with all methods that take behavior flags, you logically OR together constants to describe your desired usage. In this case, seqGrabPreview indicates that the application is only previewing the captured sound, but you can use (and combine) four other usage constants:

seqGrabRecord

Include this if you want to record the captured media to disk.

seqGrabPlayDuringRecord

Add this to play while recording.

seqGrabLowLatencyCapture

Used to get the freshest frame possible (used for video conferencing and live image processing).

seqGrabAlwaysUseTimeBase

Used by video channels to get more accurate audio/video sync.

At this point, the capture is initialized. Begin capturing audio with SequenceGrabber 's startPreview( ) method.

To create the level meter, it's necessary to get an SPBDevice, which provides low-level access to the incoming data. This object provides level meters as an array of ints by first enabling monitoring with setLevelMeterOnOff(true) and then followed by getActiveLevels( ). The returned ints range from 0 (silence), to 255 (maximum input volume). In the example, the AudioLevelMeter inner class gets the first level on each repaint and draws a box whose width is proportional to the audio level. A Swing Timer calls repaint() on the meter every 50 milliseconds to keep it up to date.

Note

There may be multiple levels in the array, usually two for stereo input.

The repaint thread also calls idle() on the SequenceGrabber , which is something you have to call as frequently as possible to give the SequenceGrabber time to operate.

Note

SequenceGrabber.idle( ) is a lot like "tasking" back in Chapter 2, except there's no convenience class to do it for you.

What about...

...defaulting the volume off with SoundChannel.setVolume() ? This is a common practice because some users' speakers will be close enough to their microphones to cause feedback when previewing the audio to the speakers. On the other hand, users with headphones probably do want to hear the preview. So, the best practice is "default off, but let the user turn it on."

Warning

One thing this demo lacks is a call to SequenceGrabber.stop() when the user quits. This is something you should usually do, but I've left it out to make a point. On Mac OS X, if you don't stop the SequenceGrabber and you leave the volume on, you will keep grabbing sound—feedback included—even after the application quits. I've even seen this behavior survive a restart. So, try it out, don't blow your speakers, and then remember to have your programs turn off the volume and call SequenceGrabber.stop( ) when they quit.

Selecting Audio Inputs

It's not realistic to think the user has only one audio input device. The computer might be connected to a headset for audio conferencing, a webcam for video conferencing, and a camcorder for dumping pictures of the summer vacation into iMovie. Ideally, it should be possible to discover connected devices at runtime and specify which is to be used for capture.

How do I do that?

To provide a list of devices, you need to query the SGAudioChannel for what devices are available, and then present the choice to the user. So, take the code from the previous lab and add an AWT Choice called deviceChoice in the constructor (replacing a line with a comment that said "reserved for next lab"). Next, after the SGSoundChannel is created in setUpAudioGrab( ), insert this block of code to search for audio devices, adding the name of each to the deviceChoice:

You need to update the itemStateChanged() callback to handle AWT events on the deviceChoice—in other words, when the user changes the selection. Fortunately, QuickTime allows you to change the input device by passing in a name, so switching devices is pretty easy. Add this to itemStateChanged( ), inside the try-catch block:

The boolean named grabbing is a simple gate to keep the repaint thread from trying to get levels while this device change is underway, because the old inputDriver will be invalid once the new device is set.

A demo of this technique, SelectableAudioCapturePreview, is shown in Figure 6-2.

What just happened?

The key to switching capture devices is a single call, SGSoundChannel.setDevice() , which lets you change device mid-grab, without pausing or doing other reinitializations. It takes a device by name, the same name that was retrieved by walking through the SGDeviceList.

What about...

...the "0" parameter on getDeviceList( ) ? This method takes flags, only one of which is even relevant to QTJ.

Actually, it's easier to explain by starting further down, with the test for whether to add a device to the Choice. The SGDeviceName used to identify the capture devices wraps not just a name string, but also an int with some flag values. sgDeviceNameFlagDeviceUnavailable is the only publicly documented flag. As seen in this example, to test for whether such a flag is set, you AND the value with the flag you're interested in and check whether the result is nonzero. If so, it means that bit is set. So, in this case, if the value is 0, the device is available (literally, it's "not unavailable"), so it's OK to let the user select it.

If we were to return to the getDeviceList( ), the only flag available would be sgDeviceListDontCheckAvailability, which skips the device availability check, meaning that flag in SGDeviceName would never be set, and thus the device would never be reported as unavailable. That's clearly undesirable behavior here—you don't want to give the user an option that's only going to throw an exception when she chooses it.

Capturing Audio to Disk

Typically, you don't just capture media and immediately dispose of it—you want to save the media to disk as you capture so that you can use it later. Fortunately, the SequenceGrabber makes this pretty easy.

How do I do that?

Adding to the previous labs' code, the calls to set up the SequenceGrabber need to be changed to prepare for grabbing to disk. Specifically, the SGSoundChannel 's setUsage() call gets a flag to indicate that it will be writing the captured audio to disk:

Next, add a call to give the user an opportunity to configure the audio capture:

soundChannel.settingsDialog( );

Warning

The settingsDialog( ) call will crash Java 1.4.2 on Mac OS X if called from the AWT event-dispatch thread. Yes, it's a bug. To work around this until the bug is fixed, you can stash the call in another thread and block on it. For instance, in this example you could replace the settingsDialog( ) call with the following:

The last step is to provide a Stop button because the data is written to disk only when the SequenceGrabber.stop() method is called. This Stop button is added near the bottom of the constructor, before the SequenceGrabber is set up:

After OKing the settings dialog, the capture begins. When the user clicks Stop, the SequenceGrabber writes and closes the audiograb.mov file and the program exits.

What just happened?

Requesting that the SequenceGrabber save to disk requires just the few extra steps detailed earlier:

Add seqGrabRecord to the channel's setUsage( ) call.

Tip

At this point, you optionally can call the channel's settingsDialog() to give the user a chance to configure the capture.

Call setOutput( ) on the SequenceGrabber.

Call SequenceGrabber.startRecord().

Also, the SequenceGrabber must be explicitly stop( )ped to write the captured data to disk.

What about...

...the SequenceGrabber.prepare() call? If the second argument is prepareForRecord, why isn't that set to true for this example? Well, inexplicably, when I did set it to true, I started getting erroneous "dskFulErr" exceptions every time I idle() d, even though I had 9 GB free. No, I don't know why—it's totally insane. But given the choice of what should work and what does work, I'll go with the latter.

And what is the deal with the settings dialog? Could that have been used in the preview examples? Yes, absolutely. In fact, it's important to let the user adjust things like gain, or to specify a compressor before grabbing begins. But that's more important when you're actually grabbing to disk, so I held off introducing it until now.

Note

Actually, it's usually best to capture uncompressed, so the CPU doesn't get bogged down with compression and possibly slow down the capture rate.

Capturing Video to Disk

Audio capture is nice, but if you bought this book because the sticky-note on the cover lists "capture" as one of the topics to be covered, you probably figured it meant video capture. Is there an iSight on the top of your monitor that wants some attention? OK, here's how to turn it on and grab some video.

How do I do that?

As with audio capture, the basics of setting up capture are:

Create a SequenceGrabber.

Create and configure (with setUsage( ) and the settingsDialog( )) the channels you're interested in—in this case, an SGVideoChannel.

Call SequenceGrabber.setOutput( ) to indicate the file to capture to.

Call SequenceGrabber.startRecord( ) to begin grabbing to disk.

Finish up with SequenceGrabber.stop( ).

There is, however, a big difference with video. With no on-screen preview component available in QTJ 6.1, you must indicate where the SequenceGrabber can draw to. The workaround is to create an off-screen QDGraphics and hand it to the SequenceGrabber via the setGWorld( ) call.

The VideoCaptureToDisk program, presented in Example 6-2, offers a bare-bones video capture to a file called videograb.mov.

When it starts up, the program shows a settings dialog for your default camera, as seen in Figure 6-4. The video settings dialog is even more important for users than the audio settings dialog, as the video dialog gives them a chance to aim the camera, check the lighting, adjust brightness and color, etc.

Just like its audio equivalent, calling SGVideoChannel.settingsDialog( ) will crash the virtual machine in Mac OS X Java 1.4.2 if called from the AWT event-dispatch thread. And just as before, you can work around this bug by firing off the settingsDialog( ) call in its own thread and blocking until the thread finishes. I've filed it as a bug, but feel free to file a duplicate to get Apple's attention.

Once you click the Stop button, the video is written to videograb.mov and the application terminates. You can view the captured movie in any QuickTime application—Figure 6-5 shows it in the BasicQTController demo from Chapter 2.

In previous versions of QTJ, this wasn't necessary because the on-screen preview provided a GWorld that the grabber could use. With no on-screen preview currently available in QTJ, this is a handy technique.

The next step is to create an SGVideoChannel from the SequenceGrabber and set its bounds. After optionally showing a settings dialog, set the usage (just seqGrabRecord this time because there's no preview) and then call prepare(false, true), which prepares the SequenceGrabber for recording but not for previewing.

Note

This time, setting the second prepare( ) argument to true is the right thing to do.

Just as with audio, the final steps are to call setDataOutput( ) on the SequenceGrabber, followed by startRecord( ). When SequenceGrabber.stop( ) is called, the file is written out and closed up.

What about...

...using this on Windows...it doesn't find my webcam! This example presupposes that a video digitizer component for your camera will be found, and a lot of video cameras don't ship with a Windows QuickTime "vdig", supporting only Microsoft's media APIs instead. However, there's hope: you can use SoftVDIG from Abstract Plane (http://www.abstractplane.com.au), which acts as a proxy to bring captured video from the Microsoft DirectShow world into QuickTime.

Capturing Audio and Video to the Same File

So, it's possible to capture audio and video in isolation. With QuickTime's editing API, it would be possible to put them in the same movie by adding each as a separate track (see Chapter 3). But wouldn't it be nice to just capture both audio and video into the same file at once, presumably keeping them in sync along the way? Fortunately, SequenceGrabber supports this, too.

How do I do that?

Starting with the previous lab's video-only example, you just need to add an SGSoundChannel in the setUpVideoGrab( ) method:

soundChannel = new SGSoundChannel (grabber);

The setUsage( ) and prepare() commands are identical to what was shown in the audio-only and video-only labs:

Beyond that, everything is the same as in the video-only case. Because the setDataOutput() call is made on the SequenceGrabber—not just on an individual channel—the grabber writes data from all the channels it's capturing into the same file, called audiovideograb.mov in this case.

Note

Run this example with ant run-ch06-audiovideocapturetodisk.

What just happened?

For once, the SequenceGrabber APIs behave pretty much as you might expect them to. With no obvious prohibition on creating both audio and video channels from the same SequenceGrabber, and assigning the grabber's output to a file, the captured data from both channels goes into a single movie file.

Making a Motion Detector

Capture isn't just about writing data to disk. You can grab images as they come in and analyze or manipulate them.

Tip

A great example of "grabbing different" is Lisa Lippincott's ScrollPlate, a demo shown at ADHOC 2004. She used her iSight camera as a scroll wheel, by holding up a Styrofoam plate with either a large green arrow (for up) or a large red arrow (for down). Her code presumably grabbed from the camera, looked at the grabbed image for an abundance of green or red, and scrolled the top window in response.

This example offers a simple motion detector, which will display an alarm message if two subsequent grabs are markedly different. The idea is that if the camera is not moving, a significant difference between two subsequent grabs indicates that something in view of the camera has moved.

How do I do that?

In this case, what you want to do is to set up video-only capture, but instead of saving the data to disk, you do a little bit of image processing each time you idle() . Specifically, there is a method in QTImage called getSimilarity( ), which compares two images (one as a QDGraphics and the other as an EncodedImage). Motion—objects entering, exiting, or significantly moving within the camera's field of vision—can be understood as a significant difference between two consecutive grabbed images.

What just happened?

This is a huge example, but much of it draws on the video-grabbing techniques of the previous two labs. setUpVideoGrab() sets up the SequenceGrabber for grabbing video, but in this case, it doesn't need to save to disk, so the setUsage() argument is seqGrabPreview, and the arguments to prepare( ) are true and false (for preview and record, respectively). A Swing Timer calls back every two seconds—the long delay is intentional, so the potential for change between grabbed frames is greater—and calls the SequenceGrabberidle( ) and update( ) methods, followed by calls to the brains of this example: scanForDifference() and updatePict( ).

scanForDifference( ) evaluates the difference between the current frame and the last one. It does this by grabbing a Pict from the SequenceGrabber and drawing it into a QDGraphics (also known as a GWorld). It compares this GWorld to an EncodedImage of the last grab, via the QTImage.getSimilarity() method. This method returns a float that expresses the similarity of the two grabbed images, where 0 means the images are totally different and 1 means they're identical. At the end of this method, QTImage.compress( ) is used to compress the grabbed GWorld into a new EncodedImage for use on the next call to scanForDifference( ).

Note

It might be better to call scanFor-Difference( ) on another thread, so the image analysis doesn't block the repeated calls to SequenceGrabber.idle( ).

updatePict( ) updates a GraphicsImporter that is used to provide the preview image in the middle of the window. This uses a Pict-to-GraphicsImporter trick that was introduced in Chapter 5sSection 5.4 lab. In this case, it's used not to get a Java AWT Image, but to get new pixels into a GraphicsImporter, which is wired up to a QTComponent for on-screen preview.

What about...

...the ideal value for triggering a difference? It probably depends on lighting, your camera, and other factors. In a professional application, you'd want to give the user a slider or some similar means of configuring the sensitivity of the detection.

Also, there seems to be a lot of inefficient code here, particularly with drawing into the newImageGW. Why is that necessary when the Grabber was initially set up with a brand-new off-screen QDGraphics /GWorld? This, admittedly, is weird. When I was debugging, I found that the GWorld used to set up the Grabber is drawn to once and never again. On the other hand, the Pict generated from SequenceGrabber.grabPict( ) is always fresh, so that's what's used for testing similarity. However, to apply the getSimilarity( ) method, you need to have a GWorld, so you Pict.draw( ) the pixels from the Pict into the GWorld.

Come to think of it, with this application updating the component with a new grab every couple of seconds, isn't that effectively an on-screen preview? Yes, it is, in an extraordinarily roundabout way. You could take out the motion-detecting stuff and make a preview component by just grabbing a Pict each time, making a new Pict with a 512-byte header, setting the GraphicsImporter to read that, and calling GraphicsImporter.draw( ) to draw into its on-screen component. I didn't split that out as its own lab because the performance is pathologically bad (one frame per second—at best), and because it's an awkward workaround in lieu of a better way of getting a component from a SequenceGrabber. Presumably, someday there will be a proper call to get a QTComponent from a SequenceGrabber—maybe another overload of QTFactory.makeQTComponent( )—and kludgery like this won't be necessary.