4. Concepts and Basic Programs

At present, the documents that are available for understanding libdc1394 are very few. Among the available documents are a FAQ, which only talks about the general concept of the camera and a few forums where people have posted their queries.

We had to read the libdc1394 source code to understand how to grab a frame from the camera. The extreme need for a complete explanation on this topic encouraged us to write this HOWTO.

On the basis of what we understood by exploring the source code of the camera, we are presenting our insights on the library.

We encourage you to keep referring following files while reading through the next section:

dc1394_control.h

dc1394_control.c

dc1394_capture.c

dc1394_format7.c

The dc1394_control.h file is the most important, as it contains the declaration of various functions and data structures. It should be read to understand the features supported by IEEE1394 cameras and the functions which access and control various features of camera.

The dc1394 library works closely with raw1394 since all the functions make use of functions of raw1394 (raw1394_read() and raw1394_write()) in order to retrieve or modify the values of various parameters of the camera. This is one of the reasons why raw1394 must be installed before installing libdc1394.

To understand how the functions provided by the library work, we need to understand few technical aspects of the IEEE1394 camera:

There can be more than one camera connected to the host card at any time, hence one need to uniquely identify the camera node.

The camera has some control registers to set camera features.

According to IEEE specifications, the camera can grab images in different formats. The image formats are defined by two parameters: format and mode. Five modes are defined:

Format_0 contains low resolution mode up to 640x480

Format_1 contains medium resolution modes: 800x600 and 1024x768

Format_2 contains mega pixel modes: 1280x960 and 1600x1200

Format_7 is the scalable image format. With this, you can change the image size, color coding and other parameters

It may be possible that a camera does not support some modes, so we need to be careful during the setup. You use setup to set various parameters like data speed, image mode, framerate that are necessary to make the camera ready to grab the image. We will shortly discuss setup functions.

In order to set or get the values of the parameters/features of the camera, the library functions will modify or read the register values.

The camera can work with or without DMA. We are greatful to Ian Peikon and Tim Hanson for contributing a section on IEEE1394 camera using DMA support

With the above background, we begin our discussion about the
various functions, data structures and other miscellaneous features
available in the dc1394_control.h file.

We would like to mention one important aspect of the dc1394_control.h file, which is that it lists out the values that can be set for the data speeds, frame rates, camera modes and image formats. All the values are in the form of enumerations to help you write readable code. For example, enumeration for speed indicates that the data speed can be set only as 100, 200, 400, 800, 1600 or 3200.

The dc1394_control.h also lists the features (brightness, hue, sharpness, saturation, etc.) supported for your camera.

4.1. Important Data Structures

The library provides some structures which are useful in
storing camera and image related information. They have been
declared in the dc1394_control.h file.

Structure

Use

dc1394_camerainfo

Helps in accessing the information about the camera model, vendor, ccr_offset (used to access the registers).

dc1394_cameracapture

Contains various parameters that are to be manipulated
while the camera setup. It also declares a buffer of int type that will be used to store the grabbed frame.

Contains the information about a particular feature (brightness, hue, gamma, white balance, etc.,) for example the availability and values for it.

We will be largely concerned with the dc1394_cameracapture structure.

4.2. Types of functions

We have categorized the functions provided by the library
into 6 types. (The categorization is purely done on our own for the
ease of discussion). Each of these functions has been declared in
dc1394_control.h. We suggest you to keep a copy of the file on hand while going through the subsequent section.

Get/Query Functions

Set Functions

Print Functions

Setup Functions

Format7 Functions

Release Functions

4.2.1. Get/Query Functions

These functions are used to get the value of various
features of the camera, the information about the ISO channel,
trigger mode, frame rate, format and mode. The functions contain
get or query in their names.

Both the raw1394handle_t
and node_t nodes: these two together identify the camera

A pointer to obtain the value, for example,
int * gamma

If we try to trace the flow of the function call, we can understand what actually happens:

Figure 1. Flow of get function call

In case the get function you called is
meant to retrieve the value of a camera feature (the value for the
parameters listed in feature enumeration), that
get function will call another
get function (GetFeatureValue), which takes the enumeration value of the feature as input. For example, in the
case of gamma value, the function passes 422 as the value (this
value can be calculated from the enumeration given in the
dc1394_control.h file).

Ultimately, the
GetCameraControlRegister function, whose task is
to get the value from the appropriate control registers, is
called. This functions takes the offset value (octlet_t
offset), which is the offset from the base register as
the input to its call. The function prototype is declared in dc1394_internal.h.

In this way, the user obtains the value of the camera parameter he queried for.

4.2.2. Set Functions

These functions are used to set the values of various
camera features. There is a corresponding set function for almost
each get function. You can recognize these functions by searching
for set string.

For example,

dc1394_set_gamma (raw1394handle_t handle, nodeid_t node, int
gamma)

Like the get function, this function needs
raw1394handle_t and nodeid_t for camera identification.

The other parameter, gamma, is the user-specified value for the gamma parameter.

The flow of the function call is quite helpful in
understanding what is actually happening. The flow is exactly
same as that of the get function. The only
difference is this time all the intermediate functions are also
set functions, and instead of
raw1394_read(), it uses raw1394_write() to write the value of the camera parameter on to the registers.

Figure 2. Flow of set function call

4.2.3. Print Functions

There are three print functions available:

dc1394_print_camera_info:
This function is used to print the values stored in the
dc1394camera_info structure. The print function is usually
called after dc1394_get_camera_info() to ensure that the dc1394camera_info structure is not NULL.

dc1394_print_feature: This function is used to print the value of any feature. It takes as an input the pointer to dc1394_feature_info structure.

dc1394_print_feature_set: This function is used to print the values of all features present in the camera. It takes as an input the pointer to dc1394_feature_set structure. The function call basically calls dc1394_print_feature repeatedly in a for loop.

4.2.4. Setup Functions

As the name suggests, these functions are used to prepare
the camera to start grabbing the images. In order to setup the
camera, some parameters must be passed to the function. The
number and the type of parameters are specific to the setup
function, but essentially three parameters must be passed in
all the setup functions: raw1394handle_t,
nodeid_t and the pointer to the dc1394_cameracapture structure (this is to provide the buffer for grabbed images and keep attributes, like height and width of frame).

As mentioned previously, raw1394handle_t and nodeid_t uniquely define the camera, while the pointer to dc1394_cameracapture provides the buffer for the image frame to be grabbed. It also stores information about the width and height of the frame which is useful at the time of image processing.

In case the camera uses DMA, the number of DMA buffers and drop frames is to be provided. In most applications you will want to set the number of DMA buffers relatively low(i.e. 1) this ensures that you are viewing frames in near real time. Drop frames is also important for real time as it causes the capture function to throw away the frames buffered in the DMA ring buffer except fot the last. We will discuss about DMA functions in the later part of this HowTo

The various setup functions are:

dc1394_dma_setup_capture

dc1394_setup_capture

dc1394_setup_format7_capture

dc1394_dma_setup_format7_capture

These setup functions have been defined in dc1394_capture.c.

You have a choice while passing the values for the
parameters, like data-speed, format, mode, and channel. Instead
of providing the value yourself, you may instruct the function to
take the value from the camera. This can be done by passing
QUERY_FROM_CAMERA in place of the actual value for that function parameter.

The basic flow of control is easy to understand:

Figure 3. Flow of the setup function call

As shown in Figure 3, the Setup function in turn calls various set functions to set the parameter values to the camera registers. In the case of DMA setup, after the set functions have been called the ioctl system call is called to allocate memory for the DMA buffers and returns a pointer capture_buffer in user space.

The ioctl system calls will fail if the DMA is not setup correctly

The setup functions also allocate memory for the camera _capture buffer:

camera->capture_buffer=(int*)malloc(camera->quadlets_per_frame*4);

4.2.5. Format7 Functions

These functions are used only if the camera is set for Format7. This format is preferred since this allows the user to define the size of the image to be captured according to his need. By default the size is 1024x768; you can set it to different dimensions, say 960x720.

All Format7 functions have format7 in
their function names, and the functions have been defined in a
separate file, dc1394_format7.c.

The setup function for Format7 has a minor difference from the normal setup since it also asks for the size of the frame, while you don't have to pass the format parameter as the setup function, as it is meant only for a particular format, for example Format7. The function call flow remains the same as discussed in the previous section.

The Format7 get functions are called
query functions. The mechanism is different from the normal
get/query functions: they don't call
GetCameraControlRegister; instead, they call GetCameraFormat7Register().

The following flowchart will make the differences evident:

Figure 4. Flow of the Format7 query function call

The Format7 query function will call
GetCameraFormat7Register, which is supposed
to read the values from the control and status registers of the
camera. This function in turn will call the
QueryFormat and
CSROffset to know the offset for the
particular information that has been queried. After getting the offset, raw1394_read is used to actually read the values.

The Format7 set functions also follow the same logic with
the obvious difference that the reading functions are replaced by
writing functions, for example, SetCameraFormat7Register and raw1394_write().

4.2.6. Release Functions

These are the final set of functions identified by us. The basic job of these functions is to release the memory allocated to the capture buffer by the setup routine. This is essential to save the system from memory leaks.

These functions are defined in dc1394_capture.c:

dc1394_release_camera()

This function in turn calls free (camera -> capture_buffer), which frees the memory.

Similarly, the release function is available for DMA setup.

4.3. Coriander: A GUI for the libdc1394 library

Coriander helps in easy handling
of the IEEE1394 cameras. It uses the above discussed functions and
libraries and provides a GUI for them. The main advantage of Coriander is that it saves time that is
normally wasted in camera setup. Also, Coriander shows only those features and
attributes that are present on the camera and hence you can judge
how useful the camera will be for your application
development. The most important feature of Coriander is its ability to display the captured image at run-time.

Coriander also allows the user to
convert a BGGR image to RGB. We will discuss the meaning of these
types of images in detail in later sections. Some files that can be
useful in understanding the functionality of Coriander are:

Our use of Coriander was limited
only to checking that the camera was working properly and confirming
the focus was correct. We will give some more usage information
for Coriander in later sections.

4.4. Example: How to grab image from the IEEE1394 camera

In this section we will demonstrate how to write a small
program to grab an image from the camera. We have taken the
program (grab_gray_image.c) given in the examples in the library tar file. We have removed some lines to increase the readability of the code. We have provided the explanation for this code below. In order to provide you with a clear picture of which section of the code does what, we grouped the code lines together by task below.

The raw1394handle_t and nodeid_t * are required to uniquely identify the camera. Moreover, raw1394handle_t is used to hold the handle that is created for the OHCI-compliant host card.

Open ohci and assign a handle to it.

This is done by:

handle=dc1394_create_handle(0)

Where handle is of the raw1394handle_ttype> type. The parameter 0 refers to the position of the camera on host card (there may be more than one slot on the host card; 0 means that the camera is on the first slot ).

If the wrong number is passed, the handle will not be created.

Get camera nodes:

There can be more than one camera node, since IEEE1394 supports multiple devices on a single port. But for our discussion we will assume that only a single camera is present. Here is how to get nodes:

Our suggestion is that the various parameters should be first queried from the camera and then passed. This is helpful in a successful setup since often the user does not know actual values of the various parameters that have to be passed in the setup function call, and ends up passing the wrong values. In such situations the setup is not done correctly and the camera doesn't get initialized.

We are listing the appropriate get functions that should be called first in order to obtain the correct values of the setup parameters:

dc1394_get_iso_channel_and_speed(handle,camera_nodes[0], &channel,&speed); /* to get the channel and the data speed */

dc1394_get_video_format(handle,camera_nodes[0],&format); /*to get the format */

dc1394_get_video_framerate(handle,camera_nodes[0],&framerate); /* to get the framerate*/

dc1394_get_video_mode(handle,camera_nodes[0],&mode); /* to get the mode */

We have passed camera.node which indicates that the dc1394_cameracapture structure is being used and only the particular node for which the camera has been setup is being referred to, but we can also use camera_nodes[0].

Have the camera start sending the data to the user. This is done by starting the ISO transmission. The following function is used:

dc1394_start_iso_transmission(handle,camera.node)

Capture one frame by calling the function:

dc1394_single_capture(handle,&camera)

Where camera is a pointer to the structure dc1394_cameracapture. This function will grab the image and store it in the buffer (capture_buffer) provided by the structure.

In order to capture more than one frame use a for loop and place the function inside it:

Use fwrite
to save the captured image, by writing the buffer
(camera.capture_buffer) to a file. The
other parameters like height and width of the image can be
extracted from the same structure: camera.frame_width, camera.frame_height.

where the values of the parameters for channel, speed, bytes_per_packet, speed, mode are found using the following functions:

dc1394_get_iso_channel_and_speed(handle,camera_nodes[0], &channel,&speed) /* to get the channel and the data speed */

dc1394_get_video_mode(handle,camera_nodes[0] &mode); /* to get the mode */

dc1394_query_format7_byte_per_packet(handle, camera_nodes[0], mode ,&bytes_per_packet); /* to get the bytes per packet which depends on the mode*/

The value of left, top can be set to QUERY_FROM_CAMERA, or can be specified directly by the user.

The value of the width and height depends on the size of frame the user wants, for example, if one wants 960x720 then pass 960 as width and 720 as height.

4.5. Example: How to grab image from the IEEE1394 camera using DMA

This section has been contributed by Ian Peikon <idp2@duke.edu>and Tim Hanson <tim@hardcarve.com>
We will now present an example of how to grab a frame by the camera using DMA calls. The example is simple to understand as the logic remains the same as the previous example. We will later compare the function calls used in the previous example (without dma) and this example (with dma) for better understanding.

Just like the previous example (Section 4.4) the program includes the necessary header files(libraw1394/raw1394.h and libdc1394/dc1394_control.h) and then declare the three variables of the following data type

dc1394_cameracapture

raw1394handle_t

nodeid_t *

After this the following steps are followed which are very similar to the previous example (Section 4.4)

Open ohci and assign handle to it using the function dc1394_create_handle(0)

Get the camera nodes using the function dc1394_get_camera_nodes

Find out various parameters need to be passed during the camera setup. This is done by querying the camera using the following functions

dc1394_get_iso_channel_and_speed(): To get the channel and speed information

dc1394_get_video_format(): To get the camera format

dc1394_get_video_framerate(): To get the framerate information

dc1394_get_video_mode(): To get the camera mode

Also we are required to set the dma buffersto 8 and number of drop framesto 1.

Once the values of the parameter have been obtained (framerate,format,mode,channel,speed etc), they are passed in dc1394_dma_setup_capture which then sets up the camera and enables it for capturing video. For DMA you must pass the number of dma buffers and drop frames parameter.

After the camera is set, dc1394_start_iso_transmission()is called. This function starts data transactions from the camera to the bus.

Now capture one frame using dc1394_dma_single_capture(). This function will capture a frame into the dma buffer

After the frame has been grabbed release the dma buffer using dc1394_dma_done_with_buffer(). This is required so that the memory can be reused. It is imperative that dc1394_dma_done_with_buffer() always follows a dma_capture function call

4.6. How to get color images: Bayer Pattern Concepts

The image grabbed by the sample code in the previous section
is not colored (we have intentionally used the words "not colored," since the image is not gray-scale either). It is actually a Bayer Pattern. We will give an overview of Bayer Patterns and how they are used to get a colored image in this section.

Digital cameras use a solid-state device called an image sensor. These fingernail-sized silicon chips contain millions of photosensitive diodes called photosites. When you take a picture with a digital camera, the intensity of light hitting each photo site on the sensor is recorded as a signal. Depending on the camera, either 12 or 14 bits of data are recorded. At 12 bits, the camera can record 4,096 levels of brightness. At 14 bits, the camera can record 16,384 levels of brightness. This is referred to as bit depth. The higher the bit depth, the finer is the detail, the smoother the transition between tones, and the higher the dynamic range (the ability of the camera to hold detail in both highlighted and shadowed areas). But at capture, digital images are grayscale, not color. To record color information, each pixel on the sensor is covered with a red, green, or blue filter, with the colors alternating. A common arrangement of color filters is the Bayer Pattern array that alternates colors, but that also uses twice as many green filters as red and blue. Twice as many green filters are used because our eyes are more sensitive to green. This pattern, or sequence, of filters can vary, but the widely adopted Bayer Pattern, which was invented at Kodak, is a repeating 2x2 arrangement. Each pixel has been made sensitive only to one color (one spectral band).

A Typical Bayer Pattern will look like this:

Figure 5. Bayer Pattern

The tile or square (pixel) labeled B means this particular tile is sensitive only to Blue light, and so on.

The Bayer Patterns may be classified into 4 types, depending on how we have arranged the colors. The naming of the Bayer Pattern is done by taking a 2x2 matrix from the top most corner of the pattern and the colors being read in (0,0),(0,1),(1,0),(1,1) order. So for the above Bayer Pattern, if we take the 2x2 matrix as:

Figure 6. BGGR Pattern

The pattern is therefore known as BGGR

The other possible patterns are:

Figure 7. Other Patterns

The image we obtained in the previous example was a Bayer
Pattern image also known as a RAW image. This
was stored in camera.capture_buffer. In order
to view what we have captured we convert this RAW image to .PGM by
adding a header (look at the explanation in Section 4.4).

In order to get a colored image, the Bayer Pattern image is
converted to a RGB image. A RGB image is an enhanced version of the Bayer Pattern image; we try to find the value of the two missing colors at each pixel (remember that each pixel of the sensor is covered by Bayer Pattern filter so we get a single color at any pixel by default). This is done by using different algorithms like Nearest Neighbor, Edge Sense, and so on:

Figure 8. RAW to RGB

where the shaded values are to be calculated by the algorithm. Subscript denotes the tile on the Bayer Pattern to which the value of R, G, and B belongs. Note that the image size will become 3 times the Bayer Pattern. In order to view the RGB image we convert it to a Bit Map, or .BMP image, by adding a bitmap header.

To get a clear picture of what's happening, we have provided the following diagram:

Figure 9. Bayer to Bitmap conversion

Let's understand how the RAW to RGB conversion algorithms
work. We will look into the Nearest Neighbor algorithm in
detail. Other Algorithms are thoroughly explained at the following
web link: http://www-ise.stanford.edu/~tingchen/main.htm.

Most of these algorithms make use of some kind of interpolations.

4.6.1. Nearest Neighbor Algorithm

In this interpolation method, each interpolated output pixel is assigned the value of the nearest pixel in the input image. The nearest neighbor can be any one of the upper, lower, left or right pixels.

An example will make the logic clear. We try to find the G values for the R, B tiles for a 3x3 block (shown as shaded region. The blank squares either bear R value or B value. We have not shown them just to make the figure easy to understand). Here we assume the left neighboring pixel value is used to fill the missing ones.

Figure 10. Nearest Neighbour

The table on the left shows the G values for the Bayer Pattern image. In order to find out the missing G values for the other squares that originally contains only R or B we use the following approach:

Find the nearest G value square and copy the G value of that square onto the R (B) square. This has been illustrated in the above figure. The square next to G7 had a value of either R or B. So to get G8 (a G value for square 8) we copied the G value of square 7, since it was the nearest so G8 = G7. Similarly we filled the other non G value squares.

The same logic is applied when finding R and B values for the green squares.

4.6.2. Example program to understand how the colored image is grabbed using IEEE1394 Camera

Now that we have presented the basic concept of the RAW, RGB and the conversion algorithm, we feel that you can understand an example program that gives us a colored image.

We have chosen the format as Format7 because the camera we
used responds to this format only. We will make use of another
example code which provides the implementation of the
algorithms. The program is conversions.cpp,
in the grabdma
folder, available for download at http://www.ptgrey.com/support/kb/data/grabdma.tgz.

To run the code, make sure that you have the following files:

conversions.h

conversions.cpp

grabcolor.cpp - also
from the grabdma
folder. We have modified the code according to our requirements
and removed some of the lines. Since most of the code is same
as the previously discussed code, we have explained only the
portions that are different. These portions have been emphasized.

As we have already discussed the use of
GetCameraControlRegister, you can understand
that it has been used to find out the value contained at 0x1040. Libdc1394 does not provide any function to query this address, so we explicitly used this call to get the value.

It is important to understand the utility of the above
function call. Refer to our discussion about the Bayer Pattern
filters in the previous section. We know that the pattern can be
BGGR, RGGB, GRBG, and GRBG. The algorithm that we use for
converting the Bayer Pattern to RGB requires to know about the
type of pattern the camera filter has got so that it can carry out
some initialization (refer to conversions.cpp for details). The fourth parameter in the function call:

BayerNearestNeighbor is the function
call for the interpolation algorithm we discussed in the last
section. This is implemented in conversions.cpp.

Having understood this, we move on to the
switch-cases. The value that is obtained for
the Bayer Pattern (qvalue) is in the hex form which needs to be decoded for setting the value of the variable pattern.

The case statements indicate the various hex values relating
to the Bayer Pattern that can be returned by the various cameras. The intimidating values are actually quite easy to decode. Here's the trick:

The hex code for various colors is as follows:

42h -> B

47h -> G

52h -> R

Now if the qvalue contains 0x42474752, it means: B (42h) G (47h) G (47h) R (52h) or BGGR. We can therefore decode all the case statements similarly.

Finally, we need to declare another image buffer that will
contain the RGB image. Remember that the size of the RGB is 3 times the
size of the Bayer Pattern image. rgbbuffer
is therefore assigned 3 times the buffer size
(camera.capture_buffer). This buffer
(rgbbuffer) will be passed as the destination buffer (the second parameter in BayerNearestNeighbor).

After the buffer has been filled with the RGB values we
write it in the file image.rgb.

In order to view this image using gimp, we need to append a bitmap header. The method we employed was as follows:

There are many algorithms that are available in conversions.cpp, so depending upon the requirements, you can call the necessary functions.

The Coriander application is useful in finding out how the RGB image will look after undergoing different conversion algorithms. For example, it provides the choice between Nearest Neighbor, Edge Sense and Down Sample conversion algorithms. The difference can be observed on a mouse click.

4.7. Common Problems Faced While Working With IEEE1394 Cameras

We now look at some of the common problems faced while working with the camera, however we have only listed those problems which we faced. These problems are likely to be faced by anyone who uses the library and the camera. Some of the problems were solved with a proper reasoning but some were just hit-and-miss attempts. The problems and their solutions are listed below.

Problem #1. After the installation of the library and the modules the
camera did not respond

Solution: We disconnected the camera and
connected it again. May be this has to be repeated multiple
times. We could not figure out an appropriate reason for this
but it works.

Problem #2. Program is not compiling

Solution: If one forgets to link
the executable with raw1394 and dc1394_control the program
won't compile. The correct way is:

Solution: Check for the
environment variable LD_LIBRARY_PATH. This should
contain the path of the shares library. On our system the path was
/usr/local/lib. The path must
be set accordingly.

Problem #4. Program hangs when executed.

Solution: This primarily happens
if the values of parameters that are passed to the setup
function are not supported by the camera. For e.g. setting the mode
mode_640x480_MONO16 when it is not supported. Our suggestion is
that one should always query for the values and then pass them
to the function (as explained in the example code, located in
Section 4.4).

Solution: (contributed by Ian and Tim). There are various reasons for this error message. Type dmesg in the command prompt to gain a better understanding of the problem. Here is a list of problem we ran into
[1]

Problem #6. Used CTRL+C
to terminate the image grabbing program, but the next time the
program is run, it just hangs.

Solution: The error usually
occurs because the camera functions like a pipe and experiences
a broken pipe situation if terminated without flushing the
memory channels. The trick is to disconnect the camera and
connect it again. One can also try debugging the problem by
reinstalling the ohci1394, video1394, raw1394, video1394
modules, which works occasionally.

Problem #7. Every parameter is okay and the installation is also
fine, but even then, when the program is executed the camera is
still not detected.

Solution: This is a strange problem which we also face frequently. The solution is based on trial and error, but it always works:

Disconnect the camera first and then reconnect it.

Now run Coriander
(that's why we suggest that you install Coriander before you start working with the camera).

Make all the necessary adjustments; for example, select the mode of the camera, the frame size, and so forth.

Close Coriander.

Execute your program.

Bingo!!! It works.

Note on first step above:

The first step is critical since we experienced an error
message from Coriander about not
finding the camera if we did not disconnect and reconnect the camera before running Coriander.

Notes

Channel 0 is not being used- This means you are listening on the iso channel and then calling a capture function again. Make sure not to call dma_unlistenuntil you are ready to stop iso transmission.

Channel 0 is already being used- This means you have set two camera nodes to the same Channel. Fix your dma_setup_capture(). This could also mean that you have tried to call the setup function more than once without releasing the camera between the calls. This error is more likely to occur when you are working with more than one camera.

Buffer 0 is already being used- This means that you have not freed the DMA buffer before trying to write to it again. Make sure to alternate dma_done_with_buffer calls with dma_capture_calls