CSI Cameras on the TX2 (The Easy Way)

I love Nvidia’s new embedded computers. The Nvidia Jetson embedded computing product line, including the TK1, TX1, and TX2, are a series of small computers made to smoothly run software for computer vision, neural networks, and artificial intelligence without using tons of energy. Better yet, their developer kits can be used as excellent single board computers, so if you’ve ever wished for a beefed up Raspberry Pi, this is what you are looking for. I personally use the Jetson TX2, which is the most powerful module available and is widely used.

One of the big fallbacks with Jetson devices is that the documentation does not (and cannot) cover all use cases. The community has yet to mature to the point where you can find some random blog’s guide on any random thing you need to do (à la Raspberry Pi and Arduino), so you’ll often have to figure out things for yourself.

But, I am here to dispell the mystery around at least one thing — using CSI cameras on your TX2. These methods should work on other Jetson devices too!

We’re going to look at utilizing the Jetson’s image processing powers and capturing video from the TX2’s own special CSI camera port. Specifically, I’ll show you:

Why you’d even want a CSI camera.

Where to get a good CSI camera.

How toget high resolution, high framerate video off your CSI cameras using gstreamer and the Nvidia multimedia pipeline.

Why CSI cameras (vs USB)?

CSI cameras should be your primary choice of camera if you are looking to push for maximum performance (in terms of FPS, resolution, and CPU usage) or if you need low-level control of your camera — and if you are willing to pay a premium for these features.

USB cameras, on the other hand, can be incredibly cheap, typically work out of the box via the V4L2 protocol, and are an excellent choice for applications where you don’t need high-performance video. You can get 720p video for only $20 using the Logitech C270, as California Polytechnic State University did in their well documented ‘Jet’ Robot Kit, which was enough to make their robot toy car identify and collect objects, find faces, locates lines, etc.

Can do a lot of the image work off-board (exposure control, frame rate, etc).

Many provide inputs/interrupts that can help time your application (e.g. interrupt on new frame).

Use CPU time due to USB bus, this will impact your application if it uses 100% CPU.

Are not optimal for use of hardware vision pipeline (hardware encoders, etc).

Can work over long distances (up to max of USB standard).

Can support larger image sensors (1″ and higher for better image quality and less noise).

CSI Bus Cameras:

Optimized in terms of CPU and memory usage for getting images processed and into memory.

Can take full advantage of hardware vision pipeline.

Short distances from TX1 only (10cm max usually) unless you use serialization systems (GMSL, FPD Link, COAXPress, Ambarella) which are immature and highly custom at the moment.

Are mostly smaller sensors from phone camera modules but custom ones can be made at a price. The added noise from the smaller sensor can be mitigated a bit through the hardware denoise in TX1/2.

Gives you access to low-level control of the sensor/camera.

I recommend you check out the full post for further insights, such as considerations for networked cameras.

Why do CSI cameras perform better than USB?

The biggest issue with USB is bandwidth and processing needs. USB 3.0 can push 5 Gbps, which is technically enough to allow push uncompressed 1080p video stream at 60fps or even 4k (3840p) at 20 fps (see for yourself). But, this based on bandwidth alone and does not reveal the truth of the additional processing and memory management bottlenecks in handling the video. For example, the See3CAM_CU130 USB 3.0 camera should be capable of 60fps 1080p, but in a real world test on the TK1 it only eked out 18 fps at 1080p compressed and a paltry 1fps uncompressed. While performance would be better on a more powerful machine, this is evidence of the problem.

In contrast, the Jetson TX1 and TX2 utilizes “six dedicated MIPI CSI-2 camera ports that provide up to 2.5 Gb/s per lane of bandwidth and 1.4 gigapixels/s processing by dual Image Service Processors (ISP).” In other words, it has the bandwidth for three 4k cameras (or six 1080p cameras at 30 fps). Again, bandwidth isn’t everything because those images need to be moved and processes, but by using the hardware vision pipeline the images skip loading into DRAM and reduces CPU load by processing video independently of the primary CPU. In my own experience, I’ve been able to run 4k video at ~20 fps by utilizing these hardware features on the TX2. This is why video works so efficiently through CSI cameras — independent hardware specialized for video, much like a GPU is specialized for 3D graphics.

Where to get CSI cameras (for Jetson devices)

In my own research, I’ve found only a handful of resources on finding CSI cameras. The Jetson Wiki has a decent page surveying different camera options and you may be able to find some tips on the Jetson developer forms, but that’s about it.

As for actual shops, there is:

e-con Systems, one of the early entrants into making CSI cameras you can plug directly into a developer kit.

I personally use the Leopard Imaging IMX377CS and find it quite capable. Plus, they have pretty good instructions for installing the drivers, which is always welcome.

Getting Video off a CSI camera

In Nvidia’s “Get Started with the JetPack Camera API” they explain that the best way to interface with the Jetson’s multimedia hardware (including the ports for CSI cameras) is via their libargus C++ library or through gstreamer. Nvidia does not support the V4L2 video protocol for CSI cameras. Since gstreamer is well documented and very common, I’ve focused on it.

GStreamer is configured using pipelines, which explain the series of operations applied to your video stream from input to output. The crux of getting video from your CSI camera boils down to being able to (1) use gstreamer in your program and (2) to use efficient pipelines.

A Note on Drivers: You will most likely need to install the drivers for your camera before any of the GStreamer functionality will even work. Since CSI cameras tend to be a smaller market, you might not find a guide online but should be able to get one from the manufacturer. Leopard Imaging, for example, provided a nice guide (over email) for setting up their drivers, but it only got me to the point of using GStreamer in the terminal. In this post, we’ll venture further and get that data into your code.

Selecting the right pipelines

As I just mentioned, one of the keys to getting quality performance with CSI cameras is using the most efficient gstreamer pipelines. This generally means outputting in the correct format. You will see me repeatedly use a pipeline along the lines of:

The very important part here is video/x-raw, format=(string)BGRx ! videoconvert ! video/x-raw, format=(string)BGR, which ensures that the raw video from the CSI camera is converted to the BGR color space.

In the case of OpenCV and many other programs, images are stored in this BGR format. By using the image pipeline to pre-convert to BGR, we ensure that those hardware modules are used to convert the images rather than the CPU. In my own experimentation, using a pipeline without this conversion results in horrible performance, at about 10fps max for 1080p video on the TX2.

Command line tools

There are a few command line tools I’ll briefly note.

nvgstcapture

nvgstcapture-1.0 is a program included with L4T that makes it easy to capture and save video to file. It’s also a quick way to pull up the view from your camera.

OpenCV

Alright, so let’s start capturing video in our own code rather than just messing with stuff in the terminal.

When setting up your Jetson device, Nvidia Jetpack installs a special, closed source version of OpenCV called OpenCV4Tegra, which is optimized for Jetson and is slightly faster than the open source version. While it is nice that OpenCV4Tegra runs faster than plain OpenCV 2, all versions of OpenCV 2 do not support video capture from gstreamer, so we won’t be able to easily grab video from it.

OpenCV 3 does support capturing video from gstreamerif you compile it from source with the correct options. So we’ll replace OpenCV4Tegra with a self-compiled OpenCV 3. Once this is done, it is quite easy to capture video via a gstreamer pipeline.

Finally, switch to the build directory to install the libraries you just built.

1

2

cd~/opencv/build

sudo make install

Video Capture from GStreamer pipeline in OpenCV

We now have an installation of OpenCV that can capture video from gstreamer, let’s use it! Luckily, I have a nice C++ example script on Github designed to capture and display video from gstreamer with OpenCV. Let’s take a look.

gstreamer_view.cpp

C++

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

/*

Example code for displaying gstreamer video from the CSI port of the Nvidia Jetson in OpenCV.

First, we define an efficient pipeline to use, using Nvidia’s nvcamerasrc interface and ensuring we pre-convert to BGR color space. Then we define a capture object that uses GStreamer. Finally, we capture each frame and display it in an infinite loop. That’s it!

Note to ROS users: If you have ROS installed, you are likely to find this script does not work. This is typically due to the fact that your ~/.bashrc includes the line source /opt/ros/<distro>/setup.bash near the end, as the ROS install guide recommends. This causes ROS to import its own ROS version over your install. If you only need video capture, then just use jetson_csi_cam from the next section with ROS. If you don’t need ROS for some particular application, one workaround this dependency issue is to remove the line from ~/.bashrc and run the setup file only when you need ROS.

Custom OpenCV and ROS together? You should not try to use a self compiled OpenCV install in a ROS program. ROS uses it’s own version of OpenCV called opencv2 and opencv3 that involved in a tangled web of dependencies that is hard to avoid. If you are willing to let things get messy, it is possible to use your own version of OpenCV, but I really would not recommend it.

Robot Operating System (ROS)

Getting your CSI camera up and running in ROS is even easier than OpenCV. All you need to do is install my ROS package, jetson_csi_cam (the README will guide you through all the steps you need). It works in the same way as the OpenCV solution but uses a different library for grabbing video from gstreamer and provides extra niceties expected in ROS. Importantly, it also uses the correct, efficient pipelines for CSI cameras.

The TX2 and TX1 are generally the same and share software compatibility, since they run the same Linux version. Double checking Jetson Hacks’s guide, the only thing I can think is you might need to change the line make -j6 at the bottom of buildOpenCV.sh to make -j4 since the TX1 only has 4 cores.

Nice tutorial! By the way, OpenCV 2 does support Gstreamer in the exact same way as OpenCV 3: it won’t support it by default but if you’ve installed the Gstreamer dev packages before you run cmake & make to build OpenCV then your OpenCV 2 or 3 will include full Gstreamer support. But like you mentioned, unfortunately OpenCV4Tegra doesn’t come pre-built with Gstreamer support.

Thanks for the great tutorial. I have some Leopard Imaging cameras for my TX2. I followed their guide on installing drivers, but I’m still unable to access the cameras with nvgstcapture-1.0 (as they recommend). Instead I get:

I’m trying to troubleshoot what I may have done wrong. If you think of anything else, let me know. I really appreciate the help.

For reference, I flashed the board with l4t R27.1 using jetpack 3.0 then followed their steps (coping Image, zImage, dtb file, 4.4.15-tegra-leopard module and camera_overrides.isp to the appropriate locations on the TX2 and editing extlinux.conf).

Hi Peter,
I followed your Tutorial. I tried to compile “another script for testing FPS at various different resolutions and FPS.” but filed and show me the following error:

In file included from /usr/include/c++/5/chrono:35:0,
from main.cpp:2:
/usr/include/c++/5/bits/c++0x_warning.h:32:2: error: #error This file requires compiler and library support for the ISO C++ 2011 standard. This support must be enabled with the -std=c++11 or -std=gnu++11 compiler options.
#error This file requires compiler and library support \

The TX2 dev board only has one USB 3 port. While you can use a USB hub, it still only allows the same amount of bandwidth, so you can only connect as many cameras as USB 3.0 bandwidth allows, aka 5 Gbps. This is technically enough to allow push uncompressed 1080p video stream at 60fp. However, the ideal is not typically achieved.

Thus as rule of thumb, I’d feel safe using two 720p cameras on one port. More might become problematic.

If you want to use more cameras, you’ll need to get more USB ports via expansion. In the end, however, MIPI cameras will give you better performance than USB.

I get a compile problem on my TX2 attempting to install jetson_csi_cam, it can’t find a couple of packages:

CMake Warning at /opt/ros/kinetic/share/catkin/cmake/catkinConfig.cmake:76 (find_package):
Could not find a package configuration file provided by
“camera_calibration_parsers” with any of the following names:

It is possible you did not install these while setting up ROS. There are a few different Linux packages, and unless you installed ros-kinetic-desktop-full, you do not have the required packages (though I’m not sure why gscam does not complain).

Try running

1

2

sudo apt-get install ros-kinetic-perception

or get eveything with

1

2

sudo apt-get install ros-kinetic-desktop_full

More details about what gets installed for each option can be found here.

That did the trick for compiling it; I had installed ros-kinetic-desktop (not full), and installing ros-kinetic-perception manually worked. I am getting an error message when I launch, though. Do you know where I should look for the cause of this error?

auto-starting new master
process[master]: started with pid [2530]
ROS_MASTER_URI=http://localhost:11311

setting /run_id to ccf8d9aa-d7b3-11e7-9664-00044b6692f9
process[rosout-1]: started with pid [2543]
started core service [/rosout]
process[csi_cam-2]: started with pid [2546]

Thanks for your post, it’s very informative. Do you have any experience/insight on how to use an HDMI camera with the TX2? There’s an HDMI to CSI-2 adaptor by Auvidea (B10x) but no driver for the TX2 at the moment. Are you maybe aware of any workarounds?

Hi, great article! I’ve only recently run into issues using ZR300 on a TX2 (V4L2), enabling USB3, patching the kernel, only to realise that Intel has abandoned ZR300.
So I started looking at camera modules from China. Seems to be an abundance of them. Just wondering, apart from drivers and gstreamer, do I need anything else? The alibaba camera modules of various makers do not seem to include any kind of logic board or circuit board, they only have a CMOS sensor, the lense and a 24pin golden finger connection. Is that enough to plug into the SDK board (or any carrier board for that matter)?

Unfortunately, I don’t have any experience with sourcing my own sensors and connecting them to the TX2. I’ve mostly stuck with stuff from Leopard Imaging and See3Cam because I knew hey should work.

The TX2 does not have a way to plug in a 24pin golden finger connection directly. In fact, it seems there is not really a standardized connector for CSI/MIPI connection, so it would be up to you to connect all the pins correctly. I don’t think there is a need for a logic board as much as a breakout board.

However, even if you were to wire things up correctly, your biggest problem would be getting the right drivers for the camera that will work on the TX2. That’s the main reason I would avoid a DIY approach without seeing someone with more experience try it first.

This is why I am confused. I am looking at similar cameras such as the one for raspberry pi 3, and it is indeed a CSI connection which plugs into the board. However I am guessing that the 24Pin gold connection is not a CSI-2/MIPI cable, and that some kind of adapter is needed?

I browsed e-consystems (I think there’s links about them on the NVIDIA forums) and I found quite a few cameras which are in a much more reasonable range ($25-$45) and claim to connect to TX1/TX2. Then I also found that Auvidea has CSI-2/MIPI adapters (branded by Toshiba) so I am going to make the assumption that indeed the 24Pin gold output from the Chinese cameras is not the correct one.

You are correct about the drivers though, this could easily lead to being a nightmare.

How to capture from GPU?
Hi Morgan Peter,
I would like to know if it is possible to read a video directly from the GPU with Open CV, using Jetson onboard TX2 camera, without capture the image with CPU and then upload it to the GPU.
Could you suggest some tutorials to follow or some codelines?
Thanks you so much for great tutorial

The analog to GPUDirect on integrated Tegra is CUDA ZeroCopy memory. On Tegra this is where the CPU and GPU are mapped to the same memory and can be accessed without memory copies. See here for an example – http://arrayfire.com/zero-copy-on-tegra-k1/