README.md

Kintinuous

Real-time dense visual SLAM system capable of producing high quality globally consistent point and mesh reconstructions over hundreds of metres in real-time with only a low-cost commodity RGB-D sensor.

Related Publications

Please cite the most appropriate of these works (in order of our preference) if you make use of our system in any of your own endeavors:

This is where things get really bad. Due to libraries constantly changing their APIs and includes, creating build processes that actually last for more than a few months between a couple of Ubuntu versions is extremely difficult. Below are separate instructions for Ubuntu 14.04, 15.04 and 16.04.

Why do you have to install OpenCV from source? Because the version in the Ubuntu repos doesn't have the nonfree module, required for SURF descriptors used in the DBoW2. Also, it is strongly recommended you build OpenCV with the following options (in particular, building it with Qt5 might introduce a lot of pain):

If you have trouble building Pangolin, disable ffmpeg support using the following CMake command:

cmake .. -DAVFORMAT_INCLUDE_DIR=""

Once finished you'll have everything you need to build Kintinuous.

15.04

On 15.04 PCL is in the official repos:

sudo apt-get install -y libpcl-dev yasm libvtk5-qt4-dev

The version of PCL in the 15.04 repos does not contain OpenNI2. You should build the Occipital maintained version, available here.

As usual, ffmpeg screws things up for everyone in 15.04. You need to build ffmpeg yourself otherwise OpenCV will fail to build. Why do you have to install OpenCV from source? Because the version in the Ubuntu repos doesn't have the nonfree module, required for SURF descriptors used in the DBoW2. Why can't you use OpenCV3? Because DBoW will fail to build. Why can't you disable ffmpeg in OpenCV's build? Because DBoW will fail if the video module isn't built. Build and install ffmpeg as follows:

Finally, build DLib, DBoW2, DLoopDetector, iSAM and Pangolin from source, in this order. If you have trouble building Pangolin, disable ffmpeg support using the following CMake command:

cmake .. -DAVFORMAT_INCLUDE_DIR=""

Afterwards, you will be able to build Kintinuous.

16.04

The instructions for 15.04 will work fine except you can't use the apt version of PCL because someone screwed up vtk. So ensure you purge/remove it completely if you have it installed from apt, then build it manually;

2. Is there an easier way to build it?

Understandably, building all of the dependencies seems quite complicated. If you run the build.sh script on a fresh clean install of Ubuntu 14.04 or 15.04, enter your password for sudo a few times and wait a few minutes all dependencies will get downloaded and installed and it should build everything correctly. This has not been tested on anything but fresh installs, so I would advise using it with caution if you already have some of the dependencies installed.

-lt : Disallow loop closures within this time period of the last (default 30s).

-s : Size of the fusion volume (default 6m).

-dg : Rate of pose sampling for deformation (default 0.8m).

-il : Inlier threshold for RANSAC (default 0.35).

-it : Residual threshold for pose graph optimisation (default 10).

-sm : Static mode (disable volume shifting).

-f : Flip RGB/BGR.

-od : Perform online deformation (required for loop closure).

-m : Enable mesh generation.

-no : Disable overlap of extracted slices.

-nos : Remove overlap when saving map.

-r : Use RGB tracking only.

-ri : Use combined ICP+RGB tracking.

-d : Enable dynamic cube positioning.

-dc : Disable color weighting by angle.

-fl : Subsample pose graph for faster loop closure.

-fod : Enable fast odometry.

Essentially by default ./Kintinuous will try run off an attached ASUS sensor live. You can provide a .klg log file instead with the -l parameter. You can capture .klg format logs using either Logger1 or Logger2.

4. Datasets

We have provided a sample dataset which you can run easily with Kintinuous for download here. Launch it as follows:

./Kintinuous -s 7 -v ../vocab.yml.gz -l loop.klg -ri -fl -od

5. License and Copyright

The use of the code within this repository and all code within files that make up the software that is Kintinuous is permitted for non-commercial purposes only. The full terms and conditions that apply to the code within this repository are detailed within the LICENSE.txt file and at http://www.cs.nuim.ie/research/vision/data/kintinuous/code.php unless explicitly stated. By accessing this repository you agree to comply with these terms.

This is because you have a slow CPU. The backend runs completely on the CPU and must process every point extracted from the frontend. This means if your map is very large, or if you're moving very fast, the backend may not be able to keep up. Additionally, turning on meshing when loop closure is enabled is very CPU intensive, but a fast modern processor will cope with this in real-time.

I saved a map, how can I view it?

Download Meshlab, which can read .ply files. If you only saved the point cloud you'll need to use PCL's viewer for .pcd files.

Firstly, if you're running live and not processing a log file, ensure you're hitting 30Hz, this is important. Secondly, you cannot move the sensor extremely fast because this violates the assumption behind projective data association. In addition to this, you're probably using a primesense, which means you're suffering from motion blur, unsynchronised cameras and rolling shutter. All of these are aggravated by fast motion and hinder tracking performance.

If you're not getting loop closures and expecting some, or getting false ones, you're at the mercy of DBoW. You can tweak some of the parameters related to it, but over all you're limited by the typical limitations of appearance-based place recognition. Feel free to splice in a different place recognition method. As an aside, ElasticFusion is much better for very loopy comprehensive scanning, which may suit your application better.

If you notice some weird slicing effect during loop closures, either turn down the volume size or increase the rate at which poses are sampled in the deformation by decreasing the -dg parameter.

Is there a ROS bridge/node?

No. In fact, if you have ROS installed you're likely to run into some truly horrible build issues.

This doesn't seem to work like it did in the videos/papers

A substantial amount of refactoring was carried out in order to open source this system, including rewriting a lot of functionality to avoid certain licenses. Although great care was taken during this process, it is possible that performance regressions were introduced and have not yet been discovered.