Trying to simplify algorithm for fitting silhouette of object to the observed object. Have closed form algorithm that might get rid of gradient descent in rotation space

3D pose from silhouette is what we'll get out. CAD model, project to plane, find rotation that explains

Victor

Transparent objects: Our algorithm have used SolvePnP to estimate a 3D pose from a 2D transformation but this approach is awkward. We have derived and implemented an analytical non-iterative estimation of a 3D pose under the assumption of weak perspective projection (valid for objects with small size comparable to the distance to the camera). The 3D pose exists and it is unique. Right now it is used as the initial guess in SolvePnP (however, this call to SolvePnP doesn't affect final poses on our dataset). The difference in the final object poses estimated by the old and new algorithms has the order of 0.1 mm.
- Removed the dependency on VTK and pcl visualization in the default build and dependency on ROS completely.
- Updated the old code of the algorithm evaluation to use the new interfaces from the transparent_objects Ecto module.
- Updated default config files to use new Ecto API.
- Filtered cases when 2d-2d ICP converges to a degenerate transformation.
- Investigated two potential bugs in the final pose evaluation when results were counter-intuitive. [Ilya]
KinFu: We have studied the paper by Steinbrucker et al (TUM) on visual odometry from dense RGB images. The implementation of this approach as a standalone algorithm is in progress. Meanwhile there is an ongoing discussion on KinFu refactoring to enable modular modifications. [Maria]

Vadim

1. (Feature Detectors & Descriptors)
- Several bugs in the recently refactored SIFT code have been fixed
- SIFT-like keypoint detector can now be used in ORB.
2. (Tests)
- The new OpenCV team member, Alexander Reshetnikov, has implemented 2 new tests:
for cv::eigen() (Jacobi algorithm for finding eigenvalues and eigenvectors of a symmetrical matrix)
for cv::findHomography().
2 more tests are in progress.
3. (OpenCV repository)
- OpenCV repository has been migrated from SVN to Mercurial at the new OpenCV hosting. After some more testing we will start using it [Alexander Shishkov]
Issues:
Unfortunately, we have not got any reply from code.ros.org admins regarding the dump of OpenCV bug tracker database. Without it we can not move completely to the new bug tracker at code.opencv.org.

Anatoly (GPU)

Accomplishments & statuses:
1) PCL.
· Minor updates of kinfu (2 minor bugs, warning for users about Fermi GPU required, etc.)
· Mesh generation from TSDF volume:
o Learned different isosurface building methods (grid projection, variants of marching cubes). Decided to start with ‘Marching Cubes 15’ (simple, all tables are available, ambiguity is not critical) and implement it right on GPU. Finished Marching Cubes GPU code. Now debugging it.
2) OpenCV GPU
· Added docs for FAST_GPU and ORB_GPU.
· Updated GPU Demo Pack according to Joe’s comments (added needle map to BROX optical_flow demo, added ORB demo).
· Fixed bug with sub-matrix in some GPU functions (remap, resize)
· Started working on Optical Flow on GPU (it will be required by video stabilization)
3) Autocalibration
· Tweaked the autocalibration method parameters. As result, ‘Point-to-epipolar-line RMS error’ in fundamental matrix estimation was decreased from ~5 pixels to ~1 pixel.
· Found and fixed a few bugs in the bundle adjustment refinement step which led to failures when some of image pairs were matched poorly.
· Implemented auxiliary functionality drawing epilines to check whether they are actually close to keypoints. Outcome: they're actually close.
· Created more new scenes using the LG camera, done more experiments. Here are the best results achieved:
Average intrinsics initial relative guess error (28 runs):
fx
fy
Ppx
ppy
0,175910054
0,119935433
0,15020694
0,134989868
Average intrinsics final error:
fx
fy
ppx
ppy
0,039920564
0,033327766
0,089192789
0,146436884
Average final rotation vector (pattern-based calibration result = [0.0007,0.0424,0.0008]):
rx
ry
rz
0,000509849
0,003470322
-0,002961118
Average final translation vector (pattern-based calibration result = [1.000000,-0.035135,-0.118919]):
tx
ty
tz
1
-0,01003881
-0,168400068
These results are quite good, but obtained by intensive parameters tweaking for the particular dataset.
Issues:
· Result of the autocalibration method depends very much on dataset, parameters, and initial guess.
· Even for the best dataset taken be the LG camera, fundamental matrix estimated isn't close to one estimated using the pattern-based approach. In the same time, for VIDERE camera both fundamental matrices are close.
Plans:
· Continue with optical flow on GPU.
· Rig autocalibration plans to be discussed with Victor (no questions to Kurt now).
· Finish Polygonal mesh extraction from Kinect Fusion. Consider implementing other 3D features or another task (registration?)

Talk with Xilinx coming up: FPGA algorithms for better sensors will be the theme

We have a 2D bar code reader in OpenCV findDataMatrix() and drawDataMatrixCodes()

similar to libdmtx

Release target: Feb.

Bug clean up

Need to do a documentation push

Expand user guide

Need index of functionality to functions. For example Camera pose => what functions will do that for you

Website

Vadim's visit

edge detection/alighnment

look at mvtech functioality

Book version 2 scope

NO MEETING NEXT WEEK

Vadim and Victor are in flight to Khronos

I'll be at Xilinx meeting

Vadim

. (API cleanup)
- A few obsolete algorithms from features2d and objdetect moved to the legacy module. Moving most of the C interface from core, imgproc, video and other modules to legacy is in progress. In goal is to have purely C++ interface in all the modules except for the legacy.
2. (API design for the CV standard)
- Developed proposal on the API for the standard. Sent by a separate e-mail. The API is inspired by the OpenCV C API, but has a few important differences, such as an abstract high-level algorithm interface.
3. (New Tests)
2 new tests have been developed - for cv::countNonZero() and cv::boundingRect() functions [Alexander Reshetnikov]
4. (OpenCV repository)
code.opencv.org is now basically ready-to-use. We now have (empty) bug tracker, regular backups, e-mail notifications and all other important features working well. For code repository we chose SVN for now; will switch to mercurial (or may be git) after the release. Next week we will start using the new hosting [Alexander Shishkov, thanks to Kirill Kornyakov for help with the setup]

Might be a change in operations since Gary has cofounded: Industrial Perception

Release at endof Feb.

Concentration on bug fix and docs

Minutes

Vadim

1. (Bug fixing)
- 16 tickets in our bug tracker have been closed: ##1345, 1415, 1422, 1442, 1486, 1512, 1514, 1518, 1519, 1521, 1531
+2 more patches from users have been applied.
2. (API design for the CV standard)
- Memory Management section has been added to the proposal document. A few minor changes made the in headers.

Victor

RGB-D visual odometry [Maria]
Improved the convergence of the initial algorithm implementation:
- added occlusions processing
- fixed the gradient computation on the border of the valid area of the warped image.
Optimized the initial algorithm implementation (cpu, speed-up is 3.5x). It takes about 3.5 sec now (for 4 levels of the image pyramid, 30 iterations per level).
Made experiments to evaluate visually the quality of the algorithm. Merged 25 sequential frames of our room by running the algorithm on adjacent frames (see the first and the last frame of the sequence in attachments). It produced the good merged cloud of all 25 frames (see the attached video merged_cloud.mp4).
Started to study the PCL implementation of KinectFusion. It's not clear yet how we should merge ICP-based and RGB-D visual odometry.
Transparent objects [Ilya]
Finished the paper about our current algorithm and submitted it to a conference.
Devised a way how can we estimate poses in transparent clutter. A test silhouette contains boundaries of several transparent objects in this case. So we should align parts of a train silhouette with parts of a test silhouette to find potential matches and get initial poses. This problem is called part-to-part curve matching that is find the longest common subcurve under similarity transformations. Studied papers on this topic. The following are the most relevant and useful for us:
H.J. Wolfson, On curve matching, PAMI, 1990.
M. Cui, J. Femiani, J. and Hu, P. Wonka, A. Razdan, Curve matching for open 2D curves, Pattern Recognition Letters, 2009.
However, Geometric Hashing is more attractive for this problem because it has lower complexity at the test stage and it is more well-known and reliable technique.
Implemented Geometric Hashing for curve matching and initial pose estimation in transparent clutter. See an example of recognition and pose estimation in attachments.
OpenCV bug fixing [Alexander Reshetnikov]
Changed the code structure of last tests.
Fixed a bug in the function cv::JacobiImpl_(), which caused incorrect work of the function cv::eigen() in the case of 1*1-matrices. Now all tests with such matrices are finished successfully (time of execution is 20 ms or less). Changed the procedure of maximum element search in the cv::JacobiImpl_() to use row and column indices of corresponding elements (thanks to Vadim).
Prepared test videos and started to write tests related to positioning in videos (tickets #203, #1335, #1419, #1497). Fixed the ticket #1335.
OpenCV infrastructure [Alexander Shishkov]
Developers site (code.opencv.org)
Added incremental backup of redmine and SVN. Full backup occurs every Sunday, and changes are backed up every two hours.
Fixed problems with IE for some themes
Tried plugins for Buildbot and repository search
Finished scripts for copying our current SVN to code.opencv.org
Buildbot (buildbot.itseez.com)
Added the Report page with full information about the current state of Buildbot, see http://argus-cv.dnsalias.org:8010/report
OSQA (alekcac.webfactional.com)
Done the initial settings for the web-server
Started to work on the visual part of this site
Other
Gave a talk with Kirill about working with Redmine and Buildbot for OpenCV developers
Updated the document about OpenCV web-sites

Anatoly

Accomplishments & statuses:
1) Autocalibration
· Created the autocalibration summary report (attached).
2) OpenCV
· Updated the stitching module docs.
· Added function ‘printCudaDeviceInfo’ and calls of it to all samples.
· Added a regression test for the multi-band blending method.
· Fixed compilation with CUDA 4.1 ( so for now we support both 4.0 and 4.1). But have not scanned NPP docs for new functions to use in OpenCV or to create wrappers for them in OpenCV API style.
· Fixed OpenCV GPU tickets:
o #1561(compilation)
o #1567 (gpu::device::transform for non-aligned sources)
o #1569 (inconsistent orientation for SURF_GPU)
3) Optical flow on GPU
· Implemented minDistance threshold for GoodFeaturesToTrackGPU point detector using CPU (as it doesn't take much time even with data transfer).
· Started learning Farneback dense optical flow method. Going to port it on GPU together with pyrLK optical flow.
4) PCL
· Added ‘gpu::copyFields’ functions (experimental), so that user can easy implement different device point cloud conversion. For example, can copy normal_[x|y|z] fields from ‘DeviceArray<Normal>’ to ‘DeviceArray<PointNormal>’.
5) PCL Registration. Decided to learn it as this cloud help with future work and with KinFu registration.
· Familiarizing with ‘pcl::registration’ and ‘pcl::sample_consensus’ in order to decide what can be ported to GPU.
· Trying to port ‘pcl::icp’ to GPU:
a. Implemented pcl::gpu::octree::knnSearch for k == 1. That is because ICP requires only the closest point. Speed-up vs. pcl::octree is 3.5x. Profiling it.
b. It seems that only thing that can be parallelized here is RANSAC iterations loop. In this case I need to implement SVD on GPU within ‘thread block’ of another ‘group of threads’, so that many SVDs cloud be performed in parallel. But don’t know if it can be done efficiently or not.
Plans:
· Continue with pyramidal Lucas and Kanade and Farneback optical flow
· Continue with PCL registration

Transparent objects [Ilya]
Finished the implementation of the paper "A Geodesic Active Contour Framework for Finding Glass" by K. McHenry and J. Ponce, CVPR 2006.
Added the texture distortion feature by using textons from the paper ”Contour and Texture Analysis for Image Segmentation” by J. Malik, S. Belongie, T. Leung, J. Shi, ICCV 2001.
Improved SVM accuracy by following A Practical Guide to Support Vector Classification from the LIBSVM.
Post-processed results of the Felzenszwalb's segmentation algorithm to remove thin regions on boundaries of objects.
The attached animated GIF shows a Geodesic Active Contour evolution on a training image computed with the implemented algorithm.
RGB-D visual odometry [Maria]
Made a lot of experiments with ICP, RGBD and different versions of merged ICP+RGBD odometry on freiburg1_desk http://cvpr.in.tum.de/data/datasets/rgbd-dataset/download#freiburg1_desk mainly. KinFu ICP is more stable and doesn't break trajectory critically on the data. RGBD is sensitive to the following problems with rgb and depth data:
- Due to insufficient synchronization between rgb and depth data, motion blur, ... 3d points don't have their real colors.
- Matched frames can be far from each other.
In such cases RGBD gives a pose far from ground truth. So RGBD breaks the trajectory several times, but it has less than ICP median drifts of translation and rotation per frame for well synchronized data (see previous report). Tried several approaches to merge ICP and RGBD. Found that more stable version of merging is the following. RGBD pose is used only if both points error and photo consistency error are less than the same ones for ICP pose. But sometimes this merging strategy gives a trajectory that has larger absolute error than ICP.
OpenCV tests and bug fixing [Alexander Reshetnikov]
Applied some solutions for fixing the bugs associated with frame count computing and video clip positioning (feature #850 and report from the ticket #120).
Results of frame count and positioning tests (default testing infrastructure is Ubuntu x64, actual frame count is 125):

Closed tickets # 850, 1154.
OpenCV infrastructure [Alexander Shishkov]
- conversation with hetzner.de support about upload speed on our server.
- modified Apache settings for OpenCV SVN server; now full checkout of OpenCV trunk (about 1GB) from office takes about 10 minutes
- modified UUID on local SVN mirror, so it is possible to do relocation from local mirror to the public repository, migrated local mirror to Apache, forbade commits to the local mirror
- updated wiki on Buildbot project (#1073), updated Android NDK on Ubuntu slave (#1245), added link to the trunk documentation to the Buildbot main page(#1239), added highgui tests tot the Android builder(#1244)
Other:
- fixed ## 1599, 1602, 1604, 1363, 1492, 1339, 1614
- started the work on FFMPEG problems ## 1615, 924

Anatoly

Accomplishments & statuses:
1) Optical Flow
· Revealed and fixed the bug in the high level Farnebeck GPU function which led to difference between CPU and GPU results. Found out that cv::GuassianBlur() on CPU uses some hard-coded sigma values in case of small kernel sizes. That also caused difference between the CPU and GPU. Ensured the GPU version behaves the same way.
· Integrated Farneback GPU to OpenCV. It works 13..22x faster than the CPU version (Intel Core i3 vs. NVIDIA GTX 470) on 640x480 size images.
· Miscellaneous:
o Added accuracy tests, performance tests, docs.
o Created sample, updated performance sample (samples are convinced as API example for developers)
o Updated demo pack with the optical flows. (Demo pack was convinced as use-case examples for users).
2) OpenCV GPU
· Revealed incorrect behavior of linear filters for kernel sizes >=11x11. Fixed it so that it supports kernel's size up to 32, also new implementation is a bit faster.
· Added support of different border extrapolation modes into my implementation of Gaussian blur -- to be similar to the CPU version.
· Created wrappers for new NPP arithmetic only functions.
3) Other
· Prepared ACM article draft version. Guess after adding Khronos process it will reach recommended size.
· Hiring activities. I never put this in reports, but I spend time for participating interview for Tegra and so on.
· Started learning video stabilization theory.
4) PCL
· Fixed the BUG in marching cubes in Kinect Fusion
· Minor compilation fixes and tests for different platforms (like C++0x flag in CMake breaks NVCC compilation and so on)
· Started learning code and papers for future people detection GTC demo.
Plans:
· Adding NPP 4.1 functions and GPU video decoding.
· Continue learning video stabilization theory.
· ACM Queue article
· Continue with pcl_people on GPU.

We have contributors already, get one or more of them to be GSoC students

Mobile vision app development:

Vision apps on phone and tablet. We want a very clear "hello world" type of example to make it easier for the rest to start developing mobile vision applications. People can apply for:

iOS (iPhone) examples

Android example

1D and 2D Barcode readers:

We have a basic datamatrix reader. Expand to other kinds of barcodes (1D, other 2D).

Natural barcodes:

OpenCV can read calibration patterns. Calibrate a camera, rectify it's output. Put a picture next to a calibration pattern in a scene. Find 2D interest points with features2D. Measure them in the image using the calibration pattern. Use the pattern of interest points and their measures to turn the picture into a calibration pattern/barcode so that the computer recognizes the picture and knows where it is in relation to the iamge.

Stretch goal, embeded and retrieve watermarks in the image.

Python example:

We're finding the python interface extremely useful and productive. It needs a lot more examples

Accomplishments & statuses:
1) OpenCV GPU
· Finished VideoCapture_GPU implementation under Linux ( but have not integrated in OpenCV)
· Improved arithm functions dispatching between NPP and our code (~2x faster now).
· Created GPU path for gpu::PyrLKOpticalFlow (cpu version has strange behavior in error calculation, created ticket #1651).
· Revealed some inconsistencies in gpu::convolve() for small kernel sizes specializations (ticket #1639). Fixed error in gpu::convolve() docs.
2) Face detection on GPU. As soon as Anton Obukhov doesn’t longer support FD code, we are going to fix its all crashes for CUDA4.1 by our own.
· Fixed HaarCascadeLoader test failure.
Active issues:
· Visualization test fails.
· Haar classify kernel crash with CUDA 4.1
3) Video Stabilization.
· Implemented trimming which ensures that there are no black border artifacts due to stabilization in output video by trying to relax estimated stabilization transformations. It works better than naive ad-hoc which doesn't stabilize at all when there are artifacts.
· Implemented Fast Marching Method which is required for in-painting steps. Wrote test comparing it with distanceTransform() (as distance map can be computed via FMM naturally).
· Implemented color in-painting via averaging of neighbor pixels colors. Implemented motion in-painting which propagates local motion into areas where optical flow isn't known. Experimented with different optical flow algorithms. As it was mentioned in the original paper motion in-painting isn't perfect technique and there are some artifacts.
4) PCL / Human Tracking
· Euclidian point cloud clusterization. Learned the algorithm and its code, understood bottlenecks. Read several papers about connected components on GPU. No idea how to accelerate the cauterization with GPU. It’s universal and slow (800ms). And even taking into consideration that input cloud is organized doesn’t reduce amount of work to be done. Instead, two lightweight algorithms were implemented on CPU which work 116ms and 14ms (floodfill-like). They produce some approximation of output from original. Experiments showed that 14-ms version is enough and doesn’t break human tracking pipeline. Accelerating this 14ms-version with GPU is also not easy and not reasonable as currently it is not a bottleneck.
· Seeded hue segmentation. The same complexities as with previous (work time 200-300ms, not idea for GPU). The approximate method used for clusterization doesn’t work here (it cuts persons hands). Accelerated CPU code by refactoring and sampling points a points queue inside (some another kind of approximation), so that it works 70-90ms for middle distance objects. Further optimization is possible with OpenMP, so expect work time to be ~30-40ms. Guess, better results are achievable by developing another smarter (than floodwill) 2D segmentation with depth constraints.
· smoothLabelImage. It’s a kind of filter. Created GPU implementation for it, so that run time reduced from 83ms to 1ms.
Overall frame time for human tracking was reduced from 2.5sec to 400ms.
Plans:
· Integrate VideoCapture_GPU to OpenCV. Fix CascadeClassifier kernel crash. Fix gpu::convolve implementation (ticket #1639).
· Video stabilization: Work on exceptional cases handling like poor in the sense of number of features datasets and datasets with large moving objects.
· Implement volume shifting for Kinfu (Kurt’s request) or something other in PCL.
· ACM Queue article

Ilya

Transparent objects [Ilya]
Studied psychological papers on perception of transparent objects. They are useful for us because they describe heuristics used by people to perceive transparent objects and we can derive features from them for glass segmentation. The most useful papers are: "Perceiving light versus material" by F.A.A. Kingdom (Vision Research, 2008) and "Low-level image cues in the perception of translucent materials" by R.W. Fleming and H.H. Bulthoff (ACM Transactions on Applied Perception, 2005). Added 3 features to our algorithm based on read papers:
1. Added Michelson contrast.
2. Added root mean square contrast.
These two features are discriminative because contrast plays the important role in perceiving glass. Look at two examples (episcotister.png and contrast.png):

(images from the paper "Toward a perceptual theory of transparency" by M. Singh and B.L. Anderson (Psychological Review, 2002))
Regions p and q on the first image look as transparent and they have lower contrast together than a and b. However, if they have bigger contrast (see the second image) then this perception is gone. In fact, the surrounding regions look as a transparent layer with the hole in the middle because they have lower contrast now.
3. Added distance between median colors in regions as a feature because a chromatic change is useful for perception too. Our regions are rather uniform after over-segmentation with Turbopixels so we can use just a median color, not full histograms.
Adapted glass boundaries computation for Kinect data. McHenry et al. classify only adjacent regions as glass-background. But our regions are much smaller so we can expect glass-background pairs to be separated by other regions. Added classification of region pairs in a neighborhood and aggregated results with a voting procedure.
See example of our algorithm working on a test Kinect image (the animated GIF):

(Note the algorithm uses RGB data only that is a depth image is not used for segmentation).
RGB-D odometry [Maria]
Refactored the RGB-D odometry implementation to put it in the OpenCV contrib module.
Found the bug in estimation of time of our RGB-D odometry working. The time was normalized on iteration number after some experiment, but the time from the original paper on RGB-D odometry is for all iterations. So optimized the implementation in more than 10 times. Main optimizations:
1. Switched to the fast Sobel version (16S).
2. Optimized image warping.
3. Chose a subset of textured pixels to estimate a transformation.
Now the implementation works the same time as in the paper (~80ms).
Started experiments on the Kurt's dataset.
OpenCV infrastructure and issues [Alexander Shishkov]
OpenCV infrastructure:
Investigated the new Buildbot API in 0.8.5.
Began to work on FFMPEG builders for the OpenCV Buildbot.
Updated settings and the theme of the test OSQA installation.
Investigated AskBot, began to install it on our hosting (we plan to compare it with OSQA internally).
Done default settings for the GSOC project on code.opencv.org.
Fixed some problems with the code.opencv.org theme.
Communicated with the designer of the opencv.org website.
OpenCV issues:
Done some infrastructure and documentation issues (##1646, 1645, 1644, 1637, 1618, 1546, 1507, 1477, 1466, 1456, 1405, 1398, 1289, 1260, 521).
Updated png and tiff libraries used for OpenCV compilation on Windows (#1609).
OpenCV tests and bug fixing [Alexander Reshetnikov]
Configured ffmpeg 0.10 on the local machine to write and test the new OpenCV wrapper. It is based on the ffmpeg class written by Andrey Morozov.
Checked the work of the Andrey's ffmpeg class on frame count and positioning highgui tests:
1. The frame count test.
Results (actual frame count is 125):

Results strongly depend on the file format because for different containers we can find the frame count property in different ffmpeg structures (now there is no universal rule to get it).
2. The positioning test.
The class gives fully correct results on positioning tests. However, the class has some problems with reading some formats (also it depends on a local ffmpeg configuration).
Began to write the new ffmpeg wrapper. Achieved compatibility of this wrapper with the old implementation of the ffmpeg wrapper.

Quaternians, Rodriges, R,t and just some common functions we face (segmentations, histograms, morphology)

Connected components

Bump hunting (Gary writing) Finding density peaks in data

ECCV application

CVPR. Gary's paper is in. Workshop dev => course ware.

Transparent

Minutes

GSoC ... find out on this Friday

Python Abid will start working on some examples. He's not going in GSoC, we'll find another for that

Latest OpenCV in ROS, Vincent to call Ethan on an issue

Kinfu in OpenCV in GPU version.

New dense optical flow from Maria is working well, document, samples, great job!

ASUS divers submitted as patches to code.opencv.org

Fast normal finder

Release April 2

Khronos ... lots of nitty gritty detials. This is going to take awhile ...

Non-profit

This is on, many efforts and considerations

We will make this happen within the month.

Next week Victor in the US

New user page for OpenCV ... should be up within a few weeks.

User site will be: OpenCV.org vs

developer code.opencv.org

Look into moin moin for the wiki

3D needs. Filters, normals, voxel filters

Eigen

Good community. Large library

2 branches for optical flow with and without

OpenCV has small matrices 4x4, 3x3 MatEx structure

Add quaternions to MatEx

Nice geometry features in Eigen

Yet another dependency

It had issues with misaligned data, goes to just C code

Might be handled in new version

Need better support for simple translation

Add a few routines to MatEx, Gary will put up a page on code.opencv.org to collect what is needed

Need general Levenberg-Marquard

Simple optimization is needed, some of this is embedded inside machine learning

Expose, normalize and justify the RANSAC that is there

lmdir lmdiff

sparse bundle adjustment

libmv will give a bunch of these functionalities

Connected components -- findContours repeated calls of floodfills

Kurt has CC for speckle components

VO, Maria is on this.

Not getting quite the same accuracy as the paper yet (missing details in paper)

Efficiency looks good, 30-50ms/frame on CPU ... could be GPU'd later

Transparent objects

Ilya has done good work. Gary will collect data from new sets and perhaps from new contacts from Proctor & Gamble

We will combine several techniques to increase reliability

Ilya

RGB-D odometry [Maria]
Added the RGB-D implementation and the sample to the OpenCV contrib module.
Implemented pure rotation and pure translation cases. Added ability to initialize transformation between frames.
Tested the implementation on the Kurt's dataset. There is only camera translation along a textured plane there. We don't have ground truth, but visually the method works well on the data. To compare RGB-D odometry results with given kinfu results we plot trajectories (see trajectories.gif). It seems RGB-D odometry is more accurate on the data, especially if pure translation is estimated. RGB-D odometry for rigid body motion accumulates a bigger error during ~350 frames than for pure translation, but it looks better than kinfu icp results.
Transparent objects [Ilya]
Wrapped SimpleBlobDetector for Python and added a full Python wrapper of findCirclesGrid.
Adapted our old dataset of 5 transparent objects for evaluation of glass segmentation. Dataset images are segmented automatically into glass, background as behind the glass, and other background by using fiducial markers. So ground truth mask is computed. The segmentation is computed using the OpenCV Python interface.
Implemented infrastructure (in Python) to evaluate glass segmentation. It runs the algorithm on all images and computes precision and recall of the glass segmentation. Experiments are on the way.
OpenCV infrastructure and issues [Alexander Shishkov]
OpenCV infrastructure:
- fixed a problem with concurrent work of Android builders on Buildbot (#1274)
- investigated a problem with MinGWLogObserver, tried to fix it, but I still don't understand the reason of such behaviour (#1303)
- installed newest MinGW/MSYS tools to the new Buildbot server, implemented alpha version of the builder for the FFMPEG wrapper (#1266)
- conversation with the Askbot main developer and different customers of OSQA and Askbot, investigation of Askbot administration. I tend to choose Askbot as framework for OpenCV Q&A forum now.
OpenCV issues:
- added support of XtionPRO/XtionPRO Live, updated a sample and documentation (#1657)
- replaces all references to code.ros.org, opencv.willowgarage.com/documentation, opencv.jp to the actual resources (code.opencv.org, opencv.itseez.com, etc..); fixed some bugs in documentation; updated cheatsheet.pdf (#1672)
- fixed some compilation and documentation errors (#1554)
OpenCV tests and bug fixing [Alexander Reshetnikov]
Added the first version of the new ffmpeg wrapper into OpenCV (for versions from 0.7 to 0.10). Now the used wrapper is chosen depending on ffmpeg version.
Current implementation doesn't work correctly with C interface (highgui tests with video i/o terminated unsuccessfully with hardware/os exception). So now the main activity is fixing the bugs in C interface of the new wrapper and checking it on existing highgui tests.
Also decided that the highgui module needs an another positioning test where we create specific video (every frame of this is binary image that has byte code of frame number).
This test will be independent of compression algorithms used in different containers and codecs.
Other
March 8 was a national holiday in Russia.

Vadim

Anatoly

Action Items

Gary

Submit ECCV workshop

Put up a wiki page on developer site for all the rotation MatEx operators that we need

Experimented with the pure voting algorithm and Geodesic Active Contours. Curves showing advantage of Geodesic Active Contours are attached.
Investigated and fixed two bugs in computation of invalid regions and aggregation of SVM predictions.
Removed hard-coded parameters, used the Python tempfile module to generate temporary files, removed debug info writing.
Combining of the RGB-based and depth-based glass segmentation algorithms is in progress.
OpenCV development and tests [Alexander Reshetnikov]
New wrapper supporting ffmpeg versions from 0.7.x was refactored (changed implementation for opening, closing, setProperty and getProperty methods, grabFrame and retrieveFrame functions).
Also it was checked on frame count test for versions 0.7.11, 0.8.10, 0.9.1 and 0.10. It has given following results (actual frame count in every video is 125):

Started the work with new positioning test (see checking new wrapper on positioning and writing tests.
Compilation errors under Ubuntu with ffmpeg 0.7.x were occurred with new wrapper. Some problems have been already fixed. Open issues with it will be closed in the nearest time.
OpenCV infrastructure [Alexander Shishkov]
Buildbot:
- updated one of the test servers to the Ubuntu 11.10 (#1068)
- added time-stamp to the Report page (#1329)
- fixed problems with highgui tests on GUI (#1300)
- added new builder for updating opencv_ffmpeg(32/64) dlls after changes in FFMPEG wrapper (#1266)
- added new builder for test highgui module with different versions of FFMPEG library 0.10, 0.5.1, 0.7.3, 0.7.8 (#1272)
Other:
- preparation to the OpenCV hackaton (instructions, issues selection, discussions)
KinFu and RGB-D odometry [Maria]
Implemented initial version of approximate synchronization of Kinect image and depth. Synchronized RGBD data is necessary e.g. for RGBD odometry. The synchronization is supported in both single threaded and multi-threaded versions. Single-threaded version may be used when there isn't TBB and frames are grabbed from video (i.e. frames aren't skipped). But reading/writing .oni video files aren't supported now. Multi-threaded version based on TBB can be used to synchronize data grabbed from the camera online.
Experiments with the initial implementation is in progress. We are trying to minimize the number of skipped online-synchronized "image-depth" pairs by balancing work between threads and playing with synchronizer parameters.

Vadim

1. A lot of changes made locally on my machine during winter have been finally committed to SVN repository. It includes quite substantial refactoring of features2d: introduction of Feature2D class that combines feature detector and descriptor extractor, using the super-abstract class Algorithm for Feature2D (Algorithm will also be used as a base for StereoMatcher, BackgroundSubtractor, OpticalFlow etc.), and many other simplifcations.
1.1. SIFT & SURF are moved to a separate nonfree module, so that it's possible to ship OpenCV binaries free of any patented algorithms (i.e. in future it's planned to move more stuff to nonfree).
1.2. Brute-force matcher is now threaded using TBB.
1.3. Some obsolete stuff, like Calonder, Ferns etc., as well as C interface of the module, has been moved from feature2d to legacy module.
2. The new module "photo" has been added. Right now it just contains inpainting algorithm; it's planned to add more computational photography stuff there in future (maybe even before 2.4 release).
3. Enhanced variant LogPolar transform (using so-called Blind Spot Model), contributed by Fabio Solari and Manuela Chessa, has been put to contrib module.

Anatoly

Action Items

Gary

Talk to Kurt et al for what we want pragmatically for rgbd

Vincent

Send emails to find mentors

Alexander

Ilya

Vadim

Create Eigen tutorial/doc, sample for using Eigen

Victor

Kurt

Get back to Maria on VO

Ethan

From Last Time

Gary

Submit ECCV workshop

Put up a wiki page on developer site for all the rotation MatEx operators that we need

Robot can grasp one by one (but robot has to be aware of where other objects are)

Does this make sense for June demo at Willow? Most of the ROS stuff is done for this

Probably go for it

Vincent coming end of May

Last week was documentation hackathon

OpenCV docs have fewer broken links etc

Vadim

OpenCV 2.4 beta is out [OpenCV team]
OpenCV 2.4 beta is finally out. Here is the announce:
http://tech.groups.yahoo.com/group/OpenCV/message/87261 and
here is the changelog: http://code.opencv.org/projects/opencv/wiki/ChangeLog
OpenCV Documentation hackathon is over [Ilya and others]
Last week we had documentation hackathon. Since beta 2.4 has been delayed by one week, and so we had some overlap in schedule, it was not as effective as the bug fixing hackaton. Still, we closed 24 documentation-related tickets in our tracker, 19 of which was closed by Ilya! The documentation will further be improved until and after the final 2.4.
EM improvements [Maria, Ilya, Vadim]
Maria, with help from from Ilya and Vadim, further improved EM implementation. The new implementation has automatically generated Python interface and much better interface. Several EM-related tickets have been closed as a result of this refactoring: #964, #1340, #1351, #1361 (according to our bug tracker, EM was quite commonly used and yet quite problematic part of OpenCV; hopefully, most issues are now resolved).
Improving ffmpeg wrapper [Alexander Reshetnikov]
Implemented 2 new highgui performance tests for video i/o.
Closed the tickets: #1771, #1790, #203, #1736.
Other bug fixes, improvements [Maria, Vadim]
Other mll and features2d bugs/improvements:
Improved MLData to read files with different space count between columns.
Fixed the ERTrees name in saving/loading;
Fixed #1798, #1795, #1805.
Fixed crash in linemod detector, reported by S. Hinterstoisser.
Extended morphological operations to handle more than 4-channel images, 16S and 64F formats, added test that checks supported formats in filtering operations.
Infrastructure [Alexander Shishkov]
Buildbot:
Updated Windows SuperPack builder (added the skipped include files, fixed incorrect include files for the stitching,...).
Transferred Android builders to MacOS.
Added the OpenCV testing with FFMPEG 0.10.2.
Fixed the problem with svnsync on Buildbot.
OpenCV 2.4 beta packages for Windows/Unix are prepared and uploaded to the SourceForge.
Eliminated the consequences of the attack on the code.opencv.org site (checked for rootkits, removed enemy processes, added the using SSH keys, added IP list for SSH).
looks like we are getting popular :)
Added DNS docs.opencv.org (link to opencv.itseez.com).
Reviewing GSoC proposals [Vadim, Anatoly]
Finished reviewing all the submitted proposals. This year we've got a _lot_ of proposals, so while the majority of proposals can be easily rejected, there are enough good ones to choose from.

Action Items

Gary

(./) Start a opencv.org "wish list": What we can do with what funding

(./) Get P&G objects for transparent test

Vincent

See about converting Maria's tracking to a generic model capture of camera pose. Talk to here

Update on the opencv.org site. We have started working with the vendor last week, we expect the first version of the site by the end of the next week and the final version one week later. Given potential delays and time for content creation, it is reasonable to expect site rollout by the end of May.

Transparent objects [Ilya]
1. Tested our pose estimation algorithm in transparent clutter to get a proof of concept whether a robot can clean out a table with transparent objects.
So several objects were placed on a table, our algorithm returned pose of the recognized object with the best confidence, this object was removed from the table and this procedure was repeated. Results are shown in the animated GIF:

2. Optimized the pose estimation algorithm. Now silhouettes of a model are computed using dot products with normals instead of morphology. Re-scanned objects to get models with more accurate normals (thanks to Maria for granting access to her computer with kinfu). Experiments show that the optimized algorithm is almost 2x faster than the old algorithm but accuracy is worse. It means we can combine two algorithms together by finding the single pose with the optimized algorithm and then refinining it with the old algorithm to get fast pose estimation with good accuracy. Results are shown in the table:
-----------------------------------------------------------------------------------------------------
| | Mean translation error (cm) | Mean pose estimation time (s) | Speed-up |
| Object |-------------------------------|------------------------------------|---------------------|
| | old | optimized | combined | old | optimized | combined | optimized |combined|
|---------------------------------------------------------------------------------------------------|
|bank | 0.41 | 0.68 | 0.40 | 31.1 | 20.4 | 20.4 | 1.5 | 1.5 |
|bottle | 1.41 | 1.33 | 1.37 | 22.6 | 10.7 | 12.1 | 2.1 | 1.9 |
|bucket | 0.34 | 0.71 | 0.45 | 55.7 | 26.4 | 26.3 | 2.1 | 2.1 |
|glass | 0.48 | 0.57 | 0.50 | 19.7 | 10.4 | 10.7 | 1.9 | 1.8 |
|---------------------------------------------------------------------------------------------------|
|mean | 0.66 | 0.82 | 0.68 | 32.3 | 17.0 | 17.4 | 1.9 | 1.8 |
-----------------------------------------------------------------------------------------------------
So the final algorithm has the same accuracy as the old one but it is 1.8x faster.
Other:
Our paper "Recognition and Pose Estimation of Rigid Transparent Objects with a Kinect Sensor" by Ilya Lysenkov, Victor Eruhimov and Gary Bradski has been accepted at Robotics: Science and Systems Conference (RSS) 2012.
RGBD Odometry [Maria]:
Started to work on the model capture demo. Before integrating the RGBD Odometry I was going to run the existing capture pipeline on a dot pattern or an ORB template. It is in progress now because there are problems with setting up the working environment and import/compilation/build of the pipeline.

Vadim

OpenCV stabilization & bug fixing [Alexander Reshetnikov, Vadim]
Closed 15 tickets, including 5 related to video capturing.
Further EM improvements [Maria Dimashova]
- added doc on cv::EM, moved doc on CvEM to legacy/doc (#8096);
- resolved failure of EM accuracy test on ubuntu x86 (32-bit);
- resolved failure of EM save-load test.
Background segmentation improvements [Vadim]
Cleaned up MOG2 code; Threaded it using TBB.
OpenCV infrastructure [Alexander Shishkov]
OpenCV Wiki from opencv.willowgarage.com has been converted to the redmine format (temporarily available here: http://redmine.alekcac.webfactional.com/projects/wiki/wiki/index)
Extended buildbot with new tests (nonfree, stitching & photo modules), all performance tests now run in "sanity check" mode to do some more regression testing of the library.
Moved windows superpack builder to a dedicated workstation and brought the build time from 23 hours down to 3 hours! (big thanks to Andrey Kamaev from Tegra team)
Proposal for Vision standard API [Victor, Vadim]
updated our proposal on vision API standard (aka OpenVL) for Khronos F2F in Dublin. The proposal is based on the collaborative work with experts from ARM and Samsung. Victor has presented this proposal on Tuesday.

Action Items

Gary

Talk to Cat about workshop support for OpenCV tutorial

Apply for ECCV tutorial

Kick people to talk to GSoC interns now. Computational photography is high priority

CVPR workshop (June . Now that 2.4 is out, we can start preparing for this

iOS is biggest risk because we don't have someone on this. Vadim is on this.

ECCV workshop proposal is in.

Post 2.4

Tegra team has grown: Want to release each month

They can help from infrastructure point of view until (hopefully if) OpenCV foundation is formed.

Short term: Mobile focus to CVPR

Computational photography

Collage, video summary

Many more modules

Library and application side

OpenCV apps site. Itseez seeds

Might turn into an apps store for OpenCV.org

Standard directory structure with email

Image stitching started as an app then turned into a module

Make a test case. Computational photography is one.

GSoC

SfM ... libmv 3rd guy willing to work for fun from Hartley's group

Vadim

OpenCV 2.4.0 is out [All the team + GPU + Tegra guys]
Finally, we have shipped it! Thanks to everybody who participated!
The changelog is here:
http://code.opencv.org/projects/opencv/wiki/ChangeLog
The packages are here:
http://sourceforge.net/projects/opencvlibrary/files
The announcement was sent to the mailing lists.
Bug Fixing [Alexander Shishkov, Alexander Reshetnikov, Vadim, GPU&Tegra teams]
~20 more tickets have been closed during the last week. A few more serious bugs in ffmpeg wrappers have been fixed and now it [finally?] works quite well.
Misc.
Holidays: as Kurt once said - lot's of national holidays is good, it helps to unite the nation. These days we have a bunch of such opportunities. This Monday and Tuesday, as well as the next Monday and Tuesday are holidays, while the last and this Saturday are working days.

Transparent objects [Ilya]
Solved the problem of false positive detections with a very small scale. Tried to make scale normalization of the cost function theoretically grounded by using a Bayesian formulation but it didn't work well in practice. Ended up with simple thresholding on the minimum scale.
Created a new dataset with transparent clutter and ground truth poses to evaluate our algorithm in clutter. The previous optimization with normals doesn't work well on this dataset so the algorithm was optimized by other means (for example, suppression of poses after alignment to a table plane, removing iterative refinement of the closed-form solution, sampling a test silhouette). Results on this dataset (the success rate is the proportion of cases when the translation error is less than 2cm):
-------------------------------------------------------------------------
Object Success rate Mean translation Mean pose estimation
error (cm) time (s)
-------------------------------------------------------------------------
bank 0.86 0.4 17.5
bottle 0.98 1.1 13.1
bucket 1.00 0.2 18.0
glass 0.88 0.6 8.7
sour_cream 0.95 0.4 13.6
-------------------------------------------------------------------------
mean 0.93 0.5 14.2
-------------------------------------------------------------------------
So results are slightly worse than pose estimation in clutter-free environment but still they are good enough for grasping. The optimized algorithm works the same time as on the dataset without clutter.
Also accuracy of initial poses computed by geometric hashing was evaluated alone. The accuracy is measured as the minimum Chamfer Distance between projected silhouettes and the ground truth silhouette. Results (the success rate is the proportion of the cases when this Chamfer Distance is less than 5 pixels).
------------------------------------------------------------------
Object Success rate Mean Chamfer Mean number of
Distance (px) initial poses
------------------------------------------------------------------
bank 1.00 0.7 27.0
bottle 0.98 1.1 41.7
bucket 1.00 1.8 33.8
glass 0.66 1.5 20.2
sour_cream 1.00 0.8 32.0
------------------------------------------------------------------
mean 0.93 1.2 30.9
------------------------------------------------------------------
So geometric hashing produces the accurate initial pose for our algorithm in the most of cases.
Other:
I got the Australian visa to attend RSS 2012.
RGBD odometry [Maria]
Built the ecto-based capture pipeline. It required some effort: to install ubuntu 10.04 (my installed 10.10 isn't supported by ros fuerte), to install ros fuerte, to fix several build problems of the pipeline - thanks a lot to Vincent for this.
Played with the pipeline scripts which are used for model capture based on orb template.
Experimented a little with the pipleline sample of rgbd odometry. The algorithm diverges in several cases:
1) there are moving objects in the scene or changing light (the algorithm supposes a statical scene)
2) there is a motion blur
3) rgb and depth are not synchronized (the ecto openni wrapper doesn't support such synchronization, approximated by time stamps).
Worked on improving the rgbd odometry accuracy:
1) Median filtering of rgb and depth increases slightly the accuracy on the TUM dataset.
2) Implemented a cpu version of icp from kinfu (only camera pose estimation, without a scene volume). Initial implementation uses a random subset of close points and works ~0.15s on one core of core i5 (I think it can be optimezed up to rgbd odometry time ~0.08s). The implemented icp gave the relative median pose error on the TUM dataset comparable with the rgbd method, but its trajectory was further from ground truth than rgbd's and kinfu's ones. Tried to combine rgbd and icp (to minimize not only photo consistency error but spatial error too) but again the way that improves the merged approach quality on the TUM dataset wasn't found. Will try the icp for model capturing (maybe it will be useful).
3) Started to study BA from Vincent.
Other:
We have 2 national holidays these weeks.

Vadim

Closing tickets [Vadim Pisarevsky]
6 tickets have been closed, mostly orphan ones.
opencv.org [Alexander Shishkov; our (contracted) web masters]
The first draft of opencv.org is ready (the link was sent by a separate e-mail; not to be disclosed yet).
Thanks everybody who provided feedback on the design and content.
OpenCV roadmap meeting [OpenCV, GPU & Tegra teams]
There was the meeting at Itseez regarding the possible and feasible release schedule for this year. I can not find any summary sent, but here is a short summary:
OpenCV 2.5 is expected to be out around Nov 1st.
Before that there will be at least two 2.4.x releases, may be more.
Also, we plan to make 2 hackathons - in the end of June and in September.
After some friendly conversation we seem to agreed to migrate to git in order to simplify the simultaneous work of 3+ teams working on OpenCV. One of the primary reasons to choose git vs mercurial - github and its "pull request" feature, using which we can react on most feature requests and bug reports with known solutions in a timely and friendly way - "please, do it yourself and submit a pull request". It's definitely better than leaving such tickets without attentions for years.

Make youtube videos of functionality in new releases. For example, the creation of 2.4 is here

Expand opencv projects channel

Need to fill a GSoC vacancy

Travel

Ilya had Visa. Might come for June

Maria will get a visa in July

OpenCV world

There are already 20 modules an growing

Nice modularlity but can make linking a pain

World is a module that includes all others like Frodo's one ring Makes linking easier

Can be customized to contain only the modules that user wants to make it smaller

Will split some modules into even smaller modules to make them even smaller

Needs to be documented

Installation guide. Important especially for Windows OS

Transparancey

Optimizing ... sped up 3x so ~5s/frame

Works in clutter

RSS presentation (Ilya is going but paying for 50%)

Intel likely to start getting involved

Application engineers from NN center

Vadim

Only 3 working days (Thursday, Friday and Saturday) this week due to Russian hollidays.
Still, here is the progress:
Bug fixing [Vadim et al]
2 bug in OpenCV 2.4 have been fixed: #1918 and #1931.
Other flavors of cv::split and cv::merge have been added (to use InputArray & OutputArray) to support vector<Mat<T> >, as requested by Vincent.
Also 5+ bugs have been fixed by other teams at Itseez that work on OpenCV.
opencv-world [Vadim and Andrey]
The new super-module modestly called opencv_world has been introduced. It combines all the other modules selected by user and thus it's much more convenient for redistribution, comparing to ~20 shared libraries libopencv_core, libopencv_imgproc etc. The module takes ~10-15Mb and it is linked from the same object files (their union) as the other modules, so the overall OpenCV compile time increased by just a few seconds. Big thanks to Andrey Kamaev from tegra team who came with the implementation idea and then significantly improved and fixed the initial version of the cmake script.
iOS port [Vadim]
I started playing with iOS port of OpenCV; With very minor modifications OpenCV 2.4.0 was built for iOS using the instructions from
http://docs.opencv.org/doc/tutorials/introduction/ios_install/ios_install.html
Next steps would be to port some of Android samples to iOS.
OpenCV site [Alexander + our contracted web masters]
The final draft of opencv.org has been created: http://opencv.top7y.ru/
You are welcome to check it out and provide feedback.
Welcome Evgeny!
Welcome the new member of our team, Evgeny Talanin, who joined OpenCV team last week. He will replace Alexander Shishkov. However, there is a good chance that Alexander will stay with us till the end of the month.
Meanwhile, Alexander Reshetnikov is on vacation till the beginning of June.

Anatoly

Accomplishments & statuses:
1) OpenCV GPU
· Fixed build under MacOS (disabled video decoding, it's not supported). Closed tickets #1754, #1846.
· Prepared draft of work plan for next 2 months.
Issues:
· GPU team still doesn’t have any Kepler to test it with GPU module.
· Some compilation errors under Mac (for static libs), to fix this need to wait until Mac Pro from our build farm will be free.
· We only check compilation for MacOS, but never executed tests there, since we don’t have Mac with CUDA capable GPU. Regarding PCL, we even never compiled it on Mac.
2) Background substation
· Investigated current OpenCV CPU implementation, learned paper. (We going to port both current functionality and new VIBE).
Plans:
· Continue work on background substation, implement first GPU version. Test if merging (using Python script) all cu-files to one reduces GPU module binary size.
· Migration to Git. Prepare mini-training for Itseez teams.

Ilya

Transparent objects [Ilya]:
Optimized the pose estimation algorithm. A lot of optimizations were implemented e.g. caching results of computations and sharing them in pose estimations for different initial poses, computing a jacobian for inliers only, scaling an image to get an approximate final pose and then refining it on the original image, using downsampled edgels in geometric hashing.
Results (the success rate is the proportion of cases when the translation error is less than 2cm):
------------------------------------------------------------------------------------
| | | Mean | Mean pose estimation time (s) | |
| Object | Success | translation |-------------------------------| Speed-up |
| | rate | error (cm) | Old | New | |
------------------------------------------------------------------------------------
| bank | 0.88 | 0.4 | 17.5 | 6.8 | 2.6 |
| bottle | 0.91 | 1.2 | 13.1 | 4.3 | 3.0 |
| bucket | 0.98 | 0.2 | 18.0 | 6.5 | 2.8 |
| glass | 0.92 | 0.7 | 8.7 | 2.9 | 3.0 |
| sour_cream | 0.97 | 0.4 | 13.6 | 4.7 | 2.9 |
------------------------------------------------------------------------------------
| mean | 0.93 | 0.6 | 14.2 | 5.0 | 2.8 |
------------------------------------------------------------------------------------
So the optimized algorithm works with approximately the same quality but it is 2.8x faster.
RGBD odometry [Maria]:
Experimented with mesh_object. Thanks to Vincent for resolving the problem with it.
Refactored the merged method rgbd + icp. Evaluated it on the TUM dataset again (see the graphics of translation errors in the attachment).
Results (the description of the errors can be found here: http://cvpr.in.tum.de/data/datasets/rgbd-dataset/online_evaluation):
RGBD ICP RGBD + ICP
Absolute trajectory error (median), m 0.387107 0.443915 0.066765
Relative transnational error (median), m 0.012085 0.013843 0.015936
RGBD has the smallest relative median error (frame-to-frame). But it also has one bad pose breaking the camera trajectory, so it has the high absolute trajectory error. RGBD + ICP has slightly worse relative median error, but its absolute trajectory error is much less, it does not break the trajectory. So on the TUM dataset merged approach is more stable than pure RGBD. But the dataset has motion blur and it's the most likely reason of the one bad pose. ICP has several poses breaking the camera trajectory.
Other:
Maria has applied for the USA visa. The interview will be on July 3 (it is the nearest free time).

Extend the stuff to more views, more camera types. No noise and noise. Missing entries.

Vadim

Bug fixing [Vadim, Alexander Mordvintsev]
closed 6 tickets. made FaceRecognizer classes derivatives of Algorithm, so that different properties can be set using the unified interface.
GSoC
The students have just started the work; there is no meaningful reports yet; the work on iOS samples started; Alexander Mordvintsev ran existing Python samples and fixed a few problems in the samples, in the wrappers and in OpenCV itself to make sure the samples run.
OpenCV site [Alexander Shishkov + our contracted web masters]
Another final draft :) of http://opencv.alekcac.webfactional.com/. It addressed some of the concerns and requests, but not all. Still, the internal representation is pretty simple (wordpress-based) and can be easily edited by us (which needs to be done anyway). Today we had meeting at Itseez regarding the further steps.
Other than that, wiki has been partially converted to redmine format: meeting notes (http://code.opencv.org/projects/opencv/wiki/Meeting_notes), Changelog (http://code.opencv.org/projects/opencv/wiki/ChangeLogHistory)

Transparent objects [Ilya]:
Finished the first draft of the camera ready paper for RSS 2012. Created a private git repository for it so we can easily collaborate on the paper.
Searched and studied new papers related to transparent objects. There are 2 interesting papers:
1. Zhong Lei, K. Ohno, M. Tsubota, E. Takeuchi, and S. Tadokoro. Transparent object detection using color image and laser reflectance image for mobile manipulator. IEEE International Conference on Robotics and Biomimetics (ROBIO), December 2011. Light detection and ranging (LIDAR) data are used to segment unknown transparent objects: highlight spots from a RGB image are used to find candidate areas of transparent objects and then the GrabCut segmentation algorithm is applied to a depth image and a laser reflectance intensity image to compute the final segmentation. The experimental setup consist of a 2D LIDAR device, a registered RGB camera and a pan-tilt unit so it is more complex than Kinect which we use.
2. Meriaudeau, R. Rantoson, D. Fofi, and C. Stolz. Review and comparison of non-conventional imaging systems for three-dimensional digitization of transparent objects. Journal of Electronic Imaging, May 2012. This paper covers different approaches to scanning transparent objects which were published after the excellent previous review on this topic: I. Ihrke, K.N. Kutulakos, H.P.A. Lensch, M. Magnor, and W. Heidrich. State of the art in transparent and specular object reconstruction. In EUROGRAPHICS 2008. Results are still the same: there is no robust and reliable technique to scan transparent objects in general case.
Other:
Bought flight tickets to/from the USA. I still have my passport in the UK visa center but they officially guarantee that I have at least 90% chance to get it the next week and I have 98% by statistics in March.
RGBD odometry [Maria]:
Installed more dependencies of the ecto capture pipeline. So now all the capture scripts work for me. Captured a couple of objects using an orb template to practice with the pipeline, got their meshes in the browser.
Replaced the existing pose estimation in the pipeline by rgbd odometry. Saw the first results and it seems I applied transformations in the wrong order somewhere, investigating this now. A warped image was stable in that experiment and it means rgbd odometry gave accurate poses.
Other:
The interview for the visa has been rescheduled for June 22.

Action Items

Gary

Check with Carol if Eduard is in

Go to guy who manages tutorials and get the link on the CVPR page above

Try registration on our site

Vincent

Send email summarizing libmv work

Ilya

.

Vadim

Look into email accounts: Google or some other? Commercial or some other?

[Bug fixing; OpenCV team et al]
18 tickets have been closed; 7 of them - by OpenCV team.
Most of them are related to documentation and various build/install issues.
[New functionality: Vadim]
Now all the OpenCV data structures can be stored to text strings and restored back

Such functionality can be useful for mobile platforms and in large-scale systems that use databases to store data instead of regular files.
[Build infrastructure: Evgeny, Alexander]
Updated our build environment to OSX 10.7, new VMWare, new Linux distros (including Ubuntu 12.04LTS), added parallel MinGW+TBB build, moved some builds to different machine, so now it's much, much faster. The ultimate goal is to have "build-test before commit" feature to make the trunk version of OpenCV always more or less stable.
[opencv.org, code.opencv.org: Alexander]
a few tweaks to opencv.org have been made to opencv.org: http://opencv.alekcac.webfactional.com/
created SVN repository for GSoC 2012 projects at code.opencv.org. Unfortunately, git still requires more time to configure it properly.
[iOS port: Eduard, Vadim]
Eduard Feicho now officially started working on the project (thanks to Gary!), he setup all the environment and built OpenCV for iOS using the already supplied with OpenCV scripts.
I created a simple app that reads video from camera and processes it using OpenCV function (laplace); will now proceed with something more complex, e.g. face detection.

RGBD odometry [Maria]
Found the bug mentioned at the last time in my initial integration of the RGBD to the ecto
capture pipeline. Made experiments to scan some models using the RGBD, see an example in the
attachment. Tried two ways of scanning: 1) static scene + moving Kinect and 2) rotated table with
an object + static Kinect. In reality it is hard to provide an ideal static scene for the whole frame
(e.g. due to other people in the room). So the RGBD was not stable in both cases and its other
results were not so good as the attached example. To satisfy this condition of the RGBD we may
run it on pixels of a table with an object only and use the second way of scanning.
Tried to get a mask of a table with an object (to pass it to the RGBD) using PlaneFitter for each
frame. But PlaneFitter returns unstable axes (rotation) if there is even a relatively small object on
the table (it is unexpected behavior of the fitter). Moreover, it slightly slows down the capturing.
So to get the stable mask the ORB-based poser is run first. The RGBD processes pixels of this
mask only which provides a stable part of a scene.
Experimented with the RGBD working on the stable part of a scene. It still does not give accurate
models because it accumulates frame-to-frame errors of a camera pose. Investigating the reasons
of the errors now.
Other:
Found and fixed a bug in ecto_opencv. This bug produced the incorrect ORB-based poser after
updating the ecto_opencv.
Transparent objects [Ilya]
Revamped the draft of the camera-ready paper for RSS.
Prepared a poster about our work with transparent objects.
Adapted a paper about our previous approach and submitted it to a conference.
Started to refactor the transparent_objects module.
Other:
Ilya will go to the USA this weekend. He will leave the office for ~1.5 month:
June 02 - June 15 Preparing the demo at Willow Garage, Menlo Park, USA
June 16 - June 21 Attending CVPR as a visitor, Providence, USA
June 25 - June 29 Participating in the BMVA/EPSRC Computer Vision Summer School (site), Manchester, UK
July 2 - July 6 Participating in the Microsoft Research PhD Summer School (site), Cambridge, UK
July 9 - July 13 Presenting our paper at 2012 Robotics: Science and Systems Conference (site), Sydney, Australia
July 14 - July 17 Vacation, Sydney, Australia

Action Items

Gary

Call Secretary of state in California to see if we can expedite a previously filed corporate filing.

Call Cat to give here a "heads up"

Schwag sites

Vincent

Check out purchase of usb keys with data

Try to get cash from Willow

Ilya

.

Vadim

.

Victor

Check out purchase of usb keys with data

Find if groups are possible with a free account

else have to pay for a couple of accounts.

Ethan

.

From Last Time

Gary

Check with Carol if Eduard is in

Go to guy who manages tutorials and get the link on the CVPR page above

Try registration on our site

Vincent

Send email summarizing libmv work

Ilya

.

Vadim

Look into email accounts: Google or some other? Commercial or some other?

OpenCV.org Foundation could sponsor improvement of vision pages on wikipedia

See the ideas google docs for more

RGBD module [Vincent and Ilya]

fast normals

fast plane finding

ICP

Code to clean up NANDS "patch nands"

Object capture is a priority going forward

Transparency

Reproduced results from the Summer onto the robot

About 5 secs/frame

GSoC Reports

2 students on iOS

4 projects using camera etc Want to improve before CVPR

Alexander working on Python samples

Documentation: Every example should have a help that tells what it does and how to call it

Deva Ramanan: Parts based code Hilton port to objdetect module

Vadim

[OpenCV 2.4.1: Alexander Shishkov and Andrey Kamaev, OpenCV team]
The first stabilization release in the 2.4.x series is out. Here is the complete ChangeLog:
The GPU module now supports CUDA 4.1 and CUDA 4.2 and can be compiled with CUDA 5.0 preview.
Added API for storing OpenCV data structures to text string and reading them back:
1 //==== storing data ====
2 FileStorage fs(".xml", FileStorage::WRITE + FileStorage::MEMORY);
3 fs << "date" << date_string << "mymatrix" << mymatrix;
4 string buf = fs.releaseAndGetString();
5
6 //==== reading it back ====
7 FileStorage fs(buf, FileStorage::READ + FileStorage::MEMORY);
8 fs["date"] >> date_string;
9 fs["mymatrix"] >> mymatrix;
cv::calcOpticalFlowPyrLK now supports precomputed pyramids as input.
Function signatures in documentation are made consistent with source code.
Restored python wrappers for SURF and MSER.
45 more bugs in our bug tracker have been fixed
[Buildbot: Evgeny Talanin]
created pre-commit script that runs three selected builds and only then allows to check the changes into the OpenCV repository.
[OpenCV sites: Alexander Shiskov]
1. some minor fixes to the prototype of opencv.org were made. The site will likely be revealed on this weekend.
2. installed and tested the new version of Askbot. It seems that all the known problems with it have been solved. It means that answers.opencv.org should be ready at the same time as opencv.org.
[GSoC 2012]
1. Eduard Feicho created 3 sample projects for iOS that will further be turned into templates for OpenCV-based applications for Apple mobile devices:
https://github.com/Duffycola/gsoc2012
- IntroCamera that shows how to use the camera/photo library (UIIImagePicker) and the video camera with autofocus (using the AVFoundation framework).
- IntroOpenGL (and LaserTowers), which should be an example OpenGL ES project.
- IntroOpenCV, which was supposed to be a hello world example for opencv+iPhone project.
the projects are now in very early alpha stage and are to be improved and cleaned up.
2. Alexander Mordvintsev created digits.py sample that demonstrates the use of OpenCV neural nets for hand-written digits recognition.

RGBD odometry [Maria]
Investigated the accuracy of the RGBD again because:
1) As it was mentioned in the last report I did not get sufficiently accurate models using the initial RGBD integration to the ecto capture pipeline;
2) The comparison with the TUM paper in February showed that our RGBD implementation gave the same errors as in the TUM paper only if I synchronized rgb and depth with a smaller time interval (0.01s) than the interval used in the TUM benchmark by default (0.02s).
So it seemed that there are unimplemented ways to improve our RGBD quality. But there was a bug in my conversion of computed camera pose to the format used in the TUM benchmark, i.e. the problem was out of the our RGBD. Recomputed the results of my previos experiments:
paper's results our results
RGBD GICP RGBD KinFu ICP (without volume, c++ version) RGBD + ICP
freiburg1/desk (573 frames)
translational_error.median (m) 0.0053 0.0103 0.0058 0.0066 0.0054
rotational_error.median (deg) 0.0065 0.0154 0.0059 0.0111 0.0055
absolute_translational_error.median (m) - - 0.1371 0.4439 0.0661
freiburg2/desk (2893 frames)
translational_error.median (m) 0.0015 0.0062 0.0022 0.0077 0.0021
rotational_error.median (deg) 0.0027 0.006 0.0033 0.0058 0.0032
absolute_translational_error.median (m) - - 0.2432 2.3209 0.2176
Сonclusions from the updated results:
1) our RGBD has accuracy close to the paper's accuracy even we use the default parameters of both the benchmark and the RGBD (from contrib module);
2) RGBD + ICP has the smaller errors (frame-to-frame and absolute) than the our pure RGBD and pure ICP.
With more confidence in our RGBD quality I am choosing the method for the global refining the camera poses and captured model, reading the papers about this now.
Other:
Maria is on vacation for a week from June 4.
Transparent objects [Ilya]
Prepared the final draft of the paper for RSS.
Improved the poster by taking into account comments from colleagues (thanks a lot to all who provided their comments).
Continued to refactor transparent_objects module.
Helped Vincent with his implementation of the fast computation of normals.
Set up the working environment on my Willow machine and reproduced pose estimation of transparent objects on a robot with the old algorithm (without grasping yet). Thanks to Vincent for help.

Action Items

Gary

Try to get corporate filing done

Vincent

Get intern to add "patch nands"

Ilya

.

Vadim

.

Victor

.

Ethan

.

From Last Time

Gary

Call Secretary of state in California to see if we can expedite a previously filed corporate filing.

[iOS port; tutorial preparations: Vadim, Eduard and Charu]
last week most of the time was spent in improving OpenCV support on iOS and preparing some demos.
First of all, a script that creates opencv2.framework, an iOS framework, has been created.
It turned out that iOS simulator does not simulate ARM instructions, it's i386 "virtual machine" with special versions
of Cocoa libraries. In other words, when you build an app for simulator, it's i386 CPU (aka 32-bit Intel arch); when
you build the same app for real device, it's ARM. So, we needed 2 different sets of OpenCV libraries to make an
OpenCV-based app to compile and run on both simulator and device. If we take into account that there are
already ~20 modules in OpenCV, setting all the include and library paths in an application may be really boring
and not forward-compatible thing. But now using the script we can create a "universal framework" called
opencv2.framework. It combines both versions of the library, together with the headers. It's super-easy to add
this framework to application, just like one adds system frameworks. Xcode automatically links application with the
correct variant of the library, depending on the target.
Meanwhile, GSoC students, Eduard and Charu, did excellent job on creating new iOS samples. Right now we have
more or less good working face detection demos. One works with still images from the photo library, another one
works with live video from camera. The screenshot is attached.
[GSoC Python interface improvements; Alexander]
Alexander Mordvintsev continues work on python wrappers and samples; he added description to digits.py,
added new sample on laplacian pyramid; fixed 2 bugs in the wrappers and other part of OpenCV causing
bugs in python samples.

Anatoly

Accomplishments & statuses:
1) OpenCV GPU
Found reasons and fixed thousands of warnings about "undefined anonymous" in device code under Linux.
(need to implement copy-constructor for device functors themselves).
Resize with INTERP_ARA flag (usually used for big scale coeff, where bilinear or others behave very bad.
Algorithm example: SURF)
Added simple implementation of cv::gpu::resize with INTERP_AREA interpolation type. Average speed-up
25-65x for scale coeff = 3..5. (GTX470 vs Core i5).
Fixed bugs in CPU version of resize function related area interpolation path. Changed behavior of CPU
resize implementation according right formula for tile with not full area.
For large scale coeffs (>10x) speed-up is very little. Working on smarter version for this case.
Spend time for dense optical flow (PyrLK) optimization ( we focues on sparse before, dense method
was a bonus)
Using shared memory for caching first image - it gives maximum 1.2x speedup in some cases.
Using texture memory for second image - it doesn't gave any speedup.
Using float source + texture built-in linear interpolation - it gives ~2x speedup.
Video stitching application. Updated to support interface changes in opencv_stitching module. Tested
the app and sent to NVIDIA.
2) Background substation
Integrated FGD foreground/background subtraction to OpenCV (total 6x speedup).
Plans to work on VIBE
Plans:
Test new dense optical flow implementation, try some other ideas. Think how to make sparse optical
flow faster.
Implement faster resize with INTEP_AREA flag for large scale coeffs.

Ilya

Transparent objects [Ilya]:
Merged a branch with the algorithm of pose estimation in transparent clutter into the master branch.
Experimented with it on the robot and made sure it works correctly.
Documented public API of classes for recognition and pose estimation. Implemented a simple sample
how to use our algorithm and described it in the Ecto Recognition Kitchen documentation. Cleaned the
code of our algorithm, submitted the camera-ready paper to RSS 2012.
Investigated and solved the issue with too large memory requirements of our algorithm. The reason
was STL doesn't release memory of deleted unordered_multimaps but holds it for future objects. We had
several temporary but huge hash tables in our algorithm and so memory was not released after their deletion.
Switched to using cv::Ptr to create hash tables only once and it solved the problem.
Vincent fixed integration of Ecto with the grasping pipeline after changes in Fuerte and the robot was
able to grasp a transparent object after these fixes.
Other:
Maria is on vacation.

Action Items

Gary

Try to get corporate filing done

Work on CVPR foils

Vincent

Get intern to add "patch nands"

Help with cvpr workshops

Ilya

.

Vadim

Work on CVPR workshop foils

Victor

Work on CVPR workshop foils

Ethan

.

From Last Time

Gary

Call Secretary of state in California to see if we can expedite a previously filed corporate filing.

iOS port Eduard had an exam, but did a lot before CVPR which helped a lot for the workshop

Camera is working for iOS, needs to go into HighGUI

Simpleflow coming along

Single scale example with algorithmic tests is done

Going to multiple scale

Might also get a side benefit of a faster bilateral filter

Website is good to go. Deploy any time.

Do a once over

Khronos committee is now fully formed and working on a vision standard. Victor is the chair.

Will work with Stefan Hinterstoisser for final version of LINEMOD by September

Priorities '12

iOS

libmv

More calibration improvements 2D, 3D, mix

Model capture

VO

optimization --

multi-threading on Windows and Mac using their native tools

Allow swapping out TBB for OpenMP for those that don't have

OpenCL use wherever possible

Many more SSE intrinsics

In general, we're moving this into single optimization class so that all code becomes the same but can specialize to architecture

We will then immediately comply with Khronos when that standard is finished.

Mobile

iOS (coming along nicely)

Computational photography

Vadim

[Tutorial about iOS+OpenCV programming: Vadim, Eduard and Charu]
Tutorial on using OpenCV on iOS has been prepared and given at CVPR'2012 in
Providence, RI. These are the slides:
http://code.opencv.org/svn/gsoc2012/ios/trunk/doc/CVPR2012_OpenCV4IOS_Tutorial.pdf.
The tutorial went well (though, because no attendees had any experience with iOS programming
and, therefore, no iOS developer account, they only experiment with iOS simulator, so not all the
samples could be run). Before ECCV'12 we should do something about it.
Eduard and Charu continue to refine iOS port of OpenCV, support for color input from camera
has been added and few bugs have been fixed.
[FREAK feature detector+descriptor]
The new feature detector, contributed by EPFL, has been integrated into OpenCV trunk, features2d module.
[Buildbot: Evgeny]
the new builders have been added: with Qt support, OpenGL, Windows with GPU;
email notifications problem resolved, now receiving information about every build
finished pre-commit script that should test working copy before committing it to trunk.
Unfortunately, since we now on SVN, the script is not very convenient to use - it's not
quite transparent for developer. Hopefully, with transition to git it will be solved.
Ilya is traveling.
Here is update from Maria:
Cleaned up an odometry evaluation application:
Removed median filtering that was applied to image and depth. For previous RGBD
implementations this filtering improved results on TUM benchmark. But current implementation
is slightly accurate without this filtering (I suppose due to M-estimator added last week).
Found experimental uncommented code in the application. Without it RGBD has less errors on
TUM benchmark.
There are no updated RGBD results, since Maria is on sick leave since June 21th.
Also, Maria came to Moscow for interview and could not get visa :(

Victor is chair, survived back wounds. Next meeting August face to face

AGAST -- feature part of BRISK

Best of ORB and BRISK paper Intern on "yet another binary discriptor" but on better distance function

OpenCV foundation could host a mini-conference/hackathon and come out with a "best of breed"

Model capture

Ilya's kin

Vadim

[OpenCV 2.4.2: Tegra and OpenCV teams]
The next OpenCV update is almost released. The code has been frozen and the packages are being created at the moment.
Here is the redmine page for the release:
http://code.opencv.org/projects/opencv/versions/6
in short, the new release will get about 45 bug fixes, OpenCV service for Android, OpenCV framework for iOS, comprehensive documentation and samples for the FaceRecognizer, some more GPU optimization etc.
Big thanks to Tegra team who actually did most of the work on preparing the release.
[OpenCV site: Alexander Shishkov]
Finally, http://opencv.org and http://answers.opencv.org are now running!
[Contributions]
pretty interesting background subtraction algorithm GMG:
"Visual Tracking of Human Visitors under Variable-Lighting Conditions for a Responsive Audio Art Installation", A. Godbehere, A. Matsukawa, K. Goldberg
has been contributed by the algorithm authors, along with the sample and tests. The screenshot is attached.
[GSoC 2012]
Charu and Eduard continue to work on iOS samples.
Charu started working on camera calibration + augmented reality demo for iOS. Camera calibration part has been written and is being debugged now.
Eduard is working on improving and accelerating VideoCameraController class that captures video from camera on iOS.
Alexander Mordvintsev continues to improve digits recognition Python sample - with better feature selection and tricks from CVPR'2012 paper:
R. Arandjelovic, A. Zisserman
"Three things everyone should know to improve object retrieval"
the performance is now much better and SVM trains much faster.
Descriptions to 2 other Python samples have been added.
[Buildbot: Evgeny]
binary compatibility between 2.4.1 version and 2.4 branch is now regularly tested. Overall verdict is 'compatible' but still no public webpage exist
added iOS framework builder, it works and seems to be good for using in XCode projects;
configured Android builds to be run simultaneously, and Android tests to be run in parallel with any non-Android builds. Total duration of build queue on Bigmac reduced by ~1.5 hours;
modified Ubuntu x64 QT builder: installed number of video I/O libraries, upgraded Qt to 4.8.2;
minor activities: run tests via run.py, correct removing old MegaPack builds result etc.

Still waiting on banking keys, codes for full access. As soon as we have this, make more noise

Priorities

MOSSE

Eulerian Mag

Mobile acceleration. TBB

Taking a theoretical look at descriptors ...

Tracking

Look at tracking code from last year's GSoC, some of it was good, get it in

Contrib or Video

Meanshift is in contrib

Meanshift alg is in video

VO 35% faster

Segmentation engine

Matlab

Mexopencv

Headers only to make it easy, Matlab to Mat

Release

2.4.2 Release is out now

2.4.3 End of Summer

2.5 end of year

Hackathon developer guide.

Sept

Khronos F2F in July in Texas

Vadim

[OpenCV 2.4.2: Tegra & OpenCV teams]
OpenCV 2.4.2 has been released on July 4th:
http://code.opencv.org/projects/opencv/wiki/ChangeLog
[GSoC2012, iOS: Eduard & Charu]
Eduard and Charu continue work on the OpenCV for iOS samples.
We decided to prepare free apps by the end of the project: "OpenCV Demo" and
may be an "OpenCV Image Effects" and put them to AppStore, in order to
popularize OpenCV.
Here is screenshot of the current version of "image effects" demo.

The idea is to make "image effects" runnable on simulator, so that people can play
with iOS and OpenCV without developer account or real ios device. "OpenCV Demo"
however, will be video based demo, demonstrating face detection, elements of AR, video
effects etc.
[GSoC 2012, Python samples: Alexander]
created very nice sample for planar object tracking using ORB features:
http://www.youtube.com/watch?v=FirtmYcC0Vc
made the how-to on using PiCloud
started experiments with Kinect (using OpenNI-MS bridge, but this could possibly
be replaced with highgui at some point)
[Python bindings: Eric Christiansen and Vadim]
A part of contrib module is now wrapped in Python too; it means that chamfer matching,
variational stereo correspondence, tone mapping and, most importantly, face recognizer
are now available for Python users.
[Buildbot: Evgeny]
done first tests of the pre-commit script and received feedback from the Tegra team.
wrote proof-on-concept hook for git to run pre-commit (actually, pre-push) testing when
we move to git.
started installation of a new, much more powerful redmine server at Hetzner.de, where
we create opencv git repository. Optimistic estimate of the transition time - 1-2 weeks.

RGBD odometry [Maria]
Made some optimizations of odometries code (Mat elements access, precomputing,
conditions order, etc). That speeded up the algorithms 25-35%.
Chose the paper
Highly Accurate 3D Surface Models by Sparse Surface Adjustment by
M. Ruhnke, R. Kümmerle, G. Grisetti and W. Burgard, ICRA 2012)
for the global refinement of camera poses and point clouds for a model capture task.
Experimented with the paper's model of Kinect depth errors due to quantization in our odometries.
There are not results yet.
OpenCV [Maria]
Simple fix #2118.
Other:
Ilya attended Microsoft Research PhD Summer School 2012. The school was starring
Antonio Criminisi, Jamie Shotton, Tony Hoare and many others.
Ilya moved to Sydney (Australia) to participate in the RSS 2012 conference and
present our work on transparent objects.

StereoBM and StereoSGBM classes not being derived from a common base class.
I realize that both of these objects take slightly different initialization variables, but there are many
better ways to handle this. In fact, most of the rest of the library uses some kind of initialization
structure which is itself derived from some common base class, such that each derived stereo
algorithm can be initialized by its own custom initiailizer.

Update on foundation.

Paypal up

Board

Credit cars or just Paypal? ... $300/year cost

Watch lists on websites?

ASIFT
*

Minutes

Stereo: refactor to inherit from base class

Can't do it right now due to backward compatibility for 2.4*

Will do in 2.5

ASIFT

Can apply to other detectors: Orb, FREAK

Implemented in python right now, worth moving into OpenCV

Next 2.4 release, 2.4.3 will be in Sept.

2.4 focus is

increasing optimization,

better camera support,

GSoC for new stuff.

Python much improved, especially samples (good GSoC work this year!)

SfM

Mobile is much better as well

Folding in last year's GSoC tracking in. Vincent cleaning it up.

Photo module for computational photography

Better modular infrastructure

2.5 might start Nov. Adrian rename conventions

Need youtube, facebook, twitter

Need a person for this. Board meeting issue, leverage community

Adrianna has left Citi. Christine as new person.

Itseez team alternates "bug" duty to close out old bugs and deal with new to keep this much more up to date.

Look into FAST code 5-9 as well as what we have 9-15

Vadim

[GSoC2012, iOS: Eduard & Charu]
Both students have successfully passed mid-term evaluation.
They wrote the reports for the passed part of the projects (http://code.opencv.org/svn/gsoc2012/ios/trunk/doc/).
In particular, Eduard created a draft version of iOS OpenCV demo app that
demonstrates various video effects:

[GSoC2012, Python: Alexander Mordvintsev]
Alexander has also passed the mid-term evaluation.
These is update from him:
spotted the problem with SURF features
modified the find_obj sample:
support SIFT, SURF and ORB features
BF and FLANN matching
interactively explore the matching points
added ASIFTdemo (which in fact is also "AORB" and "ASURF")
similar to find_obj demo, but with affine sampling
works well in rather difficult scenarios

[OpenCV, features2d: Maria, Alexander Mordvintsev]
Maria and Alexander (who works on GSoC 2012 OpenCV Python samples) have found some
severe problems in the latest version of SURF, SIFT and ORB. It appears that the coordinate
system (left or right) used for KeyPoint::angle estimation is not specified, and different
detectors use different ones. The guys mostly fixed the problem, added new meaningful
(in addition to simple regression tests) tests that cover the problem.
These are details from Maria:
===
1) Specified the KeyPoint::angle (in the rst doc and in the code comments).
2) Implemented new unit tests of 5 types:
- a simple test of output keypoints from FeatureDetector (validity of pt, angle and size),
- tests on rotation invariance for FeatureDetector and DescriptorExtractor,
- [additionally] tests on scale invariance for FeatureDetector and DescriptorExtractor.
3) Synchronized Feature Detectors and Descriptor Extractors according to given
KeyPoint::angle specification. Thanks to Alexander Mordvintsev for the fix of SURF descriptor.
4) The tests found some possible problems: ORB shows low scale invariance (especially descriptor),
FREAK shows low rotation and scale invariance. Opened the issue #2176 to find the reasons of this
(a problem in the tests or in the algorithms).
===
There are still some problems with scale and rotation invariance that we will look into.
[OpenCV + OpenCL: user contribution]
Finally, ocl module that lived in a separate branch for a while, has been merged into trunk.
The module contains some simple and fairly complex algorithms (like stereo correspondence,
face detection) accelerated using OpenCL. Architecture of the module is similar to CUDA-based gnu module.
Currently, the code can be classified as "good" in Linus Torvalds terminology; in other words, it compiles :)
Unfortunately, all the tests fail on my mac, since it uses CPU as the default OpenCL device for some reason,
while the module has only been tested on a dedicated graphic cards. The problem will be investigated.
[Migration to GIT: Evgeny, Alexander Shishkov]
We successfully installed Chiliproject on our new server at hetzner.de, migrated our bug tracker, wiki there
and installed a test git repository. We plan to finish migration to the new server and git this week.

Anatoly

Accomplishments & statuses:
1) LBP Face detection
· We continue investigation of performance bottlenecks and brainstorming optimization ideas.
· Implemented simultaneous processing of different scales. Current version is up to 2.8x faster (
GTX470 vs. Core i5-2500k, TBB)
· Tried the idea with jobs list which is compacted after each stage. Speed-up is 2.4x. Main reason
is low GPU occupancy at small pyramid levels.
· After all investigations, developed (in-mind) the method with constants grid size (its description
I sent yesterday) and working on its implementation.
Plans:
· Try to offload the algorithm to the GPU using OpenGL ES shaders.
3) OpenCV GPU
· Learned ‘Non local means denoising’ paper.
· Merged documentation and android service from trunk to 2.4.
· Background subtraction: No update
4) Other
· Final preparation for OpenCV migration to Git which will be done in 1-2 days.
· Hiring activities, most likely we have a candidate to fill the second PCL 4 Tegra position.
Plans:
· Continue LBP cascade classifier optimization.
· PCL4Tegra: optimization ‘bodyparts’ with GLSL.
· Previous work review for speed limit sign detection algorithms. Implementing non-local means
denoising on GPU.
· Try another optimization idea for ‘cv::gpu::resize’ with INTERP_AREA flag for large scales (after LBP
work has been finished)

Ilya

Transparent objects [Ilya]
Participated in the RSS 2012 conference and presented the paper "Recognition and Pose
Estimation of Rigid Transparent Objects with a Kinect Sensor" by Ilya Lysenkov, Victor Eruhimov
and Gary Bradski (5 min spotlight talk + 1.5 hour e-poster + 5 min talk at NICTA headquarters).
Other:
Our paper on pose refinement of transparent objects was accepted to the largest Russian computer
vision conference GraphiCon (http://en.wikipedia.org/wiki/GraphiCon). This paper covers the algorithm
that we developed ~1 year ago before switching to Kinect.
OpenCV [Maria]
All last week I worked on OpenCV because of detected problem with the KeyPoint::angle (#2043).
There was not a strict specification of the KeyPoint::angle before (units, a range of values and coordinate
system). Feature Detectors and Descriptor Extractors have used different own specifications of an angle
and it was a potential bug in a feature detection pipeline.
1) Specified the KeyPoint::angle (in the rst doc and the code comment).
2) Implemented new unit tests of 5 types:
- a simple test of output keypoints from FeatureDetector (validity of pt, angle and size),
- tests on rotation invariance for FeatureDetector and DescriptorExtractor,
- [additionally] tests on scale invariance for FeatureDetector and DescriptorExtractor.
3) Synchronized Feature Detectors and Descriptor Extractors according to given KeyPoint::angle specification.
Thanks to Alexander Mordvintsev for the fix of SURF descriptor.
4) The tests found some possible problems: ORB shows low scale invariance (especially descriptor), FREAK
shows low rotation and scale invariance. Opened the issue #2176 to find the reasons of this (a problem in the
tests or in the algorithms).
Last week was also my turn to be on duty of OpenCV track:
done #2165, #2152, #2155, #2156, #2159, #2160, #2167, #2043,
cancelled #2151, #2107, #2174, #2172.
Plans:
Back to model capture task.

[GIT & New Server: Alexander, Evgeny]
We finally moved to the new server (http://code.opencv.org now points to it).
The migration took a bit longer than we expected, because of the differences between Chiliproject and
Redmine and other subtle nuances.
Also, because Redmine themes are not fully compatible with Chiliproject's, we did some rough customization
of the interface.
We already tested various features, including the bug tracker, wiki, git and everything's working well.
[GSoC Python: Alexander Mordvintsev]
Alexander continues work on tracking samples in Python. Here are the details from him:
MOSSE tracker http://www.cs.colostate.edu/~bolme/publications/Bolme2010Tracking.pdf
I successfully managed to implement MOSSE correlation based tracker; it was compared with some videos
http://www.youtube.com/user/bolme2008
from the paper's author and I've got very similar results. It is quite robust and fast. The code is still need to be polished. S
ee issues section also.
notes on ASIFT
Not actually a progress, just a case of unexpected performance of ASIFT sample. Once I took a few images from a
small aircraft. I tried to run asift.py on downscaled versions of these (aero1.jpg and aero3.jpg). The result is match.png.
Commsnd line was
"asift.py --feature sift-flann aero1.jpg aero3.jpg".
I didn't expect it to find anything in the case when I myself hardly spot matches on the grayscale versions of images.
But it did found 42 homography inliers (out of 134 matches).

Issues
I found a couple of issues with DFT implementation in OpenCV.
In order to divide and take conjugates of spectrums I used DFT_COMPLEX_OUTPUT flag for cv2.dft. It seems to leave
half of a resulting array uninitialized, when the source is real:
a = np.zeros((8, 8))
A = np.zeros((8, 8, 2))
A[:] = 123
cv2.dft(a, A, flags=cv2.DFT_COMPLEX_OUTPUT)
print np.sum(A == 123) # expect everything to be zero, but some values are uninitialized
This is not a problem, when transforming back with DFT_REAL_OUTPUT flag, but gives me lots of warnings when I
manipulate spectums with numpy (overflows and invalid values). I'll add this bug to a tracker when it becomes available.
Also I found, that in theory I could use CCS-packed spectrum for my operations, but OpenCV misses the capability of taking
conjugates and division of spectrums, represented in this way (while it still can multiply them, even with conjugates with
cv2.mulSpectrums). Should we add this?
[GSoC iOS: Eduard and Charu]
Eduard is working on the planar object detection demo for iPhone. The main problem now is very low performance.
We will have to profile the code and see if it can be brought to realtime somehow.
Charu added several new effects to the "image effects" demo. See the attachment.
[Bug fixing: Maria]
Fixed bugs:
- the bilateral filter for uniform image of CV_32F type,
- Lsh index constructor in the Flann.
Supported users in the answers.opencv.org. It results to several improvements
#2181 (automatically keeping image types the same),
#2182 (allowing the norm to be passed in),
#2184 (better tests for FLANN)
#2185 (better doc of cascade detection)
and a bug #2183 (Add a norm type support to the FlannBasedMatche)
[User contributions]
Grigory Frolov, intern from Intel Nizhniy Novgorod, contributed a few SSE2/SSE4.2-optimized functions for OpenCV.
It has been integrated to OpenCV trunk, sorry, I meant, to the master.
Wongun Choi, PhD student working with Caroline, contributed the patch for calculating confidence measures from HOG
people detector (the patch is for GPU-optimized version). The patch is now analyzed by the GPU team.
[Papers from CVPR 2012: Vadim]
Studied a few interesting papers from CVPR'2012 that, we believe, may be worth implementing, since they all introduce
some techniques that can be useful beyonds the algorithms, where the techniques are used. The slides that I presented
last Friday have been shared separately. For the reference, the papers discussed are:
FREAK: Fast Retina Keypoint (already in OpenCV)
Real-time scene text localization and recognition.
A non-local cost aggregation method for stereo matching
Three things everyone should know to improve object retrieval

Anatoly

Accomplishments & statuses:
1) LBP Face detection
· Implemented the idea with fixed grid size – speed-up is the same(~3x). it seems that we are compute
bound (IPC is near 2.0, and control flow divergence ~33%).
· Integrated the functionality in OpenCV CascadeClassifier_GPU class. Added support of async. invocations
for LBP only (It's possible because LBP is implemented fully on GPU).
· Investigated approach in the paper which Shalini send us. For now we don’t see how we can apply
feature-centric approach to optimize our code.
3) OpenCV GPU
· Background subtraction: no update
· Non-local means denoising: no update.
Plans:
· Try another optimization idea for ‘cv::gpu::resize’ with INTERP_AREA flag for large scales (after LBP work
has been finished). (Marina)
· Implementing non-local means denoising on GPU (Anatoly)
· Start working on Connected Components labeling on GPU (Marina)

Ilya

RGBD Odometry [Maria]
Studied the chosen paper (Highly Accurate 3D Surface Models by Sparse Surface Adjustment by
M. Ruhnke, R. Kümmerle, G. Grisetti and W. Burgard) for global pose and point clouds refinement,
ready to start implementation.
Played with the OpenCV bilateral filter on a depth image. It improved results of ICP, but the quality
of RGBD and RGBD+ICP remained the same. Moreover ICP gives the smallest translation error on TUM datasets
(they contain rich 3D structures), but it still has the largest rotation error. So I tried a new scheme of
RGBD and ICP merging: we run ICP at first and then RGBD but with the pure rotation model.
This merging gives the smallest translation error from ICP and small rotation error after refinement by RGBD.

The filter works ~0.28s on a VGA frame (with similar to KinFu parameters). It's ~4-4.5 times slower than
our RGBD or ICP odometries. We can accelerate it technically by using TBB, SSE or algorithmically by
trying the quasi bilateral filter from the CVPR'12 paper that works O(1) per pixel (thanks to Vadim for
the talk about it on the Itseez seminar).
OpenCV [Maria]
Fixed bugs:
- the bilateral filter for a uniform image of CV_32F type,
- Lsh index constructor in the Flann.
Supported users in the answers.opencv.org. This resulted to several improvements
#2181, #2182, #2184, #2185 and a bug #2183.
Transparent objects [Ilya]
I've finally returned to Russia to my beloved transparent objects.
Prepared and submitted a camera-ready paper "Pose Refinement of Transparent Rigid Objects With a Stereo Camera"
by Ilya Lysenkov and Victor Eruhimov which was accepted to the main conference program at GraphiCon 2012.
We got good reviews with 3 overall ratings: Strong Accept, Accept and Accept.
Re-created infrastructure used at Willow to work with transparent_objects:
updated to fuerte and the latest version of ecto.
Started to experiment with Line2D on transparent objects.
Sometimes it works quite well (see attached success.png) but
sometimes it is distracted by other gradients (failure.png). Quantitative evaluation is on the way.

Action Items

Gary

Contact the book guy to deposit directly to our account

Check the bank account

Contact Google about vendor

Get a list from J-Y of what he wants

Vincent

Ask Roberta about transfer of money if it isn't there (Gary to check the account)

bilateralFilter: replace NaN with anything (max<float> ?) and not 0 should work

Vadim

[GIT, new code.opencv.org server: Evgeny]
- set up backup script for code.opencv.org data;
- configured github and local (thanks to Andrey Senin) mirrors;
github mirror appeared to be very useful feature; in a few days with already have at least 6 pull requests! That is, this technical tool really facilitates contribution process!
[GSoC OpenCV + iOS: Charu, Eduard]
Charu added 4 more effects to image effects iOS demo: cartoonish, focus, contrast/brightness adjustment, pixellation.
Eduard integrated video camera module into HighGUI.
Another project, not related to GSoC, but related to iOS: Timnit Gebru from Stanford ported some of OpenCV performance tests to iOS, so now we can profile OpenCV there. The code is at some early stage and has not be contributed yet.
[Parallelization: Ilya L, Vadim]
Non-template implementation of parallel_for primitive function has been implemented by the new intern Ilya Lavrenov. It opens very good opportunities for us - we can have different backends for the parallelization framework (e.g. GDC on Mac, OpenMP on Linux/Windows etc.) and the backends could be "hidden" in opencv_core. To test the new engine, Ilya optimized bilateralFilter (floating-point flavor of it) and got 2.7 speedup on dual-core Core i5.
[More contributions]
Grigory Frolov, summer intern from Intel, contributed some more SSE optimizations for the following OpenCV functions: invert(), boundingRect(), cvHaarDetectObjects() (i.e. the older C implementation of our cascade detector).
Vincent contributed 5/8 and 7/12 variations of FAST. Integrated via github pull request.

Ilya

RGBD odometry [Maria]
Added support of a transformation type to all odometries in ecto_opencv (rigid body motion, pure rotation, pure translation). It was in RGBDOdometry from OpenCV contrib before, now ecto_opencv also supports this.
Implemented the building of a depth pyramid with the maxDepthDifference threshold (similar to KinFu) and more appropriate processing the invalid depth values. But it gives almost the same accuracy (improvement is less 0.2% in my experiments) and it's slower than used before cv::buildPyramid in ~10 times (because my version has additional checks and does not use SSE). So cv::buildPyramid was remained in the odometries.
Tried to compare our odometries with KinFu ICP on the TUM dataset. It would allow us to estimate what refinement of previous frame by the volume TSDF gives for ICP. Compiled PCL but KinFu falls with a segmentation fault in the same environment where it worked in the beginning of this year. Shared my computer with Anatoly to fix the problem.
Other:
Maria will be on vacation from July 31 for 1-2 weeks.
Transparent objects [Ilya]
Accomplished several experiments to evaluate accuracy of pose estimation with Line2D when working with transparent objects (thanks to Maria Dimashova and Stefano Fabri for useful information about Line2D).
1. Simulated train data vs. real train data
Generated artificial templates for Line2D and compared results when real train data are used. The same train poses for templates are used in both cases. Results (success rate is ratio of cases when the estimated pose is closer than 3cm to the ground truth pose):
--------------------------------
| Train data | Success rate |
|---------------|--------------|
| Real | 64.0% |
| Simulated | 64.9% |
--------------------------------
Success rate is similar but simulated train data provide much more freedom in generating templates for different scales and rotations so simulated train data were used in subsequent experiments.
2. Accuracy of pose estimation with Line2D
Evaluated accuracy of pose estimation with Line2D on a dataset of 5 transparent objects. Line2D is able to locate an object in an image correctly but it is not able to distinguish between close templates robustly so accuracy of pose estimation is low. The following experiment demonstrates this issue:
42000 simulated train templates were generated for each object. Naive sampling is not uniform in SO(3) group so the algorithm from the following paper was used: Generating Uniform Incremental Grids on SO(3) Using the Hopf Fibration by A. Yershova, S. Jain, S. M. LaValle, and J. C. Mitchell, International Journal of Robotics Research, 2009. These generated templates were sorted according to their proximity to poses in the test dataset and they were used for Line2D training in this order. Accuracy of pose estimation with regard to number of used templates is shown in the attached line2d.png (success rate is ratio of cases when the estimated pose is closer than 3cm to the ground truth pose). One can see that if templates corresponding to test poses are used only then accuracy is high but if we start to add other templates then accuracy degrades significantly. Test poses are not known beforehand so accuracy of pose estimation with Line2D will be low in practice. For comparison our algorithm has success rate 0.98 on this dataset.
These results show that Line2D cannot be used for final pose estimation but perhaps it can be used to generate initial hypotheses of poses so additional experiments will be accomplished to check this.
Other:
Prepared and gave a talk about our work on transparent objects to Intel Summer School students.

tracking library: we have MIL, online Boosting how about feature based ? (KLT and so on from libmv)

Eric wants to know more about the wrapper scripts (so that he can help with Java and maybe get the JavaCV guys on board)

stereo code Vadim and I saw at CVPR

Minutes

adding feature tracking from libmv

wishlist of things we would like to get in OpenCV to attract new developers: simple bugs, simple features

Evgeny: bugs

Ilya: does get emails on the build farm

BA for Maria and Vincent: OpenCV needs interactive calibration. We have great automatic tests for calibration. We can use the Matlab toolbox for that too. Sample for calibration with a function that spits back the current calibration and the error.

Khronos: design of pipelines

Eric and Andrey will talk

Vadim

[GIT repository: Evgeny]
- added experimental pre-push hook for OpenCV master branch. It runs several light builders with buildbot that build OpenCV and run tests on Mac, Linux and Windows. After some testing we decided to change the strategy: run the tests _after_ pushes and merge changes from master to the to-be-created branch "stable" nightly if all the 40+ buildbot builders succeeded.
- added iOS and binary-compliance builders.
[GSoC OpenCV + iOS: Charu, Eduard]
Charu missed this week because of rellocation.
Eduard is still working on merging highgui OSX camera capture with the newly added iOS camera capture code.
[GSoC OpenCV + Python: Alexander]
Alexander created augmented reality demos; see the video: http://www.youtube.com/watch?v=pzVbhxx6aog
[Parallelization: Ilya L]
Ilya started the work on threading geometrical image transformation functions: remap, warpAffine, warpPerspective, resize, undistort. First version gives ~2.7-3.3 speedup on Core i5 machine on FullHD images.
[More contributions]
Grigory Frolov repaired SSE code in haar face detector haar.cpp and added AVX branch. The code has not been to OpenCV yet, but it gives ~30% performance boost on some configurations. Vincent contributed documentation for 5/8 and 7/12 variations of FAST.

[GIT repository: Evgeny]
- created "stable" branch of OpenCV repository. The branch "master", to which we all commit, is automatically getting
merged into "stable" once buildbot successfully built OpenCV in 40+ different configurations and the correctness tests
have passed successfully.
- changed the automated testing of user commits. Now the code is put to master anyway, and after that OpenCV is run
through express testing (3 builders vs 40+ in the full testing mode). In the case any of those builders fail,
user gets e-mail notification.
[GSoC OpenCV + iOS: Charu, Eduard]
- Charu added inpainting into the "OpenCV image effects" demo for iOS. From the screenshot below we can see that
without texture generation, which more advanced inpainting algorithms include, the results look a bit unnatural,
but still the demo could be fun to use.

Besides, he, our contributor Philipp and Vadim fixed several bugs in Python wrappers: ##2100, 2242, 2257.
[Parallelization: Ilya Lavrenov]
Ilya added multi-threaded processing to cv::resize; on 2-core Core i5 with hyperthreading we get consistent
speedup over 2x.
[More contributions]
Grigory Frolov, intern from Intel, contributed SSE-optimized bilateral filter. The code has not been integrated yet,
so there no performance numbers yet.
[Vacations]
Evgeny will be on vacation from tomorrow till 6th of September. Maria got back from vacation.

Anatoly

WR

Ilya

Transparent objects [Ilya]
Implemented a script to collect ground-truth poses of occluding transparent objects in semi-automatic way. Labeled
our datasets using this script.
Implemented computation of occlusion percentage that is proportion of a test object silhouette which is occluded
by other transparent objects in a test image. Implemented evaluation of pose estimation accuracy with regard to this
occlusion percentage. See evaluation.png for results (success rate is ratio of cases when the estimated pose is closer
than 2cm to the ground truth pose). These results show that pose estimation is robust to occlusion < 40% but success
rate drops to zero if occlusion > 60%. We need to use orientation of edges to increase robustness of the algorithm.
Started to integrate Fast Directional Chamfer Matching into the Levenberg-Marquardt ICP framework.
Found a big area of papers on edge-based 3D tracking that are directly relevant to our algorithm (we call this
step "pose refinement" but it is essentially the same problem as in tracking). Studied classic and state-of-the-art
works. There are real-time approaches that works with surface edges (Real-time visual tracking of complex structures
by T. Drummond and R. Cipolla, PAMI 2002) and also extensions to silhouette edges (Exploiting Occluding Contours
for Real-Time 3D Tracking: A Unified Approach by Gang Li, Yanghai Tsin, and Yakup Genc, ICCV 2007). It is reasonable
to implement these papers at some point.
Plans:
Finish integration of Fast Directional Chamfer Matching into the Levenberg-Marquardt ICP framework.
Improve results of evaluation with regard to occlusion percentage.
Collect more data.
Other:
Maria was on vacation.

[Optimization: Ilya Lavrenov, Vadim, Grigoriy Frolov]
- Ilya has finished optimization of warpAffine (2x), warpPerspective (2x), remap (3.2x) and resize (up to 12.77x for 2x2
bilinear resize) functions. In parentheses these are speedup factors for fullhd images on Core i5 machine. For 2x
decimation (i.e. the output image area is 25% of the input image area) we now have a special code (C version for now) that
does decimation. With SSE2 intrinsics it should be possible to get another substantial improvement.
- Now OpenCV makes use of native parallel for implementation on Windows (via Concurrency library, a part of MS runtime)
and MacOSX/iOS (via Grand Dispatch Central) when TBB is not available. So far, the new parallel for is used only in a few
functions (image warping and bilateral filter), but we plan to convert other TBB-based parallel for's to the new primitive. In
order to do that, we also added cross-platform synchronization primitives (cv::Mutex class that is actually a spinlock on most
platforms).
- The patch from Grigoriy Frolov, intern from Intel, for SSE3 optimized bilateral filter has been integrated to OpenCV. The
speedup is not huge, around 20%. The code can be further improved.
Meanwhile, Maria has made a small yet important fix in bilateralFilter to make it handle NaN's values more gracefully (which is
important for RGBD odometry)
- We fixed CMake scripts for iOS to tell the compiler to build optimized code. Previously, we apparently passed "-O0" flag to
GCC, resulting in quite slow execution. Now it's "-O3 -ffast-math". Together with GDC-powered parallel_for, we should get
much better performance of our samples.
[GSoC: Charu, Eduard, Alexander, others]
iOS:
- We finally integrated Eduard's classes for video capturing into highgui.
- Image filters and video filters samples are basically ready. The final week is devoted to reports.
Python:
Alexander put all the remaining code in work to our repository. Overall, it was excellent project with about a dozen of new
exciting samples.
Non-local means:
Victor Passichenko, GSoC student supervised by Victor and Leonid Beynenson from NVidia team, has submitted implementation
of non-local means image denoising algorithm. It has been already integrated to opencv.
"Simple flow" dense optical flow algorithm:
Yuri Zemlyanskiy submitted implementation of the state-of-art dense optical flow algorithm. The GSoC project was lead by
Victor.

iOS Port coming to an OpenCV near you soon.

Anatoly

WR

Ilya

Transparent objects [Ilya]:
Integrated Fast Directional Chamfer Matching (FDCM) into the Levenberg-Marquardt Iterative Closest Point framework (LM-ICP)
by computing jacobians using the 3D distance transform (both for silhouette and surface edges). FDCM uses orientations of edges
so it allows to get better results in clutter.
Improved filtering of outliers by considering points with high weight only.
Used estimation of edges orientations from the paper "Estimation of Curvature and Tangent Direction by Median Filtered
Differencing" by J. Matas, Z. Shao, and J. Kittler. It work with similar quality as the corresponding part of the FDCM approach
but it is faster in 3.2x times.
Collected a new dataset with 4-5 transparent objects on each image and labeled it.
See the attached plot for results of the improved algorithm (success rate is ratio of cases when the estimated pose is
closer than 2cm to the ground truth pose).
Model capture [Maria]:
Studied the g2o framework (read the paper and doc, studied its examples).
Read the paper about RGB-D SLAM that uses g2o ("An Evaluation of the RGB-D SLAM System" from ICRA 2012). Their odometry
doesn't have anything in common with RGBD odometry which we implemented. Studied their code (available as open-source)
as a more practical and complicated example of g2o usage than most of the demos from the g2o framework.
Started experiments with the graph of poses from our odometries. To get experience with g2o I built a simple graph with
edges connecting each pose with several previous poses. But this doesn't decrease the absolute translational error on the TUM
benchmark and makes it even larger. Studying the reason of this and g2o more.
OpenCV [Maria]:
Replaced the value that was set instead of NaNs in bilateralFilter_32f by some more appropriate value to get correct results
for depth maps (thanks to Vadim for the hint). This constant will be made the optional parameter of cv::bilateralFilter when we can
break OpenCV binary compatibility in 2.5 version (reported in the issue #2269). Switched to use original cv::bilateralFilter in
ecto_opencv (it's possible with some preprocessing).
Other:
Maria returned from vacation on August 14th.

Pose recognition in clutter of transparent objects.

Action Items

Gary

Talk to Cat at Google

ticket 2.4.3 for auto doc of samples

Vincent

.

Ilya

.

Vadim

Look at trained cascade classifiers to decide on what should be included

OpenCV on Mac: wait for Gary. Gary should write a tutorial about it as it is pretty easy. Vadim could help as he works primarily on Mac.

ECCV: if we cannot send people, the team needs to teach people how to present. Tutorial is on September 7th.

GSOC: everything ended well. Very complete GSOC. What did we learn ? Next year : make sure students can work full time on it (filter out students that do not have much time). We need to keep track of what to integrate.

Still have to put up Blog: wait for Gary

Foundation: wait for Gary

Vadim

[Optimization: Grigoriy Frolov]
- Grigoriy, intern from Intel, has finished the summer practice at Intel. He contributed 2 more patches, for the box filter and Haar face detector with SSE & AVX optimization. The patches will be integrated into OpenCV.
[GSoC: Charu, Eduard, Alexander, others]
iOS:
The final report has been sent. Charu and Eduard wrote tutorials about using OpenCV in iOS apps. Tutorials will be included into offical OpenCV build tree and will be uploaded to docs.opencv.org.
Python:
Alexander wrote the final report/blog entry.
Object Detection Cascades:
Cascades for detecting profile faces and silverware, trained by Attila Novak, who was supervised by Gary, have been integrated into the library. The detectors work quite well (although, they are trained for specific orientation of an object, i.e. profile faces should be directed to the right, silverware items should be oriented vertically).
Simple Flow:
Yuri Zemlyanskiy - GSoC student, supervised by Victor, has submitted simple flow dense optical flow algorithm, which is not a realtime, but reportedly quite accurate algorithm. First review round has been passed and the updated, more efficient implementation, has been submitted. It will be integrated to OpenCV, video module.
[Contributions]
7 patches from users have been integrated.

Anatoly

WR

Ilya

Transparent objects [Ilya]:
Used normals to compute silhouettes instead of morphology. It is faster (overall speed-up of the algorithm is 1.4x) and can be analyzed analytically.
Computed jacobian when normals are used to compute silhouettes. Checked validity of results by comparing with a numerical jacobian. See the attached plot for results of the improved algorithm.
Started to prepare a presentation for the Itseez seminar about our latest algorithm.
Model capture [Maria]:
Continued to study g2o and experiment with it on poses from our odometry implementations. Found and eliminated the reason why my simple graph of 3D poses (with edges connecting each pose with several previous poses) didn't give improvement last week. Some results of the camera poses refinement on TUM benchmark:
frames count translational_error.max
(m, frame-to-frame) absolute_translational_error.rmse
(m, the whole trajectory)
before g2o after g2o
freiburg1_desk 573 0.032586 0.10342 0.052493
freiburg2_desk 2893 0.025508 0.29847 0.107332
freiburg1_360 755 0.057243 0.280819 0.306784
This simple graph can improve accuracy of poses for the video sequences with relatively accurate motion estimations. But it's powerless if there are large errors in odometry. Improvement of the graph structure is in progress, e.g. a loop closure by applying Dijkstra algorithm as it's done in implementation of RGBD SLAM [http://vision.in.tum.de/_media/spezial/bib/endres12icra.pdf]. I'm going to reproduce their implementation graph construction but adapt it for our odometries (their code is available as a package for old ROS electric). This can not solve all problems due to inaccuracy of odometry but it will increase robustness of our algorithm.

Action Items

Gary

Talk to Cat at Google

write a Mac tutorial to set things up (maybe with Vadim)

Vincent

ask for a travel request for ECCV for Vincent and one guy from Itseez

Ilya

.

Vadim

.

Victor

.

Ethan

.

From Last Time

Gary

Talk to Cat at Google

ticket 2.4.3 for auto doc of samples

Vincent

.

Ilya

.

Vadim

Look at trained cascade classifiers to decide on what should be included

website, wiki -- so that users can add information. Right now, install to Mac is pretty bad. How to install with QT? What version with what? etc

ECCV

New code ... docs?

Transparent

Model capture

FABMAP

Mac

GSoC visits

Vendor datatus

Minutes

Wiki for users? Installation on Mac is missing. Installation is getting pretty standard.

We need a user wiki again. Evgeny do do?

Duplicated obsolete information

Developer

ECCV

OpenCV soon to be a vendor -- was a mail slip up.

I will be there

Alexander Shishkov, Kirill from NVidia about Android(?) Might not come.

Need to cost this out and see where we stand

New code Make sure there is documentation and examples

FABMAP -- documentation

Simple flow. Short description and sample code

Model capture

Loop closure -- extend

Transparent

Take advantage of specularities

Use the model fitting that exists

RGBD + pre pose needed.

Vadim

[GSoC: Yury Zemlyanskiy]
Finally, we've integrated simpleflow dense optical flow algorithm into OpenCV. It needed couple of
review rounds; and the latest version looks great (if only quite a bit slower (more than order of
magnitude?), comparing to the near-realtime Farnerback's algorithm). Still, it would be interesting
project to implement dense optical flow benchmark in opencv test system, similar to our features2d
nd stereo correspondence test frameworks.
[Code Contributions]
* OpenFABMAP (http://code.google.com/p/openfabmap/), implementation of FAB-MAP algorithm, has been
integrated into OpenCV. The code has been submitted by the author who agreed to change the license
to BSD. The code includes documentation and sample code.
* a small yet useful extension to PCA algorithm that computes the number of components to retain from
the specified standard deviation has been integrated via github pull request.
* another useful contribution came from NVidia tegra team; OpenEXR source code is put into OpenCV build
tree, wrapped into our CMake build engine, so that highgui is compiled with OpenEXR support by default
on several platforms (Windows, MacOSX, Android).
* parallel version of the retinafilter, a part of OpenCV contrib module that was contributed and is being
maintained by Alexandre Benoit, has been created by Alexandre.
[Testing: Ilya Lavrenov]
More strict tests of image warping functions have been created by Ilya Lavrenov. The tests for resize,
warp affine and warp perspective have been
stabilized, remap test is in progress.

Transparent objects [Ilya]
Finished the presentation about our algorithm and gave a talk about it at the Itseez seminar.
Thanks to all for activity, questions and feedback.
It helped to understand how to present the algorithm and write a paper about it.
Started to write a paper about our latest results.
Model capture [Maria]
Added to the graph construction several steps/features from RGBD SLAM
[http://vision.in.tum.de/_media/spezial/bib/endres12icra.pdf]. Now the graph structure is the following:
- each node is a frame pose,
- 3 types of edges for each node:
1) edges connecting the pose with several previous poses,
2) edges to several keyframes (if there are close such ones)
3) edges to nodes that are found by Dijkstra algorithm (if there are close such ones; excluding edges from 1) )
But results of such more complicated graph are similar to the results of last week. The possible problem of this is
that our odometries are less stable to a large transformation between frames than the method from the paper (I'm not
sure that we have a chance for a loop closure with such sensitivity to the difference between frames). I'm investigating
this and the ways to improve the graph results (e.g. initialization of our odometries by a pose from features2d + 3d-3d).
Other:
Maria took two days off for personal reasons (Aug 30, 31).

Action Items

Gary

Send note to Vadim about creating a user wiki (with log in and password)

[Bug Fixing]
12 tickets in OpenCV bug tracker have been closed. It includes crashes in the core and imgproc modules, in Python bindings, errors
in documentation and build scripts etc.
[Tests]
Ilya Lavrenov has nearly finished work on the new tests for image warping functions: remap, warpAffine, warpPerspective, resize.
The test for resize with area interpolation is still in progress.
Recently, Ilya has optimized those functions, and we plan some further optimization, since they are critical. And for that we need
more strict tests, which Ilya did.

Model capture [Maria]
Read several papers about robust back-end for pose graph SLAM because I did not get improvements from graph
optimization for some TUM datasets last week ago (I suppose the reason is outlier and close to outlier edges). The
interesting paper was "Towards a Robust Back-End for Pose Graph SLAM", ICRA 2012. Conclusion: outlier edges is
very hard problem having a particular importance for a long-time complex trajectory, so we don't plan to solve it
for our task now. At first we will suppose that the odometry does not break a trajectory (uncritical number of outliers)
and we have special kind of trajectory (one turn of/around the table).
Implemented a masker of the train images to find the pixels of the table with the object. The reason is that our
odometries suppose stable scene, so we run them only on masked area. The masker is used Vincent's RgbdPlane
(that copes very well with its task) and pcl convex hull and prism.
Experimented with the last weeks pose graph on some old TOD-like base (turning table with fiducials). Tried all
our odometries on it. The ICP and RgbdICP was not stable for such kind of scene (large plane of table) whereas Rgbd
works well even if there is relatively poor texture on the table (circles grid). The simplest structure of the graph
showed the more accurate results on the base (one edge to the previous frame and loop closure with the first frame).
See example of results in attachment: before and after graph optimization for the cup; only after graph results for
other objects. Notes: 1) circles grids existing in the images are not used, they are only texture; 2) strange pink points
around the table are my hands.
Transparent objects [Ilya]
Finished the first draft of our paper and sent it to Victor for a review.
Cleaned the code used in the paper experiments.

Action Items

Gary

Make some foils and share with Victor

Write Stefano to set dates for Rome

Vincent

Figure out a way of getting the number of clones from github talk also to Vadim

New cascades are in data/objcascades: Profile faces and silverware. Pretty good results for profile. Silverware works on only vertical, so need to rotate images

Foundation

Tune OpenCV main algorithms for MIC, contests

Marketing budget ... motivating use of algorithms/contests, tutorials

grants

GSoC

Ilya is ready, Victor still needs visa ...

Website

Cheatsheet is in code.opencv.org resource section. Needs to be off of docs and opencv.org

Hilton's code, multi-part detector Deva Ramanan's work

Detector in C++, trainer is in Matlab/Octave

need to move this along in separate module & Hilton improve along with thesis. Source and binary compatibility

Transparent

Accurate models ~1mm. But need to improve segmentation. Not independent segmentation, but info from several images

Doesn't depend on symmetry

We'll discuss transparent when Ilya is here

Model capture

RGBD odemetry for poses

Think about ways mesh and texture map

line matching

Chamfer map -- opencv sample/cpp/chamfer finds Apple logo

distance transform

Andrea going to ECCV for tutorial. Need to meet the evening before

Vadim

[Bug Fixing]
over 15 tickets in OpenCV bug tracker have been closed.
[Tests]
Numerous OpenCV test failures (~10) on Windows, Mac, fresh Ubuntu versions have been fixed. They were
mostly caused by the incompatible libjpeg library. The plan is to move to .png files in OpenCV tests to avoid
such problems in the future.
[Optimization]
Evgeny added cross-platform threading to Cascade Detector (which is used for face detection, for example)
and HOG object detector.

Anatoly

WR

Ilya

Model capture of transparent objects [Ilya]
Improved the model capture of unpainted transparent objects. Developed a more robust algorithm
to remove artifacts on top of a model and implemented a 2-step coarse-to-fine approach to produce
models.
Collected a new dataset with 10 transparent objects to experiment with model capture.
Evaluated the algorithm using an object which has a shape of a truncated cone (see attachments
for an example of a train image and the computed model). This shape allows to produce the ground
truth model analytically and compare it with the created model. Measure of quality is partial directed
Hausdorff distance that is a percentile of distances between a point of the first model and the nearest
point of the second model. Results:
------------------------------------------------------------------
| Partial directed Hausdorff distance (cm) |
Percentile |----------------------------------------------------|
| model --> ground truth | ground truth --> model |
-----------------------------------------------------------------|
1.0 | 0.276 | 0.300 |
0.8 | 0.101 | 0.102 |
0.6 | 0.068 | 0.075 |
0.4 | 0.041 | 0.055 |
0.2 | 0.017 | 0.038 |
------------------------------------------------------------------
These results show that accuracy of the created model of this object is 3mm in the worst case and
1mm for 80% of points. However, the algorithm is not robust with other objects yet and can produce s
ignificant artifacts if segmentation of an object is not stable.
Two assumptions are made when reconstructing objects:
1. Object stands on a support plane.
2. Object is hollow and has open top (the fundamental limitation of the approach is inability to r
econstruct concavities).
Other:
Arranged travels to the GraphiCon conference to present our work on transparent objects and to the
USA to attend the GSoC summit and make a new demo on the PR2 robot at Willow Garage.
Model capture of opaque objects [Maria]
Complemented the graph structure to refine locations of object points too. We model each object point using
its 3D coordinates and minimize its distances to corresponding points from several frames. The current approach
gives slightly more accurate models than the models from the graph for the camera poses only. It seems the results
can be better if we consider the local structure of each model point too (e.g. as it's done by Ruhnke).
See examples of reconstructed models in the attachments (Note: circles grids in the training frames are not used
for the odometry).

Action Items

Gary

Contact Audience

Alexander from Itseez, Andrea for Tegra coordinate with Vincent for loading USB keys

parallelization, desktop Java, OpenCL (Chinese Academy of Science ... but really plan this for 2.5, make transparent, if libraries are available, it will be used automagically).

Vadim

[OpenCV 2.4.3rc preparations]
12 more tickets have been closed; over 2 dozens of pull requests have been integrated
(most of the integration work done by Tegra team).
There are just a few things to do before 2.4.3rc is ready:
1. make our buildbot "green". This is what we are currently working on. Most of the
problems are actually "false alarms", but investigation of fixing some other problems
really unveiled problems in OpenCV OR the environment. For example, we found that
GCC 4.6 in Ubuntu 12.04 x86 is pretty buggy; when OpenCV is built with "-O3" option,
many tests fail on Ubuntu 12.04 x86 (and pass successfully on any other configuration).
2. make the actual packages for Unix/Mac, Windows, Android, iOS; test them; upload them to SF.
3. write the release notes (that's my TODO); post the announcement.
[Automatic testing of pull requests: Evgeny]
Evgeny wrote a script (it's beta for now and available only within Itseez) that allows
github.com/itseez/opencv admins to test each pull request using buildbot. In other words,
the patch from the specified pull request is applied to the specified branch, and this code is
compiled and tested in 3 configurations (Ubuntu, Windows, Mac). The next step is to make
buildbot to test each new pull request automatically and post specially formatted comment
to github. This way the process of pull request integration should be simplified a lot.

Anatoly

WR

Ilya

Model capture demo [Ilya]
Finished preparations to the Willow visit (got instructions from Maria how to use her model
capture code, collected with Maria a new dataset with a moving Kinect on a box to simulate PR2).
Attended the Google Summer of Code Mentor Summit.
Installed Maria's model capture on a Willow machine. Collected a dataset for model capture
from a PR2 robot. However, the robot was controlled manually using RViz and it resulted in
too jagged movement. Plan to use the joystick for robot control and re-collect the data.
Model capture [Maria]
Captured with Ilya a dataset of a new type: rotate a Kinect around the table with an object
(our previous datasets had a fixed Kinect and the rotated table). Current model capture code
coped with such case of a moving camera too (odometry gave good poses and this is the base
step to get accurate models). So we hope that this'll work on the robot. See the result in the
attachment (note: circle grids are not used).
Another iteration of the code refactoring was done.
Faced with the following problem on the new dataset: on stage of simultaneous refinement
of camera poses and model points we got several NaNs in Hessian of camera poses during second
iteration. I checked that we don't pass NaN values to the graph and other hypotheses about the
reasons of NaNs, but without success yet. Unlike all our previous datasets (~25) where we didn't
have this problem, the new one has bigger object, as result the graph has ~5 times more vertices
(~0.5M). Added the check of iterations with bad results to apply only success refinement.
Continue to investigate the reason of NaNs.
OpenCV [Maria]
Added 3 new tests on core module: cv::merge, cv::split, cv::phase.

[OpenCV 2.4.3rc]
The release candidate has been released. The changelog is here: http://code.opencv.org/projects/opencv/wiki/ChangeLog#243-rc
The final 2.4.3 is expected on Nov 2nd. Big thanks to Tegra team who helped with the release (as they do regularly during this year)!
[Development process]
Evgeny continues work on the technical aspects of the new workflow. It will be a page (it's now in alpha version) that will allow
developers to see the status of their patches. Perhaps, commits from regular experienced contributors will not require the use o
f too heavy pull request mechanism.

Anatoly

WR

Ilya

Model capture demo [Ilya]
Wrote a demo application for a robot which grabs data from a RGBD sensor and constructs a model using Maria's algorithm.
Visualized odometry poses in rviz. A robot path is tracked with Maria's odometry and it is visualized in rviz simultaneously (see a screenshot).
Experimented with model capture on different robots and objects. Results are good and quite robust (attached several examples).
Thanks to Maria for help with the demo.
Other:
Collected several datasets of transparent objects at IPI.
Model capture [Maria]
Experimented on the dataset from a robot to understand the differences with our datasets that can be considered in the algorithms.
Implemented online frame-to-frame processing.
Implemented refinement of no-keyframes poses within segments of already globally refined trajectory of keyframes.
The goal is to fill missing and invisible information in keyframes set and to get more accurate filter of outliers in the model
point cloud. First result of no-keyframe poses refinement is unsatisfactory. Investigating the reason of this now.

Victor is scheduling his visit week of 12. Khronos visit.!! Will coordinate time and G meeting.

Vadim

[OpenCV 2.4.3]
The final OpenCV 2.4.3 is out. The announcement is put to http://opencv.org. After the release candidate we have
fixed in total ~10 bugs together with the Tegra team (mostly various compile problems) and made our buildbot
(build.opencv.org) green. Well, sometimes it gets orange or red because of some internal instabilities, but in general
we fixed most of the test failures by 2.4.3. Currently, we are planning 2.5; the draft version of the proposal should be
ready this week.

Anatoly

WR

Ilya

Demo of model capture
Collected a new dataset from a PR2 robot for subsequent experiments with model capture at Itseez. The dataset contains
33 bag files; the robot makes one loop around a table in each bag file.
Conducted more experiments with the model capture. Attached several examples of new results (the bear was scanned
while lying on its back). However, these models don't have points for regions which a robot didn't see. For example Mini-mouse's
chin is missed.
Vincent and me finished the preparation of the demo and showed it at Willow Garage. RGBD odometry, model capture and
LINE-MOD were demonstrated as Vincent already wrote. The demo went well in overall, however, tracking was not robust and
loop wasn't closed because depth was corrupted by skylight as we checked after the demo. So we had to use a precomputed
model which we created in completely the same setup as the demo but at the night before. Still we showed several good
models produced with a robot so people could see and appreciate results of the algorithm. Also other parts of the demo
went without such problems.
Thanks a lot to Vincent and Maria for crucial help with the demo.
Model capture of opaque objects
Found the reason why my last week refinement of no-keyframe poses gave bad results. Checked several approaches to
do this refinement. The final version is rebasing no-keyframes within globally refined keyframes and optimization of camera
poses graph for each keyframes segment of the trajectory. This approach is very fast and gives accurate results (see the attachment:
green point clouds are all globally refined keyframes, other color point clouds are all no-keyframes). The other checked versions
were more complicated (with ICP and RGBD edges), but did not show improvement and they worked unacceptably long time
(even by segments).
Implementation of view angle histograms of model points (for more stable outliers filtering; using all frames) is in progress.
Other:
Monday was a national holiday in Russia.
An extended abstract about our work on transparent objects was written for a Summer School on "Robot Vision and Applications".

Android port works perfect for Vincent on first try. Kudos to the Android team!

OpenCV Manager will select what's needed, pull in the correct version na

The new descriptor is performing extremely well, beating all prior

logpolar trick for working with scales

Maybe we need to get 2.4.3.1 fix for 3x3 inversion for signal precision breaks down with illcondition. This was turned on by SSE optimization but has to be taken out.

Vadim

[post OpenCV 2.4.3: Bug fixes]
Several bugs, found in 2.4.3 have been fixed: bad accuracy in 3x3 matrix inversion, broken update method in
LBPH face detector, data races in parallel video reader/writer, broken build on Windows with OpenGL enabled ...
[Infrastructure]
Evgeny continues to improve the automated OpenCV build and testing infrastructure:
- integrated developer branches testing functionality into web GUI;
- improved the distributed testing of the submitted pull requests.
- fixed several try jobs merging into a single build;
- modfied other build definitions: e.g. no "success" if builds with warnings;
- "merge button" proof of concept prepared. The idea is that the pull requests that passed all the tests and got approved could be
automatically merged into the repository.
[OpenCV 2.5: the Proposal Draft is ready and has been discussed]
Here is the document that I prepared: https://docs.google.com/a/itseez.com/document/d/1L4pCxMXLNyjFvZAbH-xDZghQCKDhbckjBAYFssPHWD8/edit
It has been briefly discussed with several developers from various teams. OpenCL team is also in the process of reviewing it.
[The proposed changes in OpenCV roadmap: 2.4.4 and 2.5.0]
Since there is a lot planned for OpenCV 2.5, it would be very challenging to prepare some alpha release by February. So instead the
suggestion is to:
release 2.4.4 in February
release 2.5.0 in May

Anatoly

WR

Ilya

Ilya is on vacation from Nov 7 till Nov 16.
Maria is on vacation from Nov 7 till Nov 20.
Other:
Ilya won a $850 travel grant to attend "Robot Vision and Applications" Summer School. Thanks to Vincent for writing a recommendation.

Evgeny did cool work so that we can see, via the build bot whether pull requests work before we pull them

Google hates exceptions. Apparently there is no gcc pluggin that handles exceptions that is not GPL (although, note that Linux kernel is GPL).

But GPL free exception support in C++ is coming. Maybe leave for now

Send blog items to Alexander Shishkov

Updated links around the web, need to get someone on this full time

Matlab/Octave interface -- perfect for next Google summer of code

Put comments into 2.5 document prior to publicizing it

Vadim

[Infrastructure]
Some more results from Evgeny who works on OpenCV infrastructure:
- finally set tdr parameters to tegra-win32 buildslave eliminating timeout-related test failures;
- successfully carried out more tests of "merge button";
- refetched commit history at code.opencv.org to avoid any whitspaces in 'Repository' section;
The latest version our the integration page, where the status for each pull request is attached.
[Edge-aware demosaicing]
Ilya Lavrenov has implemented edge-aware demosaicing algorithm. It has been threaded with nearly
linear speedup on dual-core CPUs. The regression tests are there too.

* New system to see if pull requests work via the build bot before pulling

Anatoly

WR

Ilya

Maria will return from vacation on Nov 21.
Ilya returned from vacation on Nov 20. Started to write an extended version of our GraphiCon'2012
paper for post-proceedings which will be published in Springer Lecture Notes in Computer Science.
The new paper will contain more details about our previous work on transparent objects using a stereo
camera.

Sorry. Meeting pre-empted by OpenNI discussion and by VIP visit at Industrial Perception.

2.5 Module division proposal below

Vadim

[Infrastructure: Evgeny]
Evgeny has practically finished the work on the pull request testing page:
- "merge button" finally can be used, several pull requests were intgerated;
- fixed merge button behavior in case of any git command failure;
- published pull requests testing page at http://pullrequest.opencv.org, so it is now visible for everyone
(thanks to Alexander Tsyplenkov and Alexander Shishkov)
[Demosaicing: Ilya Lavrenov]
Ilya improved implementation of edge-aware demosaicing algorithm, now the performance should be much better.
bayer2RGBA has been parallelized. The parallel version works fine on Mac/Win and now is being debugged on Linux.
[Bug fixing, Misc]
several pull requests from users are at the negotiation stage (improved capturing from Prosillica cameras, all-in-one
connected component extraction algorithm, ...); documentation build problems have been fixed; 3x3 matrix inversion has been fixed
[OpenCV 2.5 discussion]
The rough new subdivision of OpenCV has been proposed:
https://docs.google.com/a/itseez.com/spreadsheet/ccc?key=0AoYFDB08hRs6dHFzMzRtVFppdXFtMzQzQ1JxRmZXd1E#gid=0
it will likely be refined.
[Infrastructure: Build server]
it appears that we may need the build server, but not as urgently as we initially thought, we will try to optimize the amount of
work that the current buildbot does. Terribly sorry for alarming you. We will get back to this question next year.

Anatoly

WR

Ilya

Transparent objects [Ilya]
Finished the paper on transparent objects for the GraphiCon'2012 post-proceedings. Wrote about
our algorithm of edge model creation inspired by bio-informatics and added an example of working
with textureless objects.
Prepared a poster about transparent objects for the IEEE RAS Summer School on
"Robot Vision and Applications".
Sifted through ECCV 2012 papers. Found one paper on the same problem that I was struggling
for a month in model capture of transparent objects: "N-tuple Color Segmentation for Multi-View
Silhouette Extraction" by Abdelaziz Djelouah et al. Unfortunately, It cannot be applied directly
because they wrote "Failure cases have been observed with configurations where the background and
foreground color distributions are similar". Going to study it and linked papers more thoroughly
to check for ideas that can be useful for transparent objects too.
Other:
Gave a talk on Projective Geometry in Computer Vision at a student school.
Model capture [Maria]
Returned from vacation on Nov 21. Refactoring the code and reading more papers about model
capture (stable camera tracking, in-had scanning, etc). Both are not finished yet.

2.5 have early prototype. But 2.5 is intended to be stable for several years

By next week get shell to compile

Pospone book for this. API agreed by API

Set of modules have been changed

Some functionality moved to different modules

C API moved to (new) compatibility module

Legacy module for older stuff

Style of using class instances, you use smart pointer

Like NVidia capture method

Look at code samples and start changing

Could use a maintainer

The maintainer's OpenCV time is spent on three activities.
- Communication (60%)
Mailing list discussions on general design, fielding user
questions, diagnosing bug reports; reviewing, commenting on,
suggesting alternatives to, and rejecting patches.
- Integration (30%)
Applying new patches from the contributors while spotting and
correcting minor mistakes, shuffling the integration and
testing branches, pushing the results out, cutting the
releases, and making announcements.
- Own development (10%)
Scratching my own itch and sending proposed patch series out.

Vadim

[OpenCV 2.4.3.1]
Very minor update to 2.4.3 has been prepared (by Tegra team) and is available from
https://github.com/Itseez/opencv/tags. It's so minor that we will not make any public announce.
Most of the fixes relate to Android.
[Infrastructure: Evgeny]
Some further improvements have been done to the pull request testing page (the "external"
variant of the page is available at http://pullrequest.opencv.org):
- "merge" button gets different colors, depending on the comments and build status
(e.g. when the last comment is not :+1: or :shipit:, the button gets orange rather than green)
- the last comment is printed at the status page
- the behavior is now more robust: we set time limit for each step of precommit build,
thus invalid code does not break the buildbot.
[Demosaicing: Ilya Lavrenov]
Ilya fixed the remaining bugs in the parallel edge-aware demosaicing and added parallel
version of bayer2Gray.
[OpenCV 2.5 concept]
I have started preparing the concept version of OpenCV 2.5, which is now at
https://github.com/vpisarev/opencv/tree/2.5_concept (completely broken for now).
The compile problems will likely be fixed by the end of the week; the tests will take
some more and then we will continue to refine it together.

[Infrastructure: Evgeny]
- We significantly reduced the load of our buildbot, in order to make it more responsive and free some resources for more
extensive testing. So far, the number of builders have been reduced from 50+ to ~30.
- All the invocations of the buildbot builders are now recorded in a DB, so we have more robust and much more
responsive http://pullrequest.opencv.org.
- A separate project, called meta-opencv, is now created at code.opencv.org. We will use it as a place to work on
OpenCV infrastructure. In particular, we put there our buildbot scripts.
[Demosaicing: Ilya Lavrenov]
Ilya started studying SSE2 intrinsics and as a first assignment he optimized the edge-aware demosaicing algorithm.
The SSE2-optimized version is ~4x faster than C version. Now he is working on optimizing some cases of the resize function.
[OpenCV 2.5 concept]
Vadim continues to work on the OpenCV 2.5 concept at https://github.com/vpisarev/opencv/tree/2.5_concept. So far the
modules hal (it's almost empty for now), core (except for XML&YAML I/O), imgproc (except for contours and few more functions),
bgfg (100%), optflow (50%) are ready. It can take some more weeks to make it completely ready. Given the coming holidatys,
I expect the prototype to be ready and pass some tests to be ready by the early January.
[Plans before NY]
Evgeny plans to switch to non-intensive maintainance mode of the OpenCV infrastructure and spend more time on the actual
code work, e.g. Matas approach for fast connected component analysis.
Vadim plans to take vacation from the second half of December.

Anatoly

WR

Ilya

Model capture of opaque objects [Maria]
Finished next step of the code refactoring.
Implemented missing functionality: getting a model, reading from / writing to a ply file.
Prepared and gave a talk about model capture at Itseez.
Implemented refinement of an object mask after table segmentation. Sometimes outliers in this mask were
a reason of dangling points in the final model.
Implemented the old idea to compute histograms of view angles for each point using all frames (including
refined no-keyframes) and filter the points basing on such histograms. But it appeared that this filter is not
more accurate on our data than current simple filter by correspondences count.
Made simple experiment to improve a model texture: implemented texturing version that set for 3d point a
color that is weighted sum of its projections color on several frames (the weight depends on point normal).
Again this does not give better texture visually.
Plans:
Read more SLAM papers to go to scanning with arbitrary camera trajectory.
Model capture of transparent objects [Ilya]
Studied papers "N-tuple Color Segmentation for Multi-View Silhouette Extraction" and "Automatic 3D object
segmentation in multiple views using volumetric graph-cuts" in more details. The last paper is very similar to
our current approach. However, both papers don't solve our critical issues and shouldn't work well in our case
when background and foreground are extremely similar.
Revised our problem of 3D reconstruction and formulated it more rigorously. It is energy minimization on a
factor graph. The issue is the graph has loops and factors have order of 10-200 terms (it depends on the
algorithm parameters). I don't know how to find the global minimum for such a graph. Going to try to devise
an adhoc global optimization scheme by exploiting the specific structure of the graph, otherwise use
approximate inference or re-formulate the problem.
Other:
Ilya was on vacation and he attended IEEE RAS Summer School on "Robot Vision and Applications" in
Santiago, Chile (December 3-7).

[New functionality]
- Evgeny is now working on implementing "Real-time scene text localization and
recognition" by L. Neumann and J. Matas. The draft non-optimized implementation
of the first most expensive preprocessing stage is ready; the next steps will be:
extensive optimization of the first phase, adding classification and the second stage.
- Jason Newton contributed fast algorithm for connected component extraction, which is
about to be integrated to the master branch (once we eliminate test failure on MacOSX).
[Optimization]
- Ilya implemented SSE2 version of resize area fast. Here are the execution times
in ms to process FullHD image on Core i5 2.5 GHz:
Resize Area Fast CV_8UC1 CV_8UC3 CV_16UC1 CV_16UC4
C version 502 1292 380 1982
SSE2 version 112 510 189 1218
Speed up 4.5 2.5 2 1.6
Now Ilya is studying HOG algorithm and will accelerate it further.
[OpenCV 2.5 concept]
Vadim did some more work on OpenCV 2.5 concept. Some more functions in imgproc have
been rewritten in C++, optflow module now compiles; the work on geometry module has been started.

Anatoly

WR

Vincent

on my TODO:
- not done yet with the foundation/donation page
- talk with YYYY in progress. XXXXX too
- we are now part of coverity ! My intentions were twofold:
- find new bugs in OpenCV: coverity is a static code analysis (the best out there) and they provide free analysis for Open Source software:
http://scan.coverity.com/all-projects.html
(we should be on there soon)
We did find some bugs (e.g. some memory leaks) but only under certain conditions or in C code/rare API.
Still, they should be fixed at some point. I sent a list to Vadim about the leaks.
- being on coverity is also a way to show that we care about industrial quality. Linux, Python, OpenOffice, PHP ... are on coverity.
1100 industrial companies pay for it : Adobe, ARM, EA games, EADS, LG, Lenovo ....
I still need to setup automatic uploads of a build once a week. Should be trivial, I just need internet ....
- Switchover on funding for the new year is on track

Ilya

Transparent objects [Ilya]
Discussed with Victor our problem of energy minimization on a factor graph
in reconstruction of transparent objects. Decided to not use the graphic models
framework because we have high-order factors in our graph and it is a too difficult
problem for standard algorithms. Started to think about an alternative approach.
Supported Willow Garage in usage of our transparent objects pipeline. Experimented
with training and detection on their data. Training on unpainted transparent objects
didn't work because it is still in development so a simple semi-automatic training was
implemented as a workaround. Detection worked ok, video with results was prepared and sent out.
Plans:
Develop reconstruction of unpainted transparent objects which will be less generic but
more robust (it will use the rotational symmetry assumption).
Model capture [Maria]
Read a lot of papers about SLAM to replace current one loop closure by an arbitrary graph structure
from a handheld sensor. Still there is no final decision which algorithm we should implement but we want
to have more robust odometry, false poses filter and a loop detector.
Switched to use Xtion Pro Live instead of Kinect because for the current odometry we need high FPS of
synchronized rgb and depth from a handheld sensor. Implemented a sample to capture the data from Xtion in
the base format we use for the model capture now.
Captured several datasets from the handheld device. The capture conditions of the first object were close
to ones we satisfied before (on a robot): one loop, relatively slow camera movement. But this time there was
camera jiggle from me and movements of my hands. The object was reconstructed successfully (see the attachment).
The capture conditions of other datasets were more extreme (fast movement, etc.). In these cases sometimes
trajectory was broken. Analyzing the problems of the odometry and how we can increase its robustness in more
extreme capturing.