Miłosz Orzeł - OpenCV.net, js, html, arduino, java... no rants or clickbaits.http://en.morzel.net/
http://www.rssboard.org/rss-specificationBlogEngine.NET 3.3.0.0en-UShttp://en.morzel.net/opml.axdhttp://www.dotnetblogengine.net/syndication.axdMiłosz OrzełMiłosz Orzeł0.0000000.000000Detecting a Drone - OpenCV in .NET for Beginners (Emgu CV 3.2, Visual Studio 2017). Part 3<h2>OVERVIEW</h2>
<p><a href="http://en.morzel.net/post/detecting-a-drone-opencv-in-dotnet-for-beginners-emgu-cv-3-visual-studio-2017-part-1" target="_blank">Part 1</a> introduced you to <a href="http://opencv.org/" target="_blank">OpenCV</a> and&nbsp;its <a href="http://www.emgu.com/" target="_blank">Emgu CV</a> wrapper library plus showed the easiest way to create Emgu project in Visual Studio 2017. <a href="http://en.morzel.net/post/detecting-a-drone-opencv-in-dotnet-for-beginners-emgu-cv-3-visual-studio-2017-part-2" target="_blank">Part 2</a> was all about grabbing frames from <a href="http://morzel.net/download/emgu_cv_drone_test_video.mp4" target="_blank">video file</a>. The third (and last) episode focuses on <strong>image transformations and contour detection</strong>...</p>
<p>If case you forgot:&nbsp;here's the complete <a href="https://github.com/morzel85/blog-post-emgucv" target="_blank">code sample on GitHub</a>&nbsp;(focus on&nbsp;<a href="https://github.com/morzel85/blog-post-emgucv/blob/master/EmguApp/EmguApp/Program.cs" target="_blank">Program.cs</a>&nbsp;file as it contains all image acquisition and processing code). This&nbsp;is&nbsp;the app in action:<iframe style="display: block; margin: 10px auto;" src="https://www.youtube.com/embed/XSoYPFcOwss" width="560" height="315" frameborder="0" allowfullscreen="allowfullscreen"></iframe></p>
<p>&nbsp;</p>
<h2>STEP 4:&nbsp;Difference detection and noise removal</h2>
<p>In previous post you've seen that&nbsp;<em>VideoProcessingLoop</em> method invoked <em>ProcessFrame</em> for each frame grabbed from video, here's the method:</p>
<pre class="brush:csharp;auto-links:false;toolbar:false" contenteditable="false">// Determines boundary of brightness while turning grayscale image to binary (black-white) image
private const int Threshold = 5;
// Erosion to remove noise (reduce white pixel zones)
private const int ErodeIterations = 3;
// Dilation to enhance erosion survivors (enlarge white pixel zones)
private const int DilateIterations = 3;
// ...
// Containers for images demonstrating different phases of frame processing
private static Mat rawFrame = new Mat(); // Frame as obtained from video
private static Mat backgroundFrame = new Mat(); // Frame used as base for change detection
private static Mat diffFrame = new Mat(); // Image showing differences between background and raw frame
private static Mat grayscaleDiffFrame = new Mat(); // Image showing differences in 8-bit color depth
private static Mat binaryDiffFrame = new Mat(); // Image showing changed areas in white and unchanged in black
private static Mat denoisedDiffFrame = new Mat(); // Image with irrelevant changes removed with opening operation
private static Mat finalFrame = new Mat(); // Video frame with detected object marked
// ...
private static void ProcessFrame(Mat backgroundFrame, int threshold, int erodeIterations, int dilateIterations)
{
// Find difference between background (first) frame and current frame
CvInvoke.AbsDiff(backgroundFrame, rawFrame, diffFrame);
// Apply binary threshold to grayscale image (white pixel will mark difference)
CvInvoke.CvtColor(diffFrame, grayscaleDiffFrame, ColorConversion.Bgr2Gray);
CvInvoke.Threshold(grayscaleDiffFrame, binaryDiffFrame, threshold, 255, ThresholdType.Binary);
// Remove noise with opening operation (erosion followed by dilation)
CvInvoke.Erode(binaryDiffFrame, denoisedDiffFrame, null, new Point(-1, -1), erodeIterations, BorderType.Default, new MCvScalar(1));
CvInvoke.Dilate(denoisedDiffFrame, denoisedDiffFrame, null, new Point(-1, -1), dilateIterations, BorderType.Default, new MCvScalar(1));
rawFrame.CopyTo(finalFrame);
DetectObject(denoisedDiffFrame, finalFrame);
}</pre>
<h3><strong>AbsDiff and CvtColor</strong></h3>
<p>First the current frame (kept in <em>rawFrame)</em>&nbsp;is compared to background with <em><a href="http://www.emgu.com/wiki/files/3.2.0/document/html/eb63e54b-e902-973b-588c-0b7e37a78a2f.htm" target="_blank">CvInvoke.AbsDiff</a>. </em>In other words: current frame is subtracted from background frame pixel by pixel (absolute value is used to avoid negative results). After that the difference image is converted into grayscale with <a href="http://www.emgu.com/wiki/files/3.2.0/document/html/1aa8db33-6057-9fd4-9a95-8ddd6bf6cfcf.htm" target="_blank"><em>CvInvoke.CvtColor</em> </a>call. We care only about overall pixel difference (not it's individual blue-green-red color components). The whiter the pixel is the more it color has changed... Take a look at below picture showing background&nbsp;frame, current frame and the grayscale difference:</p>
<p><a title="Frames difference... Click to enlarge..." href="http://en.morzel.net/pics/PostImg/emguapp_bg_raw_diff.png" target="_blank"><img style="vertical-align: middle; display: block; margin-left: auto; margin-right: auto;" src="/pics/PostImg/emguapp_bg_raw_diff_small.png" alt="Frames difference... Click to enlarge..." /></a></p>
<p>&nbsp;</p>
<h3><strong>Threshold</strong></h3>
<p>Grayscale image is changed into and image with only black and white (binary) pixels&nbsp;with the use of <em><a href="http://www.emgu.com/wiki/files/3.2.0/document/html/54f2f6fb-b6dc-b974-16f4-f6b4bbb578d8.htm" target="_blank">CvInvoke.Threshold</a></em>. Our intention is to mark changed pixels as white. Threshold value allows as to control change detection sensitivity. Below you can see how different thresholds produce various binary results:</p>
<p><a title="Threshold difference... Click to enlarge..." href="http://en.morzel.net/pics/PostImg/emguapp_th.png" target="_blank"><img style="vertical-align: middle; display: block; margin-left: auto; margin-right: auto;" src="/pics/PostImg/emguapp_th_small.png" alt="Threshold difference... Click to enlarge..." /></a></p>
<p>First image (left) was produced with Threshold = 1, so even the slightest change got marked - such image is not a suitable input for contour detection. Image in the middle used Threshold = 5. Drone position is clearly marked and smaller white pixel zones can be removed with erosion... Last image (right) is the result of setting Threshold to = 200.&nbsp;This time sensitivity was too low and we got just a couple of white pixels.&nbsp;</p>
<h3><strong>Erode and Dilate</strong></h3>
<p>It's hard to find a threshold that gives desired difference detection for each video frame. If threshold is too low then too many pixels are white, if it's too high then the drone might not be detected at all... The best is to pick a threshold which marks the change we need even if we&nbsp;get a bit of undesired white spots as these can be safely removed with <strong>erosion followed by dilation</strong>. When combined, these two operations create so called <strong>opening </strong>operation which can be treated as a noise removal method. Opening is a type of <strong>morphology transformation</strong> (<a href="http://docs.opencv.org/3.0-beta/doc/tutorials/imgproc/opening_closing_hats/opening_closing_hats.html" target="_blank">read this article</a> to learn more about them). These operations work by probing pixel neighborhood with a structuring element (aka kernel) and deciding pixel value based on the values found in the neighborhood...</p>
<p><em><a href="http://www.emgu.com/wiki/files/3.2.0/document/html/60ae0cbf-921e-403d-6666-f5039c1a87ca.htm" target="_blank">CvInvoke.Erode</a></em> is meant to simulate a physical process in which object area is reduced due to destructive effects of its surroundings. Detailed behavior&nbsp;depends on the parameters passed (structuring&nbsp;element, anchor, border type - never mind, this is a beginners guide, right?) but the general idea is like this: if pixel is white but has a black pixel around it then it should become black too. The more erode iterations are run the more white pixel zones get eaten away. Here's an example of image erosion:</p>
<p><a title="Erosion... Click to enlarge..." href="http://en.morzel.net/pics/PostImg/emguapp_erode.png" target="_blank"><img style="vertical-align: middle; display: block; margin-left: auto; margin-right: auto;" src="/pics/PostImg/emguapp_erode_small.png" alt="Erosion... Click to enlarge..." /></a></p>
<p>On the left is the input image and on the right we have the result of erosion which used structuring element in the shape of a 3x3 square (this is the default value used when <em>null</em> is passes for <em>element</em> parameter in <em>Erode</em> invocation). The value of output pixel was decided by probing all neighboring pixels and checking for their minimal value. If a pixel was white in the input image but had at least one black pixel in its immediate surroundings then it became black in the output image.</p>
<p>If erosion is used wisely we can get rid of irrelevant&nbsp;white pixels. But there's a catch: the white pixel zone that marks the drone is&nbsp;reduced too. Don't worry: dilation is going to help us! Just like pupils in your eyes are&nbsp;enlarged (dilated) when it gets dark the white pixel zones that survived erosion can get enlarged too... Again details vary by <em><a href="http://www.emgu.com/wiki/files/3.2.0/document/html/25f5c3ea-ed93-fcbd-20b7-b046874f8fbe.htm" target="_blank">CvInvoke.Dilate</a></em>&nbsp;parameters but generally speaking: if pixel is black but has while neighbor&nbsp;it gets white too. The more iterations are run the more white zones are enlarged. This is an example of dilation:</p>
<p><a title="Dilation... Click to enlarge..." href="http://en.morzel.net/pics/PostImg/emguapp_dilate.png" target="_blank"><img style="vertical-align: middle; display: block; margin-left: auto; margin-right: auto;" src="/pics/PostImg/emguapp_dilate_small.png" alt="Dilation... Click to enlarge..." /></a></p>
<p>On the left we have the same input image as used in erosion example and on the right we can see the result of single call to <em>Dilate</em> method (again with 3 by 3 kernel matrix). Notice how pixel obtains the maximal value of its surroundings (if any neighboring pixel is white it becomes white too)...</p>
<p>Erosion followed by dilation&nbsp;is such a common transformation that OpenCV has methods that combine the two into one opening operation&nbsp;but using separate <em>Erode</em> and <em>Dilate</em> calls gives you a bit more control. Below you can see how opening cleared the noise and enhanced white spot that marks drone position:</p>
<p><a title="Opening... Click to enlarge..." href="http://en.morzel.net/pics/PostImg/emguapp_opening.png" target="_blank"><img style="vertical-align: middle; display: block; margin-left: auto; margin-right: auto;" src="/pics/PostImg/emguapp_opening_small.png" alt="Opening... Click to enlarge..." /></a></p>
<p>&nbsp;</p>
<h2>STEP 5: Contour detection</h2>
<p>Once all above image preparation steps&nbsp;are done we have a binary image which is suitable input for contour detection provided by <em>DetectObject</em> method:</p>
<pre class="brush:csharp;auto-links:false;toolbar:false" contenteditable="false">private static void DetectObject(Mat detectionFrame, Mat displayFrame)
{
using (VectorOfVectorOfPoint contours = new VectorOfVectorOfPoint())
{
// Build list of contours
CvInvoke.FindContours(detectionFrame, contours, null, RetrType.List, ChainApproxMethod.ChainApproxSimple);
// Selecting largest contour
if (contours.Size &gt; 0)
{
double maxArea = 0;
int chosen = 0;
for (int i = 0; i &lt; contours.Size; i++)
{
VectorOfPoint contour = contours[i];
double area = CvInvoke.ContourArea(contour);
if (area &gt; maxArea)
{
maxArea = area;
chosen = i;
}
}
// Draw on a frame
MarkDetectedObject(displayFrame, contours[chosen], maxArea);
}
}
}</pre>
<p>The method takes binary difference image (<em>detectionFrame</em>)&nbsp;on which contours will be detected and another <a href="http://www.emgu.com/wiki/files/3.2.0/document/html/2ec33afb-1d2b-cac1-ea60-0b4775e4574c.htm" target="_blank"><em>Mat</em> </a>instance&nbsp;(<em>displayFrame</em>) on which detected object will be marked (it's a copy of unprocessed frame)...</p>
<p><em><a href="http://www.emgu.com/wiki/files/3.2.0/document/html/cb24a129-d9ce-57f3-19ad-0eaa27a77317.htm" target="_blank">CvInvoke.FindContours</a></em> takes the image&nbsp;and runs contour detection algorithm&nbsp;to find&nbsp;boundaries&nbsp;between black (zero) and white (non-zero) pixels on 8-bit single channel image - our <em>Mat</em> instance with the result of <em>AbsDiff-&gt;</em><em>CvtColor-&gt;</em><em>Threshold-&gt;</em><em>Erode-&gt;</em><em>Dilate</em> suites it just fine!&nbsp;</p>
<p>A contour is a&nbsp;<em><a href="http://www.emgu.com/wiki/files/3.2.0/document/html/05a55c58-b440-ca74-439c-f288410e692f.htm" target="_blank">VectorOfPoint</a></em>, because image might have many contours we keep them inside <em><a href="http://www.emgu.com/wiki/files/3.2.0/document/html/b29e1d11-e75c-6bbe-4b3b-7d8e6395a733.htm" target="_blank">VectorOfVectorOfPoint</a></em>. In case many contours get detected we want to pick the largest of them. This is easy thanks to a <em><a href="http://www.emgu.com/wiki/files/3.2.0/document/html/6ef05086-8aa7-3f1b-a27f-42594e776465.htm" target="_blank">CvInvoke.ContourArea</a></em>&nbsp;method...</p>
<p>Read the&nbsp;docs about <a href="http://docs.opencv.org/trunk/d9/d8b/tutorial_py_contours_hierarchy.html" target="_blank">contour hierarchy</a>&nbsp;and <a href="http://docs.opencv.org/3.1.0/d3/dc0/group__imgproc__shape.html#ga4303f45752694956374734a03c54d5ff" target="_blank">approximation methods</a> if you are curious about&nbsp;<em>RetrType.List</em>, <em>ChainApproxMethod.ChainApproxSimple</em> enum values seen in&nbsp;<em>CvInvoke.FindContours</em> call. <a href="http://docs.opencv.org/trunk/dd/d49/tutorial_py_contour_features.html" target="_blank">This</a> is a good read too...</p>
<p>&nbsp;</p>
<h2>STEP 6: Drawing and writing on a frame</h2>
<p>Once we've found the drone (that is we have a contour that marks its position) it would be good to present this information to the user. This is done by <em>MarkDetectedObject&nbsp;</em>method:</p>
<pre class="brush:csharp;auto-links:false;toolbar:false" contenteditable="false">private static void MarkDetectedObject(Mat frame, VectorOfPoint contour, double area)
{
// Getting minimal rectangle which contains the contour
Rectangle box = CvInvoke.BoundingRectangle(contour);
// Drawing contour and box around it
CvInvoke.Polylines(frame, contour, true, drawingColor);
CvInvoke.Rectangle(frame, box, drawingColor);
// Write information next to marked object
Point center = new Point(box.X + box.Width / 2, box.Y + box.Height / 2);
var info = new string[] {
$"Area: {area}",
$"Position: {center.X}, {center.Y}"
};
WriteMultilineText(frame, info, new Point(box.Right + 5, center.Y));
}</pre>
<p>The method uses <em><a href="http://www.emgu.com/wiki/files/3.2.0/document/html/87cf96a0-f4e9-14e0-9d1f-7c8501c84643.htm" target="_blank">CvInvoke.BoundingRectangle</a></em>&nbsp;to find minimal box (<em><a href="https://msdn.microsoft.com/library/system.drawing.rectangle(v=vs.110).aspx" target="_blank">Rectangle</a></em>) that surrounds the entire contour. Box is later drawn with a call to <em><a href="http://www.emgu.com/wiki/files/3.2.0/document/html/d43b7043-cd73-da1a-bec1-d5d78d61eecb.htm" target="_blank">CvInvoke.Rectangle</a>.&nbsp;</em>The contour itself is plotted by <em><a href="http://www.emgu.com/wiki/files/3.2.0/document/html/d53da061-2902-be71-8125-ebefd769fe9f.htm" target="_blank">CvInvoke.Polylines</a></em>&nbsp;method which takes a list of points that describe the line. You can notice that both drawing methods receive <em>drawingColor</em> parameter, it is an instance of <em><a href="http://www.emgu.com/wiki/files/3.2.0/document/html/3b69667b-68aa-01da-7a12-1960b0721dd9.htm" target="_blank">MCvScalar</a>&nbsp;</em>defined this way:</p>
<pre class="brush:csharp;auto-links:false;toolbar:false" contenteditable="false">private static MCvScalar drawingColor = new Bgr(Color.Red).MCvScalar;</pre>
<p><a href="http://www.emgu.com/wiki/files/3.2.0/document/html/fe167025-bd25-405e-7891-af1efecf39b8.htm" target="_blank"><em>Bgr</em>&nbsp;</a>structure constructor can take 3 values that define its individual color components or it can take a&nbsp;<a href="https://msdn.microsoft.com//library/system.drawing.color(v=vs.110).aspx" target="_blank"><em>Color</em> </a>structure like in my example.</p>
<p><strong>Important</strong>: <em>Point</em>, <em>Rectangle</em> and <em>Color</em> structures come from&nbsp;<em>System.Drawing</em> assembly which by default is <strong>not</strong> included in new console application template&nbsp;so you need to add reference to&nbsp;<em>System.Drawing.dll</em> yourself.</p>
<p>Information about detected object location is written by&nbsp;<em>WriteMultilineText</em>&nbsp;helper method (the same method is used to print info about frame number and processing time). This is the code:</p>
<pre class="brush:csharp;auto-links:false;toolbar:false" contenteditable="false">private static void WriteMultilineText(Mat frame, string[] lines, Point origin)
{
for (int i = 0; i &lt; lines.Length; i++)
{
int y = i * 10 + origin.Y; // Moving down on each line
CvInvoke.PutText(frame, lines[i], new Point(origin.X, y), FontFace.HersheyPlain, 0.8, drawingColor);
}
}</pre>
<p>In each invocation&nbsp;of&nbsp;<a href="http://www.emgu.com/wiki/files/3.2.0/document/html/37ada42e-a5c3-caa1-dfa7-a4a64251a059.htm" target="_blank"><em>CvInvoke.PutText</em> </a>method <em>y</em> coordinate of the line is increased so lines are not colliding&nbsp;with each other...</p>
<p>This is how frame captured&nbsp;from video looks like after drawing and&nbsp;writing is applied:</p>
<p><a title="Drone marking... Click to enlarge..." href="http://en.morzel.net/pics/PostImg/emguapp_detect.png" target="_blank"><img style="vertical-align: middle; display: block; margin-left: auto; margin-right: auto;" src="/pics/PostImg/emguapp_detect_small.png" alt="Drone marking... Click to enlarge..." /></a></p>
<p>&nbsp;</p>
<h2>STEP 7: Showing it all</h2>
<p>In <a href="http://en.morzel.net/post/detecting-a-drone-opencv-in-dotnet-for-beginners-emgu-cv-3-visual-studio-2017-part-1" target="_blank">part 1</a> you've seen that <em><a href="http://www.emgu.com/wiki/files/3.2.0/document/html/55c8f7f6-eb1d-6af2-a252-e8669fcdb41c.htm" target="_blank">CvInvoke.Imshow</a></em>&nbsp;method can be used to present a window with an image (<em>Mat</em> instance). Below method is called for every video frame so the user has a chance to see various stages of image processing and the final result:</p>
<pre class="brush:csharp;auto-links:false;toolbar:false" contenteditable="false">private static void ShowWindowsWithImageProcessingStages()
{
CvInvoke.Imshow(RawFrameWindowName, rawFrame);
CvInvoke.Imshow(GrayscaleDiffFrameWindowName, grayscaleDiffFrame);
CvInvoke.Imshow(BinaryDiffFrameWindowName, binaryDiffFrame);
CvInvoke.Imshow(DenoisedDiffFrameWindowName, denoisedDiffFrame);
CvInvoke.Imshow(FinalFrameWindowName, finalFrame);
}</pre>
<p>Displaying intermediate steps is a great debugging aid for any image processing application (I didn't show separate windows for erosion and dilation because only 6 windows of <a href="http://morzel.net/download/emgu_cv_drone_test_video.mp4" target="_blank">my test video</a>&nbsp;fit on full HD screen):</p>
<p><a title="All windows... Click to enlarge..." href="http://en.morzel.net/pics/PostImg/emguapp_all.png" target="_blank"><img style="vertical-align: middle; display: block; margin-left: auto; margin-right: auto;" src="/pics/PostImg/emguapp_all_small.png" alt="All windows... Click to enlarge..." /></a></p>
<p>&nbsp;</p>
<h2>SUMMARY</h2>
<p>This three part series assumed that you were completely new to image processing with OpenCV/Emgu CV. Now you have some idea what these libraries are and&nbsp;how to use them in Visual Studio 2017 project while following a coding approach recommended for version 3 of the libs...</p>
<p>You've learned how to grab frames from video and how&nbsp;to prepare them for contour detection using fundamental image processing operations (difference, color space conversion, thresholding and morphological transformations). You also know how to draw shapes and text on an image.&nbsp;Good job!</p>
<p>Computer vision is a complex yet very interesting topic (its importance&nbsp;is constantly increasing), you've just made a first step in this field - who knows, maybe one day I will ride in autonomous vehicle powered by your software? :)&nbsp;</p>
<p>&nbsp;</p>
<p>&nbsp;</p>http://en.morzel.net/post/detecting-a-drone-opencv-in-dotnet-for-beginners-emgu-cv-3-visual-studio-2017-part-3
http://en.morzel.net/post/detecting-a-drone-opencv-in-dotnet-for-beginners-emgu-cv-3-visual-studio-2017-part-3#commenthttp://en.morzel.net/post.aspx?id=608d226c-590d-44e9-a61b-9f086a477352Mon, 21 Aug 2017 23:40:00 +0200.NET Framework/C#CodeProjectGraphicsOpenCVmorzelhttp://en.morzel.net/pingback.axdhttp://en.morzel.net/post.aspx?id=608d226c-590d-44e9-a61b-9f086a4773528http://en.morzel.net/trackback.axd?id=608d226c-590d-44e9-a61b-9f086a477352http://en.morzel.net/post/detecting-a-drone-opencv-in-dotnet-for-beginners-emgu-cv-3-visual-studio-2017-part-3#commenthttp://en.morzel.net/syndication.axd?post=608d226c-590d-44e9-a61b-9f086a477352Detecting a Drone - OpenCV in .NET for Beginners (Emgu CV 3.2, Visual Studio 2017). Part 2<h2>OVERVIEW</h2>
<p>In <a href="http://en.morzel.net/post/detecting-a-drone-opencv-in-dotnet-for-beginners-emgu-cv-3-visual-studio-2017-part-1" target="_blank">Part 1</a> you have learned what <a href="http://opencv.org/" target="_blank">OpenCV</a> is, what is the role of&nbsp;<a href="http://www.emgu.com/" target="_blank">Emgu CV</a> wrapper and how to create a Visual Studio 2017 C# project that utilizes the two libraries. In this part I will show you how to <strong>loop through frames captured from video file</strong>. Check <a href="http://en.morzel.net/post/detecting-a-drone-opencv-in-dotnet-for-beginners-emgu-cv-3-visual-studio-2017-part-1" target="_blank">the first part</a>&nbsp;to watch&nbsp;<a href="https://www.youtube.com/watch?v=XSoYPFcOwss" target="_blank">demo video</a>&nbsp;and find&nbsp;information about sample <a href="https://github.com/morzel85/blog-post-emgucv" target="_blank">project</a>&nbsp;(all the interesting&nbsp;stuff is&nbsp;inside <a href="https://github.com/morzel85/blog-post-emgucv/blob/master/EmguApp/EmguApp/Program.cs" target="_blank">Program.cs</a>&nbsp; - keep this file opened in separate tab as its fragments will be shown in this post)...</p>
<p>&nbsp;</p>
<h2>STEP 1: Capturing video from file</h2>
<p>Before any processing can happen we need to obtain a frame from video file. This can be easily done by using <em><a href="http://www.emgu.com/wiki/files/3.2.0/document/html/5d3e2a38-d2a8-494c-37e0-387d610a87ab.htm" target="_blank">VideoCapture</a></em>&nbsp;class (many tutorials mention <em>Capture</em> class instead but it is <strong>not</strong> available&nbsp;in recent Emgu versions).</p>
<p>Check the <em>Main</em> method from our sample project:</p>
<pre class="brush:csharp;auto-links:false;toolbar:false" contenteditable="false">private const string BackgroundFrameWindowName = "Background Frame";
// ...
private static Mat backgroundFrame = new Mat(); // Frame used as base for change detection
// ...
static void Main(string[] args)
{
string videoFile = @"PUT A PATH TO VIDEO FILE HERE!";
using (var capture = new VideoCapture(videoFile)) // Loading video from file
{
if (capture.IsOpened)
{
// ...
// Obtaining and showing first frame of loaded video (used as the base for difference detection)
backgroundFrame = capture.QueryFrame();
CvInvoke.Imshow(BackgroundFrameWindowName, backgroundFrame);
// Handling video frames (image processing and contour detection)
VideoProcessingLoop(capture, backgroundFrame);
}
else
{
Console.WriteLine($"Unable to open {videoFile}");
}
}
}</pre>
<p><em>VideoCapture</em> has four&nbsp;constructor versions. The overload we are using takes string parameter that is a<strong> path to video file</strong> or video stream. Other versions allow us to connect to <strong>cameras</strong>. If you design your program right, switching from file input to a webcam might be as easy as changing <em>new VideoCapture</em> call!</p>
<p>Once <em>VideoCapture</em> instance is created we can confirm if opening went fine by accessing <em>IsOpened</em> property (maybe path is wrong or codecs are missing?).</p>
<p><em>VideoCapture</em> offers few ways of acquiring&nbsp;frames but the one I find most convenient is by call to <em><a href="http://www.emgu.com/wiki/files/3.2.0/document/html/d1c9647c-d1d7-7dca-7fe6-465f603bb68a.htm" target="_blank">QueryFrame</a> </em>method. This method returns <em>Mat</em> class instance (you know it already from <a href="http://en.morzel.net/post/detecting-a-drone-opencv-in-dotnet-for-beginners-emgu-cv-3-visual-studio-2017-part-1" target="_blank">part 1</a>) and moves to next frame. If next frame cannot be found then <em>null</em> is returned. We can use this fact to easily loop through video.&nbsp;</p>
<p>&nbsp;</p>
<h2>STEP 2: Loading and presenting background frame</h2>
<p>Our drone detection project is based on finding the&nbsp;difference between background frame and other frames. The assumption is that we can treat the first frame obtained from the video as the background, hence the call to <em>QueryFrame</em> right after creating <em>VideoCapture</em> object:</p>
<pre class="brush:csharp;auto-links:false;toolbar:false" contenteditable="false"> backgroundFrame = capture.QueryFrame();</pre>
<p>After background is loaded we can check how it looks with a call to <em>Imshow</em> method (you know it from&nbsp;<a href="http://en.morzel.net/post/detecting-a-drone-opencv-in-dotnet-for-beginners-emgu-cv-3-visual-studio-2017-part-1" target="_blank">part 1</a>&nbsp;too):</p>
<pre class="brush:csharp;auto-links:false;toolbar:false" contenteditable="false">CvInvoke.Imshow(BackgroundFrameWindowName, backgroundFrame);</pre>
<p>Is finding a (meaningful!) difference in a video always as easy as subtracting frames? No, it isn't. First of all the background might not be static (imagine that drone was flying in front of threes moved by wind or if lighting&nbsp;in a room was changing significantly). The second challenge might come from movements of the camera. Having a fixed background and camera position keeps our drone detection task simple enough for beginner's OpenCV tutorial plus it's not completely unrealistic. Video detection/recognition is often used in fully controlled environment&nbsp;such as part of factory... OpenCV is capable of handling more complex scenarios - you can read about <a href="http://docs.opencv.org/trunk/d1/dc5/tutorial_background_subtraction.html" target="_blank">background subtraction</a> techniques and <a href="http://docs.opencv.org/trunk/d7/d8b/tutorial_py_lucas_kanade.html" target="_blank">optical flow</a>&nbsp;to get a hint...</p>
<p>&nbsp;</p>
<h2>STEP 3: Looping through video frames</h2>
<p>We know that we can use <em>QueryFrame</em> to get single frame image (<em>Mat</em> instance) and progress to next frame and we know that <em>QueryFrame</em> returns <em>null</em> if it can't go any further. Let's use this knowledge to build a method that goes through frames in a loop:</p>
<pre class="brush:csharp;auto-links:false;toolbar:false" contenteditable="false">private static void VideoProcessingLoop(VideoCapture capture, Mat backgroundFrame)
{
var stopwatch = new Stopwatch(); // Used for measuring video processing performance
int frameNumber = 1;
while (true) // Loop video
{
rawFrame = capture.QueryFrame(); // Getting next frame (null is returned if no further frame exists)
if (rawFrame != null)
{
frameNumber++;
stopwatch.Restart();
ProcessFrame(backgroundFrame, Threshold, ErodeIterations, DilateIterations);
stopwatch.Stop();
WriteFrameInfo(stopwatch.ElapsedMilliseconds, frameNumber);
ShowWindowsWithImageProcessingStages();
int key = CvInvoke.WaitKey(0); // Wait indefinitely until key is pressed
// Close program if Esc key was pressed (any other key moves to next frame)
if (key == 27)
Environment.Exit(0);
}
else
{
capture.SetCaptureProperty(CapProp.PosFrames, 0); // Move to first frame
frameNumber = 0;
}
}
}</pre>
<p>In each loop iteration a <strong>frame is grabbed</strong> from video file. It is then passed to <em>ProcessFrame</em> method which does image difference, noise removal, contour detection and drawing (it will be discussed in detail in the next post)... Call to <em>ProcessFrame</em>&nbsp;is surrounded&nbsp;with <em>System.Diagnostics.Stopwatch</em> usage - this way we can measure video processing performance. It took my laptop only about <strong>1.5ms</strong> to fully handle each frame - I've told you OpenCV is fast! :)</p>
<p>If <em>QueryFrame</em>&nbsp;returns <em>null </em>then&nbsp;program moves back to first frame by calling&nbsp;<em>SetCaptureProperty</em> method on <em>VideoCapture</em> instance (video will be processed again).</p>
<p><em>WriteFrameInfo</em> puts a text in the frame's upper-left corner with information about it's&nbsp;number and how long it took to process it.&nbsp;<em>ShowWindowsWithImageProcessingStages</em> ensures that we can see current (raw) frame, background frame, intermediate frames and final frame in separate windows... Both methods will be shown in next post.</p>
<p>The <em>while&nbsp;</em>loop&nbsp;is going to spin forever unless program execution is stopped by <strong>Escape</strong> key being pressed in any of the windows that show frames (<strong>not</strong> the console window!).&nbsp;If <em><strong>0</strong></em> is passed as <em>WaitKey</em> argument then program <strong>waits until some key is pressed</strong>. This let's you look at each frame as long as you want. If you pass other number to <em>WaitKey</em>&nbsp;then the program will wait until key is pressed <strong>or a delay elapses</strong>. You might use it to automatically play video at specified frame rate:</p>
<pre class="brush:csharp;auto-links:false;toolbar:false" contenteditable="false">int fps = (int)capture.GetCaptureProperty(CapProp.Fps);
int key = CvInvoke.WaitKey(1000 / fps); // 40ms delay</pre>
<p><strong>Warning</strong>: One thing you might notice while processing videos is that moving through a file is not always as easy as setting <em>CapProp.PosFrame</em> to desired number. Your experience might vary from format to format. This is because video files are <strong>optimized for playing forward</strong> at natural speed and frames might not be&nbsp;simply kept as sequence of images. Full HD (1920x1080) movie has <strong>over 2 million</strong> pixels in each frame. Now let's say we have an hour of video at 30 FPS -&gt;&nbsp; 3600 * 30 * 2,073,600 = <strong>223,948,800,000</strong>. Independent frame compression is not enough to crush that number! No wonder some people need to dedicate their scientific/sofware careers to video compression...</p>
<p>Ok, enough for now - next part coming soon!</p>
<p><strong>Update: <a href="http://en.morzel.net/post/detecting-a-drone-opencv-in-dotnet-for-beginners-emgu-cv-3-visual-studio-2017-part-3" target="_blank">Part 3</a>&nbsp;is ready!</strong></p>http://en.morzel.net/post/detecting-a-drone-opencv-in-dotnet-for-beginners-emgu-cv-3-visual-studio-2017-part-2
http://en.morzel.net/post/detecting-a-drone-opencv-in-dotnet-for-beginners-emgu-cv-3-visual-studio-2017-part-2#commenthttp://en.morzel.net/post.aspx?id=d0329ea1-dc3d-423f-a3d0-414c78b1612cSun, 6 Aug 2017 09:44:00 +0200.NET Framework/C#CodeProjectGraphicsOpenCVmorzelhttp://en.morzel.net/pingback.axdhttp://en.morzel.net/post.aspx?id=d0329ea1-dc3d-423f-a3d0-414c78b1612c1http://en.morzel.net/trackback.axd?id=d0329ea1-dc3d-423f-a3d0-414c78b1612chttp://en.morzel.net/post/detecting-a-drone-opencv-in-dotnet-for-beginners-emgu-cv-3-visual-studio-2017-part-2#commenthttp://en.morzel.net/syndication.axd?post=d0329ea1-dc3d-423f-a3d0-414c78b1612cDetecting a Drone - OpenCV in .NET for Beginners (Emgu CV 3.2, Visual Studio 2017). Part 1<h2>INTRO</h2>
<p><a href="http://www.emgu.com/" target="_blank">Emgu CV</a>&nbsp;is a .NET wrapper for <a href="http://opencv.org/" target="_blank">OpenCV</a>&nbsp;(Open Source Computer Vision Library) which is a collection of over 2500 algorithms focused on <strong>real-time image processing</strong> and machine learning. OpenCV lets you&nbsp;write software for:</p>
<ul>
<li>face detection,</li>
<li>object identification,</li>
<li>motion tracking,</li>
<li>image stitching,</li>
<li>stereo vision</li>
<li>and much, much more...</li>
</ul>
<p>Open CV is written in&nbsp;highly optimized C/C++, supports multi-core execution and&nbsp;heterogeneous execution platforms (CPU, GPU, DSP...) thanks to <a href="https://www.khronos.org/opencl/" target="_blank">OpenCL</a>. The project was&nbsp;launched in 1999 by Intel Research and is now actively developed by open source community members and&nbsp;contributors from companies&nbsp;like Google, Microsoft or Honda...</p>
<p>My experience with Emgu CV/OpenCV comes mostly from working on <a href="https://www.youtube.com/watch?v=nI6Ef405rAg" target="_blank">paintball turret project</a> (which I use to have a break from "boring" banking stuff at work). I'm far from computer vision expert but&nbsp;I know enough to teach you how to <strong>detect a mini quadcopter flying</strong> in a room:</p>
<p><iframe style="display: block; margin: 10px auto;" src="https://www.youtube.com/embed/XSoYPFcOwss" width="560" height="315" frameborder="0" allowfullscreen="allowfullscreen"></iframe></p>
<p>In the upper-left corner you can see frame captured from video file, following that is the background frame (static background and camera makes our task simpler)... Next to it are various stages of image processing run before drone (contour) detection is executed. Last frame shows original frame with drone position marked. Job done! Oh, and if you are wondering what is the "snow" seen in the video: these are some particles I put to make the video a bit more "noisy"...</p>
<p>I assume that you <strong>know a bit about C#</strong> programming but are <strong>completely new to Emgu CV/OpenCV</strong>.</p>
<p>By the end of this tutorial <strong>you will know how to</strong>:</p>
<ul>
<li>use <strong>Emgu CV 3.2 in C# 7 (Visual Studio 2017)</strong> application (most tutorials available online are quite outdated!),</li>
<li>capture frames from video,</li>
<li>find changes between images (diff and binary threshold),</li>
<li>remove noise with erode and dilate (morphological operations),</li>
<li>detect contours,</li>
<li>draw and write&nbsp;on a frame</li>
</ul>
<p>Sounds interesting? Read on!</p>
<p>&nbsp;</p>
<h2>THE CODE</h2>
<p>I plan to give detailed description of the whole program&nbsp;(don't worry: it's just about&nbsp;200 lines)&nbsp;but if you would like to jump straight to the code visit this <strong>GitHub repository</strong>: <a href="https://github.com/morzel85/blog-post-emgucv" target="_blank">https://github.com/morzel85/blog-post-emgucv</a>. It's a simple console app - I've put everything into <a href="https://github.com/morzel85/blog-post-emgucv/blob/master/EmguApp/EmguApp/Program.cs" target="_blank">Program.cs</a>&nbsp;so you can't get lost!</p>
<p>Mind that because Emgu CV/OpenCV binaries are quite large these are not included in the repo. This should not be a problem because Visual Studio 2017 should be able to automatically download (restore) the packages...</p>
<p>Here you can <strong>download the video</strong> I've used for testing: <a href="http://morzel.net/download/emgu_cv_drone_test_video.mp4" target="_blank">http://morzel.net/download/emgu_cv_drone_test_video.mp4</a> (4.04 MB, MPEG4 H264 640x480 25fps).</p>
<p>&nbsp;</p>
<h2>STEP 0: Crating project with Emgu CV</h2>
<p>To start lets use Visual Studio Community&nbsp;2017 to create new console application:</p>
<p><a title="New project... Click to enlarge..." href="http://en.morzel.net/pics/PostImg/emguapp_new_project.png" target="_blank"><img style="vertical-align: middle; display: block; margin-left: auto; margin-right: auto;" src="/pics/PostImg/emguapp_new_project_small.png" alt="New project... Click to enlarge..." /></a></p>
<p>Now we need to add Emgu CV to our project. The easiest way to do it is to&nbsp;use Nuget to install Emgu CV package published by Emgu Corporation. To do so run "<em>Install-Package Emgu.CV</em>" command in <a href="https://docs.microsoft.com/pl-pl/nuget/tools/package-manager-console" target="_blank">Package Manager Console</a> or utilize Visual Studio UI:</p>
<p><a title="Adding Nuget package... Click to enlarge..." href="http://en.morzel.net/pics/PostImg/emguapp_nuget.png" target="_blank"><img style="vertical-align: middle; display: block; margin-left: auto; margin-right: auto;" src="/pics/PostImg/emguapp_nuget_small.png" alt="Adding Nuget package... Click to enlarge..." /></a></p>
<p>If all goes well <em>package.config</em> and DLL references should look like this (you don't have to worry about&nbsp;<a href="https://www.nuget.org/packages/ZedGraph/" target="_blank">ZedGraph</a>):</p>
<p><a title="Packages and references... Click to enlarge..." href="http://en.morzel.net/pics/PostImg/emguapp_hello_world_refs.png" target="_blank"><img style="vertical-align: middle; display: block; margin-left: auto; margin-right: auto;" src="/pics/PostImg/emguapp_hello_world_refs_small.png" alt="Packages and references... Click to enlarge..." /></a></p>
<p>Now we are ready to test if OpenCV's magic is available to us through Emgu CV wrapper library. Let's do it by creating super simple&nbsp;program that loads an image file and shows it in a window with&nbsp;obligatory "Hello World!" title:</p>
<pre class="brush:csharp;auto-links:false;toolbar:false" contenteditable="false">using Emgu.CV; // Contains Mat and CvInvoke classes
class Program
{
static void Main(string[] args)
{
Mat picture = new Mat(@"C:\Users\gby\Desktop\Krakow_BM.jpg"); // Pick some path on your disk!
CvInvoke.Imshow("Hello World!", picture); // Open window with image
CvInvoke.WaitKey(); // Render image and keep window opened until any key is pressed
}
}</pre>
<p>Run it and you should see a window with the image you've selected. Here's what I got - visit <a href="https://en.wikipedia.org/wiki/St._Mary%27s_Basilica,_Krak%C3%B3w" target="_blank">Krak&oacute;w</a>&nbsp;if you like my picture :)</p>
<p><a title="Window with image... Click to enlarge..." href="http://en.morzel.net/pics/PostImg/emgu_hello_world.jpg" target="_blank"><img style="vertical-align: middle; display: block; margin-left: auto; margin-right: auto;" src="/pics/PostImg/emgu_hello_world_small.jpg" alt="Window with image... Click to enlarge..." /></a></p>
<p>Above code loads picture&nbsp;from a file into <a href="http://docs.opencv.org/2.4/modules/core/doc/basic_structures.html#mat" target="_blank">Mat</a> class instance. Mat is a n-dimensional dense array containing pointer to image matrix and a header describing this matrix. It supports a reference counting mechanism&nbsp;that saves memory if multiple&nbsp;image processing operations act on same data... Don't worry if it sounds a bit confusing. All you need to know now is that we can&nbsp;load images (from files, webcams, video frames etc.) into Mat objects. If you are curious read this <a href="http://docs.opencv.org/2.4/doc/tutorials/core/mat_the_basic_image_container/mat_the_basic_image_container.html" target="_blank">nice description&nbsp;of Mat</a>.</p>
<p>The other interesting thing you can see in the code is the <a href="http://www.emgu.com/wiki/files/3.1.0/document/html/bb544594-4bf7-5a11-594c-f143b226f1da.htm" target="_blank">CvInvoke</a> class.&nbsp;You can use it to call OpenCV functions from your C# application without dealing with complexity of operating native code and data structures from managed code - Emgu the wrapper will do it for you&nbsp;through&nbsp;<a href="https://msdn.microsoft.com/en-us/library/aa288468(v=vs.71).aspx" target="_blank">PInvoke</a> mechanism.</p>
<p>Ok, so now you have some idea on what Emgu CV/OpenCV libraries are&nbsp;and how to bring them into your application. Next post coming soon...</p>
<p><strong>Update: <a href="http://en.morzel.net/post/detecting-a-drone-opencv-in-dotnet-for-beginners-emgu-cv-3-visual-studio-2017-part-2">Part 2</a> is ready!</strong></p>http://en.morzel.net/post/detecting-a-drone-opencv-in-dotnet-for-beginners-emgu-cv-3-visual-studio-2017-part-1
http://en.morzel.net/post/detecting-a-drone-opencv-in-dotnet-for-beginners-emgu-cv-3-visual-studio-2017-part-1#commenthttp://en.morzel.net/post.aspx?id=ddf844a8-3287-4214-aa24-ced4279c3e86Mon, 31 Jul 2017 11:51:00 +0200.NET Framework/C#CodeProjectGraphicsOpenCVmorzelhttp://en.morzel.net/pingback.axdhttp://en.morzel.net/post.aspx?id=ddf844a8-3287-4214-aa24-ced4279c3e867http://en.morzel.net/trackback.axd?id=ddf844a8-3287-4214-aa24-ced4279c3e86http://en.morzel.net/post/detecting-a-drone-opencv-in-dotnet-for-beginners-emgu-cv-3-visual-studio-2017-part-1#commenthttp://en.morzel.net/syndication.axd?post=ddf844a8-3287-4214-aa24-ced4279c3e86