OpenCV Tutorial: Real-Time Object Tracking Without Colour

Found this video useful? $1 PayPal Donations are very much appreciated, thank you. https://www.paypal.com/cgi-bin/webscr...

For tutoring rates and business inquiries, please email kyle.hounslow@gmail.com

In this tutorial we will look at real-time object tracking using the method of sequential images. This allows us to track objects without the use of colour filtering. We code in C++ using Visual Studio 2010.
Start by downloading the following zip file: https://dl.dropboxusercontent.com/u/2...

If you got stuck anywhere in this tutorial you can download the final source code from here:
https://dl.dropboxusercontent.com/u/2...

---

Closed Caption:

hello everyone Kyle here an engineer
from vancouver british columbia today
we're going to be learning how to track
objects in real time but instead of
using our color tracking that we learned
in previous tutorials today we're going
to learn how to track objects without
color
more specifically today we're going to
be looking at the method of sequential
images and what that entails is we're
going to be we're going to read one one
frame of the video feed and then another
and we're going to compare these two
frames and we're going to see which
pixels have changed between the two
images and I'd like you all to think
about this is kind of the way that the
human eye human brain will shield detect
movement
if you have a scene where the background
is still and you have let's say a
bouncing ball
you can definitely tell them it's that
the ball is moving because it's moving
in relation to the background still
background
now we're going to start out with the
most basic example you can think of the
picture above is just my desktop
actually the top of my desk my computer
rests on it's just black and in the
second image i have placed a call in the
middle
now it in opencv we can actually call a
function which is going to compare the
two images and it's going to take the
absolute difference between the two
pixels
I started the two matrices of pixels on
each of the each of the images and once
it does that
we're going to be left with a threshold
image and what the the threshold image
represents its going to represent which
which pixels have changed between the
two images that were comparing if you
follow along in the previous tutorials
you'll you'll know that if this special
image represents the object that we're
trying to detect we we know how to deal
with this we can employ the exact same
methods that we used in the previous
tutorial to track objects in reality
we're not going to be dealing with such
a simple case for the you know the coin
is there just magically appears
and were left with a perfect threshold
image of the object that were trying to
track instead since the two frames are
being compared to each other and the
object will be
doing it's not just going to have the
object and its next position is also
going to capture its previous position
so you know we'll be left with something
kinda like an oval shape from the
previous image and the next image
furthermore there is actually going to
be a overlapping region between the two
positions in this overlapping region
it's just caused by two images kind of
being blurred together and what we're
going to be left with his a bit of a
black hole in between the two positions
of the coin
so let's take a take a look at a quick
example here so you can get a better
kind of visual idea of what I'm talking
about
on the right we have a video feed of the
ball rolling by tennis ball
I'm left we have written some code such
that we can take the absolute difference
of the two frames eventually
and if we pause we can kind of see
evidence of what i was talking about
earlier that bit of overlap between the
two frames we go back one more time can
actually see that in some cases fall so
fast that between the two frames the
positions are so far apart and it
actually looks like there are two
objects but it's so time to get picky
work with what we got this method of
object tracking is a little more
advanced and that's why it's a bit more
tough to deal with
but enough talking about theory let's
let's jump into the code you can start
by downloading the the file from below
this zip file is going to contain both
the source file and a sample video that
we're going to be using the video is
going to be the one you saw earlier with
the bouncing ball and by the end of this
tutorial you're going to be able to
track that ball using the method of
sequential images that we just discussed
we're gonna start out by setting up a
visual studio project if you already
know how to do this you can just skip
ahead to the next section
but if not just follow along we're going
to select new project and then the name
of this project i'm going to call it
motion tracking
and then click Next and console
application empty project
we're gonna head to the property manager
and we're going to add in our debug or
where we're going to add our existing
property sheet
I have mine saved in my main project
directory and here you see it under
debug we learned how to make that in the
installation guide if you haven't done
that yet you can get there and check
that out
all right now that we've linked our
project to the opencv library
we're gonna head to the zip file that we
just download it we're going to extract
it to our visual studio project folder
mine is in my documents
and then visual studio 2010
and under projects and
motion tracking one more folder in
motion tracking and then we'll see okay
extract
you should see both your new source file
and your video in our project folder
I head back to visual studio and we're
going to right click on source files
we're going to add an existing item
we're going to select the motion
tracking source file that we just added
to the folder
we're going to add it to our project and
open it up
the free i click compile and run the
program will see that we have a bouncing
ball video
it's just going to be looping once it
reaches the last frame just going to
reopen that the video and play it again
so it's good making sure that all works
you scroll down I just wanna explain how
this code works right now just go down
to the main function you can see that we
have a main wildly that's infinitely
looping in this
meanwhile we open up our bouncy ball
video within this loop we have another
wildly which test the current frame
position of the video against the total
frame count of the video and when the
current frame count equals the total
frame count
you know we've reached the end of the
video at this point we can release the
capture and reopen the video and do it
all over again and that's how we get a
repeating video
so right now in our second while loop
all were doing is reading a frame and
later displaying that frame in between
you can see that we have a bunch of
comments here and what I aim to do is to
go through step by step by step and fill
in the code relative to these comments
if we go up to the top here at the
beginning of our main function we can
see that I provided a couple of boolean
variables and some matrices and we're
going to be using those you don't need
to worry about defining any of them
now what I've done with the boolean
variables is actually set up some key
stroke listeners
so that you don't need to get all this
code here it's basically just for your
convenience we can use the T button to
toggle tracking between true and false
you can also use d to turn on and turn
off debug mode or p to pause and resume
the code and when we press p to pause
the code is actually just going to enter
a infinite loop and not run any of the
code until you press p again to resume
so I hope that makes enough sense so we
can scroll up and start coding
i'm actually going to skip a couple
steps and go straight to our ABS
dysfunction that I talked about earlier
it's going to take the absolute
difference between two trains
we can see that it is asking for 11
frame then another frame that's also
asking for a matrix to store the
destination frame for the output of this
function
one good thing to note is that the first
frame and the second frame they actually
need to be a grayscale images they need
to be in basically black and white
so before we use our absolute function
we're going to convert our frames to
grayscale and how we're going to do that
is we are going to use c function called
convert color
open that up you see is asking for the
source frame going to put in frame 1 and
4 the output
I believe I've saved I i I've actually
initialized a couple variables a
scrimmage one agree image - we're going
to use those
why not so great image 1 and that is
asking for the code in which for it is
I like the conversion code if you will
for this one going to typing
I know it's one of the color one so if
we press color and then hit control
space we can see a whole bunch of
options i believe it's BGR
the GRA GRA by the time the colors been
converted our second frame should be
ready to capture is actually good to
note that if you were to just say
capture a frame 1 and then frame - there
might not be enough to lay in the code
to actually have shifted to the next
frame so i would make sure that you have
the convert color before you capture
second frame
and then we're going to do the exact
same thing going to convert a degree
scale because copy and paste change our
ones to choose
and then we're ready to use our ABS
dysfunction so if you check it out again
you can see going to put in our
grayscale images reimage one gray image
to and our destination
let's see which 1i have used for this ok
we're going to use difference image
you mean you can call it whatever you
want but you gotta you gotta initialize
the first so I've been that for you
already
now i know i said earlier that the
absolute difference function is going to
take the two frames and it's going to
output a threshold image which we are
going to be able to use to track the
object
what really is the case is that these
this absolute difference function
outputs with what's called a intensity
image or what I've called a difference
image and this type of image actually
doesn't just show black and white or
zeros and ones actually has values
between 0 and 1 so there is some gray in
there too
and what we're going to do is going to
have to use a threshold function to get
just black and white
so let's go ahead and do that now we're
going to go blow this comment and use
the cv function threshold the first
argument is going to ask for for its
input
we're going to use our difference image
and for the output we're going to want
our threshold image you have to find it
up here
I call it threshold image is copy and
pieces into their typo
the third value
now that's an interesting one actually
there is
I've renamed it to a what's called
sensitivity value because if you have
too low of a value for this threshold
it's going to actually let in too much
noise but if you have maybe let's say
the object is really small and you can't
really detect it without letting in a
little bit of noise then maybe you might
want to lower your sensitivity value and
just risk having the noise i I've set
the sensitivity value for us today at 20
it seems to be a pretty good value I
don't see any noise and is tracking the
object pretty well so let's just use
sensitivity value this max Val that's
the maximum value is between 0 and 255
so just use to 55 and the tight we're
going to use this C value called Thresh
binary i believe yep that's going to put
a binary threshold image for us
now when working with computer vision i
always like to have a debug mode because
you always want to see where your code
is going wrong so if you can see the
threshold image you know the difference
image and see what they look like you
can you can find what's wrong with your
with your image tracking
so to show the windows were just going
to use GV function i am show and let's
just call it
difference image and we're going to use
the difference image and we're also
going to show
our threshold age
now one thing to note is that once you
show these windows
if you don't destroy them after when you
click debug mode off then it's going to
leave up the blank frame there and you
don't want that so there's also a CB
function called destroy window
and we'll just use
what we typed in here these two have to
be the same you have to specify the same
name
let's repeat that again this time from
the threshold image
all right let's run the program see what
we got so far
we're going to see we just have the
bouncing ball because i have not
initialized debug mode yet so if you
click on the window and click d debug
mode will be enabled and we can see our
difference image and threshold
now you can almost barely see the
difference image because it has
everything ranging from black to white
so there's a gray in there is not just
black and white our threshold image is
black and white so I'm going to I'm
going to pause the program by pressing p
we can compare the three here
it's actually find a bit of a better
spot to pause that
there we go so if we zoom in to our
colored picture we can see tiny little
smudge here and over to our difference
image also pretty small but you can see
it and finally down to the threshold
image we can see a little bit of white
speck here and you might be thinking
right away thats doesn't seem like
enough information to give our program
to try and find the object we should
probably use something similar to last
time where we use the dilator any road
function to expand our white space you
would be absolutely right we do have to
do something about this small amount of
white space for getting instead of using
the dial eight year old functions like
we did last time I'm gonna introduce a
new one
cv function called blur and the
arguments that takes it's going to take
an input image in an output image
I we're not going to define any more
variables i'm just going to use the
threshold image
and save it in the threshold image as
well now it's asking for a size of the
blur and that's similar to the size that
we used to dilate last time so we just
use the cv type
sighs and I've already defined a size of
the top here you can you can fiddle
around with this as well just to see
what happens but you're going to want to
use their size
which is 10 for us
the center where sighs , their size and
then close the bracket so after we've
learned it is going to unfortunately
output and intensity image from this
function so we're gonna have to use the
threshold the threshold function that we
used before but it doesn't matter we can
do basically the same thing
let's use threshold image twice
we're going to use sensitivity value
same one
max Val - 55 and fresh binary as well
just like before we're going to use our
debug mode for our advantage here and
we're going to show
let's just say final threshold image i
will show our threshold image
and just like before we're going to want
to destroy the window
after debug mode is turned off
now that wasn't too complicated for you
let's let's run it and see what we get
again click key for debug mode and would
you look at that our final threshold
image
much much better than our
previous threshold image if we pause it
we can see we have increased that white
space almost made it into just one
single object he said two up top there
I think it's about time it is no time to
use our tracking function so that we can
track this object so to track our image
I've made a function very similar to one
in the last tutorial
I've named it search for movement so
it's going to search for that little
white space of the ball and basically
what we do is we use the fine contours
function again we draw a bounding
rectangle we use the centroid of the
bounding bounding rectangle around the
contour and we find the middle of the
object is pretty simple
I i encourage you to look through the
code trying to understand the best you
can
I'm not going to go through it because I
would take too much time but we have
also the drawing function which is going
to draw some crosshairs tell us that
we're tracking an object at x and y and
it looks pretty cool all together
if you scroll back down to below this
comment here of tracking enabled
we're going to do that exact thing we're
going to use our
if tracking enabled equals true I've
already defined that up top
it is also is set to false right now but
remember we have those key strokes so if
we click t is going to click tracking
enabled to true
so let's call our search for movement
function
search remove moment it's going to ask
for our threshold image which we have
prepared
it's asking for the camera feed that's
just to draw the
actually let's just see what it is
called at frame one was dropped the
frame one that's we we pass in the frame
to the search for moving function and
it's going to draw to it and give it
back to us and then we can just show it
right here
so hall is working well we should be
able to track this object
let's check it out click run
we can click d just to make sure the d
bugs are going looks great looks great
press D again turn it off or destroy
windows are working great
- and now if we press T
can see that our object is being tracked
in real time and say it's working pretty
well - it's it's definitely following
the ball along
it's moving pretty quickly here so we'll
pause it you can do so to sing p on your
keyboard
we can just see that it says tracking
object at 291 and 333
and that's just our pixel values of X&Y
pixel values
now a few things I'd like to mention
before you go off and use this code for
yourself if you go down to this wakey
statement you can change that to any
value want right now is that delaying 10
milliseconds of you figure like your
code is going too fast just change it to
30 or 40 or 50 or something
also if you scroll up to some of these
values that I have
to find here the sensitivity value now
you sensitivity value if you have
problems with let's say noise or or
anything along like little white specks
coming up on your on your feed their you
might want to increase it to something
like 40 maybe 50 and if you have a
problem
the opposite problem or maybe your
threshold image isn't really showing
anything at all and you're going to want
to lower your sensitivity value you
might let a bit of noise in but you
could probably get rid of that with with
the blur function and since i mentioned
the blur function you can actually
change the size to something bigger or
smaller to see you needs
there's one final thing I'd like to add
that if you want to use your own webcam
for this instead of video input
you're gonna have to change up the code
a little bit and i'm going to show you
how to do that
first off you're going to want to cut
and paste this capture and it's checking
for errors here you're going to want to
cut and paste that outside of the wild
because we don't need to open the video
feed every time it reaches the last
frame
because it's just a constant video or
video feed so and we're going to want to
change our input 20
that's going to get the first location
of your webcam
you're then going to want to get rid of
this while because like i said we don't
need to check for the final frame and
remove the video
you can just delete that and then we're
going to have to find the other bracket
of that wildly you scroll down here I
know which one it is
it is this one and another good thing to
add is that we're gonna have to not
release the capture because that's just
that's going to close the capture that
you just open
so delete that and you should be good to
go
alright so this concludes our lesson on
fast object tracking without color using
the method of sequential images you can
click here to subscribe for next time
we're going to be looking at
we're actually gonna make a DIY home
surveillance camera using this exact
technique as well as our video writing
tutorial that I released about a month
ago and so stay tuned for that
and as always thanks for watching
yeah