You might remember this effect from the good old Kai's Power Tools days.
But of course it has existed before that time - as usual it's
based on quite simple mathematics: a process called
Geometric Inversion

The position of your mouse controls the center of the inversion circle. If you click and drag somewhere you can change the radius of the circle

Inside the sourcecode I've already added the code that is needed to apply this effect to live video.