Steps towards a larger project

I have an idea for a project I don’t yet know how to make. To help me get there I figured I would break down each of the tasks I need to figure out and start working through them individually. I’ll get to that below, but first, I figured I would do a bit of explaining what the larger idea is all about.

Before getting to the technical side of things, I’ll describe some of the conceptual ideas. I would like to construct a system where I am able to edit out the motion from a scene (ie. people moving) and be able to create a static image void of any of it’s inhabitants. Think an image of Grand Central Station in New York completely empty on a saturday afternoon. Or, as another example, this clip from Vanilla Sky depicts Time Square completely empty.

The application of doing this I have envisioned in a number of ways. First one being that through a series of time lapse photographs a new photo could be constructed of a place that may never otherwise be empty. Also, the system could be integrated with live video so that the user’s can see themselves alone in these same spaces. Lastly I would like to input video footage into the system from movie scenes that have locked off shots. By creating a static image from this, the image can then be used for further background subtraction that would take away any characters from the scene and allow users in insert themselves into the scene.

As for the technical implementation, here is a brief explanation as to how I imagine it working out. The processing sketch will read and store all of the pixel data for each image. Once the sketch has cycled through the array of images, it will compare the values where it noticed a minimal change in pixel values and see if there is enough information to construct a new image with these values. If there are any gaps, it will continue reading images until it can construct an image without any. If not, the new image is displayed/saved.

Breaking this down further into smaller steps I can work through for now, I have listed a few key elements to familiarize myself with:

Learn how to create an array of images.

Learn how to store an image.

Learn how to implement background subtraction.

Working my way through this list I first created an array of images. I used just two to start. Alternating between a frame with an empty background and a frame with a subject in the foreground.

Next step was to figure out background subtraction. I learned a few things through this process. One of which was how to store an image off screen and into memory. The other thing which took a lot of trial an error to figure out, is that I may need to use an external library to handle the background subtraction as I couldn’t quite get the results I was hoping for. I experimented with the process a bit to also do some foreground subtraction and some frame differencing. Here is a video of background subtraction (code), foreground subtraction (code) and frame differencing (code).

Working my way through this I put together a new list. This one being some steps for the actual programming. So the steps in that go:

construct image array of 500 images

load first 5 images of array into buffer

analyze pixels across all 5 images above and below threshold

write each pixel consistently below the threshold in all 5 images to a new image on the screen

load next 5 images in array and write any pixels below threshold that haven’t already been written to the new image

repeat step 5 until either the new image has all pixels in the image filled, or the end of the image array has been reached