Converted to Conversion, I

The idea of taking a two-dimensional image and turning it into a stereoscopic image is not a new one. The idea of being able to pull this off for a motion picture sounds like a fantasy. There are two extant techniques – conversion on-the-fly and conversion I supposed one would classify as off-line, requiring some considerable effort. I am not going to consider on-the-fly technology because the state of the art is abysmally abysmal.

At this moment a serious job of conversion involves machine intelligence to a relatively minor degree (whereas on-the-fly is entirely algorithmically bases). There are rapid strides being made, especially for military applications in image analysis, that lead me to believe that a completely automatic solution may someday be possible; but right now it’s beyond the state of the art. Looking at a two-dimensional array of pixels that have various density and color values, and to determine from that array which pixel is near and which pixel is far, is something only a human being can do. There’s no machine that can look at such grouping of pixels and reconstruct a satisfying stereoscopic image.

It’s interesting that the first stereoscopic images were essentially synthetic, or converted if you prefer. When Wheatstone demonstrated the stereoscope in 1838 it was a year or so prior to the invention of photography. Only drawing could demonstrate binocular stereopsis, so he had an artist help him draw perspective views of various objects, like a staircase. Wheatstone began by using a two-dimensional view of a staircase, adding a second perspective view, and turning that into a stereopair – what we could consider to be a conversion of sorts.

“Conversion” is an interesting word, because it denotes the transformation, metamorphosis, or transmogrification of one thing into another by some process. It’s related to idea of translation or synthesis. The word “synthesis” could be applied to the transformation of a two-dimensional array of pixels into a stereoscopic image, and it’s a classier word than “conversion.” But “conversion” is a term that’s used in the workaday film business. Synthesis implies a higher-power process. (On the other hand, the related word “synthetic” implies that the result is ersatz – not quite as good as the original, or second-best – so the word “conversion” avoids that invidious connotation.)

The idea that one can take a two-dimensional movie and turn it into a stereoscopic movie, while on the face of it improbable or even ridiculous, is not completely audacious, because there are certain camera moves that lend themselves to such a feat. In fact, any lateral move or any movement in which the camera lens axis is perpendicular to the direction of the camera’s motion – in other words, a traveling shot – would produce over time, through the mechanism of motion parallax, stereo pairs. That technique has been used for aerial photography to produce stereopairs, with a single camera taking pictures through a time interval, and in that process it can work. But if there is anything moving in the shot during the interval, the result will be a temporal artifact will generate spurious parallax.

There are other related camera moves. Forward motion can also produce data that can be used for producing stereoscopic images. But generally such techniques are limited, and the conversion techniques that are in use involve something that people in the film business call “rotoscoping,” or (because it’s always important to shorten a term so that it sounds like it’s groovy) “rotoing.” “Rotoscoping” is a term that harkens back to a technique that was used in the earlier days of filmmaking in which an image was projected on a screen and outlined as a guide to animators. Think Cinderella. A term that’s more descriptive for what’s done today by the various conversion vendors is “outlining.” But “outlining” sounds like something done by kids with Crayolas and that’s not a good way for a vendor to sell a studio a service that can cost $40,000 per minute for a good job, or $15,000 a minute for a not so good job.

The techniques that are used in conversion are similar across the industry and this opinionated tutorial is not the place to emphasize the differences. The process involves first outlining objects so they can be segregated from the background, and then adding some additional relief to those objects so they don’t look like cardboard cutouts but appear to be rounded. The images that are outlined have to be shifted left or right depending upon whether they need to moved back into or out of the screen. You can think of it this way: one is essentially taking a two-dimensional image and cutting out the objects with a scissors and sliding them left or right; but then you’re left with holes in the background. In fact, the more you want the object to come off-screen, the bigger the hole you will have. That means there has to be some way to fill in the hole, or plug it. One might use some kind of a cloning algorithm which can go so far, but as of today it takes a considerable amount of operator intervention to fill in the hole; and that means an artist is required to paint in the missing background holes.

This has to be done for every frame of every shot, but there can be some help with algorithms that can follow the outlines. If you have some kind of semi-intelligent algorithm starting with a master frame, the hope is that if the shot elements remain relatively constant and there are no discontinuities, one can have – at least within one shot – a lot of work done by the computer. But turns out that conversion is labor-intensive, and that’s why a lot of it is done in areas where people get paid very little.

One approach to conversion is to create a second perspective view based on the first, by having some human being decide what’s near, what’s far, and then using a combination of the techniques that I’ve just described. Thus from one image a second perspective is created. Another approach, which is the one that was taken by Mark Feldman, who worked for me at StereoGraphics, was to essentially build a false front. I’ll tell you why we did this. Feldman, a truly creative software developer (I had a wonderful time working with him), helped me with software we needed SynthaGram content. My colleagues and I at StereoGraphics developed an autostereoscopic technique that I think was about as good as people have been able to achieve – but that’s another story.

We needed content for that lenticular stereoscopic display. In particular, we needed nine views for the SynthaGram, with a progression running from left to right. We had a lot of material that had stereo pairs, but we didn’t have much that had nine views. We envisioned the SynthaGram to be useful for advertising and there would be a requirement for stills, so Feldman worked on a way to take a 2-D shot and turn it into a 3-D shot – not simply by creating a stereo pair, but by essentially creating a false front. When the software was completed (and it was never released – we worked on this between 2002 and 2004) you could use the kinds of techniques I’ve described, create what was essentially a false front, and then have nine cameras look at it and produce a multi-perspective SynthaGram image. Mark built it in the form of a Photoshop plug-in, using paths, and other Photoshop tools, but it never got past the first generation. I did 20 or more conversions myself. I had to fill-in background, outline objects, and stuff like that.

So when I arrived in Hollywood and people were converting 2-D to 3-D I thought to myself, “Holy cow! We’ve already done that.” Of course we weren’t so audacious as to believe that we could actually do it for a feature film one frame at a time. There was one company in the L.A. area called In-Three who were truly audacious, because there wasn’t a lot of stereoscopic activity at that time; so they had to have amazing faith not only in the process but in the belief that there could be a stereoscopic business opportunity for feature films.

Flash forward a bit from 2004 or so to a year or two later, and I’m working at Real D. One of the things I’m doing is working with the theatrical motion picture business, with the studios and independent producers, trying to help them figure out how to make stereoscopic movies that look good. Because any show can play on a 3D screen (it’s up to the exhibitor what plays, Real D and other stereoscopic projection vendors have no control over the quality of what is shown in their theaters). I embarked on a program to try to get people up to speed on how to make good-looking images, and I fell in with a group from Paramount Pictures who, under the direction of Aaron Parry, were attempting to determine whether or not it was possible to shoot the Star Trek movie that was being prepared by director J.J. Abrams. What we were trying to do was figure out a production pipeline that could allow the Star Trek movie to be shot in 2-D. Abrams wanted to shoot on film, and he didn’t want to be bogged down by stereoscopic cameras.

The idea that Parry and his colleagues had hit upon was that, since so much of the movie was greenscreen, the outline procedure was taken care de facto and the CG backgrounds could be rendered stereoscopically. This sounds like an ingenious idea and in fact it is, and it probably would have worked pretty well. We sat through three or four months of tests, of six shots from The Spiderwick Chronicles, which was not a 3-D movie. Parry and his colleagues picked out most difficult shots to see how conversion houses would perform, and we worked with about half a dozen conversion houses. My memory tells me that over those months at least once or twice a week we would look at what the vendors from the conversion houses provided and critique it, and we would send back changes and then look at the changes.

One of the things I learned was that the creative and technical people in the motion picture industry can see things that I can’t, and by hanging out with them in screening rooms I’ve learned a lot. Their perception of image quality and what’s going on in a shot is obviously their obsession and their expertise, and I’ve been happy to learn. Stuff that I thought passed, when pointed out to me, clearly had anomalies. But when you’re making a feature film and you’re showing it to an audience, they don’t study the way the effects people or color timers or 3-D supervisors do. So a lot of things in a movie just have to be good enough for one viewing by a non-expert involved in the story. The results may not be great, but they’re good enough. Nevertheless, the only thing you can do when you’re working on a project like this is to apply all your critical facilities and try to get the image to be better and better and better and still meet budget.

Eventually J.J. Abrams and Paramount decided not to convert the movie, to not make the movie stereoscopic. I wasn’t privy to the decision-making process, but common sense tells me that at time there were very few stereoscopic theaters, and I don’t think the Star Trek movie needed 3-D because it had a great big built-in following. Today it’s a different story. Today you’d probably choose to produce it in 3-D because just about any tent pole or major effects movie these days, to guarantee its return on investment, needs to be released in 3-D.

That was one major learning experience. But what I haven’t said is that I walked into the process not believing that this technique could work. I walked into the process with Paramount believing that conversion was unworkable, that it couldn’t produce decent-looking images. But I got converted to conversion. Sitting there looking at the footage, I realized that it was possible to do a good job and for a reasonable price – depending upon what you call reasonable. And it didn’t seem to matter that much who the vendor was. The most important thing seemed to be who the stereoscopic supervisor was at the conversion house. If somebody had good taste and a critical eye (actually I should say “critical eyes” – you can’t say “critical eye” for 3-D, you have to say “critical eyes”), such a person would determine the look, or the outcome of the film, and its stereoscopic effect.

There are other factors to consider. A movie needs to make what is called “day and date”. It’s got to be delivered to the theaters on time. That means that conversion or turning the 2-D material into 3-D, or integrating the various assets whatever their dimensionality into the finished product probably has to be done in parallel with the post work. Or virtually in parallel with the cutting of the film as shooting progresses. If the company waits for the first cut then a good job of conversion could take six months and that means sitting on borrowed money and interest payments for too long to go down smoothly with the investors.