Harnessing the creative potential of Unity to produce a a movie and TV series!

In December of 2016 I ran a successful Kickstarter Campaign to fund the production of 5 episodes of a TV show for kids called "The Opus Joyous Show". I ended up choosing Unity as my main production tool for the show. Traditionally, used as a video game development engine, Unity has really blown me away with it's ability to create stunning, realistic film like scenes without the need for long rendering times, render farms and expensive 3D assets. The PBR (Physical Based Rendering), high quality lighting and effects, new Cinemachine Timeline and vast array of assets in the Asset Store and amazing community of developers and users there to offer support along the way has made this process really great. I think there is a big future for Unity as a film production tool and it's exciting to be part of it as small or big as this project may turn out to be.

The Decision to Use Unity

Considering that my script included ocean scenes, ships, islands, lava, volcanoes, explosions, submarines, heavenly visions and more, and also considering the fact that I didn't have a budget to shoot any of those on location, I needed to look at using 3D virtual sets as an option. At first I tried one of the leading 3D animation software tools out there to tackle this but I was finding that too much of my time was being used animating, sending things up to a render farm to preview, re-rendering and then re-rendering again. I didn't have the time or money to afford this or the expensive assets (3D models) that I would need along to get this done in time and on budget.

I decided that I needed to try something new, I needed a better way to tackle the 3D animations and environments that my script required. If I couldn't figure that out, I was going to need to re-write the script to something a little simpler. Looking for other options, I remembered my experience with Unity a year early, experimenting with a 360 VR scene of the crucifixion. I loved working in Unity at the time. The learning curve was quick, the assets were either free or really affordable and the quality really blew me away. Best of all, everything I did in Unity was in real time, no render farms needed. But, could Unity handle an entire TV series? Could I make a film within Unity? And, considering that my main actors weren't 3D model but rather puppets and real life characters that walked and talked, would that even be possible to integrate these as video assets in Unity? It's turns out the answer is yes.

It turns out that I could indeed use Unity to create a movie and TV series. And it sure helped when I found out about the new Cinemachine Timeline Asset. That was a game changer!

Cinemachine's Timeline : A Game Changer!

It turns out that I could indeed use Unity to create a movie and TV series. And it sure helped when I found out about the new Cinemachine Timeline Asset. That was a game changer! I'm not sure I could have done this without the Cinemachine Timeline asset. If you haven't heard of timeline or used it in one of the latest Unity releases, I highly recommend it. It's basically like a super powerful non-linear editing tool built into Unity which allows you to animate and activate pretty much any object or attribute in your scene. And for Camera related stuff , you simply add the "Cinemachine Brain" to your main camera, add a Virtual Camera or two or three or 4 or 5 etc. animate them and then use the Cinemachine Brain to switch between virtual cameras like you would at the healm of a Master Control Station. It's pretty amazing.
Using Video Assets and Green Screen footage within Unity.

When I originally decided to integrate 3D animations, scenes and models with real life characters and puppets etc. I thought for sure I'd have to export all of the 3D scenes and then motion track things in After Effects to add and sync up my green screen video in After Effects as well. From there I thought I might have to do some color correction and lighting effects to get things to match as best as possible. Well, I'm glad it wasn't this complicated. This would have taken a long, long time! Painfully long.
Instead, I was able to come up with a native solution in Unity. And the first time I saw it in action I almost feel out of my chair. All within Unity, I literally could import a green screen video asset and in real time, key out the green to make the background transparent, color correct the video that was just keyed out, receive and cast shadows onto the video/characters , receive lights and have the video interact within the scene, post stack processing effects and all! This is amazing! Like seriously amazing! A genuine game changer!

The Process

In addition to some of the fundamental aspects of this production in Unity such as the Timeline and Chroma Key Kit asset, there have also been some other assets that have been integral to the smooth operation and day to day task of getting this project done. For example, the Asset "VR Panorama 360 PRO Renderer" has been something I use every single time I export a scene. Using this asset I am able to export the scene as a PNG sequence and or encode it as an MP4. Now, the tricky part if you are using video assets is getting your videos to export at the same rate as your scene. By default I have found that video assets in unity look great when you are previewing but when it comes time to export and record, they go wonky and out od sync. Apparently it's well know Unity limitation thing which I don't exactly understand but thankfully, the creator of the plugin has addressed this and offers a "VideoScrub" script you can add for each video asset you include in your scene.

Another hurdle I had to overcome was figuring out how to duplicate scenes and their timelines. By default, if I created a complicated scene with a complicated timeline, and then saved it as a new file, the timeline part would actually still be shared between the two scenes and any changes I made in the new scene with the copied timeline would mess up my other scene which I learned the hard way. The simple solution to duplicate my timeline as a new file didn't exactly work either as it lost all of it's "Bindings" or references to object when I simple duplicated it. The solution: a nice little script by "seant_unity". The script is called "Duplicate Timeline with Bindings". (* As a piece of advice, I also recommend renaming the new duplicated file to something relevant for future reference as the default naming convention simply adds "clone" Also, once you have duplicated your timeline, your original timeline will still be there. I like to delete this timeline if I'm using 1 scene per timeline).

All of this probably sounds confusing if you are not using Unity, or Timeline etc. I could go on and on as I love talking about the process. But I think I'll stop here for now. If you have gotten something from this short post, that's great. I hope I helped. And if you want to know more or need some technical help, don't be shy to reach out and contact my via my Unity Connect Page.
I think I'll sign off here and leave you with the trailer for Episode 1 of the "Opus Joyous Show" if you are interested. Remember, this was all made in Unity! :)
Here's a trailer from a few months back.. new trailer and the entire series set to launch in September of 2018.