Lighting, Rendering, Shaders, Scripting

Main menu

Post navigation

Alright, today is the day!! July 23rd, 2014 – The day I decided I’m going to make a short film!

As a tiny bit of background, I made two fully animated short films at university, and I work for a film company that – predominantly – makes short films of various sorts. So I’m familiar with the process. I just want a record here so that, in a few months when I haven’t done anything, I face self-shame and feel bad about it.

My Python journey is just beginning, but I’ve gotten to the point where I’m beginning to get up to speed with the basic syntax which makes doing a lot of thinking about a lot easier! If’s, Loops, all that jazz.

But the biggest two changes compared to maxscript that I’ve found so far are as follows:

1 – You have to “import” modules to get additional functions. This is good and bad – on the one hand, there are a bunch built in and you can download or make more, which means that the possible functions around to use are not only larger, but also expandable (Again, compared to MaxScript). For company-wide distributed scripts, this means that I need to make sure everyone has the correct plugins installed but that’s easy enough.

2 – You can do Ifs, Elif’s and Else’s – this might not be anything unusual to other coders, but MaxScript only had the first and last of those – there’s no else-if. Basically it allows you to completely control the flow of a set of if conditionals in a way that you can’t with MaxScript. It doesn’t strictly allow you to do anything new, because you can repeatedly do sequential if-statements with flagged variables and stuff, but sod all that – this makes it so much easier. I likey!

My Raspberry Pi arrives tomorrow, and I have my book all about coding in Python which I’ve started to read through. I also started looking at the Max SDK documentation for the Python stuff – It’s going to be a long road, but I’m excited about it!

I have a MxS currently that remaps assets from wherever they are currently to another, single folder by copying all the assets there and then remapping, including XRefs (and nested XRefs). This is a key part of our cloud based rendering system, but the problem is that some of the XRef Max files are very large, and whilst they zip up nice and tiny, it’s not possible to remotely request a local unzip on the server they get uploaded to. So what I’m hoping to do with Python (the server also runs a WAMP stack) is to have a standalone Python script up there that listens on a port and unzips files on request. Or perhaps I can do it by making a small text file in a given folder that the Py script will check, unzip the file in that text file, then delete the text file? I’ll need to experiment…

I’ve been so, so lazy about learning Python. I’ve long since wanted to actually write standalone or web apps (And we have a WAMP stack running in the cloud courtesy of Amazon, so there’s definitely somewhere to use it – in fact, I know exactly what I want to do with it!) but I’m starting with something else – A Rasberry Pi and a book teaching Python for beginners with a Rasberry Pi in mind! So I’m adding a new tag, and hope you update this blog on how I do.

Warning: The below contains significant spoilers for How to Train Your Dragon 2 from the very start. If you haven’t seen the film yet, turn back!

I know the internet doesn’t need another How To Train Your Dragon 2 review, but I consider the volume of critical response a testament to both how much people enjoyed the first (and, actually, second) film, as well as how much they care about it as a possible franchise. Furthermore, I’d like to state that How to Train Your Dragon 2 is an excellent film and I implore everyone to see it; It’s fun, it looks (and is animated) beautifully and it’s all sewn together with another excellent sound track from John Powell. If you’re worried because you haven’t seen the first one, don’t be worried – be excited, because you have the opportunity to experience, for the first time, the joy of some of the greatest bits of animation-meets-music ever rendered to pixels (that doesn’t sound quite as good as “committed to celluloid” but digital can’t have it all). In fact, it’s because it’s a great film that the flaws (Which largely escaped the first film) are all the more disappointing. I’ll jump straight in.

The mum. I know, I know. This has been the major source of ire within the community, though I think perhaps a lot of people are, if not actually misdiagnosing the problems with her character, perhaps prioritising the wrong elements. Her major flaw as a character is thus: She is presented as an incredibly capable (moreso than the main protagonist), complicated (at least as complex as the protagonist) and able to dispense wisdom (moreso than the protagonists other “wisdom givers”, Stoick and Gobbler, the former of which’s primary trait is entirely ignoring his son) character. She remains so for approximately twenty minutes, where after the illusion is shattered and she becomes, if anything, a liability.

I say that her flaws have been wrongly prioritised because they have centered on the issue from a feminist perspective, and this view is compelling – she’s a “strong female character” who, as explained by Tasha Robinson, ends up losing all her perceived power and requires rescuing by the “real” heroes, the men. This certainly happens, and I wouldn’t deny that this just plays into the sloppy characterisation of “strong women” that Tasha eloquently described. But her major flaw here is just that her character isn’t well developed, irrespective of her gender. And don’t worry, there’s a more significant target for the ire of feminists, as I’ll explain below.

Said mum – Valka – has a few opportunities to instill some real depth of character, if only the writers were willing to let it get there. She begins by explaining to Hiccup that she left the town when they were attacked by Dragons and she discovered one (who is now her mount) fawning over baby Hiccup. She then gets accidentally dragged away but decides to stay because she realises that Dragons have real hearts and emotions and feelings and that she doesn’t want to contribute to further Viking vs Dragon bloodshed, believing Berk and her husband incapable of changing their ways. This is all wonderful, but that’s basically what we learned in the first half hour of the first film. Her absence from her family, though, indicates a strong trait within her that this is something she cares greatly about – until her husband turns up, at least

Within minutes (and after he immediately forgives her, which in itself is somewhat puzzling), she’s decided to return to Berk because they had a little sing-song and he asked her to. What I thought would happen – and I think would have created a much greater moment of tension – is if Stoick began his song and dance, she joined in as she did in the film, and then at the end said “No” when he asked her to return – it would have shown that she’s a three dimensional character, a mother and a wife who misses her family and her previous way of life but that there are other things she wants too; This is a horrific choice for a mother to have to make, but she cares so greatly about this cause that she’s willing to sacrifice her family still – just as she has for the past 15 or so years – and the song and dance stirred within her a nostalgia and memory of a time when life was simpler and possibly happier, making the decision all the harder. A similar moment happened in the Harry Potter series (excellently executed in the films, which I wasn’t anticipating) near the end when Hermione and Harry are in the tent searching for the Horcruxes after Ron has stormed off. They’re both getting a bit depressed about it all, but they have this beautiful moment where they’re able to forget about it all for a moment and just dance together, to be “normal teenagers”, to enjoy each others company before returning to the horrible reality that they faced in the tent – they didn’t just laugh and say “Oh sod it, let’s just go to Ron’s aunt’s house and ride it out!” It was a reprise, but they understood that they had responsibilities beyond their immediate happiness.

So far, not so good: We have a character who has demonstrated that she’s incredibly talented at riding dragons but otherwise has failed to actually tell our main character anything he didn’t already know. She’s immediately turned her back on her previous decision because her husband asked her to. And then, as soon as the fighting starts, she needs to be rescued twice within pretty quick succession by her husband (something she has presumably avoided in the last 15 years when he hasn’t been around). Her largest contribution to Hiccup and Toothless’ success was the “revealing” of Toothless’ blue back fins, that enabled him to turn more sharply. Thanks, Mum! This is basically something she could have tweeted to him in 140 characters.

Now, to her credit, there is an explanation for her turning her back on her previous way of life – her problem (And reason for not returning sooner) is that she didn’t believe that Berk (or, more importantly, Stoick) could change. Part of this is because she didn’t even try, but whatever. So when she learns that Berk has changed, courtesy of her son, her reason for staying with the Alpha was lost. Except that immediately after this decision, we discover that – coincidentally – the enclave is under attack in a way that it never has been before, and the Alpha dies with a gentle pat on the face. In other words, as soon as her family turns up, her entire world comes crashing down around her – and she just returns to it!

But there are other problems, too – Astrid being the more deserving recipient of feminist ire, in my view. In the first film, her role was pretty significant; Birk was unsure of who he really was, and what he could do, and she believed in both him and herself, so she kicked his arse into gear and helped him do what he had to do. She was the one to convince him to go and rescue Toothless as his father went off to go fight the dragons on the island. She was by far the best “warrior” of all the kids, and was absolutely integral to Hiccup’s eventual success – so she wasn’t the protagonist, but she a protagonist for sure.

But in the 2nd film? Hiccup still doesn’t truly know who he is, but she can’t help him this time. She doesn’t need to kick his arse, because in pursuit of some sense of identity, he seems more than willing to be reckless and act without his previous film’s characteristic hesitation. Furthermore, whilst she demonstrates her dragon-riding chops at the beginning of the film by winning the “race”, we discover at the end of the film in the closing segment that the only reason she won at the beginning was because Hiccup didn’t bother to turn up. As soon as he does, he wins pretty unequivocally. The final nail is the coffin is that she’s relegated to the same league as the other goonish screen fillers whose role on the first film was at least justifiable by dint of their helping to flesh out the society and culture of Berk. In this film, we already know what Berk is. We know what its relationship with dragons is. At no point do they really help the protagonists in their quest, they primarily just get themselves into scrapes and then get themselves out of them again. And poor Astrid is one of them! She went from being a great example of a female character who not only goes toe to toe with the male hero, but exists entirely independently of him to basically just being his trophy betrothed concubine.

Another weakness, I felt, was the relationship between Hiccup and his parents. I think it was meant to come across in a “He finally thought he could have the two-parent family lifestyle that he always wanted, but it was cruelly snatched from him”, but it came across a lot more like he’d simply swapped one parent for another, especially given Valka’s immediate resumption of previous service, sorry for the disruption, won’t happen again. I think the reason for this is that we never really get the idea that Hiccup’s life suffered as a result of his single parent upbringing – OK, so in the first film we see that his father struggles to relate to him, and his expectations of what his son would be and what his son really is diverge – but it doesn’t play a particularly significant role within the film, because Hiccup manages to overcome all of his problems more or less immediately anyway, thanks to a combination of his own actions and the urgings of Astrid. His father’s major function in the first film is to act as a hurdle to overcome. Furthermore, he was the de-facto prince of Berk, and we can therefore be assumed was granted a pretty plush upbringing. So when he gets the mother he never knew back in his life, it’s hard to avoid asking the question “Well… so what?” When Stoick then bites the dust and Hiccup is back to having a single parent, it’s hard to imagine how this’ll actually has a meaningful difference. This is compounded by his almost immediate forgiveness of Toothless – another excellent potential source of conflict, albeit internal this time – to just combine into the feeling that none of these events surrounding his relationships have much gravity, as they don’t appear to impact his behaviour at all.

Then finally, away from characters, we have the final Act 3 sequence, wherein Toothless defeats the “new” Alpha some 15 minutes after he has become the Alpha and, in turn, becomes the new Alpha. I think the idea of Toothless ending up being the Alpha is great – it makes sense in the context of the mechanics of the film, and it follows the first film’s sense of Hiccup-Toothless symmetry; In the first film, they end up with both missing a limb, not only cementing their bond due to their differences-in-common but also because, even more than before, they need each other to experience a full life – Hiccup can hardly walk let alone fly without Toothless, and Toothless can’t even get out of a small canyon without Hiccup. At the end of HTTYD2, we get that same symetry, with both Toothless and Hiccup becoming the “Alphas” of their respective groups. Hiccup’s heavy cloak over his shoulders at the end is even mirrored by Toothless’ blue back-spikes along his back as a visual sign of a coming of age. I also thought the blind-fold scene, whilst short-lived, was a really great way of re-cementing the bond between Hiccup and Toothless after their brief separation, and it was a great contextual way of demonstrating both Hiccup’s willingness to forgive and Toothless’ willingness to trust his friend.

But did it have to be so uneventful? In the first film, we had a giant, massive dragon that could physically overwhelm Toothless who, whilst powerful for his size, was no where near a match for. He was, however, quick and agile (as long as Hiccup was on his back!) and in the end, it was leveraging this trait that won the day. He faced an almost identical enemy in this film, as he was left fighting a giant, massive dragon that could physically overwhelm him. But his victory was brought about by simply sitting on a rock in a single place and blasting the Alpha in the face with his explosive shots. I understand that the other dragons slowly changing their allegiance and firing on the Alpha was a significant story point, but it didn’t seem like Toothless had really done enough to justify their respect yet. I was hoping for a Test Drive esque sequence of aerial acrobatics where Toothless showed again that it wasn’t all about pure power, but about the application of ability and skill and fighting smart – but he didn’t, he basically reinforced that it was all about brawn, a trait which the entire film had taught us to understand that the Alpha was simply superior in.

Beyond this, there were a few smaller problems that don’t really matter – the “thing” with the wingsuit that Toothless and Hiccup kept trying and failing to do until the end never really made much sense to me. What were they trying to even do? I understand that the heat from Toothless’ blasts was causing Hiccup to rise, but why did they keep flying into rock formations? What was he trying to do when Toothless “saved” him, and what did they do differently at the end to make it “work” (even though they still ended up arse-about-face)? The whole thing just didn’t read clearly to me at all. Similarly, John Snow Eret was basically an entirely irrelevant addition to the cast.

Again, I want to emphasise that this is a critical review – I actually really liked the film and want you all to see it. But these flaws just seem so… obvious and so easy to fix, and I’m not sure why they didn’t. The main story beats would hardly have to change. It could end in the same way, with the same characters in the same positions, if only they altered the journey a little! Maybe next time?!

So, now I’ve got a hold on fiddling with Backburner via MaxScript (whilst trying to avoid its…. subtleties, as described here and here) I’ve been having some fun, using Backburner for some interesting tasks.

One of the most useful but obvious – insomuch as it’s right there in the cmdjob help entry, is submitting After Effects renders to the farm. It’s very easy to write a little bit of code wrapped up in an UI to make the tasklist file described (a comma separated file detailing the name and range of a job). Submit this along with the location of the After Effects .aep file and the comp name and you’re good to go.

But it got me thinking… all you’re really doing when you do this is submit a command via cmd.exe to the machines in question. So… why not go further? The first thing I thought of was a response to a problem we had at work where we received a file from a colleague off-site that contained a plugin none of us had installed, and nor did the farm. We had the choice of either stripping out the offending objects (if we could find the damn things) and then replicating its functionality without using the (free) plugin, or we could go on the usually arduous process of installing the plugin on all the workstations and all the render nodes. It’s just copying some .dlo files into Max’s /plugins/ directory, but still, if only there were some way of giving a universal copy command across the network…

The basic code was very simple – it offered the user the ability to select a file, a destination and select one of the groups (mentioned in the first link up there) on the farm. It then creates a small .bat file which is, effectively, just a command copy the file to the destination (ie COPY “X:\network\file.dlo” “C:\plugins\”, exit 0) and sends a separate job to Backburner for each node in the group, with only that node offered as a server for each job. Once the underlying code was done, I added a few tweaks, such as the ability to add entire folders to the mix, but it’s really just an expansion on this pretty basic concept.

Putting these two examples together – After Effects jobs and Copying files – gives me some fancy ideas for a bit of fun, but the most obvious one to me was… fonts! Unfortunately when you install a Font, it isn’t just a matter of copying it to the Windows /fonts/ folder – there’s also a registry entry that gets added. Otherwise the above script would be enough to install fonts, network wide – very handy if your AE job has a non-standard font. However, it shouldn’t be hard to add a checkbox to the above script, which will effectively add a line to the .bat file that gets generated which performs the appropriate registry tweaks using regedit – I just haven’t done it yet!

But if anyone has any other cool ideas on how Backburner could be leveraged for useful or fun tasks, please let me know! If anyone wants any more information on how to do the above in detail, feel free to email me at dan-grover@dan-grover.com.

Just a quick note to mention that I have an article in this month’s 3D World (March 2013). It’s a Q&A about scene scale and how to deal with it in 3ds Max. I hope those that read it will find it useful!

Another thing that I have discovered during my trials and tribulations with getting Maxscript and Backburner to play nicely is that Groups on Backburner don’t behave as they should under 3ds Max 2013. Before product update 6, you could not really get any information about groups at all – all requests to GetGroupName returned an empty string.

With product update 6 comes some steps forwards – it now returns the correct group name! – but little else. You still can’t reliably return a list of the servers in a group. You cannot create groups, nor can you edit or delete them. This is true whether your connection to the manager has queue control or not. I have, thusly, come up with a solution that started out as a temporary fix until a new product update or 2014 comes around to fix it, but has turned out to work fairly robustly, so I see no need to change, even should groups get fixed!

The process basically involves having groups defined outside of Backburner. In my case, I have a folder full of files – text files, incidentally, though they are never seen by the user – which has, as its file name, the desired name of the “group” and the text file itself contains a comma separated list of server names. These are created, edited and deleted using a simple script that reads and writes the text files. These same files are read in my Backburner submission script, which then supplies the server list contained in the file to backburner (in the case of NetRender in the “job.submit servers:server_array” parameter, and in the case of cmdjob’s as part of the “-servers ‘server1,server2,server3′” etc flag). This has the advantage of being very very quick and simply, as well as robust – so long as no bugs are brought in that mess with the (currently functioning) use of servers in backburner submission.

The main downside is that this is set at submission and, though you can change the contents of the text files defining the groups whenever you like, this doesn’t change already submitted groups. Of course, you can always change which servers are assigned to a job in the backburner monitor, so it’s not like you need to resubmit if you realise there was a problem in the group.

Anyway, that’s my solution, and hopefully it could help someone if they find themselves in a similar position.

I have today finally made progress with a problem I’ve been contending with, on and off, for a few weeks now! The problem is of using MaxScript and Backburner with dependencies. Just submitting Max renders to Backburner using Maxscript isn’t a problem, though for some unutterable reason dependencies are not supported. There are a handful of options out there to try and solve this problem, namely…

- Sending the job to backburner suspended, without any dependencies, then setting the dependencies by editing the (unused) <DependsOn> XML tag in the backburner job folder, before archiving and unarchiving the job (so that the XML file gets re-read). This is problematic as the only way to archive and unarchive a job (also not a function available via MaxScript!) is via Telnet, either through Python or dotNET. I managed to very vaguely get this working using dotNET but Telnet is not the most elegant of things, especially when in an automated system.

- Setting post-render scripts that perform a certain task when a render is complete. The problem here is that the post-render script is called every time a render server finishes its job – which might not necessarily be at the end of the actual job, of course. Whilst this is potentially surmountable by checking to see what frame was rendered in the script, it also meant that each render node needed a licensed copy of Max if it were to open and close files.

- Using cmdjob.exe, the command-line backburner queue magician. This is what I have ended up using.

The solution is actually relatively elegant now. I can’t post actual code as this is for a paid job, but the basic process is this: cmdjob.exe can send jobs to backburner which have dependencies. So what you do is have a cmdjob task launch your actual render tasks. Instead of submitting scenes directly from Maxscript to Backburner, have cmdjob.exe run on Backburner, call up a copy of max and then run a script on it, with the script containing instructions to send a render job to backburner. Because this job is not sent until the cmdjob is executed, and because cmdjob’s can have dependencies, you can effectively have dependencies through maxscript.

Which is easier said than done! So here is a more detailed approach to the process:

- The user loads a scene they want to render.

- When they run the script, a series of other scripts are generated and saved alongside the .max file. These scripts contain all the instructions needed to alter the scene as per the users wish (for example, there may be 3 scripts – one simulates and saves out a particle sequence, one loads this sequence and pre-calculates a GI solution, the third loads both the generated particles and the pre-calculated GI solution and renders the final frames.) and then submit it to Backburner as netrender, before closing the instance of Max.

- Instead of simply sending the first job to the render farm via backburner in MaxScript, a commandline call is made to cmdjob.exe (more info here) to load a copy of max and run a certain script – in this case, the first one that we just generated. The crucial thing to know here is that cmdjob.exe jobs CAN be set to be dependent on other jobs.

- So the cmdjob.exe job is sent to backburner, and is picked up by a render node. This machine needs to have a licensed copy of Max and for that reason I recommend a special machine dedicated to these sorts of tasks. They don’t need a fast processor or fancy graphics card, but they do need a lot of RAM as they will be opening all your max scenes.

- This machine opens max and runs the script that was generated. This script basically contains all the options needed to be useful (such as turning on any particle generators, setting output paths for precalculated GI etc), then sends itself to Backburner via netrender (Job A). Cruicially, it then also submits the NEXT cmdjob.exe task to backburner, dependent on the job it’s just sent (Job B, dependent on Job A). It then closes that instance of Max, and backburner sets the task as finished.

- Next, Backburner finds itself with two new tasks – Job A and Job B from above. Job B is dependent on Job A. So Job A is set off to render, and when it’s finished, Job B begins – and the same process as the previous step begins all over again. This time it loads up the next script, submits that render (Job C), and submits another cmdjob (Job D), again, depedent on the one it just sent.

- This process continues ad-infinitum.

There are a lot of complications here. Do you want to hard-code the whole process of what submits what? My solution was the have the very original script run by the user generate all the scripts (.ms files prefixed with numbers indicating their order) to be used by the cmdjobs, and one of the last lines of each of those scripts was to “fileIn” another script (“fileIn” being the scripting equivalent of XRefing). This script deletes the .ms that called it and looks in the folder to see if there are any more. If there is, it launches a new cmdjob running the first script alphabetically. Thus, when that script runs and submits its next job, it will again delete the script which called it and look for the next. This way, I can have almost infinite scripts all daisy-chain off one another.

This took me a while to work out (thanks in no small part to a few frustrating Max bugs!) but it’s working quite well, with the added benefit of allowing the user to submit jobs to backburner that could previously not have been sent there. If you have any questions about the process, please feel free to email me at dan-grover@dan-grover.com and I’ll try and help as best I can!