Sometimes a coder who writes, sometimes a writer who codes. Formerly, a coder who edited magazines.

Main menu

Post navigation

fun-ny Faces : Face-based Augmented Reality with F# and the iPhone X

Each year, the F# programming community creates an advent calendar of blog posts, coordinated by Sergey Tihon on his blog. This is my attempt to battle Impostor Syndrome and share something that might be of interest to the community, or at least amusing…

I was an Augmented Reality (AR) skeptic until I began experimenting with iOS 11’s ARKit framework. There’s something very compelling about seeing computer-generated imagery mapped into your physical space.

A feature of the iPhone X is the face-tracking sensors on the front side of the phone. While the primary use-case for these sensors is unlocking the phone, they additionally expose the facial geometry (2,304 triangles) to developers. This geometry can be used to create AR apps that place computer-generated geometry on top of the facial geometry at up to 60FPS.

Getting Started

In Visual Studio for Mac, choose “New solution…” and “Single-View App” for F#:

The resulting solution is a minimal iOS app, with an entry point defined in Main.fs, a UIApplicationDelegate in AppDelegate.fs, and a UIViewController in ViewController.fs. The iOS programming model is not only object-oriented but essentially a Smalltalk-style architecture, with a classic Model-View-Controller approach (complete with frustratingly little emphasis on the “Model” part) and a delegate-object pattern for customizing object life-cycles.

Although ARKit supports low-level access, by far the easiest way to program AR is to use an ARSCNView, which automatically handles the combination of camera and computer-generated imagery. The following code creates an ARSCNView, makes it full-screen (arsceneview.Frame ← this.View.Frame) and assigns it’s Delegate property to an instance of type ARDelegate (discussed later). When the view is about to appear, we specify that AR session should use an ARFaceTrackingConfiguration and that it should Run:

Once the AR session is running, it adds, removes, and modifies ARSCNNode objects that bridge the 3D scene-graph architecture of iOS’s SceneKit with real-world imagery. As it does so, it calls various methods of the ARSCNViewDelegate class, which we subclass in the previously-mentioned ARDelegate class:

As you can see in DidAddNode and DidUpdateNode, we’re only interested when an ARFaceAnchor is added or updated. (This would be a good place for an active pattern if things got more complex.) As it’s name implies, an ARFaceAnchor relates the AR subsystems’ belief of a face’s real-world location and geometry with SceneKit values.

The Mask class is the last piece of the puzzle. We define it as a subtype of SCNNode, which means that it can hold geometry, textures, have animations, and so forth. It’s passed an ARSCNFaceGeometry which was ultimately instantiated back in the ViewController (new ARDelegate (ARSCNFaceGeometry.CreateFaceGeometry(arsceneview.Device, false)). As the AR subsystem recognizes face movement and changes (blinking eyes, the mouth opening and closing, etc.), calls to ARDelegate.DidUpdateNode are passed to Mask.Update, which updates the geometry with the latest values from the camera and AR subsystem:

Or we can engage in virtual soccer-hooligan face painting with mat.Diffuse.ContentImage ← UIImage.FromFile "fsharp512.png" :

The real opportunity here is undoubtedly for makeup, “face-swap,” and plastic surgery apps, but everyone also loves a superhero. The best mask in comics, I think, is that of Watchmen’s Rorschach, which presented ambiguous patterns matching the black-and-white morality of its wearer, Walter Kovacs.

I’ll admit that so far I have been stymied in my attempt to procedurally-generate a proper Rorschach mask. The closest I have gotten is a function that uses 3D Improved Perlin Noise that draws black if the texture is negative and white if positive. That looks like this:

Other things I’ve considered for face functions are: cellular automata, scrolling green code (you know, like the hackers in movies!), and the video feed from the back-facing camera. Ultimately though, all of that is just warm-up for the big challenge: deformation of the facial geometry mesh. If you get that working, I’d love to see the code!

In my opinion, the bottleneck on a DNN solution is not training, but data preparation and iterating your model to the point where it’s reasonable to start investing kilowatt-hours of electricity to the training. So I have Tensorflow on all my machines, including my Macs, even though as of Tensorflow 1.2 GPU support is simply not available for Tensorflow on the Mac. (I’m not sure what’s going on, but suspect it may have something to do with licensing NVidia’s CuDNN library.)

Having said that, GPU support for TensorFlow is much faster than CPU-only Tensorflow (in some quick tests on my Windows laptops, ~8x). With GPU-supported Tensorflow, it’s that much easier to iterate your model until your training and validation curves start to look encouraging. At that point, in my opinion it makes sense to move your training to the cloud. There’s a little more friction in terms of moving data and starting and stopping runs and you’re paying for processing, but hopefully you’ve gotten to the point where training time is the bottleneck.

Anyway…

Mac Tensorflow GPU: I’d like to think this will change in the future, but as of August 2017: Nope.

There are a very few people who seem to have figured out how to build Tensorflow with GPU support on the Mac from sources, but the hoop-jumping and yak shaving that seems necessary seems very high to me.

I think the “cp35” is the hint that you have to use Python 3.5, so if the page at https://www.tensorflow.org/install/install_windows changes to show a different .whl file, you’d have to set the python in your Anaconda environment differently.
– Validate that you’ve got GPU capability:

import tensorflow as tf
tf.Session().run(tf.constant('hello'))

This should result in a cascade of messages, many of which say that Tensorflow wasn’t compiled with various CPU instructions, but most importantly, towards the end you should see a message that begins:

Creating Tensorflow device (/gpu:0)

which indicates that, sure enough, Tensorflow is going to run on your GPU.

Sunday rant: I have a 2016 Dell XPS (high-end) laptop. I bought a Dell 25″ 4K monitor. And on Friday received a Dell Thunderbolt dock for the monitor. I plug it all together and although the monitor displays wonderfully, the dock is not passing USB through. So I start fiddling around with “unplug from dock, plug into laptop, confirm the peripheral is working,” stuff. And then the laptop BSODs. Machine boots, connects to dock, everything’s fine for 30 seconds, BSOD. Start to Google. “Update your laptop BIOS.” (For a fucking docking station!). It takes a goddamn hour to find the BIOS update on the Dell Website for their goddamn flagship laptop, but whatever.

Still BSODs. Now it’s telling me that I have to update the firmware on the dock. But I cannot update the firmware because if I attach the dock to the laptop to update it, it BSODs. So, there’s this few-second window before the BSOD where I see that I have to update my Thunderbolt Driver on the laptop.

So I download the driver and run the installer for the Thunderbolt Driver. The installer doesn’t give any option other than “Uninstall.” So I say “OK, I’ll uninstall and reinstall.” I uninstall. Fine. I go to reinstall. I’m told I don’t have sufficient permission. So I run as administrator. I still don’t have sufficient permission. So I end up editing the registery to turn off user protection. (Remember, this is all for a docking station).

I now can run the “install” option, but it refuses to continue because it sees some pre-existing value in the registry. (Which I take to mean it’s “Uninstall” function didn’t actually, you know, uninstall.) It then rolls back the Thunderbolt install and leaves me with my current situation:

A half-upgraded machine with user access protection turned off, less functionality than it had before, and it still BSODs whenever I turn on the dock. All with a respected company’s flagship hardware.

Although I am pretty deeply entrenched in the Apple ecosystem, the recently-announced $50 Dot was so inexpensive I could not resist checking it out. (Before I go further: I work for Microsoft, so take that into account as you see fit.)

Out of the box, the Echo is very easy to setup for basic queries “Alexa, what’s my latitude and longitude?” and so forth. The Echo has a relatively lo-fi speaker and the integration with Sonos (what Amazon calls an “Alexa Skill”) is not yet available, so I haven’t used it all that much.

But there’s an API so you know I had to program something. My preferred solution for “computations in the cloud” is definitely Azure Functions written in F#, but for my first Alexa Skill I used Amazon Lambda running Python.

The first thing to focus on is that Alexa Skills are a separate service that can be programmed many ways, so there’s always going to be a certain amount of integration overhead in the form of multiple tabs open, jumping back and forth between the Alexa Skills and the Web server/service where you are handling the computation.

The Alexa Skills documentation is good, but there’s a good number of parts and I think it’s wise to write your first skill using Amazon Lambda, as I did. Amazon Lambda is often the default service in the documentation and there are often hyperlinks to the Lambda-specific page to do “X.”

A Skill for Gravity

A friend was talking to me about riflery and astonishing me with the flight times he was talking about. Alexa failed to answer some basic questions about ballistics (Alexa seems to me less capable than Google Assistant, Cortana, or Siri at answering freeform questions), offering me the perfect simple use-case for my first skill.

Minimum viable query: "What is the speed of an object that has fallen for 1.5 seconds?"

SWAG achievable: "How long would it take for an object dropped from the height of the Empire State Building to fall to the ground on Mars?"

The nice thing about my minimal query is that it’s both stateless and easy to answer with some math: all you need to answer is the duration of the drop and use a gravitational constant of -9.81. (Conversions from meters/second can come later.)

I followed the documentation on building an Alexa skill with a Lambda function to create an Alexa Skill named called “Gravity.” After naming, the next page of the Skill development site is “Interaction Model.” This is where I was shocked to discover:

Alexa doesn’t do natural language processing!

I ASS-U-ME’d that I would be receiving some programmatic structure that told me the “nominal subject” of the sentence was the noun speed and would allow me to search for a “prepositional modifier” whose “object” was the noun seconds and extract its modifier. That would allow me to recognize either of these sentences:

But no! As you can see in the screenshot below, the mapping of a recognized sentence to a programmatic “intent” is nothing but a string template! You either have to anticipate every single supported structure or you have to use wildcards and roll your own. (Honestly, I imagine that it’s not a long road before the wisest interaction model is Parse {utterance}.)

To be clear: ‘just’ voice recognition is extraordinarily hard and doing it in ambient environmental noise is insane. It’s only because Alexa already does this very, very hard task that it’s surprising to me that they don’t provide for some amount of the (also hard) task of parsing. The upside, of course, is that sound-&gt;utterance is decoupled from utterance-&gt;sentence. As far as I know, no one today provides “NLP as a Service” but it’s easy to imagine. (Although latency… Nope, nope, staying on topic…)

Returning to the screenshot above, you can see that it contains the bracketed template {duration}. The matching value will be associated with the key duration in calls to the Lambda function. And, to be honest, it’s a place where Alexa Kit does do some NLP.

You can help Alexa by specifying the type of the variables in your template text. For instance, I specified the duration variable as a NUMBER. Alexa does use NLP to transform the utterances meaningfully — so “one and a half” becomes “1.5” and so forth. I haven’t really explored the extent of this — does it turn “the Tuesday after New Year’s Day” into a well-formed date and so forth?

Alexa packages session data relating to an ongoing conversation and intent data and performs an RPC-like call (I actually don’t know the details) to the endpoint of your choice. In the case of Amazon Lambda, that’s the Amazon Resource Name (ARN) of your function.

The values in the session object relate to a conversation and the values in the request object belong to a specific intent — in this case the FallingSpeedIntent with the duration argument set to “1.5”.

On the Lambda side of things

Amazon Lambda has a template function called ColorIs that provides an easy starting point. It supports session data, which my Gravity skill doesn’t require, so I actually ended up mostly deleting code (always my favorite thing). Given the JSON above, here’s how I route the request to a specific function:

[code lang=”python”]
def on_intent(intent_request, session):
""" Called when the user specifies an intent for this skill """

I remember a time when “learning about computers” invariably started with the phrase “computers only operate on 0s and 1s…” Things could vary a little for a few minutes, but then you’d get to the meat of things: Boolean logic. “All computer programs are formed from these ‘logic gates’…”

I remember a poster that illustrated Boolean logic in terms of punching. A circuit consisted of a bunch of mechanical fists, an “AND” gate propagated the punch when both its input were punched, an “OR” required only one input punch, etc. At the bottom were some complex circuits and the ominous question: “Are you going to be punched?” Because Boston. (The answer was “Yes. You are going to be punched.”)

Anyway, the point is that while there was a fundamental truth to what I was being told, it was not overwhelmingly relevant to the opportunities that were blossoming, back then at the dawn of the personal computer revolution. Yes, it’s important to eventually understand gates and circuits and transistors and yes, there’s a truth that “this is all computers do,” but that understanding was not immediately necessary to get cool results, such as endlessly printing “Help, I am caught in a program loop!” or playing Nim or Hammurabi. Those things required simply typing in a page or two of BASIC code.

Transcription being what it is, you’d make mistakes and curiosity being what it is, you’d mess around to see what you could alter to customize the game, and then your ambition would slowly grow and only then would you start to benefit from understanding the foundations on which you were building.

Now, just as “it’s all just 0s and 1s” is both true but not immediately necessary, “it’s all just weights and transfer functions,” is something for which_eventually_ you will want to have an intuition. But the breakthroughs in recent years have not come about so much because of advances at this foundational level, but rather from a dramatic increase in sophistication about how neural networks are “shaped.”

Not long ago, the most common structure for an artificial neural network was an input layer with a number of neural “nodes” equal to the number of inputs, an output layer with a node per output value, and a single intermediate layer. The “deep” in “deep learning” is nothing more than networks that have more than a single intermediate layer!

Another major area of advancement is approaches that are more complex than “an input node equal to the number of inputs.” Recurrence, convolution, attention… all of these terms relate to this idea of the “shape” of the neural net and the manner in which inputs and intermediate terms are handled.

… snip descent into rabbit-hole …

The Keras library allows you to work at this higher level of abstraction, while running on top of either Theano or TensorFlow, lower-level libraries that provide high-performance implementations of the math-y stuff. This is a Keras description of a neural network that can solve the XOR logic gate. (“You will get punched if one, but not both of the input faces gets punched.”)

I’m not claiming that this should be crystal clear to a newcomer, but I do contend that it’s pretty dang approachable. If you wanted to produce a different logic gate, you could certainly figure out what lines to change. If someone told you “The ReLu activation function is used more often than sigmoid nowadays,” your most likely ‘let me see if this works’ would, in fact, work (as long as you guessed you should stick with lowercase).

For historical reasons, solving XOR is pretty much the “Hello, World!” of neural nets. It can be done with relatively little code in any neural network library and can be done in a few dozen lines of mainstream programming languages (my first published article was a neural network in about 100 lines of C++. That was… a long time ago…).

But Keras is not at all restricted to toy problems. Not at all. Check this out. Or this. Keras provides the appropriate abstraction level for everything from introductory to research-level explorations.

Now, is it necessary for workaday developers to become familiar with deep learning? I think the honest answer to that is “not yet.” There’s still a very large gap between “what neural nets do well” and “what use-cases are the average developer being asked to addressed?”

But I think that may change in a surprisingly short amount of time. In broad terms, what artificial neural nets do is recognize patterns in noisy signals. If you have a super-clean signal, traditional programming with those binary gates works. More importantly, lots of problems don’t seem easily cast into “recognizing a pattern in a signal.” But part of what’s happening in the field of deep learning is very rapid development of techniques and patterns for re-casting problems in just this way. So-called “sequence-to-sequence” problems such as language translation are beginning to rapidly fall to the surprisingly effective techniques of deep learning.

… snip descent into rabbit-hole …

Lots of problems and sub-problems can be described in terms of “sequence-to-sequence.” The synergy between memory, attention, and sequence-to-sequence — all areas of rapid advancement — is tipping-point stuff. This is the stuff of which symbolic processing is made. When that happens, we’re talking about real “artificial intelligence.” Artifical intelligence, yes, but not, I think, human-level cognition. I strongly suspect that human-level, general-purpose AI will have a trajectory similar to medicine based on genetics: more complex and messy and tangled to be cracked with a single breakthrough.

If you’re developing for iOS 10 and your app “silently” crashes (especially if it’s an older app), the culprit could well be the increased privacy requirements in iOS 10. Namepaces such as HomeKit now require specific privacy-related keys to be in your info.plist (for instance, NSHomeKitUsageDescription). If you don’t have them, the system automatically closes your application without an exception or Console.log message (if you run in the simulator, you may see a PRIVACY_VIOLATION notice in the stack trace).

If you have the URL of a streaming video, it’s easy to display on an AppleTV, even though tvOS does not have a UIWebView (which would make it really easy). You have to use some AVFoundation code, such as: