Tag Archives: programming

Post navigation

The vCard format has served us well for encoding and exchanging contact information, but there is a better alternative – jCard. In this post I’ll describe why I think jCard can be better than vCard, and should be adopted by every software vendor who deals with contacts.

Both vCard and jCard are text-based formats, but whereas vCard has a unique legacy grammar, jCard is based on JSON. They both represent essentially the same data model that describes contact information: name, phone number, address, e-mail, and so on.

While vCard is widely supported, it is difficult to deal with. Being JSON-based, jCard has the advantage of being ready for use as soon as you parse it. You can even pluck values straight from the JSON object tree you get from the parser. Like vCard versions 3.0 and 4.0, jCard is an IETF proposed standard, RFC 7095.

Both of them represent a very complex data model, and both are semantically quite complicated. Still, I maintain that it is time to leave vCard behind and move on, because… well, read on.

vCard – hard to parse, hard to generate, almost easy to read

On the surface, vCard seems simple enough. It is a line-based format, where each line has a delimited format. Here is a simple sample of a vCard from randomuser.me, converted to vCard 3.0 by hand:

Doesn’t seem to bad, does it? Well, I’ve written my share of ad hoc vCard parsers and generators in the past (in Objective-C, Java, and C#), and never went all the way, because when you start looking into the corner cases and the special handling required for many fields, you want to run away and hide.

First of all, the lines are not just lines. They could continue on to the next, and the process of continuation is complex, but not as complex as joining them back together, so most naïve vCard handlers just ignore continuation. By the way, according to the specification the line delimiter is a CR-LF pair.

The properties are separated from the values with a colon. Some properties may have multiple fields, so those need to be separated with a semicolon. Some properties may have multiple values, so those need to be separated with a comma. The fun begins if you need to have any of those separators in a value (it could happen, you know). They need to be escaped with a backslash. If you need to have a backslash, it’s time for another backslash. And newlines in values need to be expressed just like in many programming languages, with \n. It’s complicated.

So vCard may be quite human-readable, but that is neither here nor there, because vCards are not meant for eyeballing. They are meant for information exchange between computer systems. That also brings up the not insignificant issue of character encodings.

Y U NO I18N?!

There are several versions of vCard out there. The latest version, 4.0 (which jCard is also based on), sensibly mandates UTF-8 as the character encoding of the vCard. Unfortunately nobody seems to support vCard 4.0.

Version 3.0 of the vCard specification says the character encoding must be specified in the Content-Type MIME header field, using the charset parameter. That is fine, unless you don’t use HTTP to transmit the vCard, at which point you either agree between parties or guess.

Poor old version 2.1 from 1996, still in active duty, has the ASCII character encoding as the default, unless you specify something else with the CHARSET parameter. For every property. As you can imagine, nobody does that, and then we get stuff like “KÃ¤pyaho” for my last name (and I really resent that).

The default character encoding for JSON is UTF-8, which is really nice. If you don’t trust your endpoints, you can always \uXXXX escape everything beyond ASCII in your JSON string values.

(Sorry if the formatting is a mess, I’m trying to figure out how to present it. If you see code tags, they are not part of the jCard.)

As you can see, the jCard has a standard JSON syntax, but the content itself is not typical JSON. It is a polymorphic array, containing string and arrays. The top-level object is an array, so that you can have multiple contacts in one payload.

All the vCard properties are arrays where the first item is a string identifying the property. The rest are objects, simple values or arrays, based on the property. The jCard specification goes into great detail explaining how they are constructed.

The JSON example above is verbose, because it has been pretty-printed to be eyeballed by a human. If you were to condense it for online transmission, it would be like this:

I only cheated a little by inserting a couple of newlines, but otherwise it’s tight, no-nonsense JSON.

The devil is in the details

So, vCard is difficult to parse, but jCard is not, at least for a JSON parser. However, semantically jCard is still just as complicated.

There are obviously inherent difficulties in representing contact information, and while jCard sidesteps the obvious issues of parsing and generating, the semantics are still there to be dealt with. Some might argue that it would have been better to create a whole new data model for jCard, instead of defining a JSON-based equivalent, but I don’t think it would have made much difference, going by the low adoption rate of jCard by major vendors.

Depending on the programming language, it is relatively easy or quite painful to deal with the results from a parsed jCard. If you have dynamic typing, you can just skate away. But if you have a strongly typed language like Java or Swift, you will need to be creative.

I want/need to parse jCards in an iOS application that I’m in the process of converting to Swift from Objective-C. Last time I got all the way to Swift 2.2, but then the work on the app stalled for a while, and in the meantime, Swift 3 happened. Earlier I had written some helpers to get from jCard to the iOS Contacts framework, but they did some naughty things which are not allowed in Swift 3 anymore.

So I decided to bite the bullet and create a proper library for dealing with jCards in Swift, called ContactCard. It is open sourced under the MIT License, and is up on GitHub if you would like to have a look or contribute. It’s very early alpha, so I expect it to improve a lot in a short time as I prepare it for the app. I intend to eventually publish it on CocoaPods too, and I hope to write some more about how I intend to solve the problems of polymorphic values in jCard properties.

Drones, or Unmanned Aerial Vehicles (UAVs), be they quadcopters or other type of flyer, will become more “intelligent” as themselves or by forming swarms, as this TED Talk by Vijay Kumar at the University of Pennsylvania shows:

My interest in drones lies not in flying them myself live, because I’m a lousy pilot and don’t play games much anyway, but in making them follow a predetermined route and return back to the starting point – for example, surveying an object or estate, or even carrying cargo between waypoints. The gorgeous aerial shots you get with many drones these days are great, of course, but I’ll let others play the director, and instead concentrate on the programming.

I did some web searches on the programmability of the AR.Drone, and it seems that the biggest craze has faded a little bit. Many of the libraries for Python and Node.js are not seeing as active development as I would have thought, and groups like NodeCopter are not too active either.

It also seems that some active members have moved on to do greater things, like Fleye, a personal flying robot – the result of work by Laurent Eschenauer and Dimitri Arendt:

There is also the python-ardrone library for Python, which I would prefer over Node.js.

I have tested both node-ar-drone and python-ardrone quickly with the AR.Drone 2.0, and it is an exhilarating experience to see your quadcopter come to life and rise up to hover, just by entering a few commands in the Node or Python REPL. (Just make sure you can quickly call the land() function, especially if you are experimenting indoors.)

There are also some Clojure libraries for controlling the AR.Drone, such as clj-drone and turboshrimp, but I’m not sure if I would want to add JVM to the mix.

My inspiration for programming drones actually got sparked by the O’Reilly Programming Newsletter, which featured a recent article by Greg on The Yhat Blog titled “Building a (semi) Autonomous Drone with Python“. It had a lot of tips about how to start with this kind of activity, and extending it to involve computer vision using OpenCV.

I intend to develop some applications that fly the AR.Drone automatically along the perimeter of a large object, such as a house, or along some predetermined line, like the side of a field. I hope to document some of the results in this blog.

If you’re interested in programming semi-autonomous drones, drop me a line with any ideas, tips, questions, or collaborations.

HipStyles was removed from the App Store over a year ago, but from time to time I still think it could be resurrected. However, after taking a long hard look at Apple’s new Photos framework in iOS 9, I think it is just not feasible anymore.

HipStyles and the Assets Library Framework

HipStyles was a photo finder for Hipstamatic and Oggl photos, looking for snaps taken with some Hipstamatic lens, film or flash of your choosing. It was developed because it was increasingly difficult to find and categorize the great shots you took with the many virtual vintage films and lenses available in Hipstamatic. They still publish new gear monthly, and Eric Rozen’s Hipstography does a great job in covering them.

Technically, what HipStyles did is look at your iPhone’s photo album, pull out certain metadata from the photo, and present the results. When Hipstamatic saves a photo, it writes the gear information in some fields in the {TIFF} metadata dictionary. You can observe this yourself if you use some metadata reader application on a Hipstamatic photo.

Using the old Assets Library Framework this was relatively easy, if not too fast. Basically you would just get an instance of the assets library, and then loop through the photo assets and act on the information:

In practice, it is now a lot easier to load photos from the user’s photo library, which could now be completely or partially in the cloud. If you have turned on the iCloud Photo Library feature in iOS, the master copies of your photos will be in cloud storage, and the Photos application (or some app that you developed yourself) will manage the downloading and caching of the assets as necessary.

Once you have the assets, you would then go on picking stuff from the metadata.

Initially it sounded like a no-brainer to just make HipStyles use the new Photos framework. However, the Photos framework does not expose all the metadata that HipStyles needs. Specifically the {TIFF} metadata is not available without getting all the image data first.

When you get the image data, it is possible to extract the metadata. This is a function to handle the results of an image request:

On the surface, it sounds simple: just get the image data for each image, extract the metadata, and do whatever you need with it. There is just one problem, and it’s a big one: you need to get all the image data first. All of it.

Consider a user that has a couple of thousand photos in their photo library, and they have decided to store all their photos in iCloud. Each of these photos could be multiple megabytes in size. We don’t know which of those photos are interesting for HipStyles, and we can’t know unless we get the image data of each and every one candidate. So you will create a PHCachingImageManager, set its options, configure some image request options, and send the image manager on a mission:

Then you wait… and wait… and wait. The image manager will happily retrieve all the photos from iCloud over the network. I don’t know about you, but my vintage iPhone 5 from the year 2012, with 32 GB of storage and full of apps, otherwise happily running iOS 9.2.1 with no problems, will freeze and reboot when storage gets absolutely full. So I experienced a couple of crashes when I tested the photo retrieval code above, because the image manager dutifully filled my storage with local copies of the photos from iCloud. All because of me wanting to get at the some photo metadata that was not available from the PHAsset class.

You can make the metadata retrieval easier for you if use a helper like PHAsset+Utility, but I don’t think it will make the root problem go away: to get at certain pieces of photo metadata, you need to get all the image data first.

In iOS 9, the Assets Library Framework is deprecated in favour of the new Photos framework. There is obviously no guarantee about how long the old Assets Library framework will be operational anymore, and anyway it doesn’t handle the iCloud photo library use case at all.

If you have any ideas, by all means send them my way. But at this point I’m convinced that it is not feasible to recreate HipStyles using the Photos framework.

However, Carin Meier’s Living Clojure is excellent in many ways. Get it from O’Reilly (we’re an affiliate):

My little tutorial started with part zero, in which I lamented how functional programming is made to appear unlearnable by mere mortals, and it kind of snowballed from there. Hope you like it and/or find it useful!

In the last four installments of Functional programming without feeling stupid I’ve slowly built up a small utility called ucdump with Clojure. Experimentiing and developing with the Clojure REPL is fun, but now it’s time to give some structure to the utility. I’ll package it up as a Leiningen project and create a standalone JAR for executing with the Java runtime.

Creating a new project with Leiningen

You can use Leiningen to create a skeleton project quickly. In my project’s root directory, I’ll say:

lein new app ucdump

Leiningen will respond with:

Generating a project called ucdump based on the 'app' template.

The result is a directory called ucdump, which contains:

.gitignore README.md project.clj src/
LICENSE doc/ resources/ test/

For now I’m are most interested in the project file, project.clj, which is actually a Clojure source file, and the src directory, which is intended for the app’s actual source files.

Leiningen creates a directory called src/ucdump and seeds it with a core.clj file, but that’s not what actually what I want, for two reasons:

I want ucdump to be a good Clojure citizen, so I’m going to put it in a namespace
called com.coniferproductions.ucdump.

My Git repository for ucdump also contains the original Python version of the application, which is in <project-root>/python, and I want the Clojure version to live in <project-root>/clojure.

In the previous parts of “Functional programming without feeling stupid” we have slowly been building ucdump, a utility program for listing the Unicode codepoints and character names of characters in a string. In actual use, the string will be read from a UTF-8 encoded text file.

We don’t know yet how to read a text file in Clojure (well, you may know, but I only have a foggy idea), so we have been working with a single string. This is what we have so far:

Welcome to the third installment of “Functional programming without feeling stupid”! I originally started to describe my own learnings about FP in general, and Clojure in particular, and soon found myself writing a kind of Clojure tutorial or introduction. It may not be as comprehensive as others out there, and I still don’t think of it as a tutorial — it’s more like a description of a process, and the documented evolution of a tool.

I wanted to use Clojure “in anger”, and found out that I was learning new and interesting stuff quickly. I wanted to share what I’ve learned in the hope that others may find it useful.

Some of the stuff I have done and described here might not be the most optimal, but I see nothing obviously wrong with my approach. Maybe you do; if that is the case, tell me about it in the comments, or contact me otherwise. But please be nice and constructive, because…

…in Part 0 I wrote about how some people may feel put off by the air of “smarter than thou” that sometimes floats around functional programming. I’m hoping to present the subject in a friendly way, because much of the techniques are not obvious to someone (like me) conditioned with a couple of decades of imperative, object-oriented programming. Not nearly as funny as Learn You a Haskell For Great Good, and not as zany as Clojure for the Brave and True — just friendly, and hopefully lucid.

In Part 1 we played around with the Clojure REPL, and in Part 2 we started making definitions and actually got some useful results. In this third part we’re going to take a look at Clojure functions and how to use them, and create our own — because that’s what functional programming is all about.

In this installment of “Functional programming without feeling stupid” I would like to show you how to define things in Clojure. Values and function applications are all well and good, but if we can’t give them symbolic names, we need to keep repeating them over and over.

Before we start naming things, let’s have a look at how Clojure integrates with Java. I’m assuming you are still in the REPL, or have started it again with lein repl.

Track 1: “If anyone should ask / We are mated”

As it happens, Clojure’s core library is lean and focused on manipulating the data structures of the language, so many things are deferred to the underlying Java machinery as a rule. For example, mathematical computation is typically done using the static methods in the java.lang.Math class:

user=> (java.lang.Math/sqrt 5.0)
2.23606797749979

As you can see, this is a function application like we have already seen in Part 1, but this time the function we are using is the sqrt static method in the java.lang.Math class.

Java 7 acquired Unicode character names, and they are accessed through
the getName method in the java.lang.Character class. This is no mean feat, since there are over 110,000 characters in the Unicode standard, and each of them has a name (although some of them are algorithmically generated). To find out the canonical character name of a Unicode character, such as the euro currency symbol, you would use the getName static method:

Hey, what’s wrong? Well, if you look up the documentation of java.lang.Character.getName, you will find out that it takes an int value as an argument, not a character. You can actually do this check from inside the REPL:

In my recent post Functional Programming Without Feeling Stupid I took a quick look at how functional programming can be a little off-putting for the non-initiated. I promised to provide some examples of my own first steps with FP, and now I would like to present some to you.

Advocates of functional programming often refer to increased programmer productivity. At least some of that can be attributed to the REPL, or the Read-Evaluate-Print Loop. We are basically talking about an environment which accepts and parses any code you type in, and gives you a place to experiment and see results quickly. Before interpreted or semi-interpreted languages like Python, Java and JavaScript became mainstream, the typical repeating cycle in software development was Compile-Link-Execute, and debugging meant observing special output on the console. In the 1990s integrated debuggers with watches and breakpoints became the norm, but long before that Lisp-like languages already had a REPL, and Python also acquired one.

If you are thinking about getting intimate with Clojure, you will need to get to know the REPL. It is your playground, and will always be, even if you later start packaging and organizing your code.

Clojure depends on the Java Virtual Machine (JVM) and is actually distributed as a normal JAR file (Java ARchive), like most Java libraries are. You can start Clojure from the JAR, but you will save yourself some trouble and prepare for the future if you install Leiningen, the dependency management tool for Clojure. It is simple to install and run, and I will assume that you will follow the instructions on the Leiningen web site sooner or later. Now would be a good time.

When you’re done with the installation, you only need to say

lein repl

to start a Clojure REPL. I’m using OS X, so what I describe here was done from Terminal. You don’t need to create a project with Leiningen if you just want to play around in the REPL.

Of course, if you don’t have Java installed, you need to get it first. Refer to the Java web site of Oracle for details as necessary. Furthermore, some of the things I will describe require Java version 7 or later.

If you follow software design trends (yes, they exist), you may have noticed an increasing amount of buzz about functional programming, and particularly the Clojure language. While functional programming is hard to define, almost everyone mentions pure functions, the lack of side effects and state, and easy parallelisation. As for Clojure, it is all about (a kind of) Lisp running on the Java Virtual Machine (and .NET, and transformed to JavaScript).

I’m somewhat convinced that functional programming is at least worth knowing about and trying out, even if you don’t expect to fully convert. It has been said that learning about the functional paradigm makes you a better programmer in your current imperative language. Functional languages reduce accidental complexity, and that helps you focus.

“Whoop de doo, what does it all mean, Basil?”

If you have a background in imperative languages, you will have an interesting time if and when you start digging into functional programming, because whatever else it is, it’s different. And I’m not talking about syntax only, but most of what you do. If you need to add an item to a list, you construct a new list with the new item appended to the previous list (no, it is not as inefficient as it sounds, because there is great stuff under the hood to handle that). This is because immutability is one of the cornerstones of functional programming. If you can’t change something after it is created, there is no state to mess up. You program with values, not stateful objects.

I see I’m getting myself tricked into presenting a definition of functional programming, when that has been done better elsewhere. For pointers, see Michael Fogus’ 10 Technical Papers Every Programmer Should Read (At Least Twice), including the classic “Why Functional Programming Matters” by John Hughes. But I actually wanted to talk about something else.