Protocol Buffers in your Kitura Apps

There is so much time that a full-stack developer unfortunately spends dealing with just the serialization and deserialization of his or her data model. This is why I am really excited for any tools that helps break up that monotony, get me beyond that, and allow me to focus on working on the parts of my application that makes it really unique. Fortunately, we are beyond the days of parsing XML’s from DTD and now mostly using simpler object notation, namely JSON. However, there are some notable problems with JSON:

It’s text. So it’s not as compact and efficient as it could be. This sometimes is not an issue, but when a response contains a large payload, this becomes much more important.

It’s not typed. Makes no guarantees about the existence of fields, and also the data type of the fields inside of them. You can’t guarantee that the JSON that the client sends to the server matches the JSON that the server knows how to parse.

Protocol Buffers

What we want is to be able to specify a unified way of representing your application’s data model so that it can be shared regardless of language or platform it’s used on. This is what Protocol buffers are designed to achieve. It was designed at Google, and it is a language-neutral, platform-neutral language for serializing structured data. You write your model in the Protobuf language, run the code generation tool called protoc, and an automatically generated struct (or class) that matches that structured data type will be produced for you. Protobufs have been out for a while, but only for C++, C#, Go, Python, Java languages.

Fortunately, Apple has recently released an open source plugin for protoc that enables you to build Swift structs from the Protobuf language. Once these structs are generated, it allows these structs to be serialized and deserialized both as binary Protobuf objects or as JSON. Currently, the Swift Package Manager has limited integrated support for Protocol Buffers, but in the future, we expect it to be more closely integrated. For now, you must run some of these tools manually.

Let’s direct our focus on how to integrate Protocol Buffers with a Kitura REST web service.

Create a Kitura project

Creating and structuring a Kitura application is outside of the scope of this article, but there are some good starters online. For instance, there is the IBM Cloud Kitura Starter that has everything you need to get started. You can also download a finished version of this work from my Kitura-Protobufs repository.

The basic Kitura application contains routes for getting tasks and running the web server. It is important to add the BodyParser to the routes so that the HTTP body can be read from the requests.

If you are using XCode to build you Kitura application, it needs to know how to find the Protobuf.framework file. First download the Swift Protobuf runtime from the repository, open the SwiftProtoRuntime.xcodeproj and build the project in XCode. Locate the Protobuf.framework file probably in your Derived Data directory, and drop it in your Kitura App’s XCode project.

Create a Proto file

The following model example came directly from the swift-protobuf guide. It has a BookInfo and a collection of books, called MyLibrary. Save it as BookLibrary.proto.

Build Swift structs

After you run the code generator, it will produce a Swift struct that has some nice properties. For instance, it will conform to Equatable, so that it can be compared against other structs based on the properties inside of the struct, and also Hashable so that it can be inserted in to a Dictionary and looked up quickly. Additionally, it has some mutating properties that make using the structure efficient. In particular, Copy-on-Write (CoW) semantics. Struct’s are value types in the Swift language, and properties inside of them are by default immutable. Therefore, if you make a copy of a struct, the Swift language often doesn’t have to actually make a copy of the information on the stack, and instead just use a reference instead. This makes using structs in many times very efficient. However, if you end up mutating values, a new structure has to be deeply copied. There are some interesting optimizations that the protoc generator can do to add CoW semantics on these structs so that they only have to be deeply copied if written to.

The following command will create BookLibrary.pb.swift in the present working directory. You can now import this Swift file into your Swift project by placing the Swift in Sources next to your other code in your application.

protoc --swift_out=. BookLibrary.proto

Get a book with Protocol Buffers

In this example, we look at the Accept header to either return JSON or a binary protobuf back to the client based on its setting and the MIME type for each. There currently does not exist a special MIME type for Protocol Buffers except for octet-binary. There have been proposals made, but nothing officially accepted.

Add a book

With this route, we read the Content-type. If the Content-Type is application/json, then try to create a BookInfo from JSON with the rawString. If the body contains binary information, the BookInfo is created with the Foundation struct, Data directly.

Getting back a binary protobuf for the book

Getting back JSON for the book

Add a book by sending JSON

Next Steps

Now that you have a Kitura web service that can consume Protobufs or JSON using a data model that has been automatically generated for you, you can now proceed to develop many interesting end-to-end applications like the following:

Use your generated BookLibrary.pb.swift file in an iOS project, and make calls to the Kitura server.

Build a BookLibrary.java file and import it into your Android project, and make calls to Kitura server.

I am a technical team lead in the Swift@IBM Engineering group. I am passionate about learning how to architect the entire end-to-end stack to use entirely Swift. I have interests in machine learning and IoT. Outside of writing code, I like to go out swing dancing.

Well I hoped you could tell me, I’m doing it like that now, sending MyLibrary model from the example, but it just doesn’t feel right. Or maybe I’m too acustomed to JSON. ð
Also, what content type do you set in header when sending protobuff serialized into Data? I’m sending from an iOS app, I’m using Apple’s lib for protobuf, the same one like on the server side, but I can’t get the server to receive the body of the request. Communication in other direction is working fine, I’m getting protobuf from the server, and using it in my app. I’m doing requests via URLRequest and setting httpBody with ‘try? book.serializeProtobuf()’ and headers with ‘setValue(“Content-Type”, forHTTPHeaderField:”application/octet-stream”)’, but it just doesn’t work, the request always has empty body when it comes to server.