Recently, I wrote a post about how to deploy deep learning models into production without the use of additional frameworks. This approach was simplistic and works, but there is also TFX (tensorflow x), which is meant for production use cases. In this post I will cover how to deploy a CNN (EfficientNet) into production with tensorflow serve, as a part of TFX. I want to explain why it does not convince me, when delivering predictions from a single model.

What is tensorflow serve?

TensorFlow Serving is a flexible, high-performance serving system for machine learning models, designed for production environments. TensorFlow Serving makes it easy to deploy new algorithms and experiments, while keeping the same server architecture and APIs. TensorFlow Serving provides out-of-the-box integration with TensorFlow models but can be easily extended to serve other types of models and data.

https://github.com/tensorflow/serving

So, as you can read, tensorflow serve is meant to be production ready. Honestly, I think it promises more than it actually offers, however we will see that later on. I’ve read that it is used at google for delivering a huge number of models at once. The biggest problem is that it lacks documentation (typical to google) and you don’t find much examples, where people actually use deep learning models in production. This is the often-called gap between modelling ANNs and the industry adoption. I hope I can help a bit with this post.

First attempt

Following
this tutorial it seems quite easy and for the MNIST example
and it should work. The steps to make it work are:

Make sure the directory structure is versioned. The saved model must be versioned by a subfolder named by a simple version number. MODEL_NAME/VERSION/SAVED_MODEL

Serve your model with tensorflow_model_serve

Write a rest client, which does all the preprocessing

This approach has some hints, you don’t want in real scenarios:

The input data is a raw bitmap image as json array (triples of integers for each pixel), we don’t want to send such a format, we want to send jpegs.

The hole preprocessing is done in the client, but we want an API, which does simply take images and make predictions

You either wrap the API in another or you try to include the preprocessing in TensorFlow serve.

Accepting jpegs

The problem with tensorflow serve is that you don’t have a chance to insert custom code, at least I did not find a way in the documentation. Everything must be included into the DAG of the SavedModel.

If we want to use a REST API with base64 encoded jpegs, we must somehow make the decoding of the jpeg and resizing of the image as a part of the model. It turns out that this is only possible by adjusting the model architecture (DAG) and include these operations into the model.

Example:

We have a keras model , which does image classification and the model is rather complex (EfficientNet code and paper) but has an input layer accepting 300×300 images Input(shape=(None,300,300,3)) and an output of several class activations Dense(16, activation=’softmax’). This model is not capable of accepting base64 strings as input and as we don’t have the possibility to preprocess our images in tensorflow serve additionally, we must include it into the model, which is not straight forward.

Adding the preprocessing

To make this happen, we need to combine our keras model with tensorflow tensors. The idea is to replace the current Input layer of the keras model by a tensorflow custom preprocessing layer. This must be done before exporting to the SavedModel format. Here is the code to make this happen, see the comments for explanations.

Now we have a model which accepts strings as input and does the decoding and the resizing of the image on its own.

Server and Client

Now we can serve the model the same way as in the tutorial. To activate the REST endpoint, we just supply the rest_api_port and we are done (at least this part is straight forward). Port 9000 is for gRPC calls, which would be more convenient, when directly using the API within a wrapper API.

Limitations

Unfortunately, it is not possible to change the signature of the predictions (e.g. map the output classes to meaningful names), at least I did not find any solution for it. Well, we could add a custom layer again, but in general I am not convinced that adding layers to do pre- and postprocessing, is is a good approach. It works, but it feels wrong.

For a sophisticated REST service, I would recommend to encapsulate the tensorflow serve server completely into another REST API and not customize the model itself. In this scenario it would be possible to do all the pre- and postprocessing and deliver predictions in human readable and more REST-like manner. The question arises, whether it’s better to stay with a simple flask REST API and avoid using tensorflow serve. Tensorflow serve seems to be a good option to serve multiple models at once or if you use additional features (warm-up), but for a single model it seems pretty chucky.

1 comment

[…] First let’s look at some tools, which might be feasible for the most projects. The straightforward solution for tensorflow models is tensorflow serve. I wrote about how to use the SavedModel approach and tensorflow serve HERE. […]