The big picture

As described in a previous article we (Niklas and I) are going to use Tensorflow to classify images into pre-trained categories. The previous artikel was about on how to train a model with Tensorflow on Kubernetes. This article here now describes how to use the pre trained model which is stored on Object Storage. Similar to the training we will also use docker to host our program but this time we will use OpenWhisk as a platform.

Like the first part I also use the Google training Tensorflow for Poets. This time not the code itself but I copied the important classification parts from their script into my python file.

OpenWhisk with Docker

OpenWhisk is the open source implementation of an so called serverless computing platform. It is hosted by apache and maintained by many companies. IBM offers OpenWhisk on their IBM cloud and for testing and even playing around with it it the use is for free. Beside python and javascript OpenWhisk also offers the possibility to run docker containers. Internally all python and javascript code is executed anyhow on docker containers. So we will use the same official Tensorflow docker container we used to build our training docker container.

Internally OpenWhisk has three stages for docker containers. When we register a new method the execution instruction is only stored in a database and as soon as the first call approaches OpenWhisk the docker container is pulled from the repository, then initialised by an REST call to ‘\init‘ and then executed by calling the REST interface ‘\run‘. The docker container keeps active and each time the method is called only the ‘\run‘ part is executed. After some time of inactivity the container is destroyed and needs to be called with ‘\init‘ again. After even more time of inactivity even the image is removed and need to be pulled again.

The setup

The code itself is stored on github. Let’s have a look first on how we build the Docker container:

Dockerfile

Shell

1

2

3

4

5

6

7

8

9

FROM tensorflow/tensorflow:1.4.0-py3

WORKDIR/tensorflow

COPY requirements.txtrequirements.txt

RUN pip install-rrequirements.txt

COPY classifier.pyclassifier.py

CMD python-uclassifier.py

As you can see this Docker is now really simple. It basically installs the python requirements to access the SWIFT Object Store and starts the python program. The python program keeps running until the OpenWhisk system decides the stop the container.

We make heavy use of the idea of having a init and a run part in the execute code. So the python program has two main parts. The first on is init and the second run. Let’ have a look the init part first which is basically setting up the stage for the classification itself.

\init

Python

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

@app.route('/init',methods=['POST'])

definit():

try:

message=flask.request.get_json(force=True,silent=True)

ifmessage andnotisinstance(message,dict):

flask.abort(404)

conn=Connection(key='xxxxx',

authurl='https://identity.open.softlayer.com/v3',

auth_version='3',

os_options={"project_id":'xxxxxx',

"user_id":'xxxxxx',

"region_name":'dallas'}

)

obj=conn.get_object("tensorflow","retrained_graph.pb")

graph_def=tf.GraphDef()

graph_def.ParseFromString(obj[1])

withgraph.as_default():

tf.import_graph_def(graph_def)

obj=conn.get_object("tensorflow","retrained_labels.txt")

foriinobj[1].decode("utf-8").split():

labels.append(i)

exceptExceptionase:

print("Error in downloading content")

print(e)

response=flask.jsonify({'error downloading models':e})

response.status_code=512

return('OK',200)

Unfortunately it is not so easy to configure the init part in a dynamic way with parameters from outside. So for this demo we need to build the Object Store credentials in our source code. Doesn’t feel right but for a demo it is ok. In a later article I will describe how to change the flow and inject the parameters in a dynamic way. So what are we doing here?

10-16 is setting up a connection to the Object Store as described here.

18-22 is reading the pre trained Tensorflow graph directly into memory. tf is a global variable

24-26 is reading the labels which are basically a string of names separated by line breaks. The labels are in the same order as the categories in the graph

By doing all this in the init part we only need to do it once and the run part can concentrate on classifying the images without doing any time consuming loading any more.

Tensorflow image manipulation and classification

Python

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

defrun():

deferror():

response=flask.jsonify({'error':'The action did not receive a dictionary as an argument.'})

How to get the image

The image is transferred base64 encoded as part of the Line 24-25 request. Part of the dictionary is the key payload. I choose this because Node-red is using the same name for some kind of most important key. Tensorflow has a function to consume base64 encoded data as well but I could not get it to run with the image encoding I use. So I took the little extra step here and write the image on file and read it back later. By directly consuming it I think we could same some milliseconds processing time.

Transfer the image

Line 27 reads the image back from file

Line 29 decode the jpeg into an internal representation format

Line 30 cast the values to an float32 array

Line 31 adds a new dimension on the beginning of the array

Line 32 resizes the image to 224, 244 to have a similar size with the training data

Line 33 normalize the image values

Classify the image

Line 34-35 gets the input and output layer and stores it in the variables

Line 36 loads the image into Tensorflow

Line 39 here is the magic happening. Tensorflow processes the CNN with the input and output layer connected and consumes the Tensorflow image. Furthermore numpy is squeezing out all array nesting to a single array.

Line 40 has an array with probabilities for each category.

Mapp the result to labels

The missing last step is now to map the label names to the results which is be done in line 43 and 44.

The first execution will take up to 15 seconds because the docker container will be pulled from docker hub and the graph will be loaded from the Object Store. Calls later should be around 150 milliseconds processing time. The parameter –result will force OpenWhisk to wait for the function to end and also show you the result on your command line.

1

2

3

4

5

6

7

{

"daisy":0.9998985528945923,

"dandelion":0.00007187054143287241,

"roses":4.515387388437375E-7,

"sunflowers":0.000029122467822162434,

"tulips":4.63972159303605E-11

}

If you want to get the log file and also an exact execution time try this command:

First call results in “duration”: 3805. Your call itself took way longer in the first call because 3805 is only the execution of the docker container (including init) not the time it tooks OpenWhisk to pull the docker container from docker hub.

The big picture

Modern Visual Recognition is done with deep neural networks (DNN). One framework (and I would say the most famous one) to build this kind of network is Tensorflow from Google. Being open source and specially awesome it is perfect to play around and build your own Visual Recognition System. As the compute power and specially the RAM memory raises there is now a chance of having much more complicated networks compared to the 90th where there where only one or two hidden layer.

One architecture is the Convolutional Neural Network (CNN). The idea is very close to brain structure. The basic idea is to intensively train a network on gazillions of images and let it learn features inside the many hidden layers. Only the last layer connects features to real categories. Similar to our brain the networks learns concepts and patterns but not really the picture groups.

After spending a lot of compute power to train these networks they can be easily reused to train new images by replacing only the last layer with a new one representing the to be trained categories. Training this network is only training the last connection between the last layer and the rest of the network. This training is extremely fast (only minutes) compared to month for the complete network. The charming effect is to train only the “mapping” from features to categories. This is what we are going now.

Basically the development of such a system can be divided into two parts. The first part (training) is described there. For the “use” aka classification have a look into the second part on my blog. I developed this system together with a good friend and colleague of mine. Check out Niklas Heidloff, here is his blog and twitter account. The described system has mainly three parts. Two docker containers described in this blog and one epic frontend described in Niklas blog. The source code can be found on github.

ImageNet

If you want to train a neural network (supervised learning) you need a lot of images in categories. Not ten or hundred but better hundred thousands or even 15 million pictures. A wonderful source for this is Imagenet. >14 million pictures organized in >20k categories. So a perfect source to train this kind of network. Google has done the same and participated in the Large Scale Visual Recognition Challenge (ILSVRC). Not only Google but many other research institutes build networks on top of Tensorflow in order have a better image recognition. The outcome are pre-trained models which can be used for system like we want to build.

Tensorflow for poets

Like always it is best to stand on shoulders of giants. So in our case use the python code developed by google at the codelabs. In this very fascinating and content full online training on Tensorflow Google developed python code to retrain the CNN and also to use the new trained model to classify images. Well, actually the training part is just using the original code and wraps it into a docker container and connects this container to an Object Store. So no much new work there but a nice and handy way to use this code for an own project. I highly recommend taking the 15 minutes and take the online training to learn how to use Tensorflow and Python.

MobileNet vs. Inception

As discussed there are many trained networks available the most famous ones are Inception and MobileNet. Inception has a much higher classification rate but also needs more compute power. Both on training and on classification. While we use kubernetes on “the cloud” the training is not a big problem. But we wanted to use the classifier later on on OpenWhisk we need to take care of the RAM memory usage. (512MB). The docker container can we configured to train each model but for OpenWhisk we are limited to the MobileNet.

Build your own classifier

As you can see in the picture we need to build two containers. The left one is loading the training images and the categories from an Object Store, trains the neural network and uploads the trained net back to the Object Store. This container can run on your laptop or somewhere in “the cloud”. As I developed a new passion for Kubernetes I added a small minimal yaml file to start the docker container on a Kubernetes Cluster. Well not really with multiple instances as the python code only uses one container but see it as some kind of “offloading” the workload.

The second container (will be described in the next article) runs on OpenWhisk and uses the pre-trained network downloaded from the Object Store.

The Dockerfile is straightforward. We use the Tensorflow docker image as base and install the git and zip (unpacking the training data) packages. Then we install all necessary python requirements. As all the Tensorflow related packages for Python are already installed these packages are only for accessing the Object Store (see my blog article). Then we clone the official github tensorflow-for-poets repository, add our execution shell script and finish with the CMD to call this script.

Execution Script

Shell

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

#!/usr/bin/env bash

echo${TF_MODEL}

export OS_AUTH_URL=https://identity.open.softlayer.com/v3

export OS_IDENTITY_API_VERSION=3

export OS_AUTH_VERSION=3

swiftauth

swiftdownload${OS_BUCKET_NAME}${OS_FILE_NAME}

unzip${OS_FILE_NAME}-dtf_files/photos

python-mscripts.retrain\

--bottleneck_dir=tf_files/bottlenecks\

--how_many_training_steps=5000\

--model_dir=tf_files/models/\

--summaries_dir=tf_files/training_summaries\

--output_graph=tf_files/retrained_graph.pb\

--output_labels=tf_files/retrained_labels.txt\

--architecture=${TF_MODEL}\

--image_dir=tf_files/photos

cdtf_files

swift upload tensorflow retrained_graph.pb

swift upload tensorflow retrained_labels.txt

All important and sensitive parameters are configured via environment variables introduced by the docker container call. The basic and always the same parameters are set here. Where to do the keystone authentication and which protocol version for the Object Store. The swift commands downloads a zip file containing all training images in subfolders for each category. So you need to build a folder structure like this one:

Shell

1

2

3

4

5

.|

|-Category-A/

|-Category-B/

|-Category-C/

|-NegativeExampels/

The execution script unpacks the training data and calls the retrain script from Tensorflow-for-poets. Important parameters are how_many_training_steps (can be reduced to speed up for testing) and the architecture. As the last parameter can be changed depending on how accurate the classifier has to be and also how much memory is available for the classifier this parameter is also transferred via a command line parameter.

After building the docker container and pushing it to docker hub this yaml file triggers Kubernetes to run the container with the given parameters, many taken from your Object Store credential file:

1

2

3

4

5

6

7

8

9

10

11

12

VCAP={

"auth_url":"https://identity.open.softlayer.com",

"project":"object_storage_07xxxxxx_xxxx_xxxx_xxxx_6d007e3f9118",

"projectId":"512bfxxxxxxxxxxxxxxxxxxxxxxfe4e1",

"region":"dallas",

"userId":"4de3dxxxxxxxxxxxxxxxxxxxxxxx723b",

"username":"member_caeae76axxxxxxxxxxxxxxxxxxxxxxxxxxxxxx7d",

"password":"lfZxxxxxxxxxxxx.p",

"domainId":"151fxxxxxxxxxxxxxxxxxxxxxxde602a",

"domainName":"773073",

"role":"member"

}

OS_USER_ID -> VCAP[‘userId’]

OS_PASSWORD -> VCAP[‘password’]

OS_PROJECT_ID -> VCAP[‘projectId’]

OS_REGION_NAME -> VCAP[‘region’]

OS_BUCKET_NAME -> Up to you however you called it

OS_FILE_NAME -> Up to you, however you called it

TF_MODEL -> ‘mobilenet_0.50_{imagesize}’ or ‘inception_v3’

Use Object Store to store your trained class for later use

We decided to use Object Store to store our training data and also the re-trained network. This can be any other place as well, for example S3 on AWS or your local HDD. Just change the Dockerfile and exec file to download and upload your data correspondingly. More details on how to use the Object Store can be found in my blog article.

IBM Object Store

IBM offers a S3 compatible Object Store as a file storage. Beside S3 the storage can also be accessed via the SWIFT protocol by selecting a different deploy model. As the cost for this storage is extremely low compared to Database storage it is perfect for storing sensor data or other kind of data for machine learning.

I use the storage for example to host my training data or trained model for Tensorflow. Access and payment for the Object Store is managed via IBM Cloud aka Bluemix. And as this offering is included in the Lite offering the first 25GB are for free. 🙂

As there is a problem getting the S3 credentials right now I use the SWIFT access model. Please make sure when you request the Object Store service to access the SWIFT version to select the right access model.

Access storage

Inside the IBM Cloud web interface you can create or read existing credentials. If your program runs on IBM Cloud (Cloudfoundry or Kubernetes) the credentials are also available via the VCAP environment variable. In both cases they look like mine here:

1

2

3

4

5

6

7

8

9

10

11

12

{

"auth_url":"https://identity.open.softlayer.com",

"project":"object_storage_xxxxxxxx_xxxx_xxxx_b35a_6d007e3f9118",

"projectId":"512xxxxxxxxxxxxxxxxxxxxxe00fe4e1",

"region":"dallas",

"userId":"e8c19efxxxxxxxxxxxxxxxxxxx91d53e",

"username":"admin_1xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxa66",

"password":"fTxxxxxxxxxxw8}l",

"domainId":"15xxxxxxxxxxxxxxxxxxxxxxxxxxxx2a",

"domainName":"77xxx3",

"role":"admin"

}

Important informations are the projectId, region, userId and password. The access with keystone the swift python client looks like this:

1

2

3

4

5

6

7

conn=Connection(key=VCAP['password'],

authurl='https://identity.open.softlayer.com/v3',

auth_version='3',

os_options={"project_id":=VCAP['projectId'],

"user_id":=VCAP['userId'],

"region_name":VCAP['region']}

)

Important is the version information, also as part of the authurl.

Accessing data

Objects can be read and written, containers (aka buckets) can we read and modified as described in the documentation. For example:

Over the Air update (Ota) for ESP8266

Thanks to the esp8266 project on github there is a very convenient way how an ESP can be updated over the air. There are three different ways available.

The first one is via the arduino IDE itself where the esp opens a port and is available for firmware upload just like with a serial connection. Very convenient if you are in the same network.

The second one is via http upload. So the esp provides a web server to upload the bin file. In this case there is no need to be in the same network but it is still a push and for each installed esp individual necessary.

The third and most convenient way for a bigger installation base or in case the devices are behind a firewall (as they always should be) and no remote access is possible. In this case the device can download the firmware itself via http(s) download from a web server somewhere in the internet.

For a complete dev-ops pipeline from pushing to a repository to flashing a device the third scenario it the easiest one. So we need a place to store the binary files. For convenience I use amazon s3 to host my binary files as travis easily supports s3 upload. But it can be every internet platform where files can be stored and downloaded via http(s). The necessary code on arduino side looks like this:

This arduino function can be called from time to time (at startup or on constant running systems every now and then) to check for a new firmware version and in case there is a new version available automatic flash it and restart.

Line 1 is a #define with a placeholder for the current version of the installed firmware. This placeholder is replaced in the build pipeline at travis with an increasing number. So the compiled code has something like 23 or 42 instead of REPLACE_WITH_CURRENT_VERSION.

Line 2 is the URL for a latest version of a firmware.

Line 3 is the URL to a file with only one line with the latest build number in it.

Line 7-9 loads the version file from s3.

Line 12-13 converts the file into a number which can be compared with the define from line 1.

Line 17 is the firmware update itself. A detailed description of the ESPhttpUpdate class can be found here.

There are two ways to check if there is a new version available and only flash if there is something new. The one we use here is to have an own mechanism for it. I do it because on s3 I can only host static files and therefore I place the latest build number in a static file next to the firmware itself. The other way is build in into ESPhttpUpdate. The update function can be called with a build number which will be compared on the server and the return code will reflect if there is a new version or not. In this case we would need a script on the server to check for it.

Get an increasing build version number

With a little bash script we could load the last build number from s3 and then increase it in order to have the current number for our build.

This script loads the version file (line 3), increases the number (line 4) and patches our source code file (line 11) with this number instead of REPLACE_WITH_CURRENT_VERSION. After running this script the current source code contains the latest number and also the upload folder for s3 has a new file with the newest number in order to inform the polling ESPs.

Travis config file

Travis-ci is incredible easy to use and very reliable for continuous integration. In combination with platformio it is very easy to compile arduino code for several types of hardware. Simply configure the hardware in the platformio.ini file:

1

2

3

4

[env:huzzah]

platform=espressif8266

board=huzzah

framework=arduino

In this case we use the esp8266 feather board aka Huzzah. Just set the framework to your kind of esp.

Travis itself is configured by the .travis file in the root directory of your repository on github:

Line 1: Platformio is based on python so the build environment (although the code is c++) is python for maintaining platformio.

Line 3: Right now platformio is only available for python 2.7 so this line gets the latest stable version of python 2.7.

Line 5-7: Gets the latest cache files from the last build in order to save compile time and reduce the costs for travis. As this service is for free for open source projects it is always nice to save some money for the cool guys.

Line 10: Installs the latest version of platformio itself.

Line 13: Creates the upload directory which we will upload to s3 later on.

Line 14: Calls the build number increase and patch script.

Line 15-16: Patches the wireless lan config in case it is not handled inside the arduino code itself.

Line 17: Calls platformio to download all libraries and compile the arduino code itself.

Line 18: Platformio generates a lot of files for the linker and several other files. We only need the bin file later on, so we copy it here to the upload folder.

Line 20: Travis has a build in functionality to upload files after compilation. This is the part where we upload the files to s3.

Line 22: Defines the s3 bucket to upload the files.

Line 23-26: Provides the encrypted s3 credentials. See travis documentation on how to create these lines.

Line 29: Defines the local folder to be uploaded. Otherwise travis will upload everything from the current run.

Line 30: Defines the s3 folder in the bucket where the files will be stored.

With this files in place travis monitors your github repository and creates / uploads new firmware versions each time you push changes to your repository. The arduino code checks for new versions and patches itselfs as soon as there is a new version available. A complete project can be found here in my github repository.

pushVisualRecognition.sh

I hacked a nice script for the Watson Visual Recognition service. There is already a very helpful page available here but many people (including me) like command line tools or scripts to automate processes. The script does the following processes to each picture:

Resize to max 500×500 pixel. Watson internally use only ±250 pixels, so this saves a lot of upload time.

Enhance the image (normalisation) for better results.

Autorotate the images based on the EXIF data from your camera. Watson ignores EXIF data.

The tool expects this directory structure and reads all necessary information from it:

Classifiername

Classname

<more then 10 files>.jpg

The Visual Recognition key is read from the “VISUAL_KEY” environment variable.

How to install it

The pushVisualRecognition.sh script is part of the bluemixcli docker container as described here. It basically only needs imagemagick and zip installed so you can also run it without the docker container and download the script directly from github with this link. If you want to run it with docker the command is

Being an developer advocate means to play always with the latest version of tools and being on the edge. But installed programs are getting out of date and so I always end up with having installed old versions of CLI tools. One reason why I love cloud (aka other people’s computers) computing so much is because I don’t need to update the software, it is done by professionals. In order to have always the latest version of my Bluemix CLI tools in hand and being authenticated I compiled a little docker container with my favourite command line tools. cf, bx, docker and wsk.

Getting the docker container

I published the docker container on the official docker hub. So getting it is very easy when the docker tools are installed. This command will download the latest version of the container and therefore the latest version of installed cli tools. We need to run this command from time to time to make sure the latest version is available on our computer.

Shell

1

docker pull ansi/bluemixcli

Get the necessary parameters

For all command line tools we need username, passwords and IDs. Obviously we can not hardcode them into the docker container therefore we need to pass them along as command line parameters when starting the docker container.

Username (the same as we use to login to Bluemix)

Password (the same as we use to login to Bluemix)

Org (The Organisation we want to work in, must already be existing)

Space (The Space we want to work in, must already be created)

AccountID (This can we catched from the URL when we open “Manage Organisation and click on the account)

Run the container

The container can be started with docker run and passing all parameters with -e in:

Shell

1

2

3

4

5

6

7

8

9

docker run-it--rm\

-eBX_USERNAME=<Bluemix Username>\

-eBX_PASSWORD=<Bluemix Password>\

-eBX_ORG=<Bluemix Organisation>\

-eBX_SPACE=<Bluemix Space>\

-eBX_ACCOUNT_ID=<Bluemix Account ID>\

-eWSK_AUTH=<OpenwhiskAuthentification>\

-v${PWD}:/root/host\

ansi/bluemixcli/bin/bash

Line 8 mounts the local directory inside the docker container under /root/host. So we can fire up the container and have a bash with the latest tools and our source code available.

Use the tools

Before we can use the tools we need to configure them and authenticate against Bluemix. The script “init.sh” which is located in “/root/” (our working directory) takes care of all logins and authentications.

cf

The Cloudfoundry command line tool for starting, stopping apps and connecting services.

bx

The Bluemix version of the Cloudfoundry command line tool. Including the plugin for container maintenance. By initializing this plugin we also get the credentials and settings for the docker client to use Bluemix as a docker daemon.

docker

The normal docker client with Bluemix as daemon configured.

wsk

The OpenWhisk client already authenticated.

We can configure an alias in our .bashrc so by just typing “bxdev” we will have bash with the latest cli tools available.

Debugging feature extraction of images

Histogram chart drawn on image

A big part when writing code for visual recognition in cognitive computing is to extract information aka. features from images. For example we need to understand colour information and want to see a histogram chart on a certain RGB or HSV channel. OpenCV offers a nice way to combine image analytics and image manipulation. In combination with the library mathplot it is relatively easy to analyse the picture, extract features and write debug information as a chart to the picture itself.

Getting a test picture

For this demo I will choose a picture from a webcam of the Observatory Friolzheim. On astrohd.de we can get a life picture taken from the observatory dome. Beside the webcam pointing to the building there are also weather information (very helpful when mapping images features) and a AllSkycam.

Getting the software and libraries

As described in this blog article there is a nice docker container for openCV available. We can use this one and just add missing libs and tools with a minimalistic Dockerfile:

Line 5

This one is very important if we want to run this matplotlib “headless”. Meaning without a graphic export attached to the runtime. Like in a docker container or on a remote server.

Line 8-10

We create a matplot lib figure with subplot here where we are going to add the chart.

Line 16

As openCV uses numpy to work with images, the image itself is displayed in a multidimensional array and image.ravel() flattens this to a one dimensional array. Bincount just counts the amount of unique numbers used in this array. So with this line we just create a new 1 dimensional array of all the RGB colours in the image which is basically a histogram of the image. By doing so we mix all colours together, which is ok in this sample or a classic histogram. However if we want to see only one channel we can use

Python

1

2

hsv=cv2.cvtColor(image,cv2.COLOR_BGR2HSV)

h,s,v=cv2.split(hsv)

to extract a single channel.

Line 18-19

If the picture contains very shiny or very black parts the high and low values like <10 or >250 are over proportionally high. This will distort the histogram sometimes. These two lines just resets the histogram array on the upper and lower end to 0.

Line 21-22

This just smooth the histogram.

Line 23-24

Histogram without blanked parts

Argmax returns the index of the highest number in a array. This one can be considered as a feature of the image. For example if we want to rank or sort our webcam pictures by daytime regarding to the sunlight this features gives us an indication. If we are using this feature the lines 18-19 becomes very important here to eliminate the reflexions in a picture. For example in the demo image the observation dome. The upper pictures contains the chart which already blanked out parts. The original chart looks like the one on the right.

Line 26-27

The matplotlib chart is drawn here into the subplot. Still the matplotlib chart and the image itself are “stored” in different libraries.

Line 29

Here the magic happens. fig.canvas.tostring_rgb exports the chart from matplotlib into a string and numpy.fromstring imports this string back to a numpy array which is like the picture we imported at the beginning.

Line 30

Here we reshapes the newly created image to the same size as the imported image.

Line 31

OpenCV can handle all kind of different image colour representations, like RGB, HSV aso. The fromstring importer reads in RGB but our image is in BGR, so this line just converts the colour representation.

Line 32

Here we combine both image with a weighted parameter of 70% of the original image and 30% of the chart.

Line 33

Finally we save this image in a debug folder with the same filename but just because we can as a png file.

Line 35

Finally we clear up the used memory. This is very important when we convert a huge amount of images.

How to use this for visual recognition

By extracting features in images this little piece of code helps us to add the feature directly in a debug image. In our example we only used the histogram and extracted the colour with the highest number, but by having the histogram in hand we could see what was the underlying information the sorting algorithm used. Sorting images in night, dusk/dawn, daylight can be a way of preparing pictures for further processing in neuronal networks. By browsing through the images in one of this sorting folders we can easily see what went wrong in a picture which does not belong in this group. Typical problem is night and day pictures are missorted because of reflexions.

Visual Recognition

Watson result

Computer Vision or Visual Recognition is part of cognitive computing (CC) aka Artificial Intelligence. One of the main concepts is to extract information out of unstructured data. For example you have a webcam pointing on a highway. As a human you see if there is a traffic jam or not. For a computer it’s only 640x480x3x8 (7.372.800) bit. Visual Recognition helps you to extract information out of this data. For example “This is a highway”. Out of the box systems like Watson are able to give you information what do you see on the picture. You can try it here https://visual-recognition-demo.mybluemix.net. The result can be seen on the left picture. So Watson knows it is a highway and even it’s a divided highway but it does not tell you there is a traffic jam or even a blocked road. Fortunately Watson is always eager to learn, let us see how we can teach him what is a traffic jam. This article only focuses on the picture preparation part not the train Watson part. See next postings for the Watson part.

Get pictures

There are many traffic cameras all around but I am not sure about the licence, so it is hard to use it here as a demo. But let us assume we can take pictures like this one from Wikimedia: Cars in I-70.If you live in south Germany there are nice traffic cameras from Strassenverkehrszentrale BaWue. Unfortunately they don’t offer the pictures with the right licence for my blog. If you know a great source for traffic cameras with the right licence please let me know.

Prepare pictures for training

Visual Recognition works a little bit like magic. You give watson 100 pictures of a traffic jam and 100 without traffic jam and he learns the difference. But how do we make sure he really learns traffic jam and not the weather or the light conditions. And furthermore only one lane in case the camera shows both lanes? So first we need to make sure we find enough different pictures of the road with traffic jam under different weather and light conditions. The second part can be done with OpenCV. OpenCV stands for open computer vision and helps you to manipulate images. The idea is to mask out parts we don’t want Watson to learn. In our case the second part of the lane and the sky. We can use GIMP to create a mask we can apply with openCV automatically to each picture.

Gimp

GIMP-Layers

First step is obvious to load the image in GIMP. Then open the layers dialog. It’s located under Windows/Dockable Dialogs/Layers or cmd-L. Here we add a new layer and select this one to paint on. Then we select in the tools menu the Paintbrush Tool and just paint the parts black we don’t want Watson to learn.

Blanked-image

Then we hide the original image by pressing the eye symbol in the layer dialog. This should leave us with only the black painting we did before. This will be our mask for openCV to be applied to all pictures. Under File/Export you can save it as mask.jpg. Make sure it is only the black mask and not the picture with the black painting.

Use openCV in docker

As openCV is quite a lot to install, we could easily use it within docker to work with our pictures. We can mount host directories inside a docker container, so in this case our directory with pictures:

This brings up the openCV docker container from victorhcm and opens a shell with our current directory mounted under /host. As soon es you exit the container it will be removed because of the “–rm” parameter. Don’t worry only the docker container will be deleted, everything under /host is mounted from the host system and will remain. Everything you save in other directories will be deleted.

How to mask out part of the picture

The python program to use openCV to mask out all pictures in a directory is then really easy to use:

Python

1

2

3

4

5

6

7

8

9

10

11

importcv2

importglob

mask=cv2.imread("mask.jpg",cv2.IMREAD_GRAYSCALE)

forfullname inglob.glob("pics/*.jpg"):

filename=fullname.split('/')[-1]

image=cv2.imread(fullname,cv2.IMREAD_COLOR)

dst=cv2.bitwise_and(image,image,mask=mask)

cv2.imwrite("masked/"+filename,dst)

printfilename

Basically the program iterate through all “jpg” pictures in the subfolder “pics” and saves the masked pictures with the same name in the “masked” folder. Both directories have to exists before you start the script. In order to keep the script reduced to the important parts I left the create and check directory part out of this script.

Line 4

Loads the mask images as a grayscale image.

Line 8

Loads the image to work on as a colour image.

Line 9

Here is the real work done, this applies the mask with bitwise add of all pixels. Therefore the blank will win and the transparent will let the normal picture gets through.

Line 10

Saves the new masked picture in the “maksed” folder.

Preselect pictures

For the learning process we need to sort the pictures by hand. One bucked with traffic jam and the other with ok.

Hyperledger projects code structure

Travis-CI is one of the most used continuous delivery platforms for open source projects. In order to check your hyperledger blockchain application you need to check at least two different languages in the same travis-ci configuration, as your smart contract is written in GOlang and your application most likely in Javascript or Python or whatever(tm).

Each part of the project (smart contract, backend, frontend) is in a separate directory and has a code part and a unit test part. Without getting into details on how to run unit tests for each language we concentrate now on the .travis file.

We have two tests for python (one with 2.7 the other one with the latest nightly build), one for the javascript (no unit test in this example) and the GO part.

Important are lines 29 and 30. Before we run the go build we create the directory for hyperledger and check out manually the official hyperledger code with the v0.6 branch. After that we can normally build the smart contract by calling go build ./

Last weekend the c-base and friends team had a great time doing some hacking with Narrow Band IoT (NB-IoT) from Deutsche Telekom at the nbiot-hackathon at hub:raum. We could put our hands on the BC95 chip, a dev board and access to the Telekom test network. Beside hacking we had a great time with funny hats and our obligatory overalls.

The BC95 Board

The board we could use was the BC95-B8 board which we mounted on a development board with a support controller and serial converter. Beside the board setup we also soldered a PCB with some sensors and a Teensy board to control the BC95 and the sensors.

Wiring

Pin-Connection

The dev board itself has a RS232 converter to give access via “normal” RS232 connectors. This is convenient for older laptops or desktops but luckily they also give you access via 3.3V pegel to the same UART interface. The pins are pre soldered on a 10 pin header so it’s easy to connect this to arduino or Raspberry PI via the serial connection. As you can see in the picture it is pin 1,2 and 6. No level converter necessary.

AT- Command Set

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

AT// First handshake will be used for out speed detection may be result in ERROR

AT// Repeat sending "AT" until you receive an "OK"

AT+NRB// Reboot the device in order to have a clean start

//wait for the reboot

AT+NBAND=8// Set the communication band in this case 900Mhz

AT+CGDCONT=1,"IP","NBIOT.Telekom"// Sets the APN (Access Point Name)

AT+CEREG=2// Connect to the IoT Core

AT+CFUN=1// Power on the module

AT+COPS=1,2,"12345"// Force the module to connect to the network

// wait for the network connection (some seconds)

AT+NPING=8.8.8.8// Ping a server to check if it works

AT+NSOCR=DGRAM,17,16666// Open a UDP socket

AT+NSOST=0,ansi.23-5.eu,16666,4,414E5349// Send out the data

Line 1,2

The serial protocol can be any speed at 8N1, the board auto detects the speed at first communication, therefore you need to send the AT command several times (normally 2) to set the speed. Normally the first AT command is answered with ERROR and the second one with AT. Make sure you get an OK before you continue.

Line 3

In order to get a clean setup we first reboot the board with “NRB”. It takes some seconds and it will come back with OK.

Line 7

Depending on your network you need to set the band with “NBAND”. The Telekom test network is on 900 MHz so we go with 8. Other bands are 5 (850 Mhz) and 20 (800 MHz).

Line 8

Depending on your setup and provider you need to set the APN with the CGDCONT command.

Line 9

Connect to the IoT Core

Line 10

Power on the module

Line 11

Connect to the network. This can take several seconds even a minute. You can always check the connection with at+cgatt?” and check for “+CGATT:1”. You can double check the existing connection by asking for the IP address which is assigned to you by sending “at+cgpaddr=1” to get for example “+CGPADDR:1,10.100.0.16”.

Line 15

Ping a server to test everything is fine. (In this case the google DNS server)

Line 16

Open an UDP socket to receive answers. In our case the DGRAM and 17 are mandatory but the port you are using (in our case 16666) is up to you.

Line 17

Send your UDP data package. First parameter is the out port (0). The second one is the address you want to send the data to (ip or name). Third one is the receivers port (16666). Fours is the amount of data you want to send (keep it below 100 bytes) and the last one is your data in hex notification. I recommend asciitohex.com to convert a string you want to send.

Arduino

Arduino

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

uint8_tsensordata[10];

voidsetup(){

Serial.begin(9600);

Serial1.begin(9600);

while(!Serial){}

while(!Serial1){}

Serial.println("c-base starts");

initModem();

Serial.println("DONE");

}

voidsensorfake(){

for(inti=0;i<10;i++){

uint8_td=sensordata[i];

sensordata[i]=d+1;

}

}

voidsendit(Stringin){

Serial.print(in+"\r");

Serial1.print(in+"\r");

delay(100);

while(Serial1.available()>0){

if(Serial1.available()>0){

bytec=Serial1.read();

Serial.write(c);

}

}

}

voidinitModem(){

sendit("AT");

sendit("AT");

sendit("AT+NRB");

delay(5000);

sendit("AT");

sendit("AT");

sendit("AT+NBAND=8");

sendit("AT+CGDCONT=1,\"IP\",\"NBIOT.Telekom\"");

sendit("AT+CEREG=2");

sendit("AT");

sendit("AT+CFUN=1");

delay(5000);

sendit("AT+COPS=1,2,\"12345\"");

delay(5000);

sendit("AT");

delay(5000);

sendit("AT");

delay(5000);

sendit("AT");

sendit("AT+NPING=8.8.8.8");

sendit("AT");

sendit("AT+NSOCR=DGRAM,17,16666,1");

delay(5000);

sendit("AT");

}

voidsendData(){

Stringdatastring="AT+NSOST=0,ansi.23-5.eu,16666,11,01";

for(uint8_ti=0;i<10;i++){

Stringhexstring=String(sensordata[i],HEX);

if(hexstring.length()<2){

hexstring="0"+hexstring;

}

datastring+=hexstring;

}

sendit(datastring);

}

voidloop(){

sensorfake();

sendData();

delay(2000);

}

This arduino code is really a fast and ugly hack for the hackathon in order to send out the data. It does not listen for the AT returns or anything else. So this is only an example on how NOT to do coding but it worked for the hackathon.