Memz.cohttps://memz.co
Hierarchical Notepad for Android and WebWed, 16 Nov 2016 08:16:36 +0000en-UShourly1https://wordpress.org/?v=4.7.9Service discovery for microservices using built-in Docker DNShttps://memz.co/dns-service-discovery-api-gateway-docker/
https://memz.co/dns-service-discovery-api-gateway-docker/#commentsFri, 11 Nov 2016 08:42:15 +0000https://memz.co/?p=687I have been designing and implementing various Docker-based microservices platforms for the last couple of years. One of the must-have capabilities of such platforms is service discovery. In previous set of articles I demonstrated an implementation based on such open source components like: etcd, skydns plus custom DNS-automation service built with Node.js and TypeScript. Recently, I discovered […]

]]>I have been designing and implementing various Docker-based microservices platforms for the last couple of years. One of the must-have capabilities of such platforms is service discovery. In previous set of articles I demonstrated an implementation based on such open source components like: etcd, skydns plus custom DNS-automation service built with Node.js and TypeScript. Recently, I discovered that starting from version 1.10, Docker has built-in DNS server that can be used for the same purpose. Only implementation would be much simpler.

I have already migrated some of my environments to use this solution and quite happy with the results. Today I will share my findings with you.

Microservice discovery solution overview

In result of this little exercise we will achieve pretty much the same what we achieved in the original series of 3 articles:

automated DNS entry management based on Docker container name

calling one service from another using service name as short host name

Implemented service discovery is limited to a single host. If you want to expand your platform beyond that, you will need to setup network overlay. It will create single virtual network for Docker containers running on multiple hosts, letting them to talk to each other.

Microservices discovery with Docker built-in DNS and Nginx running in a container

Enabling Docker built-in DNS

First, we need to create our custom network in Docker – this is a requirement for built-in DNS

We run curl command inside docker container to make services discoverable with Docker built-in DNS.

Setting up nginx as a reverse-proxy with DNS-based discovery

For our microservices to be useful, they must be accessible for external clients. This can be achieved thorough API Gateway pattern. In our example we will use Nginx configured as reverse proxy with DNS resolver set to Docker internal DNS (it’s available via IP 127.0.0.11 inside Docker container).

DNS load balancing

I decided to add this section in response to the received question about failover configuration. When I initially tried to implement it with Docker 1.10 it didn’t quite work. Despite 2 instances of the service available, all requests would come to the first one. I suppose we can call it failover – having 2 instances, in normal state. First one will serve all requests and second will be idle. Then if active instance goes down for some reason, second instance will start serving requests. However my goal was to achieve load balancing – when all running instances will receive a portion of traffic.

After some additional research, I found that in Docker 1.11 this issue was resolved. Here’s how DNS load balancing can be implemented.

]]>https://memz.co/dns-service-discovery-api-gateway-docker/feed/5Memz 2.0 preview is available for downloadhttps://memz.co/memz-android-preview-available/
https://memz.co/memz-android-preview-available/#commentsFri, 04 Nov 2016 08:23:53 +0000https://memz.co/?p=684Good news everyone! A complete rewrite of Memz, the best hierarchical notepad / outliner for Android – Memz 2.0 is nearly completed. There are few missing functions yet to be implemented, but the app is already useful and quite stable (I already switched to the new version myself). Today, I prepared this preview version and […]

]]>https://memz.co/memz-android-preview-available/feed/4Nginx as a reverse proxy for dockerized microserviceshttps://memz.co/reverse-proxy-nginx-docker-microservices/
https://memz.co/reverse-proxy-nginx-docker-microservices/#respondMon, 29 Aug 2016 03:53:09 +0000https://memz.co/?p=665This series is dedicated to DNS-based service discovery, reverse-proxying and load balancing of dockerized microservices using etcd, SkyDNS, nginx and a simple service built with Node.js and node-docker-monitor module. Part 1 – Setting up SkyDNS for service discovery of dockerized microservices Part 2 – Automating SkyDNS service discovery with simple Node.js service Part 3 – Setting up nginx as a […]

]]>This series is dedicated to DNS-based service discovery, reverse-proxying and load balancing of dockerized microservices using etcd, SkyDNS, nginx and a simple service built with Node.js and node-docker-monitor module.

Setting up Nginx as reverse proxy for dockerized microservices

In our pervious post we completed setting up environment for DNS-based service discovery of dockerized microservices. We implemented docker-skydns-adaptor using Node.js and TypeScript.

As we demonstrated, DNS provides a simple and effective way for services to communicate with each other. However, the main goal of any service is to provide some functionality to external clients. To achieve that, API Gateway or Reverse Proxy pattern is often used. Essentially, it means installing some web server running in reverse proxy mode. It has to be aware of microservices running in our environment and should have url rewrite rules to route HTTP requests to those microservies. You can find more information about this pattern (as well as an alternative implementation) in one of our previous articles.

In this article we will demonstrate how it can be implemented using simple nginx server setup. Most of this kind of implementations use config re-generation approach to make it microservice-aware. Firstly, a config template is prepared. This template has a dynamic section that repeats for every microservice (docker container). Then some sort of docker monitor re-generates proxy config when a docker container starts up or goes down. And finally, this monitor restarts the proxy. Here’s one of the tools that can be used for this sort of automation.

In our case, however, we can use static configuration file for the proxy. Our dynamic component providing service discovery is SkyDNS that we prepared in previous two posts.

Nginx as reverse proxy for microservices

The only thing left to do is to prepare the configuration file.

We will create it on host machine and then map it to the container running nginx.

We assume that environment created during our previous articles is in the same state that we left it: test-etcd, test-skydns, test-skydns-adaptor, test-service11 and test-service12 are still running. In this case we can see our nginx-based reverse proxy for dockerized microservices in action.

to address a specific service instance. If it doesn’t exist or malfunctioning, we will get HTTP 502 – Bad Gateway response, as demonstrated above.

As result of 3 steps described in this series: setting up basic environment, automating service registration in SkyDNS and configuring API Gateway / Reverse Proxy, we achieved both internal (using short host names) and external service discovery and HTTP request routing. Our current setup is limited to one host only, however with some additional steps, it can be extended to work with multiple hosts.

]]>https://memz.co/reverse-proxy-nginx-docker-microservices/feed/0Automating SkyDNS service discovery with Node.jshttps://memz.co/service-discovery-skydns-docker-node-js/
https://memz.co/service-discovery-skydns-docker-node-js/#commentsTue, 23 Aug 2016 00:36:29 +0000https://memz.co/?p=649This series is dedicated to DNS-based service discovery, reverse-proxying and load balancing of dockerized microservices using etcd, SkyDNS, nginx and a simple service built with Node.js and node-docker-monitor module. Part 1 – Setting up SkyDNS for service discovery of dockerized microservices Part 2 – Automating SkyDNS service discovery with simple Node.js service Part 3 – Setting up nginx as a […]

]]>This series is dedicated to DNS-based service discovery, reverse-proxying and load balancing of dockerized microservices using etcd, SkyDNS, nginx and a simple service built with Node.js and node-docker-monitor module.

Implementing automation service with TypeScript and Node.js

In our pervious article we’ve setup all components required for DNS-based service discovery and load balancing and demonstrated them in action. However, we had to manually create DNS entries for all our services (using etcdctl command). In this post we will remove that manual step by building a simple Node.js service – docker-skydns-adaptor for SkyDNS automation. This service will be listening for Docker events, maintaining DNS entries for running Docker containers.

This behaviour can also be used to address different versions of the same service, referring them like http://v1.service and http://v2.service when we want a specific version of the service to serve the request or using http://service when we don’t care.

In the next article we will add nginx to our software stack and configure it to run as Reverse Proxy / API Gateway for our microservices.

If you have any questions or want further details regarding any of the aspects of this tutorial – please leave a comment below.

]]>https://memz.co/service-discovery-skydns-docker-node-js/feed/1Service discovery for microservices with SkyDNShttps://memz.co/service-discovery-microservices-skydns-docker/
https://memz.co/service-discovery-microservices-skydns-docker/#commentsWed, 27 Jul 2016 06:21:33 +0000https://memz.co/?p=628This series is dedicated to DNS-based service discovery, reverse-proxying and load balancing of dockerized microservices using etcd, SkyDNS, nginx and a simple service built with Node.js and node-docker-monitor module. Part 1 – Setting up SkyDNS for service discovery of dockerized microservices Part 2 – Automating SkyDNS service discovery with simple Node.js service Part 3 – Setting up nginx as a […]

]]>This series is dedicated to DNS-based service discovery, reverse-proxying and load balancing of dockerized microservices using etcd, SkyDNS, nginx and a simple service built with Node.js and node-docker-monitor module.

What is DNS and how it can help with service discovery

In first approximation, DNS is a service that maps alpha-numeric, human readable and memorisable host names to corresponding IP addresses, required for establishing TCP/IP connection. Process of finding IP address for a given host name is called DNS resolution. It takes place every time when we type in a url in a browser or send an email. DNS resolvers are implemented and configured on OS level. They utilise caching to speed up the process and minimise load for DNS servers. All network client applications, whether it’s curl, sendmail or Google Chrome use these resolvers when establishing connection to destination services. Here are the most important benefits of using DNS (and host / domain names)

domain name is meaningful and easy to remember for people

domain name is permanent, while IP can change (for example when you move from one hosting provider to another)

one domain name can be backed by multiple IP addresses / hosts – for load balancing

one IP address (and server) can be used to host multiple domains – to save on hosting costs and simplify management

domain name is required for setting up secure SSL / HTTPS communication

Most of these possibilities can improve and enrich our hypothetical microservices environment. The most important, however, is that it gives any service a persistent and meaningful name by which it can be accessed. It enables easy service-to-service and consumer-to-service communication, load balancing and zero-downtime upgrades.

Dynamic nature of microservices (they get deployed, upgraded, and undeployed quite often) poses extra requirements for our DNS service – it has to be dynamic as well. SkyDNS is an API-driven DNS server backed by etcd – distributed key-value configuration store used by many modern services. It is a perfect solution for a problem we are trying to solve.

Setting up etcd and SkyDNS

I assume you have some flavour of Unix for OS and it has fairly recent version of Docker installed (I am using CoreOS 1068.8.0 with docker 1.10.3). We will install all services in docker containers just to simplify and unify instructions. Please note, that you can use etcd installed on your host machine – for production it would be more appropriate. Following instructions are not intended for any sort of production use – only for demo / dev purposes. If you want me to prepare instructions more suitable for production, please leave a comment below.

First we need to find IP address of your host. Don’t use loopback IP or virtual adapter IP. Adapter name you’re after normally starts with e*

I will use it in the following instructions, you, please replace it with IP address for your host machine.

Now, we install etcd in docker container. I am going to setup it to use non-standard port – 4002 (as I already have etcd running on my host) and non-persistent disk storage inside container (so don’t expect it to keep the data if you recreate container). We are starting docker container in bridged network mode (default) and mapping port 4002 to the same port on our host. Also, we are changing default etcd configuration by supplying command line parameters

ttl = 60 – default TTL (time to live) for DNS entries in seconds. It defines maximum time interval before DNS entry has to be revalidated by client (or for how long it can be cached). We are making it 60 seconds, which is a very short interval, because microservices are extremely dynamic by nature and we want all updates to DNS records to be propagated as soon as possible. This will, however, result in more queries coming to DNS server, having some performance impact.

domain = “cluster.local.” – “home” domain for our DNS – we will use it later to configure DNS resolution.

nameservers = [“8.8.8.8:53″,”8.8.4.4:53”] – “upstream” DNS for SkyDNS. When SkyDNS cannot resolve entry itself, it will forward request to one of those servers. This enables delegation of responsibilities – SkyDNS only resolves local resources (our microservices) and the rest is resolved by external DNS servers. 8.8.8.8:53 and 8.8.4.4:53 are IP addresses of Google publish DNS servers that should be accessible everywhere.

As we can see, when we are target host “service” we hit all 3 running services one by one – this behaviour called load balancing.

Obviously enough, there is a gap in our software stack – we have to create DNS entires manually. First getting IP address of a container, then use etcdctl tool for updating SkyDNS configuration in etcd. In the next article of this series we will develop a simple Node.js service to automate this task.

If you have any questions or want further details regarding any of the aspects of this tutorial – please leave a comment below.

]]>https://memz.co/service-discovery-microservices-skydns-docker/feed/2Memz 2.0 will get complete UI refreshhttps://memz.co/memz-2-0-will-get-complete-ui-refresh/
https://memz.co/memz-2-0-will-get-complete-ui-refresh/#respondMon, 04 Jul 2016 01:31:52 +0000https://memz.co/?p=618With next release (2.0) – Memz – our Notepad for Android will get complete UI redesign and support for Android Material look-and-feel. Navigation functionality will change slightly as current hierarchy indicator will be replaced with Tab-like indicator. Please tell us what you think about this update and leave your comments below.

]]>https://memz.co/memz-2-0-will-get-complete-ui-refresh/feed/0Android App Development with Kotlin / Ankohttps://memz.co/android-development-kotlin-anko-setup/
https://memz.co/android-development-kotlin-anko-setup/#commentsMon, 13 Jun 2016 08:34:14 +0000https://memz.co/?p=586This post is about setting up Android App project with Kotlin and Anko support. Over the last couple of years Development Tools and Frameworks for Android significantly changed and improved: Android Emulator got major performance improvement Android Studio has become an official IDE for Android Gradle is now default build tool Android SDK was redesigned and improved […]

I could continue on and on, describing all those nice changes that made Android Apps development easier, faster, more enjoyable and resulting apps nice and more stable. However this list would miss probably the most basic component – Java. We still need to use it for Android Apps development.

What’s wrong with Java, you might ask?

The short answer is – Java is not a MODERN programming language. It’s got a serious backwards compatibility burden. To make things worse, Android seems to be stuck with JDK 7 forever. With JDK 7 that doesn’t even have lambda functions. Fancy that! Android apps are event-driven by nature, and we don’t even get lambdas.

Missing lambdas is not the only problem Java has, there are many more:

null pointer unsafety

cumbersome generics

poor functional programming support

lack of extendability (operator overloading, meta-programming)

endless try-catch blocks

overall language verbosity

the list is endless, really. Many of the language’s flaws can be compensated by using specific library (for example to deal with null safety we can use Guava Optional), however it leads to other problems. As a result, we can’t write anything using Core Java anymore. To develop in Java it’s not enough to just know Java. We need to know and use bunch of frameworks / libraries, constantly upgrading them, migrating from old to new versions, be familiar with their features / bugs and so on. To summarise all of the above: Java at the moment is the most problematic component Android developers have to deal with.

Fortunately, thanks to Intellij (company behind IDEA – best IDE for Java development), we now have Kotlin – a relatively new modern language for JVM that is free of most of Java’s problems and has no performance overhead. Together with Anko – library for Android that lets defining Android layouts with Kotlin DSL, they make mobile app development truly enjoyable.

I will demonstrate how to create simple Android application with Intellij IDEA (you can use free Community version) or Android Studio (which is based on Intellij IDEA too).

Important! You don’t have to migrate the entire (existing) project to Kotlin. You can use both Java and Kotlin within single project, you can call Kotlin classes from Java and vice versa.

Setting up Android project with Kotlin and Anko support

First, we need to install Kotlin plugin for IDEA (I am using Intellij IDEA 2016.1.3 on MacOS). Go to “Preferences”, then search for “Plugins”, on Plugins page press “Browse repositories…” button and finally search for “Kotlin”. After plugin is downloaded you will be prompted to restart IDE.

Kotlin – enabling plugin

When IDE comes back up, create new project: “File” > “New” > “Project” and in following wizard select “Android”

Kotlin – creating Android project

You may change default parameters or leave default, then click “Next”. On “Add an activity to Mobile” select “Empty Activity”

Kotlin – creating activity

Move on in project creation wizard until the end. After project is created, open gradle build configuration file app/build.gradle and add these lines to enable Kotlin.

To create your first Kotlin source file, right-click on package com.example.kotlin.myapplication in folder app/src/main/java and select New > Kotlin File / Class from drop-down menu and create file MainActivityKotlin. Then type in following code to implement main activity with layout:

As you see, with Kotlin and Anko we can define layout together with event handlers and all required logic in a single file. It is especially convenient for creating highly customisable and reusable self-contained components. Unlike ordinal XML-defined layouts, layouts defined with Anko are type- and null-safe. They are, also, faster to load (XML-layouts need to be parsed and processed on mobile device during runtime).

To register our new activity, we need to update app/src/main/AndroidManifest.xml with the new activity name

]]>https://memz.co/android-development-kotlin-anko-setup/feed/2Memz – Hierarchical Notepad for Android Release 1.10.0https://memz.co/hierarchical-notepad-task-manager-android-release-1-10-0/
https://memz.co/hierarchical-notepad-task-manager-android-release-1-10-0/#respondTue, 07 Jun 2016 03:53:18 +0000https://memz.co/?p=571Memz – Hierarchical Notepad for Android has been updated – new version 1.10.0 As we previously received many complains about mandatory sync with Memz Cloud, we prepared this update. Starting with version 1.10.0 Memz doesn’t require Memz Cloud account setup on startup (for new users) It is possible to disable / enable sync at any point […]

]]>https://memz.co/hierarchical-notepad-task-manager-android-release-1-10-0/feed/0TypeScript for Node.js – using Visual Studio Codehttps://memz.co/typescript-nodejs-visual-studio-code/
https://memz.co/typescript-nodejs-visual-studio-code/#commentsMon, 23 May 2016 00:13:44 +0000https://memz.co/?p=533How to setup a simple Node.js project with TypeScript support in Visual Studio Code – Free and Open source IDE. As I explained in previous post, TypeScript brings more order to Node.js projects, it’s especially useful for defining APIs for multi-modular projects. Now I will show you how to create simple project with dependencies. Before we start […]

]]>How to setup a simple Node.js project with TypeScript support in Visual Studio Code – Free and Open source IDE.

As I explained in previous post, TypeScript brings more order to Node.js projects, it’s especially useful for defining APIs for multi-modular projects. Now I will show you how to create simple project with dependencies.

Before we start

I assume you have Unix-like OS (if you are still on Windows, you can wait until Microsoft integrates Ubuntu in Windows, you can use Linux VM with VirtualBox or other virtualisation solution, or you can use Cygwin) and fairly recent version of Node.js installed. Of course, you can use Windows as well, but you may need to slightly adapt this tutorial.

Additionally, we need to install some npm tool modules globally, as they’re used in most of Node.js projects:

Visual Studio Code project setup

Visual Studio Code is a lightweight Free and Open source IDE that supports many languages and file formats through extension modules. It’s available on all major platforms and has TypeScript support out of the box.

At the start, we open the project we’ve just created (All screenshots taken from Visual Studio Code 1.1.1 on Mac OS).

We will start by importing our project. Select “File” > “Open” from main menu, then navigate to the project folder and press “Open” button.

Visual Studio Code – open TypeScript project

Now we need to enable TypeScript compiler task for the project. To do that, open tsconfig.json file and:

Now we are ready to put our ideas into code. Right-click on src folder on left-side Explorer panel and type in desired file name – index.ts

We are going to use Node.js built-in module – fs to read data from data.json file.

Visual Studio Code – module not found

As we can see, module fs is not recognised by TypeScript compiler even though it’s a standard Node.js module. How is that possible? TypeScript compiler is independent of Node.js and doesn’t have any knowledge of what’s implemented in Node.js and what’s not. This means that any module used in TypeScript project needs to have TypeScript definition of some sort.

There are two different types of such definitions:

internal definitions – when .d.ts definition file is put inside npm module and distributed with that module. This is a preferred way.

external definitions – when .d.ts definition file is created and maintained outside of npm module it describes. This should be used only when npm module doesn’t have a TypeScript definition file and you can’t add it. This sort of definitions is managed by command line utility – typings.

In our case, fs it’s not even an npm module, therefore we have to import its external definition

In the project root folder run

typings install dt~node --save --global

Prefix dt~ before module name is source for definition. dt stands for DefinitelyTyped repository. You can find all available definitions for node with search command

]]>https://memz.co/typescript-nodejs-visual-studio-code/feed/1TypeScript for Node.js – using Intellij IDEA / WebStormhttps://memz.co/typescript-nodejs-intellij-idea-webstorm/
https://memz.co/typescript-nodejs-intellij-idea-webstorm/#commentsSun, 15 May 2016 02:29:45 +0000https://memz.co/?p=493How to setup a simple Node.js project with TypeScript support in Intellij IDEA / WebStorm As I explained in previous post, TypeScript brings more order to Node.js projects, it’s especially useful for defining APIs for multi-modular projects. Now I will show you how to create simple project with dependencies. Before we start I assume you have […]

]]>How to setup a simple Node.js project with TypeScript support in Intellij IDEA / WebStorm

As I explained in previous post, TypeScript brings more order to Node.js projects, it’s especially useful for defining APIs for multi-modular projects. Now I will show you how to create simple project with dependencies.

Before we start

I assume you have Unix-like OS (if you are still on Windows, you can wait until Microsoft integrates Ubuntu in Windows, you can use Linux VM with VirtualBox or other virtualisation solution, or you can use Cygwin) and fairly recent version of Node.js installed. Of course, you can use Windows as well, but you may need to slightly adapt this tutorial.

Additionally, we need to install some npm tool modules globally, as they’re used in most of Node.js projects:

Intellij IDEA / WebStorm project setup

Intellij IDEA is undoubtedly the best integrated development framework for number of programming languages: Java, JavaScript, TypeScript, Kotlin and many others. It’s a commercial tool, but I must say, it worth every penny. In case if you are only planning to do Node.js / JavaScript / TypeScript development, you can save some money by getting WebStorm, which is basically, stripped off and cheaper version of IDEA for Web development.

You can get a free trial license for a fully functional IDEA Ultimate for 1 month or, if you qualify, you can get discounted or even free license – for educational purposes or for working on Open source projects.

We will start by importing our project (All screenshots taken from Intellij IDEA Ultimate 2016.1 on Mac OS)

Intellij IDEA – creating TypeScript project

then select the folder we have just created

Intellij IDEA – selecting project directory

In the following project creation wizard just press “Next” button several times leaving all default options and then press “Finish”.

After project loads, we need to enable TypeScript support in project “Preferences” dialog. Make sure “Use tsconfig.json” option is enabled.

Intellij IDEA – enabling TypeScript

As we want to use interactive debugging for TypeScript files, we need to enable map files generation in tsconfig.json

Now, all preparations are done, it is time to start our coding exercise – create file – src/index.ts

Intellij IDEA – creating index.ts

First thing we were going to do – to read input data from file. We will use Node.js built-in module – fs.

Intellij IDEA – module “fs” not recognised

As we can see, module fs is not recognised by TypeScript compiler even though it’s a standard Node.js module. How is that possible? TypeScript compiler is independent of Node.js and doesn’t have any knowledge of what’s implemented in Node.js and what’s not. This means that any module used in TypeScript project needs to have TypeScript definition of some sort.

There are two different types of such definitions:

internal definitions – when .d.ts definition file is put inside npm module and distributed with that module. This is a preferred way.

external definitions – when .d.ts definition file is created and maintained outside of npm module it describes. This should be used only when npm module doesn’t have a TypeScript definition file and you can’t add it. This sort of definitions is managed by command line utility – typings.

In our case, fs it’s not even an npm module, therefore we have to import its external definition

In the project root folder run

typings install dt~node --save --global

Prefix dt~ before module name is source for definition. dt stands for DefinitelyTyped repository. You can find all available definitions for node with search command

Running with Node.js

To create “Run” configuration, right-click on index.js file and select “Create index.js …” option from drop-down menu. Then in dialog, make sure “Working directory” is set to the root directory of the project and “JavaScript file” is set to src/index.js

Intellij IDEA – create run configuration

then hit “OK” button and you are ready to go. You can run / debug project in IDEA or just run it in the console