"Micro-services is the new black" - Splitting the project in to independently scalable services is the currently the best option to ensure the evolution of the code. In Python there is a Framework called "Nameko" which makes it very easy and powerful.

Micro services

The term "Microservice Architecture" has sprung up over the last few years to describe a particular way of designing software applications as suites of independently deployable services. - M. Fowler

I recommend reading the Fowler's posts to understand the theory behind it.

Ok I so what does it mean?

In brief a Micro Service Architecture exists when your system is divided in small (single context bound) responsibilities blocks, those blocks doesn't know each other, they only have a common point of communication, generally a message queue, and does know the communication protocol and interfaces.

Give me a real-life example

Consider you have an REST API, that API has an endpoint receiving some data and you need to perform some kind of computation with that data, instead of blocking the caller you can do it asynchronously, return an status "OK - Your request will be processed" to the caller and do it in a background task.

Also you want to send an email notification when the computation is finished without blocking the main computing process, so it is better to delegate the "email sending" to another service.

Scenario

Show me the code!

Lets create the system to understand it in practice.

Environment

We need an environment with:

A running RabbitMQ

Python VirtualEnv for services

Python VirtualEnv for API

Rabbit

The easiest way to have a RabbitMQ in development environment is running its official docker container, considering you have Docker installed run:

NOTE: The init() calling will initialize the index/doctype mappings and settings, this part can be omitted and then Elastic Search will try to create this by introspection when the first document is indexed.

With the model definition in a file like myproject/models.py we can now use the model class Person to Index(insert), edit, delete and of course search documents.

What is Swagger?

Swagger is a simple yet powerful representation of your RESTful API. With the largest ecosystem of API tooling on the planet, thousands of developers are supporting Swagger in almost every modern programming language and deployment environment. With a Swagger-enabled API, you get interactive documentation, client SDK generation and discoverability.

What is Swagger UI?

Swagger UI is a dependency-free collection of HTML, Javascript, and CSS assets that dynamically generate beautiful documentation and sandbox from a Swagger-compliant API. Because Swagger UI has no dependencies, you can host it in any server environment, or on your local machine. Head over to the online demo to see what it looks like for any publically accessible Swagger definition.

What is Flask? (duhhhh!!)

Flask is a microframework for Python based on Werkzeug, Jinja 2 and good intentions. Why it is awesome? because it is simple yet powerful, talking is cheaper look at the code!

What is Flasgger?

Flasgger is a Flask extension to help the creation of Flask APIs with documentation and live playground powered by SwaggerUI. You can define your API structure using YAML files and Flasgger creates all the specifications for you and you can use the same schema to validate the data.

Contribute

Dynaconf

dynaconf - The dynamic configurator for your Python Project

dynaconf is an OSM (Object Settings Mapper) it can read settings variables from a set of different data stores such as python settings files, environment variables, redis, memcached, ini files, json files, yaml files and you can customize dynaconf loaders to read from wherever you want. (maybe you really want to read from xml files ughh?)

What is Dynaconf?

Dynaconf is a common point of access to settings variables, you import only one object in your project and from that object you can access settings variables from Python settings file, from environment variables, from parsed yaml, ini, json or xml files, from datastores as Redis and MongoDB or from wherever your need if you write a simple dynaconf loader.

How it works

Install it

Use it

By default Dynaconf will try to use a file called settings.py on the root of your project, if you place that file there all upper case variables will be read

You can also replace the file exporting an environment variable pointing to the module or location for the settings file.

# using module name
export DYNACONF_SETTINGS=myproject.production_settings
# or using location path
export DYNACONF_SETTINGS=/etc/myprogram/settings.py

Doing that when you use from dynaconf import settings the variables will be read from that file.

So how it is Dynamic?

Now think you have your program done and you want to deploy to a certain infrastructure for testing or maybe different deployment, you don't need to rewrite the settings file. Just export some variables to your environment.

export DYNACONF_MYSQL_HOST=myserver.com

Now in your project you can do:

from dynaconf import settings
print settings.MYSQL_HOST
myserver.com

The default prefix for exported envvars is by default DYNACONF_ but you also can change it if needed.

But what if I have some typed values to export?

You can also define type casting when exporting and those types will be used to parse the values.

And of course you can now read those variables in the project, all the casting wildcards also works on Redis but if you want to skip type casting, write as string intead of PORT=1234 use PORT='1234' as redis stores everything as string anyway.

There is more

Dynaconf has support for using different namespaces in the same project, you can also write your own loaders, you can find more information on the repository https://github.com/rochacbruno/dynaconf

Contribute

All contributions are very welcome!!

Acknowledgements

Dynaconf was inspired by Flask app config and also by Django settings module.

In docker-compose a common problem is starting services and daemons in containers that depends on services running on linked containers, in example: your app depends on elasticsearch but it is not ready when the app container is started. Solution is to use a wait script.

I knew that docker-compose team is working on adding a WAIT parameter, but while it is not ready we can use a wait script to load our services.

The process is simple, your container will execute a shell script that will try to execute a HEALTH CHECK many times you want before starting the main command. The health check, the command, the sleep and how many loops to try will be defined in environment variables

This is a program that needs to access an elasticsearch server located in 'elastic' host (that will be mapped by compose

Now when you start your environment with docker-compose up the app container will start and execute wait_to_start script, which will perform the test defined in WAIT_COMMAND environment variable and will retry until the test succeeds, so it will end executing the program defined in WAIT_START_CMD

As a micro framework Flask does not have built-in cache functionality, however, there is werkzeug cache API and an excellent extension to provide its caching functionality to your Flask apps,
that extension was created by @thadeusb and is very easy to implement and use.

installing

In your env install it via PyPI (recommended)

pip install Flask-Cache

You can also install it directly from source code if you need recent changes or bugfixes

pip install https://github.com/thadeusb/flask-cache/tarball/master

Configuring

There is a set of configuration keys you can put in your app settings, but the most important is the cache backend defined by the CACHE_TYPE key.

The cache type resolves to an import string which needs to be an object implementing the werkzeug cache api, but there is some aliases to the werkzeug.contrib.cache implementations

By default the CACHE_TYPE is Null which means that your app will have no cache, so you need to choose of the options below:

null | No Cache - NullCache

simple | Will use in memory pickle and is recommended only for single process development server

A Simple Flask app

Run the above with python app.py and open http://localhost:5000 in your browser and hit F5 (refresh) to see the current date and time.

Enabling Flask-Cache for views

Now we want to enable caching on that small application to avoid the refresh of the current time (for this example we are using current time as return but imagine that it could be a large dataset or huge calculations)

Run the above with python cached_app.py and open http://localhost:5000 in your browser and hit F5 (refresh) to see that the current date and time is now cached for 5 minutes.

@cache.cached decorator takes the request.path for that view and use this as cache key, if for any reason you need a different key you can pass a key_prefix argument to the decorator. In this case if you pass a key_prefix containing the %s placeholder it will be replaced by the current request.path

The above is the simplest and regular example of Flask app and the use of cache, but, if your app is designed using application factories, blueprints, class based views or views located in different modules you will need to use advanced approach.

Caching regular functions

The same cached decorator can be used to cache regular functions, but in this case you will need to specify the key_prefix argument, otherwise it will use the request.path which can lead to conflicts if you have many cached functions.

For this example we are going to use the this module and extract a random quote from the Zen of Python.

Now running python cached_function_app.py and opening http://localhost:5000 when hitting F5 to refresh you will see the current time cached for 5 minutes and the random quote updated, you can switch the cache just to see the efect.

as you can see the above file defined view functions, as it it a separated file, to avoid circular imports we are not recommended to use @app.route neither @app.cache so this views will be app agnostic and we are going to register its url rules and caching in the main app file.

That kind of structure is needed when your app has too many views and want a better organization.

NOTE: For better organization the mostly recommended pattern is Blueprints which I will explain further.

app/app.py

Now in the main app we need to import our views, explicitly decorate for caching and also register its urls.

NOTE: You can also separate the cache instance in a different file for lazy initialization as we are going to see in the next example

Caching Blueprint views

As mentioned before, the best pattern to follow in Flask applications is the Blueprint pattern which is a way to create separated 'meta-apps' that will be connected to your main application in the time of initialization, the problem here is that Blueprints are meant to be reusable by many different applications, so the delegation of cache control should be dynamized.

In order to avoid circular imports you will want to create your cache instance separate from your application instance (you may want to consider switching to the app factory module if you are building something more complex).

we can create a dummy lazy cache instance, that will be initialized in the future when the view will be called. For that in the app we are going to reimport the same cache instance and call init_app method.

Notice that we created a dummy instance of cache in cache.py and then used that instance to decorate the blueprints views, then the cache was initialized in app.py with init_app method. That is possible because of the Flask initialization cycle and the excellent implementation in Flask-Cache extension that takes care of this case, if you plan to write yor own Flask extension take a look at the Flask-Cache source code.

Run the application by calling python cached_blueprint_app/app.py and open http://localhost:5000 to see the blueprint view cached for 20 seconds.

Caching MethodView

Lets use the same cached_blueprint_app example but turning the zen_view in to a MethodView

Method views maps HTTP method names as GET, POST, DELETE to the view methos as get, post, delete etc, So all we needed to do is to create a method called get and decorate it with @cache.cached decorator.

NOTE: Due to the implicit self from the caller’s perspective you cannot use regular view decorators on the individual methods of the view however, Flask-Cache is one exception because its implementation allow the use of cached decorator in individual methods. Keep this in mind.

Alternativelly you may want to cache all the methods in a view, for that you can cache the dispatch_request method or even better you can decorate the whole view.

Now save the file and refresh to see the content cached for 30 seconds.

Caching functions and views with variant arguments using memoize decorator

Sometimes yout views and functions receives arguments which can come from url mapping or directly to the function call, yiou may want to cache the view or funtion and use the arguments as keys to cache its different results, Flask-Cache has a different decorator for doing that.

NOTE: With functions that do not receive arguments, cached() and memoize() are effectively the same.

WARNING: Some backend implementation do not support completely clearing the case. Also, if you’re not using key prefix, some implementation (e.g. Redis) will flush the whole database. Make sure you’re not storing any other data in your caching database.

There is a lot of examples and well documented API in flask-Cache website http://pythonhosted.org/Flask-Cache/ you can also create your own cache backend following the examples in the Flask-Cache docs.