Sunday, February 1, 2015

As of the 26th of January 2015, Greece has elected a new Government, which seems to strike a cord with the very heart of the European administration. Many were quick to point out that the new Greek Government is an anti-European one and came into power due to the fact that Greeks voted with a sentiment of anger. This may be true for a good portion of its voters, who felt betrayed by the two older traditional political parties (New Democracy and PASOK). However, there is a certain truth behind the reason Greek voters selected SYRIZA and that has nothing to do with sentiments against Europe. It has to do simply with the fact that Europe has mistreated Greece and many Greek people felt abandoned, been unable to make a living and look after their families.

As the Greek Government vows to re-negotiate the Greek debt on a European basis inside Europe, many key European politicians dismiss this need and refuse to listen to the Greek case, despite a wider European sentiment against the European policies that dictate austerity. In this article, I will prove why the reaction of the European administration is the wrong one. I shall present numbers that show why the Greek debt is not the problem of Europe. Europe faces an existential crisis and is currently unable to convince its member states about the sustainability of the collective European debt. Greece has of course been a problematic country, with systemic corruption and tax evasion. However, the worsening of the financial indicators in Greece and other problematic Eurozone members has very little to do with these factors and can be attributed more to the application of widespread austerity practices. These practices deprive the European South of vital abilities to restructure and develop their economies. This situation is not sustainable and its solution is not related on whether Greece remains or exits the Eurozone.

For the record, I claim no political affiliation or financial interest in the way I express my opinions. I would like to remain objective, and I welcome your comments.

The Greek Debt and its comparison to other Eurozone member debts

The Greek debt is best understood in comparison to other 'problematic' European countries. Throughout the article, I use the term 'problematic' to refer to Eurozone member states that have seen substantial worsening of their financial indicators. This worsening is of course due to the over application of austerity policies, which had the completely opposite effect than the intended one. I used the Google public data platform to fetch relevant data from Eurostat and the World Bank, in order to increase the verification of these results.

Graph 1: Government debt as percent of GDP - Source Eurostat

Graph 1 shows clearly that Greece has the highest Government debt as percent of GDP. However, Graph 1 also indicates two easily verifiable facts.

The first fact is that the problem is worsening not only for Greece, but for all other countries since the beginning of the Eurocrisis (starting from 2008). Despite the austerity measures suggested by the Troika 'experts', Spain, Italy and Portugal see their GDP debt ratios increase consistently and substantially, a sign of their worsening economies. In addition, other members of the Eurozone that are not displayed in Graph 1, have also seen notable increases of the debt/GDP ratio. A few notable examples include Ireland whose ratio jumped from 44.2% in 2008 to 124.2% in 2013 and Belgium that went from 89.2% (2008) to 102.4% in 2013. All of these countries have seen austerity measures decided by the Troika or various national governments imposing a policy dictated by Brussels.

For Greece in particular, one should note two key dates. The year 2001, when Greece started using the Euro and the year 2008, when Greece started entering the spiral of the financial crisis. Systemic corruption and tax evasion in Greece were present before these dates. However, the severe worsening of the financial indicators occurred after the year 2008 and spiraled out of control after the application of the Troika policies. A classic example of the application of too much medicine on a patient that needed it, but not in that quantity.

What did the austerity measures do for the national economies of these countries? To answer this question, we have to come to the second fact that reveals the complete picture of what happened to Greece and to some extent the rest of the problematic European economies. Graph 1 displays a clear economic deterioration, but what exactly has caused that deterioration. There are a lot of opinions about the South of Europe tied to stereotypes of lazy people, corrupt politicians. I dismiss these theories, because lazy people exist all over the world. Corruption is systemic in many South European countries. However, corruption itself is not enough to cause this type of economic deterioration. The answer lies in what happened to the GDP of these countries for the same period of time, as displayed in Graph 2 below.

Graph 2 shows the notable GDP reduction in various Eurozone countries. There is an important detail that is not easily shown in Graph 2. Starting in 2008, the Greek GDP went from 350 Billion Euros to 242 Billion Euros in 2013. In five years, 30% of the GDP disappeared from the pockets of the Greek citizens. We have rarely seen such a rapid GDP reduction in the global financial history, which displays clearly the weight Greek citizens had to lift on their shoulders, as a result of the austerity. In comparison, Ireland, Portugal, Spain and Italy lost approximately 8%, 10%, 10.4% and 5% of their GDP respectively, during the same period of time.

The ostricism of the Troika and Brussels has to do with the fact that everybody is talking about Greece, like it is the largest problem of Europe. I am afraid this is a very flawed attitude. Brussels (and Germany in particular) hide their head in the sand and avoid to see the wider picture, which consists of the the actual number of Euros owed by each one of these countries. The next graph indicates the actual amount of Euros owed by each of the problem countries, because not everyone can actually appreciate the effect of the GDP debt ratio in a common currency (unless of course you are an economist).

Graph 3: General government debt in Euros - Source Eurostat

Graph 3 shows clearly that Greece is not really the factor that could derail the Eurozone train. Italy and Spain owe collectively more than 3 Trillion Euros, a staggering amount of money that cannot be absorbed by any corrective measures. These 3 Trillion Euros exclude the extra 2 Trillion Euros of France, a large part of the European economy that is also start becoming problematic.

Let's assume that Europe does not find an agreement with the SYRIZA government and develops the legal/procedural framework to oust Greece from the EU, in order to demonstrate what happens to the bad boys that do not keep their promises. It is true that Europe can today catch the 250 billion Euro bullet owed by Greece. However, Europe will be fooling itself. Because although it may absorb the shock waves of the comparatively small Greek debt, it will not be able to absorb the trillions of Euros owed by the rest of the stagnant national economies of the Eurozone. The reasons are simple and evident.

Graph 4: Eurozone Unemployment rate - Source Eurostat

First of all, despite a buffer of 1.1 Trillion Euros that the ECB can dedicate to kickstart the economies, Europe does not have policies, mechanisms and a clear plan to make that kind of money work, because the North and South are divided in theory, practice and culture. For more than seven years now, Eurozone officials have failed to tackle vital issues such as the unemployment issue (Graph 4), more evident in the European South. The lack of developmental capital and vision to help the South retain young people in Greece, Spain, Italy and Portugal has led to massive migration of the productive workforce to Germany, the UK, the Scandinavia and even outside the European continent. How these countries can build a tax base to fund functioning states is a good question. As far as I know, nobody has built successful tax systems out of pensioners, students and people who have partial or no employment at all.

As the elections in Spain are nearing and the size of the Spanish debt is much higher than the Greek one, I am sure that these data can convince even the greatest hardliners in Europe to listen to the Greek case. I am certain that the voice of reasoning in Europe will win. Greece, even battered, will eventually exist with or without the Euro (my preference is within the Euro). I am not sure that the Euro will exist without a change of policy. Greek citizens have already lost a lot (some have nothing to loose any more) and Europe should listen very carefully this time.

Tuesday, July 1, 2014

The 10th of June 2014 was a special day for the Linux world, as Redhat released the much anticipated version 7 of its Enterprise operating system product RHEL. Code named 'Maipo' and having gone through a Beta (11/12/2013) and Release Candidate RC (23/04/2014) stage, RHEL 7 is now available (GA) and is based on Fedora 19 and the upstream kernel 3.10.

Following the blog tradition started with a 'first impression' article on RHEL 6 some years ago, I will do the same here and present a number of goodies that will keep the sysadmins/devops folks busy, in the process of deploying RHEL 7 in their production environments.

Rather than replaying in detail the excellent info of the RHEL 7 Release notes as well as the nice roadmap presentation of the 2014 RedHat Summit, I will just summarize the important changes that it brings to the daily work of sysadmin/devops teams accompanied by some hands-on videos on the concepts.

The very first thing you will notice is a 'Fedorazation' of the Graphical Installation environment. Yes, those of you that are not familiar with the new graphical Anaconda environment of the latest Fedora releases, you will discover that the old RHEL 5/6 serialized wizard menus are gone in favour of a central screen where you click to setup various aspects of the installation AND then you launch the install. You can find a video below that demonstrates a basic RHEL 7 installation on a VM environment.

Another major change you will feel from Day 1 is the obligatory-ish interaction with systemd. Faster boot times with parallel service startup and increased manageability have their price: The days of init scripts and the 'services' command are gone-ish for RHEL. I am using the '-ish' bit because although Redhat chose to maintain the traditional concepts of runlevels for backwards compatibility (read: to prevent most non systemd friendly sysadmins to start screaming), this will certainly not be the case for the near future of RHEL 7 minor releases. Thus, now is a good time to start thinking in terms of units and targets, as opposed to init scripts and runlevels. Below you will find a video where I demonstrate the concept of interacting with systemd to start and stop services, or change the default runlevel, erhh, sorry, target of your system. Once you get used to 'systemctl', your new buddy, things will not look so bad/alien.

You should get used to the idea of using Linux Control Groups (cgroups). The cgroup concept is not a new one, however now is the time to get used to the new tools for administering cgroups, as the technology is one of the conceptual blocks that RHEL 7 brings in the arena of Linux Containers (see latter paragraphs). To help you, I have made a video that demonstrates how various systemd commands can be used to administer cgroups.

In the filesystem arena, you have some notable supported volume capacity expansion. In particular, XFS which is now your default filesystem (that's right, even / on an automatically partitioned install is on XFS by default) can support volumes of up to 500 Tbytes (in RHEL 6 the limit was 100 Tb). ext4 is still an option and supported in volumes no greater than 50 Tbytes (RHEL 6 had them limited to 16 Tb per volume). My advice is to really not use ext4 again on your migrated RHEL 7 systems. As I wrote four years ago in the RHEL 6 first impressions article, XFS was the way to go in terms of performance and scalability. Today, XFS is mature enough to trust for production and can be tailored to medium/large size volumes. I just do not see ext4 fitting somewhere anymore.

I have yet to produce some comparative performance figures of XFS running on the same hardware for RHEL 6 and 7. Stay tuned for this, as it takes time and effort to do some meaningful comparison.

Staying on the filesystem arena, Btrfs is offered as a tech preview on RHEL 7, but I am not happy at all with its current state (kernel 3.10.0-123.4.2.el7.x86_64 #1 SMP Thu Jun 5 21:43:43 EDT 2014). During some iozone benchmarking, btrfs broke down and that's quite disappointing. If one considers the fact that btrfs represents really the ZFS commercial Unix arena equivalent of the Linux world, even a tech preview should have been more stable. Redhat folks, I am looking forward to a more stable btrfs tech preview, please work on that task.

For those of you that are development oriented, some major version components include:

GNU bash, version 4.2.45(1)-release

gcc version 4.8.2 20140120 (Red Hat 4.8.2-16)

Perl is on version 5.16.3 built for x86_64-linux-thread-multi

Python is on version 2.7.5

This should really update some long outdated components of RHEL 6 (especially on the gcc side of things) that were seriously hindering software development without some effort to install separate libraries, compilers, etc.

Finally, for the devops oriented folks, one of the greatest technologies that RHEL 7 is introducing is that of containers. In essence, a container is a way to package your tech apps using a lightweight environment that provides resource isolation. This cannot only seriously save some memory and CPU cycles (in comparison to hosting apps in virtualized OSes), but it can also please your developers by introducing Docker Image based containers, a technology that allows you to speed up portable application deployment. I have made a video that demonstrates the basics of using Docker (see below) and I hope this will help you orient yourselves around the technology.

Whether Docker containers will eventually replace your hypervisors is hard to say. Containers do not provide (yet) features such as redundancy and live workload migration, two business important features offered by most virtualization technologies today. Nevertheless, application serving and deployment via Docker containers is more efficient than throwing virtualized operating system images. I shall place some concrete numbers behind that claim in order to convince you to start looking at the technology.

First of all, I shall explain how I obtained those figures. I performed the tests on two identical servers ( Dell PE 1950s) with the same amount of RAM (16 Gigs), the same CPUs ( 8 cores Intel(R) Xeon(R) CPU E5345 @ 2.33GHz cores) and with identical disk drives and I/O controllers. One of these servers was running a standard KVM environment on RHEL 7. The other was running a Docker container engine on RHEL 7.

The next steps were to install:

An Apache server on the RHEL 7 KVM hosted VM (the served VM is on Fedora 20)

Docker pull and run the Fedora/Apache image from the Docker repository on the second server, to approximate the same environment as the VM on the Linux container. I also had to run the container by doing a:

docker run -p 8080:80 -i -t --name=thirdcon fedora /bin/bash

in order to give it a port on a network (testing on the local server and
bypassing the network latency would not yield a reliable measurement)

At that point, I had two web servers running, one in a VM and one in a Docker container. I configured both Apache s to serve a simple text index.html file (no images, graphics).

I then employed the ab Apache benchmarking tool from a third independent host and for each server URL, I fired a benchmarking load with different levels of concurrency like this:

ab -c 2 -n 100000 http://serverURL/index.html

ab -c 4 -n 100000 http://serverURL/index.html

ab -c 8 -n 100000 http://serverURL/index.html

representing different numbers of concurrent requests, up to the number of cores of the VM and the container. The results are summarized in the graphs below and they are quite revealing.

The first graph shows the maximum number of concurrent requests per second achieved in each concurrency level. Docker clearly wins over KVM with just over 8000 requests per second. You should also note that the more we increase the concurrency level, the greater the difference in the result between the two technologies.

The second graph plots the time-to-completion in seconds for each benchmark. Again, Docker is faster by a margin of several seconds. That might seem small, but it really is not. A few seconds here and there, in loaded servers running larger workloads concurrently means a great deal.

This concludes the original first impression assessment of RHEL 7. I hope this contributed towards convincing you to give it a go.