It was decided to use the following Roadmap to be able to provide some useful Components already and to have something to extend on.

Note, that this roadmap is not fixed yet and it could be, that some mentioned features are coming in later versions. This roadmap should just describe the rough idead where DevOpsKube is heading to.

Version 0.1

All Components mentioned in the first post should be provided using MySql. Furthermore the configuration for these components is provided and docaumented. All necessary steps to setup a single node cluster (based on CoreOS) will be documented as well.

This will be a pre-liminary version to provide all the components and steps to build up the future „development“ environment.

Version 0.2

Add additional components to the Stack to provide a fully featured SDLC Stack. These components could be:

Version 0.3

This version should already provide SSO functionality for the defined components.

Version 0.4

Additional functionality to be able to create projects via a single REST-API call. This is the first version with some unique functionality. The REST-API should include a web-based client as well as a Command-line client.

Version 0.5

Make all of the components (if upstream allows) HA-able. Furthermore integrate those as much as possible.

Version 0.6

Be self-hosted. We should eat our own dog-food, therefor we should host this project on our own Kubernetes Cluster.

At the company where I work at currently we do have several, around 70 to be precise, inter-related Components. Some of those Components are multi-modules, which makes a fully Automatic release-build even harder. The used SCM is git, awnd we do use Nexus as a Central artifact repository.

What are the Problems of the Release Generation Process? What are the manual steps, why are they manual? Where can failures happen in this Process? Are there any other Problems, which need to get resolved to make the Process easier and (hopefully) fully Automatic?

Well, identifying the Manual steps is quite easy, to replace those with automatisms will help but can be quite expensive or even plainly impossible. Lets take a look on those steps:

Check components if there are any dependencies, which are updated, and update those

Check components for changes to see where we do need to create a release

Create a release of this component

Check for updated dependencies

For some components, that are used throughout the whole Application and should be used in the same Version all over the place, we do have to update all other components, even if they do not use the latest SNAPSHOT version. This is mainly true for the Model of the core component, which is used in all Components accessing the DB and/or the Rest API. To make sure, that the same Version is used everywhere, the version is adopted in all released components manually. This is obviously rather error prone, since one can overlook some components and their dependencies.

Other Components are shared as well, but could be used in different versions in different Components. Here the developer decided which version to use in her component. This is error prone as well, since it can lead to dependencies issues via transitive dependencies.

The above mentioned points hint to architectural smells in the system, but to resolve those, would be a longer project and we wanted to make the whole release process easier quite urgently to be able to have some time to resolve those architectural issues.

We decided to just update all dependencies (internal ones, that is) to the latest version. This has still the disadvantage that some changes on one component are not probably tested on the Development and Staging Environment, but at least the reason for this is quite obvious and not hidden in „transitive“ dependencies like up until now.

How to update all dependencies to the latest version? Lets take a look at the Maven Versions plugin, which comes to the rescue.

versions:update-parent will update the parent version to the latest available version

versions:update-properties will update the versions of all dependencies in our project

That looks pretty simple, does it? Well, there are some drawbacks in these steps. One should avoid to update all dependencies (eg. used frameworks like spring) but just internal dependencies. This can be done using the „includes“ property of the plugin-goal. Problem here is, that you should know all groupIds (that is the classifier, we do use) of all internal dependencies before running this goal. Other then that, multi-module projects do have some problems here as well, in that the properties defined in the multi-module-master but used in the sub-modules are not updated correctly. That is why we defined that a new dependency has to be declared in the dependencyManagement in the multi-module-master. This is error-prone, since each developer has to follow this, but we do seldomly declare new internal dependencies in this phase anymore, so that this problem is minor.

To be able to recognize this change in the following steps, you need to make sure, that the changes are commited and pushed and to be on the safe side during the release preparation, make sure that no files (mvn versions will create a backup-file of your pom) are still lying around (mvn versions:commmit).

Check for Changes in Components

To see if a component has any changes, we should take a look into the Logs of the version control system. We should start with this on the bottom of the dependency hierarchy, so that if the component has changes, we do release it and all components above can be updated. In a manual step, we are usually going from the top to the bottom, to be sure, that all dependencies are met. This ist depending on the developers, which declare a dependency as „SNAPSHOT“, if they do need a later version. The problem with this approach is, that sometimes transitive dependencies for components, which do not use the SNAPSHOT version are updated, without the developer knowing. This could lead to problems, if the version changes method signatures, which will stay the same if the version updates are done automatically. Therefor some common rules for versioning should be defined (see Semver). Furthermore we do have to make sure, that all components are considered.

This can be automted quite easily in just comparing the last known „release“-commit with the current commit and decide if there are any changes in between.

Some helpful commands for this are:

git log --all --grep='[maven-release-plugin]' --format='%H' -n 1

The above command shows all commits containing the maven-release-plugin pattern. To see, if there are any changes after this commit, you need to grep the commit id from the latter command and do the following command:

git rev-list "$commit_id"..HEAD

Before running these commits, please make sure, that you are on the correct branch (in our case master or rc):

git ls-remote --exit-code $GIT_URL/$project refs/heads/$branch

Create a release

The creation of a release using the Release Plugin of Maven is not as painful as some people think, and works quite well for us since a long time. In the past, we did create a release manually via the Jenkins Plugin. To be able to let the release process run automatically without an interruption due to questions of the new version, you should use the command-line flag „–batch-mode“ for the maven call.

To make sure, that the release is working correctly and no changes on the git repository are made, you should use a „dryRun“ beforehand.

Automate It (Full Pull)

To automate the above mentioned steps, we do need a list of all projects in the correct sequence, which is manually created and adopted, as soon as some dependencies change and/or new dependencies are created (this make the whole process error prone as well, but up until now, there is now real alternative to it).

This list of projects is then processed in the given sequence. Each project is cloned and version bumped. After this changes are checked. As soon as this is done and there are changes, a release is created.

To make the implementation easier, we re-used the bash-libs from the kubernetes release project (mainly common.sh and gitlib.sh), but with some major extensions. For the maven calls, we created our own mvnlib extension.

To provide central data-storeage options we are using MySql for all components in need of a database.

Helper

Since a vanilla Kubernetes Cluster does not provide any Edge-Loadbalancer, we are going to provide documentation on how to setup the whole cluster including an Edge-Loadbalancer and other useful tools (eg. SSO Components). An initial set of Helpers are:

What is DevOpsKube?

A Kubernetes „cluster“ with all things SDLC (Software Development Life Cycle)/DevOps related. This Solution is going to be implemented so that it Can be installed onto a one node cluster. In later phases also big installations should be possible.

What is the Motivation behind DevOpsKube?

It was/is the dream of myself to be able to install such a full SDLC stack easily on own Hardware and/or environment. Kubernetes seems like the natural choice for this.

Up until now there is no full DevOps Stack for vanilla Kubernetes. The following „solutions“ can be found when googling:

fabric8 – highly based on RedHats OpenShift, probably the closest to my requirements

Marcel Birkner – a blog-post from codecentric (merely an blog-post with some docker containers)

GoPaddle – seems to be focused on a full CI Environment to install Container on Kubernetes and uses eg. Jenkins, but is commercial and has a different focus then the above mentioned ones

None of the above mentioned stacks do fulfill my needs/my idea, therefor I decided to put this stack together to allow to install a full SDLC stack on an own Kubernetes cluster or even on the Google Cloud.

So watch this place closely during the next couple of days, because Herr you will read First as soon as this Project is started and where you can reach it.

So what do you think? Are you interested? Would you like to contribute? Please just leave a comment and/or write me an email.

EDIT 2016-10-09 16:10: Updated the links, so that they do work in WordPress

I tried the everywhere mentioned method with https://github.com/gnab/rtl8812au, but this was not working as expcected. Therefor I tried the method mentioned on the Fedora Mailing List. This does not contain the patches for the Raspberry PI. The Makefile needs to get adopted, so that you can compile this driver on the Raspberry Pi.

The compile seems to work now. I have uploaded all sources to Github. In this repository is also the 8812au.ko, which is compiled on a Raspbian Box with Kernel 3.18.5+. Hope this will work out now.

Finally I have made it and ripped the whole CD collection of us. The whole collection can be seen on discogs. I will add new CDs in there, if those are added to the ripped cd. The whole process took more then one and a half year (well, I really starte end of Mai 2013), but I guess, it was worth it.

I am a proud owner of an Garmin etrex 30. Since I do run Linux on all my machines at home, I tried several ways to update the firmware of this device. Unfortunately this is not possible, using WebUpdater from garmin directly. A tool which should make it easy (and most probably it does on windows and/or mac) makes it hard to do on Linux.

Anyways, a download of the lates firmware can be found at http://garmin-etrex.wikispaces.com/firmware. You just need to rename the Downloaded file to GUPDATE.GCD and move it to /Garmin/ on your Garmin device.