Henrik Lynggaard's blog

Tuesday, 7 February 2012

I have previously written about the challenges of integrating an Ivy based subproject (like the play framework) into a build that is otherwise Maven based. After some work it is now working although the support feels a bit rudimentary.Here are the highlights on how to make it work

Saturday, 14 January 2012

It might seem very counter intuitive to start using Hudson's XML format to manage Hudson jobs when it has such a great web interface, especially since it is the web interface which has made Hudson so approachable and easy to use.

Don't get me wrong. I still love the web interface and think it is the right way to get people to use the tool, however I also think there comes a time when you outgrow the web interface and this is why I built the jobcreator tool.

These are the requirements and issues that caused me to outgrow the web interface:

Manual changes doesn't scale
Using the web interface to make changes to individual jobs is very easy, but it doesn't really scale if you need to change a lot of jobs on the same time. One example could be changing the git branch from "master" to a release branch for all the jobs related to a environment, Such a change could involve upwards of 30 jobs.

Another example could be that there are changes to the content and/or structure of the jobs and those must be propagated up through the environments in sync with project code.This would be even more manual changes.

Managing this with manual changes in the web interfaces introduces a big risk for human error and inconsistencies.

Hudson jobs are code
If you want to be able to reproduce a build or deployment at a later time it is important that you can also reproduce the Hudson jobs. In order to do this you need to store and version your Hudson job configurations.

This is naturally best done in a SCM like Git or Subversion. Doing so also gives you the option to do branched development of your jobs.

Testing and more than one Hudson instance.
Before making changes to the jobs being actively used the changes should naturally be tested somewhere. This normally means creating the same set of jobs somewhere else or targeted towards a different environment.

Having the jobs defined as templates makes it easy for a developer to load the jobs into a private Hudson instance to experiment or share it on the testing instance.

Overall
Of course Hudson can be managed via the web interface, but for me it is just another step in the automation.

Friday, 13 January 2012

I have finally had the chance to pull together the last changes before announcing the first version of the Hudson Job Creator tool.

The idea behind the tool is that you can write FreeMarker based templates and combine those with properties defined in a "pipeline" specification in order to generate Hudson's job config.xml files.
This is mainly useful if you maintain a number of similar jobs, or have a series of jobs that you need to specify for multiple environments.

I know working with Hudson's xml files directly can seem counter intuitive since one of Hudson's main strengths is its approachability and easy to use web interface, so later this week I will post a more in depth blog post
explaining why I chose to go this route.

This bug has now been fixed and a new version of the release plugin has been released. So if you update the release plugin to version 2.2.2, Hudson's maven3 integration will now work for releases also.

Every hour deploy and test the latest successfully build artefacts to dev #1

Every day deploy and test the latest artefacts which have been successfully tested in dev #1 to dev #2

On demand do a Maven release of the latest artefacts that have passed testing in dev #2

The tricky part of this pipeline comes in the dev #2 environment because it needs a way to select the latest artefacts which have been tested in dev #1 instead of just the latest artefacts produced. The same goes for making the release, I need some way to identify which artefacts have been tested successfully in dev #2.

This means I cannot rely on just picking the latest snapshot deployed to the internal repository.

We have considered different options:

Publish the snapshot to a internal repository and carry around the timestamp of the specific snapshot version. This would allow us to pin out the version, but gives us a problem with regards to cleaning up unused snapshots as Nexus can only do keep X days or X versions, but not remove a set of artefacts based on a timestamp. Also it has the downside of people questioning why we should even do Maven releasing if we already have a unique identifier.

A second approach would be to use the "copy artefacts" Hudson feature, but then the way we get artefacts would be different between dev and the upper environments.

The approach we have settled on is to not deploy the snapshots to the nexus repo, but use the hudson maven repository plugin. This plugin exposes each build as its own repository. In order to get the right artefacts we use a custom settings.xml to mirror the snapshot repository to the URL of the specific build as exposed by the Hudson plugin. This plugin only works with either the jobs Maven 2 jobs or the new Maven 3 integration, since it relies on Hudson understanding the build and the artefacts. We use the promoted builds plugin to identify the correct build, and we use the promotion status for easy clean up of non used artefacts.

We haven't looked too much into using nexus pro's features such as staging repositories or adding metadata, since the above approach works fairly well for us.

The new challenge:
The reason for writing this post and asking for help is a possible change to our process which I am not sure how to best integrate.
There is a wish to integrate a component (playframework based webapp) build using ivy into this framework and specifically into this project. I can see this causing some integration pains

Firstly the project is one big Maven multi module project and we prefer to keep it that way. As a very least we want to keep things as one build i.e one Hudson job for building. Is it possible to have a Maven submodule defer execution to Ivy ?

If we get the component built using ivy only, then Hudson will not be able to see the outcome as a Maven artefact, and as thus wont be able to expose it as part of the "repository per build". That is at least my strong suspicion.

What to do ?
So does anyone have a suggestions on how to resolve this ?

Can we cleanly integrate Ivy into our current build, if so how ?

Do we need to find another approach than using the "repository per build" plugin ? if so, what is the suggested alternative.

Would Nexus/Artifactory paid editions make this easier ?

Just to be clear my requirements are:

From the outside it must appear as one build i.e. one Hudson job.

For a developers desktop build it is okay to be a multi step process

We need to be able to specify a particular build to deploy in dev #2 and for releasing.

We would like to continue the use the promoted builds plugin to visualize good builds

Wednesday, 26 October 2011

This is a review of Apache Maven 3 Cookbook written by "Srirangan". I got a free copy from Packt publishing for the purpose of the review. I have been using Maven for some years now and this book is a introductory book, so it was clear from the beginning that I am not in the target audience.

The style of splitting the book into 50 recipes makes for a good format which is easy to read and breaks the book into small achievements for the reader.

Instead of focusing solely on how Maven is configured, the author tries to tie some of the subjects to software development practices e.g. covering Nexus and Hudson while explaining team collaboration. It serves the book well to put Maven into a development perspective, but it doesn't always fit with the recipe format. For instance in the Nexus case from above the "How it Work" section becomes more a "why it is good" section.

I like the fact that the book covers a wide range of different project types and topics. Many times when you read tutorials or other documentation only the simplest project types are covered leaving the reader to add plugins as needed. This book covers many project types and framework and some non-Java areas. It also covers things like setting up Nexus,Hudson, various IDEs. It even has a single chapter on plugin development.

In the first chapter the level of information in each recipe is appropriate, but as the chapters get more complex the level of information does not. This results in many of the recipes being too simplistic. A prime example the is set-up of remote repositories, it describes in great many screen shots how to install tomcat 7 and deploy nexus, but only has a single line of information on how to set-up the remote repository, plus only mentions (incorrectly) changing the settings.xml and not the required changes to the project object model. So it fails to help the reader define and use the remote repository possible leaving the reader with a broken set-up

This simplistic approach has another side effect. In many of the recipes there is clear copy-paste'able examples but very little explanation of why things are the they are. An example would be the first recipe which introduce multi-module projects. In the top level project definition the dependencies are placed in "<dependencyManagment>" section instead of the normal "<dependencies>" section without an explanation of why.

Conclusion:
While I like the style and long list of topics covered in this book, I think the decision not the explain the details of why things work like they do e.g. <dependencies> vs <dependencyManagment> or how repositories works, does the book a big disservice.

I would not recommend learning Maven from this book alone, since I think explaining the "why" is an essential part of learning a new tool. If you want to learn Maven use the free sonatype book Maven: The Complete Reference and buy a copy of this book if you like a quick introduction to the various project types and plugins.

Thursday, 13 October 2011

The tool itself is pretty good and makes it very easy to test that our soap based web-services are working as intended. The fact that is actually provides Maven and Junit integration out of the box is even better and fits very nicely with our CI environment.

There is however a few things that are not obvious when using the plugin.

The documentation page is really old, e.g. it refers to an old (2.5.1) version of the plugin. The trick here is that you should in general use the same version, as the desktop version you are running. In my case that is 4.0.0

It isn't documented on the page but there here is both a "maven-soapui-plugin" and "maven-soapui-pro-plugin" version of the plugin. In order to fully use project created using the pro version you need the pro version of the plugin

The version 4.0.0 of the plugin has a misconfiguration so you will need to manually add some dependencies to the plugin. The version I got working looks like this.