Sitecore, Digital Marketing, Work, Life and so on…

Sitecore Continuous Integration Continued

Here’s an update on the post I wrote a year ago on our Continuous Integration project here at Pentia.
To put it short: it’s turned out to be one of the most successful internal projects in our company. In the last year we’ve added 20+ new projects into the system, and we are in the process of moving our old projects from SourceSafe to SVN, and creating build files for them.
Our setup is not a conventional build setup with specifically designed build scripts for a major project, it’s more of a build framework for the many Sitecore project passing through Pentia. We’ve designed our setup around our naming and location best practises, so in order to add a project to the build system, we only need to create a single file with about 10 lines of XML specifically for the project.

Our entire setup consist of:

1: nAnt build framework
This is a set of approx. 30 nAnt scripts which is shared across all our projects. The scripts handles everything from SVN, MSBuild, FxCop, deployment, zipped releases, automatic versioning and much more. The files are versioned so that changes and new features will not influence the older projects and break the builds. In the last year we have released 5 versions of the nAnt scripts.

2: Subversion version control
In my approx. 15 years of programming I’ve only worked with Visual SourceSafe as version control (and as those of you who knows the product knows, is hasn’t changed for 15 years either :-), so the switch to SVN has been a daunting task for me. After a year, my only recommendation for those of you who consider it is to go for it! Subversion is easy to install and maintain (I recommend VisualSVN Server), easy to use (TortoiseSVN) and the integration into Visual Studio (VisualSVN) is perfect. Naturally the integration with nAnt is nice too as. E.g. it made it possible for us to automatically version our projects when releasing, by extracting the latest revision from SVN.

3: CruiseControl.NET
The hardest part to integrate into the company has been the CruiseControl.NET server. It’s not difficult to explain its purpose to the developers (to continuously check the projects on commits in SVN and build nightly releases), but the benefits of the other parts of the setup, for the individual developer has just been so much more evident. But still, its running, checking our builds and keeping our internal testing environment updated, by releasing each night.

4: Module library
The module library is a really neat feature of our build system. In short, it allows us to only maintain our own codebase in a project in version control and still allows developers to get quickly up and running, not making it necessary to install Sitecore version and thirdparty tools on their local machines. In the last year we’ve added 12 versions of Sitecore, 16 Sitecore modules (with 3+ versions each), 10+ third party modules like PDF, FTP and unit testing, and much more. The library allows us not only to add our old projects to the build system, but the cool thing is that when e.g. a new update to Sitecore is released, all our projects can be upgraded by changing a single line in the project build files.

5: Configuration merging
Both Sitecore and Microsoft has acknowledged the problem of mainting large configuration files. And the both have features which allows config changes to be merged from external files. The only thing they do not make easy, is to have different configurations for the different environments. For this we’ve developed a nAnt extension which allows the developers to create configuration merge files (configmerge files) which can be varied according to the environment we are building to. This has made a huge difference as opposed to juggling different configuration files for production/test/training/development environments for frontend and backend servers (6-7 web.configs!).

6: GUI tools.
We have lazy developers in Pentia :-) Joke aside, we quickly acknowledged that in order to successfully get the system to the developers, we had to provide them with better tools than the nAnt command-line and CruiseControl xml files. So we created a nice GUI for our nAnt scripts and for editing our CruiseControl server.

As you can hear, I am still pretty excited by the accomplishments we’ve made in the last year – and we have a lot of cool features waiting. Our next release will incorporate generation of SQL scripts for mounting Sitecore databases and setting up database access and automatic IIS setup. We also want to automate our releases even more, by generating batch files for folder security and automated FTP upload to the production servers.

I’ll still be happy to hear from anyone with experience or ideas – or you can give me or Pentia a shout if you want to hear more about our setup.

Like this:

Related

Post navigation

14 thoughts on “Sitecore Continuous Integration Continued”

Lars Nielsensays:

This is a very cool setup, – not very unlike the same setup we are using in Sitecore (I think somebody is working on a best practice for workgroups, that we might release). Anyway, this entire automated process has greatly improved our productivity.

While we do not have the same multitude of entire different projects as Pentia does, – we still work with several versions of Sitecore, and several modules.

The CruiseControl.NET tool has proved especially useful in the Sitecore case, as the nightly build results in a new version our testers continue with.
Suggestion (you might have thought of it already): Make sure to compile your unit tests belonging to the project as well. Then, after the nightly build, execute the unit test.

Thanks for an inspiring post about automated Sitecore development setup. Only thing I’m wondering; how do you deal with database changes, e.g. when a developer implements a feature which require new or changed Sitecore items? In Sitecore 6, item serialization is your friend – but what about projects implemented on former versions of Sitecore?

At the moment all developers work on a local website, connecting to shared databases. When altering content, templates or similar we use a shared website on a development server to avoid data cache issues.

It would be great if you guys were able to go into more technical detail about the module aspect. We’ve been developing on SiteCore 6.2 for aprox 6 months and the release management aspect is the bane of our existence. We’re about to kick off a new project and this time we’ve kept the items under source control to the bare minimum (i.e. just a simple ASP.NET Web Project with App_config, Layouts, XSL folders and then the web.config in the root. But this does rely on the developers installing an instance on their own machine before using SCM to “overlay it” with anything developed by ourselves. In the same way, any releases to other servers need the installation step first. In both cases updating the version would have to involve manual intervention. How you’ve managed to integrate a versioned install of SiteCore into your build process sounds like an extremely cool thing.

1. Yes, it’s important to distinguish between project files and Sitecore/Thrid-party files. Only project files or files changed by the project should be in Source control.
2. We distinguish our files from Sitecores by looking in the source control.
3. If we want to ship our code to our client, we zip a clean check-out from source control, not our workspace. This means no temporary files etc.
4. We want our development environment to be as fast as possible, i.e. build/test. When doing actual testing, we build/deploy/test to a QA environment.

We have build script that are executed to create an initial development environment on a machine (copies the Sitecore and thirdparty files) and to deploy to QA/pre-prod/prod (copies the webroot without code etc.). No custom scripts are needed while developing – which we feel hinder our development time.

great post, question though..In regards to subversion what files do you version control from the Sitecore “website” folder? Do you have a separate projects for both Sitecore system files and editable files (xslt, aspx etc). Given that each one of my developers have a local installation, I want to be able for them to get updates and make commits to a certain set files but it seems like I need to start them out with a base set of sitecore system files and then go get the remote repo (xslts, aspx) and pull it into their sitecore environment. Must be missing something, is this how your team uses subversion?

Exactly!
When working with your build and configuration management system – including version control – you have to separate the task of setting up a new (or fresh) development environment and the daily work of producting new functionality and code. Your version control system should primarily assist you in getting the latest updates from the rest of the team and integrate it into the local development environment. Furthermore it is vital for maintenance and upgradability that you separate the system files (Sitecore and other third party standard modules) from your solution files. Therefore the task of setting up a new development environment should include a number or steps (manual or automated) to retrieve and set up the system files and finally merge in the custom changes. The keyword here is merge: The more standard or system specific you can leave out of your solution files (i.e. version control) the better. This makes your solution slimmer ane more simple and makes it easier to identify standard and custom functionality – thereby reducing he cost of maintenance and upgrades.

Thanks eldblom, I took your advice and found that my editable set of files (800) compared to system files (24k) is minimal. Here’s our simple process:
* The 800 in the repo also consist of the VS2010 project files that allow for the dev to open the project upon “Get Solution…”. The 25k in zip are extracted from a remote drive that everyone has access to. Every devloper has their own DB so we make it a point to verify connectstrings.config, form.config and web.config as these files are not under version control.
* Local IIS is mirrored to QA IIS and is relatively painless as we add https (local cert in IIS) and give rights to all dirs.
* Before development against and issue/task we ask that developers make sure their on the trunk of the project in VS and create branch, saving branch to branches folder in the sandbox/Sitecore project.
* When developers are done developing locally on their issue branch, they contact an Integration Manager so the merge to trunk (QA) is completed. Once the merge is completed a simple refresh to the QA site will show the changes.

We’d like to deploy CCNET CI but think that it would only be appropriate from QA to PROD.

Question:
How do you handle a build from QA to PROD?
Do you have scripts that copy files and create packages to deploy?
Do you use CI in conjunction with all your scripts to deploy a release to PROD?

Yes, we use our build environment to create all deploys to QA, test and release, although we do not automate deploys of Sitecore items or packages. We do not run with local developer databases but run our local iis against and update on a central database – we do have automated database scripts to quickly backup and restore databases from one env. to another. Keep in mind that we support many different Sitecore solutions and therefore the primary task for our development environment is to get us quickly up and running on a project and to eliminate manual mistakes.