Pro Tips for Continuous Integration with Android and Jenkins

This should be easy, right?

Android applications are built with Java. Gradle is (finally) a mature and well-known build framework. Jenkins is a Java-first Continuous Integration platform and already has fantastic support for Gradle (as well as Maven and Ant) out of the box. So what’s the problem?

Getting an Android build set up in Jenkins is easy. Especially on a development box. If you’re part of a small, flexible team, own your company’s CI servers, or run the IT department, then there probably isn’t a problem. If you’re like us, though, you don’t always have the freedom to update your CI platform at-will. We don’t always have the rights to install Jenkins plugins or manage an Android SDK installation directly on corporate infrastructure. We don’t always own the keys to sign our applications or follow the same processes internally as we do with our big clients. That’s just the nature of big enterprise platform services—changes to infrastructure need a paper trail, and ownership of build processes, security and development are often divided amongst different verticals.

Disclaimer up front: this article is not another tutorial for on-boarding your Android application to Jenkins CI. There’s already a lot of good information out there on Android and Jenkins, with solid tutorials using Ant, Maven, and Gradle. There’s even more on Android and Travis CI, Hudson, CircleCI—even Bamboo—if Jenkins isn’t your CI platform of choice.

This article is also not another preachy persuasive essay trying to convince you to use Continuous Integration. As long as you weren’t tricked into clicking here, you’re probably already on-board with CI anyway. And if you’re not, lots of others have provided plenty of reasons you should be in a much more eloquent manner than I’d be able to.

As consultants and enterprise mobile developers, we need tools that allow us to integrate Android projects into any organization’s continuous delivery platform. What follows are some common concerns we’ve encountered time and time again on both internal projects and with our clients, along with recipes to address them.

My Jenkins don’t speak Internet

At DevNexus last week, I attended a Gradle continuous delivery presentation by a developer evangelist at Gradleware. The presentation was solid, but there was a slide in there that grated on my nerves a bit. The delivery went something like this (paraphrased for brevity and bad memory):

Gradle Evangelist: Gradle is great!

Matt: <Nods head>

Gradle Evangelist: Continuous Integration is great!

Matt: <Nods head>

Gradle Evangelist: Don’t worry about the Gradle version. We’ve got this thing called the Gradle wrapper that automatically downloads the right Gradle version from the internet!

Matt: <Roars, flips table, tackles presenter>

So… that’s not exactly how it went, though I did skip the head nod at that point. Newsflash to our amazing tool developers: our internal infrastructure doesn’t always have access to the outside internet! This is just as big of a problem with the Android SDK and SDK Manager as it is with Gradle. And in all fairness to the Gradleware rep, the Gradle Wrapper isn’t limited to pulling distributions from the internet, as we’ll see in one of the tips below.

Tip 1A: Packaging and Installing Gradle in Jenkins

Version A of this is relatively simple. Simply download the Gradle binary distribution(s) that you want to support from https://gradle.org/downloads/ and install them on your Jenkins server. Easy, right?

Alas, not really. In our experience, this introduces yet another problem: our ‘server’ is actually lots of Jenkins slaves and are controlled like a production environment. We need a repeatable process that we can ‘deploy’ to each of the slaves. Probably a no-brainer to some of you, but we found it was easy to wrap this up into an RPM, DEB, CAB, or MSI file (depending on the platform of your Jenkins servers). This makes it much easier to manage each version of Gradle as a separate module and pilot or moonlight various versions as needed.

Tip 1B: Using the Gradle Wrapper in Jenkins

As an alternative to installing Gradle distributions directly to your Jenkins cluster, you actually can use the Gradle Wrapper. This requires that you make the Gradle distributions available somewhere on your enterprise network and you update the wrapper configuration in each of your projects.

When you use the Gradle Wrapper in your project, the whole thing gets encapsulated in a gradle folder with a really nice little configuration file called gradle-wrapper.properties.

Crack that file open and voila! you’ve got a configurable endpoint for where your Gradle distribution is pulled from. If you’re in an enterprise environment where Jenkins can’t talk to gradle.org, do the following:

Get your Gradle distributions added to a local binary repo or server. Example – add them to your on-premises Nexus or Artifactory instance.

Update your gradle-wrapper.properties with the following:

distributionUrl=https\://path/to/gradle/zip

Tip 2: Packaging and Installing the Android SDK

Hope you enjoyed the warm-up. The Android SDK is a different beast – mostly because it’s not just a single package, and in Google fashion, the moving parts aren’t really documented all that well. Also expectedly, Google assumes you’ll be using the SDK Manager tool wherever you need it installed. Damn internet!

Each of the following is a different ‘component’ that makes up the Android SDK toolset – and each is required for successful builds:

Android SDK Tools – This is the toolset that includes things like the SDK Manager used, in an ideal world, to download
all other components. This is
versioned, but you really only have one version ever installed at a time unless you manage multiple ANDROID_HOME’s. This install also includes certain tools that are required to build and package your applications.

Android SDK(s) – These are the actual SDK versions that map to Android versions. You have multiple versions of these installed on a given system at the same time to support targeting multiple Android versions. Absolutely necessary to compile your Android applications. The SDK version is specified as part of your application’s build scripts.

Android Build Tools – Specific build binaries used to compile your source to dex, process your application resources, and package your apk. You can have multiple versions of the build tools installed on a given system at the same time. The build tools version is specified as part of your application’s build scripts.

The big question for us was where to get these as standalone packages. On a development machine, you’d just open up the SDK Manager, select the components you need, and the tool would go out and download everything for you automatically. To package and deploy them ourselves, we had to track down the back-end that SDK Manager was talking to.

Looks like this is incremented every now and then (tied to new SDK Tools releases?), so you may need to track down a later version if you don’t see your component in the manifest. Hint: try replacing “-10.xml” with “-11.xml”. In practice, this manifest can be used to find the download locations for SDK Tools, SDK versions, and Build Tools versions.

The idea here is that you put the contents of each of these into a deployable package, or set of packages, and deploy to your Jenkins grid as needed. Similar to the Gradle versions above, packaging each SDK version into a separate RPM, DEB, CAB, or MSI file, for example, allows you to pilot or moonlight new/old SDK versions at will without impacting working builds. Of course that’s sometimes tough in practice due to dependencies on the SDK tools version, but it’s still probably better than a single monolithic package. To summarize – downloading and packaging manually allows you to manage the SDK on your Jenkins servers completely offline, without use of SDK Manager, and allows you to install and uninstall in an automated way.

Tip 3: Backwards Compatibility and ZipAlign

On a recent engagement, we went through the packaging process above to upgrade to the latest and greatest Android SDK tools and build tools. We wanted to give developers the opportunity to use Android Studio 1.1 and the Android Gradle Plugin 1.x, which in turn required an update of Build Tools to version 21.1.2. As a part of that, we also upgraded the overall Android SDK package to the latest and greatest – version 24.0.2. Older versions of Build Tools and the SDKs remained on the server so that our existing builds targeting previous versions would still build correctly. Lo and behold, things yet again were not so easy!

On what should have been an easy validation of older builds (using Build Tools 19.0.0), we started seeing the following error:

As it turns out, Google moved the zipalign binary out of the SDK Tools package and into build-tools starting with SDK Tools r23+. Older versions of Build Tools (e.g. the 19.0.0 we were using for the majority of our builds) were still trying to reference the binary in the SDK/tools folder, though. Nice! Here’s the issue logged with Google, which has somewhat hilariously been marked as “Closed; Working as Expected”: https://code.google.com/p/android/issues/detail?id=72611

If you’re facing this scenario, you need to:

Update every single one of your existing apps to use the new build tools. Easy if you’re a flexible, small development team. No thanks if this is an enterprise platform supporting lots and lots of applications and lots and lots of different development teams.

– or –

Get the zipalign binary added back into the main SDK tools folder.

On this particular engagement, we were packaging the various SDK components into deployable RPMs, and this was a pretty easy fix based on our packaging model. Simply add an instruction to your install script – whether spec file or MSI or something else – that copies the binary from one of the new build tools revisions to the base SDK folder’s tools directory. Here’s the ‘fix’ we put into our spec file:

No, you can’t share the signing key with the rest of the team.

Your signing key can be used as a trust identifier in application code, particularly amongst separate integrated apps, so its security is a big deal. I’ve heard of lots of development shops where the team’s signing key is committed to version control with passwords embedded in build scripts or where the key lives unprotected on the local machine of the team’s development lead. We’ve found that securing the signing key and configuration on build servers provides a good balance between maintainability and security, with little impact to your development team.

Tip 4: Inject your Signing Configuration, and Make it Optional

Our goal is to:

Keep our signing configurations secure

Make sure the signing configuration is used for each release build

Reduce any impact on the development team from a process standpoint

Start by externalizing the signing configuration to an optional, global file with the following little snippet in the build.gradle script:

This basically checks to see if a signing configuration has been configured in gradle.properties or has been passed in to the build process from the command line (using a “-Pbuild_signingConfig=<location>”flag).

This does two things for us:

Allows us to use a default, non-prod signing config (or none at all) in our development environment and on developer machines. This can be defined in gradle.properties for development purposes and overwritten with a production key in our Jenkins build configuration.

Allows us to externalize our entire signing configuration to a local, secure file on our Jenkins servers, outside of version control.

With this now a part of our common build scripts, we stage a signing key and signing configuration (or set of signing configurations) out on the build box as follows:

Not only does this live in a secure place on the server, outside of user control and unwanted access, but we can also further lock down the keystore and alias passwords by injecting them into the build scripts with environment variables. We can then use the Jenkins Mask Passwords [https://wiki.jenkins-ci.org/display/JENKINS/Mask+Passwords+Plugin] plugin to mask these values and feed them into each job directly from the Jenkins job configuration.

An alternative to this approach is to pull application signing out of your Gradle build scripts entirely and call jarsigner separately, or externalize key signing completely to a separate platform.

Where the <bleep> did this APK come from?

Most of our large clients maintain a sizable portfolio of different internal applications. Each of these applications is owned by a different team with a different set of developers. However, there are almost always central support, app distribution, and release management teams, and these teams need some way of tracking where a given production artifact may have come from.

Tip 5: Embed build info directly in your release APKs

On our team, we’ve been able to take advantage of the Android Manifest’s meta-data XML tag to embed important, but not sensitive, build information. This data can be pulled directly from an APK out in the wild and has the added bonus of being presentable directly within your application (i.e. in an app splash screen or support view). Jenkins conveniently adds a lot of the data that we care about directly to the environment as variables that we can use within our build scripts. The following snippet is added directly to our shared Gradle build scripts. It checks for the Jenkins-provided environment variables and, if available, adds them directly to the AndroidManifest.xml file(s) before packaging:

Who forgot to increment the versionCode?

It can be challenging to manage the versionCode and versionName in AndroidManifest.xml on large teams, especially when multiple developers are kicking off release builds and handing over build artifacts to QA. Failing to increment the version code after a release build can result in all sorts of headaches when trying to upgrade your application on test devices and a mismatch between versionName and versionCode can make it difficult for the development team to troubleshoot defects.

Tip 6: Externalize your Application Version and Auto-Increment on Release

We’ve dealt with this on our team by pulling the version code and version name out of the AndroidManifest.xml and instead tracking a single version truth in each project’s gradle.properties file:

## Version name will get injected into the AndroidManifest.xml file at build time.
## The version code will be derived.
## Must match MAJOR.MINOR.RELEASE format.
artifact_version=1.0.0

We then update our build.gradle file to reference the versionName directly from this property and to derive a unique versionCode from the value there-in:

Lastly, we’ve added a task to our common build scripts that allows us to update the version property from our Jenkins release builds. We call this in the post release block of our job configuration so that it only gets invoked after a successful build:

Note that you’ll still need to invoke a shell script or use a VCS plugin afterward to commit the updated gradle.properties file back to version control.

Next Steps

This by no means covers an entire build and continuous delivery infrastructure – there’s a lot more material out there re: on-boarding your Android application to Jenkins, adding solid unit testing (hint: start using Android Studio 1.1 to make this easier), and publishing builds to your binary repository of choice. We’re hoping the tips and tricks above help you on your project, though, as these are common concerns that we’ve had to address over and over again on both internal projects and with our clients.

That said, there are at least a few things that I’m hoping to explore next:

Using the Jenkins Android plugin with a headless emulator to execute instrument tests

Connecting Jacoco test coverage in Android with Sonar

Direct deployment from Jenkins through a distribution platform like Airwatch, Crashlytics Beta, etc.

Bundling my common Gradle scripts into a plugin that can be referenced as a build script dependency

Look out for a second round of this article if I can accumulate enough new recipes along the way. And by all means – please jump in with your own suggestions, experience, or feedback in the comments section below.

Matt Mehalso is a software architect, manager, and senior developer with full stack experience across finance, telecommunications and public sector industries. Matt has also lead delivery teams and contributed to all stages of the software development lifecycle, including business analysis and management consulting cases. You can tweet him at @mattmehalso

Hey, this blog post has been written a while ago but I hope you can still help me.
We just migrated from Eclipse to Android Studio and we were using a Jenkins Build Server. The biggest problem now for us on a build server without Internet access are the dependencies.

Lets say you are using ButterKnife in your project by referencing it in the build.gradle file with

compile ‘com.jakewharton:butterknife:6.1.0’
Possible solutions would be to download all dependencies (and losing the best advantage gradle has over ant) or to mirror jcenter locally.

shubh

Hi, Thanks for sharing. I’m trying to setup Bamboo build system for Gradle based Android Project. But found there is not any directly support for Gradle there. Do you have any suggestion how to integrate it.