Using GreenHopper to Automate Bamboo & Bitbucket (“What the whaaaa?…”)

There are several crude expressions to describe the heat & humidity in Dallas where the Agile2012 conference recently took place –most of which involve the words “satan” and “balls”. Which-ever is your personal favorite, just mentally insert that here. Now you get the picture. So I was pretty content to be inside at the Atlassian booth demoing the heck out of JIRA, GreenHopper and Bamboo. It’s great to hear what customers are interested in, and it’s been no surprise to find that lots of them are using Git and Mercurial. This gave me an excuseentree to seriously geek out on a few victimswilling participants and show off a little proof-of-concept for automating yet another developer task: the creation and closure of feature branches. It got such a great response (“Ooh, shiny!!”) from the folks at Agile2012 that I promised to write a blog about how to do it.

“I Don’t Get It. And Why Do I Even Care?”

The basic idea is that dragging a JIRA issue to a new column on your GreenHopper board triggers Bamboo to execute a branch manipulation script. It’s a small piece of a larger development workflow which is gaining popularity and looks something like this:

A JIRA issue is created and/or pulled into an iteration.

A developer picks up the issue and moves it to “In Progress”, at which point our snazzy workflow post-function kicks in. A story branch for that issue is automatically created and pushed up to the repo.

The developer starts work on the branch. This doesn’t require pulling code down; you just switch branches in your IDE or whatever GUI-based tool you use to navigate your repo (SourceTree, anyone?). Or type ‘git checkout ${branch_name}’ from the command line.

Bamboo is continuously building and testing that branch with every push up to the repo. Remember that in this workflow pushing is safe because your work is isolated on a branch and won’t negatively impact your teammates. Ideally, the developer has enabled the auto-merge option in Bamboo so that it is merging in code from master each time it builds the branch, which provides kind of a sanity-check and helps flush out merge or integration issues early.

When implementation is complete, the developer moves the issue to “Resolved”, at which point our snazzy workflow post-function kicks in again. It triggers Bamboo to merge the branch to master and closes the branch.

Since master now has new code on it from the merge, Bamboo detects that and builds master.

Master is now contains the new user story / task / bug fix, and is ready to be deployed at any time.

Frankly, you wouldn’t give a rip about this unless your team has adopted (or is thinking about) the practice of creating a development branch for every issue. The value here is three-fold. First, the branch creation, final merge to master, and branch deletion are automated. It’s not like this is going to shave hours and hours off the time it takes to implement an issue. But it does take a couple administrative tasks off developers’ plates. It also enforces the practice of story branching, if that’s what the team has decided to do. And enforces it in a graceful way that doesn’t involve nag-mails, water-cooler grumbling or team meetings. Third, this automates a naming convention for those branches, which makes it that much easier on the eyes/brain when viewing all the branches in the team’s repo.

Now that you care… on to the good stuff!

Pre-Requisites

First, credit where credit is due: this was the brain-child of my GreenHopper marketing counterpart, Mr. Nick Muldoon. He and I collaborated on the implementation, as well. Thanks, Nick! Second, some assumptions: we used a Git repo for version control, Bitbucket to host the repo, and an Ubuntu server to host behind-the-firewall instances of Bamboo & JIRA/GH. It’s possible to use other flavors of of VCS, repository management and server OS using this post as a guide, but I can’t speak to any of the particulars or “gotchas” lurking in other technology combos.

At a high level, you’ve got 3 components that need to be able to talk to each other: your JIRA/GH instance, your Bamboo instance, and your repository. Using a post-function in our workflow transitions, JIRA calls out to Bamboo via the REST API and triggers a build containing a simple shell script that creates and commits (or merges and deletes) feature branches. Naturally, Bamboo can checkout code from your repo without any kind of special set up. But in order for it to actually manipulate the repo programatically (ie, do pushes without needing to enter login credentials) we have to set up SSH keys.

Start by going into your Bamboo host and figuring out which user account owns the Bamboo process. In our case it was the ‘ubuntu’ user. Next, find or generate a private & public key pair, and place it on the Bamboo host in ${USER_HOME}/.ssh. Refer to the docs for your server type for any naming conventions around key pairs. Ubuntu needs them to be named ‘id_rsa’ and ‘id_rsa.pub’.

Then, in Bitbucket, we added the public key (id_rsa.pub) to my account: Account > SSH Keys > Add New Key You might be able to skip this step if your repo is publicly accessible. Mine requires user login, hence the SSH key. If you’re following along at home (or at work) and are able to access a public repo sans key, post to comments and let me know. Crowd-sourcing for the win, baby.

The other piece to put in place is the HTTP Request Workflow Function plugin for JIRA, available on the Atlassian Marketplace or via JIRA’s plugin manager. What it lacks in slick naming, it makes up for (and then some) in pure, no-BS functionality. And it’s free. (Thanks, hasCode.com!!) The plugin allows you to make REST calls via HTTP as part of a workflow transition; in our case we add it as a custom post-function.

Feature Branch Creation for Lazy People (like me!)

Before we can use the HTTP workflow post-function to trigger building a Plan in Bamboo, we must set up said Plan. Actually, make that PlanS –one Plan to create new branches, one Plan to close them. One Plan to merge them all, and in the darkness bind th-… wait, sorry. Got a little carried away. *ahem* The branch creation Plan first checks out my repo so we have it in Bamboo’s current working directory. It then executes a series of Git commands, configured as an in-line script. What you see in the screenshot is the complete script, btw. Just 4 little commands. The “-u” option on the last line is necessary to push new branches that don’t have any changes vs. master yet.

Since you’re an astute reader, you’ve noticed the variables used here. I could have set them up as Plan-level variables in Bamboo, but I didn’t have to. Both their names and their values are brought in as parameters on the REST call. (I know, I know! I’m getting to that part. Keep your shirt on!) You’ll also notice that the JIRA issue key is included in the branch’s name. This is optional, but it’s done to take advantage of the wicked issue-to-branch linking feature that just came out in Bamboo 4.2. For a deep-dive on how nerds at Atlassian are using that, check out this blog by Bamboo developer, Stefan.

So! Back to JIRA where we can finally implement that post-function REST call. You need to be part of the jira-administrators group to edit workflows, so make friends with your JIRA sys-admin if you don’t have the right privileges yet. Go to Administration > Issues > Workflows and select “Edit” for your target workflow. This will create a draft to work with. We want to kick off our branch creation Plan whenever an issue moves from Open to In Progress, so in our workflow we clicked on the “In Progress” transition to edit it.

Inside the transition configs, select the Post Functions tab and click “Add”. With the plugin installed, you’ll see an option called “HTTP Request Function” –select it. Here’s where I start to feel like a kid at recess Lots of issue-related variables available to play with, and a whole set of Bamboo REST APIs to call! (The docs are here, btw.)

To trigger a build, you make a POST call to Bamboo’s queue API, loaded with authentication credentials and your choice of other parameters. We found it easiest to create a “GreenHopper” user in our Bamboo instance so we didn’t have to expose a human’s password in the call. That’s optional, but either way, make sure the user in your param string has permission to build the Plan. Our POST call ended up looking like this:

“EVAL-ANW” is our target Plan’s key. Notice that the variables shown above in our Bamboo script task are declared and assigned right there in the POST call using the presets that ship with the plugin –“&bamboo.variable.issueKey=%ISSUE_KEY%”, etc With the post-function in place, we can publish the changes to the workflow and go try it out. Drag an issue from Open to In Progress using our lovely GreenHopper board, see the branch creation script fire up in Bamboo, and viola! a new branch just especially for that issue appears in our Bitbucket repo. Pretty slick, right??

At this point, Bamboo will automatically discover the new branch and copy the master code line’s CI scheme and apply it to the new branch. No special magic needed; that’s just how Bamboo rolls. While development is ongoing, Bamboo will build and test the branch, and can even merge in code from another branch in the repo with every build. And again: using the JIRA issue’s key in our branch name lets us create a link so people can see the latest build status of the branch right inside JIRA. (Super useful for stakeholders and management types who don’t spend time in Bamboo, but do check JIRA to see how their issues are progressing.)

What Gets Opened, Must Get Closed

In similar fashion, we want to merge the story branch to master and close it when the JIRA issue goes from In Review to Resolved. So we set up a corresponding Plan in Bamboo to handle that, and call it with a post-function in the “Resolved” transition. You’ve got the idea now, so we’ll skip straight to the screenshots: In Bamboo, our script task attempts merging to master before deleting the branch. If a merge conflict arrises, the whole works stops and Bamboo’s notification system kicks in to alert the project lead of the failure. (How cool would it be if Bamboo notifications could accept variables? Then you could send the alert directly to the issue assignee! Perhaps soon…)

The workflow transition is edited in the same way, and again uses a POST call to add a build of the merge & close Plan to Bamboo’s queue. The call looks identical, except for the Plan key (“EVAL-RESOLVE”)

How Will You Use It??

Is this method a bit hacky? Yeah. It’s something that we’d like to get native in-product support for eventually. If your team implements something like what I’ve described, please please PLEASE come back and post a comment here. We’d love to hear all the use cases folks “in the wild” are finding for this, as well as just gauge general interest and enthusiasm. This same mechanic could be used to build and deploy code to test environments every time an issue is resolved, or deployed to production when issues are moved to a “Shipped” state. Or create a code review in Crucible when an issue is dragged from In Progress to Needs Review. Loads of possibilities here!

If you haven’t tried Bamboo yet, this might be just the excuse you need! Like all Atlassian products, it’s yours to try free for 30 days. So whatchya waitin’ for??

About Sarah Goff-Dupont

I've been working in and around software teams since before blogging was a "thing". When not writing about all things agile, automated, and/or Atlassian, I can be found reading contemporary fiction, smashing it out & keeping it real at CrossFit, or rolling around on the floor with my kids.
Find me on Twitter! @DevToolSuperFan

Because the merge is part of a post-function, the JIRA issue’s transition to it’s new workflow state will not be affected, tho Bamboo will register that as a build failure and notify you appropriately. Resolving the merge conflict and closing the feature branch then reverts to a hands-on task that occurs outside of JIRA.

By Sarah Goff-Dupont on August 27, 2012

Got it. Can Bamboo change the status of a JIRA issue when a build fails?

We have been do something very similar. The main difference is we use Pull Requests as a code review step and have used smart commits in our Pull Request acceptance message to transition our JIRA issues automatically.

A typical PR message with smarts commits would be “#closes [ATLAN-43] some longer description”.

The next version of DVCS connector includes support for smart commits but we are using a simple script to provide smart commit functionality until its released.

Issues are scheduled via the classic planning board for a given release.
Once QA is ready to test they go to a Bamboo plan and run it manually.
This plan is run customized as one of the parameters is the JIRA project version.
This is passed to a custom script which pulls in all issues for the project and version which are closed or resolved.
Any issue branches found are merged into a released branch.
The release branch is then pushed to the appropriate environment.

I would love to use the new rapid board but there are a number of issues that make the classic boards more functional (lack of configuring the card/views in either planning or work mode).

By beeplogic on August 27, 2012

Note that instead of adding Bamboo’s SSH key to your account, it’s better to add it as a Deploy Key on the repo in question. This limits the access of Bamboo to one repo (instead of everything under your account) and also restricts access to read-only.

I agree but it’s a very interesting proof of concept and might be an indication of built in features that could be on the way.

By Paul on August 31, 2012

Ok, this one was a KILLER. Sounds great but make sure you know your server/agent ssh client and its eccentricities *very* well (if Windows)

I suggest do *not* rename the default rsa (id_rsa) when you do an ssh-keygen, then add it to your profile on you git remote server. Then, make *sure* you go into a bash session in your local repo and do an explicit:

>ssh-add

then test:
>ssh -vT git@yourgithost

Didn’t work? Make sure you have a .ssh/config file with the yourgithost entry for user and identity file set to your newly generated pub key

Love to see more articles like this demonstrating the integration between tools and how they can be used together. Between this and the upgrade story posted for Bamboo 4.3 I got several ideas on ways to do things that hadn’t occurred to me before. Can’t wait to get a few days to breathe between releases and upgrade Bamboo to 4.3.

By Gretchen Jones on October 30, 2012

Glad to hear you’re stoked on Bamboo 4.3 – I’ll pass that along to the dev team

By Sarah Goff-Dupont on October 30, 2012

On a sidenote, i am trying to integrate workflows with crucible
on of the steps in a workflow is “Request code review” before it can be released so i want it to automaticly trigger a code review in Crucible. Not easy to do to say the least but this article pointed me in the right direction i think. Still working on it though.

I was able to get something similar to this to work using the new Webhooks feature in jira 5.2. We trigger a bamboo build, passing it only the issueKey (no other fields are available to the webhook). Inside the build we query Jira using its REST api and the given issueKey for other information (e.g. assignee) we want to use in the branch name.

We were also able to get the transition to “resolved” to create a pull request in Stash using a similar mechanism (simple webhook to a bamboo build plan that does the api call to stash). Its all a bit kludgy, but its not completely useless. The automatically created pull requests contain a link to the Jira issue in the description and a copy of the issue’s description which save us some manual effort.

Some of it would have been easier if the REST api for jira supported things like

jira.site.com/rest/api/latest/issue/PROJ-200/assignee

to get the assignee rather than having to pull it out of the larger json response given by

jira.site.com/rest/api/latest/issue/PROJ-200?fields=assignee

Also, the Stash rest api for creating pull requests doesn’t seem to be documented so that was a bit of a challenge. It would also have been nice if I could have created the branches in Stash from its REST API.

I’m not exactly sure how I would like to see this type of functionality show up natively in the tools, but maybe this use case will help.

By J. Nate Knight on January 3, 2013

Nice! Creating pull requests automatically is an idea we’ve also been tossing around internally. Not sure if it would end up involving JIRA webhooks, or if it would be a straight Stash/Bamboo (or straight Stash/JIRA) integration. I sent your comment to the JIRA, Stash and Bamboo product owners – very helpful to know your use case!

By Sarah Goff-Dupont on January 4, 2013

Great feedback – your wish for “/assignee” in the JIRA REST API is something we’d like to do more generically, like we did with the issue key, where you could address any field in the JSON webhook payload in the webhook URL (so you could pass assignee, issue key, etc) and not need a REST callback if you really only need a few pieces of information.

For example, you would be able to use “${issue.fields.summary}” to pass the issue summary in the webhook URL. To be clear, this is in the backlog but not in the product today.