We have rubocop and foodcritic configured to run on Travis CI to
automatically check our pull requests, which is awesome. I verify each
build with test-kitchen before I cut a new cookbook release, but this is
getting very time consuming to run manually each time. For the java
cookbook, testing different java versions on various platforms results in a
ton of individual runs. I’d love to set up a system to automatically run
test-kitchen and then give me the red/green light.

We have rubocop and foodcritic configured to run on Travis CI to
automatically check our pull requests, which is awesome. I verify each
build with test-kitchen before I cut a new cookbook release, but this is
getting very time consuming to run manually each time. For the java
cookbook, testing different java versions on various platforms results in a
ton of individual runs. I’d love to set up a system to automatically run
test-kitchen and then give me the red/green light.

We have rubocop and foodcritic configured to run on Travis CI to automatically check our pull requests, which is awesome. I verify each build with test-kitchen before I cut a new cookbook release, but this is getting very time consuming to run manually each time. For the java cookbook, testing different java versions on various platforms results in a ton of individual runs. I’d love to set up a system to automatically run test-kitchen and then give me the red/green light.

I just began using the (newish?) paradigm I saw in the Opscode nginx cookbook to test my cookbooks for a side project via Travis and Digital Ocean. The cookbooks at https://github.com/todonts-cookbooks are linted with rubocop and foodcritic, unit tested with chefspec, and and integration tested on DO via the kitchen-digitalocean provider and serverspec (I don’t think I’ve committed any of the serverspec tests yet). One nice thing you can do with Travis is use the build matrix to run your test-kitchen suites in parallel during the build. The most complete example I have up right now is the nginx cookbook for my project (https://github.com/todonts-cookbooks/todonts-nginx), but you can see the build-matrix stuff in action on the todonts-app cookbook (https://github.com/todonts-cookbooks/todonts-app). Props to the folks at Opscode for writing all the original harnessing.

We have rubocop and foodcritic configured to run on Travis CI to automatically check our pull requests, which is awesome. I verify each build with test-kitchen before I cut a new cookbook release, but this is getting very time consuming to run manually each time. For the java cookbook, testing different java versions on various platforms results in a ton of individual runs. I’d love to set up a system to automatically run test-kitchen and then give me the red/green light.

I just began using the (newish?) paradigm I saw in the Opscode nginx cookbook to test my cookbooks for a side project via Travis and Digital Ocean. The cookbooks at https://github.com/todonts-cookbooks are linted with rubocop and foodcritic, unit tested with chefspec, and and integration tested on DO via the kitchen-digitalocean provider and serverspec (I don’t think I’ve committed any of the serverspec tests yet). One nice thing you can do with Travis is use the build matrix to run your test-kitchen suites in parallel during the build. The most complete example I have up right now is the nginx cookbook for my project (https://github.com/todonts-cookbooks/todonts-nginx), but you can see the build-matrix stuff in action on the todonts-app cookbook (https://github.com/todonts-cookbooks/todonts-app). Props to the folks at Opscode for writing all the original harnessing.

We have rubocop and foodcritic configured to run on Travis CI to automatically check our pull requests, which is awesome. I verify each build with test-kitchen before I cut a new cookbook release, but this is getting very time consuming to run manually each time. For the java cookbook, testing different java versions on various platforms results in a ton of individual runs. I’d love to set up a system to automatically run test-kitchen and then give me the red/green light.

I’ve been running a few of our cookbooks through Jenkins with Kitchen
and Docker, since first of January. I had to use either Docker, or LXC,
because my Jenkins build host was already a VMWare VM and getting nested
virtualization working with something like Vagrant/Virtualbox wasn’t all
that straightforward.

I broke down the build jobs up into 3 stages, quick tests, integration
tests, and deploy.

Stage 1: Unit and linting tests, with Chefspec, Rubocop, and Foodcritic,
which all takes about 1 minute or less. First stage is triggered by our
Gitlab server, or upstream dependencies.

Stage 2: Use kitchen.ci and kitchen-docker to spin up baseboxes
generated with rinse https://index.docker.io/u/atomicpenguin/centos/
and then run BATS integration tests against those machines. Some of the
cookbooks are EL5 and EL6 which take about 6-8 minutes to test both.
Some just need to be tested with EL6 which take about half the time, 3-4
minutes.

Stage 3: do a berks upload, and berks apply production. If there are
data bags in the cookbook, then I test with chef-zero in Stage 2, and do
a rake databag:upload_all on Stage 3.

If there are downstream dependencies like a role cookbook which depends
on an upstream baseline role cookbook. Then the upstream cookbook
triggers a 3-stage build on downstream cookbooks.

If you want to validate branching behaviour such as in your org-java
cookbook, then a good way to test a lot of corner cases is probably
Chefspec. Kitchen works great doing just a few branching platforms and
automatically integration testing those cases. As you add more
platforms to Kitchen, two things tend to happen. Your integration tests
become more complex, and your build time grows. If your integration
tests for all your combined platforms for a given cookbook don’t fit in
a ten to fifteen minute window, then unit tests with Chefspec is
probably going to provide more value and quicker feedback.

Eric G. Wolfe
Senior Linux Administrator,
IT Infrastructure Systems

The worst sin towards our fellow creatures is not to hate them,
but to be indifferent to them; that’s the essence of inhumanity.
– G.B. Shaw

On 03/27/2014 07:18 PM, Christopher Armstrong wrote:

Ohai Chefs,

We have rubocop and foodcritic configured to run on Travis CI to
automatically check our pull requests, which is awesome. I verify each
build with test-kitchen before I cut a new cookbook release, but this
is getting very time consuming to run manually each time. For the java
cookbook, testing different java versions on various platforms results
in a ton of individual runs. I’d love to set up a system to
automatically run test-kitchen and then give me the red/green light.

I’ve been running a few of our cookbooks through Jenkins with Kitchen
and Docker, since first of January. I had to use either Docker, or LXC,
because my Jenkins build host was already a VMWare VM and getting nested
virtualization working with something like Vagrant/Virtualbox wasn’t all
that straightforward.

I broke down the build jobs up into 3 stages, quick tests, integration
tests, and deploy.

Stage 1: Unit and linting tests, with Chefspec, Rubocop, and Foodcritic,
which all takes about 1 minute or less. First stage is triggered by our
Gitlab server, or upstream dependencies.

Stage 2: Use kitchen.ci and kitchen-docker to spin up baseboxes generated
with rinse https://index.docker.io/u/atomicpenguin/centos/ and then run
BATS integration tests against those machines. Some of the cookbooks are
EL5 and EL6 which take about 6-8 minutes to test both. Some just need to
be tested with EL6 which take about half the time, 3-4 minutes.

Stage 3: do a berks upload, and berks apply production. If there are data
bags in the cookbook, then I test with chef-zero in Stage 2, and do a rake
databag:upload_all on Stage 3.

If there are downstream dependencies like a role cookbook which depends on
an upstream baseline role cookbook. Then the upstream cookbook triggers a
3-stage build on downstream cookbooks.

If you want to validate branching behaviour such as in your org-java
cookbook, then a good way to test a lot of corner cases is probably
Chefspec. Kitchen works great doing just a few branching platforms and
automatically integration testing those cases. As you add more platforms
to Kitchen, two things tend to happen. Your integration tests become more
complex, and your build time grows. If your integration tests for all your
combined platforms for a given cookbook don’t fit in a ten to fifteen
minute window, then unit tests with Chefspec is probably going to provide
more value and quicker feedback.

Eric G. Wolfe
Senior Linux Administrator,
IT Infrastructure Systems

The worst sin towards our fellow creatures is not to hate them,
but to be indifferent to them; that’s the essence of inhumanity.
– G.B. Shaw

On 03/27/2014 07:18 PM, Christopher Armstrong wrote:

Ohai Chefs,

We have rubocop and foodcritic configured to run on Travis CI to
automatically check our pull requests, which is awesome. I verify each
build with test-kitchen before I cut a new cookbook release, but this is
getting very time consuming to run manually each time. For the java
cookbook, testing different java versions on various platforms results in a
ton of individual runs. I’d love to set up a system to automatically run
test-kitchen and then give me the red/green light.

Setting up Travis-CI Test Harness
=================================
These notes were cribbed from the work that Sean O'Meara did for the Yum cookbook.
* Ensure Travis integration is on in GitHub.
* Set up Gemfile, Rakefile like the ones here
* Set up .kitchen.cloud.yml
* Set up a .travis.yml without all the encrypted secrets but with the tasks, matrix, etc. This .travis.yml needs to have env vars like DIGITAL_OCEAN_SSH_KEY_PATH set up unencrypted in the env->global section.
* Install the Travis gem on your workstation. Encrypt the secrets you will use. This is the trickiest part:
** Environment variables shorter than 90 characters are easy; just do ```travis encrypt FOO=bar --add```. For
DigitalOcean you need at least:
```
travis encrypt DIGITAL_OCEAN_CLIENT_ID=whatevs
travis encrypt DIGITAL_OCEAN_ACCESS_TOKEN=whatevs
travis encrypt DIGITAL_OCEAN_SSH_KEY_IDS=123456 # read kitchen-digitalocean's README to find out how to get this
```
** Environment variables longer than 90 characters require some incantation, e.g. for your SSH private key, do

We have rubocop and foodcritic configured to run on Travis CI to automatically check our pull requests, which is awesome. I verify each build with test-kitchen before I cut a new cookbook release, but this is getting very time consuming to run manually each time. For the java cookbook, testing different java versions on various platforms results in a ton of individual runs. I’d love to set up a system to automatically run test-kitchen and then give me the red/green light.

We have rubocop and foodcritic configured to run on Travis CI to automatically check our pull requests, which is awesome. I verify each build with test-kitchen before I cut a new cookbook release, but this is getting very time consuming to run manually each time. For the java cookbook, testing different java versions on various platforms results in a ton of individual runs. I’d love to set up a system to automatically run test-kitchen and then give me the red/green light.