Rapid7 Blog

Metasploit gems from scratch

POST STATS:

SHARE

Introduction

As Metasploit adopts community best practices related to testing and code structure, we've started breaking up our new (and part of our old) work into separate projects that can developed and tested independently. These smaller projects take the form of Ruby gems and Rails::Engines. As we've made more and more gems, we've gotten a repeatable process for making new gems, but we thought our process might we useful for other developers in the community that are looking for a newer guide on how to setup ruby gems from scratch and get the most out of all the free tools available to open source projects.

Skeleton

Github

Your gem will be open source, so you're going to want to host your source control server on one of the free source control services. We use github.com and I'd recommend it because other services are likely to have Github integration before support for the other source control service like Gitorious or Bitbucket.

Account Setup

Repository Creation

Once logged into your Github account, when you go to https://github.com, you'll see two columns, the left hand column is an activity stream while the right column lists repository. Click the big green, " New Repository" button to add a repository.

Repository Naming

What your name your repository is up to you, but you should keep rubygems naming conventions in mind:

- in a gem name should map to :: in namespace names. For example, the gem named metasploit-credential maps to the Metasploit::Credentialmodule. If you don't plan on making a much of related gems, you're unlikely to need to use the - naming scheme.

Other Options

You can leave the rest of the options with their default values and click "Create repository". After the repository is created, Github will display options for upload your code, but we haven't created any code yet, so let's do that.

RVM Installation

When developing Ruby projects on OSX (or Linux) we recommend using rvm(Ruby Version Manager), it's what we use. Go tohttps://rvm.ioand follow the installation instructions.

Local Repository Creation

The point of RVM is you can have a different version of ruby and gems that you depend on for each project you work on. We'll use the ability to setup the tools to create your gem's repository. In the below steps, replace <GEM_NAME> with the name of the repository you created on Github and <GITHUB_USERNAME> with your username on Github.

Create a gemset for your gem: rvm use --create ruby-2.1@<GEM_NAME>

Create the parent directory for your local git repository

cd ~

mkdir git

mkdir <GITHUB_USERNAME>

cd <GITHUB_USERNAME>

Create the gem skeleton using bundle: bundle gem <GEM_NAME>

cd <GEM_NAME>

bundle gem won't create a commit in the git repository, so do that now: git commit -m "bundle gem <GEM_NAME>"

Now, look back at Github in your browser and you'll see the instructions for pushing your local git repository to the remote repository on Github. I've copied the instructions here for easier use:

git remote add origin git@github.com:/.git git push -u origin master

Once the push has finished, refresh your browser and you'll see the gem skeleton in your Github repository.

Tool Integration

With the push to Github, you've now successfully started an open source gem project, but instead of stopping there, let's go on to add a common set of tools to take advantage of the project being open source.

Local Tools

We'll start with setting up some local tools that will help with development on our own machine.

RSpec

Metasploit uses [RSpec](http://rspec.info/) for a unit test framework. RSpec can set itself up on installation, but must also be declared in the gemspec file.

rspec --init will choose some good defaults and include comments on other options you can enable in spec/spec_helper.rb, but for Metasploit we want to make sure we're using the rspec 3 features as well as possible, so we turn on some additional options and our spec/spec_helper.rb looks like this:

$LOAD_PATH.unshift File.expand_path('../../lib', __FILE__)
require 'GEM_NAME'
RSpec.configure do |config|
config.expose_dsl_globally = false
# These two settings work together to allow you to limit a spec run
# to individual examples or groups you care about by tagging them with
# `:focus` metadata. When nothing is tagged with `:focus`, all examples
# get run.
config.filter_run :focus
config.run_all_when_everything_filtered = true
# allow more verbose output when running an individual spec file.
if config.files_to_run.one?
# RSpec filters the backtrace by default so as not to be so noisy.
# This causes the full backtrace to be printed when running a single
# spec file (e.g. to troubleshoot a particular spec failure).
config.full_backtrace = true
end
# Print the 10 slowest examples and example groups at the
# end of the spec run, to help surface which specs are running
# particularly slow.
config.profile_examples = 10
# Run specs in random order to surface order dependencies. If you find an
# order dependency and want to debug it, you can fix the order by providing
# the seed, which is printed after each run.
# --seed 1234
config.order = :random
# Seed global randomization in this process using the `--seed` CLI option.
# Setting this allows you to use `--seed` to deterministically reproduce
# test failures related to randomization by passing the same `--seed` value
# as the one that triggered the failure.
Kernel.srand config.seed
config.expect_with :rspec do |expectations|
# Enable only the newer, non-monkey-patching expect syntax.
expectations.syntax = :expect
end
# rspec-mocks config goes here. You can use an alternate test double
# library (such as bogus or mocha) by changing the `mock_with` option here.
config.mock_with :rspec do |mocks|
# Enable only the newer, non-monkey-patching expect syntax.
# For more details, see:
# - http://teaisaweso.me/blog/2013/05/27/rspecs-new-message-expectation-syntax/
mocks.syntax = :expect
mocks.patch_marshal_to_support_partial_doubles = false
# Prevents you from mocking or stubbing a method that does not exist on
# a real object.
mocks.verify_partial_doubles = true
end
end

To finish the RSpec setup, we want it runnable using rake. We can do this by adding the following to Rakefile:

You can test that your specs by calling rspec, rake spec, or just rake now, but no tests are defined, so rspec isn't testing anything for us. To get some benefit from rspec we'll need to add spec file. Usually, we'd test the gem versioning at this point.

YARD

By default, Ruby comes with a documentation format called RDoc, but for Metasploit, we use YARD because it allows for more structured documentation with the addition of @tags that would be familiar to anyone that's used Doxygen or Javadoc and the ability to write extensions as Metasploit has done with yard-metasploit-erd, which allows use to include ERDs (Entity-Relationship Diagram in our documentation to make working with the database easier.

At Rapid7, we've just released metasploit-yard, so we can share a common rake yard task. The READMEexplains how to set it up:

Add metasploit-yard as a development dependency to <GEM_NAME>.gemspec:

Remote Tools

With RSpec, SimpleCov, and YARD setup, we can run tests and generate documentation locally, but doing everything locally is not enough, we need a second system, that isn't our develop machine to run these steps too to ensure that they aren't passing just because of some fluke of our local machine.

Travis CI

To test remotely, Metasploit uses Travis CI. Travis CI is free for open source software and integrates well with github, as Travis CI is able to automatically test that Pull Requests pass our specs. This is a big help on metasploit-framework where we're up to 3785 pull requests and 9200 builds. Without Travis CI we'd either have had to run those all by hand, go through an error-prone and time-consuming manual process, or roll a Jenkins build pipeline to handle all of that, which would be a significant maintenance overhead.

On Travis CI, we can build against multiple implementations and versions of Ruby. Metasploit tries to build all versions of MRI Ruby and the latest stable versions of JRuby and Rubinius (rbx in the .travis.yml). We need to setup a Travis-CI account and then push a .travis.yml to Github:

Add a Travis CI badge to your README.md so users can quickly see if your project's build is failing when visiting the repository on Github: <span>[![Build Status](</span>[https://travis-ci.org/](https://travis-ci.org/)<span><GITHUB_USER>/<GEM_NAME>.svg?branch=master)](</span>[https://travis-ci.org/](https://travis-ci.org/)<span><GITHUB_USER>/<GEM_NAME>)</span>. When done correctly it will look like this:

Commit your change: git commit -am "Travis CI"

Push your changes: git push

Watch the build on Travis CI

CodeClimate

CodeClimate is a automated code quality analyzer. Metasploit uses it for both our open source and private repositories; however, CodeClimate can't cope with the sure volume of code in metasploit-framework: metasploit-framework itself is a DoS attack on static analysis tools. Like Travis CI, CodeClimate has a badge, which shows the Code Quality using a 4.0 GPA scale. CodeClimate also monitors code coverage, so we'll be adding a coverage report on top of SimpleCov.

Add the CodeClimate badge to the README: <span>[![Code Climate](</span>[https://codeclimate.com/github/](https://codeclimate.com/github/)<span><GITHUB_USER>/<GEM_NAME>.png)](</span>[https://codeclimate.com/github/](https://codeclimate.com/github/)<span><GITHUB_USER>/<GEM_NAME>)</span>

Add the Coveralls badge to your README: <span>[![Coverage Status](</span>[https://img.shields.io/coveralls/](https://img.shields.io/coveralls/)<span><GITHUB_USER>/<GEM_NAME>.svg)](</span>[https://coveralls.io/r/](https://coveralls.io/r/)<span><GITHUB_USER>/<GEM_NAME>)</span>, which will look like this:

Gemnasium

Metasploit's various gems depend on many other gems from the community. We need to keep track of new versions of those gems in case there are vulnerabilities both so we can update our usage of those gems and so we can make Metasploit Framework modules to exploit those vulnerability. To monitor our gems, we use Gemnasium. Gemnasium has sent us alerts about Rails vulnerabilities that have led to Rapid7's rapid vulnerability mitigation response with new updates across our Portfolio multiple times in the past.

Click "Allow Access" in the "Public Project" column so Gemnasium can automatically configure the Github hooks

Click "Authorize application" on Github to authorize the expanded permissions for Gemnasium

Click your username to expand the list of your repositories

Check the repositories to monitor

Click Submit

Add the Gemnasium badge to the README: <span>[![Dependency Status](</span>[https://gemnasium.com/](https://gemnasium.com/)<span><GITHUB_USER>/<GEM_NAME>.svg)](</span>[https://gemnasium.com/](https://gemnasium.com/)<span><GITHUB_USER>/<GEM_NAME>)</span>

git commit -am "Gemnasium"

git push

Rubygems

Although you haven't released your gem yet, once you do, you'll want users that visit your github repository to be able to find the latest released gem, which you can do with a Rubygems version badge

Pull Review

PullReview is another static analysis tool like CodeClimate. Metasploit uses both because PullReview has a more detailed reviews and better handling of branches than CodeClimate, but CodeClimate is cheaper for our team for private repositories. Just like CodeClimate, PullReview can't handle analysing metasploit-framework, but instead of lagging behind and reporting odd or out-of-date analysis, PullReview just won't return an analysis. Metasploit has had discussion with PullReview and they were nice enough to send us an offline analysis.

Add the badge to your README: <span>[![PullReview stats](</span>[https://www.pullreview.com/github/](https://www.pullreview.com/github/)<span><GITHUB_USER>/<GEM_NAME>/badges/master.svg?)](</span>[https://www.pullreview.com/github/](https://www.pullreview.com/github/)<span><GITHUB_USER>/<GEM_NAME>/reviews/master)</span>, which should look like this: , which is a bit cooler than the CodeClimate badge.

Conclusion

So, that's everything. Don't be intimidated by the length of this article: it takes about a day or less for me to set this all up on new projects. If there are open source tools you think Metasploit should be using (such a static analysis tool that can handle metasploit framework and our odd Metasploit module format) let us know in the comments below.

POST STATS

SHARING IS CARING

AUTHOR

Want more? Don’t miss these posts

On the night of September 7, 2014, Joe Vennix of Rapid7's Metasploit Products team wrote, "I did not believe this at first, but after some testing it seems true: in AOSP browser before Android 4.4, you can load javascript into any arbitrary frame or…

As UserInsight grows and we look to add value to more incident response teams that have already chosen the solution that serves as their "single pane of glass", this series will update you on the integrations we build to share valuable context with those solutions.…

Featured Research

The National Exposure Index is an exploration of data derived from Project Sonar, Rapid7's security research project that gains insights into global exposure to common vulnerabilities through internet-wide surveys.

Toolkit

In this toolkit, get access to Gartner's report “Overcoming Common Causes for SIEM Solution Deployment Failures,” which details why organizations are struggling to unify their data and find answers from it. Also get the Rapid7 companion guide with helpful recommendations on approaching your SIEM needs.

Featured Research

Rapid7’s Quarterly Threat Report leverages intelligence from our extensive network—including the Insight platform, managed detection and response engagements, Project Sonar, Heisenberg Cloud, and the Metasploit community—to put today’s shifting threat landscape into perspective. It gives you a clear picture of the threats that you face within your unique industry, and how those threats change throughout the year.