Open Source Web Reconnaissance with Recon-ng

During a penetration test, a big part of the success in the exploitation phase depends from how good the information gathering was performed. Since this activity, especially when dealing with a huge amount of informations, is time consuming, it is a good idea to rely on tools which make reconnaissance in automated way.

Recon-ng is an incredibly powerful tool for Open Source Intelligence Gathering (OSINT); actually, it is a reconnaissance framework written in Python built with a Metasploit like usage model (we will see what Metasploit is further on, for now it is enough to know that it is the most famous penetration testing framework).
Reconnaissance is considered as the activity of acquiring open source informations, i.e. available on the Internet, about a target in a passive way (passive reconnaissance); conversely, discovery is the activity which permits to acquire informations by sending packets directly to the target (active reconnaissance). Even if Recon-ng is mainly a passive reconnaissance framework, it includes also some elements for discovery and exploitation.

Installation

Since we will use a lot of tools during the next posts, I highly suggest to set up a Virtual Machine with a Penetration Testing distribution installed on.
Personally I use VMware Workstation 12 Player as hypervisor for server and desktop virtualization which is free and can be downloaded from the official website. Regarding operating systems, I use mainly Kali Linux, which is a Debian based distribution. This distro is very useful because it has a pretty good number of tools preinstalled and preconfigured leaving to the user a ready to use PT machine. I will not explain how to set up a VM since you can find a lot of tutorials about that on the web.

Anyway, you can still download Recon-ng on your favorite Linux distribution from author repository using git clone and installing required dependencies (this is also an option in Kali Linux in case you want the latest version available): https://bitbucket.org/LaNMaSteR53/recon-ng.

Usage

In Kali Linux, we can start Recon-ng in different ways. One is by navigating in the applications menu by clicking on Applications > Information Gathering > recon-ng like shown in the following image:

Same thing can be done by clicking on the “Show application” menu:

Another possibility is launching it by simply opening the Terminal and typing recon-ng. In any case, we are prompted with the framework banner, version and number of modules for each category:

Modules are the core of the framework and in the current version there are five categories:

Recon modules - for reconnaissance activities;

Reporting modules - for reporting results on a file;

Import modules - for importing values from a file into a database table;

Exploitation modules - for explotation activities;

Discovery modules - for discovery activities.

The good thing is that everyone can implement his own module written in Python and integrate it inside the framework.
Since we are dealing with information gathering, we will focus on recon modules.
The framework accepts commands via command line; to have a list of the commands just type help and press enter:

Consider, for example, recon/domains-hosts/google_site_web: this performs a recon activity using Google Search Engine to convert an information about a domain into data about hosts of that domain. Keep in mind that certain modules require valid API key to run; some keys can be acquired by simply registering on the related website.
To select a module we need the use command:

In this way we can read the description and take a look at the options we can set before running the recon activity. As you can see, the action performed by this module is pretty the same as the one explained in the article Information gathering with Google Search Engine, but this time it is done in an automated way.
In case we want to analyze module source code we can either use show source or navigate to /usr/share/recon-ng/modules/recon/domains-hosts where the python file google_site_web.py is located (note that folders structure reflects modules categories and data conversions).
Once all required options are set up through set command, the module can be executed with run.

We will see now an example of reconnaissance activity performed on the National Institute of Standards and Technology (NIST) domain.
Before starting, we need to introduce the concept of workspace: Recon-ng allows to define a workspace for each target subject of reconnaissance; by doing this, it will create a database containing all gathered informations about the target itself. This is the reason why in the “framework help” shown before there is the query command, which allows to examine the DB using Standard Query Language (SQL), and also why import modules are present.

We start by creating a new workspace:

workspaces add NIST

After that, the command line shows the change from the default workspace to the new one.
Then we need to associate a domain with the created workspace and finally we can check that everything is set up correctly by listing domains with show:

Adding domains and companies is the initial step because they are inputs used by modules to perform information gathering. To check all modules using these two informations as a starting point we can leverage the search command:

As shown, in minutes we have acquired tons of informations about target hosts.
Now we can lower the search level by digging even deeper: what about looking for contact informations such as names and email addresses?
We can achieve this objective by running recon/domains-contacts/pgp_search: in fact as the description reports, this module searches the MIT public PGP key server for email addresses of the given domain. After module has been executed, we can display results stored in the DB (of course names and addresses in the following table are fictional for privacy reasons):

This is not over yet: we can also search if those contacts have been involved in a databreach, like Adobe one in 2013. For this purpose there are two interesting modules, recon/contacts-credentials/hibp_breach and recon/contacts-credentials/hibp_paste: the first one leverages haveibeenpwned.com API to determine if email addresses are associated with breached credentials, while the other one uses the API to determine if email addresses have been published to various paste sites.

You can check if your email address has been compromised in data breaches by simply going on the Have I Been Pwned? (HIBP) website and launching a search. This service collects and analyzes database dumps and pastes leaked by data breaches happened over the years regarding millions of accounts.

All these informations can be useful during next phases of the attack, especially for Social Engineering (we will look into this technique in future articles).

Once collected enough informations, it is useful to report them in a document. Fortunately, Recon-ng offers modules to report results in different formats:

Recon-ng is a valuable framework for reconnaissance which has a really good system for storing and managing data for later use.
We have seen only a small part of its real capabilities, so take your time to explore and experiment with it to take advantage of its true power.