In the bin/links ruby script we check if the link disallowed is accessible or not.
Discovering disallowed urls that are accessible is important if we’re
wondering to discover service door and try to break-in

what we can do with robots.txt content (again from links rubygem)

123456789101112131415161718192021

list.eachdo|l|ifrobotsorbulkif!l.start_with?'/'l='/'+l.chompendif!target.start_with?'http://'and!target.start_with?'https://'#defaulting to HTTP when no protocol has been suppliedtarget="http://"+targetendprint"#{target}#{l}:".color(:white)start=Time.nowcode=Links::Api.code(target+l,proxy)stop=Time.nowelseprint"#{l}:".color(:white)start=Time.nowcode=Links::Api.code(l,proxy)stop=Time.nowend...

Crawling: the clean way

What about crawling a website? By crawling I mean retrieving all the possible
urls starting from the homepage, extracting all the links in the HTML and
recursive make a lot of requests.

Crawling: the bruteforce way

Even before discovering anemone rubygem, I
wrote the enchant gem to discover links
by bruteforcing the url with words taken from dictionary.

Using a bruteforce approach can be useful if an important link is not in
robots.txt (and I do suggest not to do this) and it’s likely not linked in any
of the public pages.

Enchant::Engine.get_list method is trivial, it take the words from a dictionary I borrow from Owasp Zap project.

Enchant::Engine.get_list

1234567891011121314151617181920212223

defget_listif@wordlist.nil?ifFile.exists?('../../db/directory-list-2.3-small.txt')@wordlist='../../db/directory-list-2.3-small.txt'endifFile.exists?('./db/directory-list-2.3-small.txt')@wordlist='./db/directory-list-2.3-small.txt'else@list={}endendbeginFile.open(@wordlist,'r'){|f|@list=f.readlines}rescueErrno::ENOENTputs"it seems the wordlist file is not present (#{@wordlist})".color(:red)@list={}endend

There is no real magic in the Enchant::Engine.scan method… just a bunch of get and check for error codes… I know, I won’t win the A.Turing awards for these pieces of code, but sometimes they saved me the day in real pentest.

Of course you can use Net::HTTP also in this case, but Google is not happy to
be called in an automated way without authentication and their api usage… so
it’s easy not to automate the task at all :–)

OWASP-IG-004: Testing for Web Application Fingerprint

This is a 2 years old project, may be it would a great idea to write down a new
and better fingerprinter, however
wafp script can be
used to try to detect the CMS version or a particular Application server
serving our target.

OWASP-CM-001: SSL/TLS Testing

For SSL/TSL testing I use a rubygem I wrote a couple of months ago: ciphersurfer.

However the trick behind ciphersurfer is trying to make HTTPS calls, using
standard Ruby networking APIs (against, no voodoo here).

lib/ciphersurfer/scanner.rb

12345678910111213141516171819202122

defgocontext=OpenSSL::SSL::SSLContext.new(@proto)cipher_set=context.cipherscipher_set.eachdo|cipher_name,cipher_version,bits,algorithm_bits|request=Net::HTTP.new(@host,@port)request.use_ssl=truerequest.verify_mode=OpenSSL::SSL::VERIFY_NONErequest.ciphers=cipher_namebeginresponse=request.get("/")@ok_bits<<bits@ok_ciphers<<cipher_namerescueOpenSSL::SSL::SSLError=>e# Quietly discard SSLErrors, really I don't care if the cipher has# not been acceptedrescue# Quietly discard all other errors... you must perform all error# chekcs in the calling programendendend

Here we don’t use httpclient helpers since I want to play with different
ciphers at time.

That’s it. All the magic happens there. Now, let’s look like at the bin script
to see how the scoring system has been used.

First of all, we must scan the target for all the protocols we support.

cert=Ciphersurfer::Scanner.cert(host,port)if!cert.nil?a=cert.public_key.to_textkey_size=/Modulus \((\d+)/i.match(a)[1]elseputs"warning: the server didn't give us the certificate".color(:yellow)key_size=0end

Note that we don’t make another GET here since we did it at the beginning of
the engagement when we checked if the target was alive or not.

Let's talk about this

I'm an application security specialist and this my blog about software
development, testing and security stuff. Feel free to leave a comment
telling me if you liked this post or not. You can even follow @thesp0nge and @armoredcode on twitter.

If you liked this post, don't miss any armoredcode.com update. Subscribe
to rss feed or receive new posts directly into your
mailbox.
Service is courtesy by Google, I won't store your email address in
any case.

You can discuss, upvote, downvote, and poke fun of this post over at Hacker News.

This story is also on reddit. You can comment and rate this post here.