Wednesday, 3 September 2014

Hello everybody, my name is Alberto Verza, a 23 year student from Spain, and this summer I have participated in Google Summer of Code 2014. My project was the SOAP Scanner Add-On for ZAP, in which I worked during all the Program. Let me explain you the features it includes.

One of the interesting features this Add-On provides is WSDL file scanning. Until now, ZAP could find these kind of files and it could even search URLs inside them, but further petitions to these URLs had not a valid SOAP format specified by the WSDL file. With the SOAP Scanner Add-On, detected WSDL files are now read and SOAP petitions now follow the correct format. You can provide WSDL files in many ways: using the main window "quickscan" option, through proxy navigation, or even importing single files through tools menu.

However, SOAP Scanner Add-On functionality doesn't end here. It couldn't be called "scanner" if it wasn't capable of scanning vulnerabilities after all. That's why it has some SOAP dedicated scanners, which simulates specifical attack vectors for some known vulnerabilites like SOAP Action Spoofing or XML Injection, and it raises alerts when something unusual is coming from server's response. Moreover, the Add-On not only works with scanners made in this project, but also it is compatible with previous developed ones like SQL Injection Scanner or Cross-Site Scripting Scanner, among others.

SOAP Scanner Add-On has passed GSoC final evaluations in alpha state, so if you want to give it a try, you should take a look to the User Guide [1] first.

Finally, I want to stress that OWASP ZAP and SOAP Scanner Add-On are Open Source Software under Apache License 2.0, so contributions to the code are always welcome. If you have not worked for ZAP before, then ZAP Developers will be pleased to help anyone who wants to contribute through the Developers' Group [2], so don't hesitate to ask there.

Active scan rules attack the server, and therefore are only run when explicitly invoked by the user.

You should only use active scan rules against applications that you have permission to attack.

You can also write active scan rules dynamically using scripts, as we will see later in this series, but even then its very useful to understand some of the concepts underlying classes available to you.

Where to start

As with passive rules, the easiest way to get started is to rip off an existing rule.

Active scan rules can be found in 3 locations in the zap-extensions project, depending on their status:

There are also some simple examples that we will examine in more detail.These are all in the alpha branch.

The main classes

Unlike passive scan rules there are different classes that you should extend depending on the type of rule you want to implement.

AbstractPlugin - you typically will not extend this class directly, but it provides key methods that you will need to use and abstract methods that will need to be implemented.

AbstractHostPlugin - extend this class if you want your code to be run once against the top node that the user scans. This is ideal for scanning things that are not ‘page’ related, such as the SSL certificate.

The key method you’ll need to implement is the void scan() method inherited from AbstractPlugin - this is where you perform your attacks.

AbstractAppPlugin - extend this class if you want your code to be run against every node (or page) being scanned. This is ideal for scanning elements that are not related to existing parameters, such as trying new debug flags.

The key method you’ll need to implement is the void scan() method inherited from AbstractPlugin - this is where you perform your attacks.

AbstractAppParamPlugin - extend this class if you want your code to run against every parameter in every node being scanned. This is ideal for scanning existing parameters.

The key method you’ll need to implement is void scan(HttpMessage msg, String param, String value) - this is where you attack the specified parameter.

Performing attacks

Unlike passive scan rules, active scan rules are expected to make requests to the server.

You should use the AbstractPlugin.getNewMsg() method to get hold of a new HttpMessage that you can use for your attack. If you make multiple requests then call getNewMsg() for each request. The message will be a copy of the original request but with an empty response. You can access the original request and response via the getBaseMsg() but you should not modify it.

You should use one of the AbstractPlugin.sendAndReceive(HttpMessage msg, …) methods to actually make the request. The variants allow you to choose whether the underlying code should handle redirects and/or anti CSRF tokens.

The sendAndReceive methods also handle user controlled features like authenticating as a specified user.

When you find potential issues you can raise them via one of the AbstractPlugin.bingo(..) methods.

Like passive scan rules, active scan rules support AlertThresholds which allow the user to indicate how strictly you should check for vulnerabilities.

Simple example

As you will see, it just raises an alert randomly, so it isnt of any practical use.

Like the simple example passive scanner introduced in the previous post it uses the Vulnerabilities class for the documentation associated with the vulnerability.

Most of the methods should be self explanatory, but there are 2 that are worth explaining.

The getCategory() method is used to group related rules in the UI.

And the getRisk() method affects the order the rules are run - the rules which report higher risks are run before those that report lower risks.

File based example

The ExampleFileActiveScanner class implements a slightly more complex active scan rule which is equivalent to the example file passive scan rule introduced in the previous post.

This class introduces another feature, the attack strength, which allows the user to adjust the number of attacks each rule performs.

If you are implementing a rule just for your own use then you dont need to worry about this. However if you plan to publish it for others to use then you should consider supporting it.

The getAttackStrength() method returns an AttackStrength class which can be one of:

LOW: Limit to around 6 requests per scan call

MEDIUM: Limit to around 12 requests

HIGH: Limit to around 24 requests

INSANE: No limit, although 1000s wouldnt be a good idea

You should periodically check to see if the AbstractPlugin.isStop() method returns true - this indicates that the user has stopped the scan so your code should immediately return. The infrastructure will check this before invoking your code to scan a new target (eg page or parameter).

Advanced features

If your rule depends on another rule having been run then you need to specify that via the getDependency() method.

Building and deploying

All you need to do is run the deploy-ascanrulesAlpha target
and the relevant add-on will be built and copied to the correct
location, assuming you have a ZAP core project called ‘zaproxy’.

If you want to deploy to a different location then you can change it at the top of the file. ZAP automatically extracts the files specified in the manifest into a directory underneath the ZAP user directory.An knowledgeable user can manually edit these files and any changes will take affect when ZAP is restarted.

Updating the help and manifest

As with passive scan rules its good to add a short description of the rule to the help file: ascanalpha.html and include the new class along with any files it uses in the add-on manifest: ZapAddOn.xml

Thursday, 3 April 2014

One of the easiest ways to enhance ZAP is to write new passive scan rules.Passive scan
rules are used to warn the user of potential vulnerabilities that can be
detected passively - they are not allowed to make any new requests or
manipulate the requests or responses in any way.They typically run against all of the requests and responses that flow through ZAP.Passive rules run in separate background thread so that they have as little effect on performance as possible.

You can write passive scan rules dynamically using scripts, as we will see later in this series, but even then its very useful to understand some of the concepts and the underlying classes available to you.

Where to start

The easiest way to get started is to rip off an existing rule.Passive scan rules can be found in 3 locations in the zap-extensions project, depending on their status:

This is called for every response. All details of the request and response are available via the 'msg' parameter, as detailed below. The response is also available as a DOM structure via the ‘source’ parameter.

You can implement one or both of these methods depending on your requirements.

You can examine any part of the request and response in order to find potential vulnerabilities, but you must not change anything.

If you find a potential vulnerability then you can raise it via the method:

PassiveScanThread.raiseAlert(int id, Alert alert)

An HttpMessage is passed in to both of the ‘scan’ methods. This class has methods that allow you to access all aspects of the request and response, although the latter is obviously only available in scanHttpResponseReceive.

Some examples include:

msg.getRequestHeader().getMethod();

msg.getRequestHeader().getHttpCookies();

msg.getRequestHeader().getHeaders();

msg.getRequestHeader().getContentLength();

msg.getRequestBody().toString();

msg.getResponseHeader().getHeaders();

msg.getResponseHeader().getStatusCode();

msg.getResponseBody().toString();

A Source parameter is passed into scanHttpResponseReceive - this is a DOM representation of the response generated by the Jericho HTML parser.

See the Jericho documentation or the other scan rules for examples of how to access DOM elements.

There are actually a set of vulnerabilities.xml files as it is internationalized, so ZAP will read the localized version for the language the user has selected, defaulting back to English for any phrases that have not been translated.

This is therefore a quick and easy way to fill in these details, as long as the relevant vulnerability is included in that file.

It also uses the log4j Logger class to output debug messages. This is the recommended way of outputting such messages.

Note that the pluginId needs to be unique across all active and passive scan rules. The master list of ids is scanners.md.

If you are just implementing the rule for your own benefit then you can hardcode the strings if you want, but internationalizing them is very simple and saves having to go back and change you code if you want to have your rule included in the ZAP Marketplace.

The code also makes use of the getLevel() method.

This returns an AlertThreshold class which indicates how strictly you should check for vulnerabilities.

The threshold returned can be one of:

LOW: This indicates you should report more potential vulnerabilities, which might mean more false positives

MEDIUM: This is the default level

HIGH: This indicates you should report fewer potential vulnerabilities, which might mean more false negatives

You do not have to use the threshold - especially as it might not be relevant for the vulnerability you are testing for, but it is also a useful way for the user to tune how the rules work and so its worth using if you can.

Building and deploying

All you need to do is run the deploy-pscanrulesAlpha target and the relevant add-on will be built and copied to the correct location, assuming you have a ZAP core project called ‘zaproxy’.

If you want to deploy to a different location then you can change it at the top of the file. ZAP automatically extracts the files specified in the manifest into a directory underneath the ZAP user directory.An knowledgeable user can manually edit these files and any changes will take affect when ZAP is restarted.

Updating the help and manifest

There are a couple more things that you can do to finish off a new rule.

The first is to add a short description of the rule to the help file: pscanalpha.html

This is not really necessary unless you want to publish your rules.

The second is to include the new class along with any files it uses in the add-on manifest: ZapAddOn.xml

ZAP uses this file to identify any files that need to be extracted.It also uses it when it downloads add-ons from the ZAP Marketplace, and adding your classes and files in here will allow ZAP to add, update and remove your rules dynamically.

A future post will cover how to contribute your code back to the ZAP community and progress it from alpha to beta and then release status.

However its worth understanding how the ZAP projects are structured, especially if you would like to use an alternative IDE.

Project structure

There are currently 2 main ZAP code projects, both one has been migrated to GitHub while the other is still on Google code (but will be migrated), and ongoing development takes place both on the trunks and in branches:

For each project, add all of the jars in the ‘lib’ directory to the classpath

You should update these projects at regular intervals in order to pick up the latest changes.To run ZAP run org.zaproxy.zap.ZAP.java

Each project also has a main Ant build file, build/build.xml, which we will examine in more detail later in the series.

There is more information about building ZAP on the ZAP wikiNote that if you just want to get going as quickly as possible you can just import the zaproxy trunk for core changes and/or the zap-extensions alpha branch for creating a new add-on.

The ZAP core

The zaproxy project is often called “the core”.It has 2 main high level packages within the src folder:

We try to implement significant new features in the zap-extensions project as ‘add-ons’.We do this for the following reasons:

Core changes can only be delivered via ‘full’ ZAP releases. We typically only do these a few times a year.

Add-ons can be released and updated as frequently as required. Users can dynamically install and update add-ons, often without having to even restart ZAP.

Add-ons progress from alpha through beta to release, allowing users to understand how robust an add-on is likely to be. This allows developers to release early without worrying about breaking ZAP for everyone.

Add-ons can still be included in ‘full’ ZAP releases - the WebSockets and Ajax Spider are 2 add-ons that we include by default.

Having said that, you may well find that changes you want to make can only be made in the core. That is not a problem - you can make changes to the core - but these changes will probably not be available to users as quickly as those made to add-ons.We also add new functionality to the core if we want it to be available for other add-ons to build on.Its important to note that if you make changes to code in the Paros package then you must include a comment at the top of the file mentioning your change. This is required to satisfy the Clarified Artistic License that Paros was released under.We have a standard format for these comments

// ZAP: yyyy/mm/dd Issue xyz: Description of the changes

eg see CommandLine.javaIf you make changes to the core then you typically just need to make them to the trunk.There is a branch for every release. If your change fixes a significant issue then we may also want to apply it to the latest release branch. But you dont need to worry about that.

Add-ons, Extensions and Rules

There are many ways to extend ZAP programmatically.Some of the main ways include:

Extensions, which are classes that extend the Extension.java class. They are a powerful way of adding functionality to ZAP, and much of the ‘core’ is actually implemented as extensions.

Active Scan rules, which are classes which extend Plugin.java, detect potential vulnerabilities and run as part of the Active Scanner. These rules typically make multiple requests on the target system. They only run when explicitly invoked by the user.

Passive Scan rules, which are classes which extend PluginPassiveScanner.java, detect potential vulnerabilities and run as part of the Passive Scanner. These rules cannot make any requests - all they can do is examine the requests and responses. They are typically invoked for every request that is proxied through ZAP.

Add-ons are packages which can include all of these components as well as ‘raw’ files. They are usually available on the ZAP Marketplace so that users can discover, download and install them from within ZAP. You can also install add-ons from your filestore via the “File / Load Add-on file…” menu.New add-ons should be created in the alpha branch and only move to the beta and then release branches after they have been reviewed.You can also create add-ons in your own repositories. If they are open source then they will still be eligible to be uploaded to the ZAP Marketplace, but it might take slightly longer to review them.

We will cover all of these components (and more) in more detail in a future posts.

If you have any questions about ZAP development then please ask them on the ZAP Developer group.