Digging into Citrix NetScaler IP-reputation feature

I recently had to protect a website using IP reputation feature. There is some good information about this feature, however I decided to glean information here.

Facts about this feature

IP reputation is a platinum feature. It is included in web application firewall (there are extra licenses for the WAF available, they also contain IP-reputation).

IP-reputation feature provides us with a constantly updated feed of “known” malicious IP addresses. This database maintained by webroot. This database is dynamically generated and updated every 5 minutes, so it will never be outdated. Webroot uses sensor networks for this fully automated process. You may use this database manually from here.

It is designed to check for the reputation of an IP address, so to find out if this address is a well known malicious one, or not. All IPs not found in this database are considered to be non-malicious.

IP reputation in admin partitions

IP reputation works fine in admin partition. The iprep.db database-file is shared across admin partitions. so only the root partition needs to be able to access the internet. From technical point of view the partition using ipreputation service neither needs internet access nor be able to resolve names.

Requirements

IP-reputation does HTTP call-outs to api.bcss.brightcloud.com on port 443. You therefore need to be able to:

In the end you’ll find the data base file at /var/nslog/iprep/iprep.db. This file will appear immediately after enabling the feature. This file is not human readable, it’s SQLite. So you have to mess with SQLite browser to dig into it.

NetScaler stores a copy of WebRoot’s database for off-line use (and to avoid undesired latency). It automatically checks for updates every 5 minutes.

During first start of the IP-reputation service Citrix NetScaler does an initial call-out to api.bcss.brightcloud.com from it’s NSIP via port 443 to fetch the database. This process is logged into /var/log/iprep.log

So my reputation file is round about 90 MB in size. It’s a binary file, so there is no point in looking into it.

Thread categories

NetScaler has two built in functions:

IPREP_THREAT_CATEGORY(category)

IPREP_IS_MALICIOUS

While the later is a general one, the first one is very specific. There is a set of threat categories, and you have to specify the ones you’re interested in.

In a reverse proxy deployment you would filter malicious clients: CLIENT.IP.SRC. If you want to protect your clients from connecting to a malicious server you would rather filter potentially malicious server IPs: CLIENT.IP.DST

There are several thread categories (sources: Product documentation by BrightCloud,Citrix)

SCANNERS: The Scanners category includes all reconnaissance such as probes, host scan, domain scan, and password brute force attack.

DOS: The Denial of Service category includes DOS, DDOS, anomalous sync flood, and anomalous traffic detection.

REPUTATION: The Reputation category denies access from IP addresses currently known to be infected with malware. This category also includes IPs with average low Webroot Reputation Index score. Enabling this category will prevent access from sources identified to contact malware distribution points.

PHISHING: The Phishing category includes IP addresses hosting phishing sites and other kinds of fraud activities such as ad click fraud or gaming fraud.

CLOUD_PROVIDERS: I didn’t find any information about this category. As far as I understood, this means, the IP belongs to a cloud provider like AWS, Azure, … So it does not indicate a malicious IP at all.

MOBILE_THREATS: I didn’t find any information about this category. It seems to be a collection of IPs harmful for mobile devices

How to use IP-Reputation service?

Usually I create responder policies with IP reputation feature.

Proxying outside

During proxying to outside I usually use responder policies redirecting to an error page, or respond with a predefined error message telling the user about the reason for blocking.

Action

This action responds with a HTTP 401 (unauthorized) text telling the user about the problem. This will help both, user ans help desk, understanding what’s going on. I would not reset the connection (from technical point of view: send a TCP reset), as a user would not understand, what’s going on, nor would I drop the connection, which would be more or less the same, from perspective of a user, but by far slower (as there is no reply from the server).

Policy

This policy checks if server’s IP is either known for spreading windows exploits or hosting sites used for phishing.

Proxying inside: a Reverse Proxy

if we proxy to inside (which is more common than load balancing outbound) we have several options: Responding with a HTML page (i.e. an error message), blocking or dropping.

Respond with an HTML page seems to be a good idea. My action would look simmilar to the action above. But it would inform the attacker about the reason why he is unable to connect and thereby what to do to connect even though (get an other IP).

Blocking in my opinion is stupid. What would you think if I would block you? My first guess would be: What can I do to be blocked no more? Maybe use a different IP? Use TOR network? So blocking does not seem to be the right thing!

Dropping is my choice. Using a well known bad IP my server would look like not up and running. Being a Black-Hat I’d give up (or invest a serious amount of time on examining the issue).

The policy

so this policy will drop a client request originating from either a well known source of DOS attacks, or from a well known bot network.

some concerns?

Services like IP reputation may tend to false positives. I’d strongly recommend to log, so you’ll be able to investigate issues. My policies therefore usually contain a logging policy.

But think of one of your main customers using an IP with bad reputation? You could ask WebRoot to change the reputation of this very IP address. However this does not work well and takes time. Webroot rates IPs for reason, so it’s very likely to reappear within some days. This may be an IP of a proxy also used by bad guys.

We could white list IPs by just combining the policy with&& CLIENT.IP.SRC.IN_SUBNET(98.12.43.5/26).NOT

However this would lead to an endles list of exceptions, unreadable to humans, inefficient in NetScaler and unmanageable. So I would rather use a data set.

Data sets are lists of numbers, in this case: IP addresses. I use these lists to white list IP addresses.

The mistake is about “and” and “or”. This AND allowed_OP_List is with last term (Botnets) only. You miss brackets round all your ORs, so this excludes allowed IPs from all of them.

In addition it’s wise to put “easy” terms ahead and “difficult ones” behind. To me it seems to be easier to look up a (short) list of IPs than an enormous list of categories.
In my opinion it should look like this:
“CLIENT.IP.SRC.TYPECAST_TEXT_T.CONTAINS_ANY(\\”Allowed_IP_List\\”)).NOT && (CLIENT.IP.SRC.IPREP_THREAT_CATEGORY(WEB_ATTACKS) ||CLIENT.IP.SRC.IPREP_THREAT_CATEGORY(WINDOWS_EXPLOITS) || CLIENT.IP.SRC.IPREP_THREAT_CATEGORY(SCANNERS) || CLIENT.IP.SRC.IPREP_THREAT_CATEGORY(DOS) || CLIENT.IP.SRC.IPREP_THREAT_CATEGORY(BOTNETS)”)”

Yes, it’s an “audit more”. I’d just do it to get more relevant information on a specific problem. It’s definitively not a “general use” policy. It will just give you more detailed information. Keep in mind: Additional policies cause additional CPU load (and CPU is always a concern in WAF deployments).

Sorry, maybe my question was not clear. Is there a way to turn on IP reputation and enable logging as you mentioned, but take no action on the packets? I want to see in the logs that there are connections from “known malicious” IP addresses but I dont want to drop or redirect just yet.

Ah, I see. You may use any action you like. It will just log if you use the built in responder action NOOP. NOOP does not intercept traffic at all (i.e. it will not respond). You could also create a WAF policy using the built in WAF profile APPFW_BYPASS, so WAF will log on it’s own (and you would not even need to do a logging profile)

Thank you for this post and the others !
It really helps me to implement my IP Reputation.

I have one question. I would like to silently drop packet coming from malicious IP. I made a test (creating a blacklist with a dataset containing my public IP). My connection is dropped as intended, but AFTER the SSL handshake.

Is it possible to not send the SYN/ACK packet and apply the IP REP policy before the SSL handshake ?

Cyrille, this is unfortunately not possible as SSL offloading is done prior to responding or WAF.

I could think of a lb vServer, type ANY, with a service, type ANY, pointing to your vServer. This is not a supported method, however in this case IP reputation would be done prior to SSL offloading. Maybe this one will help?