Bloggarkiv

Now I had the pleasure of talking PCI-DSS compliant XenApp enviroment for a customer. Now after working with it for the last couple of days there are lot of usefull information that I thought I would share.

Now PCI-DSS compliance is needed for any merchant who accepts credit cards for instance an e-commerce size. Or using some sort of application. So this includes all sorts of

Now instead of making this post a pure PCI-DSS post I decided to do a more “howto secure yout XenApp enviroment” and what kind of options we have and where a weakness might be.

Now a typical enviroment might looks like this.

So let’s start by exploring the first part of the Citrix infrastructure which is the Netscaler, in a typical enviroment it might be located in the DMZ. Where the front-end firewall has statefull packet inspection to see what traffic goes back and forth. The best way to do a secure setup of Netscaler is one-armed mode and use routing to backend resources and then have another Firewall in between to do deep packet inspection.

First thing we need to do with Netscaler when setting up Netscaler Gateway for instance is to disable SSL 3.0 and default (We should have MPX do to TLS 1.1 and TLS 1.2 but with VPX we are limited to TLS 1.0

Also important to remember th use TRUSTED third party certificates from known vendors, without any known history. Try to avoid SHA-1 based certificates, Citrix now supports SHA256.

Important to setup secure access to management only (since it by default uses http)

This can be done by using SSL profiles which can be attached to the Netscaler Gateway

Also define NONSECURE SSL renegotiation. Also we need to define some TCP parameters. Firstly make sure that TCP SYN Cookie is enabled, this allows for protection against SYN flood attacks and that SYN Spoof Protection is enabled to protect against spoofed SYN packets.

Under HTTP profiles make sure that the Netscaler drops invalid HTTP requests

Make sure that ICA proxy migration is enabled, this makes sure that there is only 1 session at a time established for a user via the Netscaler

Double hop can also be an option if we have multiple DMZ sones or a private and internal zone.

Specify a max login attempts and a timeout value, to make sure that your services aren’t being hammered by a dictonary attack

Change the password for the nsuser!!!

Use an encrypted NTP source which allows for timestamping when logging. (Running at version 4 and above) and also verify that the timezones are running correctly.

Sett up a SNMP monitoring based solution or Command Center to get monitoring information from Netscaler, or use a Syslog as well to get more detailed information. Note that you should use SNMP v3 which gives both Authentication and encryption.

Use LDAPS based authetication against the local active directory server, since LDAP is pure-text based, and use TLS not SSL, and make sure that the Netscaler verifies the server certificate on the LDAP server

It also helps to setup two-factor authentication to provide better protection against user thefts. Make sure that if you are using a two factor authentication vendor that it uses CHAP authentication protocol instead of PAP. Since CHAP is much more secure authentication protocol then PAP

Use NetProfiles to control traffic flow from a particular SNIP to backend resources (This allows for easier management when setting up firewall rules for Access.

Enable ARP spoof validation, so we don’t have any forging ARP requests where the Netscaler is placed (DMZ Zone)

Use a DNSSEC based DNS server, this allows for signed and validated responses. This way you cannot its difficult to hijack a DNS or do MITM on DNS queries. Note that this requires that you add a nameserver with both TCP and UDP enabled. (Netscaler can function as both a DNSSEC enabled authoritative DNS server and proxy mode for DNSSEC)

If you wish to use Netscaler as an VPN access towards the first DMZ zone, first things you need to do is

1: Update the SWOT library

Create a preauthetnication policy to check for updated antivirus software

Same goes for Patch updates

In most cases try to use the latest firmware, Citrix does release a new Netscaler firmware atleast one every three months which contains bug fixes and security patches as well.

Do not activate enhanced authentication feedback, this enabled hackers to learn more about lockout policies and or if the user is non existant or locked out, disabled and so on.

Set up STA communication using HTTPS (Which requires a valid certificate and that Netscaler trusts the root CA) You also need to setup Storefront using a valid certificate from a trusted Root CA. This should not be a internal PKI root CA since third party vendors have a much higher form a physical security.

If you for some reason cannot use SSL/TLS based communication with backend resources you can use MACSec which is a layer 2 feature which allows for encrypted traffic between nodes on ethernet.

So this is something I’ve struggeled a bit with in the past, also see it on a couple of forums post on Citrix, and there are as always not so detailed info on how to verify on “WHAT THE HELL IS WRONG WITH THE D*** CONNECTION TO DNS AND LDAP!!!”

So therefore I decided to write this post, since both DNS and LDAP are crucial in adding to the Netscaler.

So lets start with DNS. There are a couple of ways to add DNS on the Netscaler. Either its UDP, TCP or TCP & UDP. Now UDP is the one that is typical used since a default DNS uses UDP, TCP is more for Zone transfers and so on.

So what happens if we add a DNS server using UDP, Well the Netscaler is going to do a ping against the DNS server to see if it is alive (So if ICMP is blocked it will show as DOWN) It will check every 20 seconds to see if it respons on UDP/53. Also imporatant to note that it does use the SNIP address to communicate with the DNS server.

How can we verify that it can do name lookup ? (By default most of the built-in cmdlets like nslookup, dig and so on do not work with Netscaler since it has its own DNS feature built-in, and those cmdlets will only query the local DNS not the external one.

So to test DNS use the command

show dns addRec hostanem

So if we switch from UDP to TCP it will try to use TCP Handshake to verify if it is available, but not going to give use the regular DNS query. So what if we cannot reach the DNS server? Using ping from the cmdlet uses NSIP by default

but with ping in Netscaler we can define a source address (Which we can set to be one of the SNIP addresses.)

ping ip-address –S source-address

If you make a trace file you can also see that it works as it should.

If your SNIP does not have access to the DNS server you need to either define ACLs which allow it to communicate with the DNS server, create a new SNIP which has local access to the DNS server or define a policy based routing which define where the SNIP needs to go to inorder to access the DNS servers.

For instance if I want to setup a specific route for my DNS traffic from my SNIP ( I can setup a PBR) which looks like this (This is a policy route only for ICMP)

After I create the PBR I have to run the command apply pbrs

So that took take of DNS, what about LDAP ? When we setup LDAP servers in Netscaler we have the ability to do retrieve attributes button, great! well almost… it uses the endpoint client IP to retrieve attributes (not the NSIP itself) so it by default uses NSIP. So we can use PING to verify network connectivity. We can also use telnet to verify connectivity since telnet originates from the NSIP.

Shell –-> Telnet

open 192.168.60.1 389 (This can try to connect to the LDAP port 389)

How can you verify it works ? It says connected, if it stands on Trying…. the port is not available. If you want to can change that the Netscaler uses SNIP instead of NSIP, this can be done by setting up a load balanced AD server role, then point the LDAP authentication policy to that vServer.

So this is something I’ve struggeled a bit with in the past, also see it on a couple of forums post on Citrix, and there are as always not so detailed info on how to verify on “WHAT THE HELL IS WRONG WITH THE D*** CONNECTION TO DNS AND LDAP!!!”

So therefore I decided to write this post, since both DNS and LDAP are crucial in adding to the Netscaler.

So lets start with DNS. There are a couple of ways to add DNS on the Netscaler. Either its UDP, TCP or TCP & UDP. Now UDP is the one that is typical used since a default DNS uses UDP, TCP is more for Zone transfers and so on.

So what happens if we add a DNS server using UDP, Well the Netscaler is going to do a ping against the DNS server to see if it is alive (So if ICMP is blocked it will show as DOWN) It will check every 20 seconds to see if it respons on UDP/53. Also imporatant to note that it does use the SNIP address to communicate with the DNS server.

How can we verify that it can do name lookup ? (By default most of the built-in cmdlets like nslookup, dig and so on do not work with Netscaler since it has its own DNS feature built-in, and those cmdlets will only query the local DNS not the external one.

So to test DNS use the command

show dns addRec hostanem

So if we switch from UDP to TCP it will try to use TCP Handshake to verify if it is available, but not going to give use the regular DNS query. So what if we cannot reach the DNS server? Using ping from the cmdlet uses NSIP by default

but with ping in Netscaler we can define a source address (Which we can set to be one of the SNIP addresses.)

ping ip-address –S source-address

If you make a trace file you can also see that it works as it should.

If your SNIP does not have access to the DNS server you need to either define ACLs which allow it to communicate with the DNS server, create a new SNIP which has local access to the DNS server or define a policy based routing which define where the SNIP needs to go to inorder to access the DNS servers.

For instance if I want to setup a specific route for my DNS traffic from my SNIP ( I can setup a PBR) which looks like this (This is a policy route only for ICMP)

After I create the PBR I have to run the command apply pbrs

So that took take of DNS, what about LDAP ? When we setup LDAP servers in Netscaler we have the ability to do retrieve attributes button, great! well almost… it uses the endpoint client IP to retrieve attributes (not the NSIP itself) so it by default uses NSIP. So we can use PING to verify network connectivity. We can also use telnet to verify connectivity since telnet originates from the NSIP.

Shell –-> Telnet

open 192.168.60.1 389 (This can try to connect to the LDAP port 389)

How can you verify it works ? It says connected, if it stands on Trying…. the port is not available. If you want to can change that the Netscaler uses SNIP instead of NSIP, this can be done by setting up a load balanced AD server role, then point the LDAP authentication policy to that vServer.

So the last couple of days I’ve been doing a bit of research on Netscaler and prioritizing traffic based upon where the endpoint is coming from. This is where AppQoE comes in. AppQoE is just a combination of different roles into one role, HTTPDoS, Priority Queuing, and SureConnect.

So what if we have a vServer which is getting pounded by traffic, so how do we prioritize the traffic ? So in AppQoE we have two things. Policies and Actions.

Let’s say that we want to divide traffic into two priority groups. One which are android based devices and another which are windows phone devices. Android based devices are given high priority and Windows phones are given lower priority. There are four priorities we can define in AppQoE. HIGH, NORMAL, LOW and LOWEST. And the Netscaler will process traffic from top to bottom. Meaning that Android traffic is priorited over Windows Phone based traffic.

So I have an example expression here for android devices.

My action looks like this

What is does is basically bind HIGH priority traffic sign to my AppQoE policy, so not much work I have to do here. but next I have to create an AppQoE policy to my Windows Phone users.

My AppQoE action looks like this. Now important to see that the policy queue depth defines how many connenctions needs to be active before it is moved to LOWEST priority. I also have to define max connections, if there are requests over the maximum amount of connenctions I have the Netscaler display a custom wait page ( I choose NS, because then I can use a custom HTML code on the Netscaler, if I choose ACS I can choose another web server for instance.

Now i can attach this policy to a vServer. (NOTE that SureConnect cannot be enabled for a vServer if used with AppQoE)

Now stay tuned for how to setup this with HTTP DOS inorder to protect from HTTP attacks as well with AppQoe.

Ever wanted a simple way to block pesky IP-adresses which are giving you much unwated traffic on your webservers? Of course there is the possibility to use ACLs but the become cumbersone if we need to add every IP adress to an ACL (They also get unmanageable)

Another option we have is to use pattern sets. Pattern sets are basically an index with different strings which we can then use against an expression to evaluate if they fall within the category or not.

First we need to create the pattern set, under AppExpert –> Pattern Sets (Which is set to include all of those IP-adresses that we don’t want to access our websites.

Next we need an expression which has the ability to extract out the strings and evaluate them against a rule. Go into AppExpert –> Expressions –> Advanced Expressions –>

Create a new expression called CIP, where the expression looks like this

This will allow us when creating a responder policy to add a string in the expression. Next go into URL responder and create a new policy

Now the magic lies within the expression, since we created a custom saved expression we can use that, which basically just says CLIENT_IP_SRC_EQUALS_ANY”(STRING IN THE PATTERN SET nonoIPS) then RESET Connection.

Then we have to bind the policy to either a vServer or globally, and voila. Now we just have to update the pattern set next time we want to block an IP-address. But do not mistake this for an ACL it only block HTTP access.

Now the last couple of months I’ve again been involved with a Netscaler book project with Packt. This is a more advanced book then my previous book with was a more introduction to Citrix Netscaler.

This new book is called Mastering Netscaler which has more in-depth information regarding load balancing, appfirewall and such.

But… I kinda feel that this book just covers a fragement on what users want to read about when they buy a book about Netscaler.

Therefore in order to get things right, I was thinking about creating a third book about Netscaler which covers all the subjects, stuff you want to read about. Therefore this post is merely for you to give feedback to me

If you could please give me a few senteces about what YOU would want to include in a Netscaler book ? Please drop a comment below this post.

and if you are willing to help me form and maybe contribute to the outline and possibly help me write the book as well that would be great, just send me email to msandbu@gmail.com

There’s alot happening lately and therefore there has been a bit quiet here on this blog. But to give a quick update on what’s happening!

In february I just recently got confirmation that I am presenting two session at NIC conference (Which is the largest IT event for IT-pros in scandinavia) (nicconf.com) Here I will be presenting 2 (maybe 3) sessions.

One session will be primarly focused on Microsoft Azure RemoteApp where I will be showing how to setup RemoteApp in both Cloud and Hybrid and talk a little bit about what kind of use cases it has. The second session will focus on delivering high-end graphics and 3d applications using RemoteFX (using vNext Windows Server), HDX and PCoIP and talk and demo abit about how it works, pros and cons, VDI or RDS and endpoints so my main objective is to talk about how to deliver applications and desktops from cloud and on-premise…

Which will focus more in depth of the different subjects and focused on 10.5 features as well.

I am also involved with a community project I started, which is a free eBook about Microsoft Azure IaaS where I have some very skilled norwegians with me to write this subject. Takes some time since Microsoft is always adding new content there which needs to be added to the eBook as well.

So alot is happening! more blogsposts coming around Azure and Cloudbridge.

So alot is happeing with Netscaler these days. So this is a quick post to tell abit about what is happening.

1: Netscaler appliance in coming in Azure. There is now ETA to when this is coming but this is really important to Citrix workloads. I’m also guessing that this is because of Citrix Workspace Services that i coming.

Heard a rumour that this is coming reaaaaly soon. with partitions a system admin would be able to logical split up a Netscaler into different entities. Think of it like a windows computer with multuple users. Every user has the option to create their own desktop background and customized GUI, and be able to use their own appliations.

So no longer do we need an SDX to do multi tenancy. Even thou we share the same hardware and OS underneath. It is a really cool feature!

So yesterday I held a session at Citrix User Group in norway regarding Netscaler and performance tuning, not so much I can really say about performance tuning in 45 minutes but I think I managed alright.

Now most of this is core Netscaler optimization features, expect Mobilestream which is more related to features standing behind Netscaler. So therefore I wanted to write a blogpost about it as well.

Firstly is the TCP profiles. By default there is an TCP profile which hasen’t changed since 1999. So the Netscaler profile is by default there for compability and not for the best performance, but of course there are alot of different factors invovled here. For instance what kind of network infrastructure you have, packet loss, bandwidth, jitter, firewalls and so on.

But, the main thing is that the default profile does not:

Have Window Scaling activated (Window scaling is usefull send more packets inse the scaling window meaning that we can easier send more data)

Have Selective Acknoledgement activated (Means that we don’t need to resend all the data after a packet loss. Meaning that if we sendt packets 1, 2, 3, 4 , 5 and the sender didn’t receive packet 3 we don’t need to resend 4, 5)

Have Nagle alogrithm activated (Gathers up more data and waits until it reaches the full MTU and then sends the data)

So for instance the ICA-protocol which is very chatty and uses small packets (Which uses alot of overhead) means that it is not suiteable for the regular TCP-profile, so this is where the tcp profile

nstcp_xa_xd_profile (Which has all the features I mentioned above enabled in the policy) but of course you also have the mobile users who are jumping back and forth between different WLAN points or mobile antennas which means there is a point with total packet loss. In the default TCP profile it uses TCP reno, which tries to cut the congestion window in half when it detected a packet loss, not going to do the mobile users any good

Therefore Citrix impletented a variant of the TCP congestion features called Westwood+ which tries to determine the current bandwidth with the device and then it cuts the congestion window to reflect the current bandwidth. Which means that the mobile users can faster get to higher speeds again.

Now also with 10.5 ( I belive) is the option to enable MTCP (Multipatch TCP) so meaning that if you have mobile devices which support two atennas (one for mobile data and one for WIFI which can be used at the same time) we can have two TCP connections from the same device used to access content on the netscaler, its just a policy setting and we are good to go.

The problem is that you need to have specific applications written to leverage MTCP (Not all are there yet)

So go into System –> Profiles –> TCP Profiles (you can either use an existing one or create a new one)

Check for Window Scaling

And here for MTCP (If you need it) SACK and for Nagle.Now there is also an downfall for Nagle since it waits until it waits until a full MTU has been reached before it sends it across the wire and the mobile user has a lot of packet loss, in theory there might be alot of data that needs to be resent across the wire. So for SQL instances for instance, don’t use Nagle!

and the cool part is that these policies can be applied on each vServer and of course services, so dependant on the services it is hosting you can create a differnet policy.

The other thing is SSL tuning, there is a few tips here as well. First thing is quantum size. Bu default the quantum size is 8 KB meaning that the Netscaler will get 8 KB of data that is going to be sent across the wire and the sent it to the SSL chips for encrypting. We can also chance this quantum size to 16 KB meaning that more data is allowed inside the encrypted package.

So for solutions exposing for instnace downloading of large files, a 16 KB quantum size is to prefer. Regular websites which has alot of small data I recommend sticking to the 8 KB.

And then there is of course the autonegititation and duplex, which is something that everybody expects to work fine these days, but…

I still see some having issues with this and specific network devices, so you should always try to manually set the speed and duplex on the netscaler and the switch/router/firewall it is connected to.

For the VPX alot of tuning tips are the same as the MPX but….

For instnace the VPX has support for multiple packet engines meaning that you have a specific engine inside the Netscaler which runs all the different policies, handles encryption and so on. So for a regular VPX it is by default setup with 2 vCPU (One CPU for mangement and another for the packet engine) So if you have an VPX 3000 (2 vCPU and 2 GB ram might not be enough) so if you are using XenServer og Vmware you have the option to add more CPU and RAM to gain additional packet engines. (NOTE: Hyper-v does not support this feature and is capped at 2 vCPU and 2 GB ram and 2vNIC DON’T add 3 vNic)

But of course if you are running Hyper-V and Netscaler VPX make sure you have the newest drivers and make sure that VMQ (Virtual Machine Queing)

VMQ means that a VM has a dedicated Queue on the physical network card if VMQ is not working the VM has to use the default queue along with all the other VMs, with alot of Broadcom drivers that VMQ does not work.

And there is also LACP (NIC teaming, Port Channel, 802.3ad) which allows for aggreating and failover/redundacy on physical NICs (Note that this requires configuration on the switche/s and the Netscaler and it only works on the MPX and the SDX.

There is also a new feature which came with 10.5 is the suppor for Jumbo frames, this allows us to send up to 9000 MTU in an ethernet frame (the default 1500 MTU) which allows for much less overhead since there is more data in a single frame that requires less ACKs)

This only works on MPX/SDX as well, since a VPX is reliant on what the hypervisor provides.This can be configured on per interface. But note that this requires support for jumbo frames on the switch / server, but note that this does not work out over the WAN since it stops at the router or the ISP (This they mostly support the default MTU)

But note the Netscaler also has the Path MTU feature (Which allows) to Netscaler to see the path ahead and see what the lowest minimum MTU is. This feature uses ICMP to determine what the lowest MTU is on a next-hop device. Problem is that since it uses ICMP the next hop devices might be firewalls and such and therefore it might not work. This feature is used to avoid IP fragmentation on the network.