Posts Tagged 'Packets'

SoftLayer recently released "Global IPs" to a good amount of internal fanfare, and I thought I'd share a little about it with the blog audience in case customers have questions about what Global IPs are and how they work. Simply put, Global IP addresses can be provisioned in any data center on the SoftLayer network and moved to another facility if necessary. You can point it to a server in Dallas, and if you need to perform maintenance on the server in Dallas, you can move the IP address to a server in Amsterdam to seamlessly (and almost immediately) transition your traffic. If you spin up and turn down workloads on cloud computing instances, you have the ability to maintain and a specific IP address when you completely turn down an environment, and you can quickly reprovision the IP on a new instance when you spin up the next workload.

How Do Global IPs Work?

The basics of how the Internet works are simple: Packets are sent between you and a server somewhere based on the location of the content you've requested. That location is pinpointed by an IP address that is assigned to a specific server or cloud. Often for various reasons, blocks of IP addresses are provisioned in one region or location, so Global IPs are a bit of a departure from the norm.

When you're sending/receiving packets, you might thing the packets "know" the exact physical destination as soon as they're directed to an IP address, but in practice, they don't have to ... The packets are forwarded along a path of devices with a general idea of where the exact location will be, but the primary concern of each device is to get the all packets to the next hop in the network path as quickly as possible by using default routes and routing tables. As an example, let's follow a packet as it comes from an external webserver and detail how it gets back to your machine:

The external webserver sends the packet to a local switch.

The switch passes it to a router.

The packet traverses a number of network hops (other routers) and enters the Softlayer network at one of the backbone routers (BBR).

The BBR looks at the IP destination and compares it to a table shared and updated with the other routers on SoftLayer's network, and it locates the subnet the IP belongs to.

The BBR determines behind which distribution aggregate router (DAR) the IP is located, then it to the closest BBR to that DAR.

The DAR gets the packet, looks at its own tables, and finds the front-end customer router (FCR) that the subnet lives on, and sends it there.

The FCR routes the packet to the front-end customer switch (FCS) that has that IP mapped to the proper MAC address.

The switch then delivers the packet through the proper switchport.

Your server gets the packet from the FCS, and the kernel goes, "Oh yes, that IP on the public port, I'll accept this now."

All of those steps happen in an instant, and for you to be reading this blog, the packets carrying this content would have followed a similar pattern to the browser on your computer.

The process is slightly different when it comes to Global IP addresses. When a packet is destined for a Global IP, as soon as it gets onto the SoftLayer network (step 4 above), the routing process changes.

We allocate subnets of IP addresses specifically to the Global IP address pool, and we tell all the BBRs that these IPs are special. When you order a global IP, we peel off one of those IPs and add a static route to your chosen server's IP address, and then tell all the BBRs that route. Rather than the server's IP being an endpoint, the network is expecting your server to act as a router, and do something with the packet when it is received. I know that could sound a little confusing since we aren't really using the server as a router, so let's follow a packet to your Global IP (following the first three steps from above):

The BBR notes that this IP belongs to one of the special Global IP address subnets, and matches the destination IP with the static route to the destination server you chose when you provisioned the Global IP.

The BBR forwards the packet to the DAR, which then finds the FCR, then hands it off to the switch.

The switch hands the packet to your server, and your server accepts it on the public interface like a regular secondary IP.

Your server then essentially "routes" the packet to an IP address on itself.

Because the Global IP address can be moved to different servers in different locations, whenever you change the destination IP, the static route is updated in our routing table quickly. Because the change is happening exclusively on SoftLayer's infrastructure, you don't have to wait on other providers propagate the change. Think of updating your site's domain to a new IP address via DNS as an example: Even after you update your authoritative DNS servers, you have to wait for your users' DNS servers to recognize and update the new IP address. With Global IPs, the IP address would remain the same, and all users will follow the new path as soon as the routers update.

This initial release of Global IP addresses is just the tip of the iceberg when it comes to functionality. The product management and network engineering teams are getting customer feedback and creating roadmaps for the future of the product, so we'd love to hear your feedback and questions. If you want a little more in-depth information about installation and provisioning, check out the Global IP Addresses page on KnowledgeLayer.

One of the most time consuming tasks with iptables is troubleshooting a problematic ruleset. That will not change no matter how much experience you have with it. However, with the right mindset, this task becomes considerably easier.

The rules start at the top, and proceed down, one by one unless otherwise directed.

A rule must match exactly.

Once iptables has accepted, rejected, or dropped a packet, it will not process any further rules on it.

There are essentially two things that you will be troubleshooting with iptables ... Either it's not accepting traffic and it should be OR it's accepting traffic and it shouldn't be. If the server is intermittently blocking or accepting traffic, that may take some additional troubleshooting, and it may not even be related to iptables.

Keep in mind what you are looking for, and don't jump to any conclusions. Troubleshooting iptables takes patience and time, and there shouldn't be any guesswork involved. If you have a configuration of 800 rules, you should expect to need to look through every single rule until you find the rule that is causing your problems.

Before you begin troubleshooting, you first need to know some information about the traffic:

What is the source IP address or range that is having difficulty connecting?

What is the destination IP address or website IP?

What is the port or port range affected, or what type of traffic is it (TCP, ICMP, etc.)?

Is it supposed to be accepted or blocked?

Those bits of information should be all you need to begin troubleshooting a buggy ruleset, except in some rare cases that are outside the scope of this article.

Here are some things to keep in mind (especially if you did not program every rule by hand):

iptables has three built in chains. These are for INPUT – the traffic coming in to the server, OUTPUT – the traffic coming out of the server, and FORWARD – traffic that is not destined to or coming from the server (usually only used when iptable is acting as a firewall for other servers). You will start your troubleshooting at the top of one of these three chains, depending on the type of traffic.

The "target" is the action that is taken when the rule matches. This may be another custom chain, so if you see a rule with another chain as the target that matches exactly, be sure to step through every rule in that chain as well. In the following example, you will see the BLACKLIST2 sub-chain that applies to traffic on port 80. If traffic comes through on port 80, it will be diverted to this other chain.

The RETURN target indicates that you should return to the parent chain. If you see a rule that matches with a RETURN target, stop all your troubleshooting on the current chain, and return the rule directly after the rule that referenced the custom chain.

If there are no matching rules, the chain policy is applied.

There may be rules in the "nat," "mangle" or "raw" tables that are blocking or diverting your traffic. Typically, all the rules will be in the "filter" table, but you might run into situations where this is not the case. Try running this to check: iptables -t mangle -nL ; iptables -t nat -nL ; iptables -t raw -nL

Be cognisant of the policy. If the policy is ACCEPT, all traffic that does not match a rule will be accepted. Conversely, if the policy is DROP or REJECT, all traffic that does not match a rule will be blocked.

My goal with this article is to introduce you to the algorithm by which you can troubleshoot a more complex ruleset. It is intentionally left simple, but you should still follow through even when the answer may be obvious.

Here is the problem: Your server is accepting SSH traffic to anyone, and you wish to only allow SSH to your IP – 111.111.111.111. We know that this is inbound traffic, so this will affect the INPUT chain.

We are looking for:

source IP: any
destination IP: any
protocol: tcp
port: 22

Step 1: The first rule denotes any source IP and and destination IP on destination port 80. Since this is regarding port 22, this rule does not match, so we'll continue to the next rule. If the traffic here was on port 80, it would invoke the BLACKLIST2 sub chain.Step 2: The second rule denotes any source IP and any destination IP on destination port 50. Since this is regarding port 22, this rule does not match, so let's continue on.Step 3: The third rule denotes any source IP and any destination IP on destination port 53. Since this is regarding port 22, this rule does not match, so let's continue on.Step 4: The fourth rule denotes any source IP and any destination IP on destination port 22. Since this is regarding port 22, this rule matches exactly. The target ACCEPT is applied to the traffic. We found the problem, and now we need to construct a solution. I will be showing you the Redhat method of doing this.

Do this to save the running ruleset as a file:

iptables-save > current-iptables-rules

Then edit the current-iptables-rules file in your favorite editor, and find the rule that looks like this:

-A INPUT -p tcp --dport 22 -j ACCEPT

Then you can modify this to only apply to your IP address (the source, or "-s", IP address).

-A INPUT -p tcp -s 111.111.111.111 --dport 22 -j ACCEPT

Once you have this line, you will need to load the iptables configuration from this file for testing.

iptables-restore < current-iptables-rules

Don't directly edit the /etc/sysconfig/iptables file as this might lock you out of your server. It is good practice to test a configuration before saving to the system configuration files. This way, if you do get locked out, you can reboot your server and it will be working. The ruleset should look like this now:

The policy of "DROP" will now block any other connection on port 22. Remember, the rule must match exactly, so the rule on port 22 now *ONLY* applies if the IP address is 111.111.111.111.

Once you have confirmed that the rule is behaving properly (be sure to test from another IP address to confirm that you are not able to connect), you can write the system configuration:

service iptables save

If this troubleshooting sounds boring and repetitive, you are right. However, this is the secret to solid iptables troubleshooting. As I said earlier, there is no guesswork involved. Just take it step by step, make sure the rule matches exactly, and follow it through till you find the rule that is causing the problem. This method may not be fast, but it's reliable. You'll look like an expert in no time.

As I mentioned in "iptables Tips and Tricks - Port Redirection," iptables is probably a complete mystery to a lot of users, and one the biggest hurdles is understanding the method by which it filters traffic ... Once you understand this, you'll be able to tame the beast.

When I think of iptables, the best analogy that comes to mind is a gravity coin sorting bank with four rules and one policy. If you're not familiar with a gravity coin sorting bank, each coin is starts at the same place and slides down an declined plane until it can fall into it's appropriate tube:

As you can see, once a coin starts down the path, there are four rules – each one "filtering traffic" based on the width of the coin in millimeters (Quarter = 25mm, Nickel = 22mm, Penny = 20mm, Dime = 18mm). Due to possible inconsistencies in the coins, the tube widths are slightly larger than the official sizes of each coin to prevent jamming. At the end of the line, if a coin didn't fit in any of the tubes, it's dropped out of the sorter.

As we use this visualization to apply to iptables, there are three important things to remember:

The rules start at the top, and proceed down, one by one unless otherwise directed.

A rule must match exactly.

Once iptables has accepted, rejected, or dropped a packet, it will not process any further rules on it.

Let's jump back to the coin sorter. What would happen if you introduced a 23mm coin (slightly larger than a nickel)? What would happen if you introduced a 17mm coin (smaller than a dime)? What would happen if you dropped in a $1 coin @ 26.5mm?

In the first scenario, the coin would enter into the rule processing by being dropped in at the top. It would first pass by the dime slot, which requires a diameter of less than 18mm. It passes by the pennies slot as well, which requires less than 20mm. It continues past the nickels slot, which requires 22mm or less. It will then be "accepted" into the quarters slot, and there will be no further "processing" on the coin.

It's important to remember that once iptables has accepted, rejected, or dropped a packet, it will not process any further rules on it. In the second scenario (17mm coin), the coin would only be processed through the first rule; the other 3 rules would not be used even though the coin would meet their rules as well. Just because a port or and IP address is allowed somewhere in a chain, if a matching rule has dropped the packet, no further rules will be processed.

The final scenario (26.5mm coin) outlines a situation where none of the rules match, and this indicates that the policy will be used. In the coin bank example, it would be physically dropped off the side of the bank. iptables keeps a tally of the number of packets dropped and the corresponding size of the data. You can view this data by using the "iptables -vnL" command.

Chain OUTPUT (policy ACCEPT 3418K packets, 380M bytes)

cPanel even uses this tally functionality to track bandwidth usage (You may have seen the "acctboth" chain - this is used for tracking usage per IP).

So there you have it: iptables is just like a gravity coin sorting bank!

The three P's in the hosting world have always been Ping, Power and Pipe. Salespeople regurgitated them relentlessly and operations personnel just shortened them to the P's because we talked about them all the time. The three P's of hosting have changed in the recent years and those not aware of the changing landscape are doomed for failure. I propose a new three P standard (described below).

1) Power -- I list this one first because it is by far the most important. Power is the single greatest limiting factor to technology. If you don't understand the importance of power on future technology, you should exit the industry now. If you are not concerned with power, don't meter power and not fixated with power, you will be in serious trouble in the next 12 to 24 months. The entire industry has shifted to being "green" and large scale datacenter operators are so focused on power utilization, they are building and designing systems completely based on power usage and/or location. It's one of the most critical operating costs and must be understood to maximize long term success and profitability. Here at SoftLayer, we are obsessed with power utilization and efficiency and focus on mitigating power and heat (byproduct of power) to a bare minimum. We know the power usage of every server and network device located in the datacenter and track it real time. We are continuously seeking new low power technologies, engaged in industry consortiums looking for new alternatives, and actively planning our power needs through the end of 2010.

2) Packets -- Five years ago, the internet backbones were full of big fat packets that were easily passed by backbone and edge routers without issue. In the recent years, small packet technologies have greatly reduced the size of the average packet transversing the internet. For those of n00bs out there, smaller packets reduce the overall throughput of the routers processing the packets. The smaller the packets, the greater the reduction in horsepower of those routers. The fast rise in gaming, VOIP and other small packet intense applications has cut the average packet size in half in the last two years and I would expect that to occur again the next two years. Packet size can take the aggregate throughput of a router from several hundred gigs at large packet sizes to potentially single digits of gigabit throughput due to the processing required. Here at SoftLayer, we have installed and upgraded to the fastest routing technologies by Cisco to ensure the greatest network performance, but there are many legacy carrier, broadband, and enterprise routers out there that have limited capacity due to changing packet size. Hosting providers that were built on eBay surplus network equipment from the late 90's will soon begin to implode.

3) IP's (IP Addresses) -- Ok…not really a "P" but I take a little creative leeway here. IPv4 addresses are disappearing faster than norm's plate at the Hungry Heifer. ARIN has publically announced the need to shift to IPv6 and numerous articles have outlined the D-Day for IPv4 space. Most experts agree, its coming fast and that it will occur sometime in 2010 at the current pace (that's about two years for those counting). IPv6 brings enough IP space for an infinite number of users along with improved security features and several other operational efficiencies that will make it very popular. The problem lies between getting from IPv4 to IPv6. We are caught in this "chicken and egg" scenario where we can't leave one without the other being completely reliable. Although I think we will get to IPv6 without too much of a headache, I do think the IPv4 space will become extinct prior to a full scale transition and there will be a time where the cost of IPv4 IP's will skyrocket because of supply/demand. This should be at the top of your list as a hosting provider because additional IP space typically means new customer and/or expansion of existing customers. If you don't have a conservation plan for IPv4, migration plan for IPv6, and transition plan between the two – you may already be too late. Here at SoftLayer, we have been planning for over a year and 2008 will include a rollout of IPv6 to all those customers who seek to run dual stacks and will include incentives to customers who are able to shift to IPv6 completely.

The Three P's will likely change again in a few years as the industry continues to evolve and we find a way to solve the current challenges facing the industry. For now, focus and plan on these three and you should have a long successful existence.