Home Network Lag Issue

I like in a house with my brother, and each of us has a separate (Windows 7) computer that we use. We have a cable modem connection, and use a Linksys (WRT160Nv3) router to have a wifi network, which both computers are connected to.

My brother is a big World of Warcraft player. If he is playing WoW and I am not online, his connection is great. However, if I am online doing any sort of file transfers (uploading to Dropbox or Carbonite, downloading a file, etc) or sometimes even watching large videos online, my brother says his lag is far too high, and he can't play WoW anymore.

We've been "taking turns" on the network for about 3 months now, and I am sick of it. We should be able to have both computers doing whatever they want to, and neither affecting the other, and I am hoping we don't need to buy any expensive hardware to make this possible.

I would be grateful if anyone could suggest how we might be able to remedy this issue.

Is it possible to somehow segment the network traffic into sections so my brother's PC has access to a fixed amount of bandwidth, and the rest is distributed among any other computers on the network?

Your problem is caused by the ridiculously huge buffers in most cablemodems. As soon as your upload bandwidth exceeds the allotted upload bandwidth by your provider, your buffer starts filling. On many units, there's enough memory there for several seconds' worth of buffering, which just completely plays hell with TCP/IP's ability to detect packet loss and speed up/slow down connections to suit. You get into some really pathological behaviors when you have a buffer that large on a heavily saturated upstream connection. Interactive games like WoW become all but impossible to play.

To solve this, you need to do two things. First, you need to set your edge device so that it never exceeds your allotted upload bandwidth. Setting it at around 90% will typically work pretty well... you can try nudging it up to 95% to see what happens. The key is that you must never start filling the cablemodem buffer... as soon as a packet shows up, it should be on the way as instantly as possible. The choke point needs to be on a device under your control, with a very small buffer, so that TCP/IP will realize quickly that packets are being lost.

This alone will solve most of the lag issue. But you also want to avoid dropping packets; if you're using, say, bittorrent, and saturating your upload bandwidth, your roommate's packets won't be lagged, but many of them will simply be dropped, because there's not enough room to send everything. Any packet that gets through will get there quickly, but many packets won't get through, which also plays hell on games like WoW.

This is solved by QoS on your end, prioritizing his WoW packets first in the queue. Anytime a WoW packet is ready, it gets sent immediately, and then everything else is backfilled. I don't know if your router will have QoS features, but if it does, that should fix the packet loss problem. If the native firmware doesn't have this, putting DD-WRT on it should work.

Note that for really proper QoS, both sides of a slow link need to agree about prioritization. You're only controlling half the link, so it's kind of a poor-man's QOS. You can still screw him up by downloading very heavily. Even if you set a limit on incoming packets, you'll tend to flood the cablemodem's buffers if you're really downloading hard, and he'll see lag again.

Typically, however, incoming lag is not as bad as outgoing, because the speeds are so much faster that the buffer flushes much more quickly. WoW's not super-twitchy, so downloading MIGHT not mess him up too bad... that's purely dependent on how fast the connection is, and how big the buffers are.

Also note that if you speed up your incoming link, that means you'll flush buffers faster still.

You can ask your provider if they'll disable buffering for you, which will fix the lag (but not the packet loss) problem. But I've never heard of a provider being willing to do this. I'm not even sure whether they cache on their end, so buying a new cablemodem with a small cache MIGHT help, but it also might not.

Oh, another way you can definitely solve the problem: bring in a second cablemodem. If you're each on your own hardware, you won't interfere with the buffers on the other network at all. You can even still have both gateways on the same internal network, and just configure one PC to use one gateway, and the other PC to use the other gateway. If one goes down, you can temporarily switch over.

We should be able to have both computers doing whatever they want to, and neither affecting the other

What makes you think that?

Malor is on the right path, though. This is also known as the 'dark buffer' problem, since you actually have no control over any other buffering that may be going on. QoS may or may not solve the problem...I'm assuming you have newer OS's, which typically are much better are filling TCP/IP pipes, so without some sort of prioritization scheme, that can actually work against you.

For the OP, part of the problem is you have congestion- you have x amount of bandwidth. Your cable provider is probably asymmetrical (more download than upload). Your brother is actually using a pretty fair amount of upload as the game sends its state to blizzards servers (let's ignore when he's patching the game, just playing) And when you start a file transfer, your PC also attempts to use all the bandwidth it can (even upload by acknowledging data received).

Things I would do to narrow down the problem:

1) get some data from the clients. How much bandwidth do they think they are using on the wireless?

2) get some data and perform basic troubleshooting on the wireless router:

As a trivial first step, have you set upload limits in dropbox and carbonite, perhaps at 50% of your upstream bandwidth? (note that this will almost certainly be much less than your downstream bandwidth, which is typically what is advertised with domestic broadband. Does that help?

Just to be pedantic, buffers aren't a problem in and of themselves. It is buffers plus congestion.

Actually, congestion isn't entirely an issue. The spec was designed to throttle back a bit when it detects dropped packets. With such huge buffers, few get dropped so each computer or service on each computer tries to go full tilt. What you end up with is congestion which doesn't self-correct as it's supposed to do. This is my relative amateur's interpretation and is likely overly simplified.

Setting your router's max as Malor suggests is the basic workaround but, as Frennzy says, you really can't "fix" the real problem easily. There's a nifty tool made by some folks at Berkeley you can use to see how your connection handles. It will tell you, among other things, how much delay is inherent in your connection. Look for the "Network buffer measurements" part. Run this sometime when nothing else is happening and again when your brother's playing WoW. It's Java based but, IMO, is worth looking at the info.

Just to be pedantic, buffers aren't a problem in and of themselves. It is buffers plus congestion.

Actually, congestion isn't entirely an issue. The spec was designed to throttle back a bit when it detects dropped packets. With such huge buffers, few get dropped so each computer or service on each computer tries to go full tilt. What you end up with is congestion which doesn't self-correct as it's supposed to do. This is my relative amateur's interpretation and is likely overly simplified.

Pedantic fail.

You must have congestion to see the problem- it can be exacerbated by having buffers and congestion. Network congestion is almost always an issue, but there are many schemes to over come it. Typically QoS is the most common and most QoS schemes do nothing until congestion happens (or buffers begin to fill).

Just to be pedantic, buffers aren't a problem in and of themselves. It is buffers plus congestion.

Actually, congestion isn't entirely an issue. The spec was designed to throttle back a bit when it detects dropped packets. With such huge buffers, few get dropped so each computer or service on each computer tries to go full tilt. What you end up with is congestion which doesn't self-correct as it's supposed to do. This is my relative amateur's interpretation and is likely overly simplified.

Pedantic fail.

You must have congestion to see the problem- it can be exacerbated by having buffers and congestion. Network congestion is almost always an issue, but there are many schemes to over come it. Typically QoS is the most common and most QoS schemes do nothing until congestion happens (or buffers begin to fill).

Kind of, yes, but congestion relief is something that's built into the IP spec. That's why I said it's not entirely the issue. It's only a part of the issue. Any network can get pushed to the point of congestion but the IP protocol is designed to figure out where that point is by sensing dropped packets then backing down a little. Due to overly large buffers, hardly any packets are dropped so the connections just keep hogging the bandwidth, leaving little else for the rest of the systems trying to use it.

It's a weird issue and kind of counter-intuitive. Again, I'm somewhat oversimplifying, I suspect. Wikipedia has an article on it with sources and there are many others out there discussing it.

Just to be pedantic, buffers aren't a problem in and of themselves. It is buffers plus congestion.

Actually, congestion isn't entirely an issue. The spec was designed to throttle back a bit when it detects dropped packets. With such huge buffers, few get dropped so each computer or service on each computer tries to go full tilt. What you end up with is congestion which doesn't self-correct as it's supposed to do. This is my relative amateur's interpretation and is likely overly simplified.

Pedantic fail.

You must have congestion to see the problem- it can be exacerbated by having buffers and congestion. Network congestion is almost always an issue, but there are many schemes to over come it. Typically QoS is the most common and most QoS schemes do nothing until congestion happens (or buffers begin to fill).

Kind of, yes, but congestion relief is something that's built into the IP spec. That's why I said it's not entirely the issue. It's only a part of the issue. Any network can get pushed to the point of congestion but the IP protocol is designed to figure out where that point is by sensing dropped packets then backing down a little. Due to overly large buffers, hardly any packets are dropped so the connections just keep hogging the bandwidth, leaving little else for the rest of the systems trying to use it.

It's a weird issue and kind of counter-intuitive. Again, I'm somewhat oversimplifying, I suspect. Wikipedia has an article on it with sources and there are many others out there discussing it.

No it doesn't, TCP does. Not the same thing at all, especially when talking about online gaming, which is typically UDP based (not that it's the likely congestion source).

What you're describing above SPECIFICALLY refers to congestion - having more data than pipe and having to back down and/or buffer packets.

As erratic states, if you're not queuing or dropping (aka no congestion) there is no queuing latency (or extremely minimal compared to prop delay delay in most situations) or tail drop.

I think we're kind of saying the same thing except that my point is that the congestion can be mitigated (TCP/IP is specifically designed to do so). When only one thing is using the bandwidth then it's fine. It's when more than one thing needs that bandwidth and it's unavailable that it becomes an issue. With the buffer, there may actually be few enough dropped packets that congestion avoidance doesn't kick in and the symptoms the OP describes show up. QoS may help, sure, but UDP can also be affected by the problem because those packets end up in the buffer as well. It's useful to know if you have a huge buffer that's exacerbating the problem.

Buffer bloat is a real problem with real world impacts. If you don't believe me (and I wouldn't blame you; I am admittedly no expert), you may want to see what Jim Gettys has to say on the matter. If there were true Elders of the Internet, he'd be one of them.

TCP/IP is a large suite of protocols, many of which have absolutely no notion of congestion, notification processes, windowing, or any other mechanism to deal with too many packets and too little bandwidth.

TCP/IP is a large suite of protocols, many of which have absolutely no notion of congestion, notification processes, windowing, or any other mechanism to deal with too many packets and too little bandwidth.

Good catch, thanks. That's what happens when I don't proofread because of leaving to pick up the kid from school.

The version of Gargoyle router firmware I'm running has a feature called active congestion control that pings a target on the ISP side of my connection, and monitors the round trip time. If the round trip time exceeds a preset time it throttles the connection until the ping RTT drops back to normal levels. I'm thinking this feature would mitigate the problem of buffer bloat because it doesn't depend on dropped packets to trigger throttling. Does this seem correct?

Edit: This feature only works in the download direction, but I do have my upload speed set conservatively in the upload QOS settings.

Hmmm... That makes sense, I guess when you are uploading a large file from a fast local network through a much slower internet connection then your cable modem buffer would fill completely almost instantly. If the buffer is very large then you would get a large latency spike, and time critical data would have to wait in line in the buffer. I guess this could be helped by rate limiting your upload at the source pc to significantly less than your worst case upload bandwidth. I guess if I were to limit my upload in my router instead, I would potentially be moving the buffer bloat location to my router instead of my modem. Looks like a difficult issue to solve completely.

Hmmm... That makes sense, I guess when you are uploading a large file from a fast local network through a much slower internet connection then your cable modem buffer would fill completely almost instantly. If the buffer is very large then you would get a large latency spike, and time critical data would have to wait in line in the buffer. I guess this could be helped by rate limiting your upload at the source pc to significantly less than your worst case upload bandwidth. I guess if I were to limit my upload in my router instead, I would potentially be moving the buffer bloat location to my router instead of my modem. Looks like a difficult issue to solve completely.

It is a difficult problem to solve completely; see the discussions among folks like Cerf, et al. If those guys think it's hard then I don't feel bad at all about not knowing how to fix it.

You don't even have to limit the upload significantly, in my experience. Limiting it to, say, 95% seems to be pretty effective. It's worthwhile to run the Netalyzer app and see how *much* of a problem you're dealing with first, though. Run it when you're experiencing issues and when you're not then compare the two buffering readouts. In my case, it's around 600ms without much of a load and doesn't seem to go too much up from there. For others in my same general area on the same ISP, I have seen completely different results.

Part of the problem, though, is that you don't generally know where the buffering is that's causing trouble. Heck, WiFi alone is variable enough that sometimes your network issues have more to do with your neighbor's microwave dinner than anything else (OK, probably rare but you get the idea).

As a troubleshooting step, I'd probably temporarily plug one of the machines in via a wired connection to help determine whether the issue is actually saturation of upstream bandwidth or just the wireless network.

As a troubleshooting step, I'd probably temporarily plug one of the machines in via a wired connection to help determine whether the issue is actually saturation of upstream bandwidth or just the wireless network.

This was just what I was going to point out, they're both on the same WiFi AP, which is a shared medium. To say nothing of his neighbors that are on the same channel, or overlap (yuck).