Best Remote Pairing Settings 2008

About a year and a half ago, I made twoposts on the “Best Remote Pairing Settings”.

The world has moved on, and so has the state of the art in remote pairing. I work remote 75% of the time, so this is an important topic for me. The setup we have now is working pretty good, so I wanted to describe it for the benefit of other remoties. Also, I’m only going to describe one specific setup – the one that is currently working best for me. So, here it is:

Computer and OS

Macintoshes running OS X 10.5 (Leopard) and maxed out on ram (at least 2 gig+), with a second monitor, ideally a 24″.

Pivotal uses 24″ iMacsexclusively. This lets you have your remote pair’s screen up on the second monitor, while still having the primary iMac screen available for local work (configuration, looking stuff up, temporary soloing, etc). You must be very disciplined and up-front – you should be explicit about when you are paying attention to your pair and when you are not, but that’s a whole other topic.

Also, at my home office, I have an iMac which I use for screen sharing, but I usually run the Skype audio/video on my MacBook. This takes the CPU load off of the iMac, which is important, because Skype is a CPU hog (more on that later).

Screen Sharing Software

Apple has done a nice job making the screen sharing app perform well, and the most important feature for any remote pairing screen sharing tool is performance. In my opinion, it performs better than any other VNC client on any platform, assuming you have a good connection (possibly excluding some windows native-video-hook solutions like UltraVNC or Remote Desktop, but there’s no way I would ever use Windows for development). All VNC clients which use the standard RFB Protocol can only be tweaked so much, and will only give you mediocre performance. However, there are several annoying bugs and gotchas with Apple Screen Sharing:

It is hard to find. Look under /System/Library/CoreServices/Screen Sharing.app. It is easiest to use Spotlight/Quicksilver or drag it to your dock. Supposedly you can start it with iChat but this never worked for me, I had to run the app directly.

Most of the useful features are disabled by default with no way to access them via menu or toolbar buttons. This is an amazingly annoying decision by Apple, but it is fairly easy to hack the toolbar buttons back into existence. Here is a Macworld tutorial showing two options.

In general, it seems that the best remote-control tools are those with some sort of native/low-level GUI integration: Leopard Screen Sharing, NoMachine NX, UltraVNC, Windows Remote Desktop, etc. Higher-level platform agnostic tools (like standard VNC/RFB protocol) just don’t perform as well – no matter how much you tweak the available color/depth/etc settings.

Audio/Video Hardware

This is a great headset, and you need a really good headset if you are going to wear it all day, every day. Cloth earpieces, mic cover, very long cord, and I believe it also has some echo cancellation built in (there’s a huge box inline in the cord that does something). Unfortunately, I don’t see the exact model on the Plantronics website anymore. It may be replaced by the “GameCom” model, but I haven’t tried this.

Built-in iMac Microphone/Speaker

The built-in microphone and speaker on iMacs is really good. If you want to talk to a group of people remotely (for example, project standup), this is the way to do it. However, if the ambient noise gets too much, you can switch back to the headset.

You can even combine the two. For example, if you want to hear the surrounding conversations, but your pair is having trouble hearing you over the noise, then can wear the headset, but still keep the input set to the built-in iMac mic.

Sometimes you will need to adjust the input/output levels to reduce echo, and the remote pair should handle this themselves – they know what it sounds like.

Built-in iMac camera

Just like the built-in iMac mic and speaker, the iSight is a great camera. A detached iSight is just as good, if you want to be able to move it around or aim it without moving the computer.

Audio/Video Software

Skype is the best I’ve found. It does have drawbacks: it crashes rather frequently, it sucks a lot of CPU, it can do bad things to your network if you become a supernode, and it doesn’t support video in conferences.

However, it has great echo cancellation, it is free, and easy to use. The echo cancellation is really the most important thing – all other audio conferencing tools I’ve tried seem to have much more issues with echoes – even when you are using echo-cancelling hardware devices or speakerphones.

Some people seem to like iChat, but I have not had good luck with it. It takes longer than Skype to connect, the echo cancellation is not as good (sometimes it is, sometimes not), and most annoyingly, it doesn’t always close. I often end up having to force quit it – which is even more annoying when it is stuck on a freeze-frame of me making a stupid face or scratching my nose. Skype never does this – video always goes away when you shut your video or kill the call.

iChat has video conferencing, though, which is a benefit. You can sort of work around this by putting up the video preview in iChat, and having multiple remote people connect to view it via screen sharing, if you only want to see video for one of the participants (e.g. a couple of remote people calling in to a company meeting).

Network, Network, Network

This is the last but most important component to usable remote pairing. A fast, low-latency network connection is critical. I don’t have any numbers, but I believe that low latency is at least as important as high bandwidth. I also (without proof) believe that ping is not necessarily a good indicator of latency – I bet it is possible to have a good ping (ICMP) but still have issues with TCP/UDP latency. Who knows what’s going on in the tubes between you and your pair? Any data, tools, or insight on this would be very welcome.

As empirical evidence, for the first year or two at Pivotal I had DSL, which was pretty fast with low ping, but had continual problems with performance. Then, I switched to corporate-grade cable with a significantly higher bandwidth limit. My experience improved dramatically and my problems decreased greatly. This was about the same time I switched to Leopard screen sharing, so I think that had something to do with it, but the better connection definitely made a huge difference. Again, sorry I don’t have more concrete numbers, but I will guarantee that the better your connection, the better your experience will be.

Also, if you are in a corporate network, this may cause you problems. Even if there is a big pipe to your location, there may be saturation on your local LANs or intranet. Again, no hard data, but this is backed up by experiences of having consistently better performance when connecting to another remote at-home pair with a good connection as opposed to connecting to the Pivotal office which has a much larger pipe.

Remote Pairing Presentation at RailsConf 2008

At RailsConf 2008, Michael Buffington and Joe O’Brien did a good presentation on Remote Pairing. This is a very good presentation which covered many important aspects of remote pairing, as well as presenting some innovative ideas. Unfortunately, I don’t see a link to the preso, please post one in the comments if you have it.

Summary

This isn’t meant to be the be-all, end-all set of recommendations, it’s just what is working pretty well for me now. By “pretty well”, I mean that I can be an efficient pair, even when I’m driving the remote machine.

However, I’ve learned to cope with a lot, and adapted my work habits. It has forced me to become much better at communication, and describing what, why, and how I am programming. In general, though, I believe that remote pairing is physically, emotionally, and intellectually taxing. Regardless, I personally deal with it because Pivotal is such an awesome place to work and Pivots are such incredible developers. Most importantly, I come out in person for a week every month, attend retrospectives and brownbags, have some beers, and generally stay “entangled” with the rest of the team in person. If I was 100% remote, I don’t think I could handle it long-term.

So, I hope this helps out all the other remoties out there. Please let me know what you think, your experiences, and what works well for you.

14 Comments

Regarding latency and protocol, you’re right on both counts. Low latency is vital for real-time communication. And IMCP often gets prioritized differently. There are TCP and UDP ping programs out there, definitely worth trying depending on what protocol your chat client uses.

Screen sharing is pretty great if everyone is on os x. We typically use screen -x to share terminals, combined with Skype.

December 19, 2008 at 10:02 am

Chad Woolley says:

@william – Cool, I’ll try out some tcp and udp ping tools.

@luigi – I’ve briefly tried the new google talk with video. I’ll see how it works for pairing.

@sam – Yes, Gnu screen is handy – I use it when I’m pairing on console-only sysadmin stuff. It’s definitely faster than any gui solution, but you do need to learn the keys and quirks. However, for web development, you have to have full gui screen sharing for browser testing, IDEs, etc.

Thanks for the comments!

— Chad

December 20, 2008 at 12:06 am

peter hessler says:

ICMP is *NEVER* prioritized differently on the transport networks. Yes, it can happen. But they simply don’t care. ICMP is only filtered differently on either endpoint network (and sometimes, not even there). However, the filter is not applied (unless the network is intentionally degrading), until the bandwidth is fully utilized. Once your router is holding packets, you’re toast anyways.

Regardless, ICMP will tell you the *best case* network latency. It will never get better than that. If your ICMP pings are at 350ms, you cannot get better than 350ms delay each way using TCP or UDP.

–phessler

(btw, blabs breaks when javascript is turned off)

December 22, 2008 at 6:31 pm

Chad Woolley says:

@peter – Thanks. I know you’ve said this before, which is why I made sure to qualify my comments indicating that I don’t really know what I’m talking about.

However, I’m still unclear on how I can tell “best case” from “real case”. For example, say I have a lot of bandwidth to spare on both ends, and pings are low, but I still get bad performance (releative to past performance at the same location, or current performance at another location). How do you track that down? If the TCP or UDP pings are returning different metrics than straight ICMP, it must mean something, right?

Then again, as I said, I don’t really know what I’m talking about here. Fortunately, I haven’t had enough painful instances lately to motivate me to look into it further (which I attribute to Leopard Screen Sharing and/or a better pipe on my end), and that is a Good Thing :).

— Chad

December 22, 2008 at 8:13 pm

peter hessler says:

You can check UDP/TCP pings (traceroute defaults to UDP packets, I think fping is able to do TCP pings). You could even try a network game (Quake, or the like), and view the latency across the game. Have the other person host it, to get the full demo. You should also try different sized packets. ICMP is very small, which may be handled better than larger packets. `ping -s1400 [host]` is a good metric, since most of your packets will be about that size (close to the 1500 byte boundary of IPv4).

It could also be that your bandwidth requirements are simply too high. Comparing things when you’re right next to them will give you a way to view the situation when the bottleneck is certainly not the network (gigabit ethernet is fast enough for full-frame HDTV). Sadly, you can’t use Jumbo Packets (that would insanely speed up your connection, since you will only have the 40byte header once per 9000 bytes, instead of per 1500 bytes), as those require every one of the routers between you and your destination to support them.

Then, again, as this article says, network is everything. Recently at Pivotal, the upstream network provider discovered some setting incorrectly set to half-duplex vs. full-duplex. I didn’t get the details, but it made a significant different in Remote Desktop performance.

Skype still crashes all the time, though :)

July 4, 2009 at 1:01 am

jonathanpberger says:

Another quick hack: change the video to black & white (use the filters in iChat; not sure if it can be done in gChat or Skype) is a quick and dirty way to squeeze some extra performance out of the video link. I never bothered to measure it, but I’d guess it chops the video by a third (1 channel instead of RGB) and anecdotally speaking, the gains are pretty noticeable.