Seattle, 1990

Bob leaned back, stretched his arms and took in the view
from his window. It was a lazy Sunday, and he'd just finished
reading his messages on his favorite BBS. An aging Amiga decorated
with pink stickers courtesy of Bob's daughter sat at his feet. His
sleek new home phone was perched prominently on the corner of the
desk, the cord snaking its way to the wall by way of a splitter
shared with his modem.

As Bob surveyed the scene, his phone started to
ring. Pausing a moment, he smiled, enjoying the harmonic ring of
the new phone that he lovingly customized earlier that morning
before answering.

"Hey Bob, it's Alice!" Alice, a co-worker, was one of
Bob's closest friends. They had both graduated from the computer
science program at the same school, managing to stay in touch over
the years, usually by trading "war stories". Recently, Alice
had joined Bob's group from another company. To the chagrin of Bob
and his co-workers, she was brought in directly as a lead
programmer. "Bah, it's just a title", Bob was fond of
saying. He hid his resentment well when interacting with Alice
socially.

"Bob, my life just changed forever." Bob's eyebrow
slowly began to raise. His mind raced through the
possibilities. Had she been promoted again?Was she
moving on to another opportunity?I wonder if I'll make
lead if she does!. On the verge of Bob's moment of
imagination turning into an uncomfortable pause, Bob snapped out
of it.

"Alice, my goodness, I've never heard you so
excited."

"Okay, you know I'm not normally ahead of the technology
curve," Alice continued. "But today, I just got a cellular
telephone!".

Bob's eyes narrowed. A cellular phone? He'd recently seen a
story about them on the evening news, and thought they were the
most ridiculous thing he'd ever seen. Who on earth needs to
make phone calls in the middle of the street?, Bob had
thought to himself when he saw the story. Bob responded
forcefully. "A cellular phone? Those crazy looking things with
the huge antennas? Alice, you've got to be kidding
me." Bob admiringly moved his hand across the smooth plastic
shell of his phone's handset.

"I'm serious! I know they're a bit different from what
you're used to, but to be honest I kind of like the styling. I
didn't realize how much fun it could be to be on the cutting edge
of technology. You should have seen the looks I got in the park
down the street this morning!"

Bob rolled his eyes and interrupted. "Look, I'm sure it's
fun to be mistaken for an FBI agent, but you can't possibly think
this is rational. When the heck would you need to make a phone
call in the middle of the street? I have two phones at home, and
another at work. What else could I possible need?"

"I can see where you're coming from. I'm sure that to you
it doesn't sound that different from your home phone. I mean, you
pick up the phone, dial the numbers, and it connects to the other
side. Nothing groundbreaking there. But it's so freeing to be able
to talk anywhere. I can completely understand how this might seem
like a novelty. To be honest, listening to myself explain it to
you, it doesn't really sound that compelling. But imagine the
possibilities! I'm always connected. I can have meetings in my
car, and I can leave the office for a long lunch on a sunny day
without worrying about missing out on anything
important."

Bob chuckled, making sure he drew it out long enough to
suggest that he'd heard enough. As the final note of his chuckle
diminished, Bob tried to finish the conversation.

"Okay, Alice, I guess I can sort of see how that might be
nice. But learning a whole new speed dial system, and working with
a phone I don't understand just doesn't seem worth it for those
few situations where it might be useful. I'm glad your happy with
your purchase, in any case."

Alice shook her head slowly, and put in a last word before
hanging up. "You'll see, Bob, just wait."

Seattle, Present Day

Bob was crouched beneath his desk, reaching for the mini-USB
cable attached to the back of his computer. He plugged the other
end into his battery-drained Blackberry, and its bright welcome
screen came to life. Come on, come on he urged. After a
brief delay, the phone's LCD dimmed and the network connected. A
high pitched bell ring alerted him to a new message, and Bob
quickly pressed the "Read" button. She remembered!,
Bob thought to himself excitedly. His daughter had sent the text
message announcing her SAT results, just as promised. I knew
it!, thought Bob, proudly inspecting the score. She's
smarter than her old man after all.

As Bob turned to his computer, he heard the familiar sound
of his favorite classical melody. He had only last week taken an
MP3 of the song and made a custom ring tone, something that
impressed his daughter far more than his last promotion. Who's
calling me during lunch?, he thought. Leaning back, Bob
peered under the lip of his desk, eyeing the blazing white LCD
screen of his phone, now perched atop his computer
chassis. Recognizing the name on caller ID, he immediately grabbed
the phone and answered.

It had been several months since Bob last spoke to
Alice. The time between their phone conversations had grown
progressively longer of late, and Bob was happy to hear her
voice. It had been five years since Alice left for Silicon Valley
to run the software group at a small search startup. Having moved
his way up to middle management, he was always excited to hear
from someone who still had their finger on the pulse of
technology.

Alice got right to the point. "Bob, I am about to change
your life." Bob sensed excitement in her voice. In the two
decades they had known each other, Alice had embraced the role of
technology evangelist, and Bob that of technology skeptic. They
each enjoyed their roles, and Bob could tell immediately that
Alice couldn't wait to tell him about the next big thing. He
gathered up as much feigned skepticism as he could in a feeble
attempt to mask his genuine curiousity, and offered a
response.

"Alright, what is it this time?"

"DVCS. Distributed Version Control Systems", Alice
responded. "We just migrated our entire source control system to
Mercurial. I think in the first week using it we've already gained
100 hours of productivity. The developers love it. I've made it my
mission to tell everyone I know."

Bob pursed his lips. She's got to be kidding, he
thought, this is what she's so excited about?

Bob announced his skepticism. "Alice, Alice, Alice. I've
been around a long time, and it's not quite so easy these days to
put one past me. What are you really so excited about?". Alice
laughed long and hard.

"Oh you old curmudgeon, here we go again. Look, we've gone
through this before, and I am going to convince you that
distributed version control is serious stuff. Why are you so
skeptical already? What have you heard about it?"

"To be honest", he started, "I first started hearing
about distributed version control when we hired on a new developer
who'd been working on an open source project. He was
almost as excited as you about it. He had some trouble
explaining what was so great about it, so he sent the entire team
a link to a video where the presenter bashed our current version
control system for an hour. The guy who manages our VCS server was
really offended. So as far as I can tell, distributed version
control is only relevant in open source projects run by
opinionated bullies who see diff & patch as a perfectly
acceptable source control system. I'm an old corporate soul -- I
haven't used patch since college, Alice, and I don't miss
it."

"Ah, I know the video you're talking about," Alice
said. "Forget about that. I'm going to make the DVCS case for
you, right here, right now. Pretend you've never heard of
it."

"You've got your work cut out for you. To be honest with
you, nobody has ever really bothered to tell me what problem this
thing solves for me."

"That's always the first question!", Alice
said. "I've actually practiced this speech on a few other
ex-colleagues, and everybody asks that question. Unfortunately,
that's the wrong question to be asking. Instead of asking what
problem it solves, you should be asking what new possibilities it
offers. That's been the real win for us."

Bob leaned as far back as his chair would allow, and propped
his feet up on his desk. He had been through this many times
before, and he knew he was in for a long ride. He gave Alice her
opening: "Alright, I'll admit to being slightly intrigued. But
we're really happy with our current VCS, everybody here knows how
to use it, and it handles everything we need without any
issues."

Alice smiled widely. She normally had to work much harder
than this to get Bob on the hook. She knew that beneath the
curmudgeon, he still had a passion for technology. It was her job
to make sure that he didn't lose that, and she embraced it
proudly.

"Okay, I'm going to ask you a very important question, and
I expect an honest answer. Has your source control server ever had
any downtime?"

Bob thought for a moment. "I suppose, sure, but nothing
more than the usual. I mean, our server is up all the time,
really. We plan all our maintenance for the weekends, so other
than the occassional hiccup ..."

Before he could finish his sentence, Alice
interrupted. "Yes! The hiccups! Those 15 minutes of downtime
because of the urgent security patch, the 10 minutes of slowness
when two machines are pulling down copies of the repository! They
happen, and you shrug them off because it's just the way it
is. But do you know how much productivity you lose when somebody
loses their train of thought because the server isn't available
for their 'annotate' command?"

"Look, I get what you're saying. But really, these hiccups
aren't very common", Bob retorted. "I mean, even if they
were, you have to compare them against the cost of having
everybody switch over to a whole new system. The last time we did
that was a total nightmare. One of our developers even quit over
it! And that didn't involve explaining to everyone this crazy new
distributed source control model. I'd have a revolution on my
hands."

Alice responded reassuringly. "I completely understand. I
had my own reservations until very recently. One of our developers
had been trying to get us to adopt a distributed source control
system for a few months. She liked DVCS so much that she had found
some way to use it on her local system and still interoperate with
our centralized system. I was skeptical for the same reasons as
you, though, Bob. I convinced myself that the server was more
reliable than it was, and I tried to forget that our VPN would
sometimes be down all weekend, forcing developers to come into
work to make fixes. Like you, I didn't really think it was worth
it."

Bob interjected impatiently. "OK, so what changed your
mind?"

"Last week our system administrator told us they were
going to be rebooting the server during lunch time. This wasn't a
big deal at the time. Everybody had advance warning, and they made
all of their checkins before lunch, just in case something went
wrong. When we returned, we found our source control system in
pieces, its parts splayed all over the desk. The admin
had installed a few security patches, and the server wasn't
booting. He said he thought it might be bad memory, but he wasn't
really sure."

Alice's horror story had reminded Bob of a similar situation
a year ago when something remarkably similar happened at his own
company. They expected the server to be repaired in 30 minutes,
but it dragged to 60, 90, then 120 minutes. By 3:30 PM most of the
office had cleared out, leaving a few frustrated developers
emailing files to each other as they raced towards the important
deadline which was now in jeopardy.

"Bob, are you still there?" Bob was growing
increasingly nervous as he enumerated in his mind the many things
that can go wrong with a server.

"Yes, yes, sorry, you just reminded me of
something. Please, go on."

"Okay, so the admin couldn't give us an ETA on the
repair. We were feeling really helpless, and called a team meeting
to decide on a protocol for getting work done while the server was
down. Fortunately, one of our team members had a plan. You
remember that developer I told you about earlier who was using the
DVCS on her own system?" Alice was talking excitedly now, and
didn't bother to wait for Bob's answer. "So, she immediately
took control of the meeting, and laid out the plan. She had
created a complete copy of the source system just before it went
down, you know, just in case of emergency, and it was checked into
the DVCS repository on her system. She gave the group some brief
instruction on how to copy the repository, and in less than thirty
minutes our entire team was back to work with their own copies of
the source."

"Wait a second," Bob interrupted. "You're telling me
that you temporarily switched your entire source control system
over in a half hour in the middle of an emergency? Forgive me for
being skeptical, but I'm not buying it. For starters, what happens
to your revision history?"

"When she took a backup of the centralized system, she had
done it via a script that preserved the entire revision
history. She said that migrating from centralized systems was
common, and writing the script was a breeze since most of the
'heavy lifting' was built into Mercurial."

Bob made no attempt to mask his skepticism. "Okay, I get
that, and it's all very impressive, really. But what about all the
work you did while the server was down? How did you get it back to
the main repository once it was revived?"

"That's the best part. We didn't." Alice heard
a squeak on the other end of the phone, and knowing that Bob was
about to ask her if she was crazy, continued without
hesitation. "The admin kept working on the server throughout the
day. During this time, after everybody had their servers
up ..."

Bob cut her off. "Wait, what? Servers? I thought they just
checked out the source code from their co-worker's machine? Where
did all of these servers come from?"

"That's exactly the difference between DVCS and a
centralized system. You don't check out the source code
from the server, you clone the repository from the server
to your local machine. Once you've made the copy, you're a server
too."

Bob was finding this DVCS concept more and more ridiculous
as Alice went on. "Wait, wait, wait. Last I checked you had a
team of 20 developers."

"We're up to 25 now", Alice corrected.

"And you're telling me that in response to a temporary
server outage, you created 25 separate source control
servers, and that's somehow a good thing? How on earth does
anybody know what state the source code is in? Please tell me I'm
missing something. I'm starting to believe you've gone
mad."

"I know, it sounds like chaos. And frankly, it could be if
we let it. That's where process comes in. In this case, everybody
initially cloned the repository from the same server. So, we
designated that as the official server where everybody shared
their changes during the downtime."

"Aha!" Bob was sure that by the end of this
conversation, he'd would bring Alice back down to earth. "So
you're taking this confusing and complicated distributed system,
and making it act just like a centralized system! What happens
when that server goes down? You're no better
off!"

Alice let out a slightly annoyed chuckle. "Not exactly. If
the server we've designated as our central server goes down, which
it surely will at some point, so what? That's yet another
wonderful feature of a distributed system. We've got backups of
the code all over the place, without even trying. Every
developer's server contains a backup copy of most or all of the
code from the shared repository. So, if the main server goes down,
we can designate another server as the central server temporarily,
or we can not worry about it, since developers can make checkins
to their own servers, only sharing changes when needed."

"Wait a second!", Bob interrupted. "If everybody is
making checkins to separate servers, how do you get those changes
back together in the so-called central server once it's back
up?"

"That, my skeptical friend, is called a merge."

"Oh, great," Bob murmered, "you mean like merging
branches?"

"That's exactly what I mean. In fact, repositories and
branches are pretty much the same thing in Mercurial. Whether or
not there is a central server, everybody commits changes to their
own servers. Period. That's the only place you can commit to. When
you want to combine your changesets with those from your
repository, you need to merge them."

Bob groaned. "Yuck! Branches are such a huge pain. And
you're seriously suggesting that making people merge branches
every single time they want to share changes is a
good thing? I'm serious, Alice, if you need some help, I
know some really good people."

"You actually have a point there. We used to think
branches were a pain too. In fact, we only kept a single branch
aside from our main repository because merging branches
was such a pain. This is another huge difference between
distributed and centralized systems. Because you merge branches
all the time, distributed version control systems make it
incredibly easy. Easy to create branches, easy to share
them. Flexibility, Bob, that's what it's all about."

"I don't see what's so flexible about chaos. All this
complexity, just to compensate for a few minor blips of downtime?
I'm not buying it."

"Yes, initially it was just a response to the
downtime." Alice sensed that she was losing Bob, a phenomenon
she was quite familiar with. She softened her tone. "On the day
we started using Mercurial, it was going to be a temporary
thing. My thought was the same as yours, that it was handy as a
temporary crutch, but too complex to keep around. However, our
temporary outage ended up being not-so-temporary."

"What a nightmare." Bob said. As Alice spoke, he had
been working hard trying to convince himself that this was an
incredibly uncommon scenario. Geez, I guess I'd better confirm
that we're backing our server up daily, Bob thought. And
I wonder if I can get budget for a backup source control
server?

Bob had neglected to mention that he had only a year ago
spent $25,000 on licenses for his company's source control
software. While he was proud to now be using the same source
control system as big companies like Google and Microsoft, it left
him with very little budget for hardware. As a result, their
source server was running on an underpowered machine, and it was
not uncommon to hear complaints about its sluggishness. Bob had
also recently received the bill to renew his yearly contract for
support and upgrades. That was going to take another $5,000 out of
his budget, at the cost of some much needed upgrades to his
developers' machines.

Alice continued. "It turned out that our source server's
RAID card had died, and we were going to be down for three more
days before the part could be delivered. It was in these three
days that we discovered the great value of DVCS, not before. One
group of our developers, for example, had previously been trying
to do 'buddy builds', where they share their changes with
each other before committing to the main repository. They had
initially tried to coordinate this work on a separate branch, but
the pain of merging so frequently was killing them, not to mention
the fact that they often forgot to make their changes on the
branch. Then they started emailing each other patches, but this,
too was prohibitively cumbersome. To get around this, they ended
up sharing their source directories to each other directly, and
emailing the names of files that needed to be copied over. Changes
got lost all the time, it was a total mess."

Alice was talking quickly now, hoping to stop Bob from
interrupting. "The afternoon we started using Mercurial,
however, all this stopped. Each developer would finish their work,
commit the relevant changes to their own servers, and
then make the changesets available for their colleagues to pull
and merge into their repositories."

Alice had said the magic word. "Ack! More
merging!"

"Yes! More merging! I'm making it sound too complicated
though. You only have to actually merge files if the changes
you're pulling overlap with your own changes. When you do
inevitably have to merge, though, you do so after you've checked
in your own changes. Thus, even if the merge goes awry, you never
lose your work because of it, something that is extremely
important to us. Regardless, merging is so simple and fast that it
doesn't even matter."

"Many of the new and interesting workflows enabled by DVCS
are possible in centralized systems, but are simply too
much of a pain in practice to have any chance of adoption. Plus,
they can grow organically. You don't need to explicitly decide to
start working on a branch for buddy builds, you can just make some
checkins and choose where to send them. You're always
working on a branch, in effect, since your local repository is a
branch. Not to mention the fact that you don't need to ask a
server administrator for permission to create a branch."

"Okay, so you can do buddy builds", protested Bob,
"and if the server hiccups it's not that big of a deal. But so
what? Those things aren't very useful for us. People around here
just come in and do their work. Even if these benefits are as
profound as you suggest, I really don't think it's worth training
my entire team on some complicated new source control
system."

Alice hadn't expected to sway Bob in only a single
conversation, but she was nonetheless growing a bit frustrated
with her progress. She had used the same sales pitch on others to
great effect, but there was something missing. It just wasn't
having the same effect on Bob. She continued with her defense.
"The things I've mentioned so far are 'big
deals'. Having a server down even for only a minute or two is a
huge deal, in fact. Developer time is expensive, and
knocking a developer out of the zone because of a server hiccup
is, in my mind, totally and completely unacceptable. Heck, even a
bit of server latency can ruin a developer's ability to
stay in the zone."

"One thing I think you're forgetting is that we're using
the best centralized system money can buy," Bob said. "I'm
sure our server is significantly more speedy and reliable than
your free, open source centralized server was. Everybody here is
really happy with it. I've even heard them bragging about it to
friends at other companies."

"Okay," said Alice. "I'll play by your rules, even
though it's unfair. Let's assume that your server is up 100% of
the time, never any downtime, never a hiccup, never a bit of
slowness. Oh, and everybody has constant access to it. DVCS
still offers advantages beyond simply not needing live
server access to get work done."

"Oh? Like what?", queried Bob, now feeling proud of his
progress towards the goal of dragging Alice back into
reality.

"Like no more 'check-in races', for starters. It's a
common complaint every place I've worked. The first person to
commit their changes to the central server avoids having to be the
one who merges changes. Thus, the next committer is forced to
perform the merge, whethero r not they are the best person to
merge the changes. I've actually seen people switch workstations
for ten minutes so the changes could be merged by the proper
resource. How ridiculous is that? With DVCS, because changes are
committed locally, nobody is ever denied the ability to do so
because of someone else's changes. And once they are committed,
anyone can pull the changes together and merge them. If
you had two developers racing towards a deadline, another less
busy developer could volunteer to do all of the change
merging."

"Yeah, well," Bob smirked, "I guess our needs aren't
quite as sophisticated as yours. Don't get me wrong, some of the
stuff you're talking about sounds interesting, and hey, if we
weren't using the best centralized system that money can buy, I
might be more interested. Really, whatever flaws there are in our
current system, we've made the necessary adjustments. People here
are happy. We're doing great work, the team gets along well. I
just don't see the case for DVCS."

Alice realized that there was no more she could do. She bit
her lip, resisting the urge to respond defensively. Planting the
seed, she realized, was the best she could hope for with Bob. She
leaned back in her chair, shook her head softly, and ended the
conversation. "You'll see, Bob, just wait."

"What?", said Bob, "three days! where was the standby server, ready for recovering from the backup? You did take backups, right?"

"Oh, kind of, and we always meant to get around to a standby server" confessed Alice with a toss of her head, "but, anyway, now our build machines each have a complete repository on them, so we use them for releases."

"How do you choose which one?" asked Bob.

"The one that we used last time" smiled Alice, impishly.

"Wait a minute, hold on now." said Bob, "I've just remembered. What about access control? How do I stop just anyone changing the code?"

"Well, whichever machine we declare to be the server for holy. blessed builds that we release, we're really careful about who is allowed to merge changesets to it."

"Really careful? I want a bit more than that" humphed Bob.

"Ok, we have the usual authentication using SSH against our LDAP server and then we do authorization based on file paths in the changesets."

"Sounds better; I guess you do know what you're doing. But does the authorization stay with each cloned repository?"

"Er, we're working on that." admitted Alice, "but once it does, we'll be able to move our official release repository to any machine we like."

"Groovy!" said Bob, glad he had been able to add to the story of DVCS.

Tuesday, May 20, 2008 11:13:19 PM by John

Alice, stunned by the turn of events, thanked Bob for his time. She knew the transition to DVCS was inevitable and was happy to see that Bob and his team would get a productive head start. Always a student of humanity, the level of resistance she encountered was immense yet the conversations were illuminating.

Wednesday, May 21, 2008 2:26:48 PM by A

I think more use cases for the offline access might help -You are demoing some middleware to a customer and need to make many fixes / changes at his place. You still can have your version control with you! Not all companies will allow offsite access to the version control server. Further even within a company many times some locations cannot access the version control or its extremely slow.

Wednesday, May 21, 2008 7:09:30 PM by Jack9

The biggest problem with Version control is merging time, not downtime. More DVCS nonsense.

Thursday, June 12, 2008 6:49:42 PM by John Stracke

Bob had only once heard of cellphones in 1990? What hole was he living in? By 1990, they were showing up on TV (I remember a pocketable one on LA Law), and car phones were going for one penny plus contract.

Saturday, June 14, 2008 11:56:40 PM by Aaron

Jack9, have you used a DVCS? If not, you're as bad as "Bob"; blind and ignorant and unaware. Go try hg. It's not as slick and fast as bitkeeper, but it sure beats svn, cvs, p4, etc. Every time I have to go back to a centralized system, I'm depressed.

Saturday, June 28, 2008 8:38:39 PM by Jakub Narebski

While "one branch per repository" (or "repository / clone is branch") workflow is default Mercurial workflow, distributed version control systems of now (for example Git, but this includes Mercurial with localbranches extension) can use "multiple branches in repository" workflow. This workflow has grown at demand of users (as you can read for example in blog of Junio C Hamano, current git maintainer).

One issue of note is that while push based workflows are not that different from first glance from centralized SCM workflow (but for the backups, commit then merge, developing feature in multiple commits on topic branch, off-line commits, etc.), pull [request] based workflow require having project maintainer (like Linus Torvalds for Linux kernel, or Junio C Hamano for git itself; he/she does need to do all the work if he/she sets up "network of trust" Linus Torvalds was talking about in his Git Talk at Google Tech Talk). But if you read old but (in some) good "The Mythical Man-Month" having maintainer for a project is one of recommendations, I think one that survived passage of time.

Saturday, June 28, 2008 8:41:54 PM by Jakub Narebski

> "The biggest problem with version control is merging time, not downtime. More DVCS nonsense."

@Jack9: with distributed version control systems you can commit, _then_ merge when you have time to resolve it; heck, you can ask somebody else to resolve merge in his/her repository if you are not able to do it for some reason (Linus does it sometimes, by the way).

Saturday, June 28, 2008 11:06:40 PM by Jakub Narebski

> "The biggest problem with version control is merging time, not downtime. More DVCS nonsense."

@Jack9: An alternative to merge (well, not exactly aternative, more like alternate workflow) is something that is called "rebase" in Git, "transplant" in Mercurial, and "graft" in Bazaar. See for example http://www.kernel.org/pub/software/scm/git/docs/git-rebase.html

Git has also additional feature making repeated merging easier, namely git-rerere (reuse recorded resolution of conflicted merges). This tool records how did you resolved merge so you don't have to repeatedly do the same resolving. As far as I know this feature is unique to Git. Lately there was talk on git mailing list about extending feature list of git-rerere, among others deleting resolutions and enabling sharing recorded resolutions...

Friday, July 25, 2008 6:45:07 PM by Tristan

"Sounds better; I guess you do know what you're doing. But does the authorization stay with each cloned repository?"

"No, does your centralised system stop a developer from making changes to parts of his/her checked-out files? No, so I guess stopping changes to a set of files you're not going to use for releases isn't even theoretically possible. Bit odd to require it of the DVCS and not the centralised system."