is a "DevOps" Engineer at Mozilla Research

Folklore and fallacy

I was a student employee at the OSU Open Source Lab, on and off between
internships and other jobs, for 4 years. Being part of the lab helped shape my
life and career, in almost overwhelmingly positive ways. However, the farther
I get from the lab the more clearly I notice how being part of it changed
the way I form expectations about my own technical skills.

To show you the fallacy that I noticed myself falling into, I’d like to tell
you a completely made-up story about some alphabetically named kangaroos.
Below the fold, there’ll be pictures!

Once upon a time, some kangaroos lived in a desert. One day, for some
inscrutable marsupial reason, a bunch of young kangaroos got together to
practice jumping. Since they’re not particularly creative creatures, they
called the group jumping school.

Every kangaroo who came to jumping school started out only being able to jump
1 foot, and got better at a rate of 1 foot per year, and stayed for 4 years.

A kangaroo called Aggie was one of the school’s first students. She came in
only able to jump 1 foot, but she improved by 1 foot per year.

At the start of the second year that the school was around, a kangaroo named
Bill joined. Bill could only jump 1 foot when he started, and improved at a
rate of 1 foot per year. But Aggie could always jump 2 feet farther than
Bill while she was still in school, because they were both improving at the
same rate.

Nobody new joined for a while, and Aggie left after her fourth year to go
cross roads in front of unwary motorists, but Bill stayed in school and kept
improving. When she left school, Aggie could jump 5 feet. She knew she’d
worked hard, and could always jump farther than Bill, so she felt pretty good
about herself.

At the start of the school’s fourth year, after Aggie had left, a new student
named Claire joined. Claire could only jump 1 foot at first, but improved at
a rate of 1 foot per year.

Claire and Bill would chat about school sometimes, and Claire observed that
Bill could always jump 2 feet farther than her. When she commented on it,
Bill said “If you think I can jump far, you should have seen Aggie! She was a
student here before you came, and she could always jump 2 feet farther than
me!”.

At the end of the school’s 6th year, Bill finished up and went away to fight
in boxing matches. (When Bill left, he could jump 5 feet. He always
suspected he could have done better, since he remembered Aggie always being
able to jump just a bit farther than him.)

A new student, Dave, joined after Bill left. Dave started
out being able to only jump 1 foot and was able to jump 5 feet by the time
he’d been at the school for 4 years. Dave knew that Claire could always jump 2
feet farther than him while they were in school together. Dave heard stories
from Claire about Bill and Aggie, who could both jump even farther than her!

2 years after Dave joined, Claire left jumping school for a full-time job
tearing up farmers’ crops and gardens. She knew she’d tried her best, and she
could jump 5 feet after she’d been at school for 4 years, but she also knew
that Bill had always been a better jumper than her and Aggie had been even
better than Bill. Her 5 feet didn’t seem particularly impressive, since (she
even double-checked her math on it!) Aggie would have been able to jump 4
feet farther, so that must have meant Aggie was a student who could jump 9
feet.

A couple of years later, Dave was finally finished with school. He’d come in only
being able to jump 1 foot, and left being able to jump 5 feet! But he
wasn’t sure if this was better or worse than normal, so he thought about the
other kangaroos who’d also gone to the school.

Dave knew that Claire was a student, and she was able to jump 2 feet farther
than him whenever they studied together. Bill was a student who could jump 2
feet farther than Claire, and Aggie was a student who could jump 2 feet
farther than Bill! Since Dave could jump 5 feet, he concluded that Claire
could jump 7 feet, Bill could jump 9, and Aggie could jump 11! Not only was
Dave the worst of the lot, he wasn’t even half as good as the school’s first
student! He felt pretty bad about his accomplishments, and wondered why
students were getting worse every year.

Epilogue

Once upon a somewhat later time, Jumping School had a reunion and all of the
former students attended. Claire and Dave were a little afraid of meeting
Aggie, since they’d heard such impressive stories of how far she could jump.
All 4 kangaroos tested how far they could jump, just for old times’ sake, and
they found that they could all jump about 6 feet! They compared stories about
their experiences since leaving school, and found that their rates of
improvement had slowed down as they got closer to the limit of how far their
species was able to jump.

The Moral

I think that a similar effect distorted my perception of my own competence when
I compared myself to past OSL students. One can spot the fallacy pretty easily
when everything is spelled out with cute photos: Relative skill levels don’t
translate reliably into absolute ones over time. It’s tricker to spot the same
fallacy in real life, but you might have an easier time now that you’ve seen
the pattern once before.