The New York Times published an article in recent months detailing how employers increasingly rely on internal referrals in hiring. In the age of “spray and pray” job seekers, where the cost of submitting an application is a few recycled electrons, this may seem like a rational solution in the war against resume spam.

When the herd starts doing one thing, the smart sheep sees an arbitrage opportunity. Hiring should follow rational economic principles — the goal is to hire the best workers as cheaply as possible. Unusual hiring tactics are the way to score big in the labor arbitrage market.

Which brings me back to my story. I am now the chief scientist at Bright, leading the group that is refining the Bright Score technology (the Bright Score is a scientifically driven way to quantify the match between a job candidate and an open position). In 2007, however, I was a research scientist at UC Berkeley and an experimental nuclear and particle physicist. I decided to make a career change and targeted two areas where my expertise in big data would seem a natural fit — hedge funds and web search. In retrospect, I am lucky I didn’t get a job at a hedge fund.

Since I decided to go into web search (information retrieval would be the technical term), I immediately applied to Google. To their credit, they dug my application out of the pile and arranged for a phone, and then an in-person interview. Along the way, however, the same question came up over and over “Who do you know at Google?”

I didn’t get that job, but did get a job at a search startup (Surf Canyon) that built enhanced search functionality on top of Google. We even published research showing a technique that could dramatically improve search results within Google search.

Every year or so a clever Google recruiter contacts me. I’d really like to work at Google at some point. Google’s like Mecca for an information retrieval specialist.

But each and every time they contact me, I still get the same question: Who do you know at Google? They beg me for names.

Herein likes the problem — I don’t know anyone at Google. They are remarkably thin in terms of former high energy nuclear physicists on their staff. At IR industry events, they are usually poorly represented. When I attended one of the premier research conferences in the field (CIKM), there were about 30 people there from Microsoft, but exactly one researcher there from Google. I tried to befriend some Google IR researchers I got paired with on the golf course, but they were reluctant to have a conversation about their work.

From my interaction with Google recruiters, I can tell that by not knowing anyone at Google I am unfit to work at Google. This reminds me of the classic Economist article on the topic of teacher credentialing that shows a picture of Einstein with the caption “unfit to teach.” A hiring heuristic that gives massive preference to internal referrals means you’ll throw away some of the best candidates.

Reading all three-million unsolicited resumes is certainly not the solution either. Using technology like the Bright Score, TalentBin, Entelo, or Linkedin, smart companies can mine the pool to find the best talent. It is surprising, however, that the company that revolutionized the way we find information is still basing its hiring decisions largely on internal word of mouth.

Good people who don’t have internal connections do apply for jobs. But finding those great candidates among the crowd is not easy. Reviewing every resume is not possible, so it’s hard to sort the good from the bad, so a common trend is to increase the importance of internal referrals. This is dangerous.

Ideally, a company finds a healthy balance between taking the time to find that needle-in-a-haystack candidate and relying on internal referrals. Companies that want to be the most creative and profitable need to recognize this and adopt practices that keep a healthy mixture in their workforce.

David Hardtke is the chief scientist at Bright.com, a San Francisco-based company that is building algorithmic tools to match candidates and employers. Prior to joining Bright, he founded a new meta-search engine called Stinky Teddy that utilizes the real-time web to improve the timeliness and pertinence of web, video, and image search.