Opinions

Do Web 2.0 Companies Really Have The Best Technical Talent?

There are a lot of cool companies with products on the web that millions of people are using. I’ll wondered whether I should label them “web 2.0”, “silicon valley”, “cool startups”, or something else, but I think it’s clear which ones I’m writing about.

The assumption is that these companies attract the best technical talent. And even despite my criticism of their interview process, I still used to think that indeed the best developers and software engineers go to these popular web companies. But due to a couple of observations, I’m no longer certain. These companies have been making junior-developer mistakes in many areas and I can’t imagine that a “top talent” could allow this to happen.

What raised my concerns is security. And it’s not computer-science heavy-cryptography-algorithm-design type of security. It’s the simple web security that every developer out there should actually know – how to store passwords. I was astonished to learn that Yahoo, LinkedIn, and recently Pandora were revealed not to be hashing and salting the passwords, which lead to password leaks. (Note: the case with Pandora is a bit less trivial as noted in the HN comments). It’s the bare minimum thing you should do, and any sane developer who sees that the company stores passwords in plaintext, or uses MD5 hashes, or doesn’t add salt, should just go and fix that (with management approval, yes). “Legacy” is not an issue here – you can hash&salt plaintext passwords, and you can add salt to a hash the next time a user logs in (by getting his actual password on login in order to generate a new hash). But many companies haven’t done that. I would be ashamed to work in a project that doesn’t follow these practices widely known for years, especially if it has millions of users.

But enough with the security. Let’s talk about API design. API design is hard, but it is manageable by top engineers. And yet, there are many instances of unstable, not well-designed APIs. Facebook, for example. They are improving it now, but it used to be horrible. The Android core APIs look (or used to look in the first versions) as if written by a freshman (just a simple example from my android experience: cursor.getInt(cursor.getColumnIndex(CallLog.Calls.TYPE));. Many methods with more than 5 arguments, etc.) Related to APIs – salesforce XSDs sometimes cannot be parsed, because they are invalid – we have been fixing their XSDs in order to communicate with them.

And these are the things that we see on the surface. I’ve heard “horror” stories like writing web projects in C++, and the other day I took a look at the code of reddit, which (even though I’m not a python developer) struck me with some really odd stuff (won’t go into details). I guess many people have heard or seen a lot of “wtf” moments, that a “best developer” just wouldn’t do.

So is it really the case that these silicon valley/web 2.0 companies have the best developers, or they are just regular companies that have average developers doing stupid things? There are certainly some great developers in these companies that do “magic” and “insane” stuff, but apart from the stars, are the rest of the developers also “the best”? I’m no longer sure this is the case.

17 thoughts on “Do Web 2.0 Companies Really Have The Best Technical Talent?”

I’d say it depends on your definition of the best. Your comments make it seem like you think the most experienced people who will think of everything. I think really though when we say we hire the best, we’re looking for people who are willing to, and can tackle any problem and can learn and understand business needs.

The plaintext password cases are blatant incompetence without the slightest vetting for experience or knowledge, and do in fact indicate these companies are hiring extremely poorly qualified developers, with no ability to discern competence, and no ability at the company to review or correct these practices. It’s not just that they have such poor practices in a draft implementation, the big problem is for such a public facing thing as password security, at no point did any of the dozens, hundreds, or even thousands of “engineers” at these companies notice the problems and bring them up to be fixed. That means everyone is incompetent, not just a few.

With API design, that is an extremely difficult skill that one really needs considerable experience to be good at. I share your opinion that most of these appear to be created by recent graduates with little real experience. Apple is another company with baroque APIs (and abysmal documentation) that appear to be thrown together at random with little consistency. Microsoft is one of the few that spends considerable attention and reflection to API and framework design. It seems this is related someone to their hire of Anders Hejlsberg, who is one of the best at this.

I’m at Salesforce and noticed that you wrote “Related to APIs – salesforce XSDs sometimes cannot be parsed, because they are invalid – we have been fixing their XSDs in order to communicate with them.” If you can send me some examples of the problems you’re having I’ll make sure the right people get poked to fix them.

Working at a very successful startup, I learned that it’s much like stock speculation… Pump and dump.

The top echelons of the company are interested in getting features pushed out there as quickly as possible. ZERO interest is paid to anything else. Make an unmaintainable mess, badly designed, horribly inefficient, full of bugs, but do it quickly, and you’re the golden child. Nobody, from bottom to top, plans to stay at the company more than a few short years, so push out junk, and then more and more workarounds to keep the junk just barely working, with new features coming, and use it to justify big raises for yourself. You can NEVER cut too many corners. It’s the next guy’s problem to keep the mess working.

Properly hashed passwords don’t take much effort, but it does take some… So it would be absolutely irrational for the developers to work on it. Your company doesn’t make money from proper security, so higher-ups won’t ever demand wasting developer time on it, either. Start-ups are just highly amplified capitalism… Everybody working in their own short-term interest, does not make for good long-term sustainable decisions. Bigger, older companies are more likely to have a long-view, in my experience, and make decisions that will avoid long-term maintenance nightmares, avoid bad PR that will damage the brand, and more.

For a possible fix, I’d point to PCI-DSS, Sar-Ox, etc. To be allowed to accept credit-cards, you must comply with a list of regulations on how you handle all related data. To be allowed to handle medical records, you must comply with onerous regulations. Of course I’ve heard horror stories of companies violating these agreements, because paying a fine is cheaper, or their horde of lawyers finds loopholes, but the threat of the nuclear-option that is potentially available in both cases is likely to keep companies from going too far, if either is directly relevant to their business.

It’s very easy to criticise individual components or decisions from the outside, with time and hindsight. People learn from experience and even with the best knowledge and intentions, bad decisions can be made. In many startups, the goal is to get things done which is often at the expense of “perfect” architecture. A sign of a good company/engineer is continual improvement. This is easier with the passwords example because you can go back and rehash/encrypt/salt/etc but with APIs, the goal is to avoid breaking existing integrations (unless you go new versions).

There also has to be a business case. Engineers often make the mistake of only considering isolated engineering decisions but a lot of things aren’t broken from a business perspective. Going back to improve something “just because it could be better” is a luxury and even a company with the resources of the likes of Apple has to make a resource allocation decision.

You’re on to something that many seasoned programmers have been saying for quite some time. There is a balance to be played between innovation and operational stability/sustainability.

The younger startup crowd falls very heavily on the innovation side. More experienced/enterprise talent is vastly better at sustainable systems. But the real catch lies in how flexible any given individual is on the scale. Most people are very inflexible, and do not even realize it.

At the end of the day, hiring strategies should make a decision of where they want their people to fall on that scale. It is not a technical question at all, but a strategic business decision.

I think you’re on to something, and looking organizationally, your reasoning implies that the vast majority of ‘experienced’ senior programmers (who have perhaps been bitten by an unsustainable project themselves) spend a large portion of their careers enforcing maintainable code practices on themselves and, more critically, on the new staff who might otherwise be more inclined towards sloppy experimentation more appropriate for new product development, and that restrictive style incurs a huge institutional price in time-to-market.

If that were not so, we wouldn’t see so many acquisitions.

It’s possible for big companies to err in both directions… Anyone remember google’s Orkut? It was (is) a social network, and not bad for it’s day, but just didn’t scale during a critical time for competing for users in the US. It was started as a 20% time project using a totally different stack than google’s normal software, so couldn’t quickly leverage the experience of more seasoned, sustainability-focused developers.

It IS surprising to see big companies make these kind of security errors with their products.

In the end, we need to reward the successes captured by the sloppy mockups, and also strongly reward the successes sustained by the seasoned cleanup crews.

I believe the point that Bozho is making is not about enforcing some mythical “best practices”, as some of the commenters imply.

what I believe he is saying is that making compromises while on a hurry is a fact of life, which many of us developers must learn to live with (if we are to become “senior” at all, that’s a pre-requisite, at least in my career it was), but that there are several things you must NOT allow in your project/organization NO MATTER WHAT, because it makes you look like an idiot, or what’s much worse — makes you look very unprofessional, which is not a good PR for a big company, or a startup which did spread some propaganda and/or manifesto saying how they believe in making XYZ a better area.

In my opinion, storing passwords in plain-text or not salting their hashes, is one of those things.

I guess you are too centered on the geeky technical/engineering aspect in creating a product, which is part of your job, and not the other 90% of what make the final product.

The “web 2.0” companies are not out there to create McLaren Mercedeses, but are out there trying to invent the next VW Beetle, which while not technically perfect even for its time, was probably the fruit of many bright minds and a product which become a legend.