Career advice regarding tools

A few weeks ago, J. D. Long gave some interesting advice in a Google+ discussion. He starts out

Lunch today with an analyst 13 years my junior made me think about things I wish I had known about the technical analytical profession when I was 25. Here’s some things that popped into my head:

The entire list is worth reading, but I want to focus on two things he said about tools.

Use tools you don’t have to ask permission to install (i.e. open source).

Dependence on tools that are closed license and un-scriptable will limit the scope of problems you can solve. (i.e. Excel) Use them, but build your core skills on more portable & scalable technologies.

I would have disagreed a few years ago, but now I think this is good advice.

In the late 90’s I used mostly Microsoft tools. That was a good time to be a Microsoft developer. Windows was on the rise; Unix and Mac OS were on the ropes. Desktop applications were the norm and were easier to write on Windows. Open source software was hard to install and hard to use. People who used open source software often did so for ideological reasons, not because it made their work easier.

Of course times have changed. Mac recovered from its near death experience. Unix didn’t, but it has been resurrected as Linux. The web made it easier to write cross-platform software. And above all, open source software has matured. The open source community is more positive, focused on promoting good software rather than trying to give some corporation a stick in the eye.

Now the advantages of open source are clearer. There’s not the same hidden cost in frustration that there was a few years ago. Now I would say yes, it’s a great advantage to use tools you can install whenever and wherever you want, without having to go through a purchasing bureaucracy.

It’s interesting that JD equates open source with scriptability. Open source software often is scriptable, not because it’s open source, but because of the Unix aesthetic that pervades the open source community. Closed source software is often not scriptable, not because it’s closed source, but because it is often written for consumers who value usability over composability. Commercial server-side products may be scriptable. If I were to restate JD’s advice on this point, I’d say to keep composability in mind and don’t just think about usability.

I appreciate JD’s attitude toward applications such as Excel. He’s not saying you should never defile your conscience by opening Excel. Some tasks are incredibly easy in Excel. The danger comes from pushing the tool into territory where other tools are better. There are still some in the open source community who believe that opening Excel is a sin, but I’m much more in agreement with the people who say, for example, that Excel isn’t the best tool for statistical analysis.

Portability is funny. In the early days of computing, there were no dominant players, and portability was important (and difficult). Then for a while, portability didn’t matter if you were content with only running on the 95% of the world’s computers that ran Windows. Now portability is important again. Windows still has a huge market share on the desktop, but the desktop itself is losing market share.

And portability matters for more than consumer operating systems. JD mentions portability and scalability in one breath. You may want to move code between operating systems to scale up (e.g. to run on a cluster) or to scale down (e.g. to run on a mobile device).

There’s also the aspect of career portability. You want to master tools that you can take with you from job to job. I would be leery of building a career around a small company’s proprietary tools. If I were in that situation, I’d learn something else on the side that’s more portable.

In closing, I’ll give the rest of JD’s career advice without commentary. These points could make interesting fodder for future blog posts.

Be a profit center, not a cost center.

Use tools you don’t have to ask permission to install (i.e. open source).

Dependence on tools that are closed license and un-scriptable will limit the scope of problems you can solve. (i.e. Excel) Use them, but build your core skills on more portable & scalable technologies.

Learn basic database tools.

Learn a programming language.

Your internal job description may say, “Analyst” but get something else on your business cards. Analyst is so vague as to be meaningless. My external title is currently “Sr. Risk Economist.” I like the term “Data Scientist” for now. I expect that term will be meaningless in 5 years.

Large organizations do not properly appreciate agile and smart analytic types. Time at large firms should be seen as subsidized learning. Learn lots, but get out.

Ensure you can explain any of your projects to your wife or non-technical friends. It’s good practice for board meetings later in your career.

Be sure you know the handful of things that you can do better than most anyone else. Add something to that list every year. Make sure you can explain these things to non techies.

Be a profit center, not a cost center. At least be as close to the profit center as possible. The chief analyst for the sales SVP is closer to the profit center than an IT analyst supporting billing operations.

Get really good at asking questions so you understand problems before you start solving them.

Yes, that bit about being a profit center not a cost center is in there twice. It should probably be in there 5 times.

While I tend to agree with the open-source argument, I am still not able to find suitable open-source alternatives to my main analytical engine — however, I do have a separate computer laboratory setup where I am trying to recreate the same tools in an open-source environment — I anticipate that I will eventually be successful — but not yet…

It is not always true that you can install open source tools without permission. On your own system, yes; on a company’s system, maybe not. I’ve gotten a non-trivial amount of flack at least twice for installing open source tools on my primary work desktop, because it was owned and regulated by my employer and they didn’t want the tools there, so the tools had to be uninstalled.

Of course I’m sure someone like JD would take that as prima facie evidence that the employer is not worth working for, but everyone’s mileage varies.

That being said, I think the advice is very good. Back when my environment was Unix which changed flavors regularly, I always kept the GNU tools first in my path, so that as little as possible changed when the mainframe changed.

Without rebutting anything here, and without wishing to defend or promote Microsoft, I must correct a technical point. Excel is quite scriptable, from the inside (using Visual Basic for Applications) and from the outside (via COM, I have scripted it with Tcl). It goes way out of its way for interoperability, too — I have plugged pieces together to achieve live AIM -> message parsing -> Excel ->pricing model -> send alert IM.

I would be interested in reading future blog posts on the rest of JD’s advice, especially as someone in her mid-20s who would be on the receiving end, but I hope you won’t repeat his implicit assumption that all members of the “technical analytical profession” are men who should endeavor to explain their work in sufficiently simple terms that even their wives can understand.

I don’t meant to nitpick language here, and I understand that JD’s point is the importance of being able to explain one’s work to non-technical people. I think he’s given some great advice in that list, but it’s disheartening to me that someone in a mentoring role would default to assuming that all those reading advice on technical or quantitative careers are men.

Cost vs. profit center is important as far as it goes, but not as big a deal as getting paid a salary vs. getting paid a percentage vs. being paid a salary. The whole reason successful programmers aren’t well paid compared to successful hedge fund managers, investment bankers, contract lawyers, salespeople, and executives is that we’re not getting a percentage. This is true even at companies whose product is software. Stock options can give you a small to large percentage depending on when you get in, but they’re very high variance compared to, say, brokering a deal at a percentage.

John Venier’s spot on. Most large companies (e.g., AT&T or IBM) control their desktop environments pretty tightly and don’t let employees install whatever they want. We were exempted from this policy at Bell Labs, but the researchers aren’t always. They’re primarily worried about intellectual property liability in case any of the open source code is stolen — they have some protection from vendors through indemnification.

The term “data scientist” always makes me think of a suitable contrast, such as “making things up scientist”.