Monday, May 24, 2010

The Secret Lives of Professors

I came to Harvard 7 years ago with a fairly romantic notion of what it meant to be a professor -- I imagined unstructured days spent mentoring students over long cups of coffee, strolling through the verdant campus, writing code, pondering the infinite. I never really considered doing anything else. At Berkeley, the reigning belief was that the best and brightest students went on to be professors, and the rest went to industry -- and I wanted to be one of those elite. Now that I have students that harbor their own rosy dreams of academic life, I thought it would be useful to reflect on what being a professor is really like. It is certainly not for everybody. It remains to be seen if it is even for me.

To be sure, there are some great things about this job. To first approximation you are your own boss, and even when it comes to teaching you typically have a tremendous amount of freedom. It has often been said that being a prof is like running your own startup -- you have to hire the staff (the students), raise the money (grant proposals), and of course come up with the big ideas and execute on them. But you also have to do a lot of marketing (writing papers and giving talks), and sit on a gazillion stupid committees that eat up your time. This post is mostly for grad students who think they want to be profs one day. A few surprises and lessons from my time in the job...

Show me the money. The biggest surprise is how much time I have to spend getting funding for my research. Although it varies a lot, I guess that I spent about 40% of my time chasing after funding, either directly (writing grant proposals) or indirectly (visiting companies, giving talks, building relationships). It is a huge investment of time that does not always contribute directly to your research agenda -- just something you have to do to keep the wheels turning. To do systems research you need a lot of funding -- at my peak I've had 8 Ph.D. students, 2 postdocs, and a small army of undergrads all working in my group. Here at Harvard, I don't have any colleagues working directly in my area, so I haven't been able to spread the fundraising load around very much. (Though huge props to Rob and Gu for getting us that $10M for RoboBees!) These days, funding rates are abysmal: less than 10% for some NSF programs, and the decision on a proposal is often arbitrary. And personally, I stink at writing proposals. I've had around 25 NSF proposals declined and only about 6 funded. My batting average for papers is much, much better. So, I can't let any potential source of funding slip past me.

Must... work... harder. Another lesson is that a prof's job is never done. It's hard to ever call it a day and enjoy your "free time," since you can always be working on another paper, another proposal, sitting on another program committee, whatever. For years I would leave the office in the evening and sit down at my laptop to keep working as soon as I got home. I've heard a lot of advice on setting limits, but the biggest predictor of success as a junior faculty member is how much of your life you are willing to sacrifice. I have never worked harder than I have in the last 7 years. The sad thing is that so much of the work is for naught -- I can't count how many hours I've sunk into meetings with companies that led nowhere, or writing proposals that never got funded. The idea that you get tenure and sit back and relax is not quite accurate -- most of the tenured faculty I know here work even harder than I do, and they spend more of their time on stuff that has little to do with research.

Your time is not your own. Most of my days are spent in an endless string of meetings. I find almost no time to do any hacking anymore, which is sad considering this is why I became a computer scientist. When I do have some free time in my office it is often spent catching up on email, paper reviews, random paperwork that piles up when you're not looking. I have to delegate all the fun and interesting problems to my students. They don't know how good they have it!

Students are the coin of the realm. David Patterson once said this and I now know it to be true. The main reason to be an academic is not to crank out papers or to raise a ton of money but to train the next generation. I love working with students and this is absolutely the best part of my job. Getting in front of a classroom of 80 students and explaining how virtual memory works never ceases to be thrilling. I have tried to mentor my grad students, though in reality I have learned more from them than they will ever learn from me. My favorite thing is getting undergrads involved in research, which is how I got started on this path as a sophomore at Cornell, when Dan Huttenlocher took a chance on this long-haired crazy kid who skipped his class a lot. So I try to give back.

Of course, my approach to being a prof is probably not typical. I know faculty who spend a lot more time in the lab and a lot less time doing management than I do. So there are lots of ways to approach the job -- but it certainly was not what I expected when I came out of grad school.

Let me push a bit in the positive side: I think the freedom to pursue your own research agenda is incredibly liberating. Going after the problems you care about and those that drive you, that's a special opportunity rarely found outside academia, including the best, most academic research labs.

NSF funding is rough, particularly when you have to drive things all by yourself. And while it's awfully hard to get a large core together (a la SD), I've found that just having 1-2 people who you can work well with and trust has a huge impact on making the load bearable.

20% of your NSF grants funded is simultaneously not bad, and very frustrating. My rate is about the same. 4 awarded out of 15 submitted (until two weeks ago, it was 3 out of 14, so don't be deceived by the small numbers variance). :)

And that's not mentioning how much of the funding is cut during the process.

My "professor anxieties", in order, are:a) Making sure I have enough funding to support my students and staff;b) Making sure I do well by my Ph.D. students in terms of helping them find good problems to work on.

Note that that's not my priorities -- but those are the things I get most anxious about, in part because of the funding rate that you mentioned.

Thanks for the post; I really like the "behind the scenes" aspect of this blog.

I am curious to know if you have a prescriptive rather than descriptive take on your frustrations as a professor: Are there aspects of the system that you believe could be improved (e.g., through changes in standards for tenure or grant evaluation) without compromising others? For instance, could your "endless string of meetings" be made shorter without some your department or research community losing out as a result?

Dave - I think it depends a lot on the department. We have a small CS faculty so more administrative burden falls on each professor. At Berkeley I got the impression that there were some very productive faculty who spent very little time on non-research-related administrivia, but maybe that was a carefully-concocted illusion intended to keep the students working hard :-)

Regarding your point about: "It has often been said that being a prof is like running your own startup", I recently wrote an article that makes a similar analogy, except comparing the life of a CS Ph.D. student to someone hacking on an early-stage start-up ...

As someone who came from industry to become a professor and then went back, the biggest shock was how much LESS professors get paid. My first job as assistant professor paid less than I made as an engineer BEFORE I went to graduate school.

Running a start-up is the same as being a professor in that you spend an enormous amount of time on proposals, meetings with potential clients, etc. that may never pay off. Second, there is always something else to do - a new product, new marketing campaign, new contract to bid.

Having been in both positions, I feel like I have more freedom at a small company (especially since I now own it). There is, in my experience, less opportunity for intellectual growth. Every ten years or so I take a post-doc or position at university just to get back at the cutting edge.

LoseThos is a simple 64-bit PC Operating System with no paging. It runs everythying in Ring 0, so your students can very easily play with virtual memory. There are a few instructional exercises for students.

I had the idealized view before I went to grad school but I was pretty clear on what was involved by the time I graduated. The banal answer is that there are trade-offs: (somewhat) more intellectual freedom in exchange for (somewhat) lower pay and having the stress of looking for funding.

Industry vs Avademia has some analogies with "applied" research vs "theoretical" research within academia (or within a discipline)

One the one side you have more funding and more "usefulness" but also more constraints (the research has to be applicable in some manner). On the other hand you have less funding, less usefulness but also fewer constraints (the research just has to be interesting in some way).

So you could see it as a continuum in some way.

Wall Street quant on one extreme and a postmodern English professor on the other end ... ;-)

As a counterpoint, I continue to think being a professor is one of the best jobs in the world. Its unmatched with respect to personal freedom and research freedom, it permits a broad range of career paths and lets you work on many different things with many different people. That said, its not for everyone and, just like every job, there is a bunch of scut work. However, there are quite a few coping strategies for maximizing the positive aspects and minimizing the negative ones. It is not the case that fund raising must take 40% of one’s time or that endless meetings are the only strategy for being a faculty member :-)

However, by far, the most important strategy is a very simple one: try to work with people you really like. I can’t emphasize enough how much this matters since this colors almost everything else. If you have a choice of places to work, I’d pick the place with one or two people you really really like, over a “higher ranked” place any day. If you are hiring people, don’t fail to consider how much you’d like working with them. If you don’t have people you like where you work, then work with people elsewhere who you do like. I think I’ve done all of these and have continually underestimated the value each time.

Let me put an end to this dis-information for those interested in an academic career. There are many many ways for professors to run their research group.

In my research group for example, most of my time goes into: discussing research with students, writing papers, reading papers, reading/writing email, preparing/teaching class, mentoring students on presentation writing etc., proposal writing, meetings, giving talks in that decreasing order. And depending on how targeted your proposal writing is, your success rate can be as high as 90% or 80%.

So don't conclude this description is "representative" in anyway whatsoever as Prof. Welsh himself has alluded to.

VictoryX2 - I actually did a fair bit of due diligence, interviewing at something like 15 universities and 4 industrial labs (Google, HP, IBM Watson and IBM Almaden). At the time, Google was a scrappy startup that mostly did search, and did not seem like a viable long-term career option (this was pre-IPO days - I reckon that was a multimillion dollar mistake :-) HP and IBM are somewhat old-fashioned and seemed a lot less glamorous than a faculty job. I saw the full spectrum of universities from big (CMU) to small (Caltech). My biggest mistake was not interviewing at MSR, which has a great mix of academic culture and lots of resources.

Stefan - I actually love all of my colleagues here although not enough of them work in my area.

Anon "dis-information" - I guess you didn't get to the end of my post where I say explicitly that my experience is atypical?

Just to add a different personal experience, I have spent very little of the past 6 years sitting on stupid committees, and probably spend only about 10% of my time writing proposals or chasing funding. I find the grant writing process quite helpful actually as it gets me thinking about what I want to do next.

I wholeheartedly agree with what Stefan said -- you have to find ways to minimize the scut work.

I do spend a lot of time meeting with students, but that's the fun part of the job.

I still write code; I always try to have some kind of (sometimes small) hacking project.

I do spend a lot of time on my job, but not to the exclusion of an outside life.

Overall I can't imagine a job I'd rather have (and I've worked in research labs and been involved in startups -- though not as a programmer.)

Seriously, I really think it has a lot to do with the institution and department. From what I can tell, we seem to have a higher load of scut work, and as for fund raising, we don't have big corporate consortia like MIT. It's also true that I have taken on more than my fair share of external service work.

I am an about-to-graduate Ph.D student with a lot of friends who are also in the same boat. One thing I have repeatedly encountered in my conversations with friends, colleagues etc., is the whole notion that "the best and brightest go to academia". I think the engineering community would very well-served if this perception disappeared.

Some graduate students end up going the academic route even if they are not cut out for it, just to prove that they are indeed the cream of the crop. Others, who decide not to go the academic route end up feeling like failures just because they chose a different career.

In my mind, this dogmatic view is very similar to the attitude in China and India - you aren't successful if you are not an engineer or a doctor!

Matt -- I agree that chasing money is one of the less fun aspects of the job. But some of the other points you raise I'd disagree with.

Having talked, for example, to a number of young lawyers, I can't complain about working too hard, or my time not being my own. For the most part, I work hard because I'm excited about the things I work on. And while there's always more to do, for the most part, I can schedule it how I want. So far, it's an unusual day when I don't walk my kids to school or I'm not home for dinner; I don't mind doing some more work after they go to bed.

I may be extrapolating too far here, but since (relatively speaking) I'm an "old man" and you're still a "young Turk" (in the best possible way), I'm suspecting that things will re-balance a bit for you both because you're now a parent and after you get tenure. I'm not saying you won't work hard, or that the job is always perfectly wonderful -- it's not, and you're right to tell people about your experiences -- but I'm betting you'll find your day's structure changing, with a bit more focus on the important things -- both at work, and outside of work.

Matt -- I agree with you in general, and I concur with other colleagues' (e.g., Stefan) about their more optimistic views. I also agree with Michael M (about being old, and looking at other professionals). When I compare my working hours with my 5th year surgery resident daughter, I feel that I never worked hard enough! I figured (not necessarily adopted) that the key is to be able to say "NO" at the right times... Speaking of completing your tenure application and having a baby, congratulations (again), but if you think that you'll find "peace" at work, don't relax too much... It'll get worse...;-)

Matt, I find it odd that Harvard makes its researchers spend so much time scrounging for funding when it's sitting on a - what - $50B foundation?

Too bad the tax payers have to subsidize what you do too. Public institutions could use that funding.

Not that I have no sympathy for you. It is interesting the way thinking for a living looks so easy from the outside. Being a student is the same way. For those who have not tried it, higher education looks like a piece of cake.

Some this, especially wrt money, is pretty far from my experience but I was similarly surprised by one aspect of the job, despite the fact that my father was a history prof.

* In stark contrast to the popular notion of the comfortable tenured academic, being a successful CS faculty member requires a strong entrepreneurial streak. You write this only in terms of raising money but it applies much more to ideas. It is not sufficient to do good work. One also has to establish one's personal "brand" so that one becomes known as a "leading expert in X" for some set of relatively specialized X.

In terms of time, pre-tenure it is difficult to say "no" to potentially career-advancing opportunities. I personally still find it difficult to say "no" but that is not universal. Your time is your own in ways that you just aren't comfortable utilizing at this stage. These days, non-academics are also constantly connected to their work in ways that I did not anticipate; I view my "always on" mode as part of a personal choice, not something that is required for the job.

Lorre - the Harvard endowment is around $25B now but this is not the kind of money that can be directly tapped for research. It is complicated, but one way to look at it is that Harvard is a very large hedge fund with a university attached to it (which gets a tiny fraction of the fund's payout each year).

Paul - one of my future posts will be on branding your research - something both profs and grad students need to learn to do.

(1) I like Stefan's advice on deliberately trying to minimize the scut work.

(2) Size of the department and personality are also factors: e.g. a larger department might have less administrative overhead and more support staff for applying for large grants and whatnot. Also some professors get stretched thin and have a hard time saying no.

(3) You can't directly compare hours worked by in different professions because the nature of the work is so different. e.g. just because a CEO is in meetings for 12 hours a day doesn't mean he is working harder than someone who was writing a proposal for 4 hours and thinking hard about research for 6 hours. That said, obviously many people work hard besides professors.

(4) Good point from Paul about "branding" ... I think a lot of people struggle with that ... I know I do...

Nice post Matt. I understand that this reflects your experience but I thought I would share some of my perspectives as well.

To me, the greatest thing about being a professor is that you get to define your job. If you particularly love teaching, then you can focus your energy there. If you love service, you can make a career of that. You could go down the administrative track, aiming to become Chair, Dean, Provost, etc. And, of course, if you want to perform a lot of research, then there are straightforward ways to focus on research. You do have to choose of course; I actually spend a measurable fraction of my time saying "no" to a variety of requests. I suspect that one problem you have is that you are capable of being great at *all* of the above things. In this case, you have many factions who want your services. You need to pick what aspects of the job make you happiest.

On committees: I cannot say what things are like at Harvard, but most departments I am familiar with limit required "service" or "scut" work to one committee/year. And there is no reason to have any committee take more than 1-2 hours/week, certainly if averaged out over a year. If a faculty member is required to serve on a committee that requires more 100 hours of meetings a year, something is dreadfully wrong.

My solution to the committee problem was to volunteer to serve on (and then run) a committee that I deeply cared about, graduate student admissions. To me, the number one predictor of my happiness is the quality of the people that I work with. And while I get to work with great faculty (like Stefan above :-), the vast majority of my time goes into working with graduate students. Improving the process of selecting, and then recruiting great graduate students was something that I could easily get excited about. At UC San Diego, this is one of the more demanding committees, but I do not believe that it requires anywhere close to 100 hours/year of meeting time. And the outcome is quite tangible both personally and for the department as a whole.

Many people have mentioned the downside of going after funding. I am with Sam where I actually enjoy the grant writing process. Having the chance to think about future directions for research and lay out a reasonable plan that external reviewers can buy into is extremely worthwhile. And getting feedback that some line of work is potentially misguided has been valuable to me on at least one occasion.

For me, the research drives grant proposals. That is to say, if I am doing research that the community is excited about, then a proposal around those themes holds a pretty good chance of receiving funding. That is the beauty of the NSF funding model based around peer review. A mistake I have seen others make is to have grant-writing drive research. This is more of the old DARPA model (and by old I mean the period from ~2000-~2009) where there were deliverables that someone else had defined and was excited about.

I have much more to say on this topic and of course there are downsides to being a professor (there is no perfect career). But it sounds like I should put together a blog post of my own...

I guess since I left they've been making you do more stupid committee work. Maybe you (and your readers) might like hearing about another experience that was similar but different.

I feel lucky that during my untenured years I spent almost all my time with students or preparing to be with students. And I have been very lucky with funding: my success rate at NSF was high, and once or twice some industrial money just fell into my lap.

It also sounds like your dials go to 11 :-) When I had 2 postdocs, 3 grad students, and 1 undergrad, my group was way too big. It felt like I was exceeding my design specs. Afterward I made sure never again to let my group get bigger than 3 or 4 people, so that I would not have to let the grad students have all the fun. There's a big perception in academia that more is better. I've been lucky to work at two places where that wasn't so---and one of them is the same place where you work.

The stuff you write about building your brand is absolutely true. People have to know who you are and what you do. Your science won't speak for itself; you have to speak for it. Writing proposals actually helped me here, because it forced me to think carefully about what I wanted to do and why it was important. But I had the luxury of never having to churn out proposals on a schedule just to keep the wheels moving.

And that's another good point about the startup analogy: your burn rate controls you. One nice thing about postdocs is that when their funding is used up, they can move on. Grad students have to be fed until they finish or until they stop producing. These days I am very, very careful about whom I take on as a PhD student and how many students I take on. It makes a difference.

Now that I am a year post-tenure, I find myself, like your tenured colleagues, working harder than ever. But again I have been lucky to avoid stupid committees and to serve on committees that are making a difference. (It's mostly been good luck and not good management.) On the other hand, I also learned early on (I won't say where) how to get off time-wasting committees as soon as possible.

The good news is that once you get tenure, stress level goes way down, even if workload does go up. It's easier to say no to the deans (which isn't the same as easy). And you have the privilege of being on one of the last Athenian democracies left standing. As Larry Summers found out to his cost, when push comes to shove it's the professors who run the shop. That's a real privilege.

Amin - I've lost count of the number of committees (both formal and informal) I've served on, both within Harvard as well as external things. I'm including all of the service work that professors do in that category. It is more than just formal committees that eat into research time, of course - it includes ad hoc paper and proposal reviews, reviewing PhD and senior theses and design projects, writing recommendation letters, countless trips to DC and to companies, even meeting with the parents of prospective freshmen who visit off-cycle. Being the one guy in my area I can't pawn most of this stuff off to anyone else here, so I get tapped to do a lot (as do all of the other faculty here). I need a clone.

Norman - Thanks for the encouragement. Also keep in mind that the work I do requires not just a big group but a lot of equipment and infrastructure (we are building robotic bees after all). I'd love to downsize but then I couldn't tackle the problems I'm interested in, so it is kind of a Catch 22!

Hi Matt, I just wanted to throw one more thought into the mix--teaching colleges. You contrast industry with research academia, but also state that your favorite part is teaching and mentoring students. I couldn't agree more, which is why I ended up choosing a teaching college. No, I don't run a world-changing research program (though I still do research I am proud of), and yes, I work very hard, but probably 80% of that work is directly dedicated to my students. I spend very little time doing work that I don't love. And I agree with you that the freedom of the job is definitely one of the biggest selling points. I also very much agree with Stefan that the best part of the job is that I get to work with people I really like (students, faculty and staff).

Anyway, for people reading this blog and thinking about what to do next, please don't rule out the idea of a teaching college as a career. It's often overlooked as an option when you're coming from the big research schools.

"At Berkeley, the reigning belief was that the best and brightest students went on to be professors, and the rest went to industry -- and I wanted to be one of those elite."

1. I thought about this quite a bit: if a grad student is really that good, that must mean he/she thinks out of the box, which usually means he/she doesn't blindly assume that the 'current general belief' is right (in this case for his/her future). So, anyone who wants to be in academia just because 'everyone else says so' can't be that good, right? :)

2. I think there's a rather unhealthy cycle: Top universities churns out faculties right out of grad school. These poor blokes don't see the real world, all they know is their tiny worlds filled with fellow academics. Most likely, nobody else outside of the PC committees would ever have heard of them. But that's okay, what matters is that they live in their own little worlds, and pat themselves on their backs.

And these elite scholars teach the next generation. Having never seen the outside world, they perpetuate the idea that 'only the best and brightest' become faculties. Because, isn't that what they are?

Sorry about the harsh words :) That's the good thing about 'single-blinding' blog posts: we can speak our minds.

"... These poor blokes don't see the real world, all they know is their tiny worlds filled with fellow academics."

There is no such thing as the "real world". No matter what you do, you'll get exposed to only a small fraction of the possible experiences on earth.

Universities are indeed the places where most cutting edge scientific research is done. (At least on most branches of science, I don't know if CS systems is an exception.)

So, if the definition of "best and brightest" is the best scientists then yes, generally speaking the best and brightest do become professors (though there are always exceptions). If the definition the people that will make the most money, or the people that will have the most impact on industry/society, then this is not generally true (though again there are exceptions).

"if a grad student is really that good, that must mean he/she thinks out of the box..."

The fact that a grad student has a talent for doing research, doesn't necessarily mean they are any better in making decisions about their own lives than any other person.

I would just like to echo Christine's comments. Having spent over a decade as a post-doctoral researcher, first in industry and then at an R1 university, I have come realize, like you, that CS research in academia is BUSINESS.

I think the deadline-driven research, that our ultra-competitive conferences foster does not especially help and takes our field away from the deliberate and considered progress that other sciences enjoy. Just think about the fads that we have been/are going through: AI, Grids, Clouds, etc. There is something to be said about "slow" science, as in slow food movement. :-)

Hence, I have decided to move to a liberals arts college to focus on teaching. I'm taking a huge pay-cut, but I'm expecting happiness to go up considerably.

Nice post, Matt. What struck me years ago at Dartmouth, and what continues to be true, is how little of what I do as a faculty member was I trained to do in graduate school. In grad school, I learned how to do research (whatever that means), how to write it up, and how to give talks on it. (And I was fortunate to have an advisor who viewed working with students on writing and talk-giving as an integral part of advising.)

I was not trained in any aspect of teaching--how to give lectures, how to grade, how to plan a syllabus, how to design exams, etc. I was not taught how to write grant proposals. I was not taught how to do administrative work.

I was not taught how to deal with students crying in your office. I was not taught how to deal with male students crying in your office. (The very first crying student I had to deal with was a guy. Awk-ward!)

I was not taught how to hear sexual assault cases on judicial panels. (I have had to do this at Dartmouth.)

There's lots of other stuff that I wasn't taught in grad school, but you get the picture.

Matt, your candor is certainly appreciated! Having very recently achieved ABD status, I am suddenly feeling quite fortunate to have relatively few illusions about what I'm getting into as a career academic.

On the funding front, my advisor's strategy is to have students work with him on grantwriting, so over the last three years I have written the bulk of 3 NSF proposals, although my name does not appear on them, of course. Two of the grants were successful, so I have contributed substantially to securing my own funding, as well as another PhD student and a small flock of MS and BS students. My additional rewards include summer funding, travel funding, dissertation data collection funding, and a significantly better pay rate than any of my fellow students, although still quite modest.

More valuable yet is the experience - I'm fairly certain that this will provide a substantial advantage on the job market. If your students are not working on grantwriting with you, it might be a worthwhile strategy to consider.

A thought provoking post. I can certainly relate to the difficulties of balancing management, research, development, teaching, bid writing...

I think some of the answer' to this situation may lie in closer links between academia and business. There is fertile ground, in universities providing consultancy, knowledge transfer, sharing facilities etc. Business links also provide placement opportunities for students. With the right partnerships (and that's the difficult bit), business + academia working together can be win-win.

Hi Matt Not much to add but what you speak of rings true for many of the people I know in academia. Some Universities (such as the one I am moving to) recognize the grant chasing computer scientist often have to do and provide supports to discourage it. More time for considered reflection and active research than committee life and fund raising.

I've had to restrain myself in the past when sitting with our esteemed tenured faculty when they make that comparison with running a startup. It isn't. It's whiffle-ball startup: you won't end up ruined and jobless if you don't bring in a grant, you'll just end up with fewer graduate students and fewer publications.

Google+ Badge

About Me

Matt Welsh is a software engineer at Google, where he works on mobile web performance. He was previously a professor of Computer Science at Harvard University. His research interests include distributed systems and networks.