(Guest Post by Claire Kenyon)
This is the time of year when job candidates are getting ready to send out their applications. I have done this many times over the years. Here are a few suggestions based on my experience.
Candidates want to find a job where they will be successful; department want to hire candidates who will be successful in their job: this is not inherently adversarial; it's just a question of finding the right match.
Cover letter:
1) Don't call a place "College" if it's a "University", and vice-versa. Get the names of committees and of people right. Obvious, yet, it took me a few years to learn this!
2)How to get the reader to look beyond the cover letter? Catch their attention. Give a specific reason why you are interested in that place; preferably personal (something that says something about you, and that few other candidates are likely to say.) "I am interested in the computer science department at University Lambda because of its unique research interest in Reducing the Number of Greek Symbols in Analysis of Stuff, and its joint project with the Humanities department on that subject. My publications have a lot of greek symbols in them (see [1,2,3,4,5] for example), and I would be very interested in applying the lambda methodology to my work."
3) How to get the reader to forward your application to the right person? Give specific names. For example:
Professor Big Shot, who I met during the 2007 Symposium on Theory of Unreasonable Protocols for Integer Data (see [3]), encouraged me to apply.That context will help Big Shot place you in their memory when they get the application.
4) Be self-consistent. Do not tell UC Big
I just love the idea of public service in the rich environment of a large university
and simultaneously tell Happy University
I love the idea of mentoring a small group of select students in the focused environment of a small high-quality university.
The reasons are that this may become known (we do talk to one another) and would cost you your credibility; that you won't be able to follow through by arguing convincingly both ways; and that such blatant mis-representation of yourself makes it more difficult to find the right match.
5) Read the ad, and address obvious issues upfront.
You said you're looking to hire a researcher in human-computer interaction using ergonomic mouse pads, and my area, cryptanalysis of public-key cryptosystems using elliptic curves, may at first sight look somewhat remote; however there are surprising connections that I intend to reveal in my future research: elliptic curves
Claire, with all due respect, the cover letter seems like the least important part of your application (even from just the perspective of getting your foot in the door). Many schools do not even look at cover letters (or perhaps an administrative person looks to see what bin your application should go in). At my university, our online system doesn't even allow me to see the cover letter.
ReplyDeleteI suspect that it's impossible to get a job at a university unless (1) somebody on the faculty already knows who you are, or (2) somebody on the faculty knows and has great respect for one of your letter writers.
(My comments refer to a top 25 department like Brown.)
Thank you, Claire. This is good advice for the cover letter.
ReplyDeleteOne minor addition (in the spirt of this blog :):
3.b. Run a grammar check:
Professor Big Shot, *whom* I met during the 2007 Symposium on Theory of Unreasonable Protocols for Integer Data (see [3]), encouraged me to apply.
Anon#1:
ReplyDeleteYou're absolutely right. I've served several years on the hiring committee in my department. I have *never* heard *any* discussion about an applicant's cover letter, good or bad. Nobody gets an interview unless either someone on the faculty (and preferably on the committee) already knows and loves their work, or they get a glowing letter from God.
Anon#2: Except for "whom". Applications with "whom" in the cover letter get tossed in the trash.
Sorry, Claire, but I disagree with much of your advice.
ReplyDelete1) Yes, don't make silly mistakes in your cover letter. On the other hand, I got a job offer at university XXX even though my cover letter contained the typo "I really want to work at University YYY". I agree with the other posters that cover letters barely get looked at.
2) Most people want to go to the best place they can, perhaps taking into account geographical preferences. I think it would sound odd to say that in a cover letter. If natural collaborations are of interest, they may be good to talk about, but I don't view the potential for such collaborations
as essential to getting an offer.
3) I don't agree with dropping names. I guess if you happen to know someone well, this may work. But if you met someone at a conference who gave you generic advice to apply, don't include that. And don't let the fact that you don't know anyone discourage you from applying.
4) Instead of "be consistent", how about "be truthful".
5) It is definitely good to read the ad, but again don't let this put you off. If you are in a non-targeted area, don't expect an interview but you can still apply without trying to bend over backwards to fit what you do into the areas of interest.
I think Anon 1 and 3 are a bit pessimistic. You do NOT need to know someone on the faculty of the school you are applying to. And while you DO need strong recommendations from people who are reasonable, the letters do NOT need to be from the absolute best people in the field.
The cover letter debate (pro or con) will continue forever. The situation seems to be:
ReplyDelete(1) At many top research universities, nobody will even bother to read the letter. If they do read it, all they want is to learn your research area and career status (junior or senior) even more quickly than they could from your CV. If you clutter up the cover letter, you won't help your chances and might slightly hurt them.
(2) At many liberal arts colleges, they expect the letter to make a compelling case for why you want to teach there and consider it a good fit. If you don't explain this in detail, they will not consider your application seriously.
(3) Outside of top-20 research universities and self-proclaimed liberal arts colleges, nobody knows which category any given school falls into. It's usually not hard to guess, but if you propose a general rule, then the next commenter will give you a counterexample. This may also change from year to year, depending on who is on the committee.
The one useful thing I think fits well in a cover letter is addressing a specific need the school has (even if the need is unstated). For example, schools in the middle of nowhere are often concerned that nobody wants to live there. If you can offer a good reason why you do, then you'll become a more serious candidate. Don't do this unless it is true (of course) and useful(don't bother to tell Berkeley how much you like the bay area).
If I remember correctly, all I said in my cover letter is that I am so and so, with such and such interest and am aplying for such and such job. I got interview calls only from the third of the universities I applied to.
ReplyDeleteI felt a university might be running many different job searches, e.g., non tenure track or senior faculty jobs. Often different postings gave different post-box addresses still a cover letter helps minimize the errors for your application to go in the wrong pile.
In today's world of electronic application even this is not needed. The chance of your application to go in the wrong job folder is small.
My strongest advice is to send a personal email to an appropriate faculty member telling him/her that you have applied in his/her department. This is not necessary but may sometimes help, as I realized after my job search was done that faculty members who could be interested in me having in their department did not even know that I actually applied there. Faculty members from two different universities asked on conferences' lunch tables, how come I did not send my application to their universities.
Well, I did. The package perhaps was not routed to them. Now I am on the other side of the line. I always appreciate if applicants let our group members know that they have applied or will be applying for a job in our group.
And if you are not well-connected, you can always get a real job (nobody except for universities evaluates scientists based on who their friends are).
ReplyDeleteThis comment has been removed by the author.
ReplyDeleteGASARCH:
ReplyDeleteBush is not a scientist.
And if you are not well-connected, you can always get a real job (nobody except for universities evaluates scientists based on who their friends are).
ReplyDeleteYes, in the real world social connections are meaningless. That's why socially inept people tend to flourish in business, while in academia everybody is remarkably well-adjusted.
I think Anonymous #7 was joking. Everybody knows that one's prospects for getting a real-world job based on her skills in analyzing PCP constructions is essentially zero. :)
ReplyDeleteI am interviewing 1-2 candidates a day for "real-world" jobs now. I never ask for references. The cv is only a guidline for the interview. The #1 factor is technical and problem-solving skills, which we evaluate ourselves by spending hours with each candidate. I can't imagine rejecting a candidate who is technically good for lack of pedigree or connections, bad table manners or any of the other things I saw mentioned in advice for academic job searches over the years.
ReplyDeleteWhat is striking is how many unemployable candidates are coming out of CS phd programs -- people who can't program or answer simple algorithm design questions.
Anonymous #12:
ReplyDeleteYou're deluded, and probably doing your company more harm than good in your interviewing.
You can't really test problem-solving skills in an interview, even one that lasts all day. (Well, you can check for someone's ability to solve a problem in a 1/2-1 hour interview, but odds are that is not actually the skill you want to test for. Or perhaps it is, but then perhaps you might not be the sort of company the candidate is interested in working for.) That's why I give assignments, and not just a 3-hour final exam. (I do give both, and am explicit with students that these test different skills.)
On the theory side, many people who come out of PhD programs can't necessarily code very well -- that's not what they've been trained to do. Most coding, however, is not that difficult, and they'll pick it up quickly. But they won't do well on your interview. I'm glad to hear you're at least asking algorithm design questions -- a more reasonable thing to do. Although again, my guess in undergrads may do better than many grads on such interview tests. They've done more basic algorithms more recently, and they've been coding more. (Most interviews ask people to sketch out the code.) It doesn't mean they'd be smarter or better employees.
Letters of recommendation might not be perfect, either, but to ignore them is a really poor idea.
perhaps the anonymous industrial interviewer is the guy that talked to me... I was asked to solve what sounded like a neat streaming algorithms problem, but after a while I couldn't see how to do it.
ReplyDeleteIt turned out that the interviewer wanted to make a whole slew of vague, unsubstantiated assumptions about the distribution of the input.
Anon 12 replying to Michael Mitzenmacher:
ReplyDeleteWhen I was still in academia I had the same attitude you express in your post. You will be surprised, perhaps as stunned as I was, about the kind of mistakes graduates as well as phds can make (and insist on). Just from the last week: A recursive computation of fibonacci numbers takes linear time, an algorithm that checks n^2 cases takes O(n log n) time.
About programming: I am talking about not being able to write pseudo-code for reversing a string, or for printing the sum of every consecutive sub-sequence of an array in whatever order you want. A candidate who cannot do that, will not be hired for a job that requires even a minimum amount of non-production programming, such as the positions I interview for.
You can argue with me (or think that you are smarter and I am short-sighted; I know your line by heart because I was a know-it-all academic not long ago). Or, you can listen to what I have to say and think about whether you are doing your students a good service by preparing them *only* for academic jobs that most of them won't get.
Anon 14:
ReplyDeleteI was not the person you talked to, but I also have some questions with which to check mathematical / computation skills and some with which to check how the candidate attacks real-world problems, those that do not have clean solutions like the homework problems she saw in school.
On the job you need to solve the problems you are facing - not like in academia where you can change the problem if it's too hard.
Usually such problems do not have a "correct answer". You can, e.g., make assumptions or give a solution that will work in many cases. It's more about the process than the answer.
One important piece of advice to theorists in the computer science job market: if you can teach applied computer science courses, say so! (or say which ones). If you can back that up with TA experience, even better.
ReplyDeleteThis is especially important when applying to small departments: theory faculty might want more colleagues, but will only succeed if they can show the new hire will be able to teach a wider range of CS courses that the department might need covered.
When I was still in academia I had the same attitude you express in your post. You will be surprised, perhaps as stunned as I was, about the kind of mistakes graduates as well as phds can make (and insist on).
ReplyDeleteI teach them, and you think I'd be surprised about what kinds of mistakes they make? Not at all. I've seen mistakes you couldn't believe.
But that's part of my point. In a short interview, anyone can make a surprising, stupid mistake. And you're apparently choosing to ignore their entire educational history because they've made a mistake in front of you.
Just from the last week: A recursive computation of fibonacci numbers takes linear time, an algorithm that checks n^2 cases takes O(n log n) time.
So of course I teach in lecture #2 of my undergraduate class how to compute Fibonacci numbers, including the fast Matrix-mult Fibonacci way. (BTW, what do you mean, exactly, about time? The nth Fibonnaci number has size exponential in n, so it does take at least linear time (in n) to write down? Or am I just making a stupid mistake? I'm an academic, after all. Though perhaps my mistake is not one of mind, but in understanding what you expect the answer to be -- a common interview problem when you're giving out quizzes.)
But again, your example proves my point. Anyone who's lucky enough to have seen your question in a class will get it quickly enough. So are you testing how well I teach to your interview? I could do that if you like, but I don't think it's best for the students.
About programming: I am talking about not being able to write pseudo-code for reversing a string, or for printing the sum of every consecutive sub-sequence of an array in whatever order you want. A candidate who cannot do that, will not be hired for a job that requires even a minimum amount of non-production programming, such as the positions I interview for.
That's your choice. Personally I find having to write pseudo-code on the board on the fly for an interview annoying. If you're concerned about how I code, which is reasonable, ask for a code sample. If you're concerned about how I'll code for your company 3 months from now, ask about my willingness to code, my experience coding, and figure out how smart I am at coding-related activities.
You can argue with me (or think that you are smarter and I am short-sighted; I know your line by heart because I was a know-it-all academic not long ago). Or, you can listen to what I have to say and think about whether you are doing your students a good service by preparing them *only* for academic jobs that most of them won't get.
I do agree with you, in that in my undergraduate algorithms class, I do try to prepare my students for something other than academic jobs. Remember, I'm the one on these blogs that actually makes students program in their undergrad algorithms class, and teaches them heuristics that we can't prove things about but work well in practice, and I still find what you say about how to interview completely misguided.
For graduate students, I would disagree, in that most graduate students want academic or research-related jobs. Of course they should know how to code, but from my experience, successful graduate students can learn a new programming language in about a week, and do neat things in it within a month. Testing whether they can code what's in your head on a given day is, in my opinion, a poor choice.
I'd like to agree strongly with Anonymous #17. Too often I see teaching statements that say "I can teach one or two of your theory courses but what I really want to do is introduce a specialized course in -my narrow specialty-".
ReplyDeleteYou can distinguish courses that you are willing and qualified to teach at the undergraduate level from those you would feel capable of teaching at the graduate level.
Michael:
ReplyDeleteThe interview, unlike a homework assignment, is an interactive process. After you give me the textbook answer I will twist the problem, present variations, ask you why you replied the way you did, discuss, and so on. With a few hours of that, "I saw this before" cannot take you very far.
One of the bad signs is when a candidate does not know something he thinks he knows or claims to know or has taken courses about. We *are* assuming that basic programming is on the CS curriculum of most universities (to the level or writing a 5-line piece of pseudo-code, as annoying as that is. How fascinating do you think this is for the interviewer?)
I appreciate your suggestion to *ask* the candidate whether they can program instead of annoying them by asking them to demonstrate it with 5 lines of PSEUDO-CODE. But I think the reply would be even less useful than a letter of reference from their favorite professor (which at least proves that they are talented in either science, politics or arse-licking).
The debate between Michael and anonymous is sort of besides the point, given that in fact many companies do have interviews exactly as anonymous says. This includes wall street firms, consulting firms, and even google and microsoft, places that most CS grad students are happy to go to. (On the other hand, this is typically in addition to recommendation letters.)
ReplyDeleteYou can argue about whether this is a good thing or not from a philosophical point of view. But Michael, don't you think companies would change their interviewing strategy if it was not yielding successful results? Also, I believe such interviews do test something, in that a candidate who does well on these questions is demonstrating intelligence, logical reasoning, communication skills, and an ability to think "on her feet". There may be false negatives, and there may be qualities such an interview does not test, but I would argue that such an interview is a better indicator of quality than recommendation letters (have you seen how inflated such letters are these days?).
For graduate students, I would disagree, in that most graduate students want academic or research-related jobs.
ReplyDeleteEven if you are right that they want them (and that's quite an if), they are not likely to get them. When people go into a phd program they think it will improve their employability, and universities don't seem to care that this is not the case for the majority of the phds they produce. You certainly don't seem to care.
I believe that IQ tests are illegal in job interviews in the US. The "think on your feet" type questions are a way of getting around US discrimination law.
ReplyDeleteJonathan --
ReplyDeleteI understand that people use these "quiz tests" as part of the interview process. I don't object to that, just like I don't object to my own using a 3-hour final exam to evaluate students. It does test something. The anonymous involved said that they actively ignored the CV and letters of recommendation, which I think most employers do not do. Just information-theoretically, it's wrong, unless you believe that information is uncorrelated to performance. But also, short-answer interviews test one type of skill (the ability to perform well on short-answer interviews). I agree that this skill is also correlated to performance, and could be used as an input. I was objecting to the idea that it was the sole input, or even a better input than CV/letters.
Personally, I've found that looking at a CV and letters I can get a pretty good idea of a candidate's quality, and I can tell which letters are overblown and which are reasonable. I do think employers should ignore uninformative letters, of course. So, Jonathan, I think we agree up to the point where you suggest these one-day on-your-feet interviews are somehow more informative than a record of the person's entire past history.
A point nobody has brought up, which I will bring up, although it may appear sexist, is that I have heard anecdotally from multiple sources that this short-question-code-on-the-board type interview is actually biased against women. Certainly I have known talented women who would not perform well on such an interview who would be considered extremely compelling candidates. (Apparently, many women are less aggressiveness and appear according to social cues to be less confident, and these quiz-interviews often test aggressiveness and the willingness to act confident as much as logical reasoning, etc. Some women have actually told me that they like to actually think about questions carefully before answering, which these interviews are not designed for. I imagine some highly qualified men may feel the same, of course.)
Recent PhD -- I care a lot about the employability of my students, undergraduate and graduate. Heck, one of my courses is designed to be a feeder straight into Google, and seems to be reasonably successful. (I will admit that it does help that Harvard students are, on the whole, quite smart without my help.) I take pains when graduate students come in to let them know about the job market, prospects, skills they should have, etc. Why would you suggest otherwise? Just because I suggest that a day mixed of brain-teasers and writing pseudo-code on the board might not be the best indicator of talent?
A point nobody has brought up, which I will bring up, although it may appear sexist, is that I have heard anecdotally from multiple sources that this short-question-code-on-the-board type interview is actually biased against women. Certainly I have known talented women who would not perform well on such an interview who would be considered extremely compelling candidates.
ReplyDeleteI think the key issue is this: would someone who is better at thinking on his/her feet also be better when thinking in a more relaxed setting?
If so, then such interviews make sense, although they may violate US anti-discrimination law.
Michael:
ReplyDeleteI did not say the CV is ignored. Of course we look at it, and that is what gets people invited to interviews (letters we never ask for, and we are not the only ones - when I was job hunting I was never asked for letters, and I interviewed at the types of places mentioned above by other people).
The CV is also used to know what to ask people about - we ask them about what they should know.
But I saw some very bad candidates with very good looking CVs. This includes CS phds who could not answer a single question posed to them, whether programming, math, statistics, algorithms, anything. So the CV is there to guide the interview, but it cannot make up for a bad interview.
As for your fear of false negatives, all I can say is that we are so keen to find good people, that we continue the process as long as we think there might be a chance for a fit. My firm is notorious for endless interviewing for this reason precisely.
On the gender issue, I think the academic hiring process is much more discriminating against women who are less likely to forge informal relationships with male professors and therefore are not as well connected.
Anon 23:
Is it legal to admit students based on SAT and GRE exams? If so, is it later legal to accept them for jobs based on where they studied? And, do algorithm design and math questions count as "IQ-type" questions anyway?
Is it legal to admit students based on SAT and GRE exams? If so, is it later legal to accept them for jobs based on where they studied?
ReplyDeleteI didn't propose this law nor am I a lawyer. Presumably you have looked into this issue already?
And, do algorithm design and math questions count as "IQ-type" questions anyway?
Definitely.
But Michael, don't you think companies would change their interviewing strategy if it was not yielding successful results?
ReplyDeleteYou are giving companies far too much credit. Think on your feet questions work very well for undergrads, which is a largely homogenous group of people with almost no record to speak of. On the other hand I know of a several companies that apply such questions throughout the entire corporate hierarchy, including hiring of senior research and managerial positions. These are people with extensive records of their skill set which futhermore are not highly correlated to IQ (being a good manager and having a high IQ are almost negatively correlated :-).
Now consider this from the perspective of any one of these companies: 90%+ of the hires are undergrads, so the feedback data they get from the field is unbelievably good: 90% of their hires are properly selected by this method. Wow!
The discussion seems to go off-topic. I'd love to hear more about academic job hunting things...
ReplyDeleteI'd love to hear more about academic job hunting things...
ReplyDeleteDo you have any specific questions?
Michael, I like your information theoretic argument. First, at least at MSR we do not ignore any information. We carefully look at the letters and sometimes correlate with our past experience. In a CV, I myself look at both the consistency and the exceptions. I look for a consistently good performance and also give a lot of weight to exceptional uniquenesses. For an example, won prize for this*, lead this* group, or thought of this*.
ReplyDeleteWe also try to be informed ourselves. We keep in mind good potential candidates we meet at conferences etc. Or good words somebody might have said about somebody.
But as the anonymous said, we give a lot of weightage to our interviews. CV, letters and our general knowledge about you as a candidate are the most important indicators to get you an interview, but once you are given the interview then the interview becomes the primary thing (other variables remain in the equation though).
Note that, the interview is not a single interview. It is a set of multiple interviews done by multiple people and also include the informal discussions at the formal dinner table. So we do get a diversity in the feedback. It has happened that we almost selected a candidate when we detected some fundamental shortcoming in the candidate's understanding of the subject at the dinner table. It has happened with PhD students from top schools. Of course the reverse has also happened.
Even though entropy function is submodular, but its implication such as the confidence in the information and contrast in the information gathered can be supermodular.
Hi Kamal. I'm familiar with the MSR hiring method, in that I think it's similar to what we did at Digital SRC back when I was there. However, please feel free to correct me below.
ReplyDeleteThe interview process, as I recall, is a lot less "quiz-like" in that atmosphere. Some people like to give brain-puzzles and see how the candidate reacts, and again, that's useful information. But other interviewers test depth in a person's field. All interviewers try to determine how the person might contribute to their research/the research of the lab (including leading new projects). That is, different aspects of the candidate are being probed to try to get more information than the letters and CV might themselves provide. And if there remains doubt, people often try to gather more information (by contacting letter-writers directly, for instance).
Of course an interview is important, and can give useful information. Why, we even use interviews in academia!
So Kamal, I have no beef with you and the MSR interview process, which I understand works just fine! What I was objecting to, at the beginning, were the following points from Anonymous 12:
The #1 factor is technical and problem-solving skills, which we evaluate ourselves by spending hours with each candidate.
I doubt at MSR you generally presume to completely judge a candidate's "technical proficiency" in one interview day. Particularly in the case of graduate students and recent PhDs, this seems a bad idea. It's not so bad an idea for hiring undergrads, as anonymous a few above suggests, but surely other information would be useful as well.
What is striking is how many unemployable candidates are coming out of CS phd programs -- people who can't program or answer simple algorithm design questions.
Again, I'm skeptical that (at least for people coming out of theory) many recent PhDs can code on the fly on the board. Personally, I code sufficiently rarely and in particular don't do so few 1st year programming exercises that I might have trouble reversing a link list on the board on the fly quickly, though about 15 years ago that sort of interview question would take about 10 seconds. I think I'd be more valuable to a company now (though I suppose not if they just wanted me to code, but that's because I don't just want to code).
I never ask for references.
...letters we never ask for...
But I think the reply would be even less useful than a letter of reference from their favorite professor (which at least proves that they are talented in either science, politics or arse-licking).
I personally spend a lot of time preparing letters for students, and I try to properly explain their strengths and weaknesses. My interpretation of the above comments was that letters were deemed valueless, and in particular of substantially less value than the interview. Which I disagree with. (In academia -- and I would guess MSR -- if the letters and the interview seem wildly disparate, we at least consider the possibility our interview was off somehow.)
Could you anonymous people let me know what companies you work for where you don't care about letters? It will save me plenty of time to not send them to you any further...
I also thought it was funny that anonymous said they look at CVs but not letters. Of course CVs are overstated -- the students are writing those themselves! Letters give at least semi-independent information; of course, it takes some effort to learn how to gauge individual letters.
Finally, recent phd really just ticked me off with his/her slander. But hey, that's the anonymous comments for you....
Staying with the theme of industry hiring approaches, I urge anyone interested in getting such a job to read Joel Spolsky's discussion.
ReplyDeleteJoel is writing for the hirer, not the applicant, but the advice seems very useful for all (it also should have relevance in academic hiring, but I doubt whether it really does).
A very interesting discussion. Just to add my bit: at AT&T, we used the "MSR/DEC/..." style interviews, with multiple people probing the candidate during the course of a day or two.
ReplyDeleteInternally, we had specific goals: for example, it was important at our lab that whoever we hired had enough breadth of interests to collaborate widely among groups, and by including a diverse interviewer panel, it was fairly easy to detect people who couldn't engage with other people's work, grasp quickly what the key questions were, and make useful suggestions. These are still very smart people, but we felt (rightly or wrongly) that without other meritorious features they wouldn't fit in well or add value.
The thing is though, the same argument that Michael is making about Anon's interview technique (which I'll call the MS/Google method) could be made about this one: it's too hard in 30 minutes to assess the essence of a person, they might be tired by the end of the day, etc etc.
However, we do it anyway, because it does tend to filter out people who don't have the qualities we want, and if there are false negatives, well so be it.
Joel Spolsky (a well known software developer and blogger, for those who don't know) often argues that the cost of hiring the wrong person is MUCH higher than the cost of losing the right person: his argument being that people (and jobs) are sticky, and it's hard to undo mistakes, but that there's a reasonably large volume of smart people above a threshold.
I suspect the reason the MS/Google technique works is because of this:
a) in a fast moving corporate environment, you really don't want people who'll slow you down
b) if your job involves lots and lots of coding, you should have basic tools and tricks at your command (just like I'd be vaguely surprised if someone couldn't even hazard a chernoff bound when scribbling on my board)
c) The fraction of people who are good and yet fail at this test is small, and is worth the risk.
I wouldn't use this model for a research lab, and from the sound of it, MSR doesn't, and neither does AT&T or Lucent (and DEC apparently). But at places like Google/MS, where the "research element" is of a fundamentally different nature, maybe this does work.
Personally though, I do speculate as to how much of this is peer pressure: MS was legendary for its programmers and so Google started doing the same thing, and so on...
Michael:
ReplyDeleteI doubt at MSR you generally presume to completely judge a candidate's "technical proficiency" in one interview day.
We very rarely hire someone after just one day.
Some people like to give brain-puzzles and see how the candidate reacts, and again, that's useful information.
I don't ask quiz questions. I ask mathematical, algorithmic and programming questions of the type the candidate will need to solve on a regular basis on the job.
Could you anonymous people let me know what companies you work for where you don't care about letters? It will save me plenty of time to not send them to you any further...
There is no need - we never waste your time by asking for them.
I also thought it was funny that anonymous said they look at CVs but not letters. Of course CVs are overstated ..
Which is why we interview the candidate and ask her to demonstrate the skills she claims on her CV to have. I think it makes sense to hold people to what they say about their skills, and not to what their professors say about their skills.
Anon 23/27:
It is legal to ask questions that relate to a person's fit for the job she is being interviewed for. For a job that requires algorithm design and programming, you can ask relevant questions. In fact, these are almost the only questions you *can* ask.
Off to interview another top-U grad student. I'll continue later if I won't feel that I am repeating myself.
Michael, one more thing:
ReplyDeleteOf course graduate students are asked and probed about their research. On that, they are expected to show expertise at a much higher level than on other fields. The way they present their research to a non-expert is also important.
Why would you suggest otherwise? Just because I suggest that a day mixed of brain-teasers and writing pseudo-code on the board might not be the best indicator of talent?
ReplyDeleteNo, because you disagreed that grad students should not be prepared only for academic jobs (comment 18).
Think of hiring a pilot. If he/she is just out of flying school, you might ask a few simple general questions (the equivalent of brain teasers) to confirm that they were awake in class (e.g. what to do if the landing gear does not come down on its own). If, on the other hand, you are hiring an experienced pilot you get them straight into a flight simulator and throw them a real-life-type curve ball situation which requires many advanced skills to solve, including at some point deploying the landing gear manually.
ReplyDeleteFrom what I hear Google and other such companies never get past asking the "deploy the landing gear manually" style question regardless of the target position or presumed skill set of the candidate. The end result is that their junior set of people are top notch, but their seniors are a lot more spotty, as they've been selected under the wrong parameters, that is ability to solve minor puzzles rather than ability to think through problems which are orders of magnitude larger and cannot be solved by one person alone. Some people can do both, and those are the good hires, on the other hand many are good at the former but rather bad at long term thinking (think great sprinters but terrible endurance runners) and they seem to have plenty of those too.
Michael,
ReplyDeleteYou are not arguing about quiz-style interview questions. You are arguing to preserve the importance of your letters of reference and hence yourself.
You lose a lot of your power if your students know that you don't have total control of their destiny, don't you?
I urge anyone interested in getting such a job to read Joel Spolsky's discussion.
ReplyDeleteJoel is writing for the hirer, not the applicant
Quote from Spolsky:
Finally, avoid brain teaser questions like the one where you have to arrange 6 equal length sticks to make exactly 4 identical perfect triangles. Or anything involving pirates, marbles, and secret codes. Most of these are “Aha!” questions—the kind of question where either you know the answer or you don’t. With these questions knowing the answer just means you heard that brain teaser before. So as an interviewer, you don’t get any information about “smart/get things done” by figuring out if they happen to make a particular mental leap.
In that paragraph he echoes what Michael is saying.
It is legal to ask questions that relate to a person's fit for the job she is being interviewed for. For a job that requires algorithm design and programming, you can ask relevant questions. In fact, these are almost the only questions you *can* ask.
ReplyDeleteYou know what I mean. Some algorithm design and programming questions test knowledge while others test IQ. It is the latter that might be illegal as they essentially act as an IQ test.
BTW, you may find this interesting:
http://chicagoboyz.net/archives/2147.html
The question from anon 39 says a lot more about himself than about Michael.
ReplyDeletewritter of comment 40:
ReplyDeleteYou (and probably michael too) should go and get interviewed before you write nonsense on blogs. Who asks "a-ha!" quiz-questions anymore? People ask math and algorithms questions that anyone who studied CS should be able to solve.
I wasn't going to comment again -- I have to say, for the most part, I think the discussion has reached the happy medium it should -- but student from comment 39 made me laugh so hard I just had to.
ReplyDeleteYou lose a lot of your power if your students know that you don't have total control of their destiny, don't you?
Anyway, on the off chance you were serious...
Student: I love NOT writing letters of recommendation. Writing letters for students takes away time from research (and teaching, and other important tasks, and unimportant tasks, like blogging). I write letters because I consider it an obligation that goes with my job, and I take it quite seriously. So I'm quite serious when I state I would like to know what companies don't take them seriously, as then I can save myself time and effort.
You (and probably michael too) should go and get interviewed before you write nonsense on blogs.
ReplyDeleteI have, many times, on both sides of the table (hirer and potential employer), in industry and in academia.
People ask math and algorithms questions that anyone who studied CS should be able to solve.
ReplyDeleteSure they do, no one questions that. What we are arguing about is if they give you real insight into the candidate's ability.
Something like a linear time fibonacci suggested earlier is more of an aha-style question, even though on its face is a "programming question". A much better question is to give a piece of code that can be optimized in ten different ways. Good programmers will find at least six or seven of them, even if they don't have the Aha insight for the other three. Bad programmers will only see one or two.
By the same token, it would be ridiculous to ask this question to Michael if you were looking to hire a tornado-codes expert, yet you'll find companies out there who do.
Something that puzzles me about the corporate interview puzzle question ("Write a storage allocator in FORTH while rubbing your stomach counterclockwise"): what do such questions in an interview uncover, that a reasonably good standardized test would not? Why not simply have an application component that involves such an examination, either an existing one, or one composed by or for your company? Wouldn't that be more consistent and reliable, and less humiliating?
ReplyDeleteFor at least some candidates, isn't the interview just as much about selling the job to them, as it is about testing if those ivory-tower, namby-pamby, hoity-toity "professors" with their "textbooks" and their fancy-pants "book learning" have "taught" them anything really *useful* out here in the *real world*, where the *rubber meets the road* and the *shit hits the fan*? Is it really the case, as implied by Spolsky's advice, that there is a vast sea of really good people just begging to work at your company?
From the discussion one can get the impression that the only purpose of the job interview is for the employer to test the abilities of the candidate. We must remember that in many cases the candidate is also testing the employer. With this in mind, asking clever puzzles and 'Ah-a' questions helps to make the impression (especially for junior candidates) that the job itself will involve solving such puzzles, and thus makes the job more 'cool' and attractive to the candidate - I was explicitly told by a guy who interviews people for a software/algorithms company that this is a tactic that they use in order to attract good candidates.
ReplyDeleteAnother issue - asking 'Aha' questions has a value even if the candidate does know the puzzle and its answer - it means that he is more likely to many such puzzles and be interested generally in solving math/cs riddles, which is a characteristic positively correlated with mathematical abilities
With this in mind, asking clever puzzles and 'Ah-a' questions helps to make the impression (especially for junior candidates) that the job itself will involve solving such puzzles, and thus makes the job more 'cool' and attractive to the candidate - I was explicitly told by a guy who interviews people for a software/algorithms company that this is a tactic that they use in order to attract good candidates.
ReplyDeleteSo this is a form of deception? Or is the job really like that most of the time?
Michael, I would have to say that for your average job (not academia and not research), a technical interview provides significantly more information than a CV and letters. A CV gives you very little useful info: a list of published papers means essentially nothing to someone who doesn't understand the titles or the relative rankings of conferences. Someone who is very bright, but not exceptional, may not have won many awards. Someone who worked in a hard area may not have published very much (and, of course, the converse too).
ReplyDeleteLetters, too, are mainly useful at the extremes: "this candidate is exceptional". Every one else falls in the middle. As an advisor, you may know the quality of your student's papers but do you really have a good sense for how smart they are, or how well they program?
On another note, interviews in academia, in my experience, are mostly a waste of time (I've been on both sides of them). Yes, the interviewer might learn something about the candidate (they have a good personality?), but typically this is irrelevant when the final hiring decision is made.
Let's return this thread to its original intent (and probably of more interest to the audience of this blog)
ReplyDeletere: anon 50
When have you seen an academic interview matter? What can a candidate do to come out of the interview a stronger candidate than before the interview? Can a savvy candidate use an interview to ease concerns that a person looking only at the CV might have?
I have been interviewing lately and have found that I have a difficult time working through problems that are new to me in such a high stress position. I find that my brain processes seem to be very different from those that occur when I'm working through a problem at work and I seem to have a difficult time confidently making mental leaps without questioning my reasoning very carefully. Of course, by the time I'm finished questioning my reasoning the interviewer has been twiddling his/her thumbs and has interrupted my thought processes five times trying to give me 'hints' which I ignore because they're distracting me from my current mental road. It is very frustrating to me at times because I have been told many times by many people (managers/professors) that I am an especially valuable employee/researcher; however, this is not going to come through in an interview process until I become more adept at this process. So perhaps I can be chalked up as a false negative but who's to say that I would not be a more valuable employee in the end than someone who passes the test? Not to mention that I have a personal stake in this because I want challenging jobs but have a difficult time 'thinking on my feet' amongst all of that stress (I think on my feet just fine in my current job where my next very important turn in life is not entirely hinged upon my answer). Just another perspective on the matter.
ReplyDeleteJust to add to comment 52, I have an extremely hard time sleeping the night before an interview because, again, I know that my performance 'tomorrow' will dictate whether or not my life moves in a positive direction. These are just biological traits of mine that I have been trying to overcome but meanwhile I have been rejected from various positions and the corresponding companies have been missing out on my skills and ability to contribute. Let's just assume, for the sake of argument, that I do have unique skills and I am particularly valuable as an employee (of course you have no idea who I am, but if you assume that I am run-of-the-mill then this argument becomes uninteresting).
ReplyDeleteIn response to comment number 43:
ReplyDelete> Who asks "a-ha!" quiz-questions anymore?
Most Wall Street firms still ask questions about pirates and marbles.
Anon. 52 & 53: Don't be afraid to work on the problem aloud, they like that. Also, if you blank during the interview but realize the answer that night, drop them an email. It's always good to follow up anyway.
ReplyDeleteChuck:
ReplyDeleteI interview for a wall st. firm. The only circumstance in which I would resort to stupid questions would be if the candidate doesn't know anything about what they claim to know about. In that case, I might politely pass the time with pirates and dungeons instead of walking out. Otherwise, if the candidate studies math I would ask which area of math they know about and ask questions about that. And so on.
Anonymous 55:
It is good to think aloud, as long as you are focused on the problem you are given. Don't babble about everything the problem reminds you of.
I think it is unfortunate that this discussion derailed into a debate of pro-and-con pirate questions. Some important issues were raised and brushed aside, namely the question of whether grad schools are doing a good job in improving (or at least preserving) the employability of their students.
Michael's position seems to be that (1) he knows better than us what we need and how to find what we need, and (2) there is an inverse correlation between a candidate's basic skills and overall quality.
Can anyone come up with anything more useful?
You misstate my position completely, which is:
ReplyDelete1) I probably have very useful information you should consider -- information of comparable value to an interview. (How comparable depends on the interview.)
2) Testing "basic skills" in a time-sensitive, pressured situation is not as simple as it sounds, and unless you're an expert (and to be clear, I'm not an expert in interviewing) you're probably not testing the basic skills you're really after.
I think if you read my comments again, you'll see that those are my points. You seem to be reading what others say my points are.
Michael, IMHO, you are mostly right. But I think the other anonymous has a pinch of correctness too. I have an impression that Microsoft (non-research) is moving/has moved away from asking tricky questions. Still letters have much less weightage than CV. CV is really the primary indicator.
ReplyDeleteThere are a couple of things you need to realize in your arguments. First, the number of available jobs matter a lot. Microsoft hires tens of employees a day. In tech industry in general there are more jobs than the number of qualified candidates. You can estimate this by the enormous salaries techies get compare to other professionals. So the employers do not need a precision in their evaluation. They need to identify candidates above a very low quality bars. Say whether a candidate is in the top 25% of the class (at large not only at a specific school). The letters mostly help distinguish between top 5% and top 10% of the class. I have never seen a letter distinguishing a candidate from bottom 75%, may be because in research we look for the toppers of the class. CV is used for this distinction. For an example, I have heard that Google has an explicit cut-off on GPA.
The second point you are missing is that performing well on a job also include performing well in high pressure situations. Sales jobs are mostly high pressure. Approaching deadlines for engineers is also a high pressure situation. In industry there are situations in which people could be called from their death bed. An open exploit in Windows which is being currently targetted, a data server not behaving properly and causing 404 error etc. I have friends in Amazon (engineering) who have to be on their toes all the time. A friend broke down once when Amazon servers were down for 30 minutes at night. The failure to act in a high pressure situation can cause more harm in a day than the benefit the company can accrue from the lifetime work of an employee.
So thinking fast and keep oneself intact in a high pressure situation is a part of the job duty which should also be tested appropriately. Your skills should not leave you alone in such situations, otherwise you are much less valuable than the other candidate. I think we do not see such situations often in universities and research labs. But they happen often in various degrees in the wider wild world.
Michael,
ReplyDeleteI don't think I *COMPLETELY* misunderstood you. As you wrote, you don't do interviews of this type, you are not an expert on interviews (I'm guessing that you don't have a very good idea what kind of jobs we are interviewing for either, it's either "research" or "coding" for you), but you know that we are not interviewing well.
Kamal,
ReplyDeleteFirst, let me thank you for responding with clear and cogent arguments. I may quibble with you, but I appreciate a reasoned response.
I agree with your point about numbers. Particularly for hiring undergraduates, I can see the benefits of the interview approach (quiz/puzzle style questions) suggested by various anonymous, and that the tradeoff between its strengths and weaknesses may differ from what I suggest in my comments. However, the original context of the post (and anonymous #12, which started this all) was in the context of hiring graduate students/recent PhDs. In this context, I think the tradeoff is much clearer, and stand by my comments.
While your point about "handling pressure" is also reasonable, I have some objections. It's not clear to me that these interviews test that. Handling brain-teasers or writing code at a board in front of an interviewer is a different kind of pressure than what you're talking about. It's also not clear to me that for most jobs this skill is really a priority. For example, wouldn't it better to hire someone (or at least some fraction of employees) who had the tendency to get projects done early, rather than someone who procrastinates until a deadline but then seems to do well under deadline pressure? I know I prefer working with the former rather than the latter, though I appreciate both skills.
Anon 59: I obviously don't know anything about how you personally interview. But I do have my own experience interviewing (on both sides), I do get feedback from students, and I do read articles about hiring online, all of which are consistent. See, for example
ReplyDeletehttp://money.cnn.com/2007/08/29/technology/brain_teasers.biz2/index.htm
although you can find many similar and recent articles online detailing such interview tactics. Perhaps you interview using other techniques.
I have admitted in past comments that I'm sure there is some correlation between being able to answer such questions and some of the skills people might be hiring for. I just think such tactics ignore other skills that might be (more) valuable. For hiring massive number of undergraduates, as Kamal suggests, you might attempt to argue that this is as good a way as any, which is the direction Kamal was going. It appears that many of you doing the hiring feel this way, although you haven't provided what I feel is convincing evidence, or good answers to my objections. I'm happy to be pointed to some.
For hiring graduate students, I think this type of interview is much less informative, for reasons I've explained.
Most of the time grad students are not in high pressure situations, so its more difficult to adapt to the high pressure demands of industry. But that doesn't rule out the possibility that recent PhD can't adjust -- especially if they previously have industry experience.
ReplyDeletePerhaps the best way to prepare for a Google interview is prepare like it is an algorithms exam. In grad school you may not have to do many 3 hour exams, but there shouldn't be any excuse for preparing for such interviews especially when you know what to expect.
The dichotomy between "undergraduates" and "grad students" is funny and shows a lot about the mentality of the people it is coming from.
ReplyDeleteIn "the real world", where the military-like titles of academia carry less weight, you can see - imagine - people with and without phds working on the same team in similar positions. And guess what? Often you can't even tell the difference!
Michael:
ReplyDeleteI interviewed with Google and was not asked a single question of the type described in the article you link to. All the questions were either math or algorithms.
My guess is that these questions are occasionally asked, and they are the only ones you hear about because CNN will not report that google asks about data structures.
For you to conclude from such articles that these types of questions are a major factor in hiring is very bad science.
That you rely on CNN to know what your students need in order to land a job goes a long way to explain why you (general you, as in universities) are often doing such a poor job at preparing them.
"The second point you are missing is that performing well on a job also include performing well in high pressure situations."
ReplyDeleteSurely you are not trying to say that every software/research job includes handling high pressure situations in the job description? You make a good point for those that do, but I don't run into too many of those in software development.
I think the conclusion is clear: Industry needs to find out if people are good, does not trust the word of others that do not work within the company and has only a very short amount of time that they can allocate to make the decision. The choice is natural, interview the guy and see if he/she can pass your test. This, of course, weeds out really great people, but then again companies do indeed seem to be (in general) big moronic giants. Indeed, just as I would expect, the cogs that carry out the policy also defend the policy since, obviously, otherwise they must admit that they are powerless cogs within a giant machine simply spinning their wheels all day. They rationalize and delude themselves into thinking that their policy is powerful and smart - "high stress situations require people that can handle it," etc. All along entirely missing the possibility that a more cautious developer/researcher who does not spew garbage from his/her mouth without carefully verifying its validity may have caught the problem before it arose.
ReplyDeleteA repeated point in the comments seems to be that we professor-types are doing a poor job training our students for real-world jobs. Although I personally disagree, I'd like to hear more about this; please see my blog post at
ReplyDeletehttp://mybiasedcoin.blogspot.com/2007/12/preparing-students-for-jobs.html
and feel free to discuss in the comments.
This has to be one of the more interesting recent discussions. Claire, please post more often. :)
ReplyDeleteThere is one positive aspect of companies like Microsoft and Google asking quiz questions of interviewees:
ReplyDeleteIt has made undergraduates particularly attentive in their Data Structures and Algorithms classes since the subject matter and modes of thinking required in these classes are what seem to be most frequently tested. (Our data structures class has significant programming as well as theory.)
The respect of the material that this engenders is good for theory and a positive development for CS overall.
I really noticed a change in attitude in the early 90's as word spread about Microsoft's interview questioning and it has gotten even more so with Google interview questioning added to the mix. (For example, one of the early assignments in my algorithms course this fall had a question closely related to one that many got during their Google interviews.)
BTW: It is clear from the magazine inserts that Google produces that these quiz questions have a dual purpose of selecting smart people and advertising that only smart people get to work there, which helps the brand. They are a bit like initiation rites, too, making employees feel more like part of a select club.