Thursday, January 12, 2006

Time to Write the Letters

Every year I seem to write more letters, letters for those applying to graduate schools, letters for those looking for their first postdoc or assistant professor jobs, letters for tenure cases. Why the continual increase? More and more we see young researchers taking multiple postdocs, where each round of searching requires a set of letters. Letters from senior people carry more weight, and each year I get another year senior.

But the Internet has led to an increase in requests because one can now have automated processes that send requests for letters. Harry Buhrman has two complaints about this model.

  1. Often these automated requests seem cold. Even automated they could ask in a nice way, be grammatically correct and address the person directly.
  2. Universities should pay at least some small amount of money to letter writers or their institutes. This will keep down the number of requests and reimburse the letter writers for some of their time.
I second Harry's first point, though for me I never even look at words in a request, I just want to know where to send the letter.

I don't agree with Harry's second issue. We have a responsibility to write letters for our students and colleagues. The marginal cost of a sending an additional letter for someone is rather small, though the universities should make the process as painless as possible. A URL I can click and then upload is best. Having to cut and paste a username and password sent in an email is already adding effort for me with no increased security. For one graduate program I had to go through ten web pages of forms to fill and verify; there is no excuse for that. Ideally I would like some place I can just deposit the letter which legitimate universities could just download as needed.

For tenure letters perhaps a payment scheme would make sense. These letters require much more effort and we only write one of them for each candidate.


  1. If Profs get paid for writing letters, then people should get paid to referee papers.

  2. Note to anon: The purpose of the cost is to discourage universities from over-consuming the service of letter writers. Think of it more of a tax than any means of real compensation.

  3. Thank you, Lance, for disagreeing with Harry on the second point. From a student's perspective, it is often a somewhat awkward process to get letters of recommendation from professors for summer REUs, graduate schools, and the like. Please realize that it is not our choice to have to get these letters, but they are nevertheless vitally important to our future career. Myself and many of my friends (at various universities) have had experiences where we have asked one or another professor for a letter, and s/he will agree and then despite reminders, will not submit it on time, be difficult to get in touch with about it, and in some cases it becomes too late. This causes major panic attacks for students, fallout in relationships with faculty, etc. In conclusion (and to echo Lance), please seriously consider the humanistic side of the affair in your outlooks on letter-writing.

  4. Berkeley apparently has a "Letter Service." I haven't used it, but apparently profs here can write one letter to the service, then the service takes care of sending it out to places of the student's choice. I don't know if they handle online recommendation forms, though. Does this exist at other universities?

  5. CMU seems to have a similar letter service, but it is less formal. An administrative assistant collects all the letters and I ask her when I need them sent out. And I thank her profusely. Thanks, Nancy!

  6. Seems like a number of schools are now requiring online submissions, which those services don't handle. Plus they don't exist anyway at some schools (we didn't have something like that at my undergrad, for instance). We don't all go to a Berkeley or CMU, I guess =)

  7. The Berkeley system would be useful for a student planning to take a number of years off before applying but it is not something I would use if I could contact the faculty directly. The difficulty is that it does not do any tailoring of a recommendation to the places where it will be sent.

    The same recommendation to a top ten university and to one way down in the middle of the pack might not be appropriate. The recommender doesn't know where the recommendation will be sent and the institution receiving the recommendation gets less information because of this.

  8. Ideally I would like some place I can just deposit the letter which legitimate universities could just download as needed.

    The mathematics community has just that, see the AMS site
    As mentioned above, a self inflicted problem occurs of course when you want to tailor a letter to a specific place ("X would be especially suited for your department Y because ...").

    As for the anonymous comment on the 'humanistic' side of the matter: pretty much by definition every letter writer has gone through the process by him or herself, so they do know how it feels to have to ask for letters, the frustration, and so on. Harry wants the receiving universities to pay, not the applicant.

    - Wim van Dam

  9. Two comments:

    (1) Maybe the number of letters being requested is simply too high. I know places that require 4 letters for a faculty applicant, and optionally 5! That seems a bit ridiculous to do the 8 or more letters needed for a tenure case.

    (2) A nice model adopted by some schools which should be used more universally is the following: perform a first pass over the candidates to narrow the field, then request letters only for those who make this "first cut." A variation on this uses a single letter for the first pass, and then requests additional letters for the latter stage.

  10. My letter writers were extremely accommodating and made the process as easy as they could for me. They went out of their way to make me feel like I wasn�t burdening them even though I applied for many (and many different) things�graduate school, graduate fellowships, and fellowships abroad. An example for us all. Thanks again!

  11. A fee for letters of recommendation would likely be passed on to the candidate in the form of an increased application fee. Such an application fee increase would nullify any benefits of a fee for recommendations. De facto, it would be the applicant paying his/her recommender (or recommender�s institution) for the recommendation.

  12. "As for the anonymous comment on the 'humanistic' side of the matter: pretty much by definition every letter writer has gone through the process by him or herself, so they do know how it feels to have to ask for letters, the frustration, and so on.".

    I guess that was not appaerent to me based on the experiences I described =)
    It probably depends on the professor in question, and also since people have different experiences with with stuff it would depend on the experiences the professors themselves had...

  13. (1) Maybe the number of letters being requested is simply too high. I know places that require 4 letters for a faculty applicant, and optionally 5! That seems a bit ridiculous to me...

    Hiring someone in a faculty position is for maybe the next forty years (unless you don't hire people you expect to be good enough to get tenure). There are hundreds of applications to choose from. This makes it pretty important to get good data. Given the variation in the amount of effort and candor that people put into letter wrtiting, any smaller number of letters is risky.

    On the other hand if you've already got more than a couple of corroborating well-written letters then there is no reason to hold things up just because a fourth or fifth letter isn't there.

  14. My letter writers were extremely accommodating [...] An example for us all. Thanks again!

    ... Who of course have no inklink that they are being thanked because the post is anonymous.

    - Wim van Dam

  15. I wonder if the academic hiring process could use some ideas from corporate hiring. If you apply for a position of an engineer, you get invited for an interview, talk to several interviewers who ask you questions to test your skills and abilities. Then each interviewer writes up a report on the candidate, and the hiring committee makes a decision based on those reports.

    Why can't the faculty at a department interviewing a candidate do a similar thing? The candidate spends a lot of time meeting with faculty members, yet all these meetings are not enough for a firm recommendation. If professors were better trained in interviewing skills, and made sure to use face time with the candidate efficiently, they could come up with a good decision without relying on outside recommendations.

  16. Re: Palenica's comments:

    I think academic hiring already does something like face-to-face interview. Although it is far from "testing skills on the spot"-kind of interview, faculty do write reports (at least those in the hiring committe and the conscientious ones) on candidates. Recommendation letters are extremely important in addition to such face-to-face evaluations for several reasons (i) How good a researcher is hard to evaluate based on a half-hour or even a one-hour polite conversation. It is difficult to ask deep questions in candidate's area unless you are also in it. (ii) Research contributions generally tend to be very specialized and unless you are exactly in the same area you may not be able to appreciate the relevance and significance of the candidate's work. Experts' opinions help you in that evaluation. (iii) Recommendation letters allow you to pre-screen candidates *before* the actual interview; this is, of course, done at companies as well.

    Basically, the strengths being tested for an academic position are quite different from the skills tested in a company job. I think the current model is good as long as departments are not overly prejudiced about what areas they want to hire in.

  17. I totally agree with palenica. The problem with the system is that when 10 different people are writing nice things about 10 different candidates, it's impossible to interpret the results in a consistant manner. How do you decide whether the "good" of advsor-A is same as the "average" of advisor-b. If you can't, then the whole process is a joke. (BTW. if there is no one who can judge the qualification of a person, then it's better to hire a set of experts and let them review each candidates. Having different people write different things is not helpful--in fact it can hurt.)

    In fact, when the time came for me to apply for the Ph.D. I decided not to, because my advisor for MS thesis was very anti-social and racist, (and as a foreign student, I didn't know this at the time of joining the university) and he would not write anything about me (good or bad!). Even though I had 4.0 GPA during my MS from an ivy league school, I was stuck with this advisor. To me the system of getting recommendations seemes so brain dead, I decided not to apply. (How can you not have checks and balences against such abuse of power?)

    Fortunately, not going for a Ph.D. has not hurt me so much (although I would have wanted to become a prof., which I can't do now). But now I work in one of the "best" research labs in the US, and things are good.

  18. Today I was submitting a reference letter for a faculty candidate at a school that shall remain unnamed. The web form asked me to rate the candidate's research ability on a scale from 1 to 10, without giving any further guidance. I just left it blank. How, HOW, can one do this meaningfully? I WANT to contribute to the system in a way that's fair and honest, and helpful to both the candidate and the school, but how can I contribute to THIS system?

  19. The utility of using letters of recommendation at all is dubious at best, and frankly using more than two is absurd on its face. It would sure be nice if some school as a whole made a policy of not writing any letters of recommendation at all, so that the students wouldn't be subjected to that arbitrary and humiliating process, and wouldn't be penalized for it.

    The high school I went to tried to get everyone into college. Every student got multiple letters of recommendation from random high school teachers, each of whom was writing about a hundred others from random students they'd had over the course of the previous four years. Oh man what a bullshit process that was.

  20. There is not much sense behind the concept of Recommendation letters, as pointed out by several people.

    For graduate students applying for jobs in academia and industry, I think that jobs should be handed out based upon their publication record, and nothing else.

    For admissions to graduate school, I think that the best solution is to have a written or oral exam, testing the candidate's ability in his desired area. And no, I do not mean GRE or the subject GRE, those tests are a farce, and test nothing. I mean a more challenging test (and one-on-one oral tests are the best) that challenges the fundamentals of the student. Though given the administrative hassles of the same, I don't think it will be feasible.

  21. I disagree with previous posts implying that letters are always "meaningless." Judging faculty applicants based on publication record alone is not even possible, unless there is an expert in the appropriate area who can judge the quality of those publications.

    A problem, though, is when 3 letters are required and, after getting one from your advisor and someone else you worked with, you are left with the third letter being from someone you took a class with 4 years ago (utterly meaningless...).

  22. With respect to needing 3 letters of recommendation -- yes, it probably is meaningless to make your third letter be from someone who had you in class four years ago when applying for a faculty job. As a result, a candidate who obtains 3 letters relating to research collaborations is going to be more attractive than a candidate that doesn't. Therefore, the letter system rewards people who collaborate more often and more widely. Is that a bad thing?

  23. Therefore, the letter system rewards people who collaborate more often and more widely. Is that a bad thing?

    Two comments: first, I would suggest that an applicant who can only get two letters from people who know him well should only have to send two letters (but still be considered, rather than automatically ignored for lack of a "no-op" third letter).

    Second, and more to your point, I am not convinced that collaborating with many different people is inherently any better than collaborating just with your advisor (or, hell, publishing a bunch of single-author papers). Also, note that a graduate student at, say, Berkeley will have a much easier time finding multiple people to work with than an equally-good student at a lesser school (more faculty, more connections, etc.). Maybe this illustrates the importance of picking a good graduate school, but still.

  24. Anonymous, I actually tend to agree with you. There are many researchers I respect who work primarily on their own. I'm just observing that our system for finding and promoting people via recommendations seems to work against this style. I don't actually know if that's a good thing. I do notice that it hasn't extinguished that style of research, so the effect can't be overwhelming.

    With respect to picking a good grad school, you need letters to get into grad schools, too. So unless you figure out that you need good recs early, you may end up with fewer choices in grad schools than you might "deserve."

  25. There is a lot of talk at cross-purposes here with faculty and grad applicant situations being conflated.

    For grad applicants: Some letters are necessary: Many undergrads get serious research experiences and the breadth and depth of these research experiences are not something discernible from transcripts. Other students have relevant outside activities or internships. Even many of the "got an A in my course" letters are relevant because they often give much more detail than the mere numerical grade (courses at different universities are different, with very different student populations, and there are different ways to earn an A).

    But why ask for 3 letters? To my mind it is because undergrads are often poor judges of whom to ask for letters. All that the grad school is looking for is at least one really good and informative letter out of the bunch. Students have no way of knowing if the faculty they have asked will be responsible so even if they think they know who this good letter will be from they should get more letters.

    For faculty applicants: For a good candidate there should be no problem in getting enough letters. The recommenders don't have to be at the candidate's home institution, supervisors on internships, or collaborators. They can be people elsewhere who work in the same area as the faculty candidate and who are likely to know and appreciate the candidate's work. If not enough of those people exist then it is certainly not a good sign!

    Paul Beame

  26. Well, I still fail to see how the letters of recommendation help? If
    different people are writing different things about different
    people, how can you compare? If you can't, then what is the purpose
    of the letters of recommendation.

    You might say, recommendations show how good the researcher is at
    personal networking? But you are planning to fill a faculty
    position, not a car salesman position. For MBA admissions, I would
    understand (with some reservations), why these letters might have
    some value, but for a CS faculty position or for a CS grad studies, I
    fail to see how it helps?

    In fact, I think a lot of people hate applying to the grad schools because the whole admission process seems like a complete BS. Then people keep asking, why aren't there enough students in Grad schools? Well, why shall there be, if the admission process itself is so unintellectual.

  27. It remains that letters of recommendation may contain factual information that is not otherwise present in the application and is relevant to making a decision.

    Also, experienced faculty can spot intangibles that (in their experience) are an indication of proclivity in research.

  28. I can understand that recommendation letters have some factual information, which in some cases might help that candidate.

    However, a lack of such information in some other candidate's letter doesn't say anything about that other candidate, especially when someone else doing the writing. In other words, a lack of information is not the same as negative information. If that weren't the case, then P vs. NP would have been settled by now.

    Hiring/Admission processes are already quite random. Taking recommendation letters into account does not make the system any better, but does create a lot of anxiety among candidates. (1 letter of recommendation from the advisor I can understand, but more than that is pure torture--although I can understand the motivation behind it.)

    --Chris Clanton

  29. As Paul said, there's a difference between grad school letters and faculty position letters. Let me focus on the latter. For the sake of argument, I will take a strong stand: letters are an incredibly helpful part of the process, and I am assuming people who are arguing otherwise simply have not been involved with hiring!

    Here is a partial list of information you get from (honest) letters that you don't necessarily get from the rest of an application:

    1) In a student's joint papers, how much of the work was his/hers, and how much was the work of others? Was the student coming up with the big ideas, or just doing the grunt work?
    2) How does the student rank compared to others from that institution that year and in previous years? (This info is noisy but still helpful; many places seem to be on a rising staircase, so every year they have a student who is the best in the last 10 years...)
    3) What are the student's weaknesses? Honest letters can suggest weaknesses; smart institutions realize the people they are hiring may not be perfect right out of grad school (or ever).
    4) How much impact is the work having on the established experts in the area?
    5) What work does the student have in the pipeline that may not have been published, and what is it's likely importance?

    Notice that these issues will generally require letters from people other than the advisor. Also, the advisor is generally the most biased letter-writer; letters from others are mandatory for getting good information.

    The arguments against letters seem to be:
    1) The information is noisy. This is true, but there's still good information there, and faculty are pretty good at screening out the noise. (I personally bring a healthy dose of skepticism to those "... best student in 10 years..." statements.) The noisy information is better than no information.
    2) Letters reward the well-connected and help shut out the hidden geniuses from less prestigious institutions. This is less true than you think. Indeed, letters are perhaps one of the few ways though hidden geniuses will get noticed appropriately! All folders from top 10 schools get read very very carefully, and other schools maybe less carefully. But a student from a non-top-10 school who manages to get a letter from Dick Karp saying that they are brilliant will get a lot of attention. Letters are actually a great way to level the playing field.
    3) Some students, through circumstances that are not their own fault, will not be able to get good letters, and are therefore hurt by the process. I find this point specious. The ability to get good letters is, in my experience, highly correlated to the impact of your work and your current status as a researcher. Also, you know well ahead of time you need good letters for a job. Graduate students should consider it part of their job description to put in the legwork to make sure they will have appropriate letters when they get their degree.

  30. Let me strongly second Mitzenmaher's observations. I have been involved with both faculty and grad student recruiting for many years, and letters are probably the most important part of the process. He argued strongly for letters about faculty, so I'll take the "letters for grad school."

    One of the difficulties of judging grad student applicants, is that the qualities that make someone excel in college are not the same as the ones needed to excel in grad school (which are not exactly the same that make a researcher be superb.) There is, of course a large overlap: Turing was a superb undergrad student. But, for the sake of argument, a student who stayed up nights to make a project perfect, or to solve a really tricky problem, might have obtained a better GPA if she spent more time on another course.

    Letters CAN point out indicators that do not get included in grade reports. A student may write solutions that have an unexpected twist, or an elegant shortcut, or an interesting insight. Another, less inventive student, may simply write down the standard solution. Both will get an A. A letter can distinguish the two. Similarly, a project may be late because the student was lazy, or because he was overly ambitious and the system crashed the night before the due date. Again, a letter may provide such information.

    The information from letters is noisy: so are grades, or standard tests, or acceptance criteria at conferences. People in admission committees are aware of this. Moreover, we might have seen the same letter-writer previously. Comparisons to previous letters can do a lot to narrow things down.

    You can even do the unthinkable, and contact the letter writer asking for more information.

    The more data one has, the better the process.

    As a final aside, the idea of being paid for refereeing is not at all absurd. Book publishers often pay for careful reviews of book manuscripts, and some journal pay for editors.

  31. Do people really put THAT much effort when reviewing prospective grad students? For example, Berkeley has 2000-3000 applications. It seems hard to believe that the committee goes over all these letters with a fine-tooth comb.

  32. Of course we don't, but after narrowing it down to 20-30 of the top applicants in our sub-discipline (this is how things are partitioned at Berkeley), letters/research become the primary way of distinguishing between the best students.

  33. Isn't it that it is the professors themselves who are asking for these recommendation letters from other professors to prove how important they are, even if not for teaching/research!

    Seriously, this business of getting 3 or more letters is a pain in the ass. Maybe your undergrad adviser will be happy to give a letter for you, but what about the other letters? And if students have to pay for all this shit, they will be even more away from the graduate school.