Saturday, May 07, 2005

Ranking CS Departments

A few years ago, one ranking listed Harvard as the number one engineering school. The schools were ranked by average starting salary of their graduates and both of Harvard's engineering students that year got good jobs.

I could do several posts on rankings but let us focus on rankings of Ph.D. programs in computer science. One cannot give a linear ordering of departments: One department might have a strong theory group and another strength in systems. Even within theory, one department could have strong algorithms while another group has strong complexity. Even then one's graduate experience depends more on their individual advisor than the department as a whole.

Nevertheless Americans like statistics and ordering things including CS departments. There are two major rankings of Ph.D. programs: The US News and World Report ranking last updated in 2002 and uses a methodology of giving questionnaires to department chairs and directors of graduate study. The NRC ranking looks at a slew of different statistics but hasn't been updated since 1993. I hear rumors that the NRC will do a new ranking soon.

The US News ranking captures perception of strength instead of strength itself, often based one's opinion of a department from a few years back. On the other hand, the purpose of rankings are perception as we wish to be perceived as a strong department. We all complain about the rankings but they affect us greatly, in recruiting students (Americans especially use the rankings to choose a Ph.D. program) and recruiting faculty. When hiring faculty we often rightly or wrongly give extra weight to Ph.D.s from higher-ranked departments. Deans and provosts use the rankings to allocate resources as higher rankings lead to more prestige for the university as well.

For example, if a mid-level department wishes to have the strongest quality faculty they should hire mostly in one area, as people like to join groups with people they know, respect and can work with. But this approach won't help much in the rankings so most departments try for a broad faculty with likely lower overall quality but a better ranking.

At least forty departments have a stated goal of being a top-ten department. The pigeonhole principle guarantees many won't end up happy.

13 comments:

  1. Two things:

    1. "both of Harvard's engineering students that year got good jobs."

    What do you mean by both? Are you saying that only two students graduated from Harvard with a degree in engineering? That seems like an awfully low number.

    2. Say you have two people applying for a faculty position in the CS dept at UC. Let one of them be from say MIT or Berkeley and have completed an "average quality" dissertation. Let the other be from say UNLV and have completed a "high quality" dissertation. Would you still hire the graduate from the elite school even though their capability may not be as high as that of the graduate from the non-elite school?

    ReplyDelete
  2. Rankings can be useful if one keeps in mind all the caveats that come with them. For one not much weight should be given to the linear ordering of closely matched departments. One can make a solid case for any of the top four departments (Berkeley, Stanford, MIT, CMU) as the number one in the country (in fact US News has them tied for #1), and which one comes ahead is entirely subjective.

    On the other hand, we can be reasonably confident that a 3.x CRA ranked department is in a different league than a 4.9 one.

    During the interview process, in general too much weight seems to be given to the institution of origin. It is important to keep in mind that the average graduate from a top four department is below a top graduate from a department ranked 5-20.

    Having said that, a quick (and unscientific) look at the math genealogy suggests that about 5 out 6 very famous researchers graduated from a top 4 school. Perhaps a good strategy to become a better department is to actively seek that 1-in-6 candidate which comes from a lesser known school but is nonetheless top notch.

    ReplyDelete
  3. What is going on here is that Harvard has many degrees that one might think of as "engineering," but only one of them is actually an ABET-accredited bachelor of science. The rest, including Computer Science, Applied Math, etc. are actually bachelors of arts (AB) and not officially "engineering" degrees. You can still earn a master's of science in many of these areas, but not a bachelor's.

    Most people, at least when I was an undergrad, elected not to go for the "engineering science" degree. There are many fixed requirements, and unless you have a specific need for it, there are other degrees with more flexibility that allow you to study similar things. It wouldn't surprise me to learn that one year only two seniors were graduating as "engineers." If the survey was limited to bachelors degrees, then that could explain the survey results.

    -David Molnar

    ReplyDelete
  4. Continuing David Molnar's comments, let me state for the record that (as a counterpoint to what I assume was a joke by Lance) computer science and engineering are alive and well at Harvard. Indeed, while I can't speak about starting salaries elsewhere, it wouldn't surprise me if starting salaried for Harvard CS graduates was among the top -- our undergraduates are highly sought after by top computer companies like Microsoft and Google, and consulting and financial companies (as well as the top CS graduate schools, and sometimes the top law schools, medical schools, and so on).

    The only other thought I have is in regard to Lance's conclusion:
    "At least forty departments have a stated goal of being a top-ten department. The pigeonhole principle guarantees many won't end up happy."

    No, no, Lance, you misunderstand. It just guarantees that universities will encourage and back the rankers. We need at least 4 ranking organizations (perhaps using different criteria) so each of those 40 schools can claim that they are the top 10 under some ranking.

    In a similar vein, I've noticed, for instance, that even a single ranking (the US News and World Report ranking for undergraduate) changes significantly from year to year. I doubt schools are changing that much year to year. I assumed it was because they needed the rankings to be different from year to year so they could keep selling them. (Much like textbooks authors who need to revise their textbook to avoid an oversaturated used book market...)

    ReplyDelete
  5. "Having said that, a quick (and unscientific) look at the math genealogy suggests that about 5 out 6 very famous researchers graduated from a top 4 school. Perhaps a good strategy to become a better department is to actively seek that 1-in-6 candidate which comes from a lesser known school but is nonetheless top notch."

    What is not clear from this statistic is how much of this is really reinforced by the idea of hiring only people from top schools. It seems to me that, without good jobs, it is difficult for researchers from other schools to compete.

    ReplyDelete
  6. "What is not clear from this statistic is how much of this is really reinforced by the idea of hiring only people from top schools. It seems to me that, without good jobs, it is difficult for researchers from other schools to compete."

    Do you mean that, for example, you have to be a good researcher to be an Stanford professor, but then that position makes you into an even better researcher because you have the best students to assist you with your research? How do you think "good jobs" make it easier to compete? More resources? Better students? More exposure to other people's research? Less teaching? I want a "good job", but I wonder if it would really make me do better research.

    ReplyDelete
  7. Does anyone know of any legitimate world-wide ranking of CS departments (as opposed to just those in the US)?

    ReplyDelete
  8. Some world-wide rankings (check yourself for their criteria): http://www.thes.co.uk/worldrankings/. The access is not free, though. Google reveals however some surveys of their statistics, http://www.freewebtown.com/lyen/rankingu.html is one of these. In the "Engineering and IT" list, the order is Berkeley, MIT, Stanford, IIT, ICL, Caltech, Tokyo, Cambridge, NU Singapore, Beijing.

    ReplyDelete
  9. Research, especially in CS, is a social endeavor. If you are in a department with strong people it is easier to collaborate, to have good students, to know what is going on in the field, to be known by the editors of journals and members of committees. All this provides a positive feedback to the career of a young faculty. The lack of any of these, on the other hand, works against the researcher in comparison with his competitors in other institutions.

    ReplyDelete
  10. I would just like to completely agree with those who argue that positive (and negative!) feedback effects can have dramatic impacts on a career, I think much more than we tend to assume. We like to believe that "merit" defines success, but my viewpoint is that there is a lot more to it than that.

    ReplyDelete
  11. How do you think "good jobs" make it easier to compete?

    More resources?

    Yes.

    Better students?

    Yes.

    More exposure to other people's research?

    Yes.

    Less teaching?

    Not so much. Most of the top twenty departments have comparable teaching loads.

    ReplyDelete
  12. Just to add one more advantage of
    being in a better/larger department:
    A vita with 40 4-author papers is
    generally viewed as almost twice as
    good as a vita with 20 2-author papers of the same quality.
    If you in a group with
    many researchers, it is way easier to make
    some modest contribution to someone else's results and end up with co-authorship.

    ReplyDelete
  13. "Research, especially in CS, is a social endeavor. If you are in a department with strong people it is easier to collaborate, to have good students, to know what is going on in the field, to be known by the editors of journals and members of committees. All this provides a positive feedback to the career of a young faculty. The lack of any of these, on the other hand, works against the researcher in comparison with his competitors in other institutions."

    I'm a Chinese student, and in my impression US academy has a fair air, isn't it true? And I think relationship with strong people is mainly undeterminable by your own effort. So a industrious researcher
    in a not-so-good environment can still get success. Otherwise it's really frustrating to average researchers, IMO.

    ReplyDelete