Wednesday, September 14, 2016

Academic Rankings Foster Competition

This week US News and World Report released their undergraduate rankings earlier this week. A time for schools to brag. US News and World Report used to publish an actual weekly news magazine, now they mostly just focuses on rankings. Besides various categories of undergrad institutions USN&WR ranks engineering and business programs. There are many other ranking systems of varying quality but in the US we take the USN&WR rankings the most seriously.

Computer Science does not get an undergraduate ranking. Computer engineering does--not the same. CS does get rankings as a PhD program, last time in 2014. I've posted on rankings in 2005, on the failed NRC rankings of 2010, on using metrics for rankings, and Bill had his own so-called non-controversial thoughts on rankings.

In the September CACM, Moshe Vardi wrote his editor's column entitled Academic Rankings Considered Harmful
Academic rankings, in general, provide highly misleading ways to inform academic decision making by individuals. An academic program or unit is a highly complex entity with numerous attributes. An academic decision is typically a multi-objective optimization problem, in which the objective function is highly personal. A unidimensional ranking provides a seductively easy objective function to optimize. Yet such decision making ignores the complex interplay between individual preferences and programs' unique patterns of strengths and weaknesses. Decision making by ranking is decision making by lazy minds, I believe.
No potential grad student should decide based solely on rankings but neither can we expect them to solve a highly-complex multi-objective highly-personal optimization problem over all 266 PhD-granting CS departments. They will find ways to narrow down their list of schools somehow and a reasonable independent ranking of CS departments can certainly help.

More importantly rankings cause us to compete against each other. Every CS department wants to raise their rankings (or stay on top) and use that goal to work on strengthening their departments and use rankings to make the case to upper administration and alumni to get the resources needed to continue to grow. By the nature of rankings, not everyone can rise up but we all get better in the process.

22 comments:

  1. I think you are making the implicit assumptions that rankings actually measure quality and, therefore, by trying to raise in rankings a department gets better.

    I disagree with the initial assumption, and afaik Moshe does as well.

    ReplyDelete
    Replies
    1. Reread the paragraph above. Moshe is not claiming any such thing. There is a high correlation between quality and rankings, except in some rather notable cases where a few noted institutions have gone out of their way to game them. What the objects to is boiling down all those quality parameters to a single number, linearly ranked.

      In fact, for any one who has actually played with the data one of the most interesting and surprising effects is how little the rankings change when you change the weights of the different parameters.

      Delete
  2. And competition is always good? A very American attitude, I have to say!

    In Germany, it is still quite common for students to go to the university "around the corner" instead of trying to get into the best university in the country (or the best one they can get into). I can say from first-hand experience that this makes life a whole lot nicer for those of us who are NOT working at a top university. For instance, you can recruit PhD students from among your own students.

    ReplyDelete
    Replies
    1. 1. An American attitude? I thought the lack of speed limits on the Autobahn was to foster competition for high-end cars... Competition, properly used, does foster excellence.

      2. The reason we try NOT to recruit our own students is because we want to give them views of science different from our own (I thought this was a popular idea in Europe--cf. Erasmus programs). Your policy may be good for faculty, but I thought universities were created to serve students, and foster progress in science, not to make life nice for faculty...

      Delete
    2. I thought 1. was so that you can get from A to B faster, not for competition. 2. may be good for the student as well, if it allows them to pick a decent University based on other considerations (family/lifestyle/landscape/etc.) without the feeling of FOMO.

      Delete
  3. A more nuanced version of the same argument: there is *some* correlation between rankings and quality. However, many of the ranking criteria are known, and they can be manipulated. In order to do that, departments may adapt strategies (hiring, supporting specific areas, etc.) that may be far from optimal.

    There is tremendous strength in having many, very diverse departments, with different emphasis, and clashing philosophies. A linear classification works against this.

    ReplyDelete
    Replies
    1. I agree, however rankings will happen. Every student choosing a school, every candidate selecting among many job offers, every donor choosing a target to fund... they will rank the institutions whether we like it or not. Hence the best solution from our part is not to boycott all rankings, but rather make the raw data available so that each person can compute the linear ranking that best reflects their needs at that time.

      A universal boycott like Moshe suggests is pointless, and hard to believe coming from the people whose calling is the science of information.

      We should be in favor of making all this info available and providing the programmatic tools so that people can mine it in meaningful ways.

      A further advantage of making the raw data available is that is very difficult to manipulate more than a handful of such measures. After that it is easier to just become a better department in the first place.

      Delete
    2. I think the boycott on rankings is reasonable and advisable.
      You say that a ranking is inevitable, and your aim is to make it even more inevitable by giving it a stamp of approval from the community.
      I think the opposite should happen: rankings may be inevitable, and thus our task should be to fight to minimize their validity and force people to take other parameters into consideration.

      Delete
    3. I did not say a ranking was inevitable. I said rankings in the plural are inevitable since people must make individual choices. We can either make those choices on the basis of data or consult the daily horoscope.

      I give the stamp of approval to any carefully constructed personally tailored ranking for that person's use alone.

      As I said, we should provide the data that allows people to construct such rankings while refusing to endorse any specific ranking.

      Delete
    4. Anonymous 6:29 PM, September 14, 2016, wrote:

      "A universal boycott like Moshe suggests is pointless, and hard to believe coming from the people whose calling is the science of information.

      "We should be in favor of making all this info available and providing the programmatic tools so that people can mine it in meaningful ways."

      In Moshe's editorial you will find exactly what you suggest:

      "Instead, CRA should help well-informed academic decision making by creating a data portal providing public access to relevant information about graduate programs. Such information can be gathered from an extended version of the highly respected Taulbee Survey that CRA has been running for over 40 years, as well as from various open sources. CRA could also provide an API to enable users to construct their own ranking based on the data provided."

      If memory serves me right the AMS has a small web portal where students can find some information on graduate programmes in mathematics in the US (which areas are covered where and how many graduates there have been over a chosen time span).

      Delete
  4. In France the national discussion has been focused for years on the Shangai academic rankings. This was an important factor behind the decisions to create mega-universities by merging several existing universities from the same region (the rationale being that "size matters" for the Shangai rankings).

    I don't know if the folks in Shangai have noticed, and I very much doubt that this has increased quality of research or teaching. But it did create new mega-bureaucracies on top of the existing bureaucracies.

    ReplyDelete
  5. As far as I understand, rankings are a necessary feature of the US system, at least on the undergraduate level.

    When anyone can call their school a university or a college, there is no way one can tell on their own whether a particular school is good or bad. Top universities are an exception, but they are also completely irrelevant, except to the small minority of applicants with reasonable chances of getting in one. The vast majority of applicants need a way of choosing between the University of Western Obscuristan and the Central Obscuristan College, and rankings can be quite helpful in that.

    Contrast this with many European countries, where the state basically guarantees that no matter which university you choose, you get more or less the same quality of education.

    ReplyDelete
  6. It is interesting to contrast the situation in CS with that of Math. In CS, while there are good people at a number of US institutions well down the ranking order who contribute top notch research, their number as a percentage seems a lot smaller than in Math. With the exception of an institution or two, the drop off in PhD production as one goes down the rankings in CS also seems much more precipitous than in Math. The fact that CS departments have grown up in an age of rankings may have played in part in the level of concentration. However, I think that the lucrative industry options that CS bachelors graduates have without further study is likely a bigger factor in causing this drop off.

    If I am a prospective graduate student who is deciding on graduate school, the kind of information I would like to know is: What is the likelihood that a PhD from this department will be worth the investment of years of my life? What is the likelihood that those years spent earning that PhD will be enjoyable?

    While rankings can't help much with the latter (except maybe to say that having a sufficiently large cohort of PhD students around like oneself is likely to give one a better experience), it is hard to find an alternative proxy for the former.

    Rankings do help perpetuate the current equilibrium and there is also some sense in which the rankings may help the rich get richer. Indeed, they can also allow a department to slide for a time and live on its reputation. They also have an impact in faculty recruiting: PhD graduates interested in academia often seem to prefer industry jobs to faculty positions at lower ranked institutions. We are losing potential faculty researchers as a result but I am not sure that it is worse for the overall success of CS research.

    ReplyDelete
    Replies
    1. Something I did when I was applying to graduate schools was to look at where faculty members at various schools had done PhDs, mostly to convince myself that it was okay to go to grad school at somewhere that wasn't one of the usual places. And it is true that amongst those who went to American universities, a large portion of faculty in cs theory did their PhD at MIT or Berkeley. But it seems as if the diversity of schools increases as you go down with age, and recently there have been many people who have been very successful in the job market that did not go to MIT or Berkeley, or even places like CMU, Stanford, Cornell, Princeton, etc. I think a lot of this is that cs theory is still relatively young, I think it is still the case that a large portion of active researchers are academically descended from Manuel Blum. As the community grows, I suspect this drop-off will become less steep.

      Delete
    2. While Manuel Blum is an awesome person and scientist, there are many distinguished theoreticians that are not his descendants: Knuth, Valiant, Karp, Rivest, Cook, Levin, Shamir, Babai, Szemeredi, Razborov, Dana Scott, Hopcroft, Rabin, to cite a few of the older generation, as well as much if not most of the Algorithms community.

      It is not clear to me that Math is less hierarchical: there are few top-ranked departments that do not have a Fields medalist....

      Delete
    3. The way I see it (people from lower ranked departments are expected to disagree for the obvious reason): your chance of finding a faculty position if you go to top 4 (Berkeley/CMU/MIT/Stanford) is ~10 times more than if you go to a university below fifth (which is UIUC). In a job market where less than 10% of graduates find academic positions it is important to maximise your chance by all means possible. If you however know exactly what you want to work and the typical student of the supervisor you have in mind have been successful in finding good positions then that makes sense.

      @Paul, it also has to do with the fact that mathematicians are used to publish single author papers and wait for years to get a paper into a journal. In CS it is extremely difficult to compete with other researchers if you do not have a good number of coauthors. If you are stock in a low rank university without much reputation you don't get good colleagues to work with and you seldom get bright students and postdocs. That can still work out if you are the kind of person who do not pay attention to the game and can work alone on your own project, maybe you solve a very difficult problem and become famous, more likely you will keep teaching medicare students and live an OK life without any big events and if you love doing research and are not ambitious that is completely fine thing to do. Many CS graduates look at that and compare that with high salaries they can earn in the industry plus the impact they can have on people's lives and they go for the later. There are many differences between math and CS that make it unlikely that things would work for CS the way it works for math.

      Delete
    4. I see several things wrong with the post. First, in many respects TCS is more like Mathematics than, say, Computer Architecture. For many theoreticians, going to industry means a big change in the area one works in. Second, the number of departments with superb TCS faculty is much bigger than 4, and clever students go to those departments because of the exceptional faculty. Princeton, Cornell, Chicago, Yale, Harvard are just some of the first names that come to mind, but the list of places that a Theory student could get superb opportunities is much longer. If you want supporting data, look at the list of Godel Prize winners, Nevanlinna Prize winners, STOC/SODA/ICALP committees, etc.
      Also, do not underestimate the quality of students at "smaller" institutions. The typical state flagship university will attract some extraordinarily talented students. They are not all "medicare", as you suggest.

      Delete
    5. You are looking at exceptions, I am looking at statistics. Everything else being the same, your chance of finding a position at a reasonable university is 10 times higher (for various reasons, some related to the quality of your research, some not).

      Yes, some people graduating from schools ranked below 5 do find positions, that doesn't change the fact that if you graduate from Berkeley or CMU or MIT or Stanford you are 10 times more likely to find a position than if you graduate from Princeton or Cornell or Chicago or Yale or Harvard.

      You misunderstood the part of about students. If you are a faculty in a school which is not in top 50, almost always you are going to be dealing with students of quality much lower than than if you were in top 25 schools. Yes, there will be exceptions from time to time, but they will be EXCEPTIONS! If you are a genius you can probably find a position no matter which school you. If you are not then you would like to pay attention to all the side issues that increase your chance of finding a position after graduation if that is what you are after and that includes the name of the school you graduated from for various reasons.

      Delete
    6. Anonymous:

      The numbers (at least in Theoretical Computer Science) are way too small to get meaningful statistics. The "10 times more likely" is unlikely to hold up anyhow. For fun, look at theoreticians at Stanford:
      my list is Boneh, Charikar, Guibas, Reingold, Roughgarden, Greg Valiant, Ryan and Virginia Wiliams, and Wootters. Of these 9, 4 (Boneh, Reingold, Roughgarden, Wootters) come from PhD programs not on your "top 5."
      Right now, if you are very good, you are going to find positions. Remember that research--oriented departments look for the "exceptions" -- and the reason that if you were number 4 at Stanford, it is less likely that you are going to be great than if you were number 1 at Harvard....



      Delete
    7. Here is the data if you want to check it yourself.
      http://cs.brown.edu/people/alexpap/faculty_dataset.html
      The data has been updated since the last time I checked it. Still there is a significant gap that is not explainable by the quality of research or quality of students. We tell our students that they will be fine and will find a good academic position if they do good research. That is a complete lie. If you are an exceptional researcher you will most likely find a position. If you are a good researcher then your chance of finding a good academic position drops significantly and depends on all kinds of factors that have nothing to do with your quality of research, and whether we like it or not, the name of the school you have graduated from does play a significant role.

      Here are the top 10 schools according to the number of faculty in top 50 US schools who have graduated from them:

      Massachusetts Institute of Technology - USA 251
      University of California - Berkeley - USA 181
      Stanford University - USA 146
      Carnegie Mellon University - USA 121
      University of Illinois at Urbana-Champaign - USA 82
      Cornell University - USA 68
      Princeton University - USA 66
      University of Washington - USA 63
      Georgia Institute of Technology - USA 54
      Harvard University - USA 51

      Delete
  7. What is your opinion on http://csrankings.org ? If I understand correctly, only recent research publications are counted here? From the perspective of a grad student, isn't this the most appropriate ranking?

    ReplyDelete
    Replies
    1. It is much better than other rankings, but it is not the whole story. Even between schools where the quality of research and students are not very different side factors not having anything to do with quality of research can significantly increase your chance of finding an academic position significantly. If you count the number of CS faculty who have graduated from MIT in top 50 US schools and compare it with Berkeley or Stanford or CMU you will notice that they are almost twice as many. Arguably the quality of research is similar at these top schools, the quality of faculty and students as well, then you can wonder why MIT graduates are so much more successful at finding academic positions than Berkeley or CMU or Stanford graduates let alone the rest of schools.

      Delete