- When making a ranking one must ask: What is it for? For Academic depts it may be to help guide potential ugrads or grads in their choice of where to go. Rankings of the most influential people of all time (Michael Harts book here), or in a given year (Time magazine does this) are made to (I think) organize our thoughts and start debates. Michael Hart also did a book about the most influential people as soon from the year 3000 (so half are fictional) as a way to speculate (see here). My own ranking of Bob Dylan satires here was done for my own amusement.
- Transparency sounds like a plus. But if a ranking is too transparent, and is considered important, than organizations might game the system. Recall Goodhart's law: When a measure becomes a target is ceases to be a measure. On the other hand, if the measure really is good then it may be okay if it becomes a target. Some measures are hard to game- like surveys of what people think.
- There have been a variety of rankings of presidents (see here). These ranking say something about the times they were done. Studying how they change over time could itself be a historical project of interest. Another thought: the book Hail to the chiefs it notes that James Buchanan and Andrew Johnson usually rank as the worst presidents, while Lincoln ranks as one of the best--- but this is unfair!--- Buchanan could not stop the civil war (but nobody really could) and Johnson had to clean up the mess after it (but nobody really could). The Lincoln presidency was almost entirely the civil war which Ameican won, so he gets the credit. More to the point--- ranking presidents is odd since it may depend very much on the times they govern.
- Bill James (KEY Baseball statistician who I think should go into the Hall of Fame for changing the way we think about Baseball) has tried to have lists of GREAT TEAMS. But there is a problem (which he fully notes)- some teams are GREAT in terms of having great players, but didn't win the world series, or have only one pennant. Less than the sum of its parts.
- Numerical ratings may be odd in that they lump different items together. GPA is a bit odd--- do you prefer a student who got an A in Theory and a C in Operations Systems, or a student who got a B in both? I don't know the answer, but GPA wipes out the distinction.
- Rankings that compare very unlike objects are useless. Here is a ranking of CS blogs--- the criteria seems to be just one guys opinion. I disagree with his ranking, but I have no idea what he cares about. Also, he includes Harry Lewis's fine blog BITS AND PIECES, which is often about academic stuff, and also Terry Tao's fine blog WHATS NEW which is really a math blog. Very hard to compare those two to each other or to others.
- The tigher the focus the more useful a ranking can be. Ranking the best novelty songs of all time would be impossible (Number one is PDQ Bach's Classical Rap) but if you restrict to, say, best science fiction Satires I(Luke Ski's Grease Wars part 1, part 2, Part 3)- then its easier (Trivia note- Science fiction satire songs are often called FILK SONGS--- the urban legend is that at an early Science Fiction Convention Science fiction Folk Songs was mispelled as Science fiction Filk Songs and hence the term was born.)
- SO, what really would help potential CS Grad Students in theory? Perhaps a grid where for every department is listed the theory faculty, and for each one the number of pubs in top tier confs, second tier confs, and journals in the last 5 years, and their area, and a pointer to their website. Then RESIST the urge to boil it down to one number.
- I"m reminded of being on the Grad Admissions committee. I get to look at the transcript (much more informative than the GPA), letters, possibly papers. Fortunately I don't have to boil it down to one number--- there are very few categories (accept, wait list of some sort, reject, but there can be a few others involving scholarships, but VERY few categories really).
- Finding what you want: I think that Raiders of the lost ark has tone of the best ending-of-a-movie ever. So I Googled best movie ending and variants of it, and alas, Raiders did not do well. One of the rankings didn't have it in the top 50. So I then Googled best movie endings Raiders of the lost ark and I found a ranking that had Raider in the top 10. Yeah! But this is all silly- I found some person who agrees with me.
- Steve Skienna and Charlie Ward have written a book Who's bigger: Where historical figures really rank which has a transparent and reasonable way to measure... not clear. Probably fame. For a review see here
Tuesday, November 11, 2014
Non controversial thoughts on rankings
US News has a ranking of CS depts and various subcategories. Recently MohammadTaghi Hajiaghay and Luca Trevisan have suggested alternative rankings here (Moh) and here (Luca). These rankings inspire me to record some thoughts about rankings.
Subscribe to:
Post Comments (Atom)
I think it's interesting to flip this around and view it as a mechanism design problem. Assuming the entities being ranked (e.g., departments) receive utility from being ranked highly, what measure(s) would you want to use to encourage broadly beneficial behavior? (E.g., in the case of departments, behavior beneficial to the advancement of their discipline, behavior beneficial to the students who go through them, behavior beneficial to humanity, etc.) One thing for sure is you probably would want a measure that includes many different positive things that people can do.
ReplyDeleteFrom your concern about GPA, maybe schools should also report GPV -- Grade Point Variance.
ReplyDeleteIt would be a plus because (1) it conveys more information to colleges / grad schools about the applicant, and (2) some students will obsess over it, meaning they'll probably understand variance better.
How do you compute this GPV? Is it just the weighted (biased? unbiased?) variance, or something else?
Delete"Will this rank as one of Bill's top posts?" Fortnow.
ReplyDeleteSorry to see that is not happening. High rank needs high controversy.
I propose the following ranking (or rather "ordering"); it's one that all theory people should be familiar with.
ReplyDeleteAlphabetically.
You are suggesting information that we should put for each department without asking what a grad student cares or should care about. Isn't it better to do a survey of grad students and recent graduates and ask them what is important for them? What would they like to know before choosing a grad school?
ReplyDeleteHere are some that comes to my mind: the fields each department is active in, the star faculty (people like Valiant, Blum, ...), some measure of research activity (like number of papers in top conferences), the prospect of finding a tenure track academic position in one of top schools (with break down), etc.
I personally like to know the likelihood of a graduate finding a good tenure-track academic position a few years after graduation for each school. This would penalize departments who take large number of students who do not find good tenure-track academic positions. It will also force departments to do more to help students find positions. At the end of the day what most prospective grad students want is to continue doing research after they finish their PhD. at a good school. I believe this is as important as the quality of research and the list of research areas of schools. An it is a quite objective and meaningful measure and is not easy to manipulate.
There are other factors that one should take into account when selecting a school for graduate studies and an adviser many of which are not academic issues. But the prospective of ending up in a good academic job plus the quality of research are the top academic measures. The second one includes the subjective evaluation of departments by experts in an objective way: how likely is your department to hire someone from each school?
At the end of the day what most prospective grad students want is to continue doing research after they finish their PhD. at a good school.
ReplyDeleteIs that so? Over the past few years several of the top graduates from the big four went voluntarily to industry. We have seen the same phenomenon at my institution (top 20), where some of our top graduates spurned academic offers to go to industry. These star candidates got industry offers in the range of $130-200K.