Brynjolfsson works extensively on metric-based decision making and gave early warnings on the loss of jobs to ML. I still have the HiPPO (Highest Paid Person's Opinion) from his EC 2010 invited talk. I like it for the complexity class on its back.In this article, Brynjolfsson and his co-authors use properties of the co-author and citation network graphs to predict future paper-citation based metrics, like the h-index for professors at top ranked OR departments. The paper claims that their metrics do a better prediction job than tenure committees.
The IHE article captures an alternative view:
Many professors oppose the use of bibliometrics in hiring, tenure and promotion decisions, saying that scholarly potential can’t be captured in a formula most often applied to pursuits with a bottom line, like winning games or playing the stock market. Such a system inevitably will be “gamed" by academics, critics say, and time-consuming but ground-breaking research will be sidelined in favor of “sure things” in terms of publishing -- the academic equivalent of clickbait.I have a different issue--the use of bibliometrics to judge the current value of a professor. In baseball, the original Moneyball, there are measurable goals: runs, wins, championships. So you can use data analytics to both measure the current and future potential of a player. In academics, what makes a great professor? I hope the sole answer isn't the h-index. And without a subjective method to measure a successful professor, how do you train a model to predict for it?
I'm not against using metrics to help with tenure and hiring decisions but I still put most of the weight on the letters.
The article does talk about predicting INFORMS fellows from the network centrality model, predicting a subjecting decision from objective statistics, though doesn't compare that to tenure decisions. I wonder how well one can predict ACM Fellows as well. Another challenge here: As CS is a constantly changing field, can one use an algorithm that predicts today's fellows from 20 year old data to predict future fellows from today's data?
A department needs to decide on an objective metric, and then it could (at least in principle) train a model that hires people to optimize that metric. Different departments might have different metrics. Maximizing total grant money might be a reasonable one. Maximizing reputation (i.e., rankings) seems to be one that a lot of departments care about.
ReplyDelete(FWIW, I think the discussion should focus on hiring rather than tenure since -- anecdotally, at least -- it seems that 90+% of faculty in CS get tenure once they are hired.)
Is baseball fundamentally a zero-sum game? The "Moneyball" answer is "yes' (literally), in the sense that zero-sum win-lose records are the foundation of Moneyball metrics.
ReplyDeleteThis immediately explains why the Chicago Cubs have fewer fans than any other team in baseball … oh wait, never mind.
Is teaching fundamentally a zero-sum game? Is mathematics fundamentally a zero-sum profession? The military literature provides an explicit algorithmic answer: universalize Gen. Victor Krulak's 1957 answer to the question "Why does the United States need a Marine Corps?" to "Why does <the world> need <profession>?".
For example:
----------------------------------------
Why does the world need mathematicians?
Mathematics exists today — flourishes today — not because of what we know we are, or what we know we can do, but because of what the world believes we are and believes we can do.
Essentially, because of the unblemished achievements of mathematics over centuries, the world believes three things about mathematicians.
First, they believe that when trouble comes to the world, there will be mathematicians — somewhere — who through hard work have made themselves ready to do something about it, and do it at once. They picture mathematicians as mature individuals — dedicated members of a serious professional community.
Second, they believe that when mathematicians bend their minds to a task, they invariably turn in a performance that is dramatically and decisively successful — not most of the time, but always. The world's faith and convictions in this regard are almost mystical. The mere association of the word ``mathematics'' to a challenge is an automatic source of encouragement and confidence everywhere.
The third thing that they believe is that training in mathematics is downright good for young people; that mathematicians are the masters of an unfailing alchemy that helps convert unoriented youths into proud, self-reliant stable citizens — citizens into whose hands the planet's affairs may safely be entrusted.
The people believe these three things. They believe them deeply and honestly, so much that they are willing to pay for mathematicians to solve problems, and to teach young people.
Therefore, for reasons that completely transcend cold logic, the world wants mathematicians. These reasons are strong, they are honest, they are deep-rooted, and they are above question or criticism. So long as they exist — so long as people are convinced that mathematicians can really do the three things I have mentioned — we are going to have a mathematical profession.
And likewise, should people ever lose that conviction — as the result of the mathematics community's failure to meet their high — almost spiritual — standards, the profession of mathematics will swiftly disappear.
Is there a chance that such a thing might happen? I think there is. I think that we ourselves can shake these convictions and the accompanying faith which really sustain us. By a lack of attention we can lose the inspirational personal relation that is shared between our senior members and our rank-and-file. Also, by carelessness or inordinate attention to less important things, we can lose the attributes of professional dedication and unfailing preparedness which, in centuries past, has deservedly made the mathematical profession one of humanity's treasures.
How serious it is, I don't profess to estimate to you, but it certainly worries me. It does, because if the world wanted to try, she could get along without professors of mathematics.
----------------------------------------
Now we are equipped to consider whether Moneyball-managed professions foster Krulak's three virtues. A charitable answer is "not obviously" … and my own opinion is more critical than that.