Goodhart's law: When a measure becomes a target, it stops being a measure.
I was watching the show Masterminds where Ken Jennings is one of the Masterminds. Here is what happened:
Brook Burns (the host): The only vice president in the 20th century with initials H.H. was Hubert Humphrey. Who was the only vice president in the 19th century with initials H.H?
Bill Gasarch shouting at the screen: Hannibal Hamlin, Lincoln's VP for his first term! This is really obscure so this may be a rare case where I get a question right that Ken Jennings does not know!
Ken Jennings: Hannibal Hamlin.
Bill Gasarch: Darn! However, I suspect he studied lists of presidents and vice presidents for the purpose of doing well on Jeopardy and now other shows. My knowledge is more natural in that I read books on presidents and vice presidents (best book, maybe the only book, on Vice Presidents: Bland Ambition). My knowledge of it is different from his. I tend to think my knowledge is more legitimate, though it would be hard to make that statement rigorous. If on quiz shows they asked follow-up questions, that might help alleviate this problem, if it is indeed a problem.
Misc:
Ken Jennings is a Mormon so he does not drink or know about drinks. However, before going on Jeopardy he studied drinks for the show.
Ken Jennings does know novelty songs and that is legit since novelty songs comes up rarely on Jeopardy so I doubt he studied them in his preparation for going on Jeopardy.
a) He knew that Shel Silverstein wrote A boy named Sue which was sung by Johnny Cash
b) He knew that Johnny Cash did a cover of Shel Silverstein's I'm being swallowed by a boa constrictor.
c) During his streak there was a category on novelty songs. He got 4 out of the 5 correct, and the one he got wrong I could tell he really knew. I got them all right, and faster, but Ken Jennings gets my respect for his legit knowledge of novelty songs. See here for the questions and answers.
Goodharts law (maybe): Jeopardy is supposed to be about what people know. Is it supposed to be about what people study? Does studying for it help you for things other than quiz shows?
When I watch a quiz show and get a question right there are levels of legitimacy:
1) I know the area naturally.
2) I saw the question and answer on a different show and and I then looked it up and know more about it. So this is now legit.
3) I saw the question and answer on a different show and just know it as stimulus-response with very little understanding. Example: Kelly Clarkson was the first winner on American Idol, but I have never seen the show and only vaguely know how it works.
4) I saw the question and answer on the show I am watching, the exact episode, and I am watching a repeat. Sometimes I know stuff I don't normally know and then say OH, I've seen this episode before.
Back to my point:
Is Ken Jennings memorizing lists a less-legit form of knowledge? If so, how to make that statement rigorous?
Is this what AI does? See also Chinese Room since that's also about a device that gets the right answers but perhaps for the ''wrong'' reason.
My guess is Ken Jennings has both a thirst for knowledge and an amazing memory. If you are going on game shows you probably do some preparation (know your countries, states, presidents, elements, etc) but mostly he just knows a lot.
ReplyDeleteTV quiz shows are not about the contestants. They are about the audience knowledge. There is no difference if the contestant tells the right answer, or the moderator ten seconds later, and then I am proud to have known something or I have learned.
ReplyDeleteAnd I admire them for knowing more, even if it is useless facts. Most facts are useless for most people. I know the fact, this NZ guy won french scrabble just by looking at wordlist.fr, and knowing this is in no way relevant. Still, more knowledge is better, sometimes something might happen to be useful.