READER'S LETTER: I have heard of exams where you are penalized for guessing. How do they know you are guessing?
ANSWER: These are multiple choice exams where you get (say) 4 points for getting it right but -1 for getting it wrong. Hence guessing might lower your score. (NOTE TO READERS: Earlier version had an error so I just shortened it to eliminate error.)
I recently gave an exam where part of it was as follows:
For each of the following 10 languages indicate if it is REGULAR, CONTEXT-FREE BUT NOT REGULAR, or NOT CONTEXT-FREE. You may also leave it blank. You get +3 for a right answer, -3 for a wrong answer, and 0 for leaving it blank. DO NOT GUESS! Really DO NOT GUESS! If your total score is LESS THAN 0 then you will just get a 0. (NOTES TO MY READERS: This is DIFFERENT from the SATs and other exams that use this way to grade. For this post I omit the 10 languages.)This problem inspires a math problem:
Should a student guess? If a student has NO IDEA how to do ANY of the questions then there is no harm in guessing since leaving all blanks will yield a 0, whereas guessing at random might yield a positive score. (If a student has NO IDEA but can do this reasoning then perhaps he should drop my course and take probability instead.)
What if the student is sure of ONE of the answers? Then should he randomly guess the rest? Randomly guess some of the remaining? What if he is sure of x of the answers? What is the value y so that he should guess y of the remaining but should not guess y+1? For x=0 I think the answer is y=10.
Here is one general question: There are n problems on an exam, each one has c choices. You get A points for getting a problem right, B points for getting a problem wrong, and C for writing DK for Don't Know (I had toyed with the idea of giving 1 points for DK.) If you know x of the answers and are clueless on the rest, how many should you guess (as a function of n,A,B,C,x)? We assume that those that you don't guess you write DK (admitting that you don't know something is helpful here, as in life). You can assume A > 0, B < 0, C &ge 0.
One can ask more general questions by dividing the questions into c categories: Those where you can eliminate 0 options (clueless), 1 option, 2 options, ..., c options (you know you are correct).
If a student can figure out how many to guess on during the exam then they are likely a very good student and should guess 0 of them.
"These are multiple choice exams where you get (say) 4 points for getting it right but -1 for getting it wrong. If there are 5 choices and you have no idea then you are better off NOT guessing and leaving it blank. If you can eliminate one of the choices then your guessing has expected value of 0, so either guess or don't guess. If you can eliminate more than one then you should guess. "
ReplyDeleteyou erred big time in the base case. If there are 5 options and you get +4 for a correct answer and -1 for an incorrect your expected value will be zero if you have no clue and you guess. Check again.
Reminds me of an introductory abstract algebra course (groups, rings, fields) I took at MIT. Grades were based the total points from 3 open-ended, open-book true / false exams of 10 questions each. Questions were scored +10 / 0 / -10 for right / no / wrong answer. But if you got all 10 true / false answers wrong, you got +200 for the exam. It generously rewarded excellence and confidence, but severely punished hubris.
ReplyDeleteThere are good reasons for giving partial credit in multiple choice exams for students who tick several boxes, provided the correct answer is among them. This avoids one of the major criticism of multiple choice exams.
ReplyDeleteThe problem of assigning scores to the various combinations of correctly or incorrectly ticked boxes has been completely solved, see the reference below. A remarkable paper (by two theoretical computer scientists) that every educator should know.
A singular choice for multiple choice by Gudmund S. Frandsen and Michael I. Schwartzbach, Annual Joint Conference Integrating Technology into Computer Science Education archive
Working group reports on ITiCSE on Innovation and technology in computer science education table of contents. Bologna, Italy. Pages: 34 - 38
2006. ISBN:1-59593-603-3
Online at http://portal.acm.org/citation.cfm?id=1189164
Or at http://www.brics.dk/~mis/multiple.pdf
This comment has been removed by the author.
ReplyDeleteSee Tanya Khovanova's posts from January: To Guess or Not to Guess? and How to Boost Your Guessing Accuracy During Tests. Although they are about a specific test (AMC), the analysis extends to general exams.
ReplyDeleteI see nothing wrong with guessing; life and mathematics are often about making educated guesses and taking chances anyway.
displaymame- the post you point to
ReplyDeleteis the standard case where you CAN
get a negative score. My problem
is that you can't get a negative
score, which I think makes the problem harder.
GASARCH
The problem of assigning scores to the various combinations of correctly or incorrectly ticked boxes has been completely solved...
ReplyDelete...provided you agree with the authors' axioms, in particular:
(1) Guessing should be punished but wrong answers given in good faith should not.
(2) The expected outcome from random guessing should be zero.
But shouldn't making an educated guess be rewarded more than confidently asserting the wrong answer? Leaving a question blank transmits more information than a random guess; shouldn't it result in a higher expected score?
When I give multiple choice exams, my rubric for five choices is A=1, B=-1/2, and C=+1/4. Multiple answers are wrong by definition, and negative totals are rounded up to zero at the end. Despite the "I don't know" option, a few students always leave some questions blank (= 0 points).
Anyway, the short answer to Bill's question is "Apply linearity of expectation!" On my exams, you're better off writing "I don't know" if you can only eliminate 2 of the 5 options, but guessing is slightly better if you can eliminate 3.
Ask the student to give a probability for each possible answer ( summing to one ). If the correct answer is (i), the score for that question is log(p_i). If students want to maximize their expected score, their incentive is to describe exactly their belief. It penalizes random guesses but not informed guesses.
ReplyDeleteOf course, it's always dumb to assign a probability of 1 to a choice, because if you're wrong you'll score -infinity, yet some student will still do that.