question but did not want them to guess. Hence I had +4 for a right answer, -3 for a wrong answer.
Here is the question:
------------------------------------------------------------------------------------------
For each of the following say if its TRUE, FALSE, or UNKNOWN TO SCIENCE. No Proof Required BUT you get +4 for every right answer and -3 for every wrong answer and 0 for an answer left blank. So
DO NOT GUESS!!!!!!!!!!!!!!!!!!!!!!!!!!!
a) If A is regular and F is a finite set then A UNION F is regular.
b) If A is in P and F is a finite set then A UNION F is in P.
c) If A is in NP and F is a finite set then A UNION F is in NP.
d) If A is decidable and F is a finite set then A UNION F is decidable.
e) If A is undecidable and F is a finite set then A UNION F is undecidable
-------------------------------------------------------------------------------------------------
Note that they are all TRUE. A student who (as many did) answered T-T-T-T-F had the following conversation with me:
BILL: You guessed! I told you DO NOT GUESS!!!!!!!!!!!!
STUDENT: No. I REASONED that you would not make them all T, so by this reasoning the last one had to be F. I now see that my reasoning is wrong--- you would make them all T, but it was not guessing, it was reasoning.
I claim the student was guessing, he claims he was not. What do you think?
Having said that, the following IS a rational strategy:
If I don't know the answer but it has nothing to with P vs NP then it has to be T or F. In this case guess since exp val is positive. If the answer has to do with P vs NP then do not guess.
This also raises a question- if they honestly thought (say) e was F I want to just give them 0, whereas if they are guessing or using reasoning about `Bill wouldn't ...' I want to give them -3. But alas, we cannot read their minds or souls.
I can see a few more issues. One is that if they know that they get an A already if they get 13 points, but a B for 12, then if someone knows only 3 answers, it's worth guessing for he last two.
ReplyDeleteI think I would make a quiz where you also have the option to say 'certainly not option a, but I don't know whether b or c.' Because, as you wrote, in your example it was worth guessing if they could eliminate one option. Scores could be:
Empty: 0
Correct: +5
Incorrect: -5
One option eliminated: +1
One option eliminated error: -5
Btw, I've just realized that no matter how you assign the scores, there will be students for whom it's worth guessing, if we allow someone to be 'only' 99% sure. Is it possible for anyone to be ever 100% sure? I guess we're all guessing...
Determining the internal mental state of a student from their response to a three-state prompt seems intrinsically difficult. Even if the measurement you want to make is RIGHT-WELL-REASONED/RIGHT-ILL-REASONED/WRONG, you need both an incredibly well-worded problem and incredibly well-worded distractors.
ReplyDeleteEveryone who designs and gives tests should, at some point, read the published validation study on some standardized test. Even if we're never going to achieve (or even attempt) that on the tests we write, it's a good way to notice just how hard it is to know what the data means.
Usually, to get a good internal-state measurement, you want each response the student could make to correspond to a single internal state. This isn't quite possible, but you can come closer than this item did.
In any case, proposing a true statement and asking a student to determine whether it is true, false, or unknown gives practically no measurement of whether the student has "guessed," under any interpretation of the term.
Doesn't attempting to match up answers to predicted internal states end up heavily advantaging students who think about the problems in unusual ways or ways that differ from those taught? If your thought process is unusual the wrong answers are less likely to correspond to a potential confusion you might have.
DeleteWouldn't it be more fair to not bother trying to guess at other internal states that might lead to wrong answers and instead just ask a sufficiently large number of questions and count on statistics to let you infer their ability to do whatever kind of problems you want grades to reflect?
Note that my question is presuming that (as in the post) the test is for the purposes of determining a final grade. Obviously, other tests may have the purpose of giving the teacher feedback about what various students do and don't understand rather than being primarily evaluative in which case I see a stronger case for each response corresponding to a single internal state.
DeleteBTW I tried searching standardized test validation study and I don't think I found what you were referring to...any hints?
Determining the internal mental state of a student from their response to a three-state prompt seems intrinsically pointless.
ReplyDeleteEither the student can correctly solve the problems you want them to solve, or they can't. Why does their internal state matter?
What if you wrote: "If A is decidable and F is finite, then A \ F is decidable" instead? It is essentially the same as (e) but I suspect more people would get it right. That said, it shouldn't be too hard for students who got the first 4 to see that A UNION F decidable => A decidable; but there's exam pressures etc so it makes sense many might miss it.
ReplyDeleteMore to the point, the student KNEW that a wrong answer would get -3; regardless of how much of a "guess" it was, they took that gamble. Reasoning on "Bill wouldn't..." seems to suggest a very low confidence in the answer, even if one wants to argue it doesn't constitute a guess. They knew the gamble they were taking, so they shouldn't complain --- on the other hand, if you chose to do -3 only in instances where a student guessed, then the student might have cause (and justification) to complain that it wasn't really a guess. But given they had enough confidence to risk the points loss, maybe that already suggests it wasn't a total guess? Does putting an answer imply that it wasn't a guess? What if the penalty was something more like -8? If it's too high (e.g. instant 0 for the entire exam) then no one will even bother answering (or maybe some would?), but as you said if it's less than the reward for right answers, it's a good strategy to guess (if the student has reasonably high confidence it isn't UNKNOWN TO SCIENCE).
If you really wanted to avoid guessing, why not ask for a brief justification? None of these would require much more than a few words or a sentence to explain. Then you can take penalize for any incorrect answer without justification.
Okay, how about the students decide the scores they get for each question based on their confidence? They not only select an option, but also a confidence level k, say between 1 and 10. If They get it right, they get k points, while if they err, they get, say -k^2.
ReplyDeleteWhat's the difference? Are you saying that if a student does not guess, but instead is simply wrong, you would not deduct points?
ReplyDeleteI'm a bit puzzled as to the underlying guess/no-guess distinction you have in mind. I mean anytime we answer questions, even in math, we reach some non-extremal probability assignment over the answers. It's always possible to make mistakes and students have a continuous range of confidence from vaguely remembering 'yah something like this is true for finite sets' to having what they believe is a proof. Moreover, it's surely reasonable for the student to sometimes inform their probability assignments by psychological regularities about tests.
ReplyDeleteSo how high a probability do you want the student to assign to the answer for it to no longer count as a guess?
More generally, why bother? If the goal is to evaluate student knowledge then I don't see why you can't just adjust the penalty for wrong answers and credit for correct ones so their performance (assuming they aren't dumb about strategy) reflects their level of knowledge?
I don't think your definition of a "Guess" is well defined.
ReplyDelete