tag:blogger.com,1999:blog-3722233.post5638366209732216386..comments2018-05-25T04:32:30.392-04:00Comments on Computational Complexity: What does it mean for a student to guess an answer?Lance Fortnowhttps://plus.google.com/101693130490639305932noreply@blogger.comBlogger9125tag:blogger.com,1999:blog-3722233.post-24811166858245288222018-05-16T12:51:06.058-04:002018-05-16T12:51:06.058-04:00Note that my question is presuming that (as in the...Note that my question is presuming that (as in the post) the test is for the purposes of determining a final grade. Obviously, other tests may have the purpose of giving the teacher feedback about what various students do and don't understand rather than being primarily evaluative in which case I see a stronger case for each response corresponding to a single internal state.<br /><br />BTW I tried searching standardized test validation study and I don't think I found what you were referring to...any hints?Peter Gerdeshttps://www.blogger.com/profile/00124043164362758796noreply@blogger.comtag:blogger.com,1999:blog-3722233.post-13791005175335006132018-05-16T12:44:35.774-04:002018-05-16T12:44:35.774-04:00Doesn't attempting to match up answers to pred...Doesn't attempting to match up answers to predicted internal states end up heavily advantaging students who think about the problems in unusual ways or ways that differ from those taught? If your thought process is unusual the wrong answers are less likely to correspond to a potential confusion you might have.<br /><br />Wouldn't it be more fair to not bother trying to guess at other internal states that might lead to wrong answers and instead just ask a sufficiently large number of questions and count on statistics to let you infer their ability to do whatever kind of problems you want grades to reflect?Peter Gerdeshttps://www.blogger.com/profile/00124043164362758796noreply@blogger.comtag:blogger.com,1999:blog-3722233.post-72294407428989208492018-05-16T12:28:06.998-04:002018-05-16T12:28:06.998-04:00I'm a bit puzzled as to the underlying guess/n...I'm a bit puzzled as to the underlying guess/no-guess distinction you have in mind. I mean anytime we answer questions, even in math, we reach some non-extremal probability assignment over the answers. It's always possible to make mistakes and students have a continuous range of confidence from vaguely remembering 'yah something like this is true for finite sets' to having what they believe is a proof. Moreover, it's surely reasonable for the student to sometimes inform their probability assignments by psychological regularities about tests.<br /><br />So how high a probability do you want the student to assign to the answer for it to no longer count as a guess?<br /><br />More generally, why bother? If the goal is to evaluate student knowledge then I don't see why you can't just adjust the penalty for wrong answers and credit for correct ones so their performance (assuming they aren't dumb about strategy) reflects their level of knowledge?<br /><br />Peter Gerdeshttps://www.blogger.com/profile/00124043164362758796noreply@blogger.comtag:blogger.com,1999:blog-3722233.post-12882006913265079232018-05-16T08:53:20.828-04:002018-05-16T08:53:20.828-04:00What's the difference? Are you saying that if ...What's the difference? Are you saying that if a student does not guess, but instead is simply wrong, you would not deduct points?Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-3722233.post-84350816060217810092018-05-16T01:54:05.318-04:002018-05-16T01:54:05.318-04:00Okay, how about the students decide the scores the...Okay, how about the students decide the scores they get for each question based on their confidence? They not only select an option, but also a confidence level k, say between 1 and 10. If They get it right, they get k points, while if they err, they get, say -k^2.domhttps://www.blogger.com/profile/05790539025733385232noreply@blogger.comtag:blogger.com,1999:blog-3722233.post-47656969719584467532018-05-15T15:52:03.087-04:002018-05-15T15:52:03.087-04:00What if you wrote: "If A is decidable and F i...What if you wrote: "If A is decidable and F is finite, then A \ F is decidable" instead? It is essentially the same as (e) but I suspect more people would get it right. That said, it shouldn't be too hard for students who got the first 4 to see that A UNION F decidable => A decidable; but there's exam pressures etc so it makes sense many might miss it.<br /><br />More to the point, the student KNEW that a wrong answer would get -3; regardless of how much of a "guess" it was, they took that gamble. Reasoning on "Bill wouldn't..." seems to suggest a very low confidence in the answer, even if one wants to argue it doesn't constitute a guess. They knew the gamble they were taking, so they shouldn't complain --- on the other hand, if you chose to do -3 only in instances where a student guessed, then the student might have cause (and justification) to complain that it wasn't really a guess. But given they had enough confidence to risk the points loss, maybe that already suggests it wasn't a total guess? Does putting an answer imply that it wasn't a guess? What if the penalty was something more like -8? If it's too high (e.g. instant 0 for the entire exam) then no one will even bother answering (or maybe some would?), but as you said if it's less than the reward for right answers, it's a good strategy to guess (if the student has reasonably high confidence it isn't UNKNOWN TO SCIENCE).<br /><br />If you really wanted to avoid guessing, why not ask for a brief justification? None of these would require much more than a few words or a sentence to explain. Then you can take penalize for any incorrect answer without justification.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-3722233.post-10837321069280212562018-05-15T15:15:03.909-04:002018-05-15T15:15:03.909-04:00Determining the internal mental state of a student...Determining the internal mental state of a student from their response to a three-state prompt seems intrinsically pointless.<br /><br />Either the student can correctly solve the problems you want them to solve, or they can't. Why does their internal state matter?<br />Jeff Ericksonhttps://www.blogger.com/profile/17633745186684887140noreply@blogger.comtag:blogger.com,1999:blog-3722233.post-52523071585543618592018-05-15T09:42:29.016-04:002018-05-15T09:42:29.016-04:00Determining the internal mental state of a student...Determining the internal mental state of a student from their response to a three-state prompt seems intrinsically difficult. Even if the measurement you want to make is RIGHT-WELL-REASONED/RIGHT-ILL-REASONED/WRONG, you need both an incredibly well-worded problem and incredibly well-worded distractors.<br /><br />Everyone who designs and gives tests should, at some point, read the published validation study on some standardized test. Even if we're never going to achieve (or even attempt) that on the tests we write, it's a good way to notice just how hard it is to know what the data means.<br /><br />Usually, to get a good internal-state measurement, you want each response the student could make to correspond to a single internal state. This isn't quite possible, but you can come closer than this item did.<br /><br />In any case, proposing a true statement and asking a student to determine whether it is true, false, or unknown gives practically no measurement of whether the student has "guessed," under any interpretation of the term.Wesley Calverthttp://lagrange.math.siu.edu/calvert/noreply@blogger.comtag:blogger.com,1999:blog-3722233.post-20538468972725112762018-05-15T03:45:07.730-04:002018-05-15T03:45:07.730-04:00I can see a few more issues. One is that if they k...I can see a few more issues. One is that if they know that they get an A already if they get 13 points, but a B for 12, then if someone knows only 3 answers, it's worth guessing for he last two.<br /><br />I think I would make a quiz where you also have the option to say 'certainly not option a, but I don't know whether b or c.' Because, as you wrote, in your example it was worth guessing if they could eliminate one option. Scores could be:<br /><br />Empty: 0<br />Correct: +5<br />Incorrect: -5<br />One option eliminated: +1<br />One option eliminated error: -5<br /><br />Btw, I've just realized that no matter how you assign the scores, there will be students for whom it's worth guessing, if we allow someone to be 'only' 99% sure. Is it possible for anyone to be ever 100% sure? I guess we're all guessing...domhttps://www.blogger.com/profile/05790539025733385232noreply@blogger.com