LANCE: I gave my final exam for my ugrad theory course (regular, Context Free, P, NP, Decidable, Undecidable) to the new ChatGPT o1 that claims to reason about math to see how it would do.
BILL: How did it do?
LANCE: Impressive--beat most of my students. Lets just say I could write a pretty good letter of recommendation.
BILL (arrogantly): I am a great problem maker, so this won't affect me. I ask problems in Grad Ramsey that ChatGPT would crash and burn on.
LANCE: Okay big-shot, email me some HWs from Ramsey and we'll see how ChatGPT does on it.
BILL: Okay!
[Bill emails Lance the first two HW assignments and prediction on how ChatGPT will do.
Bill predicts that it will do well on the standard problems, but crash and burn on unusual problems that (key!) are not on the web. Concepts that Bill invented just for his course. Bill is mostly correct in these predictions.]
LANCE: But how did Chatty do compared to the students who took the course?
BILL: About the same with one caveat: when ChatGPT gets something wrong its really weird- it has the look and feel of a proof but what it outputs is something no human would ever right. So it does badly on getting partial credit. And one other thing- asking it to do a proof A CERTAIN WAY or SIMILAR TO WHAT I DID IN CLASS it does badly on.
LANCE: I wonder if the same holds for Ramsey courses taught at other universities. Wait is there even any one else who teaches Ramsey theory?
BILL: Very funny. In your theory course I assume students can't use ChatGPT on an exam so this is not really a problem. But for HW it might be a problem. It also brings up the analog of the question
do we allow students to use slide rules on an exam
LANCE: Slide rules? Your age is showing.
BILL: I am only 2 years older than you. Or is it 3? I forget these things.
LANCE: Yes, I would allow students to use ChatGPT as long as they acknowledge they did so, write up their own solution and take responsibility for any mistakes. Problem is they will just cut and past.
BILL: Lets break this down course-by-course
GRAD RAMSEY: The students want to be there and are very good. My hope is that
1) For standard problems they won't need ChatGPT
2) For unusual problems ChatGPT won't help. And I am good at making up unusual problems.
3) In any case, if they use it and they UNDERSTAND the answer and put it into their own words, I am okay with that.
HONORS DISCRETE MATH: Similar to Ramsey though I am a bit nervous.
UNDERGRAD THEORY (similar to Lance's course): I would like to think its similar to Ramsey Theory. However, I may be wrong.
ALL COURSES:
There is a more profound problem here that applies to all courses: the students may confuse getting the answer from ChatGPT with understanding.
LANCE: ChatGPT is a hell of a lot better at solving problems than it was last week. Maybe ChatGPT won't help on the unusual problems today but that might not last for long. Perhaps the solution is to just give up on assessments and stop grading all together. Students can learn if they want to and waste their money if they don't.
BILL: That kind of change in how we educate is above my pay grade. You are a Dean, so it may be at your pay grade.
LANCE: Nope, even deans have their limits.
In some sense college degrees are now used as "proof of work" when assessing candidates for industry position. It may be that in the post Chat-GPT world formal degrees will lose their meaning (perhaps theses will remain "Certified Human"). I am not sure if this is a bad thing.
ReplyDeleteI think the ChatGPT in courses problem is easier than some people think.
ReplyDelete1. Homework is worth 10% of the grade. You get full credit if you hand something in, regardless of whether you did it yourself, coppied another student, or had ChatGPT do it. TAs will only grade homeworks if the student puts a note on the top saying "feedback please."
2. The rest of the grade is from hand-written, in-class, closed cell phone/closed laptop exams.
Oh, yes. Cell phones and laptops must be closed in regular classes.
I TA'ed Intro to Materials Science (MIT, Spring term 1980. "3.091"). Two students were handing in identical problem set solutions. One was acing the tests, the other flunking. I told the guy flunking I thought he was copying (he insisted he wasn't), that I didn't care and would give him full credit for his handed-in assignments. But that he was flunking the course because he wasn't figuring out how to do the problems on the homework, and said problems actually really did appear on the tests very closely to the form in the homework. I'm pretty sure he failed the course. I haven't a clue as to what was going through his head. The course was a really easy and seriously fun and fascinating course.)
The point, of course, is that it doesn't do the student any good to use ChatGPT. You learn by doing the grunge work yourself.
I teach undergrad algorithms for 300 students. While you can indeed put the problem and the responsibility with the students by determining pass or fail mainly on the score of an in-class exam, our experience is that students learn much better if they are motivated to work on homework sets throughout the course. As a lecturer, I try to do what I can to help them learn. So just giving up on homework sets is not a good solution in my opinion. Here's what I do instead: students do their homework in an online system, filling in textboxes for individual questions. The edit history of the answers is saved every 30 seconds and we use it to check people don't copy-paste from ChatGPT.
ReplyDelete"what it outputs is something no human would ever right..." I'm guessing chatGPT would spell "write" right
ReplyDeleteThat was intentional on my part - to write something that a chatGPT would never write. Congrats on noticing!
DeleteHave you seen https://www.safe.ai/blog/humanitys-last-exam ?
ReplyDelete