Bad brainwaves: ChatGPT makes you stupid
In a new study, MIT Media Lab measured 55 people over four months on how well they could write an essay — either with ChatGPT, with a search engine, or just their unassisted brain. The researchers hooked them to an EEG to see which parts of their brains were active or not. [arXiv, PDF]
The ChatGPT group had less interconnected brain activity. Specifically, they didn’t use the parts of their brains that normally engage in language processing, working memory, or creative writing as much as the other groups.
ChatGPT users “consistently underperformed at neural, linguistic, and behavioral levels.” ChatGPT makes you stupid.
Only 20% of the chatbot group could remember a quote from their own essay. Shown their own essay afterwards, 16% of the chatbot group denied it was theirs.
That shouldn’t be surprising — if you’re not doing the work, you won’t remember the work.
But then the researchers switched the groups up. The search-engine and just-their-brain groups did just as well when they switched to using the chatbot. But the chatbot group didn’t get any better when they switched to search-engine or just their brains. They were still stuck thinking as badly as they did when they were trying to write with the chatbot.
This strongly suggests it’s imperative to keep students away from chatbots in the classroom — so they’ll actually learn.
This also explains people who insist you use the chatbot instead of thinking and will not shut up about it. They tried thinking once and they didn’t like it.
“They tried thinking once and they didn’t like it.”
*Chef Kiss*
I’d approach this result with a certain amount of caution.
Studies of this nature are notoriously difficult to replicate, especially due to the small sample size (55) and somewhat select pool of participants (elite Boston universities) [see section on Participants].
Given that all the participants were relatively well educated and almost 40 percent of them either in graduate programs or having completed them, it’s difficult to believe that they had never written an essay of any sort before and/or that an exposure or two to an LLM have the effect claimed.
Admittedly, I’ve only skimmed the paper but it’s also not clear whether the participants had been exposed to LLMs before the study (highly unlikely given the selection pool), what their use levels might have been or how these confounders have been factored into the study analysis.
Oh, and if you check the NLP ANALYSIS section of the paper, they say they used Natural Language Processing (a branch of AI) and LLM agents to “generate classifications of texts produced, as well as scoring of the text by an LLM as well as by human teachers.”
One final note: The lead author of the paper went to work for Google, a competitor of OpenAI in the LLM business [see Conflict of Interest section].
I messed up the sense of the 4th paragraph. I meant the parenthetical to read “highly likely” and not “highly unlikely.”