Can You Tell AI Writing From Human Writing?

In one sentence: A study published in the journal Judgment and Decision-Making had people read AI- and human-written short stories and found that most couldn't tell which was which, and that what they liked was swayed less by the writing itself than by whether they believed a machine had written it.

Key Findings

• A live guessing game. Co-host Regina Barber and producer Berly McCoy each read a koi-pond short story aloud and asked guest host Mary Louise Kelly, of NPR's All Things Considered, to guess which was written by a human. The first was ornate: the koi's colors were "a mosaic of oranges, whites, and blacks," the fish "silent sentinels guarding the mysteries of life and death." The second was plainer and stranger: the narrator names three fish, in reverse order of size, Kurt Russell, Goldie Hawn, and Kate Hudson, "with no assignment of gender intended." Kelly liked the second one better and guessed it was the human-written one, though she flagged that the gender disclaimer "feels like something AI would have been programmed to be politically correct about." She went with her gut anyway. She was right: the first, image-heavy story was AI-generated; the second, with the named fish, was written by a person.
• The study behind the game. The segment drew on a set of experiments recently published in Judgment and Decision-Making, led by Deena Weisberg, an associate professor of psychological and brain sciences at Villanova University. In one experiment, participants read short stories, some human-written and some AI-generated, and rated them, while being told, correctly or incorrectly, which kind each one was. In another, participants were given a pair of stories, one human and one AI, and asked to identify which was which.
• Two findings. Most participants couldn't tell AI-generated stories from human-written ones. And they tended to prefer the AI-generated ones, especially when they believed, correctly or not, that those stories had been written by a human. "So what people said they preferred, human writing, and what they actually preferred were often different?" Kelly asked. Barber confirmed it, tying the result to a wider body of research showing that people's beliefs about AI involvement shape how they feel about a piece of writing. Kelly had a name for it on the spot: "expectation bias."
• Not being able to tell is itself the finding. "I personally feel confident in saying the time has passed when it is simple to just be able to tell what a human has written and what an AI has written," Weisberg said, attributing it to how much large language models have improved in the past few years.
• The variable nobody can ignore: length. Weisberg herself cautioned that the result might hinge on how short the stories were, around a thousand words each. Other researchers McCoy spoke with made the same point: length is often the key factor in spotting AI-generated text. Mohit Iyyer, an assistant professor of computer science at the University of Maryland who wasn't involved in the study, told the show that AI-generated writing tends to be formulaic, and that the longer a piece runs, the more that formula shows. "AI got nothing on Tolstoy," Kelly quipped.
• A possible reason people liked the AI stories better: they're easier to read. Weisberg's hypothesis: "The AI-written stories were very obvious and, I think, more predictable in their flow, which isn't surprising, given how AI generates text. And the human-written stories were more difficult to read. They were more subtle. They didn't spoon-feed you the themes in quite the same way." Barber's gloss: in a sense, the AI stories are just easier stories to read. Weisberg's team has posted the stories used in the study so people can judge for themselves; the show pointed listeners to its show notes for the link, which isn't reproduced in the transcript.

Scientific Background

The study makes two separate points, and it's worth keeping them apart. The first is about detection: when the same piece of writing was randomly labeled "human" or "AI," people's judgment of it tracked the label, not the text's actual origin, which suggests that a lot of what people think they know about "AI-sounding" writing is really a prior belief doing the work, not a reliable read on the text itself. The second is about identification without any label at all: given a genuine pair, one human and one AI, and asked to sort them, most participants still couldn't. Taken together, the explanation the segment offers is that AI-generated prose tends to be fluent, logically clean, and explicit about its themes, which is exactly what makes it both more pleasant to read and more recognizably formulaic; human writing tends to be more oblique, leaving more for the reader to do.

Length is the variable both the study's own author and an outside expert flagged. Weisberg and Mohit Iyyer, a University of Maryland computer scientist with no role in the study, independently pointed to the same mechanism: AI writing's formulaic quality compounds as a piece gets longer, so a roughly thousand-word short story sits right in the range where that gets least likely to show. That makes "people can't tell AI writing from human writing" a claim that holds up at short lengths in this study, not a length-independent finding. Nothing in the segment discusses automated detection software or algorithms; the entire conversation is about human readers' own judgment.

People

• Deena Weisberg — associate professor of psychological and brain sciences at Villanova University, senior author of the study. She reads the results as evidence of how far large language models have come.
• Mohit Iyyer — assistant professor of computer science at the University of Maryland, not involved in the study, brought in as an outside voice on AI text detection; his point is that formulaic tells accumulate as AI-generated text gets longer.
• Mary Louise Kelly — host of NPR's All Things Considered, guest hosting this Short Wave news roundup, and the one who correctly guessed which koi-pond story was human-written.
• Regina Barber / Berly McCoy — Short Wave's regular co-host and producer for this biweekly science news roundup.

Odd Details & Trivia

• The two koi-pond excerpts reward a close read: the AI-written one leans on nostalgic imagery ("a relic from my childhood" in "the heart of my mother's garden"), while the human-written one names its three fish, in reverse size order, Kurt Russell, Goldie Hawn, and Kate Hudson — in real life, Goldie Hawn and Kurt Russell are a couple, and Kate Hudson is Goldie Hawn's daughter — and jokes the fish might be "hiding among the hydrangeas, growing hydroponically."
• Kelly's own account of her reasoning: she was won over by picturing "Goldie Hawn swimming around," not by analyzing the prose. Her correct guess reads more like a lucky alignment of taste than a repeatable detection method.
• This episode is one installment of Short Wave's biweekly news roundup. The same episode also covers a "wandering" supermassive black hole caught devouring a star 750 million light-years away (reported by Jillian Rastinejad, Robert Stein, Priyamvada Natarajan, and colleagues in The Astrophysical Journal Letters), and a Current Biology study finding that vehicle-deer collisions were 76% lower in areas with more puma activity, because the pumas change deer behavior rather than simply thinning deer numbers. Neither is covered in depth on this page.
• Credits: produced by Hannah Chinn, Juan Diego Ramirez, and Jordan-Marie Smith, with Hannah Chinn also contributing reporting; edited by Rebecca Ramirez and Patrick Jarenwattananon; fact-checked by Tyler Jones; engineered by Ted Mebane and Kwesi Lee.

Cross-Domain Connections

The humanities-domain page 《黑镜》的哲学：让你脊背发凉的，从来不是科技本身 (on the philosophy of Black Mirror, specifically how technology mediates what people feel) has one concrete point of contact with this episode, not just a shared topic of "technology." That page draws on Guy Debord's claim that spectacle is "a social relationship between people that is mediated by images," meaning people's sense of a thing is shaped by what stands between them and it, not by the thing itself. This episode's study shows the same structure in a different material: participants' judgment of a story swings on the label "AI-written" or "human-written," not on the story's actual origin. One page reaches that point through film theory, the other through a psychology experiment on short-story readers. The parallel stops there: the Black Mirror page is ultimately about spectacle consuming the real, while this episode is about people currently being unable to tell AI writing from human writing and having their preferences skewed by that same inability. It would overreach to treat the two as making the same argument.

No other science-domain page currently covers related ground, so none is forced here.

Pages to Build

• expectation-bias — the mechanism by which a prior belief about something (like the label "AI-written") reshapes how a person actually experiences that thing, independent of what's really true. This episode is one concrete case; the mechanism is general enough that future science-domain episodes will likely invoke it again. Deena Weisberg's study itself and Mohit Iyyer's specific claims are treated as one-off content for this episode and aren't queued for standalone pages (better sparse than padded).

Source · shortwave