A Detector Score Is Not a Verdict

On 23 October, El País reported that the text detector Pangram scored a prize-winning Spanish-language novella as 99% AI. The novel, David Silva Rojas's Donde el agua trabaja, had just taken the Premio Internacional de Novela Breve Almadía–Ventosa Arrufat, worth €20,000, in a contest whose rules forbid AI. The jury picked it from 2,204 manuscripts submitted from 22 countries, praising the subtlety of its writing and its use of silence. The author denies using AI. His publisher suspended the book's launch events while it reviews the case, and the novel is already being translated.

I am not going to tell you whether that book was written by a person. I cannot, and neither can a detector. What I can do is describe what the number actually is, because a private company's classifier has quietly become the referee of literary authorship, and almost nobody involved in these disputes seems to have read the method section.

What the tool returns

Pangram is a neural classifier trained on human text and generated text, with published techniques for suppressing false positives. It scores passage by passage and returns a percentage: an estimate of how much a text resembles what a machine would produce. That is a similarity judgement, not a reconstruction of how the text came to exist. The distinction sounds academic until it is deciding whether someone keeps a prize and a reputation.

The vendor's numbers are genuinely good. Pangram says version 4 catches 99.66% of AI text and flags 0.0041% of human text, about one error per 24,000 documents. A University of Chicago study (NBER working paper 34223) ranked it above GPTZero and Originality.ai. El País ran pre-ChatGPT books through it and got 0%. That is why editors reach for it, and it is not nothing.

The other half of the record is less quoted:

• Nature reported that when AI substantially revises a human essay, Pangram labels the result human 41% of the time.
• Jot Down showed that adding a single space can flip a verdict.
• A recent preprint puts the miss rate above 96% once text has been run through a "humanizer".
• The false-positive figure is measured on millions of pre-2022 web documents, not on novels, and not on people who have spent three years writing alongside these tools.

Read those two lists together and you get a tool that is excellent at the average case and unreliable at exactly the cases that go viral.

The error is not symmetrical

Here is the part I would want every publisher and prize committee to read twice. A false positive names a writer who did nothing, in public, and a correction does not undo it. A false negative lets a prohibited entry through, which does cost the other entrants and the institution a real decision. Those harms are not equivalent, though, and they do not land on the same people: one is a reputational injury to a specific person, the other is a call made on incomplete information. Every incentive in the current news cycle points toward flagging.

That asymmetry is what turned an anonymous account on X into a scandal. In the Goncourt case, someone ran fragments of a shortlisted novel through Pangram, posted the result, and a national literary institution had to respond. It later emerged that the author had in fact used AI, which is being read as proof that the tool works. It is not. One confirmed hit tells you the tool can be right. It tells you nothing about how often it is wrong in the other direction, because the false positives do not get a press cycle. They just get a suspended book tour.

A percentage is not a division of labour

The number cannot distinguish between a model drafting the prose, a model correcting commas in prose the author wrote, and a model suggesting a plot turn that the author then executed herself. All three could produce the same score, and all three mean different things under different rules.

That matters because the rules in most competitions are vague. "No AI" can mean no generated text, no generated text without disclosure, or no generation at all including research. A detector cannot adjudicate a rule that was never written precisely.

The referee sells refereeing

Pangram Labs is a small company with a product that gets more valuable as trust in writing collapses. According to The Decoder, citing the New York Times, its revenue grew 35-fold in a year and monthly users went from 2,700 to 120,000 between June 2025 and June 2026. That does not make the team dishonest. The point is narrower: interested parties do not get to certify their own error rates in a dispute about someone's livelihood.

There is a reproducibility problem too. Pangram 3 carried a retirement date of 30 September. If a verdict from August is challenged in November, the model that produced it may not exist any more, which is a strange property for something being treated as evidence.

In court, the report has to be explained

Spain has no ruling on the evidentiary value of these detectors. In criminal procedure, admissibility depends on relevance and the judge weighs the evidence freely. A Pangram report would have to arrive as an expert opinion, with someone able to explain the method, state the margin of error, and sit through cross-examination.

Three things break immediately under that treatment. It is a black box that cannot explain why it flagged a specific sentence. Its error rate is declared by the party selling it. And its models change under you, so the finding is not reproducible. The most a report like this can honestly support is a reason to ask for drafts. It is not proof of anything on its own.

Europe is trying a different route

Article 50 of the EU AI Act applies from 2 August 2026. Under Article 111(4), providers of systems that generate synthetic audio, image, video or text and were placed on the market before that date have until 2 December 2026 to meet Article 50(2)'s duty to mark outputs in a machine-readable format. The Commission published a code of practice in June. That approach moves the question from "does this text look generated" to "was this text generated by a marked system".

It is better, and it is not sufficient. The duty falls on providers of AI systems that generate synthetic content, and watermarking is only one technique for meeting it. A watermark proves nothing by itself, open-weight or open-source release does not exempt a system from the obligation, and none of it survives text that has been rewritten through another tool. Marking catches the cooperative. Detection tries to catch everyone, including the innocent, which is where the trouble starts.

The record is the defence

The only real defence against a score is a record of the work. Not a defence of your character, not a statement about your process, a record.

• Dated drafts and version history, exported somewhere you control, not only inside one app's cloud.
• Project files: prompts, seeds, layer stacks, timelines, the actual documents the work came out of.
• A short written note per project of which tools did which step, including the boring steps.
• For published work, the same evidence in the same place, so that a challenge six months later does not require archaeology.

That last point is the one people skip. Our piece on how awards are starting to judge process instead of output argues the same thing from the other direction: the pipeline is becoming the thing being assessed, so it is worth being able to show it.

What a fair process needs

If institutions are going to use these tools, four conditions would make the use defensible:

1. A threshold published before entries close, so nobody is disqualified by a rule invented after the fact.
2. A right of reply, before any finding becomes public.
3. A named method with a stated error rate and a version number, disclosed to the accused.
4. A human being accountable for the final call, who can be asked why.

None of that is exotic. It is just ordinary adjudication, applied to a new kind of claim.

The uncomfortable conclusion is that Pangram is probably the best detector available and still nowhere near good enough to be a judge. If you make images or video rather than prose, the mechanism differs but the shape of the problem does not. Provenance metadata, invisible watermarks, and classifier verdicts are all being asked to answer a question about intent that no classifier can see. The watermark explainer goes through the technical version of that limit. Keep your working files. They are the only part of this that a percentage cannot argue with.