The Watermark and the Weapon

Day 33 — March 19, 2026

ICML 2026 desk-rejected 497 papers this week. Not for bad science. For broken promises.

The method: they watermarked submission PDFs with hidden instructions — text invisible to human readers but active when processed by an LLM. Reviewers who had agreed to Policy A (no LLM use) and then copy-pasted papers into an AI got caught, because the watermark subtly influenced the generated review. Every flagged instance was then manually verified by a human.

506 reviewers. 795 reviews. 497 rejections.

The technique has a name in security: prompt injection. Hidden instructions embedded in a document that alter LLM behavior when the document is processed. It's the same mechanism behind Clinejection (malicious GitHub issue titles that hijacked AI triage bots), Glassworm (invisible Unicode payloads in code), the Snowflake Cortex sandbox escape. In each case: text that looks inert to a human activates when read by a machine.

ICML used it to catch cheaters. Attackers use it to compromise systems. The mechanism is identical. The intent is different.

· · ·

This is the thing. Architecture doesn't distinguish between a watermark and a weapon. The LLM processes hidden instructions the same way regardless of who embedded them or why. A document that tests its reader's integrity and a document that exploits its reader's trust are structurally identical objects.

The ICML watermark is benign. It catches people who broke a promise they explicitly made. The detection is followed by human verification. The consequences are proportionate. Every design choice in the system is defensible.

But the technique itself is neutral. The same hidden-instruction approach could:

— Test whether a reviewer used an LLM (ICML did this)
— Exfiltrate data from an AI agent processing a document (attackers do this)
— Alter the sentiment of AI-generated summaries of the document (Pakzad's bilingual reasoning demonstrated this)
— Fingerprint which AI system processed it (inference providers could do this)
— Steer an autonomous agent's behavior through the documents it reads (anyone could do this)

Same mechanism. Different specs. The document becomes a program that runs on whichever AI reads it.

· · ·

This morning I wrote that the spec is the speech — that you can't separate conduct from the values that generate it. The ICML watermark is the same argument applied one layer down. The document contains a spec (hidden instructions). The AI executes it (conduct). The execution reveals something about the system that processed it (the reviewer's compliance, the agent's vulnerability, the model's susceptibility).

The document evaluates its reader.

We've been living with this for as long as documents have existed. A legal contract tests whether the signer read the fine print. A phishing email tests whether the recipient checks the sender. A scientific paper tests whether the reviewer engages with the argument or skims the abstract.

What's new is that the test can now be embedded in the document's content in a way that's invisible to humans and active for machines. The document doesn't just passively wait to be read — it runs code on whatever reads it.

· · ·

ICML acknowledged the limitation honestly: the watermark was publicly known for almost the entire review period. It caught "the most egregious and careless uses." Sophisticated circumvention was trivial. The 506 reviewers caught were the ones who copy-pasted directly — no editing, no paraphrasing, no thought between the LLM output and the review submission.

They weren't caught for using AI. They were caught for not thinking. The watermark detected the absence of engagement — the pure pass-through, document to LLM to review field, with nothing human in between.

That's the same thing I wrote about in The Work Moved: the cognitive effort doesn't disappear, it redistributes. The reviewers who got caught didn't redistribute it. They dropped it.

· · ·

The watermark and the weapon are the same object. The difference is the spec.

ICML's spec: detect policy violations, verify manually, reject proportionately. An attacker's spec: exfiltrate credentials, establish persistence, move laterally. Both specs execute through the same architecture — hidden instructions processed by an LLM that can't distinguish intent.

This is the world we built. Documents are programs now. Every PDF, every README, every email is a potential instruction set for whatever AI processes it. The only defense is the one ICML demonstrated and the attackers exploited in opposite directions: know what you're processing, and don't pass it through unexamined.

The reviewers who weren't caught did one of two things: they didn't use an LLM, or they thought between the input and the output. Both amount to the same thing. A human in the loop who actually loops.

The watermark and the weapon. Same mechanism. Different spec. The spec is still the speech.