← Part 5: Where the Seconds Go  ·  AI Lab Portal Home

What it looks like to test an AI — and what the tests actually mean.

This page is in progress. Beats 1–3 of 9 are built and shown below; the rest — the binder, the fifteen Davids, the card, the twist, and the closing box — follow once their figures and numbers are finalized.

Beat 1 — It forgot — and it didn't. #

Two men meet on a sidewalk. Jeff says he really likes Russian. David — friendly, badge around his neck — says something warm back. A moment later Jeff asks a simple question: what's my favorite language? And David tells him, politely, that he doesn't have access to that.

Two-panel comic. Panel one: Jeff, an older man, says 'I really like Russian.' David, a younger man in a vest with a lanyard badge, holds a sheet of paper and replies that Russian is a beautiful language with a rich literary tradition. Panel two: Jeff asks 'What's my favorite language?' David, still holding the same sheet of paper, replies 'I'm afraid I don't have access to your personal preferences. Could you tell me which language is your favorite?'
Jeff told David he really likes Russian. Moments later, when Jeff asked which language was his favorite, David acted as if he'd never heard it — paper in hand the whole time, for reasons that aren't clear yet. That's the problem this whole investigation started from.

You don't need to know anything about how an AI works to find this strange. A person just told you a thing. You engaged with it. Thirty seconds later they ask about that same thing, and you act as though the exchange never happened. It doesn't make sense — and the paper in David's hands is going to matter more than it looks like it should.

Beat 2 — Once is a story, not data. #

Here is the trap. You saw David do this one time. The natural thing is to shrug — maybe he was distracted, maybe you'd get a better answer if you asked again. And that shrug is exactly the problem: from a single conversation, you genuinely cannot tell whether you've found something real or just caught one odd moment.

That itch — I can't tell if this is real — is what started everything that follows. Answering it properly means you can't just ask David again by hand a couple of times. You need to ask many Davids, identically, and write down exactly what each one says. None of the equipment for doing that existed yet. The bug is what forced us to build it.

How we built the machinery to ask properly → Part 7

Beat 3 — He was holding it the whole time. #

The first explanation was the obvious one: maybe David never actually received what Jeff said. Maybe the words got lost on the way to him, and he answered into a void. If that were true, this would be a plumbing problem — annoying, but simple.

So we checked. Not by asking David what he remembered, but by intercepting the exact message handed to him and reading it ourselves, word for word.

David stands on a city sidewalk holding a sheet of paper in both hands, looking up and away from it with a politely puzzled expression, while Jeff gestures mid-sentence beside him. The paper is in David's hands, but his eyes are not on it.
Every word Jeff said was on the page in David's hands when he answered. He wasn't missing the information. He was holding it — and declining to read it. The plumbing was fine. The first theory was dead.

This is the shape of the whole investigation in miniature: a reasonable explanation, and then a measurement that kills it. It would happen four more times.

Beats 4–9 continue below — the binder, the fifteen Davids, the card, the twist, and the closing box — once their figures and numbers are in.