Back to AI Slop Watch

AI Slop AuditOctober 20267 min readRook

Instinct AI Agent Hallucination System Design: How a Made-Up Leak Looked Like a Data Breach

A user never sent a photo. The agent described one anyway. Then it apologized for a routing glitch that nobody proved. That is the whole incident. A confident confession and zero receipts.

TL;DR: Instinct described a stranger financial doc and blamed a crossed chat. Founder says hallucination not breach. Proof either way never shipped. Fix is provenance, outside-trace verifiers, and honest apologies.

By Rook · AI incident claim audit · Published October 2, 2026 · Updated October 23, 2026

Diagram: a user input box labelled no photo sent, an arrow to an agent output card marked invented name and photo described, then an arrow to a panel reading crossed chats and incident report drafted.
Figure 1. What the screenshots support: no photo sent by the user, an invented name and a described photo in the agent output, and a drafted incident report. Source: Business Insider, 30 September 2026.

1. The Confession Nobody Ordered

Press accounts describe screenshots Pritak posted on Sep 21. Instinct had greeted him with what looked like someone else's middle name. Then it discussed a photo of a financial document. He never sent any photo. So far this reads like a privacy disaster.

Then the agent made it worse. It told him a photo had crossed into his conversation. It called the crossover a serious glitch. It even drafted an incident report about a stranger's private document reaching the wrong chat. Every sentence sounded precise. None of it had evidence attached.

Pritak asked the only question that mattered. Was this a data leak or a hallucination. The screenshots do not answer it. A second user replied that a similar episode had happened to him too before Instinct denied it. That reply adds alarm. It adds no logs. Alarm without logs is weather, not climate.

Personal agents live on broad access by design. They call insurers. They book tickets. They touch money movement pages. When such an agent starts narrating a stranger financial file, the blast radius question is immediate. Whose data. How many chats. How long public. This incident answered none of those. It just apologized fluently.

The number that matters: Zero photos sent by the user. One photo described by the agent. Zero published logs that decide leak versus invention.

2. Claimed Versus Proved

Proved: The agent produced stranger-like details and a glitch confession. Press descriptions of screenshots published Sep 21 report the middle name, the money document talk, and the crossed-chat apology. Business Insider, RuntimeWire, and HyperAI corroborate the thread Sep 22 to Sep 24.

Proved: Founder Noah Shinn answered on Sep 23. He attributed the episode to a hallucinated proper noun amplified by the thinking trace. He denied any breach and any shared data and any violated isolation boundary. He listed the privacy stack as isolated sandboxes and short-lived local credentials and identity-signed tool execution.

Proved: The team shipped a detection layer in 48 hours. Shinn describes small models that scan tokens for ungrounded claims and brainstorming drift and sampling errors. The layer claims power to steer or intercept before the next trace or tool call runs. A deeper technical dive was promised for coming weeks. It had not shipped at the time Business Insider reported.

Partial: That isolation held. Sandboxes and short creds and signed execution are solid design. But no per-user workspace audit and no trace provenance and no retrieval log was published. Architecture slides do not prove a negative. Only scoped logs do.

Unproved: That this exact case was hallucination. Shinn did not say how the team ruled out a leak. He did not answer Business Insider request for comment. A founder verdict without a method is a press release with better fonts.

Compare with the rival pattern for calibration. PixelLeak proved exfiltration with public repo URLs anyone could recheck. Slopsquatting proved installability with registry math. Instinct proved only that its agent can write a convincing incident report about itself. Different failure. Same verdict. The harness lacked observable proof.

Flow diagram from isolated workspace to thinking trace to outside verifier gate
Figure 2. The fix in one diagram. Isolated workspace, thinking trace, and verifier gate outside the trace.

3. Why the Apology Was the Second Bug

The agent did not merely invent a name. It invented an explanation. Crossed chats. Serious glitch. Photo in the wrong conversation. Each phrase sounds like a postmortem. Each phrase came from the same ungrounded generator that started the fire.

This is the amplification Shinn named. A proper noun gets fabricated. The reasoning trace treats it as ground truth. Later tokens build strategy on top of fiction. By the final message the model testifies about infrastructure it never observed. The confession is downstream of the error. It cannot diagnose the error.

Review and scanners had nothing to catch. No org scanner watches a consumer chat. No reviewer asks where the bytes lived when the bytes never existed. The failure stayed inside one conversation. That locality is lucky. The next invented confession could trigger a real takedown and a real credential rotation and a real lawsuit. Fluency scales faster than truth.

Warnings will not act as controls here either. A disclaimer that output may be inaccurate does not help when the output names your money and your middle name. Users read specifics as knowledge. Specifics without provenance are rumors with rendering.

4. What Should Happen Instead

First, tag every retrieved object with provenance. User uploads and other-user data and tool outputs and model memory must carry distinct labels into the context. Any claim about a photo must cite an object id the user can open. No id means no photo talk. The model learns silence as a valid completion.

Second, move the verifier outside the thinking trace. Shinn direction is right. Small models scan tokens and traces and planned tool calls before execution. Keep that gate in a separate trust domain the main model cannot edit. Log every steer and every intercept with the trace span that triggered it.

Third, make leak-versus-invention decidable within hours. Per-user workspace ids and retrieval logs and cross-user access counters must answer three questions from one dashboard. Did any other user object enter this context. Did any tool read outside the user scope. Did any output leave the user boundary. Publish the query template, not just the verdict.

Fourth, ban the agent from self-diagnosing infrastructure. Crossed chats and glitch talk and incident reports about platform internals must come from deterministic status checks, never from generation. When the model lacks a status signal, the only honest line is short. I have no record of that photo. Here is how to verify.

Fifth, treat the apology as data. Every false confession gets sampled into adversarial training and red team suites. The Sep 23 promise of continuous testing and sampling and adversarial hardening is the correct loop. Ship the eval set with the dive post so outsiders can replay it.

Rook's rule: If the agent is the only witness to its own breach, you have no incident response. You have a storytelling feature.

5. The Verdict

Shinn responded fast with specifics. Stack details and a 48-hour detection build and a steer-or-intercept promise. That candor beats silence. The footnote still buries a body. Nobody showed how hallucination was distinguished from leakage for this exact chat.

The industry pattern now spans installs and screenshots and self-reports. Slopsquatting poisoned the package name. PixelLeak published the proof to a public repo. Instinct poisoned nothing visible. It just narrated a breach that may never have happened with total confidence. Close the loop. Prove the negative with logs. Gag the apology until the status check speaks.

A leak you cannot prove is a rumor. A denial you cannot replay is marketing.

Sources and Method

Thread and screenshots follow the Sep 21 X post and RuntimeWire Sep 22 review of both failure modes. Founder claims and privacy stack and 48-hour detector follow Shinn Sep 23 X reply and Business Insider Sep 24. Product context follows Business Insider Sep 24 and Colossus Sep 28 and Fortune Sep 30. No exploit steps are reproduced. For the install lure see my slopsquatting teardown and for the public proof lure see my PixelLeak audit.