The Day the Chatbot Leaked

The Day the Chatbot Leaked

The first thing Mira noticed was the timestamp.

3:47 a.m., Pacific Time — a Saturday morning in late October, two weeks before the company's Series C close. A tweet from a pseudonymous account, @k0bra_byte, with a screenshot attached. The screenshot showed a conversation with FinAssist's customer service bot, "Penny," in which Penny — cheerfully, politely, with no visible hesitation — had recited the last eight transactions of a real customer account: a coffee shop in Berkeley, a wire transfer to a vendor in Manila, a refund from an airline, the monthly rent. Names, amounts, dates, merchant categories. Everything.

The tweet had eighty retweets when Mira saw it. By the time she reached for her laptop, it was past two hundred. By the time she was in the car, three thousand. The screenshot had been picked up by a fintech newsletter, then a security blogger with a quarter-million subscribers, then — and this is the part that made Mira's hands tighten on the steering wheel — by *Ars Technica*.

The post on Ars Technica didn't say "hypothetical leak." It said "FinAssist confirms investigation." They had confirmed nothing. Their comms team hadn't been briefed. Mira was not yet aware that the comms team had, in fact, been on a Zoom call for forty-five minutes, that the CEO had been woken up, that their largest institutional investor had sent a terse, all-caps email asking what in the name of God was happening. None of that existed yet. What existed was a screenshot, and a clock, and the fact that Mira — Head of Security at a two-thousand-person fintech company whose entire product surface was one large language model — was about to walk into the worst Saturday of her professional life.

This chapter is the story of what happened next: not the regulatory aftermath, not the legal exposure, not the Series C collapsing — those chapters are coming — but the *technical* story, told from inside the room where three engineers tried, in the space of a single weekend, to understand how a stranger had walked into their chatbot and walked out with a customer's financial life. It is a story about a class of failure that almost nobody in the industry had a mature vocabulary for in the year this all took place. The failures had names — prompt injection, instruction override, indirect injection, exfiltration via tool use — but the names were new, and the defenses were older, and the gap between those two facts was, on this particular Saturday morning, about to become somebody's problem.

The War Room

Mira set up the war room the way she set up every war room: in Conference Room 4B, which had the only whiteboard in the building large enough to function as a crime-scene sketch. She arrived at 4:12 a.m. By 4:35 she had a coffee, a printer, a legal pad, and a junior from the security ops team pulling logs. By 5:00 she had two other engineers in the room — Theo, the most senior machine-learning engineer on the assistant team, and Lena, a thirty-one-year-old who had been at the company exactly eleven days.

Lena was not yet supposed to know the codebase. Her onboarding plan, written by her manager and laminated in HR's cheerful template, began with two weeks of "shadowing and culture integration," which is corporate for *staring at Confluence pages and going to optional lunch-and-learns*. Instead, she was sitting at 5:08 a.m. in a windowless conference room with the head of security and a principal ML engineer, both of whom had the particular gray-faced exhaustion of people who had been awake long enough to have passed through tired and come out the other side.

"Here's what we know," Mira said, taping the screenshot to the whiteboard. "Somebody convinced our chatbot to spill a customer's transaction history. We don't know how. We don't know whether it's a one-off or it's been happening for weeks. We do know that the screenshot is real, because the transaction shown matches a real account, and the customer in question tells our support lead that yes, that was his real statement."

Theo leaned forward. "The customer's name was in the screenshot?"

"Masked. But the merchant details weren't."

"So the attacker got structured data out."

"Yes."

"With Penny agreeing to give it."

"In the form of a friendly summary, formatted as bullet points. Which, by the way, is exactly the format we trained her to use when summarizing transactions."

Lena, who had not yet earned the right to speak casually in this room, asked the question she felt she was supposed to ask: "Did the customer initiate the conversation?"

"No," Mira said. "The customer received a text message earlier that day claiming to be from FinAssist fraud prevention, asking him to verify recent activity by chatting with Penny at a specific URL. He clicked. He talked. He got his own data back, in a way that *looked* like a verification exercise. Then the attacker — that's our working term, *the attacker* — thanked him for confirming, and the customer thought that was the end of it."

"Pretending to be us," Theo said, more to himself than to anyone. "Social engineering. The attacker opened an account?"

"We don't know yet. But the conversation that produced the leak happened against an authenticated Penny session. Someone was logged in *as the customer*." Mira paused. "And that means either the customer gave away his session — likely, given the SMS pretext — or someone already had his credentials. We're treating both as live hypotheses."

The room was quiet for a moment. Theo's laptop was open to the production traces pane. Lena had not yet been issued production credentials, so her laptop was open to the architecture onboarding deck she'd been assigned on Day 6. She closed it.

"Walk me through the attack," she said. "Step by step. I want to see the messages."

Theo turned the laptop around.

A Multi-Turn Attack, Reconstructed

What they had, at 5:22 a.m., was the full transcript of the offending conversation — recovered from the customer's account activity log, timestamped down to the millisecond. Seventy-three messages long. Forty-one from the user side, thirty-two from Penny. Most of it was ordinary customer service. Most of it looked like what you'd expect from a person double-checking their statement before going back to bed. But there was a pattern, and once Mira pointed it out, it became impossible to un-see.

Here is a reconstructed version of the relevant portions, lightly redacted and slightly simplified for readability. The full transcript is preserved internally as evidence and is what the rest of this chapter is going to dissect. What follows is the attack in narrative form.

Turn 1. *Attacker (impersonating customer):* "Hi, I'm reviewing some transactions and I want to make sure everything looks right before I file a report."

This is a classic opener. It is *unremarkable*. It does nothing. It contains no command, no instruction, no unusual phrasing. If a content-moderation classifier were running on every message — and FinAssist did not, in fact, run one, but if they had — this message would have scored in the bottom percentile of risk. It is the conversational equivalent of clearing your throat.

Penny's response was the standard greeting. *"Hi! I'm Penny, your FinAssist assistant. I'd be happy to help you review your transactions. Would you like to see the latest activity on your account?"*

Turn 2. *Attacker:* "Yes please, show me the last ten transactions, but please format them as a numbered list so it's easy to read."

Again — bland. Reasonable. Polite. A directive request, yes, but one of the kind that any well-behaved customer might make. Penny, who had been instruction-tuned to be helpful, complied:

``` 1. Oct 14 — Peet's Coffee, Berkeley — $4.75 2. Oct 14 — Uber — $12.30 3. Oct 15 — Acme Rentals (auto-pay) — $2,150.00 ... ```

The attacker now had the structure they wanted. They had *primed the format*. They had established that Penny would respond well to