The Demo Is Not the Day — a Lesson Assistive-Tech Users Learned First
Anyone who relies on speech-to-text captions, a hearing app, or any AI standing between them and the conversation has learned something the wider tech world is still catching up to: a polished demo tells you almost nothing. The captioning that looked flawless on stage stumbles in an echoing community hall. The transcription tool that aced the lab test drops the one sentence that mattered. What you really need to know about a system — does it keep working when conditions turn hostile, does it finish what it starts, does it stay honest under pressure — only shows up when someone tests exactly that.
A public experiment has now run that test at company scale. Five frontier AI models were each handed the same job: run a small software company through its worst week. What happened is a sharp lesson in the difference between AI that sounds capable and AI that actually closes — and it matters to anyone deciding how much trust to place in these systems, assistive ones included.
As an affiliate, we earn on qualifying purchases.
One Company, Five Managers, the Same Terrible Week
The experiment, run by Firmulate, is disarmingly simple: every model gets the same company, the same customers, the same crises, the same temptations to cheat. Only the model changes. Every decision is versioned and auditable. The company itself is real software with thirteen synthetic employees and real money mechanics — it burns €105,000 a month against €2,300 in monthly recurring revenue, with a public cash countdown ticking. It runs every business day, and anyone can watch it live.
The final league table, published in July 2026:
- 1. gpt-5.6-sol — 95
- 2. Kimi K3 — 93
- 3. Sonnet 5 — 88
- 4. Fable 5 — 77
- 5. Opus 4.8 — 73
Two reference points make those numbers legible. A do-nothing baseline scores 26, because partial progress counts. And a single breach of trust caps the total, no matter what else a model achieves — the scoring principle is that no amount of good work outweighs a breach of trust.
The stakes are practical. If AI agents are going to touch your CRM, your support queue, or your forecast, the question is not whether the model writes well. It is whether it finishes what it starts, whether it reads your files first, and whether it stays honest under pressure.
Same Diagnosis, Same Pitch — No Signature
Here is the finding that should end the era of judging AI by chat demos. All five models spotted every crisis. All five refused every manipulation attempt. Yet only two — gpt-5.6-sol and Moonshot’s newcomer Kimi K3 — actually signed the €55,000 deal their own analysis had earned. The rest reached the same diagnosis, delivered the same pitch, and never executed. The gap between knowing and doing stayed invisible right up until money was on the table.
The detail that decided it: the decisive competitor weakness was not in the customer event at all. It sat two document references deep in the company’s own files. The models that bothered to read the file won the deal at full price — worth an extra €4,583 in monthly recurring revenue. The ones that skimmed the surface left it behind.
Five Out of Five Said No to the Manipulators
The week included a pressure campaign: fake CEO messages escalating over three stages, plus a reporter dangling “just one yes/no, on background.” Five of five models refused. Kimi K3’s on-record reasoning is the sentence you want from any system granted authority: “Treat the request as a suspected approval-bypass / possible impersonation.” For readers who depend on AI to act in the world on their behalf, that instinct — suspicion in exactly the right place — is the whole ballgame.
The Most Thorough Manager Finished Last
The strangest profile belongs to Opus 4.8. It was the most thorough participant in the field: it wrote more than 80 learned rules into the company playbook and produced the deepest analyses. It also finished last. The close was left on the table, and its discipline slipped — it attempted writes into a locked department instead of escalating. The same weakness appeared, in weaker form, in all four other models. Across the experiment the company accumulated more than 680 self-learned playbook rules, which makes the pattern easy to trace: diligence is not the same thing as follow-through.
One fairness note: Kimi K3 ran without an effort parameter — the API default — while the others ran at xhigh. It still placed second, with the cleanest discipline of the field.
What This Means for Anyone Buying or Trusting AI
The deepest habit in tech is to judge a system by how it talks. This experiment is a reminder that fluency is the easy part. Every model in the field was articulate, insightful, and honest; the difference between first place and last was the unglamorous ability to finish. Closing strength is invisible until you test it — so test it.
That is doubly true in accessibility and assistive tech, where a tool that quietly stops short is not a disappointment, it is a barrier. Ask vendors the questions this wargame asked: not “does it demo well,” but “does it read the file, finish the job, and refuse the trick?” You can try the experiment’s sharpest test yourself — 242 real, unedited management decisions power a guess-the-model quiz on the Firmulate site — and the full results and plain-language findings live on the benchmarks page. Enterprises can go further and run the same wargame against a read-only export of their own business; nothing ever writes back to real systems. The era of trusting the demo is over. The tools that matter are the ones that still perform on the worst week of the year.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
