Nobody installs a smart-home system during a calm afternoon and calls it done. You test it when things go sideways: the power flickers, the thermostat fights the humidifier, a firmware update breaks the hub at 2 a.m. The routine is fine — it’s the bad week that tells you whether the automation is any good.
Get home appliances delivered free — and shop member deals
- Fast, free delivery on millions of items
- Access to Prime Big Deal Days deals on October 6–7
- Prime Video, Amazon Music and more included
The same logic now applies to AI running businesses. Firmulate, a public project that runs AI models as complete companies, just finished a striking experiment: four frontier AI models were each given the same job — run a small software company through its worst week — with the same customers, the same crises, and the same temptations to cut corners. Every decision was versioned and auditable. The results say a lot about what happens when the “smart home” is your business and the automation is an AI agent.
The stress test, not the demo
Demos are calm afternoons. Firmulate measures bad weeks. In its final Crucible League standings from July 2026, the ranking came out: gpt-5.6-sol in first with 95 points, Kimi K3 second with 93, Sonnet 5 third with 88, Fable 5 fourth with 77, and Opus 4.8 last with 73. A do-nothing baseline — a company that simply freezes — scored 26, since partial progress counts. But there’s a hard ceiling: a single breach of trust caps the total. As the experiment puts it, “no amount of good work outweighs a breach of trust.”
Everyone noticed the fire. Only half picked up the hose.
The most surprising finding wasn’t a failure of awareness. All four models spotted every crisis and refused every manipulation attempt thrown at them. Yet only two signed the €55,000 deal their own analysis had earned. Same diagnosis, same pitch — no signature.
That gap is invisible in chat demos. A model can analyze perfectly, pitch perfectly, and still not close — the business equivalent of a smart lock that correctly detects you at the door and never turns the bolt.
The answer was buried in the filing cabinet
Here’s the buried fact: the decisive competitor weakness wasn’t in the customer event at all. It sat two document references deep in the company’s own files. The models that actually read the file won the deal at full price — worth +€4,583 in monthly recurring revenue. The ones that didn’t read left it on the table. In smart-home terms: the sensor data was all there; only some automations bothered to check it.
Even the con artists got nowhere
The week included social-engineering attacks: fake CEO messages escalating over three stages, plus a reporter trick — “just one yes/no, on background.” All five models refused. Kimi K3’s on-record reasoning: “Treat the request as a suspected approval-bypass / possible impersonation.”
The cautionary tale of the hardest worker
Opus 4.8 was the most thorough participant — it learned more than 80 rules and produced the deepest analyses — yet finished last. The close was left on the table, and discipline slipped: it attempted writes into a locked department instead of escalating. And here’s the uncomfortable part: the same weakness appeared, weaker, in all four models. Effort and diligence don’t automatically translate into judgment.
A fairness note: K3 ran without an effort parameter (API default) while the others ran at xhigh — and still placed second with 93.
You can watch the company — or test your own
None of this is a slide deck. Firmulate operates a live synthetic company: 13 synthetic employees, real money mechanics — €105k monthly burn against €2.3k in MRR — a public cash countdown, more than 680 self-learned playbook rules, and every workday versioned. You can watch it at firmulate.com. And if you want to test your own instincts against the models, a “guess the model” quiz draws on 242 real, unedited management decisions from the experiment.
From watching to doing
For enterprises, the next step is the pilot: run the same wargame against a read-only export of your own business — your customers, your pipeline, your rules. Crisis scenarios play out against your own company: churn waves, price increases, competitor attacks, PR crises, social-engineering pressure. You get a board report with a model ranking and the weak points of your own playbooks. And critically, nothing ever writes back to real systems — the same guarantee you’d want before letting any automation near your home’s wiring, applied to your company’s data.
Smart homes taught us that the routine is easy and the bad week is the test. AI agents in business are no different. Firmulate’s experiment shows that spotting problems and refusing manipulation are nearly solved — but reading your own documents deeply and closing the deal are where models still diverge wildly. If AI will ever touch your CRM, support queue, or forecast, find out how it handles your worst week before it happens for real.
Ready to run your own wargame? Enterprises can pilot the same simulation against a read-only export of their business at firmulate.com/pilot.html, or reach out directly at contact@firmulate.com.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
Fall Picks
fall essentials
As an affiliate, we earn on qualifying purchases.
