When you think of AI, you might imagine chatbots that sound convincing or witty. But in the world of business, the real test of an AI’s worth isn’t just its ability to chat — it’s whether it can actually finish the job and deliver tangible results, even under pressure.
The Testing Ground: An AI-Run Company Facing Its Worst Week
Imagine a small software firm under siege — with demanding customers, tight deadlines, and the temptation to cut corners. This is the scenario where four advanced AI models were put to the test, each running the same company through its most challenging week. The goal? See if these models could identify crises, resist manipulation, and importantly, close deals to keep the business afloat.
As an affiliate, we earn on qualifying purchases.
The Benchmarks: Who Was Watching?
The experiment was rigorous. Each AI model scored in a league table based on a comprehensive benchmark, with scores ranging from 73 to 95. The top performer, gpt-5.6-sol, scored a 95. It not only detected every crisis but also refused every manipulation attempt, including sophisticated social engineering scenarios involving fake CEO messages and reporter tricks. Meanwhile, other models also identified issues and resisted manipulations, but only two managed to close the deal worth €55,000 — the one that earned its own analysis.
What Separated the Winners from the Others?
All models demonstrated solid crisis detection and resistance to deception — a kind of surface competence. But closing a deal required going deeper, into the company’s own files. The decisive edge came from reading and understanding internal reference documents buried two layers deep in the company’s files. Models that successfully accessed and analyzed these documents won the business at full price, adding over €4,583 in monthly recurring revenue (MRR).
The Invisible Weaknesses
Interestingly, the models’ failures to close the deal weren’t due to a lack of understanding of crises or manipulation. For example, one model, Opus 4.8, displayed meticulous analysis but faltered during the final step — leaving the deal unexecuted because discipline slipped and the closing attempt was written into a locked department instead of escalated. This highlights a crucial insight: the ability to recognize problems is not the same as the ability to finish the job.
Why Chat Demos Aren’t Enough
Many companies evaluate AI capabilities based on chat demos, where models perform well in simulated conversations. But this experiment reveals the flaw: chat proficiency doesn’t necessarily translate into execution strength. The models could identify crises, refuse manipulations, and even analyze internal documents — yet only some could follow through and close deals. The ability to stay honest, disciplined, and complete tasks under pressure is invisible in a simple chat or demo.
The Lessons for Business Leaders
What does this mean for organizations considering AI solutions? The key takeaway is that measuring an AI’s ability to generate convincing conversations isn’t enough. Instead, focus on its capacity to finish what it starts, access critical information, and resist shortcuts, especially when real money and reputation are on the line. A unit of useful work — closing deals, making decisions, executing plans — is the true measure of an AI’s business value.
The Experiment Continues
Firmulate’s live experiment, available at firmulate.com, allows enterprises to run the same wargame against their own business scenarios. It’s a safe, read-only environment that simulates real crises, decision pressures, and manipulations. The goal? Help organizations identify whether their AI workforce can actually deliver results, not just chit-chat.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
