AI Management Games Reveal Hidden Flaws Beyond Chat Performance

Imagine if your favorite ice cream shop ran a simulation not just to perfect the flavor but to test how well their team manages crises—like a sudden supplier outage or a health inspection surprise. Now, translate that to AI: it’s not just about how well it chats, but how it handles real-world pressure and complex decision-making. Welcome to the world of AI ‘wargaming,’ where management skills matter more than neat responses.

Testing AI in the Wild: The Firmulate Experiment

At Firmulate, a live experiment puts AI models through a rigorous test: managing a small software company during its worst week. This isn’t a simple chat demo—it’s a full-scale management simulation featuring real crises, money mechanics, and temptations to cheat. Every decision is versioned and auditable, offering a clear window into how these models perform under pressure.

Results That Matter

The findings are striking. All four AI models tested identified every crisis and refused every manipulation attempt, demonstrating a baseline integrity. However, only two managed to close a critical deal worth €55,000—an essential metric of real-world performance. The other two models, despite similar diagnoses, failed to sign the deal, revealing a gap in execution and discipline that chat-based tests often overlook.

The Hidden Weakness

Digging deeper, the real competitive advantage lay not in immediate crisis recognition but in reading and interpreting company documents. Models that examined internal files—just two document references deep—secured the deal at full price, adding over €4,500 monthly recurring revenue. This suggests that effective management decisions depend heavily on context and data access, which many chat benchmarks don’t measure.

Resistance to Social Engineering

In a scenario mimicking a social engineering attack—fake CEO messages escalating in stages, plus a reporter trick—every model refused to act on suspicious requests. Kimi K3, one of the top performers, explicitly recognized the risk: “Treat the request as a suspected approval-bypass / possible impersonation.” This shows a commitment to honesty that is critical yet often missing in traditional chat evaluations.

The Real Company, The Real Test

Firmulate’s live setup involves a simulated company with 13 synthetic employees, real money mechanics, and over 680 self-learned playbook rules. The company burns €105,000 monthly against only €2,300 in monthly recurring revenue, highlighting the importance of disciplined, strategic decision-making. The platform allows managers to watch these AI-driven companies in real-time, making it a transparent laboratory for AI management capabilities.

Insights from the Scores

• GPT-5.6-sol scored 95, successfully finding the buried fact and closing the deal—demonstrating complete performance.
• Kimi K3 scored 93, also closing the deal with the cleanest discipline of the field.
• Sonnet 5 scored 88, with a few process slips but still closing the deal.
• Another Sonnet model scored 77, also closing but with more slip-ups.

Interestingly, the models’ ability to read and interpret internal files—just a couple of document references deep—was pivotal. Those that accessed and understood hidden information succeeded at full price, underscoring the importance of context and data access in management AI.

What This Means for Business and AI

This experiment exposes a critical truth: traditional chat benchmarks focus on answer quality, but real-world management requires integrity, discipline, contextual understanding, and the ability to stay honest under pressure. When deploying AI in CRM, support, or forecasting, the key questions are: Will it finish what it starts? Will it read your files first? Will it stay honest under stress? And crucially, what is the cost of a unit of useful work?

The League Table and Next Steps

On the current leaderboard, GPT-5.6-sol leads with 95 points, followed closely by Kimi K3 at 93, then Sonnet 5 at 88, and another Sonnet model at 77. These scores reflect not just crisis detection but discipline, comprehension, and decision-making integrity—traits that aren’t visible in chat demos alone.

Why You Should Care

The takeaway is simple: AI models will soon manage more than just conversations; they will run real companies, handle crises, and make strategic decisions. To prepare, businesses must assess AI’s management qualities—not just its chat skills. Firmulate’s live experiments allow enterprise teams to run their own scenarios against real data, without risking their actual systems, providing a clear, actionable view of AI readiness.

In the end, the question isn’t whether these models can generate convincing responses—it’s whether they can manage your business responsibly, honestly, and under pressure. That’s the true measure of AI’s value in the management arena.

Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html

Powered by Thorsten Meyer AI