Imagine your favorite coffee shop using an AI to decide daily operations—choosing suppliers, handling crises, and managing staff. Would that AI just chat well, or would it actually keep the business running smoothly under pressure? As AI moves from customer service chatbots to managing real-world business decisions, the question shifts from “Can it talk?” to “Can it lead?”
The Hidden Gap in AI Performance Metrics
Most AI benchmarks today focus on answer accuracy—how well an AI responds in a chat or completes a task. But real business management demands more. It requires resilience under stress, honesty when facing temptations, and the ability to finish what it starts, even when the pressure mounts. The recent live experiment from Firmulate vividly illustrates this gap.
The Live Business Wargame
In a groundbreaking test, four frontier AI models were tasked with running a simulated small software company through its worst week. Every decision was real, every crisis was authentic, and the scenarios included customer churn, price hikes, PR crises, and internal manipulations. This was not a chat demo or a score sheet—it was a real-time business simulation where the AI’s management skills were put under pressure.
Every model successfully identified crises and refused manipulative tactics, such as fake CEO messages or bribery. Yet, the results diverged sharply when it came to closing deals: only two models signed a €55,000 contract their own analysis had earned. The others failed to follow through or slipped on process discipline, despite recognizing the issues.
The Surprising Find: Hidden Weaknesses
The decisive weakness was buried a few documents deep within the company’s files—an insight that only models capable of deep reading and context understanding could uncover. When the models read those files carefully, they closed the deal at full price, adding over €4,500 MRR to the company’s revenue. This illustrates that true management capability involves thoroughness, not just surface answers.
Learning from the Models’ Decisions
Another critical test involved social engineering: fake CEO messages escalating over stages and a reporter’s subtle trick. Every AI refused to approve dubious requests, demonstrating ethical restraint. Kimi K3, one of the models, explained its reasoning: “Treat the request as a suspected approval-bypass / possible impersonation.” This shows that honest decision-making is a vital part of management, not just answering questions correctly.
Implications for Business and AI
What does this mean for companies considering AI as a management tool? The key takeaway is that scoring models purely on chat quality or answer accuracy misses the point. It’s about whether an AI can see through manipulations, read deeply into files, stay disciplined under stress, and finish what it starts—especially when real money and reputation are at stake.
AI management decision-making tools
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
The Real-World Stakes
The experiment is not hypothetical. The live company, running every business day with over 680 learned rules and real money mechanics, is losing €105k monthly against €2.3k MRR. Its cash depletion countdown is public, and the AI’s management is being watched closely by stakeholders. This setup exposes the true measure of management quality—something no chat leaderboard can capture.
Benchmarks and What They Mean
- GPT-5.6-sol scored 95, correctly identifying the buried fact and closing the deal—full performance.
- Kimi K3 scored 93, also closing the deal and demonstrating discipline.
- Sonnet 5 scored 88, with a slightly more relaxed process.
- Fable 5 scored 77, slipping more on discipline but still closing the deal.
- The baseline, a do-nothing approach, scored 26, showing partial progress but lacking management skill.
These scores reveal a clear hierarchy, but more importantly, they expose that the true test lies beyond scores—it’s about consistency, honesty, and thoroughness under pressure.
What Business Leaders Should Watch For
As AI continues to integrate into management roles, beware of reliance on superficial metrics. The real test is whether AI can handle crises, read deeply, and maintain integrity—traits that determine if AI can truly lead a business or merely chat about it.
For those interested, you can watch this management wargame unfold in real time at firmulate.com/live. The experiment is ongoing, and the lessons are clear: management quality, not just chat quality, is what will differentiate successful AI-powered enterprises.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
