Mark Zuckerberg said this week in an internal meeting that AI agents haven't progressed as much as Meta expected. The phrase might sound like a corporate excuse, but it points to a real problem: many agent projects work well in presentations and poorly in production.
The gap usually isn't in the model. It's in how you evaluate the project.
What to measure (and what not to)
Most teams evaluate their agent with the wrong metrics. They ask "does it work?" when they should ask "does it work for which case, how often, and at what cost?"
Autonomous resolution rate. Out of every 100 interactions, how many does the agent resolve without human intervention? A support agent that resolves 30% of cases is only generating extra work for your team if the other 70% requires complete manual review.
Average resolution time. If the agent takes 45 seconds to respond but your team takes 10 minutes to correct the response, the net savings are negative. Measure the total case time, not just the agent time.
Cost per interaction. Models charge per token. An agent that uses 2,000 tokens per response and processes 500 queries per day costs more than a part-time junior employee. Do the math before scaling.
Escalation rate. How often does the agent hand off to a human? If 60% of cases end in escalation, the agent isn't automating. It's filtering. And filtering has a cost.
Signs your project isn't going well
There are early indicators that an agent won't work as expected. These aren't technical failures. They're signals that the use case doesn't fit.
Your team trusts the agent less than on day one. If after three weeks your employees prefer doing the work manually, the agent isn't delivering value. Internal adoption is the most honest metric.
Exceptions are more frequent than standard cases. If your agent works well for domestic orders but not international ones, and 40% of your orders are international, you have a design problem, not an implementation problem.
Costs rise every month without resolution improving. If the agent processes more queries but resolves the same percentage, you're scaling inefficiency.
Nobody knows who owns the agent. If the product team says it's IT's responsibility, and IT says it's product's responsibility, the agent has no owner. And without an owner, it doesn't improve.
When to pivot or stop
An agent that doesn't meet metrics after 60 days won't improve on its own. You have three options.
Reduce scope. If the agent tries to cover five use cases and only works well for two, cut it down. An agent that does two things well is worth more than one that does five poorly.
Change the model or prompt. Sometimes the problem is that the model lacks the capabilities for your case. A larger model, a better-structured prompt, or a more complete RAG system can change the outcome.
Stop the project. If after adjusting scope and model the metrics don't improve, the use case isn't suitable for an agent. That's not a failure. It's information. Redirect the budget to a case with better fit.
At Luxion, we evaluate AI agent projects with real metrics from week one. We show you a prototype connected to your data, measure results for 30 days, and decide together whether to scale, adjust, or stop. No long-term commitments or promises of total automation.