The conversation about AI models has shifted. A year ago, the question was "which model is the best?" Today, with price wars between OpenAI and Anthropic, open models competing with closed ones, and cheaper options appearing every month, the real question is different: "which model works for my specific use case, at the lowest possible cost?"
That question sounds simple, but most companies answer it wrong. They pick a model because they read a generic benchmark, because "everyone uses that one," or because the provider offered a discount. Then they find that the model works well on general tasks but fails on theirs, or that they're paying for capabilities they don't need.
Why public benchmarks aren't enough
Standard benchmarks (MMLU, HumanEval, GSM8K) measure general capabilities: reasoning, code, math. They're useful for comparing models against each other, but they don't tell you if a model will work for classifying support tickets in your industry, generating product descriptions in your brand voice, or extracting information from your contracts.
Your use case has its own rules. A model might be excellent at general English but mediocre at Spanish technical jargon. It might answer simple questions well but fail when it needs to follow complex instructions. It might be fast and cheap, but sacrifice accuracy on tasks where a mistake is expensive.
The only way to know is to test with your own data and real tasks.
How to set up a practical evaluation
You don't need a research team or weeks of work. You can set up a basic evaluation in a day or two if your use cases are clear.
Step 1: Define 3-5 specific tasks. Not "generate text" or "answer questions." Specific tasks: "classify a support ticket into one of these 5 categories," "extract the customer name and order number from this email," "generate an 80-word product description in this tone." The more specific, the more useful the evaluation.
Step 2: Prepare a test set. Between 20 and 50 examples per task. You don't need thousands. What matters is that they cover the variety of cases you'll encounter in production: the easy ones, the hard ones, the ambiguous ones. If you can, include some cases where you already know the correct answer, so you can measure accuracy.
Step 3: Test 2-3 models. Pick a mix: one of the leaders (GPT-4, Claude), a cheaper one (GPT-4o-mini, Claude Haiku), and if it makes sense for your case, an open model (Llama, Mistral). Run your examples with each model and save the responses.
Step 4: Measure what matters. For tasks with a correct answer (classification, extraction), calculate accuracy. For open-ended tasks (text generation), review a sample and evaluate quality, relevance, and instruction adherence. Also measure latency and cost per call. Don't choose on accuracy alone: a model that's 5% less accurate but 10x cheaper might be the right choice if the error isn't critical.
Step 5: Validate with real users. Before deploying, show the responses to the people who will use or supervise them. Their feedback will tell you things metrics don't capture: whether the tone is right, whether the format is useful, whether there are subtle errors you missed.
What to do with the results
The evaluation will give you a clear picture of which model works best for each task. Sometimes you'll find that an expensive model is necessary for critical tasks, but a cheap one handles 80% of the volume. Other times, an open model outperforms closed ones in your specific domain.
What matters is that the decision is based on data from your business, not hype or what competitors are doing. And when models change (and they will), you can re-evaluate without starting from scratch: your test set and methodology are still there.
At Luxion, we help companies run this kind of evaluation connected to their real use cases. We start by identifying the tasks with the best effort-to-result ratio, set up the evaluation with your data, and show you the results before committing to anything. If you want to apply this to your business, we can help you land it in a small, measurable prototype.