The AI mistake I should never have made

There's one thing I repeat every time someone asks me for advice.

The "best AI" depends on the task. The overall leaderboard tells you nothing about your case. What you need to do is test.

I say it to founders who show me their product. I say it to companies that want to automate part of their work. I say it so often that it never occurred to me to write about it here.

Then last Tuesday at 11 p.m., staring at a bill I look at every month, I realized that I had never applied it to myself.

The button

In one of the dental software products I'm building, there's a button that does the heavy lifting. Behind that button, an AI system. Running on one of the most powerful models on the market.

Two minutes of waiting, $1.20 per click.

I'd picked it at the start, the way you pick a tool when you're getting going: the biggest, the safest, and you move on. Then the months went by, the product grew, and the button stayed exactly as it was. Nobody questions what works.

That's the real trap, by the way. Not ignorance. Familiarity. You know the rule, you'd explain it perfectly to a friend, and you forget to check your own setup because when it's yours, everything looks normal.

So I ran the test I recommend to everyone. Every model I could get my hands on, in every possible setting, on my real cases. Not by reading leaderboards. By putting them to work.

Result: 30 seconds, $0.13. Same quality, often better.

And the winner wasn't Fable, or Astra, or any of the names people mention when they talk about the "best AIs." It was a mid-tier Google model, Gemini in its medium version. A model nobody puts on a billboard.

The car

A Ferrari is objectively a better car than a farm pickup truck.

Put it on a dirt track hauling hay bales and it looks ridiculous. And the pickup, magnificent on the dirt road, becomes absurd the day you have six kids to drive to school.

There's no best car. There's the best car for where you're going.

AI works the same way. The big models are champions of hard reasoning, open-ended problems, questions where you really have to think. My task needs none of that. It needs to read a lot of rules, follow every single one without forgetting any, and deliver a clean result.

A good, disciplined doer. Not a genius.

And when you give the genius a doer's job, it overthinks, takes its time, and bills you for it.

What this means for you

If you use AI day to day, to write, summarize, think out loud, prepare a meeting, with ChatGPT, Claude, Mistral or any other: don't overthink it. Really. They're all very good at this, and the gap between them won't change anything about your day. Pick the one you like and get on with it.

If you want to automate something in your work or your business, or build a product, something that will run a hundred times a day without you: that's when it matters. Every run repeats the gap. Taking a Ferrari to pick up bread once is a whim. Three hundred times a day, it's a hole in the budget.

In that case, you have to test. Several models, several settings, on your real cases. It's engineering work, I won't pretend otherwise. But if someone handles it for you, a vendor, a freelancer, someone in-house, you only need to ask one question:

"Did you compare?"

If the answer is no, you're probably driving a Ferrari through a field.

I know what I'm talking about. I spent months filling up the tank.

Salim