I wrote the tie-break before the bake-off

CyclingHero Admin has an import that reads documents. A tour arrives as a PDF brochure, an email, a table, or a page of prose, and the admin turns it into a tour proposal: days, towns, hotels, routes, climbs, airports. Version one was one big Claude call that read the whole document and wrote the whole itinerary, and it worked well enough to prove the idea.

Version two was supposed to make it cheap, accurate, and EU-only where that matters. My hypothesis had a nice ring to it: an EU-hosted open model gets 90% of Claude’s quality for 10% of the cost. That turned out to be true. Then the eval told me not to ship it as the default anyway.

Write the rule first

Before the first run, I wrote the picking rule into the plan:

Highest mean score wins. Anything within two points of it counts as tied. Ties go to EU-only, then to the cheapest, then to the fastest.

That’s the whole post, really. The rest is what happened when the numbers came in and I had to obey it.

I wrote the rule first because I knew which answer I wanted. I’d just moved the sumo newspaper desk onto gpt-oss-120b on Scaleway, and it was better and cheaper there. I wanted the same story twice. A bake-off where you pick the rule after seeing the table is just a way to launder a preference.

Round one: everyone passes

The first eval was five real documents with known answers, scored check by check. Ten models took a turn at the structuring step, the one that reads the document and writes the itinerary:

• gpt-oss-120b (low reasoning): 94.8%, at €0.003 of model cost per import
• Sonnet, the reference: 95.0%, at €0.037. Twelve times the price for 0.2 points.
• qwen3.5-397b topped the table at 96.8%, but it always thinks: four times the output tokens and 70 seconds an import.
• deepseek-v4-flash scored better with thinking switched off (94.6 vs 92.6) and ran in 8 seconds.
• gemma-4-26b reasoned until it hit the 32k output budget on every document. It ignored the effort setting, took four minutes, and scored 0%.

Six models sat within two points of each other. Under the rule they were all tied, so the cheapest would win, and gpt-oss looked like it had it.

But I didn’t trust the table, because the misses were the same misses on every model, Sonnet included. A hotel it couldn’t place. “Palma de Mallorca” where the answer key said “palma”. The differences between the top six were one or two checks out of about a hundred. The eval had saturated. It could no longer tell good from slightly better, and a saturated eval always says “the cheap one is fine.”

Round two: harder documents

So I made harder ones: eight documents generated from real published tours, written the way operators actually send them. A brochure table, a lowercase email, a page of prose, three of them in miles, one with no year, one with “or 42 km via the coast” on a day. The old five still scored 97–99%. The new eight scored 85–90%, which meant the eval had something to say again.

Four profiles, thirteen documents, two runs each:

profile	score	model € / import
Claude	93.3%	€0.050
Claude + Scaleway for extraction	92.9%	€0.043
Scaleway only	90.3%	€0.003
Scaleway + a dedicated decider	90.4%	€0.003

The hypothesis held. The EU profile reached 97% of Claude’s score at 6% of its model cost. It also lost by three points, and three is not two. On the generated documents the gap was 86 against 90, and it came from somewhere specific: gpt-oss sometimes dropped the tour’s region (which switches off the whole route library), left airports empty, and turned a riding day into a rest day.

The rule said Claude + Scaleway stays the default. I’d have liked a different answer. I didn’t get to pick one, because I’d already picked how to pick.

I gave it one more chance. I wrote code that infers the region from the towns, which is the exact failure the EU model kept hitting. That helped, and it helped Claude just as much, because Sonnet had been reading a Liguria start as “out of region” all along. The gap stayed at 3.3 points. Then I re-ran the hardest documents with gpt-oss on medium reasoning (92.8%) and mistral-medium (96.5%, close to Sonnet’s 98.3%), but mistral-medium costs about what Sonnet does. So it’s only worth it if EU-only matters more than money for a given client, and that makes it a switch they can turn on, not a default.

The model wasn’t the bill

Here’s the number that changed how I think about the cost: about 90% of the cost per import isn’t the model at all. It’s Google Places, looking up hotels and towns. Swapping Sonnet for gpt-oss saves about four and a half cents. A better library lookup that skips one Places call saves more than that.

I’d been optimising the line item that was fun to optimise.

The model wasn’t the misses either

Most of the remaining errors were identical across all four profiles. The route library holds near-copies of the same ride under different codes. Some hotels have no coordinates, so nothing can place them. Two documents scored 78% and 87% for every model, every run. That’s the resolver, not the model.

That became the first principle in the plan: accuracy is a data problem first. Fix the data and the deterministic lookups before you pay for a smarter model. You’ll pay for the smarter model on every import forever. You pay for the data fix once.

A model is only as smart as what it can reach

This is the part I’d tell anyone who’s about to buy “AI” for their business: the smartest model available, with nothing to look things up in, is a confident guesser. Ask it which hotel the tour stays at on night three and it will give you a hotel, a nice-sounding one, and maybe one that doesn’t exist. That’s where slop comes from. The model isn’t stupid; it just had nothing to check against, so it filled the gap with something plausible.

So the import doesn’t ask the model to know anything about the world. It splits the work between code and model by what each is good at:

• Code reads the boring parts. It strips the terms and conditions, normalises the text, and splits the document into days.
• A small model reads one day at a time and writes down what it says: towns, kilometres, the hotel’s name as written. Short input, a strict schema, and code that checks the numbers are plausible.
• Code finds the candidates. Towns go through our own library, then a gazetteer, then Google. Hotels have to be near that night’s town. Routes are ranked by how close their stated distance is. Every answer is a real record with a real code.
• A model picks from that list, with “none of these” always on it, and says how sure it is. It can choose. It can’t invent.
• Low confidence becomes a card for a person, with the top three candidates as one-click choices. Every correction is remembered, so the next import of the same thing is just a lookup.

The same rule holds where the model talks to a person. In the conversation pane it gets tools like search_routes(near, km) and search_hotels(near, radius), backed by the same deterministic queries. The only write tool it has, propose_change, applies nothing. It draws a card, and a person clicks it. Google results get the same suspicion: if the top hit’s name doesn’t match the hotel in the document, nothing is created, and the import says so: couldn’t find hotel X, Google’s best match Y doesn’t look like it.

That’s how a model costing a third of a cent per import scores around 90%, and Sonnet around 94%, on documents written by people who never imagined a machine would read them. It isn’t zero errors. Some wrong picks still come back sure of themselves, and the eval counts every one. But they’re a different kind of error: a wrong choice from a list of real things, not a hotel nobody ever built. A reviewer can catch a wrong choice in a second. An invented hotel ends up in a customer’s itinerary.

The model brings judgement. The tools bring the facts. If you only pay for one of them, pay for the tools.

Confidence is a vibe

The import asks a model to choose between candidates: which of these five routes is the day’s main ride? I wanted a confidence score with each pick, so that a sure pick applies silently and an unsure one becomes a card for a human to check.

The general-purpose models’ confidence clustered between 0.8 and 0.95 whatever the question. Haiku and mistral-small put wrong picks at 0.9–1.0 and right picks at 0.5. A threshold on that number flags nothing useful, so for those models I don’t use one: they pick and never ask.

The dedicated decision model (Jev, through Vercel’s AI Gateway) was the only one whose number meant something. Every pick it made below 0.48 was wrong. A threshold there flags six of its ten wrong picks with zero false alarms. It changed seven picks in the eval, and all seven were right.

It isn’t the default either. It retains data on the plan we’re on, so it’s opt-in and never chosen automatically, whatever the score. That rule was also written down before it came up.

What it cost to find out

All three rounds came to under €10: about €1.80 for the first, €3.20 for the second, €2.60 for the region pass. That’s less than one engineering hour, to settle a question that would otherwise have been settled by whoever argued loudest, which in this case was me.

The rule

Decide how you’ll decide before the numbers arrive. Write the tie-break into the plan: the margin that counts as a tie and the order of what breaks it. Then make the eval hard enough to disagree with you. If every candidate passes, the eval isn’t telling you the candidates are good. It’s telling you the eval is easy.

In the sumo post the cheap EU model won, because the job was rephrasing a paragraph that was already correct. Here the same model lost by three points, because the job is reading a messy document and getting the facts out of it. Same model, different verdict. The workload decides, and the rule is how you let it.