# Vision model bench - results of 2026-09-03

Nine plates x 3 passes per model, English, the production request body from
`vision.ts`, scored end to end (model -> `usdaSearch` -> `aiEstimateMacros`).
Run after the matcher fixes of the same day, so these are the numbers the
pipeline at the time produced. Fixture portions were visually estimated, so
these scores measure agreement with estimated references. Small differences do
not establish a reliable accuracy ranking. See method-notes.txt for the limits
and the plan to source published photos with documented reference data.

Gates: p50 over 5 s or p95 over 15 s is TOO SLOW and ranks below every
usable model. Cost is per scan in USD.

| # | Model | Recall | Grams ok | Extras | Kcal within 25% | Kcal median err | p50 | p95 | $/scan | Note |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | google/gemini-3.1-flash-lite | 100% | 98.8% | 0 | 74.1% | 11.4% | 1.3 s | 2.7 s | 0.00074 | **Primary.** Cheapest usable, fastest, never invents an item |
| 2 | google/gemini-3.5-flash-lite | 97.6% | 92.7% | 0 | 81.5% | 12.7% | 1.4 s | 2.4 s | 0.00098 | Best kcal-ok rate but misses an item per pass and grams are worse; inside fixture noise, not adopted |
| 3 | google/gemini-3.8-flash | 100% | 100% | 0 | 70.4% | 15.2% | 4.0 s | 8.3 s | 0.00417 | **Fallback 1.** Ties on accuracy at 5.6x the cost and 3x the wait |
| 4 | google/gemini-3.7-flash | 100% | 97.6% | 0 | 70.4% | 14.2% | 4.4 s | 9.1 s | 0.00361 | Same as 3.8, no reason to prefer |
| 5 | google/gemini-3-flash-preview | 100% | 96.4% | 6 | 70.4% | 12.5% | 1.9 s | 3.9 s | 0.00117 | Invents items (soy sauce, nori) |
| 6 | openai/gpt-5.4-mini | 95.2% | 92.5% | 5 | 59.3% | 18.1% | 2.2 s | 4.0 s | 0.00161 | **Fallback 2**, purely the non-Google outage path |
| 7 | anthropic/claude-haiku-4.5 | 89.3% | 74.7% | 19 | 48.1% | 30% | 3.8 s | 7.7 s | 0.00320 | Worst usable: invents the most, misses the most |
| - | meta/muse-spark-1.3 | 100% | 97.6% | 4 | 51.9% | 21% | 10.7 s | 28 s | 0.00790 | TOO SLOW, dearest of all |
| - | z-ai/glm-5.3-flash | 100% | 92.9% | 5 | 59.3% | 22.4% | 7.6 s | 37 s | 0.00069 | TOO SLOW (one call hit 190 s) |
| - | openai/gpt-5.6-sol | 100% | 42.9% | 10 | 29.6% | 36.3% | 7.4 s | 35 s | 0.00694 | TOO SLOW and the worst grams of any model |
| - | anthropic/claude-fable-5.1 | invalid | | | | | 8.4 s | | 0.05 | Ignores the JSON schema, returns prose; 0 of 27 valid |
| - | anthropic/claude-opus-5 | invalid | | | | | 10.3 s | | 0.037 | Same, markdown instead of JSON |

Earlier pass the same morning, before the matcher fix (grams-only scoring, so
kcal columns are not comparable): google/gemini-2.5-flash 40.7% kcal-ok with
4 extras, gemini-3.6-flash 66.7% at 4.3 s, openai/gpt-5.4-nano 81% recall and
7 extras, openai/gpt-5.6-luna 35.8% grams-ok, qwen/qwen3.5-flash-02-23 a 400
from every provider, openrouter/free valid JSON on 4 of 9. None of them is
worth rerunning.

## What runs in production

`OPENROUTER_MODELS` = `google/gemini-3.1-flash-lite,google/gemini-3.8-flash,openai/gpt-5.4-mini`
(secret digest verified equal to the `vision.ts` default on 2026-09-06).
Gemini free-tier fallback when OpenRouter is down: `GEMINI_MODEL` =
`gemini-3.5-flash`, then `gemini-3.5-flash-lite`. Baselines for the primary
and first fallback are in `baselines/`.

Rerun before touching the list or the scan prompt:

```
deno run --allow-net --allow-read --allow-env --allow-write supabase/bench/run.ts --models <candidate> --baseline google/gemini-3.1-flash-lite
```