# Nutrition5k findings, September 18, 2026 Keep Gemini 3.1 Flash Lite as the production primary model. Gemini 3.5 Flash Lite is the strongest challenger in this pilot: its calorie error is lower at similar speed, but the paired uncertainty interval includes no improvement. Gemini 3.8 Flash does not justify its extra latency and cost here. No production model or fallback settings were changed. This is a completed comparison on existing published images, with no meal preparation or weighing by this project. We used 48 fixed Nutrition5k source groups, two passes per model, the current matcher v4 and each model's own macro fallback. That produced 288 pipeline outcomes from 388 API requests, including 100 macro calls. Total confirmed API spend was **$0.602433**, with no unresolved charges. Jev was excluded because this experiment evaluates photo understanding. ## Results | Model | Valid scans | Within 25% of reference calories | Calorie MAE | Median calorie error | Warm pipeline median / p95 | API cost per 1,000 scans | | --- | ---: | ---: | ---: | ---: | ---: | ---: | | Gemini 3.1 Flash Lite | 96/96 | 48/96 | 103.0 kcal | 25.1% | 2.04 / 3.36 s | $0.70 | | Gemini 3.5 Flash Lite | 96/96 | 52/96 | 94.9 kcal | 20.3% | 1.94 / 3.30 s | $0.98 | | Gemini 3.8 Flash | 95/96 | 47/96 | 95.8 kcal | 25.6% | 4.89 / 15.86 s | $4.59 | Median calorie error means median absolute percentage error across successful scan outputs. MAE is mean absolute error. Costs include vision and any macro fallback call, including unsuccessful attempts. These are observed API costs, excluding infrastructure. Two passes do not turn 48 groups into 96 independent meals. The 3.8 accuracy metrics omit its failed result; the success and within-25% denominators retain all 96 attempts. | Model | Protein / carbs / fat MAE | Protein / carbs / fat WAPE | Total mass MAE | | --- | ---: | ---: | ---: | | 3.1 Flash Lite | 7.6 / 11.2 / 8.6 g | 29.7% / 40.7% / 48.0% | 75.6 g | | 3.5 Flash Lite | 8.5 / 9.2 / 7.4 g | 33.2% / 33.4% / 41.2% | 67.3 g | | 3.8 Flash | 8.0 / 8.8 / 7.2 g | 30.9% / 32.4% / 40.5% | 68.5 g | WAPE divides total absolute error by total reference quantity. It includes zero-reference components without dividing individual errors by zero. ## Why the recommendation stays unchanged For each source group, we averaged calorie percentage error across both passes, then compared the models on the same groups. A 10,000-resample bootstrap sampled whole groups, not individual calls. - **3.5 versus 3.1:** mean calorie error improved by 8.7 percentage points. The paired 95% interval was -19.8 to +1.0 points, so this does not establish a reliable improvement. Carbohydrate and fat errors improved; protein error increased. Its extra cost was about $0.29 per 1,000 scans and its median speed was similar. It is a credible candidate, but it missed the frozen confidence criterion for switching. - **3.8 versus 3.1:** mean calorie error improved by 8.6 points over 47 complete pairs, with an interval of -20.8 to +1.3. Its warm median was 4.89 seconds and p95 was 15.86 seconds, exceeding the frozen 15-second p95 target. Total API cost was about 6.6 times the baseline. - All 288 vision calls parsed successfully. One 3.8 macro call failed because the upstream provider returned a rate-limit error inside an HTTP 200 response with no answer content. It was reported by the production helper as `OpenRouter returned no content`. There was no paid retry or model substitution. One event is insufficient to estimate a general provider failure rate. This experiment does not establish which model should handle production fallback traffic: those requests may differ from ordinary primary-model requests. Keep that configuration unchanged too. No additional paid comparison is scheduled. ## Scope and integrity The [Nutrition5k authors](https://github.com/google-research-datasets/Nutrition5k) provide photographed portions with measured ingredient masses and USDA-derived nutrition references under CC BY 4.0. We froze the source selection, original RGB bytes, prompts and scoring before requests. We filtered inconsistent source records and conservatively grouped related ingredient-addition stages before selecting one image per group. Grouping is inferred, not an official independent meal identifier. All 48 selected records come from cafe1. Warm pipeline time adds measured vision execution to measured nutrition execution against frozen USDA candidates. It excludes USDA network retrieval, upload, edge startup, rendering and hosted food-cache hits. Fresh USDA retrieval took a median 0.85 seconds per query, logged separately; it cannot simply be added once to each scan because scans contain different numbers of queries and production can use caches or concurrent requests. This is not a measured phone-to-result latency. Successful outputs contained 341 USDA / 43 AI items for 3.1, 312 / 43 for 3.5 and 340 / 38 for 3.8. These are coverage counts, not source-specific accuracy scores. Reference ingredients and predicted item nutrition are preserved, but independent item correspondence labels are absent. We therefore cannot cleanly separate portion, food-identification and nutrition-record errors, or claim USDA and AI are equally accurate from this run. Whole-plate errors retain omitted foods. The specialized camera, single-cafeteria subset, consistency filter and possible public training exposure limit generalization. USDA-derived reference nutrients can also favor agreement with the same database. These results are stronger references than visually guessed portions, but not universal app accuracy claims. The integrity audit reconstructed every request hash, checked all 395 pool hashes, replayed recorded food selections through matcher v4, verified meal macro calculations, reconciled all charges and checked for leaked credentials. A local cache serialization failure during USDA retrieval was repaired without repeating paid vision work. The original runner, original freeze and storage-only amendment remain available. See computed results, machine-readable report, frozen protocol and reproduction guide.