KcalSnap Journal, field note 002 We tested Jev in our calorie tracker. Here's why we didn't ship it. Completed September 18, 2026 https://kcalsnap.com/journal/jev-food-matcher/ Scope ----- This is KcalSnap's own development experiment, not an independent evaluation. No private user photos or meals were used. Public Wikimedia Commons fixture examples and their credits appear at https://kcalsnap.com/journal/ai-calorie-count/#sample-meals. All code scores use the matcher before the four September 18 fixes. The fixes are regression-tested corrections, not a new calorie-accuracy benchmark. Jev chose a USDA food record or none from a text query and candidate list. It did not inspect photos, estimate grams or calculate calories in this test. Models: typesafe/jev-1.13 (recorded resolved version typesafe/jev-1.13-20260917) and google/gemini-3.5-flash-lite via OpenRouter. The photo experiment kept google/gemini-3.1-flash-lite as its vision model. Research assistance ------------------- Astra at xhigh effort orchestrated the benchmark analysis. Fable 5.1 at medium effort contributed review recommendations along the way. These were research roles, separate from the food selectors under test. Figures come from recorded benchmark runs and calculations. KcalSnap remains responsible for interpretation and the production decision. Matcher test ------------ 60 queries in 20 food families: 20 existing photo-fixture queries and 40 synthetic contrasts against the same saved USDA pools. This stresses known traps. It measures selection given retrieval, not search recall or real-user query frequency. Labels and prompts were fixed before calls and code scoring. Whole food families were split into 39 development and 21 held-out queries. Production rules had historical exposure to some families. Labels accepted generic meat pasta sauce for bolognese, plain hamburger bread for a sesame bun, and regular sour cream for an unspecified simple sour cream sauce. Major named ingredients, dietary constraints and preparation state still had to match. Labels are semantic judgments, not nutritional equivalence tests. Models made three identical-input passes; code was deterministic. Full-list models saw all candidates with usable macro data, at most 100, and an explicit none option. They saw the same semantic policy, no labels or reference calories. Original USDA rank was preserved for code scoring. All-query results: Code: 44/60 correct, 7 wrong foods, 9 false rejections. Jev full: 168/180 correct, 9 wrong foods, 3 false rejections. Flash Lite 3.5 full: 164/180 correct, 13 wrong foods, 3 false rejections. Correct none when nothing fit: code 13/14; Jev 33/42; Flash Lite 36/42. Median selector request time: Jev 333 ms; Flash Lite 585 ms. Mean billed selector cost per lookup: Jev about $0.000112; Flash Lite about $0.000486. These exclude the rest of a scan. Held-out correct: code 12/21; Jev 60/63; Flash Lite 57/63. Accepting a practical cooked-ground-beef approximation instead of requiring a patty-specific record makes both models 60/63. The entire held-out gap is one query repeated three times. The completed shuffle reran five selection arms on all 60 queries. It reordered candidates, renamed positional IDs and moved none to the front. This does not isolate order alone. Each full-list model changed record on 12/60 queries. Jev had four correctness flips (three improvements, one regression); Flash Lite five (three improvements, two regressions). Some changed picks were equivalent. The code's top-eight shortlist retained an acceptable record in only 12 of 16 held-out queries that had one. It did not advance to photos. An initial Jev approval gate on the code's single top pick rejected too many usable records. This is distinct from the later code-veto hybrid. Veto composition ---------------- When code said none, retain none and skip model selection. Otherwise, let the model choose from the full list. In the query test, the veto blocked 22 queries, including nine with acceptable candidates that a model behind it cannot rescue. Query-level results reuse the identical recorded model calls offline, applying a 1.5-second fallback sensitivity to elapsed times. They are not fresh evidence. Jev hybrid: 151/180 correct, 29 false rejections, zero wrong foods. Flash Lite hybrid: 144/180 correct, 30 false rejections, six wrong foods. Paired photo experiment ----------------------- Nine public photos, three fresh vision draws each: bolognese, English breakfast, burger and fries, cepelinai, oatmeal, pancakes, pizza, Caesar salad and sushi. All use visually estimated portion references and have prior development exposure. Each shared vision result supplied identical foods, grams and USDA candidates to five separately executed nutrition arms. Vision used the production prompt, schema, temperature and routing, with a common 4096 output-token cap. Nutrition used the real existing AI macro estimator when no food record was selected. Each arm made fresh selector and fallback calls. Items ran concurrently, photos sequentially, and arm order rotated. Totals: 27 shared scans, 151 detected items per arm, 135 nutrition outcomes, 528 selector calls, 69 macro calls. Selector deadline: 1500 ms, then return the exact baseline code choice. An intentional model none still uses the AI estimator. All photo timeouts recovered: Jev full 2/151, Jev veto 1/113; Flash Lite 0/264 across both variants. All 27 vision and 69 macro requests succeeded. An earlier Jev shortlist request reached a separate 20-second timeout and stopped that batch. The continuation retained that failure and made one later successful attempt. The three photo deadlines and earlier timeout are different observations. These small counts do not establish vendor reliability rates. Meal calories sum each predicted portion times its resolved nutrition per 100 g. References use visually estimated portion ranges and reference nutrition values. Within tolerance means within 25% of that estimated reference. The references do not contain measured portion weights. Three repeats do not make each photo three independent meals. Arm Within band Median deviation Added paired time Code before fixes 21/27 10.9% Baseline Jev full 22/27 15.2% +330 ms Flash Lite 3.5 full 23/27 18.4% +579 ms Code veto + Jev 22/27 14.5% +450 ms Code veto + Flash Lite 3.5 22/27 16.3% +630 ms Mean absolute reference deviations, same order: 15.94%, 15.82%, 16.23%, 15.32%, 15.99%. AI-estimated items, same order: 38, 7, 14, 41, 42 out of 151. Means, medians and broad tolerance counts answer different questions. None establishes an accuracy winner here. Fresh macro estimates vary across arms; calorie changes cannot all be attributed to record selection. Compensating errors are possible. Similar results do not establish that AI and USDA paths are interchangeable. Timing ------ Pipeline time is measured vision duration plus shared maximum USDA retrieval duration plus independently measured nutrition execution. Four of 53 pools needed fresh USDA retrieval, up to 3402 ms. Most retrieval hit a warm local raw cache. The common retrieval barrier differs from production per-item overlap. Timing excludes phone upload, server startup, database food-cache hits and rendering. Logging adds local overhead. This is not deployed mobile UX timing. Pipeline p50 / p95 seconds: Code: 2.690 / 5.599 Jev full: 2.749 / 5.969 Flash Lite 3.5 full: 2.666 / 6.387 Jev veto: 3.098 / 6.468 Flash Lite 3.5 veto: 3.376 / 6.714 Added times compare arms on the same scan. The median of paired differences is not the difference between overall medians. Full-list Jev was faster than Flash Lite on 21/27 pairs, with a median 275 ms saving. A descriptive 10,000-resample bootstrap clustered by photo gave 232 to 337 ms for that saving. This describes this small run, not future SLAs or whether a user would notice the difference. Decision -------- Do not add either selector or the blanket code veto to production on this experiment. Jev improved semantic matching and was faster than Flash Lite, but neither showed a clear calorie benefit over the complete existing pipeline. The four deterministic bugs were fixed and deployed separately. Next: source published meal photos or datasets that already include measured portion weights and documented nutrition references for the pictured food. Check provenance, image-to-portion correspondence and reuse terms before using them. Separate portion error, nutrition error, coverage and whole-plate results. Report USDA-matched and AI-estimated items separately, without assuming different food groups are equal. Public aggregate data: results.json in this directory. Raw operational logs, credentials and deployment records are not part of this public release.