# DFood-TJ development diagnostic and confirmation

Authorized 7 October 2026. New research budget: USD5 total, including durable
reservations for unavailable billing records. First diagnostic stage: at most
USD1. No customer API credits, hosted scan records or private photos are used.
The existing KcalSnap research credential is used; billing settings stay fixed.

## Before inference

Use the 179 accepted images from the versioned masking audit: 61 development,
118 confirmation. Keep original references unchanged. `prepare.py` verifies
every prepared image hash and creates separate provider inputs: RGB JPEG,
maximum edge 1600, no upscale, LANCZOS, quality 90, subsampling 0. These fit the
paid API's 2 MB / 2048-pixel limits. This encoding is distinct from the lossless
mask preparation. Save actual request-image hashes and encoder provenance.

Programmatically audit reference sums, finite positive mass/energy, source
participant separation and identical food/weight vectors across splits before
any calls. Export opaque participant cluster codes, never source participant
identifiers or clinical fields. Flag a >25% discrepancy between source kcal and
4/4/9 macro-derived energy for sensitivity analysis, not label repair/exclusion.
The duplicate-record label audit already found exact agreement in four pairs.

## Development diagnostic

The first 12 accepted development images in the frozen order, one repeat of
`google/gemini-3.1-flash-lite`, form the initial diagnostic. Use the current
production prompt, matcher v5, schema, model parameters and validation. Inspect
these outcomes to identify recognition, mass, nutrition retrieval and reference
alignment issues. Mass/density ratios are arithmetic diagnostics, not causal
proof of error sources. Source prepared-dish labels need not enumerate every
visible ingredient. Never teach the model this dataset's answer vocabulary.

The benchmark runs the actual paid API pipeline with a stand-in database and
an evidence-backed fetcher. Internal model/reasoning injection defaults to the
unchanged production configuration and is not exposed by the HTTP handler.
Synthetic fixtures verify request identity, fallback and rounding. Preserve
integer grams and rounded per-item/meal totals. Recognition deadline 25 seconds,
nutrition deadline 20 seconds, maximum 4096 output tokens, no automatic retry or
model replacement. Same-model nutrition fallback uses the production body.
Normalize food queries exactly as production. Retrieve and save missing USDA
pools outside the timed model path, with a 250-request production quota reserve.
Use frozen pools for all arms. Report the sum of recorded model-stage latency separately
from cold retrieval; this is not the full hosted-service latency or availability.

Cache each recognition and nutrition call immutably by actual image/request
hash, requested model and repeat ordinal. Preserve provider, resolved model,
response ID, usage, latency, raw response and original timestamp. Each food
retains recognition and nutrition provenance. Save responses before scoring.
Failed/unknown-charge calls remain recorded and are never silently repurchased.
Missing evidence, budget or infrastructure errors pause the run for repair;
they do not become model failures or compatibility stops.

After diagnosis, choose at most one candidate configuration for this experiment
and give it and the incumbent the same bounded development harness pilot. A
candidate may be a justified prompt/matcher change or one alternative model.
Version changed requests, keep old responses, test shared fixes on both arms.
Declare development expansion and numeric confirmation criteria in a frozen
phase file before its new calls. Do not repeatedly search confirmation outcomes.

## Confirmation gate

Before any confirmation inference, freeze source/code hashes, arms, order,
repeats, numeric success/reliability/latency/cost gates and failure rules.
Primary outcome is attempted-meal fraction within 25% of source calories;
failures count as unsuccessful. Also report measured mass, macro error, signed
bias, masked/no-cue strata and reference-energy sensitivity. Bootstrap paired
participant clusters, keeping repeats and meals together. Report complete-case
accuracy only with denominators and both-arms-success diagnostics.

If development identifies no credible candidate, complete a declared baseline
confirmation to characterize accuracy without a switching claim. If a candidate
advances, interleave both arms within the same time window. Confirmation is
consumed once; later improvements require new evidence, not relabelled reruns.
Keep production settings unless the frozen evidence supports a useful change.

Public training exposure, hospital-meal domain bias, source table assumptions,
incomplete ingredient annotation and undetected repeated physical meals remain
limitations. Two repeats are not two independent meals. Model coordination is
network waiting; no model training or heavy CPU/GPU batch runs on the Mac.


# DFood development and confirmation decision rules

Frozen before fresh paired development. The initial diagnostic pilot and its source snapshot remain immutable at commit a5efbf7 (runner source 392de4e). Reproduce it at that commit. Later phases require matcher v6; they never overwrite the pilot.

## Changes and funding

Both model arms use the same v6 matcher, prompts, production pipeline, deadlines, normalized queries, frozen USDA pools and empty food cache. v6 fixes explicit recipe-component exclusions and steamed bread identity. These are semantic corrections, not fitted nutrient or portion calibration.

Phase spending caps total $5: pilot $1, cached-recognition repair $0.25, fresh development $0.75, confirmation $3. The ledger reserves unknown charges at the request bound. No paid API key, customer credits, billing limits or subscriptions are changed. No retries or substituted model IDs. Concurrency is one.

Repair reuses the 12 pilot recognition calls (repeat 0), buying only newly required fallback nutrition. Development runs all 61 development photos through Gemini 3.1 Flash Lite and Mistral Large 4.0, with reasoning none explicitly on both Mistral stages, once each (repeat 1). Order is hashed independently of outcomes. New development and confirmation refuse historical cache responses; same-phase resumes reuse completed requests. Every reply and requested/resolved model, provider, image hash, request contract, timing, cost and per-food source are saved.

An arm stops for compatibility after at least eight attempted meals if more than half fail. All planned but unattempted meals remain unsuccessful for the primary endpoint. Infrastructure, missing evidence, credentials, budget and retrieval faults pause the experiment, rather than becoming model failures. No outcome-dependent sample additions or removed meals.

## Development selection

At most one challenger. It advances only if every gate passes:

- Successful scans within 25% of the source calorie reference increase by at least 5 percentage points, over all planned meals including failed/stopped attempts.
- Completion falls by no more than 3 percentage points.
- Conservative cost per completed meal (all attempts, unknown charges at their reservations) is no more than 2 times baseline.
- Model-stage p95 latency is no more than 1.2 times baseline. This excludes cold USDA retrieval, database and hosted API overhead.
- On paired meals both arms completed, protein, carbohydrate and fat WAPE each worsen by no more than 5 percentage points.

These are practical promotion filters, not evidence that a rejected model is intrinsically inferior. Development estimates are exploratory and have selection bias. If the candidate fails any gate or compatibility stops, confirmation characterizes the corrected baseline only. The successful-scan fraction within 25% of measured mass is a key secondary result.

## Confirmation

Use all 118 frozen confirmation photos, twice (repeats 2 and 3). Do not tune after observing them. If the challenger qualified in development, run both arms, otherwise only baseline. The image is the unit: each image's success/completion score is the mean of its repeats. Report a fixed-seed, 10,000-resample participant-cluster percentile bootstrap 95% CI (34 participants), preserving all photos and repeats within sampled participants. Report image-weighted point estimates and intervals; also group identical ingredient-weight vectors for a sensitivity interval, since repeated menus may correlate across participants.

A model switch requires all development gates on confirmation plus a strictly positive lower bound for the paired primary improvement CI. The direction of primary improvement must remain nonnegative after separately removing the source energy-discrepancy flags and cross-split identical ingredient-weight vectors. These subsets are sensitivity analyses, not replacements for the full denominator. Report per-repeat and masked/no-cue results. No imputed nutrition for failed scans; conditional calorie/macro errors must name their denominator.

Freeze the report/statistics source hashes, model config, image and label hashes, source code and this protocol before the first confirmation request. Development and confirmation use distinct repeat IDs; no historical calls are reused. No extra confirmation repeats to seek significance.

## Interpretation limits

DFood is public hospital food, not a fresh private benchmark. Images and participant groups are disjoint after documented duplicate exclusions, but repeated physical meals cannot be ruled out. Calories and macros are prepared-dish table labels, not laboratory measurements; the ingredient lists can omit visible components. Source calorie/4-4-9 discrepancies are flags, not proof of erroneous labels, and remain in the main analysis. Portion mass is independently informative but whole-meal mass/density cancellation does not identify a cause by itself.

This experiment estimates performance on these photos and this provider window. It does not establish global dietary accuracy, causal acquisition gains, customer retention or business profitability.


# Pre-confirmation development amendment

Recorded after the fresh v6 development inference and label-blind query/USDA audit, before computing the development calorie gates or any confirmation call. The previous phases and their freezes remain immutable.

The 83 distinct successful development query/USDA pairs exposed further semantic defects in both arms: colored rice could select rice-and-beans, plain beef selected bacon, a dressed coleslaw selected dressing alone, steamed cabbage selected kimchi, a breading query selected yeast bread, and bulb onion selected green onion. Follow-on full-pool tests exposed offal and cured meat alternatives behind the original top matches. These corrections become matcher v7; there is no prompt, portion multiplier or nutrient-label adjustment.

Replay all61 development photos for both arms using their existing repeat1 recognition responses and the same production core with v7. Preserve unchanged nutrition calls; buy only newly required fallback batches. The replay is exploratory development, including rebuilt nutrition-batch sampling noise. It is not a fresh latency experiment. Report the original v6 comparison alongside this corrected comparison, including the49 photos outside the diagnostic pilot. A candidate must pass the fixed practical gates on the corrected pipeline; no failed or timed-out scan is repurchased. The one Mistral recognition timeout remains unsuccessful and its unknown cost remains reserved.

The five phase caps remain a total of $5: pilot1, repair0.25, development0.75, development-v7 0.25, confirmation2.75. Cost per completed meal uses the original recorded charges for reused stages and reservation bounds for unknown charges, while new experiment spend counts each purchased request only once.

Retain the original p95 and cost gates. Add a stricter measured-mass guard: the candidate's successful-within25%-mass fraction may not drop more than3percentage points. On61 development images, the calorie improvement requires at least4 additional successes; completion and mass can each lose at most1 success. This added guard addresses possible calorie-error cancellation. No gate is loosened after seeing results.

Before confirmation, freeze the exact analysis code and all runtime dependencies including reservation code. Unknown harness errors pause; only explicit provider-response/parse/validation failures and full stage timeouts count against a model. Null-status connection failures before a deadline pause. Credential/billing failures remain immutable evidence and pause, requiring a documented amended repeat before another purchase. No implicit cache deletion or retry.

If the challenger fails the development gates, run only corrected Gemini on all118 confirmation photos twice. If it passes, run both arms with v7. All original confirmation rules apply, including failure-inclusive denominators, fixed participant bootstrap, recipe-vector sensitivity and explicit public-data/reference limitations. Do not tune after confirmation outcomes. Completion, mass, calories, macros, summed model-stage time and conservative costs remain separate results.


# Model selection before confirmation

All61 development photos ran once per arm with matcher v6. Gemini completed61/61, Mistral60/61 with one25-second recognition timeout. Gemini had33/61 scans within25% of the source calorie label; Mistral29/61. The timeout is a failure, never a free replacement scan.

After the documented label-blind query audit, matcher v7 was applied to both arms using the same saved recognition calls. Only newly required nutrition batches were purchased, costing $0.00548227. Results became33/61 versus28/61. Paired participant-cluster primary difference is -8.20percentage points, 95%CI[-21.15,+3.85]. This does not establish that Mistral is intrinsically inferior; it supplies no positive evidence to justify promotion.

Mistral also misses the predeclared cost, p95 latency, macro and measured-mass gates. Conservative cost per completed meal is $0.00076052 Gemini versus $0.00282681 Mistral, including the original timeout reservation. Model-stage p95 is4.111 versus9.929seconds. These mix recorded stages in the v7 replay and exclude cold retrieval and hosted overhead. Measured mass within25% is38/61 versus32/61.

Confirmation therefore evaluates only corrected Gemini3.1FlashLite with matcher v7, all118 confirmation photos, repeats2 and3. No production model, prompt or prices change. This is baseline characterization, not a held-out win over Mistral. No more tuning after confirmation results.

The original and corrected development reports, diagnostic rows and food provenance are preserved. The original v6 recognition and macro evidence can be replayed at0f097c2; the corrected replay uses its own immutable freeze. All report confidence intervals are resampling uncertainty conditional on this public dataset, not a promise of future service reliability.


# Final review before confirmation

Claude Opus 5.5 at xhigh completed three advisory passes on the experiment method, actual production pipeline, saved-pool audits and analysis implementation. The selected account was verified by the worker. Review metadata and full private transcripts remain in the session scratchpad; no credentials, customer data or confirmation photos/labels were provided.

Accepted fixes include actual production-core injection, strict model identity, same-model nutrition fallback, frozen USDA pools and reply cache, participant-cluster bootstrap, failed/stopped denominators, unknown-charge reservations, isolated harness versus model failures, and a label-blind review of both models' matched queries. The latter exposed concrete shared matcher problems, fixed before confirmation.

The final pass caught lexical collisions: `corned` must not be compared as the stem `corn`, and offal words such as kidney/heart must be restricted to animal descriptions. It also caught salad-dressing queries being mistaken for whole dressed salads, a weak redundant fixture, and body-read transport failures after HTTP headers. These are corrected and regression-tested. They do not change any of the 685 saved query/pool decisions relative to the preceding v7 draft. A zero-budget, network-denied development-final replay verifies the full pipeline again without purchasing responses.

Confirmation configuration is checked before analysis: all118 prepared confirmation IDs, repeats2/3, v7, fresh cache, mass guard and compatibility stop. An all-failed synthetic preflight exercises every reference join, cluster/vector/menu key, denominator and result path without inference, saving no synthetic result files. All real source weights and labels remain unchanged.

The primary uncertainty interval remains participant-cluster resampling. Exact ingredient-weight-vector clustering is only an exact-recipe-vector sensitivity and does not establish menu independence. A separate sensitivity groups the sorted source food names, ignoring weights, to capture repeated source menu labels. It is still an approximate grouping, not proof of physical-meal independence. All-success bootstrap completion intervals can be degenerate and are not a guarantee of future reliability.

Original p95/cost thresholds are retained; no gate was relaxed after development began. One timeout's reservation raises Mistral's reported conservative cost, but even excluding that unknown charge its recorded cost per completed meal remains over2times Gemini's, and its accuracy, mass, macro and latency gates also fail. No rerun is purchased to erase that failure.

Rejected suggestions: do not silently delete cached credential/billing failures or repurchase ambiguous calls; preserve evidence and require a documented amended repeat after fixing infrastructure. Do not tune portions, inferred oil, query language or nutrient labels to fit these photos. Public calorie references and generic food matches retain uncertainty.

Known generic matches still include braised beef mapped to short ribs, steamed cabbage to red cooked cabbage, generic bean sprouts, and generic steamed chicken. These are not claimed to be exact recipes. Record occurrence counts in confirmation; do not tune them using confirmation outcomes. The returned food sources and cached responses make subsequent independent tests possible.