We tested Jev in our calorie tracker. Here’s why we didn’t ship it.

Our food matcher could turn “sugar-free syrup” into caramel popcorn. It could choose raw blueberries for dried blueberries, or barbecue beef for bolognese sauce. Those are real bugs, and a fast model that understands food names seemed like a promising way to fix them.

So we tested Jev, TypeSafe’s model for structured decisions, against our code and Gemini 3.5 Flash Lite. This follows our first journal on AI calorie estimates, but asks a different question: once we have identified a food, can a model choose a better nutrition record?

We kept Jev out of production and shipped four targeted matcher fixes instead. The experiment made a good case for better matching. It did not establish that adding another model would give users better calorie estimates.

Where Jev would fit

Jev accepts text, so it was choosing between food records after the photo had already been read. We gave it a food query, every usable candidate returned by the USDA search, and a “none fits” option. Calories and macros were still calculated in code.

The first result looked compelling

We built 60 labelled queries across 20 food families: 20 queries from our existing photo fixtures and 40 variations designed to catch mistakes. Models saw the full usable candidate list and made three passes. The code made one deterministic decision per query.

This was deliberately heavy on known traps. Its score describes this test set, not the proportion of real meals that KcalSnap gets right.

Jev was faster. The accuracy difference between the two models was much less persuasive: on the 21 held-out queries, their entire gap came from one beef query repeated three times. Accepting a reasonable cooked-ground-beef approximation made them tie.

There were weaknesses too. When no candidate fit, the code correctly declined 13 of 14 cases. Jev did so in 33 of 42 repeated decisions, and Flash Lite in 36 of 42. Reordering candidates while changing their option IDs changed each model’s chosen record on 12 of 60 queries. Some were equally acceptable alternatives, but a few changed correctness.

Then we tested the calorie total

We ran nine public meal photos three times each. For every scan, all five alternatives shared the same recognized foods, estimated grams and USDA candidates. Each then ran its own nutrition selection and actual AI fallback. Model selection had a 1.5-second deadline, falling back to the code’s choice on timeout.

A model moved one or two more scans inside a broad tolerance band, while median deviation increased. Mean deviations stayed close, around 15% to 16%. Those mixed results, against visually estimated references, gave us no defensible accuracy winner.

The missing piece in the first headline was our fallback: a rejected USDA match already goes to an AI nutrition estimator. It does not become zero calories or a failed scan.

That is worth investigating with independently documented meal references. It does not prove the AI estimator and USDA are equally accurate, or that we should remove the database. Portions were uncertain, fallback estimates varied, and changing one component can compensate for an error elsewhere.

Jev’s speed advantage was real in this test

Full-list Jev was faster than full-list Flash Lite on 21 of 27 paired scans, saving a median 275 ms. It was still a median 330 ms slower than the code baseline. We have not measured whether that difference is noticeable in the actual app.

Across both photo-test variants, Jev hit the 1.5-second deadline three times in 264 requests; Flash Lite hit it zero times in 264. Every timeout recovered through the code fallback. An earlier Jev shortlist request also reached a separate 20-second timeout. These are observations from a small run, not a provider reliability ranking.

Both models were inexpensive at this scale. Jev’s full-list selector cost about $0.000112 per lookup in the first test, versus $0.000486 for Flash Lite. Cost was not the deciding factor. Adding another model needs a benefit that survives the whole workflow.

What we actually shipped

The four concrete bugs were worth fixing regardless of which model won. These matcher changes are now live:

• Keep sugar-free meaningful. Sugar-free syrup cannot become caramel popcorn, and sugar-free is not treated as interchangeable with reduced sugar or no added sugar.
• Respect dried food. Dried blueberries cannot use the nutrition of raw, fresh or rehydrated berries. A suitable dried record can still be selected.
• Keep bolognese away from barbecue beef. A shared “beef” or “sauce” word cannot override an incompatible food identity. When no defensible match remains, use the existing AI estimator.
• Preserve lean-beef percentages. A 93% lean query keeps that constraint. It cannot quietly become 85% lean or unspecified beef.

We added regression tests using the captured candidate lists and refreshed the matching cache version. Existing saved meals keep their recorded values. We did not add Jev or change the production vision model.

The next benchmark starts with better source data

The next step is to find published meal photos or datasets that already pair the pictured portions with measured weights and documented calorie and macro references. After checking the source and reuse terms, we can use those references to compare portion estimates, nutrition matches and whole-meal totals.

For now, the four matcher fixes are live and the existing pipeline stays in place. Another model comparison should use a stronger reference set.

Read field note 001: Your AI recognizes lunch. Can you trust its calorie count?