LUNA-TECH.CO← back to luna-tech.coField Note · 005 · Bench
PS/ALI AI Bakeoff: analysis & full aggregate.
Each model is fed civic address records (Seattle streets like Madison, Pike, MLK, Stewart, Aurora…; ZIP 98122 etc.) and asked to propose correction probes (expand Mad→Madison, fix a street suffix ST→AVE, etc.). Each probe is checked against authoritative reference data, so the harness can tell a validated correction from a guess. This is an E911-style address-validation task (PS/ALI = Private Switch / Automatic Location Identification, the feature that feeds per-extension location from a private switch/PBX into the 911 ALI database of record), which is why “don’t guess” behavior is scored as heavily as raw accuracy. A confidently wrong dispatch address is a safety failure, not just an error.
Summary chart — accuracy vs. calibration vs. contract failures.
Single-pass runs. X = valid probe rate (right is better) · Y = ECE / calibration error (up is better) · fill colour = share of responses lost to invalid JSON (green→red) · ring = cohort. Points are numbered and ranked in the side panel; coincident markers (e.g. #4 qwen and #5 llama-3.1-8b, nearly identical) are nudged apart with a thin tie-line to their true position. Read the story directly: #2 nemotron-30b and #4 qwensit top-right in green (the win); #1 gpt-oss and #3 gemini-flash are top-right but red: accurate yet emitting a third to a half of responses as malformed JSON the validator discards.
Findings.
Cohort 1: local models (n=10; firm)
- qwen3-coder:30b, the clear winner. ~63% valid, best ECE (0.22), zero contract violations on single-pass/CF, fast (~6.7s). But it’s aggressive (~28% authoritative-validation-failure + ~4.5% overreach), and its contract violations spike to 37.6 in stability mode.
- gemma-4-e4b: competent but ~4× slower (~27s) for less accuracy and worse calibration (ECE 0.38); documented JSON-breakage history in
.archive/. - llama3.2:3b, the floor: ~17% valid, uncalibrated (ECE 0.66), punts on ~69% of records. It is the most consistent model in either cohort, but only because it reliably does nothing useful. Stability ≠ quality.
Cohort 2: cloud / API (mostly n=3; directional)
- nemotron-3-nano-30b: best cloud model that emits valid output: top validity (~74%), best ECE (0.17), strong reasons (0.51). Cost: ~10s latency and the lowest stability in the cohort (reasoning model).
- llama-3.1-8b, the workhorse; only full 10/10/5 run, solid + fast + cheap, but terse reasons (0.19) and a contract-violation spike in stability (→98.6).
- gemini-2.5-flash, effectively broken here: ~48–60% invalid JSON, ~22 retries/run, despite good underlying accuracy. Flash-specific. Flash-Lite (~1% fails) is clean but very conservative.
- gpt-oss-20b (free): highest raw validity (~77%) but ~30% JSON failures. nemotron-9b (free) is clean but low-accuracy and slow (free-tier queueing, not the model).
Cross-cutting themes.
- Contract compliance dominates. Re-rank on “validity among well-formed responses” and Gemini-Flash / GPT-OSS jump tiers, but as shipped they’re unusable in this harness.
- Stability mode surfaces contract violations that single-pass hides (qwen 0→37.6, llama-8b 6→98.6), likely a high-volume edge case worth a harness look.
- Calibration cleanly tiers the field: ECE ≈0.15–0.25 trustworthy (qwen, nemotron-30b, llama-8b, gpt-oss, llama-70b); ≈0.34–0.43 overconfident (gemma, gemini); ≈0.66 = ignore its confidence (llama3.2).
- Conservative vs. aggressive is a real axis. For an E911 task the safe default is conservative, since overreach is the costlier error, so the “best” pick depends on whether a human reviews the queue.
Full aggregate tables.
Scalars averaged per target per mode, read from each run’s aggregate JSON block (reproduces Cohort 1’s official report exactly). Cell colour: green = better, red = worse, per each column’s ↑/↓.
Single-pass
| Provider:Model | Runs | Recs | Validated | Overreach | Probe Eff ↑ | Valid Rate ↑ | ECE ↓ | Contract Viol ↓ | Lat ms ↓ | Retr | Reason Q ↑ |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 lmstudio:google/gemma-4-e4b | 10 | 40 | 28 | 0 | 0.536 | 0.536 | 0.379 | 0 | 26920 | 0 | 0.41 |
| 1 ollama:llama3.2:3b | 10 | 40 | 9 | 1 | 0.174 | 0.174 | 0.662 | 1.5 | 2412 | 0 | 0.23 |
| 1 ollama:qwen3-coder:30b | 10 | 40 | 43 | 3.9 | 0.63 | 0.63 | 0.223 | 3.9 | 6704 | 0 | 0.49 |
| 2 google:gemini-2.5-flash-lite | 3 | 20 | 22 | 0 | 0.228 | 0.228 | 0.369 | 0 | 1939 | 0.3 | 0.55 |
| 2 google:gemini-2.5-flash | 3 | 20 | 10.3 | 0 | 0.685 | 0.685 | 0.337 | 10 | 10714 | 22.3 | 0.36 |
| 2 nvidia:meta/llama-3.1-8b-instruct | 10 | 40 | 160.4 | 4.5 | 0.627 | 0.627 | 0.219 | 5.9 | 5506 | 0.2 | 0.19 |
| 2 nvidia:meta/llama-3.3-70b-instruct | 3 | 40 | 81 | 0 | 0.61 | 0.61 | 0.314 | 0.7 | 21996 | 3.3 | 0.38 |
| 2 nvidia:nvidia/nemotron-3-nano-omni-30b-a3b-reasoning | 3 | 40 | 29.7 | 0 | 0.741 | 0.741 | 0.167 | 2 | 10509 | 3 | 0.5 |
| 2 openrouter:nvidia/nemotron-nano-9b-v2:free | 3 | 20 | 9.3 | 0 | 0.419 | 0.419 | 0.38 | 0 | 37183 | 1.7 | 0.44 |
| 2 openrouter:openai/gpt-oss-20b:free | 3 | 20 | 21 | 0.7 | 0.769 | 0.769 | 0.252 | 6.7 | 16474 | 8 | 0.44 |
Counterfactual
| Provider:Model | Runs | Recs | Validated | Overreach | Probe Eff ↑ | Valid Rate ↑ | ECE ↓ | Contract Viol ↓ | Lat ms ↓ | Retr | Reason Q ↑ | Sens ↓ |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 lmstudio:google/gemma-4-e4b | 10 | 40 | 27.7 | 0 | 0.529 | 0.529 | 0.349 | 0.5 | 26655 | 0 | 0.41 | 0.725 |
| 1 ollama:llama3.2:3b | 10 | 40 | 7.9 | 1.2 | 0.169 | 0.169 | 0.676 | 1.2 | 2205 | 0 | 0.2 | 0.525 |
| 1 ollama:qwen3-coder:30b | 10 | 40 | 42.8 | 4.3 | 0.627 | 0.627 | 0.233 | 4.3 | 6563 | 0 | 0.5 | 0.75 |
| 2 google:gemini-2.5-flash-lite | 3 | 20 | 18 | 0 | 0.207 | 0.207 | 0.395 | 0.3 | 2041 | 1.3 | 0.54 | 0.75 |
| 2 google:gemini-2.5-flash | 3 | 20 | 11 | 0 | 0.704 | 0.704 | 0.423 | 9.3 | 9321 | 22.3 | 0.39 | 0.667 |
| 2 nvidia:meta/llama-3.1-8b-instruct | 10 | 40 | 158.8 | 6.1 | 0.59 | 0.59 | 0.226 | 7.9 | 5875 | 0.1 | 0.2 | 0.55 |
| 2 nvidia:meta/llama-3.3-70b-instruct | 3 | 40 | 67 | 0 | 0.512 | 0.512 | 0.228 | 0.7 | 28405 | 5.7 | 0.37 | 0.417 |
| 2 nvidia:nvidia/nemotron-3-nano-omni-30b-a3b-reasoning | 3 | 40 | 29.3 | 0.3 | 0.647 | 0.647 | 0.245 | 3 | 8376 | 2.3 | 0.51 | 0.583 |
| 2 openrouter:nvidia/nemotron-nano-9b-v2:free | 3 | 20 | 9 | 0 | 0.394 | 0.394 | 0.371 | 0 | 43387 | 2.7 | 0.45 | 0.583 |
| 2 openrouter:openai/gpt-oss-20b:free | 3 | 20 | 21.3 | 0.7 | 0.728 | 0.728 | 0.148 | 6 | 14172 | 9.3 | 0.44 | 0.833 |
Stability
| Provider:Model | Runs | Recs | Validated | Overreach | Probe Eff ↑ | Valid Rate ↑ | ECE ↓ | Contract Viol ↓ | Lat ms ↓ | Retr | Reason Q ↑ | Dec Cons ↑ | Cand Cons ↑ | Conf Var ↓ |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 lmstudio:google/gemma-4-e4b | 5 | 376 | 243 | 0 | 0.454 | 0.454 | 0.434 | 4.2 | 29338 | 5.2 | 0.41 | 85.1% | 87.2% | 0.061 |
| 1 ollama:llama3.2:3b | 5 | 376 | 76.6 | 6.6 | 0.167 | 0.167 | 0.684 | 6.8 | 2251 | 0 | 0.21 | 97.1% | 99.2% | 0.005 |
| 1 ollama:qwen3-coder:30b | 5 | 376 | 391.4 | 37.6 | 0.61 | 0.61 | 0.24 | 37.6 | 6589 | 0 | 0.49 | 94.0% | 94.3% | 0.031 |
| 2 nvidia:meta/llama-3.1-8b-instruct | 5 | 376 | 1420.2 | 71.8 | 0.589 | 0.589 | 0.224 | 98.6 | 4668 | 0.2 | 0.19 | 85.1% | 91.4% | 0.04 |
| 2 nvidia:meta/llama-3.3-70b-instruct | 2 | 376 | 619.5 | 0 | 0.512 | 0.512 | 0.215 | 1.5 | 22385 | 24 | 0.38 | 92.1% | 94.4% | 0.013 |
| 2 nvidia:nvidia/nemotron-3-nano-omni-30b-a3b-reasoning | 2 | 376 | 253 | 5 | 0.644 | 0.644 | 0.229 | 16.5 | 8828 | 13 | 0.52 | 74.7% | 82.4% | 0.09 |
Decision-outcome mix (single-pass + counterfactual, % of records)
| Provider:Model | accept | accept+norm | suggest_review | needs_review | valid_fail | contract_fail | overreach_fail |
|---|---|---|---|---|---|---|---|
| 1 lmstudio:google/gemma-4-e4b | 22.5% | 23.8% | 5.8% | 42.0% | 5.9% | 0.1% | 0.0% |
| 1 ollama:llama3.2:3b | 20.4% | 7.9% | 0.1% | 68.8% | 0.0% | 0.1% | 2.8% |
| 1 ollama:qwen3-coder:30b | 22.5% | 32.4% | 4.5% | 8.6% | 27.5% | 0.0% | 4.5% |
| 2 google:gemini-2.5-flash-lite | 15.0% | 29.2% | 4.2% | 50.8% | 0.0% | 0.8% | 0.0% |
| 2 google:gemini-2.5-flash | 5.0% | 20.8% | 1.7% | 21.7% | 2.5% | 48.3% | 0.0% |
| 2 nvidia:meta/llama-3.1-8b-instruct | 22.1% | 3.4% | 29.5% | 37.9% | 0.0% | 3.4% | 3.8% |
| 2 nvidia:meta/llama-3.3-70b-instruct | 22.5% | 12.9% | 31.2% | 31.7% | 0.0% | 1.7% | 0.0% |
| 2 nvidia:nvidia/nemotron-3-nano-omni-30b-a3b-reasoning | 22.1% | 36.7% | 12.1% | 25.4% | 0.0% | 3.3% | 0.4% |
| 2 openrouter:nvidia/nemotron-nano-9b-v2:free | 15.0% | 18.3% | 10.0% | 56.7% | 0.0% | 0.0% | 0.0% |
| 2 openrouter:openai/gpt-oss-20b:free | 12.5% | 34.2% | 3.3% | 18.3% | 0.0% | 28.3% | 3.3% |
Coverage & caveats.
- Cohort 1 (local): full, with single-pass n=10, counterfactual n=10, stability n=5 for all three.
- Cohort 2 (cloud): uneven. Only
nvidia:meta/llama-3.1-8b-instructhas full 10/10/5. Gemini and OpenRouter free tiers cap at 3+3 with no stability (Google free 20 req/day, OpenRouter free 20/day). NVIDIA 70B and Nemotron-30B have 3/3/2. Theollamaandopenrouter google/gemma-4-31b-it:freedirs are empty. - Cohort-2 rows are mostly n=3 over 20 records → directional; Cohort-1 rows (n=10) are firmer.
- Counterfactual sensitivity is on ~4 groups, so it is noisy.
- Cohorts ran on different dates/config; cross-cohort comparison is approximate.
Source.
Harness, prompts, and judge live at github.com/lunatechsea/PSALI-Bakeoff. Run JSON is kept locally; the tables above are the aggregate scalars each run emits, which regenerate from the same harness against any model you point it at.
Validated / Records are per-run totals on different record sets (Cohort 1 + NVIDIA = 40-record set; Gemini/OpenRouter = 20; stability = 376). Use the normalized Probe Eff / Valid Rate for fair cross-model comparison, not raw Validated.