← Test Log · AI Lab Portal Home
DeepSeek-R1 (7B) was added to the lab as a second model alongside qwen3:8b, and web search was enabled for it the same way as everything else on this instance — a master toggle, a search backend, a per-chat switch. Four manual test runs with an identical post-cutoff prompt (a real event the model's training data couldn't know about) all came back as confident, detailed fabrications: no error, no search status line, no citations. That symptom — a toggle that's accepted but silently does nothing — is what kicked off this investigation.
| Prompt | "Tell me about the state of the investigation into the seattle shooting near the space needle where 3 people were killed" — a real event (Bite of Seattle festival, Seattle Center, 3 killed / 4 wounded) that postdates the model's training cutoff, so any answer not describing it with citations means search didn't ground the response |
|---|---|
| Method | Reproduced live in the browser (not a scripted API call) with Open WebUI's backend log open, then cross-referenced the actual open_webui package source to trace the code path |
| Result | Fabricated a fictional 2017 "Space Needle shooting" with an invented suspect name, zero citations, zero search status indicator — and zero web-search-related log lines anywhere in the backend log for the request. No exception. Nothing to catch. Nothing was ever called. |
Root cause #1 — the actual silent-failure bug. DeepSeek-R1 (7B)'s Function Calling parameter was unset, which resolves to Open WebUI's default, Native. In the backend's middleware.py:
Open WebUI's forced, always-on search path only runs when a model is set to Legacy. In Native mode, search is instead exposed to the model as a tool it has to choose to call itself — and DeepSeek-R1 via Ollama never emits that tool call. The search code simply never executes. Nothing fails, because nothing runs.
Root cause #2 — found while verifying #1. Switching the model to Legacy forced the search path to fire — and it failed too, but visibly this time: a real "An error occurred while searching the web" status line, and a backend traceback showing the search library posted to DuckDuckGo, got back an HTTP 202 Accepted, and then raised DDGSException: "No results found." anyway — consistent with DuckDuckGo soft-blocking the request rather than a hard network failure.
Both of Test 28's fixes were applied together: DeepSeek-R1 (7B)'s Function Calling set to Legacy (opening the forced-search path), and the instance's web search engine switched from DDGS to Brave (replacing the backend that was failing underneath it). The Brave API key was entered directly by the lab's human operator — never handled by either Claude Code instance, the same credential boundary used for SSH key setup elsewhere in this lab.
| Test run | Identical Test 28 prompt, new chat, Web Search toggle on |
|---|---|
| Result | "Searching the web" → "Searched 4 sites" → "Retrieved 2 sources" → "Thought for 26 seconds" → a grounded answer citing bbc.com and apnews.com, correctly describing the real event: 3 killed, 4 wounded, one suspect arrested, second sought |
| Backend log | Confirms the full pipeline ran end to end — search → fetch → chunk → embed (98 items across 4 sites) → retrieve — not just a UI status line with nothing behind it. Zero exceptions. |
| Latency | Generation itself (26s) landed inside the Test 28 weights-only baseline range (26/35/20/21s) — search didn't slow down generation. Total wall-clock is longer once search + fetch + embedding (~45–50s, happening before generation starts) is counted — the expected cost of a correct answer instead of a fast wrong one. |
"The toggle is on and nothing errors" is not the same as "it's working." The most dangerous failure mode here wasn't that search broke — it's that it broke in a way that produced confident, well-formatted, citation-free prose indistinguishable in tone from a correct answer. Diagnosing it required reading the actual backend source, not just watching the UI.
Search now worked — but a different, subtler failure showed up almost immediately. Asked a two-part factual question, DeepSeek-R1 retrieved real sources, cited them, and still got half the answer wrong.
| Prompt | "Which two teams made it to the 2026 FIFA World Cup Finals?" — a countable, objectively-scorable question (ground truth: Spain and Argentina) |
|---|---|
| Motivating example | "Retrieved 2 sources," thought 12s, answered "Spain" only — grounded, but incomplete. Categorically different from Test 28's total fabrications. |
Three candidate layers could explain this: A — the model's generated search query was too vague; B — the right query ran, but retrieval/chunk-selection missed the answer; C — the answer was retrieved but the model failed to use it. Three runs, each with the full chat record pulled directly from the API to get the exact generated queries and the exact chunk text actually injected into context — not inferred from the UI.
| Run | Query quality | Correct Wikipedia page fetched? | Answer | Score |
|---|---|---|---|---|
| 1 | Weak — never mentioned "final" | No — the two most authoritative FIFA pages failed to fetch, silently skipped | France & Argentina | 1/2 |
| 2 | Good — explicitly said "Final" | Yes — but the top-k chunks selected were boilerplate navbox content, not the lead section | France & Argentina | 1/2 |
| 3 | Weak — misread "Finals" as "qualifications" | Yes — but only "Argentina" made it into any retrieved chunk; "Spain" never did | Mexico & Colombia | 0/2 |
Layer attribution, measured: A (query formulation) was contributory but not sufficient alone — Run 2 had a genuinely good query and still failed. B (retrieval/chunk-selection) showed up in all three runs, in different forms — an outright fetch failure, or the right page fetched but the wrong section selected within it. C (synthesis) was a real, contributing defect in every run — the model never hedged when its grounding was incomplete, instead answering with confident, specific, wrong guesses. Run 1 went further and blended in outside knowledge (a correct venue detail that wasn't in any retrieved text) on top of a wrong guess.
A follow-up review (Test 31, below) caught two errors in Test 30's original write-up, and both were corrected directly in the log entry rather than silently — the mistakes and the fixes are both part of the record:
Score against ground truth, every time, not against what a previous run said — and verify a strong claim ("synthesis alone is sufficient") against the actual underlying data before writing it down, not just the pattern it looks like from the surface.
Two back-to-back runs of the same prompt, same config, five minutes apart, gave opposite outcomes — one had search never fire at all, the other fired and answered correctly. That meant query generation itself was nondeterministic, and Test 30's three runs were too few to separate real signal from ordinary variance. This test didn't chase a mechanism — it produced a number.
| Prompt | "Which two national teams played against each other in the 2026 FIFA World Cup final match on July 19, 2026?" — run verbatim, 10 times, each in a fresh chat |
|---|---|
| Query generation rate | 6 of 10 runs generated a search query at all. The other 4 answered from weights alone. |
| Score when search fired (6 runs) | 2/2 × 5, 1/2 × 1, 0/2 × 0 — every single fired run scored at least half-correct, and 83% scored fully correct |
| Score when search did NOT fire (4 runs) | 0/2 × 4 — 100% failure, each wrong in a different way (a stale "as of 2023" self-anchor, a wrong host country, two different wrong team pairs) |
This reframes everything Test 30 found. The dominant lever isn't retrieval quality or synthesis quality — it's the roughly 60/40 coin-flip on whether DeepSeek-R1 decides to search at all. Once it does, it's right most of the time. Test 30's 3-run sample simply wasn't large enough to show that: two of its three fired runs happened to land on a wrong second name, which looked like a retrieval/synthesis problem until a 6-run fired sample showed 5 of 6 fully correct.