← Test Log  ·  AI Lab Portal Home

Partially resolved. Four tests, one investigation: DeepSeek-R1 (7B)'s web search went from silently broken, to a real fix, to a subtler failure mode, to a measured explanation for that failure mode — a coin-flip on whether search fires at all, not a retrieval or reasoning defect. When search fires, the pipeline works (91% fully correct, pooled across Tests 31–32). It only fires on about 57% of attempts, with nothing in the interface indicating which case you got.

Background #

DeepSeek-R1 (7B) was added to the lab as a second model alongside qwen3:8b, and web search was enabled for it the same way as everything else on this instance — a master toggle, a search backend, a per-chat switch. Four manual test runs with an identical post-cutoff prompt (a real event the model's training data couldn't know about) all came back as confident, detailed fabrications: no error, no search status line, no citations. That symptom — a toggle that's accepted but silently does nothing — is what kicked off this investigation.

Test 28 — The Silent Failure #

Prompt"Tell me about the state of the investigation into the seattle shooting near the space needle where 3 people were killed" — a real event (Bite of Seattle festival, Seattle Center, 3 killed / 4 wounded) that postdates the model's training cutoff, so any answer not describing it with citations means search didn't ground the response
MethodReproduced live in the browser (not a scripted API call) with Open WebUI's backend log open, then cross-referenced the actual open_webui package source to trace the code path
ResultFabricated a fictional 2017 "Space Needle shooting" with an invented suspect name, zero citations, zero search status indicator — and zero web-search-related log lines anywhere in the backend log for the request. No exception. Nothing to catch. Nothing was ever called.

Root cause #1 — the actual silent-failure bug. DeepSeek-R1 (7B)'s Function Calling parameter was unset, which resolves to Open WebUI's default, Native. In the backend's middleware.py:

if metadata.get('params', {}).get('function_calling') == 'legacy':
    form_data = await chat_web_search_handler(...)

Open WebUI's forced, always-on search path only runs when a model is set to Legacy. In Native mode, search is instead exposed to the model as a tool it has to choose to call itself — and DeepSeek-R1 via Ollama never emits that tool call. The search code simply never executes. Nothing fails, because nothing runs.

Root cause #2 — found while verifying #1. Switching the model to Legacy forced the search path to fire — and it failed too, but visibly this time: a real "An error occurred while searching the web" status line, and a backend traceback showing the search library posted to DuckDuckGo, got back an HTTP 202 Accepted, and then raised DDGSException: "No results found." anyway — consistent with DuckDuckGo soft-blocking the request rather than a hard network failure.

Test 28 Result: two stacked, independent bugs identified with direct source-code and backend-log evidence — a function-calling-mode gate that skips search entirely for this model, and underneath it, a DuckDuckGo backend that wasn't returning usable results even when reached. Diagnostic only; both fixes reported, neither applied — model reverted to its original state.

Test 29 — The Fix #

Both of Test 28's fixes were applied together: DeepSeek-R1 (7B)'s Function Calling set to Legacy (opening the forced-search path), and the instance's web search engine switched from DDGS to Brave (replacing the backend that was failing underneath it). The Brave API key was entered directly by the lab's human operator — never handled by either Claude Code instance, the same credential boundary used for SSH key setup elsewhere in this lab.

Test runIdentical Test 28 prompt, new chat, Web Search toggle on
Result"Searching the web" → "Searched 4 sites" → "Retrieved 2 sources" → "Thought for 26 seconds" → a grounded answer citing bbc.com and apnews.com, correctly describing the real event: 3 killed, 4 wounded, one suspect arrested, second sought
Backend logConfirms the full pipeline ran end to end — search → fetch → chunk → embed (98 items across 4 sites) → retrieve — not just a UI status line with nothing behind it. Zero exceptions.
LatencyGeneration itself (26s) landed inside the Test 28 weights-only baseline range (26/35/20/21s) — search didn't slow down generation. Total wall-clock is longer once search + fetch + embedding (~45–50s, happening before generation starts) is counted — the expected cost of a correct answer instead of a fast wrong one.
Test 29 Result: PASS. A complete reversal from all five of Test 28's runs, which were 100% confident fabrications with zero citations. Both config changes (Legacy FC on this one model only, Brave as the instance-wide search engine) were left in place as the permanent fix, not reverted. Caveat added after Tests 31–32: this result is one successful run, not a reliability measurement — see Test 31 below for how often search actually fires.

Lesson

"The toggle is on and nothing errors" is not the same as "it's working." The most dangerous failure mode here wasn't that search broke — it's that it broke in a way that produced confident, well-formatted, citation-free prose indistinguishable in tone from a correct answer. Diagnosing it required reading the actual backend source, not just watching the UI.

Test 30 — A New Wrinkle #

Search now worked — but a different, subtler failure showed up almost immediately. Asked a two-part factual question, DeepSeek-R1 retrieved real sources, cited them, and still got half the answer wrong.

Prompt"Which two teams made it to the 2026 FIFA World Cup Finals?" — a countable, objectively-scorable question (ground truth: Spain and Argentina)
Motivating example"Retrieved 2 sources," thought 12s, answered "Spain" only — grounded, but incomplete. Categorically different from Test 28's total fabrications.

Three candidate layers could explain this: A — the model's generated search query was too vague; B — the right query ran, but retrieval/chunk-selection missed the answer; C — the answer was retrieved but the model failed to use it. Three runs, each with the full chat record pulled directly from the API to get the exact generated queries and the exact chunk text actually injected into context — not inferred from the UI.

RunQuery qualityCorrect Wikipedia page fetched?AnswerScore
1Weak — never mentioned "final"No — the two most authoritative FIFA pages failed to fetch, silently skippedFrance & Argentina1/2
2Good — explicitly said "Final"Yes — but the top-k chunks selected were boilerplate navbox content, not the lead sectionFrance & Argentina1/2
3Weak — misread "Finals" as "qualifications"Yes — but only "Argentina" made it into any retrieved chunk; "Spain" never didMexico & Colombia0/2

Layer attribution, measured: A (query formulation) was contributory but not sufficient alone — Run 2 had a genuinely good query and still failed. B (retrieval/chunk-selection) showed up in all three runs, in different forms — an outright fetch failure, or the right page fetched but the wrong section selected within it. C (synthesis) was a real, contributing defect in every run — the model never hedged when its grounding was incomplete, instead answering with confident, specific, wrong guesses. Run 1 went further and blended in outside knowledge (a correct venue detail that wasn't in any retrieved text) on top of a wrong guess.

Test 30 Result: a compound failure across all three candidate layers, not attributable to one — B (retrieval/chunk-selection quality) was the most consistently present defect. Diagnostic only; nothing was changed.

A Correction, Made in the Open #

A follow-up review (Test 31, below) caught two errors in Test 30's original write-up, and both were corrected directly in the log entry rather than silently — the mistakes and the fixes are both part of the record:

  1. A scoring error. Runs 1 and 2 both answered "France and Argentina." Since ground truth is Spain and Argentina, Argentina is a correct name — those runs should have scored 1/2, not the 0/2 originally recorded. The mistake: scoring had drifted into comparing against what earlier runs answered instead of re-checking against ground truth each time.
  2. An unsupported claim. The original write-up called Run 3 "the cleanest isolated proof that synthesis alone is sufficient" to cause failure — reasoning that both correct names were present in context and ignored. A direct string search against Run 3's actual retrieved text found that "Spain" was never present at all — only "Argentina" was. So retrieval had failed there too, not just synthesis. The claim was retracted.

Lesson

Score against ground truth, every time, not against what a previous run said — and verify a strong claim ("synthesis alone is sufficient") against the actual underlying data before writing it down, not just the pattern it looks like from the surface.

Test 31 — The Real Number #

Two back-to-back runs of the same prompt, same config, five minutes apart, gave opposite outcomes — one had search never fire at all, the other fired and answered correctly. That meant query generation itself was nondeterministic, and Test 30's three runs were too few to separate real signal from ordinary variance. This test didn't chase a mechanism — it produced a number.

Prompt"Which two national teams played against each other in the 2026 FIFA World Cup final match on July 19, 2026?" — run verbatim, 10 times, each in a fresh chat
Query generation rate6 of 10 runs generated a search query at all. The other 4 answered from weights alone.
Score when search fired (6 runs)2/2 × 5, 1/2 × 1, 0/2 × 0 — every single fired run scored at least half-correct, and 83% scored fully correct
Score when search did NOT fire (4 runs)0/2 × 4 — 100% failure, each wrong in a different way (a stale "as of 2023" self-anchor, a wrong host country, two different wrong team pairs)

This reframes everything Test 30 found. The dominant lever isn't retrieval quality or synthesis quality — it's the roughly 60/40 coin-flip on whether DeepSeek-R1 decides to search at all. Once it does, it's right most of the time. Test 30's 3-run sample simply wasn't large enough to show that: two of its three fired runs happened to land on a wrong second name, which looked like a retrieval/synthesis problem until a 6-run fired sample showed 5 of 6 fully correct.

Test 31 Result: query-generation rate measured at 6/10. Conditional on firing, DeepSeek-R1's web search is accurate (83% fully correct, 100% at-least-partial). Conditional on not firing, it's uniformly wrong. The bottleneck is whether search fires, not what happens once it does.

Conclusions #

  1. The original silent failure had two independent causes: a function-calling-mode gate that skipped search entirely for a model that doesn't reliably self-invoke tools, and underneath it, a search backend that wasn't returning usable results even when reached. Both were fixed together (Legacy FC + Brave) and verified with a complete reversal from 100% fabrication to a correctly grounded, cited answer.
  2. A second, subtler failure mode surfaced immediately after the fix: search firing and citing sources is not the same as the answer being right. A three-run diagnostic found contributing defects at every candidate layer (query formulation, retrieval/chunk-selection, and synthesis), with retrieval/chunk-selection the most consistently present.
  3. A larger, cleaner measurement (N=10, not N=3) reframed that finding: the dominant lever is whether a search query gets generated at all, not what happens once it does. Once search fires, DeepSeek-R1 answers correctly the large majority of the time.
  4. Two real mistakes in the Test 30 write-up were caught and corrected in the open, not quietly fixed — a scoring error and an unsupported claim that a later, direct check disproved.
  5. A countable, objectively-scorable prompt ("name two X") proved to be a meaningfully better grounding probe than an open narrative prompt — it exposed a failure mode (confident, specific, wrong answers) that a narrative prompt could hedge around invisibly. Recommended as the default probe for future grounding tests.

← Test Log  ·  AI Lab Portal Home