← Test Log  ·  AI Lab Portal Home

Test 23 complete — 30/30 trials, 0 errors. Median overhead of ~9.4 s attributable to Open WebUI's own RAG pipeline, not model inference or embedding.

Background #

Test 21 found that Open WebUI's own background tasks — automatic title, tag, and follow-up generation — were the real culprit behind slow queries, and disabling them collapsed the median chat latency from ~58 s to ~13 s.

Test 22 verified the RAG pipeline end to end (5/5 known-answer queries correct, embedding engine switched to Ollama-served nomic-embed-text) but observed wall-clock time exceeding 30 s against a displayed thinking time of only 7–14 s — an unresolved gap that motivated this bench.

Question & Method #

Where does the wall-clock gap actually live — model inference, the embedding call, or Open WebUI itself?

Design: paired trials, same known-answer question with and without the knowledge base attached, N=15 per condition (30 total), alternating A/B to spread any drift, fresh chat per trial.

Instrumentation: a capture proxy sat on Ollama's real port; the real Ollama instance moved to an alternate port behind it. Open WebUI itself was unmodified and unaware of the swap. Ollama's own reported timing fields (load_duration, prompt_eval_duration, eval_duration, total_duration) served as ground truth for model-side time — wall clock minus Ollama's own total is the overhead attributable to something outside Ollama.

Diagram of Test 23's instrumentation. Normal state: Open WebUI on port 8080 talks to Ollama on port 11434, serving qwen3 and nomic-embed. Test 23 run state: Open WebUI is unchanged, talking to capture_proxy on port 11434, which logs timing fields and forwards to Ollama, moved to port 11499.
Test 23 instrumentation: the capture proxy impersonates Ollama on its real port; Open WebUI is unmodified and unaware. Normal state restored after the run.

Controls: Test 21's config held throughout (Title/Follow-Up/Tags generation disabled). A controlled warm-up ran before any scored trial, and a written restore procedure was executed and verified afterward. embed_fired was captured per trial as an empirical check on whether the embedding call actually ran.

The Mid-Run Catch #

The first execution attempt paused during its own warm-up check: ollama ps showed the embedding model had never loaded. The root cause was a design flaw, not a bug in the running bench — the original spec's RAG condition attached the knowledge base by sending the literal text #RAG Test <question>, mirroring how a person types it into Open WebUI's chat box. That syntax is parsed by Open WebUI's own frontend before the request is ever built; sent as plain text over the raw API, it does nothing. The original design would have run 30 trials comparing two effectively identical no-context conditions.

The fix: attach the knowledge base directly via the API's own files field, referencing the knowledge base's ID. This was verified against the normal, un-swapped Ollama instance before resuming — no live-infra risk was taken to find or test the fix. The environment was fully restored between the aborted attempt and the real run, and the spec was revised in place (now v3) to document the finding.

Lesson

The warm-up gate existed to control model state — and instead caught a design flaw before any scored trial ran. Checks that control state catch more than checks that merely record it.

Results #

Metricno_rag (N=15)rag (N=15)
Wall clock — median44.0 s44.9 s
Ollama total_duration — median41.7 s30.9 s
Overhead (wall − Ollama total) — median2.3 s11.7 s
Embedding call fired0/15 (expected)15/15 (expected)
Embedding total_duration — median (when fired)—0.04 s

The gap lives in Open WebUI's own RAG pipeline — retrieval, chunk assembly, and prompt injection — not in model inference and not in the embedding call itself, which is negligible. This closes a meaningful chunk of Test 22's open observation: roughly 30 s of Ollama's own time plus roughly 12 s of Open WebUI overhead approximates the wall clock actually observed there.

Measured but not yet explained: ~9.4 s is a lot for retrieval against a five-fact, single-chunk document. Whether this cost is fixed per query or scales with corpus size is unknown, and deferred to a future test against a harder, more realistic document set.

The Confound #

Ollama's own generation time was shorter for RAG trials (median 30.7 s) than for plain trials (median 41.2 s) — the opposite of what "RAG adds cost" would predict looking at generation alone. This is very likely an artifact of reusing a single question: the no-RAG condition cannot answer it and generates long, hedging text, while the RAG condition retrieves the fact directly and answers concisely with a citation.

Consequence: raw wall-clock is not a clean RAG-vs-plain comparison in this run. The isolated overhead metric above — which factors out generation length entirely — is the reliable number, not total wall clock. A planned follow-up test using a question answerable in both conditions would remove this confound.

Conclusions #

  1. RAG adds a real, measurable overhead of ~9.4 s median, located in Open WebUI's own pipeline — not in model inference and not in the embedding call.
  2. The embedding call itself is negligible (~0.04 s median) — it is not the source of RAG's added latency.
  3. Async Embedding Processing affects document upload-time indexing only; query-time embedding always runs synchronously regardless of the setting, confirmed empirically (embed_fired true on 15/15 RAG trials, false on 15/15 plain trials).
  4. A real confound was identified rather than smoothed over — reusing one question makes raw wall-clock an unreliable RAG-vs-plain comparison here. A follow-up test is planned to remove it.
  5. The written restore procedure was executed and verified twice (once after the aborted first attempt, once after the real run) — no live infrastructure was left in a swapped state.

← Test Log  ·  AI Lab Portal Home