← Part 3: Reliable Remote Access  ·  Part 4: Always-On Server  ·  AI Lab Portal Home

This page is the readable view of the lab's master test log. Tests 1–13 cover getting a phone to reach the AI Lab from anywhere — not just over the home Wi-Fi — and that story is told in Part 2 and Part 3. Tests 14 onward chase down a different problem: the connection would quietly stop working after the computer sat untouched for a few hours, and Part 4 walks through finding out why and fixing it. Prefer the raw spreadsheet? Download the working log (.xlsx).

Test 23 (RAG-path timing bench) has its own dedicated write-up: Test 23 — RAG-Path Timing Bench. Tests 28–31 (web search grounding for DeepSeek-R1) are told together as one story: Silent Failure to Grounded Answers — Web Search on DeepSeek-R1.

How to read this log

Every test has a number, a client device, and a validity ruling. Tests marked Invalid were run from the HP laptop while its Tailscale client was silently stuck in NoState — they measured nothing and are kept only for honesty. The standard test prompt is "Answer in one sentence only: why do cats purr?"; timing comparisons use that prompt unless noted.

Summary — All Tests #

Every row below is individually linkable — right-click a test number for its direct URL.

TestClientResultNote
1HPFailed — InvalidHP Tailscale later found in NoState
2HPFailed — InvalidSame NoState invalidation
3iPhoneStoppedNo response after 4 minutes
4iPhoneFailedPage could not be loaded (DuckDuckGo, IP)
5iPhoneFailedPage could not be loaded (DuckDuckGo, MagicDNS)
6Lab PC 2HealthyLocal stack status check after waking PC
7HPFailed — InvalidNoState invalidation
8HPFailed — InvalidNoState invalidation
9iPhoneSuccessFirst clean remote query: 63 s / 23 s thinking
10iPhoneSuccessImmediate repeat: 71 s / 31 s thinking
11iPhoneSuccessOne-sentence prompt: 40 s / 13 s thinking
12iPhoneSuccess — VerifyPossible duplicate of Test 11 in source log
13Lab PC 2SuccessLocalhost comparison: 44 s / 24 s thinking
14iPhoneFailed~4 h idle → node offline; the failure that drove Part 3/4
15iPhoneSuccessLocal wake instantly restored access; 33 s / 10 s
16iPhoneSuccess6 h 02 m untouched idle survived after power fix
16biPhoneSuccessWarm second query, 55 s / 40 s (heavier prompt)
19iPhoneSuccessMagicDNS short name works; 121 s cold / 31 s thinking
19biPhoneSuccessWarm repeat 44 s — cold/warm gap ≈ 77 s
17aiPhoneFailed → SuccessNight 2: connectivity held; cold-load timeout, retry OK
17bHPSuccessHP recovered from NoState; MagicDNS works from desktop
19ciPhoneSuccessPhone-only stopwatch protocol: 104 s / 52 s thinking
17iPhonePending12+ hour untouched idle endurance
18iPhonePendingReboot resilience, Tailscale unattended mode
20iPhoneSuccessCold model load measured at 4.5 s, not the assumed 77-90 s — full detail on Part 5
21HPMeasuredDisabling Title/Tags/Follow-Up collapsed chat path ~58 s → ~13 s median — full detail on Part 5
22Lab PC 25/5 PASSRAG known-answer baseline verified end-to-end (Open WebUI → nomic-embed-text → qwen3:8b) — see the .xlsx for full observations
23Lab PC 230/30 COMPLETERAG-path timing bench — ~9.4 s overhead traced to Open WebUI's own pipeline — full write-up on its own page
24Lab PC 2 + HPBug foundOpen WebUI's displayed thinking-timer confirmed broken (upstream bug); real latency 20-49s, matched by Ollama's own timing
25Lab PC 2MeasuredThinking-off and output-length levers quantified against Ollama directly — up to ~53x faster combined, token rate steady ~13-15 tok/s throughout — see also the CPU/Memory profile
28Lab PC 2Bug foundDeepSeek-R1 (7B) web search silently failing — two stacked bugs found — full write-up on its own page
29Lab PC 2PASSWeb search fixed for DeepSeek-R1 (7B) — Legacy FC + Brave engine — grounded, cited, correct when it fires; firing rate measured at 57% across Tests 31–32 — full write-up
30Lab PC 2Compound A+B+CNew failure mode: grounded-but-wrong answers — 3-run diagnostic, corrected 2026-07-28 — full write-up
31Lab PC 26/10 fire rateQuery-generation coin-flip is the dominant lever — 83% correct once search fires — full write-up

Test Detail — 14 Onward #

Tests 1–13 are covered in narrative in Parts 2 and 3. Details below begin where the extended-idle investigation starts.

Test 14 — Remote access after ~4 hours idle

Client / targetiPhone Chrome → 100.115.162.13:8080
ResultFailed (timeout in two browsers); ai-lab-b gray/offline in Tailscale
DiagnosisModern Standby (S0 Low Power Idle) — SleepStudy later showed the machine entered standby a median of 17 seconds after display sleep, with zero network wakes across 77 sessions

Test 15 — Local wake restores access

Client / targetiPhone Chrome → 100.115.162.13:8080
ResultSuccess — 33 s total, 10 s thinking
ObservationMoving the monitor/USB switch back to the Beelink restored Tailscale and Open WebUI instantly, with no service restarts — proof the stack never crashed; only reachability was lost

Test 16 — First overnight after the power fix

Client / targetiPhone browser, cellular → 100.115.162.13:8080
Idle window6 h 02 m untouched (11:30 PM Jul 7 – 5:32 AM Jul 8)
ResultSuccess — first query answered immediately (18 s thinking); warm follow-up (16b) at 7:43 AM: 55 s / 40 s on a heavier prompt
Fix under testpowercfg /change standby-timeout-ac 0 + monitor-timeout-ac 10, applied July 7

Test 19 / 19b — Name-based access and the cold/warm gap

Client / targetiPhone browser, cellular → http://ai-lab-b:8080 (MagicDNS short name)
ResultSuccess — bare hostname resolved; cold run 121 s (31 s thinking), warm repeat 44 s (17 s thinking)
FindingThe ~77 s difference is Ollama reloading qwen3:8b from SSD after its default 5-minute keep-alive eviction

Test 17a — Second overnight, and a new failure mode

Client / targetiPhone browser, Wi-Fi → ai-lab-b:8080
Idle window~6.5 h untouched (midnight – 6:25 AM Jul 9)
ResultConnectivity: success — page loaded, node online, history rendered. First response: "Load failed" — a cold-load timeout, not a connectivity failure. Retry at 6:40 succeeded; follow-up ran with 11 s thinking
SignificanceThe Test 14 failure mode did not recur on night 2; the new failure is a latency problem with a chosen mitigation (OLLAMA_KEEP_ALIVE=2h, to be applied after Tests 17–18)

Test 17b — The HP redemption

Client / targetHP desktop, Edge → ai-lab-b:8080
ResultSuccess — 26 s thinking
SignificanceThe machine whose NoState Tailscale client invalidated Tests 1–2 and 7–8 is now healthy and resolving MagicDNS — name-based access confirmed on a second client platform

Test 19c — Phone-only protocol

Client / targetiPhone browser, cellular → ai-lab-b:8080, timed by stopwatch
ResultSuccess — 104 s total, 52 s thinking on an open-ended prompt
FindingCold-ish start after a 47-minute gap cost ~52 s of overhead vs. ~90 s the day before — reload cost varies; call it 1–1.5 minutes

Tests 17 and 18 — Pending

Test 1712+ hour untouched idle endurance — two ~6-hour nights already banked; a deliberate full-day window remains
Test 18Reboot resilience: enable Tailscale unattended mode, reboot, do not log in, test from the phone
Tests 17 and 18: Pending

Test 24 — Open WebUI's displayed thinking-timer is wrong

Client / targetBrowser on Lab PC 2, then browser on the HP desktop over the network → ai-lab-b:8080, plain query, no RAG
ResultOpen WebUI's on-screen "Thought for Xs" timer showed <1s on every round, 10+ rounds across both machines. Real latency, confirmed two independent ways (stopwatch and Ollama's own total_duration captured via a proxy), agreed within 1–3s every round: 20–49s per round
Root causeConfirmed upstream Open WebUI backend bug — not the frontend, not RAG, not the lab's proxy, network, or infrastructure. A persisted chat record showed the broken displayed value (duration: 0) and Ollama's real timing (total_duration = 21.80 s) side by side in the same message, proving the two timestamps used to compute the display don't bracket the real thinking phase. Mechanism is a working theory, not proven — see Part 5 for where this investigation started
Where the seconds actually goNot hidden or mysterious once measured: qwen3:8b generating at ~10–12 tok/s accounts for the real 20–49s directly. Not RAG, not the network, not a proxy tax
Test 24 Result: Displayed-timer bug confirmed and root-caused — real latency matches Ollama's own reported timing throughout; not a lab infrastructure issue

Test 25 — Quantifying the thinking-off and output-length levers

Client / targetLab PC 2, querying Ollama directly (:11434) — no Open WebUI, no proxy, so Test 24's timer bug can't contaminate the numbers
Design2×2: qwen3:8b thinking on/off × prompt unconstrained/one-sentence-constrained. 10 scored trials per condition, median reported
ResultMedian of 10 trials per condition:
A — think ON, unconstrained: 89.29 s total, 1206 tok, 13.4 tok/s
B — think OFF, unconstrained: 34.80 s total, 476 tok, 13.8 tok/s
C — think ON, constrained: 21.54 s total, 301 tok, 14.1 tok/s
D — think OFF, constrained: 1.69 s total, 19 tok, 15.1 tok/s
FindingTurning thinking off cuts time 61–92% depending on condition — bigger than the ~halving guess that motivated the test. The visible answer stays about the same length either way (~2300 vs ~2110 characters), so the extra tokens with thinking on are essentially all hidden reasoning, not a longer real answer. Token rate held in a tight 12.8–15.1 tok/s band across all four conditions, confirming it's token count driving time, not generation speed. Combined levers (A→D): ~53x faster
ScopeQuantify-only, per the decision on the open questions in the test spec — no persistent thinking-off configuration change made or proposed as a follow-up
Test 25 Result: Both hypotheses confirmed — disabling thinking and constraining output length combine for up to ~53x faster responses, with token throughput steady throughout

What was actually happening on the hardware during these runs — CPU, memory, and how they line up with the steady tok/s finding: Test 25 — CPU & Memory Profile.

Conclusions After Tests 1–13 #

Tests 1–13 established that the AI Lab stack can provide both local and remote inference, while revealing an unresolved intermittent remote-access reliability issue. Open WebUI listens on all interfaces (0.0.0.0:8080), Ollama loads and runs qwen3:8b, and the complete iPhone → Tailscale → Open WebUI → Ollama path completed multiple queries. Successful iPhone tests ran 40–71 seconds against 44 seconds locally — Tailscale is not the bottleneck; most delay is model inference on CPU.

Remote access was temporarily unavailable during Tests 3–5 despite the local stack later being found operational. Tests 1, 2, 7, and 8 are invalid for evaluating Lab PC 2 because the HP laptop's Tailscale client was later found in NoState. The open question — whether remote access survives extended S0 Low Power Idle — became Test 14.

Conclusions After Tests 14–19 #

Test 14's open question is answered. The SleepStudy baseline confirmed the root cause: the Beelink enters Modern Standby a median of 17 seconds after display sleep, and across 77 standby sessions (122.6 hours) it was never once woken by network traffic — remote requests were never wake candidates at the OS level.

The fix — standby-timeout-ac 0 with monitor-timeout-ac 10, applied July 7 — removed the standby transition while letting the display sleep. Remote access then succeeded after 6 h 02 m untouched idle (Test 16) and again after ~6.5 h a second night (Test 17a): the exact scenario that failed in Test 14. MagicDNS name-based access is confirmed from iPhone-on-cellular and from the recovered HP desktop (Tests 19, 17b).

New secondary finding: with Ollama's default 5-minute keep-alive, the first query after a long idle can exceed the request timeout while the model reloads from SSD ("Load failed," Test 17a) — a latency failure, not a connectivity failure; retry succeeds. Measured reload cost ≈ 1–1.5 minutes (121 s cold vs. 44 s warm). Planned mitigation: OLLAMA_KEEP_ALIVE=2h, after Tests 17 and 18 complete.

What's Next

Part 4: Always-On Server

The full narrative behind Tests 14–19 — diagnosis, the Modern Standby fix, and the tests that proved it.

Read →

Test 23: RAG-Path Timing Bench

Where the RAG wall-clock gap actually lives — a mid-run design-flaw catch, and the confound honestly flagged along the way.

Read →

Test 25: CPU & Memory Profile

What the resource graphs show while qwen3:8b is generating, and what they say about the ~13–15 tok/s ceiling.

Read →

Tests 28–31: Web Search Grounding

A silent failure, two stacked bugs, a fix that worked — and a subtler failure mode that turned out to be a coin-flip, not a bug.

Read →

← Part 3  ·  Part 4  ·  AI Lab Portal Home  ·  Download the .xlsx

Test Log: Live document — Tests 17 and 18 pending