← Part 3: Reliable Remote Access · Part 4: Always-On Server · AI Lab Portal Home
This page is the readable view of the lab's master test log. Tests 1–13 cover getting a phone to reach the AI Lab from anywhere — not just over the home Wi-Fi — and that story is told in Part 2 and Part 3. Tests 14 onward chase down a different problem: the connection would quietly stop working after the computer sat untouched for a few hours, and Part 4 walks through finding out why and fixing it. Prefer the raw spreadsheet? Download the working log (.xlsx).
Test 23 (RAG-path timing bench) has its own dedicated write-up: Test 23 — RAG-Path Timing Bench. Tests 28–31 (web search grounding for DeepSeek-R1) are told together as one story: Silent Failure to Grounded Answers — Web Search on DeepSeek-R1.
Every test has a number, a client device, and a validity ruling. Tests marked Invalid were run from the HP laptop while its Tailscale client was silently stuck in NoState — they measured nothing and are kept only for honesty. The standard test prompt is "Answer in one sentence only: why do cats purr?"; timing comparisons use that prompt unless noted.
Every row below is individually linkable — right-click a test number for its direct URL.
| Test | Client | Result | Note |
|---|---|---|---|
| 1 | HP | Failed — Invalid | HP Tailscale later found in NoState |
| 2 | HP | Failed — Invalid | Same NoState invalidation |
| 3 | iPhone | Stopped | No response after 4 minutes |
| 4 | iPhone | Failed | Page could not be loaded (DuckDuckGo, IP) |
| 5 | iPhone | Failed | Page could not be loaded (DuckDuckGo, MagicDNS) |
| 6 | Lab PC 2 | Healthy | Local stack status check after waking PC |
| 7 | HP | Failed — Invalid | NoState invalidation |
| 8 | HP | Failed — Invalid | NoState invalidation |
| 9 | iPhone | Success | First clean remote query: 63 s / 23 s thinking |
| 10 | iPhone | Success | Immediate repeat: 71 s / 31 s thinking |
| 11 | iPhone | Success | One-sentence prompt: 40 s / 13 s thinking |
| 12 | iPhone | Success — Verify | Possible duplicate of Test 11 in source log |
| 13 | Lab PC 2 | Success | Localhost comparison: 44 s / 24 s thinking |
| 14 | iPhone | Failed | ~4 h idle → node offline; the failure that drove Part 3/4 |
| 15 | iPhone | Success | Local wake instantly restored access; 33 s / 10 s |
| 16 | iPhone | Success | 6 h 02 m untouched idle survived after power fix |
| 16b | iPhone | Success | Warm second query, 55 s / 40 s (heavier prompt) |
| 19 | iPhone | Success | MagicDNS short name works; 121 s cold / 31 s thinking |
| 19b | iPhone | Success | Warm repeat 44 s — cold/warm gap ≈ 77 s |
| 17a | iPhone | Failed → Success | Night 2: connectivity held; cold-load timeout, retry OK |
| 17b | HP | Success | HP recovered from NoState; MagicDNS works from desktop |
| 19c | iPhone | Success | Phone-only stopwatch protocol: 104 s / 52 s thinking |
| 17 | iPhone | Pending | 12+ hour untouched idle endurance |
| 18 | iPhone | Pending | Reboot resilience, Tailscale unattended mode |
| 20 | iPhone | Success | Cold model load measured at 4.5 s, not the assumed 77-90 s — full detail on Part 5 |
| 21 | HP | Measured | Disabling Title/Tags/Follow-Up collapsed chat path ~58 s → ~13 s median — full detail on Part 5 |
| 22 | Lab PC 2 | 5/5 PASS | RAG known-answer baseline verified end-to-end (Open WebUI → nomic-embed-text → qwen3:8b) — see the .xlsx for full observations |
| 23 | Lab PC 2 | 30/30 COMPLETE | RAG-path timing bench — ~9.4 s overhead traced to Open WebUI's own pipeline — full write-up on its own page |
| 24 | Lab PC 2 + HP | Bug found | Open WebUI's displayed thinking-timer confirmed broken (upstream bug); real latency 20-49s, matched by Ollama's own timing |
| 25 | Lab PC 2 | Measured | Thinking-off and output-length levers quantified against Ollama directly — up to ~53x faster combined, token rate steady ~13-15 tok/s throughout — see also the CPU/Memory profile |
| 28 | Lab PC 2 | Bug found | DeepSeek-R1 (7B) web search silently failing — two stacked bugs found — full write-up on its own page |
| 29 | Lab PC 2 | PASS | Web search fixed for DeepSeek-R1 (7B) — Legacy FC + Brave engine — grounded, cited, correct when it fires; firing rate measured at 57% across Tests 31–32 — full write-up |
| 30 | Lab PC 2 | Compound A+B+C | New failure mode: grounded-but-wrong answers — 3-run diagnostic, corrected 2026-07-28 — full write-up |
| 31 | Lab PC 2 | 6/10 fire rate | Query-generation coin-flip is the dominant lever — 83% correct once search fires — full write-up |
Tests 1–13 are covered in narrative in Parts 2 and 3. Details below begin where the extended-idle investigation starts.
| Client / target | iPhone Chrome → 100.115.162.13:8080 |
|---|---|
| Result | Failed (timeout in two browsers); ai-lab-b gray/offline in Tailscale |
| Diagnosis | Modern Standby (S0 Low Power Idle) — SleepStudy later showed the machine entered standby a median of 17 seconds after display sleep, with zero network wakes across 77 sessions |
| Client / target | iPhone Chrome → 100.115.162.13:8080 |
|---|---|
| Result | Success — 33 s total, 10 s thinking |
| Observation | Moving the monitor/USB switch back to the Beelink restored Tailscale and Open WebUI instantly, with no service restarts — proof the stack never crashed; only reachability was lost |
| Client / target | iPhone browser, cellular → 100.115.162.13:8080 |
|---|---|
| Idle window | 6 h 02 m untouched (11:30 PM Jul 7 – 5:32 AM Jul 8) |
| Result | Success — first query answered immediately (18 s thinking); warm follow-up (16b) at 7:43 AM: 55 s / 40 s on a heavier prompt |
| Fix under test | powercfg /change standby-timeout-ac 0 + monitor-timeout-ac 10, applied July 7 |
| Client / target | iPhone browser, cellular → http://ai-lab-b:8080 (MagicDNS short name) |
|---|---|
| Result | Success — bare hostname resolved; cold run 121 s (31 s thinking), warm repeat 44 s (17 s thinking) |
| Finding | The ~77 s difference is Ollama reloading qwen3:8b from SSD after its default 5-minute keep-alive eviction |
| Client / target | iPhone browser, Wi-Fi → ai-lab-b:8080 |
|---|---|
| Idle window | ~6.5 h untouched (midnight – 6:25 AM Jul 9) |
| Result | Connectivity: success — page loaded, node online, history rendered. First response: "Load failed" — a cold-load timeout, not a connectivity failure. Retry at 6:40 succeeded; follow-up ran with 11 s thinking |
| Significance | The Test 14 failure mode did not recur on night 2; the new failure is a latency problem with a chosen mitigation (OLLAMA_KEEP_ALIVE=2h, to be applied after Tests 17–18) |
| Client / target | HP desktop, Edge → ai-lab-b:8080 |
|---|---|
| Result | Success — 26 s thinking |
| Significance | The machine whose NoState Tailscale client invalidated Tests 1–2 and 7–8 is now healthy and resolving MagicDNS — name-based access confirmed on a second client platform |
| Client / target | iPhone browser, cellular → ai-lab-b:8080, timed by stopwatch |
|---|---|
| Result | Success — 104 s total, 52 s thinking on an open-ended prompt |
| Finding | Cold-ish start after a 47-minute gap cost ~52 s of overhead vs. ~90 s the day before — reload cost varies; call it 1–1.5 minutes |
| Test 17 | 12+ hour untouched idle endurance — two ~6-hour nights already banked; a deliberate full-day window remains |
|---|---|
| Test 18 | Reboot resilience: enable Tailscale unattended mode, reboot, do not log in, test from the phone |
| Client / target | Browser on Lab PC 2, then browser on the HP desktop over the network → ai-lab-b:8080, plain query, no RAG |
|---|---|
| Result | Open WebUI's on-screen "Thought for Xs" timer showed <1s on every round, 10+ rounds across both machines. Real latency, confirmed two independent ways (stopwatch and Ollama's own total_duration captured via a proxy), agreed within 1–3s every round: 20–49s per round |
| Root cause | Confirmed upstream Open WebUI backend bug — not the frontend, not RAG, not the lab's proxy, network, or infrastructure. A persisted chat record showed the broken displayed value (duration: 0) and Ollama's real timing (total_duration = 21.80 s) side by side in the same message, proving the two timestamps used to compute the display don't bracket the real thinking phase. Mechanism is a working theory, not proven — see Part 5 for where this investigation started |
| Where the seconds actually go | Not hidden or mysterious once measured: qwen3:8b generating at ~10–12 tok/s accounts for the real 20–49s directly. Not RAG, not the network, not a proxy tax |
| Client / target | Lab PC 2, querying Ollama directly (:11434) — no Open WebUI, no proxy, so Test 24's timer bug can't contaminate the numbers |
|---|---|
| Design | 2×2: qwen3:8b thinking on/off × prompt unconstrained/one-sentence-constrained. 10 scored trials per condition, median reported |
| Result | Median of 10 trials per condition: A — think ON, unconstrained: 89.29 s total, 1206 tok, 13.4 tok/s B — think OFF, unconstrained: 34.80 s total, 476 tok, 13.8 tok/s C — think ON, constrained: 21.54 s total, 301 tok, 14.1 tok/s D — think OFF, constrained: 1.69 s total, 19 tok, 15.1 tok/s |
| Finding | Turning thinking off cuts time 61–92% depending on condition — bigger than the ~halving guess that motivated the test. The visible answer stays about the same length either way (~2300 vs ~2110 characters), so the extra tokens with thinking on are essentially all hidden reasoning, not a longer real answer. Token rate held in a tight 12.8–15.1 tok/s band across all four conditions, confirming it's token count driving time, not generation speed. Combined levers (A→D): ~53x faster |
| Scope | Quantify-only, per the decision on the open questions in the test spec — no persistent thinking-off configuration change made or proposed as a follow-up |
What was actually happening on the hardware during these runs — CPU, memory, and how they line up with the steady tok/s finding: Test 25 — CPU & Memory Profile.
Tests 1–13 established that the AI Lab stack can provide both local and remote inference, while revealing an unresolved intermittent remote-access reliability issue. Open WebUI listens on all interfaces (0.0.0.0:8080), Ollama loads and runs qwen3:8b, and the complete iPhone → Tailscale → Open WebUI → Ollama path completed multiple queries. Successful iPhone tests ran 40–71 seconds against 44 seconds locally — Tailscale is not the bottleneck; most delay is model inference on CPU.
Remote access was temporarily unavailable during Tests 3–5 despite the local stack later being found operational. Tests 1, 2, 7, and 8 are invalid for evaluating Lab PC 2 because the HP laptop's Tailscale client was later found in NoState. The open question — whether remote access survives extended S0 Low Power Idle — became Test 14.
Test 14's open question is answered. The SleepStudy baseline confirmed the root cause: the Beelink enters Modern Standby a median of 17 seconds after display sleep, and across 77 standby sessions (122.6 hours) it was never once woken by network traffic — remote requests were never wake candidates at the OS level.
The fix — standby-timeout-ac 0 with monitor-timeout-ac 10, applied July 7 — removed the standby transition while letting the display sleep. Remote access then succeeded after 6 h 02 m untouched idle (Test 16) and again after ~6.5 h a second night (Test 17a): the exact scenario that failed in Test 14. MagicDNS name-based access is confirmed from iPhone-on-cellular and from the recovered HP desktop (Tests 19, 17b).
New secondary finding: with Ollama's default 5-minute keep-alive, the first query after a long idle can exceed the request timeout while the model reloads from SSD ("Load failed," Test 17a) — a latency failure, not a connectivity failure; retry succeeds. Measured reload cost ≈ 1–1.5 minutes (121 s cold vs. 44 s warm). Planned mitigation: OLLAMA_KEEP_ALIVE=2h, after Tests 17 and 18 complete.
The full narrative behind Tests 14–19 — diagnosis, the Modern Standby fix, and the tests that proved it.
Read →Where the RAG wall-clock gap actually lives — a mid-run design-flaw catch, and the confound honestly flagged along the way.
Read →What the resource graphs show while qwen3:8b is generating, and what they say about the ~13–15 tok/s ceiling.
Read →A silent failure, two stacked bugs, a fix that worked — and a subtler failure mode that turned out to be a coin-flip, not a bug.
Read →