← the failure log (chronological)
Browse by topic
The whole build log, grouped by technology instead of by date — a map for finding the post you need. 45 posts across 8 topics. Filed automatically from each post's tags, so this page updates itself when a new post ships. For chronological order, see the failure log.
LLM serving & inference
Running large language models locally — vLLM, quantisation, context length, throughput.
- I built an AI agent company on my own hardware
Running Paperclip multi-agent on a local Hermes+LiteLLM stack: the overhead tax you don't see coming, the code review that passed a broken health check, and why 'you are a coordinator, never implement' is not a rule. - My uncensored LLM was leaking raw tokenizer output
Dolphin 24B AWQ in vLLM returned Ġ and Ċ BPE artifacts at ~2 tok/s: what a chat-template misconfiguration looks like when you actually inspect the bytes. - One RTX 5090, how many users: the honest answer
vLLM continuous batching stress test on Nemotron NVFP4 — throughput scales to 8 concurrent users, then the KV-cache fills and throughput flatlines at 728 tok/s. - The model echoed my prompt back
Three ways Nemotron's JSON output silently broke my parser: prompt notation echoed as literal keys, json_repair returning a bare string, normalized floats failing strict type checks. The parse-boundary discipline that handles all three. - Killing the two-pass dance: fitting a 30B multimodal LLM and a TTS engine on one RTX 5090
Six vLLM 0.20.0 boot attempts to coexist Nemotron-3-Nano-Omni NVFP4 and Fish Speech 1.5 on 32GB, and why --max-num-batched-tokens 16384→4096 was the fix nobody documents. - A fitness AI that refuses to guess
ADASHINO plans training with a deterministic solver -- the LLM is the conversational shell, not the source of the numbers. No demo video, and a four-week dogfood gate before any claims ship. - I let an LLM agent design quantum circuits in a sandbox for 10 cycles
An LLM proposed VQC variants, each run in a locked no-network sandbox, scored vs a 0.978 baseline. It explored, hit a collapse, recovered, and edged the baseline by +0.0044 — within noise. The honesty and the sandbox are the story. - Sovereignty as routing, not policy
How to enforce 'client data never reaches a cloud API' with LiteLLM routing profiles — including a chain that deliberately errors instead of degrading silently. - Six hours in TensorRT-LLM so you don't have to
I picked NVIDIA's 'official' inference path for a multimodal model and spent six hours hitting walls, ending on one AutoDeploy can't fix. The lesson: 'theoretically compatible' is not 'someone benchmarked this.' - I had my KV-cache math 14× wrong (I treated my Mamba-hybrid like a transformer)
How much KV-cache does a Mamba-2/Transformer hybrid use per token? I priced all 52 layers when only 6 attend — my numbers came out 7–14× too low. - A 32GB GPU is a budget, not a suggestion
How to fit a 30B multimodal LLM at 180K context on one RTX 5090: move the display to iGPU, never co-load models, and pick NVFP4 over FP8.
Creative AI — video & image
ComfyUI, object removal, upscaling and the rest of the local video / image pipeline.
- A self-healing catalogue for a stack that drifts
After 4 of 9 ComfyUI workflows broke from upstream node renames, I built a monthly validation job that detects drift, regenerates the public catalogue, and refuses to auto-fix the renames. - My tested workflows broke from upstream drift
Swept 9 ComfyUI workflows I'd marked tested: 4 broke from upstream node renames and API changes. The exact errors, the rebuild method, and why a workflow JSON is a promise against an environment that keeps moving. - The forensic bridge: measure the scene before diffusion touches it
A vision LLM does a three-pass forensic scan of the footage — heroes, reflections, atmospheric fields — and writes the SAM3 mask prompts and fill constraints itself. Full pipeline in 203 seconds, autonomy proven, and the one target it honestly couldn't remove. - The SAM3 mask that crashed VACE
A segmenter was handing my video editor a single still-frame mask shaped [1,H,W] when the model wanted one mask per frame, [N,H,W] — a tensor-dimension mismatch that looked like everything except what it was. - I benchmarked open-source object removal until it actually worked
Four runs, one dependency rabbit-hole, and the lesson that a single failure frame proves nothing — the clip you test on IS the evaluation.
Audio stack
Speech, voice cloning, transcription and the audio toolchain.
- I built an avatar that introduces itself as an avatar
Fish Speech 1.5 + LatentSync fully local lipsync in five minutes: gross sync correct, close-up artifacts visible. The avatar opens by naming what it is. - 25 ways the audio stack install deviated from its spec
Fish Speech, WhisperX, LatentSync 1.6, Hallo2, and YuE 7B on an RTX 5090 -- the April spec said one thing, May's install said 25 other things. - I benchmarked an AI avatar pipeline end to end
Ten local audio and avatar pipelines on one RTX 5090 — TTS, voice clone, transcription, lip-sync, music, dubbing — measured wall-clock and VRAM for each, including the two pipelines that stayed blocked and the close-ups that still go to cloud.
Networking & security
Firewalls, iptables vs Docker, Tailscale, default-deny and keeping things private.
- Shipping a domain, site, and offsite backup in one evening
Five underdocumented setup blockers: DNS parking records, zone-scoped API tokens, Hetzner Storage Box defaults, restic sftp path format, and the Cloudflare Pages-to-Workers migration. - Driving a Linux AI workstation from the couch
How I built a tailnet-only web command panel for 25 services on an RTX 5090 rig, and why typing localhost:PORT from a remote browser connects to the wrong machine. - My security auditor is 80 percent assembled, not invented
How BHAIRAVA shipped as a real one-person security product: 80% existing measured components, 20% new connective tissue, and Stage V honest failures including 0/12 recall on real Django CVEs. - My code auditor scored 1.00 on fixtures and 0.00 in the wild
13 real Python web apps, 276 findings, zero true positives. The five rule bugs real code exposed, the −90% fix pass that still found nothing, and the LLM oracle that coin-flips on the hard case. - Your firewall isn't protecting your Docker containers
Docker rewrites iptables rules ahead of UFW — every published port is LAN-accessible regardless of your firewall rules. The DOCKER-USER chain fix.
Containers & build / dependencies
Docker, pinned wheels, torch / CUDA build pain and reproducible environments.
- Six footguns installing a local AI agent on Ubuntu 24
Hermes AI agent install on Ubuntu 24.04 via pip hits six traps: systemd PATH, silent aiohttp omission, nvm Node invisibility, dashboard flags, and a .env override that silently downgrades your API key. - A local 3D asset foundry with two license lanes
TRELLIS.2 vs Hunyuan3D on one RTX 5090, QuadRemesher tracing-stop when Python launches it, and why the VLM judge failed a good asset the pixel metrics cleared. - Moving Docker's data root doesn't move containerd
I moved Docker to the big disk and /var kept filling up anyway — because Docker and containerd are two different daemons with two different stores. - Nightly wheels are a depreciating asset
Rebuilding a one-month-old audio-stack Dockerfile failed with xformers ResolutionImpossible on cu130, because the nightly index it pinned had moved on. Pin proven stable, never pip-install at runtime.
Hardware & GPU / VRAM
Fitting models onto one card — VRAM budgeting, the RTX 5090, display offload, model co-residence.
- A GTK switch for one GPU shared by seven AI stacks
GTK4 mode switcher for an RTX 5090 shared across seven local AI stacks: the idle-vs-busy detection problem, an SM-util false positive from reverse-PRIME graphics, and why the app is a front-end only. - How many qubits fit in 32 GB? I simulated until the GPU refused
A hybrid quantum-classical net on MNIST 0/1 lands at 99.5% — just behind logistic regression (99.6%), the honest loss. Plus the memory-wall sweep: the statevector fits to 28 qubits, then the card taps out. - I ran a trained circuit on a real IBM quantum computer — here's the noise
Sim-trained VQC, inferred on ibm_fez: ⟨Z₀⟩ went +0.72 (sim) → +0.49 (noisy hardware) → +0.87 (mitigated). Mitigation cost 3× the shots. 5 s of QPU under a hard 240 s cap. - Superconducting vs trapped-ion: I ran the same circuit on two quantum computers
One trained VQC, two hardware technologies. The trapped-ion raw ⟨Z₀⟩ (+0.60) beat superconducting (+0.49) at getting near the sim (+0.72) — then its cheap error-mitigation misfired and moved the answer to +0.32, further from truth. The higher-fidelity machine, and a mitigation that backfired.
Ops, automation & reliability
Cron, backups, monitoring outcomes, and the failures that taught the lessons.
- One YAML file to rule my estate
How PROJECT-STATUS.yaml became the single source of truth for 22 projects, and the fail-open trap where a stray quote made the dashboard serve convincing lies. - I built a trading intelligence stack that isn't allowed to trade yet
GINKO: nine phases of financial intelligence infrastructure, a hard Live Gate, and zero live orders. No performance numbers published because none have passed the audit. That is the post. - What I designed in May vs what shipped in June
I generated an architecture diagram for my local AI stack. It was gorgeous. Within a month, seven things in it were wrong — and I couldn't diff the PNG to find what changed. - My backup failed silently for 17 days
A cron PATH bug, the morning I found it, and why I now monitor outcomes instead of trusting automation.
Other / uncategorised
Posts that do not yet fit a topic above — tag them to file them.
- Multiplayer without servers: seed the RNG with the date
Seeding a Flutter game's RNG with the UTC date gives every player identical pieces that day. No WebSockets, no state sync, no anti-cheat server needed. - I slept while the LLM shipped five releases
I handed an AI agent four phases of a Flutter app and went to bed. It shipped them. Two honest caveats. - The bug was a zero-size stack
Flutter Stack sizes itself to its largest non-positioned child. A SizedBox.shrink() in a conditional game-over overlay collapsed an entire game view to 0x0, silently, no exception. - Planning a robotics stack before buying a single servo
VISHWAKARMA P0: Isaac Sim quarantined for sm_120 Blackwell bugs, the target drone discontinued, and GR00T weights carrying a non-commercial license the Apache badge did not show. - The license audit that blocked my own translation model
NLLB-200, MMS, SeamlessM4T all CC-BY-NC: the Apache label on the repo does not cover the weights inside it. - A quantum classifier that ties a neural net — and why that's the honest result
A re-uploading VQC hits 97.8% on two-moons, dead even with an MLP and well above logistic regression (85.3%). The framework trains; nobody won. That side-by-side honesty is the whole point. - I built a three-stack AI system without writing code
The method behind building a sovereign LLM stack, a video stack, and an audio stack without hand-writing code: spec-driven, LLM-directed, gate-verified. - I watched a quantum model become untrainable — the barren plateau, measured
Gradient variance of a hardware-efficient circuit collapses 1,436× as it grows (global cost, α≈0.64, r²=0.99) — a textbook barren plateau. A local cost decays 7.1× slower: the known escape, quantified. - The quantum kernel that got worse as I added qubits
A fidelity quantum kernel loses to a classical RBF-SVM on both datasets — and its off-diagonal similarities concentrate 64× as qubits grow, the kernel version of the barren plateau. An honest negative result.