Llama Manager

What. A management, debugging, and monitoring platform I run for local LLM inference on an AMD Strix Halo (gfx1151) box. It puts multiple inference engines behind one OpenAI-compatible API — llama.cpp in multi-model router mode with on-demand loading and LRU eviction, plus DeepSeek V4 Flash via antirez/ds4 — and wraps them in a web UI. The dashboard shows 1-second CPU/RAM/GTT/GPU/thermal gauges and rolling five-minute charts for temperature, power draw, memory, and generation speed, with minute-level history kept for up to a year. A models page loads and unloads GGUFs (including multi-part files) on demand; a processes page shows the router and worker PIDs with live CPU/RSS and a kill switch; presets activate a tuned single-model configuration in one click; per-request logs break down latency and status codes; a live server-log stream exposes llama.cpp slot activity; and a full streaming chat UI with token-throughput stats rounds it out. An auto-paging fullscreen dashboard drives a wall-mounted display.

Why. Local inference on a shared, thermally-constrained CPU+iGPU die fails in ways a cloud endpoint never does: OOM, thermal throttling, leaked slots, a model that simply will not load. Every stability guard in here — memory watchdog, thermal governor, restart governor, queue admission, slot reaper — was written from a real incident. The telemetry exists so the next incident is diagnosable instead of mysterious.

How. An OpenAI-compatible proxy on port 5250 resolves request-time default-big / default-small aliases, then routes to a local engine or offloads to a remote OpenAI-compatible backend based on queue depth, thermal state, or policy. Adaptive context scaling and SSD expert-streaming fit an 80 GB DeepSeek V4 Flash model onto the iGPU. KV-cache prefix state persists across model reloads. HuggingFace GGUF search and download, per-model presets, a dedicated embeddings server, an MCP server for agent tooling, and a full streaming chat UI round it out. Runs as a systemd user service.

Stack: Node.js, React, llama.cpp (ROCm), antirez/ds4, distrobox/ROCm 7.2.4, JSONL time-series storage, MCP, systemd.

Links: Repo

Built Feb–Jul 2026 across ~255 commits.