Systemd-tmpfiles vs Plex: When /tmp Aging Breaks EAC3 Transcoding
Why long-running Plex servers suddenly fail EAC3 transcodes with exit code 187, and how a tmpfiles.d exclusion rule fixes it permanently.
guatulabs/dev
Technical walkthroughs, open-source projects, and real-world guides on AI agents, homelab infrastructure, Kubernetes, and IIoT.
Tailscale Kernel TUN in Unprivileged LXC: Direct SSH Without Userspace Networking
Two lines in a Proxmox LXC config give Tailscale a real kernel TUN device: direct peer-to-peer SSH, working UDP, and the container stays unprivileged.
Pushgateway Heartbeat Gotcha: When ndots and NetworkPolicy Silently Eat Your Alerts
How ndots:5, a wildcard DNS record, and a default-deny NetworkPolicy combine to swallow CronJob heartbeats without a single error or alert.
FastMCP Agent Mail: RBAC Tokens vs Anonymous Access, and the 403 Errors in Between
Why a FastMCP agent mail server that works anonymously in dev returns 403 behind TLS ingress, and how to wire bearer tokens without leaking them.
Kyverno allowLatestTag Pitfalls: How Policy Can Stop Pods From Restarting
A disallow-latest-tag policy only blocks new admissions. That gap turns a routine pod eviction into a permanent outage. Here's the failure mode and the fix.
Hybrid Retrieval v2: Qwen Embeddings, BM25, and RRF with a FastEmbed Reranker
Dense vectors miss exact identifiers. Here's the hybrid Qdrant setup I run: Qwen3 embeddings, BM25 sparse vectors, RRF fusion, and a CPU FastEmbed reranker.
Longhorn Read-Only Mounts: Detection, Recovery, and Closing the Silent Failure Window
A Longhorn volume can report Healthy while the filesystem inside your pod has been read-only for hours. How to detect it, recover it, and alert on it.
ForwardAuth with Authentik: Wiring SSO Across Multiple Apps Without Breaking Ingress
Authentik ForwardAuth on Traefik fails in ways that look like success. Outpost bindings, NetworkPolicies, secret shape, and header mapping across apps.
Backing Up Qdrant Snapshots Correctly: From emptyDir to NFS Persistent Backups
Qdrant snapshots on emptyDir vanish on restart, and snapshots on the data PVC die with the disk. Here's the NFS-backed pattern that survives both.
Traefik Host Collision: Why Duplicate Host Rules Break IngressRoutes
In Traefik v3, a wildcard router can outrank an exact Host match, because default priority is rule string length. Here's how it fails and how to fix it.
We Almost Deployed a Temporal Knowledge Graph. The Eval Said No.
A temporal knowledge graph passed every static RAG test and still failed. The gap was temporal reasoning, and one eval caught it before prod did.
I Benchmarked My Homelab Memory Stack: Hybrid Search + Local Reranker Took LoCoMo from 63% to 80%
How adding BM25 hybrid search and a local BGE reranker to a vector-only agent memory stack moved LoCoMo accuracy from 63% to 80%, and what it cost in latency.
The Write Policy Is the Hard Part: Promotion Pipelines for Agent Memory
Storing agent memory is easy. Deciding what earns a permanent write, and keeping the write-path alive through RBAC and network policy, is the real work.
Moving Scheduled LLM Curation from Cloud APIs to Local Models
A comparison of cloud APIs vs local models for scheduled agent memory curation, and the headless-cron gotchas nobody warns you about.
Silent Drift: Why Re-Embedding Only on Count Changes Rots Your Semantic Index
Count-based re-embedding leaves stale vectors pointing at edited text. Here's how semantic drift creeps in and how hash-based validation stops it.
Your Vector DB Snapshots Are Landing on the Same Disk That Will Fail
Qdrant snapshots default to the same volume as your data. One disk failure takes both, and even a surviving snapshot can fail restore with a 400 error.
Eviction Without Deletion: Running an ACT-R Decay Policy for Agent Memory
Stop deleting agent memories to fight vector-store noise. An ACT-R decay policy lowers activation instead, so cold context can re-activate later.
Prometheus Alerting Rules That Don't Cry Wolf
How to write Prometheus alerts that carry context, tolerate transient scrape blips, and page only when something is actually broken.
SealedSecrets: Storing Secrets in Git Without the Risk
How SealedSecrets lets you commit encrypted secrets to Git safely, why the real risk is losing the controller's private key, and how to back it up.
MetalLB on Bare Metal: LoadBalancer Without a Cloud Provider
Your bare-metal LoadBalancer Service is stuck Pending. Here's how MetalLB hands out real LAN IPs, why L2 mode works, and the traps that cost me hours.
OpenTofu State in MinIO: An S3-Compatible Backend for IaC
Store OpenTofu state in self-hosted MinIO: S3 backend config, native locking, least-privilege policies, and guards against a CI apply that destroys infra.
Langfuse for LLM Observability: Tracing Agent Calls Instead of Guessing
Using Langfuse to trace multi-step agent workflows, replace custom eval logic, and consolidate LLM observability into one tool that actually scales.
CloudNativePG: Running PostgreSQL in Kubernetes Without the Pain
Running PostgreSQL on Kubernetes with CloudNativePG: the Kyverno policy traps, SSL gotchas, read/write splitting, and the 16.4 segfault to avoid.
Proxmox Backup Server: Incremental Backups for Your Whole Cluster
Setting up Proxmox Backup Server for incremental, deduplicated backups across a multi-node cluster, including the token auth and locking gotchas.
Network Policies with Calico: Default Deny and Namespace Isolation
Rolling out default-deny NetworkPolicies and namespace isolation with Calico without breaking DNS, ingress, or admission webhooks.
Agent Glass-Break Patterns: Controlled Escalation for Production
How to implement controlled escalation for AI agents using safeBins and network-level constraints to prevent production catastrophes.
Velero + MinIO: Kubernetes Backup Strategy for Bare Metal
Moving beyond cloud S3 to a bare-metal backup strategy using Velero, MinIO, and Longhorn. Lessons on CSI snapshots and ETCD persistence.
When Agents Should Stop: Designing Safety Boundaries That Work
Stop conditions are a feature, not a failure state. Budget caps, progress checks, and exit contracts that make autonomous agents quit cleanly.
Edge Computing for IIoT: When to Process at the Source
Stop sending raw sensor data to the cloud. A deep dive into edge processing, local LLM routing, and the privacy hard-wall for IIoT.
Grafana Dashboards: Information Density vs Readability
Stop cramming every metric into one screen. A practical look at balancing information density and performance in Grafana dashboards.
Kubernetes RBAC: Building Least-Privilege Service Accounts
Moving beyond cluster-admin for everything. A practical approach to scoping ServiceAccount permissions for production workloads and AI agents.
Building Agent Skills: A Pattern for Discoverable Capabilities
Moving beyond basic tool-calling to a composable skill pattern that makes agent capabilities discoverable and reusable across different frameworks.
Tesla P40 in a Homelab: 24GB of Inference on a Budget
Running a Tesla P40 for LLM inference. Why I ditched GPU passthrough for host-level drivers to stop the constant Proxmox crashes.
Condition-Based vs Time-Based Maintenance: Making the Switch
Stop replacing parts that aren't broken. A deep dive into moving from calendar-based schedules to real-time condition-based maintenance in IIoT.
Longhorn Volume Health: The Gap Between 'Healthy' and Actually Working
Stop trusting the Longhorn UI blindly. Learn to monitor replication, fix stale mounts, and manage snapshot bloat in production K8s storage.
Proxmox Cluster Quorum: How Many Nodes Do You Actually Need
Stop guessing about Proxmox quorum. A deep dive into 2-node vs 3-node setups, QDevice, and why your cluster suddenly goes read-only.
Privacy-Routed LLM Inference: Keeping Sensitive Data Out of the Cloud
How to build a routing layer for AI agents that ensures sensitive data stays on local hardware while leveraging cloud LLMs for non-private tasks.
Kyverno Admission Controllers: Policy-as-Code That Actually Works
Moving beyond the happy path of Kubernetes policy enforcement. Real-world Kyverno pitfalls, mutation loops, and the gap between docs and production.
Tailscale Subnet Routers: Accessing Your LAN Without the VPN Headache
Stop installing Tailscale on every single device. Use a subnet router to expose your home network and IoT gear without traditional VPN complexity.
PCIe Passthrough: NIC Name Instability and MAC Pinning
Stop your network configs from breaking after a reboot. How to fix unpredictable NIC naming in Proxmox PCIe passthrough.
Vibration Monitoring Architecture: From Sensor to Dashboard
Building a production-ready vibration monitoring pipeline. Lessons on sampling rates, MQTT bottlenecks, and avoiding dashboard lag.
GPU PCI Address Instability: When Your Card Moves Between Reboots
Dealing with shifting PCI addresses in Proxmox and how to stop your GPU from disappearing or changing IDs after a reboot.
Cognitive Memory for Agents: Vector Search vs Activation-Based Recall
Comparing vector databases and activation-based memory for AI agents. Trade-offs in latency, scale, and interpretability.
AdGuard Home: Network-Wide DNS Filtering with Failover
Setting up AdGuard Home for network-wide DNS filtering with a robust failover strategy to prevent total internet outages.
Unprivileged LXC + Docker: The runc Sysctl Permission Trap
Fixing the silent failure of sysctl settings when running Docker inside unprivileged Proxmox LXC containers.
Three-Layer Safety for Autonomous Agents: Stopping the Infinite Loop
Moving beyond prompt engineering to implement token-level schema enforcement, pre-execution gates, and shell-safe execution pipelines for AI agents.
Stop Merging Broken YAML: Kubernetes Manifest Validation in CI
Don't let invalid manifests break your GitOps pipeline. Learn how to use kubeconform and Kyverno exclusions to catch errors before they hit production.
GPU D3cold Power States: How to Brick Your Card Without Trying
Bricking GPUs with D3cold: Real-world gotchas and fixes for Proxmox users
cert-manager + Cloudflare DNS-01: Automated TLS for Everything
Automating TLS with cert-manager and Cloudflare DNS-01 in Kubernetes
SealedSecrets Key Backup: Don't Lose Your Encryption Keys
How to back up and recover SealedSecrets encryption keys in Kubernetes
Ollama on Kubernetes: Recreate Strategy and Single-GPU Deadlock
Deploying Ollama on Kubernetes can lead to GPU deadlocks. Here's how to avoid them.
MQTT Broker Selection: HiveMQ vs Mosquitto for Industrial Use
Comparing HiveMQ and Mosquitto for industrial IoT: scalability, security, and reliability
Wildcard DNS + ndots:5: The TLS Nightmare and How to Fix It
Kubernetes default DNS settings can cause TLS certificate mismatches when using wildcard DNS. Here is how to debug and fix it.
Self-Improving AI Infrastructure: How Your Homelab Wiki Updates Itself
How to automate your homelab wiki with self-improving AI infrastructure
The 6-Layer Memory Architecture I Run for Claude Code
Open-sourcing the memory system behind my Claude Code setup: CLAUDE.md, path-scoped rules, wiki, vector search, cognitive memory. With the mistakes.
Building Karpathy's LLM Wiki: A Production Homelab Implementation
Implementing Karpathy's LLM Wiki in a homelab with real-world lessons and gotchas
Proxmox API Tokens: Bash History Expansion and the ! Character
Bash history expansion breaks Proxmox API tokens — here's how to fix it
AMD iGPU Stealing Your RAM: UMA Frame Buffer on Headless Servers
AMD iGPU steals RAM on headless servers, here's how to fix it
Agent Credential Management: Two-Tier Service Accounts for Secure AI Agent Workflows
Managing agent credentials with two-tier service accounts: a secure approach for AI agent orchestration
Infrastructure as Code, but Automated: OpenTofu and GitHub Actions
Stop manual applies. Learn how to build a production-ready CI/CD pipeline for your infrastructure using OpenTofu and GitHub Actions.
Equipment Health Scoring: How One Number Made My Operators Stop Checking the Dashboard
Turn raw sensor data into a single, actionable health score operators actually use
Pod Disruption Budgets: Why kubectl drain Gets Stuck on Longhorn
Pod Disruption Budgets can block kubectl drain on Longhorn. Here's how to avoid it.
Attention Residuals: How Kimi Is Rethinking Transformer Depth
Kimi's Attention Residuals replace fixed residual connections with learned layer aggregation. What it means for LLM depth.
Helm fullnameOverride: Naming Sanity in ArgoCD
Avoid naming chaos in ArgoCD by using Helm fullnameOverride effectively
NVIDIA Container Toolkit: Why the Default Runtime Matters
Fixing default runtime misconfigurations in NVIDIA Container Toolkit for GPU workloads
DOCP/XMP: Why Your Proxmox Node Runs at Half RAM Speed
DOCP/XMP enabled but Proxmox still runs at half RAM speed? Check your kernel and BIOS.
AMD Ryzen C-State Freezes: How `processor.max_cstate=1` Saved My Proxmox Node
Ryzen freezes in Proxmox? Learn how to disable deep C-states and stop random system lockups.
Kubernetes Storage on Bare Metal: Longhorn in Practice
How I configured Longhorn storage for a Kubernetes cluster on bare metal — what worked, what didn't, and what I'd do differently next time.
GPU Passthrough on Proxmox: A Field Guide to the Gotchas That Bit Me
The documentation won't warn you about D3cold bricking, PCIe bus renumbering, or why the NVIDIA device plugin silently fails. This is that guide.
Building MCP Servers with FastMCP: Stop Writing Boilerplate, Start Writing Tools
FastMCP makes building Model Context Protocol servers feel like FastAPI. Here's how to go from zero to a working MCP server in under an hour.
GitOps for Homelabs: How ArgoCD App-of-Apps Scales Your Cluster
How the ArgoCD app-of-apps pattern brings real GitOps discipline to homelab Kubernetes — repo structure, examples, and what I'd do differently.
Multi-Agent AI Systems: Architecture Patterns That Actually Work
A practical guide to designing multi-agent AI systems — orchestrator patterns, trust boundaries, and the tradeoffs I learned running agents in production.
Building a Production Homelab: Multi-Node Proxmox Cluster with Kubernetes
How I built a multi-node Proxmox cluster running Kubernetes with GPU passthrough, GitOps, and dozens of services — and what broke along the way.
