Infra Magician

Two tenants, one GPU, and no wall between them

root@harshit.cloud (Harshit Luthra) — Sat, 25 Jul 2026 00:00:00 GMT

Run `nvidia-smi` inside a pod in one namespace, then inside a pod in another, on a cluster using GPU time-slicing. Look at the GPU UUID and the PCI bus ID. ![A dark terminal screenshot showing two kubectl exec commands. The first, in namespace team-a, runs nvidia-smi and shows an NVIDIA H100 80GB on PCI bus 00000000:17:00.0 with GPU UUID GPU-4a8f2c1e-7b93-4d21-a0e5-9c1f2b3d4e5f, using 1024MiB, one vllm process. The second, in namespace team-b, runs nvidia-smi and shows the same H100 on the same bus 00000000:17:00.0 with the identical GPU UUID, using 896MiB, its own vllm process. A comment at the bottom reads: different namespaces, same die, no DRAM partition between them.](/images/gpu-deployments-part-8-multi-tenancy-security/same-gpu-two-namespaces.png) *Fig. 1 · two pods, two namespaces, and the exact same GPU UUID on the exact same PCI bus. Each thinks it has the whole 80GB card. Neither can see the other. That invisibility is the danger, not the comfort.* This is the last part. Seven parts built the fleet, wired it, scaled it, watched it, routed to it, and kept it alive through deploys. This one is about the moment more than one team shares it, which is where the assumptions that held for a single tenant quietly stop being true. The load-bearing one, the thing most teams get wrong: a Kubernetes namespace is not a wall on the GPU. ## the isolation you don't have A namespace is an API-scoping and RBAC boundary. It has nothing to do with the hardware. The device plugin advertises `nvidia.com/gpu` resources and the scheduler places pods onto them regardless of which namespace asked, so "one GPU, four replicas" under time-slicing means the scheduler can land a pod from `team-a` and a pod from `team-b` on the same physical die with no memory partition between them. What isolation you actually get depends entirely on the sharing mechanism, and it's worth stating exactly: ![A comparison matrix titled 'what actually isolates two tenants on a GPU', four rows by three columns. Rows: whole GPU (exclusive), MIG (hardware partition), MPS (multi-process service), time-slicing. Columns: memory isolation, fault isolation, compute QoS. Whole GPU: yes, yes, full device. MIG: yes (dedicated DRAM partition, hardware-enforced), yes (fault contained to the instance), guaranteed per slice. MPS: no (software soft caps only), weak (shared MPS server failure domain), soft percent cap. Time-slicing: none, none, none (equal time-share only). A note reads: only MIG or a whole GPU gives a hardware memory wall; MPS and time-slicing are software conventions, not walls.](/images/gpu-deployments-part-8-multi-tenancy-security/isolation-matrix.png) *Fig. 2 · the only two rows with a real memory wall are "whole GPU" and "MIG." MPS caps are software. Time-slicing has nothing. If your mental model was "different namespace, different memory," this is the row that corrects it.* The config that creates this situation is ordinary and common. This is a device-plugin time-slicing setup, the kind people turn on to raise utilization: ```yaml sharing: timeSlicing: resources: - name: nvidia.com/gpu replicas: 4 # one physical GPU -> four schedulable "GPUs", zero isolation ``` MPS swaps that block for per-client soft caps (`CUDA_MPS_PINNED_DEVICE_MEM_LIMIT`, a compute percentage), which are enforced by the driver, not by hardware, and share a failure domain. This isn't only a noisy-neighbor concern. Security researchers have shown covert and side channels through the GPU's shared uncore engines, readable with unprivileged NVML calls, that bypass both MPS and MIG partitioning, and GPU DRAM that isn't zeroed on context teardown has leaked data across tenants under time-slicing. The one-line defensive setting: turn on `renameByDefault: true` so a shared GPU advertises as `nvidia.com/gpu.shared`, and a tenant can't request a "shared" GPU thinking it's a private one. > **Key Insight:** A Kubernetes namespace isolates the API and RBAC, never GPU memory. Under time-slicing or MPS, two tenants can sit on the same physical die with no hardware wall between their VRAM. If tenants don't trust each other, only MIG or a whole-GPU allocation is safe. This is the correction most GPU-cluster security models are missing. ## MIG when tenants can't trust each other MIG is the answer when the boundary has to be real. It carves the physical card into hardware partitions, each with its own DRAM slice, L2, and SMs, so one instance cannot read another's memory regardless of driver or kernel bugs. Expose it through the GPU Operator in `mixed` strategy, which advertises each profile as its own resource: ```bash kubectl patch clusterpolicies.nvidia.com/cluster-policy --type='json' \ -p='[{"op":"replace","path":"/spec/mig/strategy","value":"mixed"}]' ``` The node then advertises MIG profiles as first-class schedulable resources, and a tenant requests a hardware-isolated slice by name: ```yaml resources: limits: nvidia.com/mig-1g.10gb: 1 # one hardware-isolated 10GB partition ``` The tradeoff is the one from part 1: partitions are fixed at configure time, reconfiguring drains the node's GPU pods, and a workload can't burst past its slice. That rigidity is the price of a real wall. For untrusted multi-tenancy it's a price worth paying; for a single trusted team it's usually not. ## quotas that don't waste the cluster Isolation stops one tenant from reading another's memory. Quotas stop one tenant from eating the whole cluster. The blunt version is a `ResourceQuota` capping GPU requests per namespace (extended resources are quotable only via `requests.`): ```yaml apiVersion: v1 kind: ResourceQuota metadata: { name: team-a-gpu-quota, namespace: team-a } spec: hard: requests.nvidia.com/gpu: "8" ``` That's static, though, and it strands GPUs whenever a team is idle. Kueue fixes that by admitting jobs against quota and letting teams in a shared cohort borrow each other's idle GPUs while guaranteeing each its floor back on demand. A `ClusterQueue` holds the quota; the `cohortName` is what enables borrowing: ```yaml apiVersion: kueue.x-k8s.io/v1beta2 kind: ClusterQueue metadata: { name: team-a-cq } spec: cohortName: "gpu-pool" # teams in one cohort lend/borrow idle GPUs resourceGroups: - coveredResources: ["nvidia.com/gpu"] flavors: - name: h100-flavor resources: - name: nvidia.com/gpu nominalQuota: 8 # guaranteed floor borrowingLimit: 4 # cap on what it can borrow from the cohort ``` One version trap worth checking before you copy that: current Kueue serves `v1beta2`, where cohorts are a named field (and can be their own CRD with fair-share weights). Plenty of clusters still run `v1beta1`, where the field is a plain `spec.cohort` string and the standalone Cohort doesn't exist. `kubectl get crd clusterqueues.kueue.x-k8s.io -o jsonpath='{.spec.versions[*].name}'` tells you which you have. NVIDIA's KAI scheduler (the open-sourced core of Run:ai) models the same idea as a hierarchical `Queue` with a `quota` (the deserved floor) and an `overQuotaWeight` (your proportional claim on idle GPUs above it), plus time-based fair-share so a team that under-used earlier gets favored later. ## the endpoint nobody locked The most common GPU-cluster security hole isn't exotic. It's a vLLM pod with a Service and no authentication, reachable from anywhere on the cluster or, worse, the internet: free inference, prompt exfiltration, model theft. Lock it at three layers. Default-deny ingress so only the gateway can reach the model server: ```yaml apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: { name: default-deny-ingress, namespace: team-a } spec: podSelector: {} policyTypes: ["Ingress"] ``` Then an authenticated, TLS-terminating gateway in front (Gateway API `HTTPRoute` with auth attached upstream), and a least-privilege ServiceAccount so a compromised inference pod can't read the rest of the cluster's secrets: ```yaml apiVersion: v1 kind: ServiceAccount metadata: { name: vllm-sa, namespace: team-a } automountServiceAccountToken: false # the model server never calls the API server ``` Which connects to the last quiet failure: secrets. Gated models return a 403 at download time when the pod has no valid Hugging Face token, and that surfaces as exactly the CrashLoopBackOff from part 7, except this time it really is the app. Mount the token from a Secret (or an external manager), never bake it into the image, and pin the model by digest so you don't silently load a tampered checkpoint: ```yaml env: - name: HF_TOKEN valueFrom: secretKeyRef: { name: hf-token, key: token } ``` Prefer `.safetensors` over pickle formats (loading a pickle can execute arbitrary code), pin the model revision to a commit, and verify its checksum. The supply chain for a 140GB weight file deserves the same suspicion as any other dependency, which is the whole thesis of the lazy-security series if you want the longer version. ## the frontier: confidential computing One emerging piece, worth knowing exists even if you're not deploying it yet. Confidential computing extends a CPU trusted execution environment to the GPU, so even the cloud operator hosting your node can't read the weights or activations in VRAM. H100 CC-mode is GA: the driver, running inside a confidential VM, encrypts everything crossing the PCIe bus, and CUDA apps run unmodified once trust is established. Blackwell extends it with NVLink encryption for multi-GPU confidential domains. Independent benchmarks put the overhead under 5% for typical LLM inference (it's the CPU-to-GPU I/O encryption that costs, so small-batch workloads pay a bit more). On Kubernetes it lands through Kata Containers and Confidential Containers, and it's still maturing operationally. For regulated or sensitive-IP inference, it's the direction; for most workloads today, it's a section to file away. ## who's paying for the idle GPUs The last shared-cluster problem is money, and it's the biggest one, because cluster GPU utilization commonly sits at 30 to 40%. The gap between GPUs bought and GPUs doing work is enormous, and the fix is making teams see their own idle GPU-hours. Enforce a `team` label on every GPU pod (a policy engine like Kyverno rejects pods without one), export per-GPU metrics with DCGM, and attribute GPU-hours per team: ```yaml validate: message: "GPU pods must carry a 'team' label for chargeback." pattern: metadata: labels: { team: "?*" } ``` With the label enforced, DCGM's per-pod metrics roll up into GPU-hours per team, and tagging inference requests with an `X-Team` header at the gateway takes it down to token-level attribution. Showback doesn't reclaim a single GPU by itself. It just makes the waste visible to the people who can, which turns out to be most of the battle. That's eight parts, and it's time to say the thing the whole series was circling. A GPU deployment is a dozen layers under one pod, a small network in one box, a large one between boxes, a set of graphs that tell you the truth, an autoscaler that turns it off, a router that doubles it for free, a set of probes that keep it breathing, and a tenancy model that decides who it hurts when it breaks. The silicon was the easy part. Everything that makes it hard lives in the layers around it, and every one of those layers is a place you can be the person who saw it coming, or the one explaining it in the incident review. That was the job the entire time. The GPUs were never the point.

Your model isn't crashing, your probe is

root@harshit.cloud (Harshit Luthra) — Thu, 23 Jul 2026 00:00:00 GMT

The pods were in `CrashLoopBackOff` and the logs said nothing. vLLM started, printed its usual banner, began loading weights, and then died. Restarted, loaded again, died again. It looked exactly like a broken build, so the first hour went into the model, the image, the CUDA version, everything except the actual culprit, which was a fourteen-line probe config nobody had looked at. This is part 7. The first six parts built the fleet, wired it, scaled it, watched it, and routed to it. This part is about keeping a model server alive through the three things that routinely kill it: a health probe that fires too early, a node drain that cuts a request in half, and a model-version rollout that drops traffic on the floor. None of it is glamorous. All of it is what stands between you and a 3am page. ## your model isn't crashing, your probe is Here's the tell, and once you've seen it once you never miss it again: ![A dark terminal screenshot. First command: kubectl get pods -l app=vllm-llama3-70b shows two pods with STATUS CrashLoopBackOff and RESTARTS 4. Second command: kubectl describe pod shows Last State Terminated, Reason Error, Exit Code 137, and an Events section with a Warning 'Liveness probe failed: Get http://10.0.3.14:8000/health: connect: connection refused' followed by 'Container vllm failed liveness probe, will be restarted'.](/images/gpu-deployments-part-7-serving-ops/probe-crashloop.png) *Fig. 1 · the diagnosis is in two lines: exit code 137 (killed by the kubelet, not a segfault) and "Liveness probe failed ... connection refused" during load. The model was fine. The probe shot it.* vLLM binds its HTTP port almost immediately, but `/health` on `:8000` only returns 200 once the weights are loaded and the engine is warm. For a 70B on a cold pull that's several minutes. A default liveness probe with a small `initialDelaySeconds` starts checking during that window, gets a connection refused, decides the container is dead, and the kubelet kills it. It never finishes loading, so it never passes, so it loops forever. The exit code is 137 (128 + SIGKILL) which is the kubelet's fingerprint, not an application crash. The fix is a `startupProbe`. It holds the liveness and readiness probes off entirely until it succeeds, and its `failureThreshold × periodSeconds` is your total load budget: ```yaml startupProbe: httpGet: { path: /health, port: 8000 } periodSeconds: 10 failureThreshold: 60 # 60 × 10s = 600s (10 min) to finish loading livenessProbe: httpGet: { path: /health, port: 8000 } periodSeconds: 10 failureThreshold: 3 # after startup, catch a real hang in 30s ``` The reason this beats just cranking `initialDelaySeconds: 600` on the liveness probe is that once the startup probe passes, liveness reverts to its tight interval and still catches a genuine deadlock in thirty seconds. A giant liveness delay would blind you to real hangs for ten minutes after every restart. With the startup probe in place, the pods ride through the load and come up clean: ```text $ kubectl get pods -l app=vllm-llama3-70b NAME READY STATUS RESTARTS AGE vllm-llama3-70b-7f4b9c6d8f-9m2tq 1/1 Running 0 8m03s vllm-llama3-70b-7f4b9c6d8f-c8xvn 1/1 Running 0 8m03s ``` > **Key Insight:** A model that takes minutes to load needs a startupProbe, not a bigger liveness delay. Without one, the probe that's supposed to detect a dead server is the thing killing a healthy one, and the crash loop looks exactly like an application bug. This is the single most common self-inflicted LLM serving outage. ## draining without dropping a stream The next one bites on every deploy and every node scale-down. When Kubernetes deletes a pod, it sends `SIGTERM` and removes the pod from the service endpoints at the same instant. vLLM handles `SIGTERM` correctly (it stops accepting new requests and finishes the in-flight ones), but if the load balancer is still routing to it during that beat, new requests land on a server that's shutting down. And if the grace period is too short, the kubelet `SIGKILL`s the process mid-generation, dropping a half-finished response the client has to retry from scratch. Two fields fix it. A `preStop` hook that sleeps long enough for the endpoint removal to propagate before `SIGTERM` reaches vLLM, and a `terminationGracePeriodSeconds` set above your longest in-flight generation: ```yaml terminationGracePeriodSeconds: 210 # preStop(15s) + longest decode(~180s) + margin lifecycle: preStop: exec: command: ["sh", "-c", "sleep 15"] ``` The sequence is worth internalizing, because the parallelism is the point: the endpoint removal and the `preStop` sleep happen at the same time, so by the time `SIGTERM` actually reaches vLLM, the load balancer has already stopped sending it traffic. ![A horizontal timeline titled 'what happens when a vLLM pod is deleted', showing the shutdown sequence left to right. t=0: pod marked Terminating, which forks into two parallel tracks: top track 'removed from EndpointSlice, load balancer stops routing new requests', bottom track 'preStop hook runs: sleep 15s'. Both converge at t=15s: 'SIGTERM sent to vLLM, which stops new admissions and finishes in-flight requests'. Then 'in-flight decode drains'. Finally at t=210s a marker 'SIGKILL if grace period elapses' shown in red as the backstop. A note reads 'endpoint removal and preStop run in parallel, so traffic stops before the process does'.](/images/gpu-deployments-part-7-serving-ops/graceful-drain.png) *Fig. 2 · the drain, in order. The whole job of `preStop` is to buy time for the load balancer to stop routing before vLLM stops answering.* ## surviving a node drain Parts 3 and 5 leaned on Karpenter to consolidate and remove idle GPU nodes. That same consolidation, unguarded, will happily evict every replica of a service at once and take it to zero. The guard is a `PodDisruptionBudget`, and for a small pool of expensive GPU replicas you want `maxUnavailable: 1` so at most one goes down for any voluntary disruption: ```yaml apiVersion: policy/v1 kind: PodDisruptionBudget metadata: name: vllm-llama3-70b-pdb spec: maxUnavailable: 1 selector: matchLabels: { app: vllm-llama3-70b } ``` Karpenter drains through the Eviction API, which respects PDBs, so this is enough to keep the service serving through a consolidation. For a pod that's mid-critical-work and must not be interrupted at all, there's a stronger lever, the `karpenter.sh/do-not-disrupt` annotation, which excludes its node from consolidation entirely: ```yaml metadata: annotations: karpenter.sh/do-not-disrupt: "true" ``` Use it deliberately, though. Leave it on permanently and you've told Karpenter it can never reclaim that node, which is how you end up back in part 5's problem of expensive GPUs that never scale down. The PDB is the always-on floor; the annotation is for pods you're actively protecting. ## one replica, many pods When a model is too big for one node (the 405B from part 3, tensor-parallel across eight GPUs and pipeline-parallel across two nodes), a single replica *is* a group of pods, and a normal Deployment can't express that. LeaderWorkerSet can. It treats a leader plus N-1 workers as one unit: `replicas` is the number of these groups, `size` is the pods per group, and `RecreateGroupOnPodRestart` means if any pod in the group dies the whole group restarts, which is correct, because a tensor-parallel replica missing one member is dead weight: ```yaml apiVersion: leaderworkerset.x-k8s.io/v1 kind: LeaderWorkerSet metadata: name: vllm spec: replicas: 2 # two independent model replicas leaderWorkerTemplate: size: 2 # 2 pods each: one leader + one worker restartPolicy: RecreateGroupOnPodRestart ``` The leader starts the Ray head and vLLM's OpenAI server with `--tensor-parallel-size × --pipeline-parallel-size` equal to the total GPUs across the group; the workers join the Ray cluster via the injected `LWS_LEADER_ADDRESS`. The catch from part 2 still applies with force: the group has to land on NVLink-connected GPUs or the cross-node collective crawls, so pair LWS with gang scheduling and topology-aware placement (Kueue's TAS, or the KAI scheduler). Rolling updates are first-class through `rolloutStrategy` with `maxUnavailable` and `maxSurge`, so you can roll a multi-node replica without taking the whole service down. ## rolling out a new model without an outage The last way to drop traffic is deploying a new model version badly. KServe makes the careful version cheap: set `canaryTrafficPercent` on an `InferenceService` and point `storageUri` at the new weights, and it splits traffic between the current good revision (which it tracks automatically) and the new one. ```yaml apiVersion: serving.kserve.io/v1beta1 kind: InferenceService metadata: name: llama3-chat spec: predictor: canaryTrafficPercent: 10 # 10% to the new revision, 90% stays on last-good model: storageUri: "s3://models/llama3-chat/v2" ``` You watch the split live, and the metrics from part 4 (TTFT, error rate, and any quality signal) decide whether it graduates: ```text $ kubectl get isvc llama3-chat NAME URL READY PREV LATEST LATESTREADYREVISION llama3-chat http://llama3-chat... True 90 10 llama3-chat-predictor-00002 ``` Promotion is deleting the `canaryTrafficPercent` field and re-applying: all traffic shifts to the new revision and the old one scales to zero. Blue-green is the same mechanism flipped straight to 100; shadow is mirroring live traffic to the candidate without returning its responses, so you can compare outputs with zero user risk. Two things to know before you rely on it. First, this traffic-splitting is a serverless (Knative) mode feature; in raw deployment mode you split with Gateway API route weights instead. Second, version the weights, the serving config, and the prompt template together as one revision, or your canary metrics are comparing two things that differ in ways you didn't track. (And don't reach for ModelMesh for multi-model serving; the project is archived.) That's the service staying up through loads, drains, and deploys. The last thing between you and a calm on-call isn't the software at all, it's other people: the tenants sharing your cluster, the ones who can reach your endpoint, and a GPU-memory boundary that turns out to be nowhere near where you think it is. That's the final part, and it's the one most likely to end up in an incident review.

The cheapest speedup is your load balancer

root@harshit.cloud (Harshit Luthra) — Sat, 18 Jul 2026 00:00:00 GMT

A team I know had a slow chatbot and did the obvious thing: added replicas. TTFT barely moved. They added more. Still slow, and now the bill was worse. Every GPU showed healthy utilization, the queue metrics from part 4 were climbing, and adding capacity wasn't buying the speedup capacity is supposed to buy. Then someone swapped the Kubernetes Service in front of the pods for a prefix-cache-aware router, changed nothing else, and the same eight GPUs got more than twice as fast. The problem was never the GPUs or the count. It was the load balancer, quietly throwing away the most expensive thing the servers had built. This is part 6. Parts 1 through 5 built the fleet, watched it, and scaled it. This part is about the layer in front of the fleet: how requests get assigned to replicas, and why the default answer (round-robin, the same load balancing you'd use for a stateless web app) is the wrong one for LLM inference. It's the highest ratio of payoff to effort in the whole series, because the win comes from a routing decision, not from hardware you have to buy. ## the load balancer that throws away your cache Here's the thing a normal load balancer doesn't know: LLM replicas are not stateless. Each vLLM replica keeps its own KV cache (from part 2, the running memory of tokens it has already processed), and it also keeps a **prefix cache**: if two requests share the same opening tokens (a long system prompt, a RAG document, the history of a chat), the second one can reuse the first one's cached computation and skip prefill entirely. Prefill is the compute-bound, expensive half of a request. A prefix cache hit makes it nearly free. But each replica has its *own* prefix cache. A round-robin load balancer scatters requests across replicas blind to what each one holds. So a request that shares a 2,000-token prefix with something served thirty seconds ago lands on a different replica that never saw it, re-runs the entire prefill from scratch, and fills its KV cache doing redundant work. Multiply that across a prefix-heavy workload and the fleet spends most of its compute re-prefilling prompts it already processed, on a different pod. The caches sit there full of answers to questions that keep getting routed elsewhere. ![A two-panel diagram contrasting request routing. Left panel titled 'round-robin': four vLLM replicas, incoming requests that share a common prefix drawn as colored tokens, arrows scattering them evenly across all four replicas; each replica shows a mostly-full KV cache and a low prefix-cache-hit badge (~11%), with a note 'every replica re-runs prefill for prefixes its neighbors already cached'. Right panel titled 'prefix-cache-aware': the same requests and four replicas, but arrows route each request to the replica that already holds its prefix; caches show headroom and a high hit badge (~93%), with a note 'a cache hit skips prefill entirely'.](/images/gpu-deployments-part-6-inference-routing/routing-contrast.png) *Fig. 1 · the same four replicas, two routers. Round-robin scatters shared prefixes and re-computes them; prefix-aware routing sends each request to the replica that already has the answer half-built.* ## the numbers, from one honest benchmark This isn't a theoretical gain, and the cleanest measurement of it is a benchmark Andy Golubev ran on EKS in June 2026. The setup was deliberately boring: Qwen2.5-7B on vLLM, eight `g5.xlarge` nodes with one A10G each, one decode replica per node. Two runs, identical in every way except the front door. First run: a plain Kubernetes Service doing round-robin. Second run: an llm-d prefix-cache-aware router. Same model, same eight GPUs, same eight replicas. The workload was `vllm bench serve` replaying 9,000 prompts that shared a pool of 150 long (2,048-token) prefixes, which is exactly the RAG-and-chat shape real traffic takes. The gap was not subtle. ![A before-and-after stat panel titled 'same 8 GPUs, only the router changed' comparing round-robin to prefix-cache-aware routing across five metrics. Output throughput: 2,742 to 6,423 tokens per second (2.34x). Wall clock for the run: 840 to 359 seconds. Mean TTFT: 19.0 seconds to 0.86 seconds (about 22x). Prefix cache hit rate: 11 percent to 93 percent. Requests waiting in queue: about 180 to 0. KV cache utilization: pinned near 99 percent down to 64 to 71 percent, showing headroom instead of thrash.](/images/gpu-deployments-part-6-inference-routing/routing-benchmark.png) *Fig. 2 · the Golubev EKS numbers. The one that matters most is the last one: round-robin ran the cache pinned at 99% and thrashing; the smart router left it headroom, because it stopped generating redundant work.* Read the cache-hit line again: 11% to 93%. Round-robin was reusing almost nothing; the aware router reused almost everything. Mean TTFT dropped from a genuinely broken 19 seconds to under a second. And the fleet stopped queueing (waiting requests went from ~180 to zero) not because it got more capacity, but because it stopped wasting the capacity it had. Google reports the same shape from GKE's managed version: TTFT improvements up to 96% at peak load on prefix-heavy workloads. The lesson is uncomfortable and freeing at once. Before you buy a ninth GPU, check whether your router is making the eight you have re-do each other's work. ## routing to the replica that already knows The mechanism is simpler than it sounds. vLLM emits events about what its prefix cache holds. A cache-aware router consumes those events and keeps a live picture of which replica has which prefixes cached. When a request arrives, instead of picking the next replica in rotation, the router picks the one most likely to already hold the request's prefix, so the cache hit actually happens. That decision point has a name in the Kubernetes world: the **Endpoint Picker**, or EPP. It doesn't just look at prefix locality. A good EPP scores every candidate replica on several signals at once and sends the request to the best total score: - **prefix-cache locality**: does this replica already hold the request's opening tokens? Longer match, higher score. - **load**: what's this replica's KV-cache utilization and queue depth right now? A replica that's already saturated scores lower even if it has the prefix, so you don't stampede one hot pod. - **LoRA affinity**: if you serve multiple fine-tuned adapters, is the right adapter already loaded here? Loading one is not free, so prefer the replica that has it. The router balances those against each other, and it can queue or shed when everything is overloaded. This is the difference between "which pod is next" and "which pod will serve this request fastest given what it already has warm." Same request, same pods, a decision made with information the round-robin balancer never had. ## the standard nobody had two years ago The reason this is worth a whole chapter now, and wasn't in 2024, is that it stopped being a bespoke hack and became a Kubernetes standard. The **Gateway API Inference Extension** is a SIG-Networking project that adds LLM-aware routing on top of the ordinary Gateway API. It introduces an `InferencePool`, which is a group of pods that share the same accelerator, base model, and model server (the LLM-shaped version of a Service), and it wires an Endpoint Picker into the request path through Envoy's external-processing (`ext-proc`) protocol, so the proxy calls out to the EPP for a decision on every request. ![An architecture diagram titled 'the inference gateway request path'. A request enters from the left into an Envoy-based Gateway (L7 proxy handling TLS and connection management). The gateway makes an ext-proc call out to the Endpoint Picker (EPP), which reads live signals from the replica pool (prefix-cache state, KV-cache utilization, queue depth, loaded LoRA adapters), scores each replica, and returns the chosen endpoint. The gateway then forwards the request to the selected replica inside an InferencePool of four vLLM pods. A note reads 'InferencePool is the LLM-aware Service; the EPP is the brain'.](/images/gpu-deployments-part-6-inference-routing/inference-gateway-arch.png) *Fig. 3 · the request path. The gateway is a normal Envoy proxy; the intelligence lives in the Endpoint Picker it consults per request, over the same ext-proc hook Envoy already uses for auth and rate limiting.* The project has reached GA, with `InferencePool` graduating to a stable v1 API. (One honest caveat: the exact GA milestone landed over early-to-mid 2026, so pin the version you're deploying rather than trusting a blog's date, this one included.) Google productized it as **GKE Inference Gateway**, which is GA and is literally powered by the llm-d router underneath. So the "advanced" thing here is also increasingly the default, k8s-native thing, which is exactly the production framing this series cares about. ## the whole stack, named "llm-d" gets thrown around as if it were a server. It isn't; it's an assembly, and it's clearer to name the pieces: - **vLLM** is the model server. It owns the per-replica KV and prefix cache and emits the cache events the router needs. - **KServe** is the serving control plane, exposing an `LLMInferenceService` custom resource so you describe the model and its serving config as a normal Kubernetes object. - **The inference gateway** is Envoy plus the Gateway API Inference Extension: the data plane plus the LLM-aware routing above. - **The router itself** is that L7 proxy plus the Endpoint Picker, making the per-request decision from cache, load, and LoRA signals. - **Disaggregated prefill/decode** from part 3 is an optional add-on here, with the KV cache handed between pools over a connector. If you're not ready to adopt the full gateway, the vLLM project's own **production-stack** helm chart is a lighter on-ramp: it deploys a router service in front of your vLLM pods that already does model-aware and prefix-aware routing, plus KV-cache offload through LMCache. Same idea, smaller commitment. Either way, the thing you're installing is a router that knows what a KV cache is. ## watching the router Part 4 said the metrics that matter live in the serving engine. Add one more surface: the router. The signals that tell you whether the routing layer is earning its keep are the ones that moved in that benchmark. Prefix cache hit rate is the headline. A round-robin fleet sits low (that 11%); a well-routed prefix-heavy fleet should sit high (the 93%). If you turned on cache-aware routing and the hit rate didn't climb, either your workload doesn't actually share prefixes or the router isn't seeing the cache events. > **Key Insight:** Prefix cache hit rate is the one dashboard number that proves the router is working. Watch it alongside per-replica KV utilization and waiting-request count. If hit rate is high and evenly spread, routing is doing its job. If one replica is pinned while others idle, your scorer is over-weighting cache locality and stampeding a hot pod. If hit rate is flat near zero, your traffic isn't prefix-heavy and this whole chapter buys you little. That last line is the honest boundary. Cache-aware routing is close to free money for workloads with real prefix reuse: RAG over a shared corpus, long system prompts, multi-turn chat, agents replaying context. For traffic where every prompt is unique and short, there's no cache to reuse and the fancy router mostly just adds a hop. The KV-connector interfaces underneath are also still moving, so expect some churn if you build on the bleeding edge. Know which workload you have before you reach for this, the same way you'd check the topology in part 2 before promising a throughput number. The router is the front door. What sits behind it, the probes that decide when a replica is ready, the way you roll a new model version out without dropping a request, the tenants all fighting over the same pool of GPUs, is where a production inference platform actually gets hard. That's where the series goes next.

Scaling GPU inference to zero and back

root@harshit.cloud (Harshit Luthra) — Thu, 16 Jul 2026 00:00:00 GMT

The finance dashboard is what started it. A staging cluster running two H100 nodes around the clock, mostly to serve a demo that got maybe forty requests a day, all of them during business hours. The nodes sat idle from 7pm to 9am and every weekend, billing the whole time. Somebody added scale-to-zero over a Friday. Monday morning the first person to open the demo waited six minutes for a response, assumed it was broken, and filed a bug. The bill went down and the product got worse, which is scale-to-zero working exactly as designed and nobody being happy about it. This is the last part of the series. Parts 1 through 4 covered building GPU infrastructure and seeing what it's doing. This part is about the thing that actually shows up on the invoice: a GPU you're paying for while it does nothing. Scaling GPU inference elastically, all the way to zero when there's no traffic, is the biggest lever on the bill. It's also genuinely hard, because unlike a stateless web pod that starts in a second, a GPU replica has to drag a hundred gigabytes of model weights onto the card before it can serve a single token. The whole post is about that gap. ## the cold-start tax Scaling a web service to zero is free because starting a new pod is nearly instant. Scaling an LLM to zero is expensive because starting a replica is not. Add up what has to happen before the first token, on a cold node: - **Node provisioning.** The cluster asks the cloud for a GPU node, waits for it to boot and join. One to five minutes, and that assumes the GPU is even available to hand you (more on that at the end). - **Image pull.** A CUDA plus vLLM container image is commonly 5 to 15 GB. On a fresh node with a cold cache, pulling and unpacking it is minutes, not seconds. - **Model load.** Llama-3-70B in BF16 is about 140 GB of weights, usually sitting in object storage. Reading that over the network and moving it onto the GPU is the big one, and done naively it's several minutes. - **Warmup.** CUDA graph capture, `torch.compile`, and a few dummy forward passes to populate caches. Tens of seconds before the first real request is fast. Stack those up and a naive 70B cold start runs six to nine minutes, more if those 140 GB of weights come cold over a slow path. That's the tax. Everything else in this post is a way to stop paying it, so that scale-to-zero saves the money without the six-minute Monday. One cruel interaction hides in here: a naive Kubernetes health probe treats that multi-minute load as a failure. A liveness probe with default timing restarts the pod mid-load, and now you have a crash loop that looks like a vLLM bug but is really a probe firing too early. Set the liveness `initialDelaySeconds` above your worst-case load time, keep it longer than the readiness delay, and add a `preStop` sleep so in-flight requests drain before the pod goes down. It's the single most common self-inflicted serving outage, and it costs nothing to avoid once you've seen it once. ![A horizontal stacked-bar diagram titled 'the cold-start tax: what the first user waits for'. A single bar is divided into four segments sized by time: node provisioning (1–5 min), image pull (2–4 min for a 5–15GB image), model weight load (several minutes for ~140GB), and warmup (tens of seconds). The total is labeled 'naive cold start: 5+ minutes'. Below, a second much shorter bar labeled 'after the fixes in this post' shows the same stages compressed, annotated with 'lazy image pull + model streaming + warm nodes → tens of seconds'.](/images/gpu-deployments-part-5-scale-to-zero/cold-start-budget.png) *Fig. 1 · where the minutes go on a cold start. Each stage has a fix; the trick is that you have to attack all of them, because the slowest one sets the wait.* ## scale on the queue, not the GPU Before scaling to zero, get scaling to *anything* right, and the first mistake is scaling on GPU utilization. It's the obvious metric and it's the wrong one, for the reason part 4 laid out: a GPU can sit at 95% while the queue is empty, or at 40% while requests pile up. Scaling on GPU-util adds replicas late and removes them at the wrong time. Scale on the serving signal instead. The metric that actually tracks unmet demand is the queue: `vllm:num_requests_waiting`, or KV-cache utilization as a leading indicator. You'll still find plenty of setups scaling on GPU-util or cache percentage, and those aren't wrong so much as indirect; the point is to scale on demand rather than on how busy the chip happens to look, with tail latency as the guardrail. In practice that means KEDA (the Kubernetes event-driven autoscaler) with a Prometheus trigger reading that metric, or an HPA wired to the same value through the Prometheus adapter. For traffic that's genuinely request-driven and spiky, KEDA's HTTP add-on can scale on in-flight request count directly. The shape is the same: pick the metric that means "users are waiting," set the threshold at the per-replica capacity you measured with the load test in part 4, and let it add replicas before the queue becomes TTFT. ## actually reaching zero Scaling to zero is a special case of scaling, with one extra problem: when you're at zero replicas, there's nothing running to receive the request that's supposed to wake you up. Something has to catch that first request and hold it while a replica spins up. KEDA does this with an activation threshold: below it, the deployment sits at zero; the first event scales it to one. Knative Serving builds the pattern in more deliberately, with an **activator** component that buffers incoming requests while a cold replica starts, then releases them once it's ready, so the request is slow but not dropped. KServe (which runs on Knative) exposes this as a simple `minReplicas: 0` on an InferenceService, and it's the most common way teams get GPU model servers to zero on Kubernetes. ![A lifecycle state diagram titled 'scale-to-zero, and the request that pays for it'. Four states in a loop: 'ZERO: no replicas, no cost' at rest; an incoming request triggers an arrow to 'ACTIVATING: activator buffers the request, autoscaler asks for a GPU node'; then 'COLD START: pull image, load weights, warm up' (annotated 'the first user waits here'); then 'SERVING: replica ready, requests flow'; and after an idle timeout an arrow back to 'ZERO'. The buffered first request is shown held at the activator through the cold-start state.](/images/gpu-deployments-part-5-scale-to-zero/scale-to-zero-lifecycle.png) *Fig. 2 · the lifecycle, and the unlucky first request. Everything after this figure is about shrinking the cold-start box so that request waits seconds, not minutes.* The honest catch is that reaching zero and the cold-start tax are the same coin. Zero replicas is where the savings are, and it's also where every request pays the full six minutes. Everything below is about making that first request cost seconds instead, because a scale-to-zero setup with a six-minute cold start is a cost win and a product loss, and you rarely get to keep both. There's also a breakeven worth running before you build any of this. Scale-to-zero wins when utilization is low and spiky; it loses somewhere around half-time, where a dedicated node is both cheaper than paying serverless rates by the second and free of cold starts entirely. If a GPU is busy more than half the day, don't scale it to zero. Right-size it and leave it on. ## killing the image pull The container image is the first fixable minute. Two families of fix. The first is to stop pulling the whole thing before you start. **SOCI** (Seekable OCI), lazy-loading via a containerd snapshotter, lets a container start running against an index of the image and fetch the actual bytes on demand, so the model server boots while the layers it doesn't need yet are still downloading. `estargz` (the containerd stargz snapshotter) and Nydus do the same lazy-pull trick with different formats; GKE's Image Streaming is the managed version, and it cut a 5.4 GB Triton image's start from 191 seconds to 30. There's also a variant for the case where you *will* read the whole image anyway (AI images touch most of their bytes immediately): SOCI's parallel-pull mode just parallelizes the download and unpack instead of lazy-loading, and AWS measured roughly 60% off a 10 GB image. The second is to not pull over the network at all. Pre-bake the image into the node's disk image so it's local before the pod schedules. Run an in-cluster pull-through cache or a peer-to-peer image mirror (Spegel and friends) so the second node to need an image gets it from a neighbor instead of the registry. Pre-pull hot images with a DaemonSet so they're warm before traffic arrives. None of these is clever; all of them beat pulling 15 GB cold from a registry when a hundred replicas try it at once. ## killing the model load The bigger minute is the model weights, and this is where the newer tooling has moved fast. Loading 140 GB of safetensors off a disk or object store the default way is serial and slow. The fixes stream instead. **NVIDIA's Run:ai Model Streamer** reads weights from object storage in many parallel streams straight onto the GPU, overlapping download with load, and vLLM supports it directly (`--load-format runai_streamer`). NVIDIA's own benchmark took an 8B model's S3 load from 28 seconds at four streams to under five at thirty-two. CoreWeave's **tensorizer** (`--load-format tensorizer`) serializes the model into a format that streams from S3 or local disk with near-zero deserialization overhead. Both turn a multi-minute load into tens of seconds. Underneath, `safetensors` already supports memory-mapped zero-copy loading, which helps when the file is local. And local is the other half. Cache the weights on the node's NVMe (an instance-store disk) so a restart reads from local flash instead of re-downloading. Or mount a shared, fast filesystem so every replica reads the same warm copy, but mind the access mode: a `ReadWriteOnce` PVC serves one replica and then silently blocks the second pod from mounting, so anything that scales past a single replica needs `ReadWriteMany` (EFS, FSx for Lustre, NFS, CephFS). That RWO-to-RWX switch is a classic thing to discover the first time an autoscale event never becomes a second pod. Some teams ship the model as its own OCI artifact and let the image machinery above handle it (KServe's modelcar pattern). The principle is the same one from part 2: the bandwidth between the weights and the GPU is the bottleneck, so shorten that path. The frontier technique skips loading altogether. GPU memory snapshots (Modal and Cerebrium both ship this, built on the CUDA checkpoint/restore API in recent driver branches) checkpoint a fully warmed replica, weights on the card and CUDA graphs already captured, then restore that image straight onto a GPU. Because it bypasses weight load, `torch.compile`, and graph capture in one move, it's the only approach that also kills the warmup tax. Modal reports a vLLM Qwen model dropping from 45 seconds to 5, and Cerebrium measured cold starts down 71 to 88 percent. The catch is portability: a snapshot is pinned to a specific GPU model and driver branch, so it's a per-SKU artifact, not a universal one. > **Key Insight:** Scale-to-zero is a tradeoff, not a free win. Every second of cold start is latency the first user eats; every minute of warm idle is money you burn. You buy down the cold start with lazy image pulls, model streaming, and a warm node pool, then set a minimum replica floor high enough that your p99 cold start stays inside the SLO. "Zero when truly idle, one when it might not be" beats a dogmatic zero. ![A diagram mapping each cold-start stage to the fixes that attack it, three columns. Column one 'image pull (minutes)': SOCI / stargz / Nydus lazy pull, pre-bake into the node image, P2P mirror (Spegel), slim the image. Column two 'model load (minutes)': Run:ai Model Streamer, tensorizer, local NVMe or shared PVC cache. Column three 'warmup (tens of seconds)': GPU memory snapshots, limited CUDA graph sizes. A banner across the bottom reads 'you have to attack all three: the slowest unfixed stage sets the wait'.](/images/gpu-deployments-part-5-scale-to-zero/cold-start-fixes.png) *Fig. 3 · the fix menu, one column per stage. Snapshots are the only trick in the third column, which is why they're the frontier: everything else leaves the warmup tax standing.* ## turning off idle nodes Scaling pods to zero doesn't save anything if the expensive GPU node they were on keeps running. The node has to go too. This is the autoscaler's job below the pod level: Karpenter consolidates workloads and removes nodes that are empty or underutilized (`consolidateAfter`, disruption budgets so it doesn't yank capacity mid-request), and the older cluster-autoscaler scales node groups down on the same principle. A GPU node that's been empty for a few minutes is thirty to a hundred-plus dollars a day; letting it linger is the single most common way GPU bills quietly balloon. For predictable traffic, the laziest win is scheduled scaling. If the demo only serves business hours, a cron trigger that scales the floor to zero at 7pm and back to one at 9am captures most of the savings with none of the cold-start risk during the day. And to hide cold starts when you do scale up, keep a warm node in reserve with low-priority placeholder pods (over-provisioning): real work evicts the placeholders instantly and lands on a node that already has the image, so the pod cold start doesn't also pay the node cold start. ## spot, and the capacity trap Two closing realities that scale-to-zero runs into. Spot instances make idle capacity cheap, and for inference they can work (unlike the gang-scheduled training from part 3, a single inference replica dying is survivable). But a spot GPU can be reclaimed with about two minutes' notice, so you need a node-termination handler that drains in-flight requests and a plan for where the replacement comes from. Which is the trap: scaling up assumes there's a GPU to scale up *onto*. In 2026, popular GPUs are not always available on demand, and a scale-to-zero service that can't reacquire an H100 at 9am Monday is worse than one that never scaled down. The mitigations are the capacity blocks and reservations from part 3, an on-demand fallback when spot is dry, and a warm floor for anything with a real SLO. Scale-to-zero is a cost strategy, not a capacity strategy, and confusing the two is how you save money right up until the morning you can't get your GPUs back. That's the cost side handled. The fleet scales with demand and turns itself off when it's idle, and the first user back doesn't wait five minutes for the privilege. But there's one lever left, and it's the strangest one, because it costs nothing to pull. Everything so far has quietly assumed that a request, once it arrives, lands on some replica and gets served. Part 6 is about that word "some." It turns out that which replica you pick, out of a pool that all look identical from the outside, can make the exact same hardware more than twice as fast. The load balancer, of all the unglamorous things, is where the last big win hides.

What a green GPU dashboard hides

root@harshit.cloud (Harshit Luthra) — Sat, 11 Jul 2026 00:00:00 GMT

The GPU dashboard was a wall of green. Every card pinned at 90-something percent utilization, power near TDP, temperatures fine, no XID errors. By every signal from part 1 of this series, the box was healthy and working hard. Meanwhile the on-call channel had three messages from product asking why the chatbot took eight seconds to say its first word. The hardware was busy. The users were furious. Both were true. That is the trap of monitoring GPU inference with GPU metrics. A card at 92% utilization tells you a kernel is running. It tells you nothing about whether requests are piling up in a queue, whether the KV cache is full and requests are being evicted, or whether the first token is landing in 200 milliseconds or eight seconds. This is part 4 of the series. Parts 1 through 3 built the thing: the stack under a pod, the wires in a box, the network between boxes. This part is about seeing what it's doing once real traffic hits it, and the short version is that the numbers users feel live in the serving engine, not on the GPU. ## the four numbers users actually feel Before any dashboard, get the vocabulary straight, because LLM latency is not one number. A request has a shape, and four measurements describe it. **TTFT**, time to first token, is how long the user stares at a blinking cursor before anything appears. It's dominated by prefill (processing the whole prompt) plus however long the request sat in a queue before the server picked it up. This is the number product complains about. **ITL**, inter-token latency, is the gap between consecutive tokens once generation starts. (Some tools report TPOT, time per output token, the averaged version of the same thing; benchmarks like GuideLLM show both side by side, so don't treat them as interchangeable in a report.) It's what makes text feel like it's streaming smoothly or stuttering out. Decode is memory-bandwidth-bound, so ITL degrades as you pack more concurrent requests onto a GPU. **End-to-end latency** is the whole request, first byte to last. For a long generation it's mostly `output_tokens × ITL`, which means it's as much about how much the model says as how fast it says it. **Throughput** is output tokens per second across all requests, and it moves in the opposite direction from latency. Batch more requests together and throughput climbs while per-request latency gets worse. There's no single "fast" setting. There's a frontier, and where you sit on it is a product decision (a chatbot wants low TTFT, a batch summarization job wants raw throughput) rather than a tuning default. The number that captures both at once is **goodput**: the requests per second you can serve while still meeting your latency SLO. A server can post gorgeous raw throughput while quietly blowing p99 TTFT for half its users, so goodput (throughput, filtered by "did it meet the SLO") is the only throughput figure worth quoting. ![A timeline diagram of a single inference request showing where each latency metric is measured. A horizontal track runs left to right: the request arrives and sits in a queue (labeled 'queue time'), then prefill runs (labeled 'prefill'), then the first token emerges (a marker labeled 'TTFT ends here: queue + prefill'), then a series of output tokens stream out with the gaps between them labeled 'ITL / TPOT', and the far right marks 'end-to-end latency'. Below, a note reads 'TTFT is what the user waits for; ITL is how smooth the stream feels'.](/images/gpu-deployments-part-4-observability/request-latency-anatomy.png) *Fig. 1 · one request, four measurements. Queue time hides inside TTFT, which is why a busy GPU and a slow first token go together.* ## what vLLM actually tells you vLLM exposes a Prometheus `/metrics` endpoint, and once you've read it a few times the health of the server is obvious at a glance. The metrics that matter split into two groups: what the queue is doing, and what the KV cache is doing. The queue first. `vllm:num_requests_running` is how many requests are being served right now; `vllm:num_requests_waiting` is how many are stuck in line because the server can't fit them yet. A healthy server has a running count near its batch capacity and a waiting count near zero. When `num_requests_waiting` starts climbing and staying up, that eight-second TTFT has arrived, and no GPU metric will show it. `vllm:request_queue_time_seconds` measures the wait directly. Then the KV cache (the model's running memory of the tokens it has already processed, which grows with every active request and every token generated). `vllm:kv_cache_usage_perc` is the fraction of KV cache in use. This is the one to stare at, because when it approaches 100% vLLM has to start **preempting**: evicting a half-finished request to free memory, then recomputing it from scratch later. `vllm:num_preemptions_total` counts that happening. A rising preemption rate means the server is thrashing, doing the same work twice, and every latency number is about to get worse. ![A side-by-side comparison of two vLLM server states drawn as metric panels. Left panel labeled 'healthy': running requests near batch capacity, waiting requests at zero, KV-cache usage around 60 percent, preemptions flat at zero, p99 TTFT well under the SLO line. Right panel labeled 'drowning': running requests flat at capacity while waiting requests climb, KV-cache usage pinned near 100 percent, preemptions rising, p99 TTFT crossing above the SLO line. A note reads 'the GPU looks equally busy in both; only the serving metrics tell them apart'.](/images/gpu-deployments-part-4-observability/healthy-vs-drowning.png) *Fig. 2 · the same server, healthy and drowning. GPU utilization is high in both; the queue, the cache, and the preemption counter are what separate a busy server from a failing one.* The rest fill in the picture. `vllm:time_to_first_token_seconds` and `vllm:inter_token_latency_seconds` are histograms, so you alert on the p95 or p99, not the average (the average hides the user who waited twelve seconds). `vllm:prompt_tokens_total` and `vllm:generation_tokens_total` give you real throughput, computed with `rate()` (vLLM removed its old pre-averaged throughput gauges, so you do the division yourself). And `vllm:prefix_cache_hits_total` over `vllm:prefix_cache_queries_total` tells you how much prompt reuse you're getting, which matters enormously for the RAG and multi-turn workloads from part 2. It's also the number a smart router watches to decide which replica to send a request to, which is the whole subject of part 6. One warning that will save you an afternoon: these names drift between releases. The V1 engine renamed the KV-cache gauge (it used to be `gpu_cache_usage_perc`) and swapped the per-output-token metric to `inter_token_latency_seconds`. Diff the live `/metrics` output of your exact build before you copy anyone's PromQL, this post included. > **Key Insight:** GPU utilization tells you the chip is busy; it never tells you users are waiting. The metric that predicts an angry inbox is vllm:num_requests_waiting climbing, usually because vllm:kv_cache_usage_perc hit its ceiling and the server started evicting half-finished requests. Watch the queue and the cache, not GPU-util. ## the same story in SGLang and Triton The engine changes, the questions don't. SGLang exposes its own Prometheus metrics behind `--enable-metrics`: `sglang:num_running_reqs` and `sglang:num_queue_reqs` are the running-and-waiting pair, `sglang:token_usage` is the KV-cache fraction, and `sglang:cache_hit_rate` reports how often its RadixAttention prefix cache paid off. (One gotcha of exactly the kind above: SGLang flipped the metric prefix from `sglang:` to `sglang_` in v0.5.4, and the bundled Grafana dashboard hasn't caught up, so a fresh install can read "No Data" until you fix the prefix.) If you built on SGLang for its prefix-sharing (the reason to pick it in part 2), that last one is how you confirm the bet is paying. Triton with the TensorRT-LLM backend reports through Triton's metrics endpoint instead: request and queue durations, inflight-batcher stats, per-model success and failure counts. Different names, same three questions every time: how long are requests waiting, is the cache saturated, is the tail latency inside the SLO. The lesson worth internalizing is that these are all the same dashboard with different labels. Queue depth, cache utilization, tail TTFT, tail ITL. Learn to read one engine and you can read all of them. ## compute-bound, memory-bound, or queue-bound The reason to keep the GPU metrics from part 1 next to the serving metrics is that together they diagnose *why* the server is slow, which is the only thing that tells you what to do about it. Three shapes cover most incidents. If TTFT is high and `num_requests_waiting` is high but `gpu_cache_usage_perc` is low, you're **queue-bound**: requests are backing up faster than you can start them, and the fix is more replicas (which is part 5). If ITL is degrading and `DCGM_FI_PROF_DRAM_ACTIVE` is pinned while tensor activity isn't, you're **memory-bound** on decode, and the fix is a smaller batch, quantization, or better KV-cache management. If tensor cores are saturated during prefill and DRAM isn't, you're **compute-bound**, which for inference usually means very long prompts and points at chunked prefill or a prompt-length limit. ![A three-way decision diagram titled 'why is the server slow?'. A central question branches on the combination of serving and GPU metrics into three outcomes. Branch one: high queue (num_requests_waiting up) plus low KV-cache usage routes to 'queue-bound: add replicas'. Branch two: degrading inter-token latency plus DRAM_ACTIVE pinned but low tensor activity routes to 'memory-bound on decode: smaller batch, quantize'. Branch three: tensor cores saturated during prefill routes to 'compute-bound: chunked prefill, cap prompt length'. Each branch pairs a serving metric with a DCGM metric.](/images/gpu-deployments-part-4-observability/bottleneck-diagnosis.png) *Fig. 3 · the serving metric tells you something is wrong; the GPU metric next to it tells you which kind of wrong. You need both panels on the same screen.* Neither set of metrics is enough alone. GPU metrics without serving metrics miss the queue entirely. Serving metrics without GPU metrics can't tell a memory-bound stall from a compute-bound one. The dashboard that works has both, side by side, on one screen. ## what to alert on Most teams over-alert on the GPU and under-alert on the experience. The page that matters fires on the user's SLO, not the hardware's vitals. A working starter set: - **TTFT p99 over budget** for N minutes. This is the customer-facing SLO. Everything else is a leading indicator of this. - **`num_requests_waiting` sustained above zero.** A brief spike is fine; a standing queue means you're under-provisioned and the next thing to break is TTFT. - **Preemption rate climbing.** `num_preemptions_total` moving means the KV cache is saturated and the server is recomputing evicted work. It's the early warning before latency falls off a cliff. - **Error rate.** Request failures and, specifically, CUDA out-of-memory events, which on an inference server usually mean a batch or context-length setting is too aggressive. - **The part-1 hardware alerts still stand.** XID errors, thermal throttle, ECC. A dying GPU shows up as latency variance long before it shows up as an error, so keep those wired. Tail latency is the whole game here. Alerting on average TTFT is how you find out about an outage from the customer instead of the pager, because the average stays calm while your p99 is on fire. ## tracing a single slow request Aggregate metrics tell you the fleet is unhealthy. They don't tell you why *this* request took nine seconds. For that you want per-request tracing, and the ecosystem has standardized on OpenTelemetry's GenAI semantic conventions: spans carry `gen_ai.*` attributes (the model, input and output token counts, the request parameters) so a single request's journey through the gateway, the queue, prefill, and decode is one connected trace. When a specific user reports a slow response, a trace tells you whether it sat in a queue, hit a cache miss, or just asked for a 4,000-token essay. The metrics say the kitchen is slow; the trace shows you which order got lost. ## proving it before prod You don't want to discover your latency frontier during a launch. Load-test the serving endpoint before it sees real traffic, with a tool that speaks LLM rather than plain HTTP. vLLM ships `vllm bench serve` (the old `benchmark_serving.py`), which replays a request distribution, reports TTFT, ITL, and throughput percentiles, and computes goodput directly if you hand it SLO thresholds. GuideLLM (now a Red Hat project) does the same with a `sweep` mode that finds your safe operating range on its own; NVIDIA's genai-perf covers the Triton side, though NVIDIA is steering new work to its successor AIPerf; and the Kubernetes serving working group's inference-perf standardizes the numbers across engines so you can compare vLLM to SGLang honestly. Whatever you pick, the output you care about is the same: the curve of tail latency against offered load, and the load at which p99 TTFT crosses your SLO. That crossover is the goodput ceiling, the real per-replica capacity, and it's the input to everything in part 5. Because that's the thing this post sets up. Once you can see the queue building and the cache saturating, the obvious next question is: why am I staring at these graphs manually at 2am instead of having the queue depth add a replica by itself, and drop it again when the traffic goes home. That's scaling, and scaling GPUs that cost six dollars an hour is its own kind of problem.

Scaling GPUs past one box

root@harshit.cloud (Harshit Luthra) — Thu, 09 Jul 2026 00:00:00 GMT

When Meta trained Llama 3 405B on 16,384 H100s, the cluster hit 419 unexpected interruptions over 54 days. That's one failure every three hours or so, for 54 days straight, and about 78% of them were hardware. GPUs and their HBM3 memory accounted for roughly half. They also watched the datacenter's power draw swing by tens of megawatts as thousands of GPUs idled and resumed in sync, which is a sentence that should make any infrastructure engineer sit up. This is the part of the series where the GPU stops being the interesting component. Part 1 was the stack under one pod, part 2 was one box and the wires inside it. Once you cross the node boundary, the network becomes the machine, failures become continuous rather than exceptional, and the scheduler decides whether your very expensive cluster does useful work or deadlocks against itself. Everything here is about the stuff between the boxes. ## the network is the machine Synchronous training does an all-reduce of the entire gradient (every parameter, billions of them) every single step. That collective is a hard barrier: the slowest link gates every GPU in the job. Add nodes and your compute scales, but the all-reduce volume grows and so does the chance that one link is slow or dead. This is why, past one node, communication rather than FLOPs sets your scaling efficiency, and why the network gear costs as much attention as the GPUs. ![A network topology diagram titled 'rail-optimized fabric'. It shows four GPU nodes at the bottom, each with eight GPUs and eight NICs. The NICs are grouped into rails: NIC-0 of every node connects up to the same leaf switch, NIC-1 of every node to the next leaf switch, and so on across eight leaf switches. The leaf switches connect up to a row of spine switches forming a non-blocking fat-tree. An annotation notes 'GPU_i on any node reaches GPU_i on any other node in one switch hop'.](/images/gpu-deployments-part-3-scaling-out/rail-optimized-fabric.png) *Fig. 1 · a rail-optimized fat-tree. Each GPU gets its own NIC ("rail"), and rail N of every node lands on the same leaf switch, so the GPUs that talk most take the shortest path.* The fabric itself is a two-horse race. InfiniBand is the incumbent for dedicated training superclusters: the generation ladder runs EDR 100Gb, HDR 200Gb, NDR 400Gb (Quantum-2 switches, ConnectX-7 NICs), and now XDR 800Gb (Quantum-X800, ConnectX-8). Most DGX SuperPODs and the big named clusters run it. RoCE v2, RDMA over Ethernet, is winning share on cost and on letting existing Ethernet teams reuse what they know. The catch with RoCE is that it needs a carefully tuned lossless fabric (Priority Flow Control plus ECN marking) or you get congestion storms, and getting that right at scale is its own discipline. Meta is the proof that Ethernet can do it. They built two 24,576-GPU H100 clusters, one on RoCE and one on Quantum-2 InfiniBand, and trained Llama 3 405B on the RoCE one with no network bottleneck, after co-designing the topology, the PFC/ECN thresholds, and an all-reduce-aware load balancer. That's the honest framing: RoCE works at scale, but Meta spent real engineering to make it work. Two more pieces earn their keep on either fabric. GPUDirect RDMA lets the NIC DMA straight into GPU memory, skipping a bounce through host RAM, and without it every hop stages through system memory. SHARP does the reduction inside the switch ASIC, so gradients get summed in the network instead of shuttled between every node, which on the newest Blackwell fabrics is a large multiplier on effective all-reduce bandwidth. ## NCCL across the wire The same NCCL from part 2 handles inter-node collectives, and the failure mode here is specific and common: NCCL silently falls back to TCP sockets when it can't find or use the RDMA path, and the job "works" while running an order of magnitude too slow. The env vars that prevent that are worth pinning in your launcher. `NCCL_IB_HCA` names which RDMA NICs to use, and getting it wrong means NCCL picks one NIC and loses your rail parallelism. `NCCL_SOCKET_IFNAME` has to point at the real data-plane interface, not `eth0` or `lo`, a classic container and Kubernetes trap. `NCCL_CROSS_NIC=0` on a rail-optimized fabric keeps a ring on the same rail instead of hopping across them. `NCCL_IB_GID_INDEX` is the RoCE gotcha: the wrong GID index gives you no traffic or a silent slow path. On a fresh cluster the first all-reduce is routinely two to ten times slower than optimal until the env vars, the GID index, the PFC/ECN config, and the topology file are all correct. The bring-up ritual is always `nccl-tests`: run `all_reduce_perf`, measure the achieved bus bandwidth, compare it against what 400 or 800 Gb should give you, and don't trust the cluster until the number is close. ## who schedules the gang Here's the failure that surprises people coming from web infrastructure. Vanilla Kubernetes schedules pods independently, one at a time, with no concept of a job that needs all its pods at once. Give it a 4-node training job and it will happily place 3 pods and leave the 4th Pending forever, holding three nodes of GPUs idle. Run two such jobs and they can each grab most of what the other needs and starve each other indefinitely. Distributed training is all-or-nothing, and a scheduler that doesn't know that will deadlock your cluster. The first time it happens you assume you're out of GPUs. You're not. They're all sitting idle, reserved by pods that will never get their partners. ![A two-panel diagram titled 'why training needs gang scheduling'. Left panel, labeled 'vanilla Kubernetes': a four-pod job where three pods are Running on GPU nodes and one pod is stuck Pending, with the three running pods marked as holding GPUs idle while waiting, and a red 'deadlock' label. Right panel, labeled 'gang scheduling': the same four-pod job where all four pods are admitted together as one unit or none at all, marked 'all-or-nothing placement, no partial allocation'.](/images/gpu-deployments-part-3-scaling-out/gang-scheduling.png) *Fig. 2 · gang scheduling in one picture. The left side is how a cluster quietly wedges itself; the right side is the fix, and the reason every GPU scheduler below exists.* The fix is gang scheduling: admit all N pods together or none, so partial allocations can't happen. The tools that provide it each have an honest drawback: - **Slurm** is the HPC default and has gang scheduling and topology awareness built in, plus new block scheduling for aligning jobs to NVL72 racks. Its weakness is that containers and multitenancy are bolted on (Pyxis plus Enroot), and it's a poor fit for long-running inference services. That gap is why Slurm-on-Kubernetes projects like SchedMD's Slinky and CoreWeave's SUNK exist. - **Kubernetes vanilla** has no gang scheduling and no topology awareness by default. You add a batch scheduler on top; you don't run training on the default scheduler. - **Volcano** is the de facto CNCF choice: gang scheduling via PodGroups, queues, fair-share. It runs as a second scheduler that bypasses the default, which complicates coexistence with normally-scheduled workloads, and gang scheduling itself costs maybe 10–15% utilization because resources sit idle waiting for the full gang. - **Kueue** is the Kubernetes-native answer for queueing and quota, and it cooperates with the default scheduler instead of replacing it. The tradeoff is that it does admission and quota, not fine-grained placement, so you still need scheduler plugins underneath for the actual gang and topology binding. - **Run:ai** is the commercial option, now NVIDIA-owned, with fractional GPU and pooling. NVIDIA open-sourced the core as KAI Scheduler (Apache 2.0, now CNCF Sandbox), so a free path exists, but the full enterprise feature set stays paid and KAI is young as a standalone project. - **YuniKorn** brings strong hierarchical-queue multitenancy from the Spark world, at the cost of being another full scheduler replacement with a smaller AI-specific ecosystem than Volcano. The cross-cutting truth is that gang scheduling trades utilization for progress. Holding GPUs idle while you wait for the full gang is the price of not deadlocking, and it's a price worth paying. ## splitting the model When a model outgrows one GPU, there are three axes to split it on, and real training combines them. Data parallelism replicates the whole model on each GPU and all-reduces gradients; it's the simplest and only works when the model plus its optimizer states fit on one card. Tensor parallelism splits individual matrix multiplies across GPUs and is communication-heavy, so you keep it inside a node on NVLink (TP=8 is the common ceiling). Pipeline parallelism cuts the layers into stages across nodes and passes activations point-to-point, which is cheap enough to cross the network. ![A diagram titled '3D parallelism: how a frontier model maps to a cluster' showing a grid of GPU nodes. Tensor parallelism is shown splitting a single layer across the eight GPUs within one node, connected by NVLink and labeled 'TP=8, stays inside the node'. Pipeline parallelism is shown splitting the model's layers into stages across several nodes, labeled 'PP across nodes, cheap point-to-point'. Data parallelism is shown replicating the whole arrangement across groups of nodes with an all-reduce between replicas, labeled 'DP / FSDP on top'.](/images/gpu-deployments-part-3-scaling-out/3d-parallelism.png) *Fig. 3 · the standard frontier recipe: tensor-parallel inside the NVLink domain, pipeline-parallel across nodes, data-parallel on top. Match each split to the bandwidth it can afford.* For the common case of "my model doesn't fit but I want to stay in native PyTorch," FSDP2 shards the parameters, gradients, and optimizer states across GPUs and reconstructs each layer on the fly via all-gather, prefetching the next shard to overlap communication with compute. DeepSpeed's ZeRO does the same idea in stages: stage 1 shards optimizer states, stage 2 adds gradients, stage 3 adds parameters and is functionally equivalent to FSDP. For frontier scale and maximum MFU you reach for Megatron-Core and combine tensor, pipeline, and sequence parallelism into the 3D (now 4D, with expert parallelism for MoE) recipe: TP=8 inside the node, pipeline across nodes, data parallelism on top. Checkpointing is the reliability workhorse and used to be the tax that made frequent saves unaffordable. Async distributed checkpointing fixed that by writing state in a background thread that overlaps the next iterations; TorchTitan reports 5–15x lower checkpoint overhead than synchronous saves. That matters directly because of the failure rate: at one interruption every three hours, you want to checkpoint on the order of tens of minutes, and torchrun's elastic mode restarts the job from the last snapshot when a node dies. The newer torchft goes further, recovering a failed replica from a healthy peer without restarting the whole job. ## when the model won't fit one node Serving crosses the node boundary for the same reason training does: the model, or its KV cache, exceeds one node's total HBM. DeepSeek-V3 at 671B, Llama 405B, the big mixture-of-experts (MoE) models. You split them with tensor and pipeline parallelism across nodes, and for MoE you add wide expert parallelism, spreading experts across many nodes so each GPU holds few experts but sees a large batch per expert. The pattern that's become standard is disaggregated prefill and decode. Prefill (processing the prompt) is compute-bound; decode (generating tokens one at a time) is memory-bandwidth-bound. Running them in one pool means prefill work stalls decode latency. Splitting them into separate worker pools lets you scale each for its own bottleneck and transfer the KV cache between them over RDMA. It isn't a free win, though. Moving the KV cache between pools costs bandwidth, so disaggregation pays off when prefill interference is genuinely the bottleneck (long prompts, high concurrency) and can be net-negative when it isn't. DeepSeek runs it because at their scale it clearly is; a chatbot with short prompts might not need it at all. ![A diagram titled 'disaggregated prefill and decode' showing an inference request flowing left to right. The request first hits a pool of prefill workers, labeled 'compute-bound: process the whole prompt', drawn as a small cluster of GPUs. The resulting KV cache is transferred over RDMA to a separate, larger pool of decode workers, labeled 'memory-bandwidth-bound: generate tokens one at a time'. An annotation notes 'scale each pool independently; prefill no longer stalls decode', with a footnote citing DeepSeek-V3's production split of a 32-GPU prefill unit feeding a 320-GPU decode pool.](/images/gpu-deployments-part-3-scaling-out/disaggregated-prefill-decode.png) *Fig. 4 · prefill and decode want different hardware ratios, so modern stacks run them as separate pools and ship the KV cache between them. DeepSeek-V3 runs a 32-GPU prefill unit in front of a 320-GPU decode pool.* DeepSeek's own deployment runs a 32-GPU prefill unit (4 nodes, expert-parallel across 32) feeding a much larger decode pool, and an LMSYS reproduction on 96 H100s hit 52,000 input and 22,000 output tokens per second per node. On Kubernetes the primitive that expresses "this is one model replica made of many pods" is LeaderWorkerSet: one leader, N workers, scheduled and scaled as a unit, which is exactly what gang scheduling and topology-aware placement need to bite on. NVIDIA Dynamo and llm-d sit on top: Dynamo for distributed serving with a KV-cache-aware router, and llm-d for KV-cache-aware routing on the Gateway API Inference Extension. That routing layer turns out to be one of the biggest free wins in the whole stack, which is why it gets its own part 6. ## everything fails at scale Reliability stops being a checkbox and becomes the main event. The Llama 3 numbers from the top of this post are the reference point, and ByteDance's MegaScale run on 12,288 GPUs tells the same story: 55.2% MFU and more than a hundred failure-recovery events over a few weeks. The failure you can't see coming is silent data corruption, where a GPU computes a wrong number without erroring. It doesn't crash or log anything. It just hands back the wrong answer, and your loss curve grows a mysterious kink a few hours later. Meta caught six such events in 54 days; Google reports SDC-related disruptions roughly every one to two weeks. A single corrupted gradient contaminates the global update across every worker, and it's now a first-class reliability topic with its own whitepapers. > **Key Insight:** At cluster scale, hardware failure is the normal state, not an exception you engineer away. A 16k-GPU run loses a GPU every few hours. You can't stop that, so the whole game is checkpointing often enough and keeping enough hot spares that a dead node costs you minutes instead of the whole run. The straggler is the SDC's cousin: a GPU or link that's degraded but not dead, quietly throttling a synchronous job because collectives move at the speed of the slowest member. Detecting it at scale is genuinely hard, so systems run periodic self-check diagnostics that pause the job, measure NVLink and compute per node, diagnose, and resume from checkpoint. Imbue open-sourced their bare-metal playbook for a 4,088-H100 cluster: check VBIOS and baseboard firmware, the Mellanox OFED stack, PCIe link speed and width, then run matmuls to measure actual NVLink bandwidth, cordon anything that fails, and swap in a hot spare automatically. That last part is the operating model at scale. You don't fix nodes in the critical path; you drain them and pull from a spare pool, because the cluster is always partially broken and the job has to keep moving. ![A donut chart infographic titled 'Llama 3 405B: what interrupted 54 days of training' showing the breakdown of 419 unexpected interruptions. Segments: GPU faults 30.1 percent, HBM3 memory 17.2 percent, network switch and cable 8.4 percent, GPU SRAM 4.5 percent, GPU processor 4.1 percent, and other or software causes making up the remainder. A center label reads '419 interruptions, ~1 every 3 hours, ~78% hardware'.](/images/gpu-deployments-part-3-scaling-out/llama3-failure-breakdown.png) *Fig. 5 · the failure breakdown from Meta's Llama 3 405B run. GPUs and their memory are roughly half; the rest is the long tail an at-scale operator plans around, not against.* ## paying for it The economics are why all of the above matters. At $2–6 per GPU-hour, a 16,000-GPU cluster idling during a recovery burns thousands of dollars a minute, and gang scheduling means one bad node can idle the whole job. That's the real argument for the reliability engineering: not uptime for its own sake, but goodput, the useful training throughput net of failures and restarts. ![A horizontal bar chart infographic titled 'GPU cost per hour, on-demand ballpark (2026)' comparing neocloud versus hyperscaler pricing for three GPUs. H100 SXM: neocloud about 2.5 to 3.5 dollars, AWS about 6.88 dollars. H200: neocloud about 3.8 to 4 dollars, Azure about 10 to 13 dollars. B200: neocloud about 5 to 6 dollars, AWS about 14.24 dollars. A note reads 'reserved commits cut 16 to 40 percent; spot is 30 to 70 percent cheaper but nearly unusable for gang-scheduled training'.](/images/gpu-deployments-part-3-scaling-out/gpu-cost-comparison.png) *Fig. 6 · the on-demand spread between neoclouds and hyperscalers is wide, and it moves monthly. Big training runs almost never pay on-demand; they live on reserved capacity or capacity blocks.* The pricing spread is wide and moves every month. Neoclouds like Lambda, CoreWeave, and Nebius run an H100 around $2.5–3.5 per hour; AWS lists closer to $6.88. B200s are $5–6 on neoclouds and north of $14 on AWS. Reserved commitments of one to twelve months cut 16–40% off on-demand, and that's where most large training capacity actually lives. Spot is 30–70% cheaper and nearly unusable for gang-scheduled training, because losing any one node preempts the whole synchronous job and reacquiring N contiguous, topology-aligned nodes on spot is a fantasy. That gap is why capacity blocks exist: AWS EC2 Capacity Blocks and GCP's Dynamic Workload Scheduler let you reserve co-located GPUs for a fixed window, booked weeks ahead, because on-demand can't guarantee the topology and reserved is too long a commit for one run. AWS raised those block prices about 15% in early 2026, which tells you which way demand is going. That's the whole arc of building this stuff. A GPU deployment is not a GPU. It's a dozen layers under one pod, a small network inside one box, and a large one between boxes, and the interesting failures always live in the wiring rather than the silicon. Meta's cluster lost a GPU every three hours and still trained a frontier model, because the whole apparatus around the GPUs (the fabric, the scheduler, the checkpointing, the spare pool) was built to keep moving while parts of it were on fire. That's how you build it. The next part is how you watch it once real traffic arrives, which turns out to be a different problem than watching the GPUs.

One box, eight GPUs, and the wires between them

root@harshit.cloud (Harshit Luthra) — Sat, 04 Jul 2026 00:00:00 GMT

We bought two boxes that were supposed to be identical. Eight H100s each, same rack, same image. On one of them a tensor-parallel serve of Llama-3-70B did about 3,000 tokens a second. On the other it did 900, with every GPU pinned at high utilization the whole time. Same model, same code, same card count. The difference was a PCIe switch and a BIOS setting nobody had checked, and it took most of a day to find because every dashboard said both boxes were healthy. That's the thing about a multi-GPU box. It looks like a bag of eight GPUs. It behaves like a small, opinionated network, and the wiring between the cards matters more than the cards. This is part 2 of the series. Part 1 was the twelve-layer stack under a single GPU pod. This one stays inside one chassis: how the GPUs talk, how to read the topology, why NCCL is slow, and how a 70B model actually lands on the hardware. Part 3 leaves the box. ## the box is a network, not a bag of GPUs The first question about any multi-GPU server is how the GPUs are wired, because that sets a hard ceiling on everything above it. There are two very different animals sold as "8-GPU servers." ![A diagram contrasting two eight-GPU server layouts. Top, labeled 'HGX / DGX baseboard': eight GPUs each connected to a bank of four NVSwitches so every GPU reaches every other at full 900 GB per second, drawn as a dense fully-connected mesh. Bottom, labeled 'PCIe box': eight GPUs hanging off PCIe switches and the CPU root complexes, GPU-to-GPU traffic forced through PCIe at a fraction of the bandwidth, drawn as a sparse tree routed through the CPUs.](/images/gpu-deployments-part-2-single-node-multi-gpu/intra-node-topology.png) *Fig. 1 · the same GPU count, two topologies. The NVSwitch baseboard is why 8-way tensor parallelism works; the PCIe box is why it sometimes doesn't. Check which one you bought before you promise a throughput number.* An HGX or DGX baseboard wires all eight GPUs through a bank of NVSwitches. Every GPU reaches every other GPU at the full NVLink rate, non-blocking. On H100 and H200 that's NVLink 4 at 900 GB/s per GPU. On B200 it's NVLink 5 at 1.8 TB/s. That flat, full-bandwidth mesh is the reason you can split a model eight ways and have the halves talk fast enough to keep up. A cheaper "8x PCIe" box has no NVSwitch. The GPUs hang off PCIe switches and the CPU root complexes, and GPU-to-GPU traffic crawls through PCIe, often routed up through the CPU. PCIe Gen5 x16 is about 128 GB/s, Gen4 about 64. NVLink 4 is roughly seven times faster than Gen5 and fourteen times faster than Gen4. That gap is the entire reason tensor parallelism cares about your topology. The two "identical" boxes in the opening weren't identical: one had the NVSwitch mesh, the other routed two of its GPU pairs across a PCIe switch with a BIOS feature quietly strangling them. One units trap that trips up everyone reading spec sheets: NVIDIA quotes NVLink bandwidth bidirectionally. A100's "600 GB/s" is 300 each way. Pick a convention, state it once, and don't compare someone's unidirectional number to your bidirectional one. > **Key Insight:** Two servers with the same eight GPUs can differ by 3× on the same job. Before you promise a throughput number, run nvidia-smi topo -m and confirm the GPUs talk over NVLink (the NV-prefixed rows), not over PCIe routed through the CPU (the SYS rows). ![A horizontal bar chart infographic titled 'GPU-to-GPU bandwidth, per GPU' comparing five interconnects. Bars from shortest to longest: PCIe Gen4 x16 at about 64 GB/s, PCIe Gen5 x16 at about 128 GB/s, NVLink 3 (A100) at 600 GB/s, NVLink 4 (H100/H200) at 900 GB/s, NVLink 5 (B200) at 1800 GB/s. A note reads 'NVLink 4 is roughly 7x PCIe Gen5, which is why tensor parallelism wants NVLink.'](/images/gpu-deployments-part-2-single-node-multi-gpu/nvlink-vs-pcie-bandwidth.png) *Fig. 2 · bandwidth per GPU across the interconnects you'll actually meet. The jump from PCIe to NVLink is the one that decides whether a split model keeps up with itself.* ## reading nvidia-smi topo -m You don't have to guess at any of this. `nvidia-smi topo -m` prints the whole connection matrix, and learning to read it is the single most useful GPU-ops skill after `nvidia-smi` itself. Every cell tells you how one GPU reaches another, and the symbols form a quality ladder from best to worst: - `NV#`: connected by # bonded NVLinks. Best. `NV18` means eighteen links, full H100 mesh. - `PIX`: a single PCIe bridge, same switch. Fine. - `PXB`: multiple PCIe bridges, but not across the CPU host bridge. - `PHB`: crosses a PCIe host bridge, through the CPU, same NUMA node. - `NODE`: crosses host bridges within a NUMA node. - `SYS`: crosses the inter-socket link between CPUs. Worst. This is CPU-to-CPU-to-GPU. If the GPUs in your tensor-parallel group show `NV18` to each other, you're golden. If any pair shows `SYS`, your collective operations are dragging across the socket interconnect and you've found your 900-tokens-a-second box. The same matrix has a column for GPU-to-NIC affinity, which matters enormously for the multi-node story in part 3: you want the NIC on the same PCIe complex as the GPU it feeds. ## NCCL picks a road Every framework that splits work across GPUs (PyTorch DDP and FSDP, DeepSpeed, Megatron, the tensor-parallel path in vLLM) does its cross-GPU communication through NCCL, NVIDIA's collectives library. NCCL is where "the GPUs need to agree on a number" turns into actual bytes on actual wires, and it auto-picks the road. Inside one box it prefers, in order: peer-to-peer over NVLink (best), peer-to-peer over PCIe, shared host memory (staged through RAM), then network sockets (worst, and a sign something's misconfigured). When an all-reduce (the step where every GPU merges its numbers with all the others and ends up with the combined result) is slow, the debugging loop is almost always the same handful of moves: ```bash # 1. what did NCCL actually choose? NCCL_DEBUG=INFO python train.py 2>&1 | grep -iE 'via|transport|channel' # "via P2P/direct pointer" = good. "via SHM" or "via NET/Socket" intra-node = bad. # 2. confirm the GPUs are NVLinked, not routed over SYS nvidia-smi topo -m # 3. benchmark against the ceiling all_reduce_perf -b 8 -e 4G -f 2 -g 8 # from nccl-tests # 4. if a hang clears when you disable P2P, you have an ACS/IOMMU problem NCCL_P2P_DISABLE=1 python train.py ``` The env vars worth knowing are few. `NCCL_DEBUG=INFO` tells you what topology and transport NCCL chose. `NCCL_P2P_LEVEL` and `NCCL_P2P_DISABLE` control peer-to-peer. `NCCL_SOCKET_IFNAME` picks the bootstrap interface, and pointing it at the wrong one (`lo`, `docker0`) is a classic way to make init hang. `NCCL_TOPO_FILE` lets you hand NCCL a topology description, which you sometimes need on cloud VMs because virtualized PCI hides the real affinity and NCCL guesses wrong. On a healthy 8-GPU NVSwitch box you usually touch none of these and it just works. The trouble starts on anything cheaper or virtualized. ## the invisible tax: NUMA and ACS Two settings below the framework quietly decide whether your bandwidth numbers are real, and neither shows up in a GPU dashboard. The first is NUMA pinning. A multi-socket server splits its PCIe lanes and RAM between CPU sockets, and each GPU is physically wired to one socket. Run your process on socket 0 while it drives a GPU hung off socket 1, and every host-to-device copy crosses the inter-socket link. NCCL's low-latency protocol stages data through a pinned CPU buffer, so this hits communication, not just data loading. The fix is a one-liner: `numactl --cpunodebind=0 --membind=0 `, matched to the socket that owns the GPU. `nvidia-smi topo -m` prints the affinity so you know which socket that is. It's genuinely deflating to spend an afternoon profiling and find the answer was a `numactl` prefix, but that's most of this job. The second is PCIe ACS, Access Control Services, and it's the one that cost us most of that day. ACS forces PCIe peer-to-peer transactions to route up through the CPU root complex so the platform can police them. That defeats direct GPU-to-GPU DMA across a PCIe switch: latency climbs, throughput collapses, and NCCL can hang outright. ACS has to be off for peer-to-peer to work across a switch. You check it with `lspci -vvv | grep -i acsctl` and disable it in BIOS or with a `setpci` loop over the bridges. Its cousin, the IOMMU, does the same routing-through-the-root-complex thing, which is exactly why passthrough GPUs on cloud VMs often show degraded peer-to-peer: the isolation that makes virtualization safe is the isolation that makes GPUDirect slow. If `nvidia-smi` insists your GPUs "are not P2P capable," the shortlist is ACS enabled, IOMMU on, consumer cards, or GPUs on different root complexes. `p2pBandwidthLatencyTest` from the CUDA samples confirms which. ## fitting a 70B model on the box The practical question most teams actually have is: how many GPUs does my model need, and how do I split it. Tensor parallelism splits every layer's weight matrices across GPUs, which means every token, at every layer, triggers a collective to recombine the partial results. That makes TP communication-bound and latency-sensitive, which is the real reason it wants NVLink. On a PCIe-only box, tensor parallelism is frequently slower than just pipelining the layers, and vLLM's own guidance says as much: no NVLink, prefer `--pipeline-parallel-size` over `--tensor-parallel-size`. The memory math for Llama-3-70B is a worked example worth carrying around: - Weights: 70B params times 2 bytes for BF16 is about 140 GB. That already doesn't fit on one 80GB card. - KV cache (the model's running memory of the tokens it has already processed): roughly 2.5 GB per sequence at 8K context, climbing to tens of GB at 128K. This is what eats whatever VRAM the weights left behind and sets your max batch size. - Real footprint with activations and framework overhead lands north of 200 GB. So Llama-3-70B at BF16 wants two H100 80GB cards (`--tensor-parallel-size 2`), or four A100 40GB, or a fistful of 24GB cards. Tensor parallelism splits the KV cache too, so two GPUs buys you roughly double the batch headroom, not just room for the weights. Quantize to FP8 on Hopper or Blackwell, or INT4 with AWQ, and the weights roughly halve or quarter, which can drop a 70B onto a single card at some quality cost. You can quantize the KV cache too (`--kv-cache-dtype=fp8_e5m2`), which shrinks the biggest consumer of leftover VRAM once batch sizes climb. And a rule worth internalizing: tensor-parallel a model only when it genuinely doesn't fit one GPU. For a model that does fit, run several replicas at `--tensor-parallel-size 1` instead, because TP's per-layer communication is pure overhead you're paying for nothing when one card already holds the whole model. The constraint people forget: your TP size has to divide the number of attention heads evenly, so you don't get to pick arbitrary GPU counts, and on Kubernetes it also has to equal the pod's `nvidia.com/gpu` limit or the server won't start. (You learn this the moment `--tensor-parallel-size 6` refuses to start and the error message does nothing to help.) ## the caveats that page you Dense GPU boxes fail in ways that a single card never does, and most of them present as "one GPU is a little slow" rather than a clean error. Thermal throttling is first. Eight cards share airflow, and a fully loaded chassis runs hot. Data-center GPUs start clocking down before about 85°C, and `nvidia-smi --query-gpu=clocks_throttle_reasons.active` will tell you whether it's a software thermal slowdown, a hardware one, or a power cap. Power capping is the sibling: `nvidia-smi -pl` sets a limit below the card's TDP (700W on an H100 SXM), and a rack without enough power budget will cap every card and slow the whole box uniformly. Then the straggler problem, which is the nastiest because it hides. Collective operations synchronize every step, so the slowest GPU sets the pace for all of them. One card that's throttling, power-capped, or quietly degraded drags the entire tensor-parallel group, and the symptom is maddening: every GPU shows high utilization, throughput is low, and nothing errors. You find it by looking at per-GPU clocks and temperature and hunting the outlier. The rest of the list is worth a scan before you sign off on a box: - PCIe lane starvation. A card silently negotiating Gen3 x4 instead of Gen5 x16 is pure mystery slowness. `lspci -vv` shows the negotiated `LnkSta`. - NVLink errors. `nvidia-smi nvlink -e` shows CRC and replay counters. Rising counts mean a flaky link or cable and degraded bandwidth. - Oversubscribed PCIe switch. Cheap boxes put several GPUs behind one switch uplink, and they contend for it. - The `/dev/shm` trap, if you're on Kubernetes. NCCL stages some intra-node transfers through shared memory, and a container's default 64 MiB `/dev/shm` hangs multi-GPU serving with no useful error. Mount an `emptyDir` with `medium: Memory` at `/dev/shm`. It's in every production vLLM manifest, and it's why tensor-parallel works on a bare VM but hangs in a pod. - Xid 79, the "fell off the bus" from part 1, shows up here too when a card overheats or loses power delivery under a full load it never saw in acceptance testing. ## picking an inference engine If the box is for serving rather than training, the engine choice matters, and by 2026 the field has settled. All the live engines converge on the same two tricks: continuous batching (decide the batch membership every decode step so the GPU never idles on the slowest request) and paged KV cache (manage the cache like OS virtual memory so you don't waste most of your VRAM on fragmentation). The differences are in the scheduler and the compile strategy. ![A two-by-two quadrant diagram titled 'choosing a single-node inference engine'. Axes are ease of operations (horizontal) versus raw performance on NVIDIA hardware (vertical). vLLM sits high on ease and solid on performance, labeled 'the default: broad model + hardware coverage, sane defaults'. TensorRT-LLM sits highest on performance but lower on ease, labeled 'max throughput on NVIDIA, but you compile an engine per model'. SGLang sits mid-high on both, labeled 'RadixAttention: wins on shared-prefix, RAG, agents, MoE'. TGI sits low, greyed out, labeled 'maintenance mode: HuggingFace now points you to vLLM or SGLang'.](/images/gpu-deployments-part-2-single-node-multi-gpu/inference-engine-quadrant.png) *Fig. 3 · where the four engines land. Start at vLLM; move only when a profiled bottleneck points somewhere specific.* Start with vLLM. It has the broadest model and hardware coverage, it invented PagedAttention, and it stands up with sane defaults faster than anything else. Reach for TensorRT-LLM when a profiled bottleneck justifies the cost, because it delivers the highest raw throughput and lowest latency on NVIDIA hardware but makes you compile a per-model, per-GPU engine and run a heavier ops burden for it. Reach for SGLang when your workload shares prefixes: RAG, multi-turn chat, agents, or high-concurrency MoE, where its RadixAttention prefix cache pulls meaningfully ahead. And don't start new work on TGI; HuggingFace put it in maintenance mode and points people at vLLM or SGLang themselves. One layer up from the engine, NVIDIA's Dynamo wraps any of these for distributed, disaggregated serving with a KV-cache-aware router, and NIM packages them as prebuilt microservices. On a single box the raw engine is what you're tuning, but those are the names you'll meet the moment you scale out. The two boxes from the opening ran the same vLLM. Same engine, same flags, one served three times the traffic. The engine was never the variable. The wires were. Which is the whole lesson of a single node, and also the reason part 3 is about the wires between nodes, where the same story plays out at a hundred times the scale and a hundred times the cost.

The dozen layers under a GPU pod

root@harshit.cloud (Harshit Luthra) — Thu, 02 Jul 2026 00:00:00 GMT

The pod was stuck in `Pending` and the node had eight H100s sitting idle. `kubectl describe node` said `nvidia.com/gpu: 0`. `nvidia-smi` on the host printed all eight cards, healthy, 40°C, nothing running. So the hardware was fine, the driver was fine, and Kubernetes was convinced there were zero GPUs in a box that cost more than my car. That gap is the whole job. A GPU pod doesn't run on a GPU. It runs on about a dozen layers stacked between the silicon and your container, and any one of them can be quietly broken while every layer above and below it looks green. If you've shipped normal apps on Kubernetes, a GPU pod looks identical right up until it doesn't: underneath sits a stack of hardware and driver pieces a web pod never touches, and that's where the surprises live. This series is about running those layers in production without getting paged. Part 1 is the anatomy: what the layers are, what breaks at each one, and which numbers actually tell you the truth. Part 2 is a single box with eight GPUs and the wires between them. Part 3 is scaling past one box, where the network becomes the machine. Part 4 is watching the whole thing under real traffic, part 5 is scaling it to zero when nobody's using it without the bill or the cold start eating you, part 6 is the routing layer in front of it all, where the right load balancer buys a 2× speedup for free, part 7 keeps it breathing through loads and deploys, and part 8 is sharing it with other teams without the tenants, or a security boundary that isn't where you think, burning you. ## the dozen layers between silicon and your pod Start at the bottom and climb. Each layer trusts the one under it and lies to the one above it when things go wrong. ![A vertical layer-stack diagram titled 'what a GPU pod actually sits on', showing twelve stacked layers from bottom to top: GPU silicon and NVSwitch, firmware (VBIOS and GSP), kernel driver nvidia.ko, userspace driver libcuda, CUDA runtime and cuDNN and NCCL, NVIDIA Container Toolkit, container runtime containerd, device plugin, node feature discovery, GPU Operator, scheduler, and the workload pod on top. Each layer has a short label describing its one job and a red note describing its characteristic failure.](/images/gpu-deployments-part-1-anatomy/gpu-stack-layers.png) *Fig. 1 · the stack under a GPU pod. Read it bottom to top; the failure notes on the right are where you'll actually spend your nights.* The **silicon** is the GPU itself plus the NVSwitch fabric that wires the GPUs on a board together, plus the NIC (a ConnectX-7 or BlueField-3) that carries traffic off the box. This is where hardware faults live: ECC errors (bit-flips in memory the card catches and corrects), thermal throttle, a card that stops answering on the PCIe bus. **Firmware** sits on the card. Modern GPUs have a GSP, a GPU System Processor, a little RISC-V core that runs firmware and offloads work the host driver used to do. When you hear about a GPU "hanging" for no visible reason, the GSP firmware timing out is a common culprit. The **kernel driver** is `nvidia.ko` and friends (`nvidia-uvm`, `nvidia-peermem`). It's a kernel module, which means it's compiled against your exact kernel headers. Upgrade the kernel without rebuilding the module and the driver won't load. This is the layer that breaks on a Tuesday because someone patched the base image. The **userspace driver** is `libcuda.so`, the CUDA driver API. It ships with the driver, not with CUDA, and this trips people up constantly. `nvidia-smi` talks to the driver through NVML, which is why `nvidia-smi` can work while your actual CUDA program fails: they're using different entry points into the same stack. The **CUDA runtime** is `libcudart` plus the math and collective libraries: cuBLAS, cuDNN, NCCL. Here's the thing nobody tells you on day one. PyTorch and TensorFlow wheels bundle their own copy of all of this. When you `pip install torch==2.x+cu128`, you are installing CUDA 12.8, cuDNN, and NCCL inside the wheel. The host node doesn't need a CUDA install at all. It needs a driver new enough to satisfy that bundled runtime, and nothing more. Once that clicks, half the version confusion evaporates. The **NVIDIA Container Toolkit** (`nvidia-ctk`, `libnvidia-container`, currently 1.19.1) is the bridge between the host driver and the container. At container start it injects `/dev/nvidia*` and the driver libraries into the container's filesystem. Miss this and you get the classic symptom: `nvidia-smi` works on the host, fails inside the pod. The **container runtime** is containerd or CRI-O running runc underneath. It has to be told to use the `nvidia` runtime. If `default_runtime_name` in `/etc/containerd/config.toml` isn't set, pods land on the node with no GPU access and no obvious error. (Recent GPU Operator versions wire this through the NRI/CDI plugin instead of a default runtime, but the failure mode is identical: get it wrong and the pod sees no GPU.) The **device plugin** (0.19.x) is the piece that talks to the kubelet. It counts the GPUs and advertises them as `nvidia.com/gpu: 8`, or `nvidia.com/mig-1g.10gb: 56` if you're slicing. When this crashes or can't reach the driver, you get `nvidia.com/gpu: 0` and pods stuck Pending. That was my incident at the top. The device plugin had crash-looped after a driver-container restart and never re-registered. **Node Feature Discovery** and **GPU Feature Discovery** label the node with what it has: GPU model, memory, compute capability, MIG profile, driver version. The scheduler reads those labels to place pods. Wrong labels, wrong placement. The **GPU Operator** (v26.3.x) is the thing that installs and manages every layer above the kernel as a set of DaemonSets. It runs the driver as a container, wires the toolkit, deploys the device plugin, DCGM, the MIG manager. It's a huge convenience and one more control loop to debug when it gets stuck reconciling. On top, the **scheduler** (kube-scheduler, or Kueue, Volcano, Run:ai, or Slurm) decides which pod lands on which GPU. And finally the **workload**: a training job that needs all N GPUs at once or nothing, or an inference server that would happily take a seventh of one card. Twelve layers. The reason GPU infra feels harder than normal infra isn't any single layer. It's that the failure at layer 3 shows up as a symptom at layer 9, and the tooling at layer 9 has no idea layer 3 exists. > **Key Insight:** The layer that breaks and the symptom you notice are rarely the same layer. The pod is Pending up at the scheduler, but the cause is a driver that didn't load down near the metal. When a GPU pod misbehaves, debug from the bottom of the stack up, not the top down. One part of this stack is quietly being rebuilt under you. The device plugin advertising `nvidia.com/gpu: 8` is a flat count: a pod asks for a number and gets whatever GPUs the node has. Kubernetes 1.34 (September 2025) made Dynamic Resource Allocation (DRA) generally available, and it's the eventual replacement for that model. DRA is a `ResourceClaim` API, the way a `PersistentVolumeClaim` is for storage, so a pod can ask for "two NVLink-connected GPUs with at least 40GB each" instead of a bare count, and it's how rack-scale multi-node NVLink (GB200 NVL72) gets scheduled at all. NVIDIA ships a DRA driver through the GPU Operator. The device plugin is still the common path in mid-2026, but a stack diagram drawn today should treat it as the model DRA is replacing, not the permanent one. ## the version matrix that pages you The single most common self-inflicted outage is a version mismatch, and the matrix has four axes: driver, CUDA toolkit, cuDNN, and framework. The good news is that three compatibility mechanisms mean you rarely have to line up all four exactly. The bad news is that nobody explains which mechanism they're relying on, so it feels like luck. Backward compatibility is the easy one. A newer driver runs older CUDA binaries, always. An R580 driver runs a CUDA 12 app and a CUDA 13 app without complaint. So keeping the driver ahead of everything is safe. CUDA minor-version compatibility is the one you lean on daily. Any CUDA 12.x toolkit runs on any driver that supports 12.0. A driver from the 525 era will run a CUDA 12.8 binary, because they share the CUDA 12 major. This is why the framework-bundles-its-own-CUDA pattern works: the wheel carries CUDA 12.8, your node has some 12-capable driver, and they meet in the middle. Forward compatibility is the escape hatch. The `cuda-compat` package ships updated `libcuda` stubs that let a *newer* CUDA major run on an *older* driver branch. It's how you run a CUDA 13 app on a node still pinned to an R535 driver you can't upgrade yet. It only works on data-center GPUs, and it's a deliberate override, not something to build on. Here's the current data-center driver picture as of mid-2026: | Branch | Type | EOL | CUDA | |---|---|---|---| | R535 | LTS | June 2026 | 12 | | R580 | LTS | June 2028 | 13 | | R595 | Production | March 2027 | 13 | | R610 | New Feature | Aug 2026 | 13 | If you take one thing from this section: manage the driver, let the framework carry the rest. Pin your training and serving images to NGC base images (`nvcr.io/nvidia/pytorch:25.xx`) where NVIDIA has already matched CUDA, cuDNN, and NCCL for you, and you delete an entire class of 2am pages. One current gotcha worth flagging if you're on Blackwell. Drivers from 580.65.06 turn on Coherent Driver Memory Management by default for GB200 and GH200 on Kubernetes, and CDMM is incompatible with both MIG and GPUDirect Storage right now. If you buy GB200s planning to slice them with MIG, check that first, because it's not going to be on the datasheet. ## slicing one GPU three ways A single H100 has 80GB of HBM (its fast on-board memory) and enough compute to serve dozens of small models. Handing a whole card to a workload that uses 8% of it is how you set money on fire. There are three ways to share a GPU, and they are not interchangeable. The difference is isolation. ![A diagram showing one physical GPU sliced three ways side by side. Left: MIG, the GPU split into seven hardware partitions each with its own walled-off memory and compute, labeled 'hardware isolation'. Middle: MPS, multiple processes sharing one GPU context running concurrently, labeled 'concurrent, soft limits, weak isolation'. Right: time-slicing, several pods taking turns on the whole GPU in round-robin, labeled 'no isolation, round-robin'.](/images/gpu-deployments-part-1-anatomy/gpu-sharing-three-ways.png) *Fig. 2 · the same GPU shared three ways. Isolation drops as you move right; utilization convenience goes up. Pick by how much you trust the tenants.* **MIG** (Multi-Instance GPU) is hardware partitioning, available on the data-center cards (A100, A30, H100, H200, B200) but not the smaller inference GPUs like the L4 or A10G, which fall back to MPS. It cuts one GPU into up to seven instances, each with its own SMs (the GPU's compute cores), its own dedicated slice of HBM, its own memory controller and L2. A fault in one instance doesn't touch the others. That's real hardware isolation, the kind you want when tenants don't trust each other. The profiles are named `[compute]g.[memory]gb`. On an H100 80GB you get seven `1g.10gb` slices, or two `3g.40gb`, or one `7g.80gb` that's just the whole card back. An H200's 141GB gives you seven `1g.18gb` slices; a B200's 180GB gives seven `1g.23gb`. The quirk that catches everyone: compute fractions go 1/7, 2/7, 3/7, 4/7, 7/7. There is no 5g or 6g profile. Memory is quantized in eighths. So a `1g` slice gets one-seventh of the compute but one-eighth of the memory, and the arithmetic never quite lines up. **MPS** (Multi-Process Service) is a daemon that multiplexes several processes into one GPU context so their kernels run genuinely concurrently, not round-robin. You can cap each client's memory (`CUDA_MPS_PINNED_DEVICE_MEM_LIMIT`) and compute share (`CUDA_MPS_ACTIVE_THREAD_PERCENTAGE`). What you don't get is hardware memory protection or clean fault isolation. One client that OOMs can take its neighbors down with it. MPS is for high-throughput inference where you own all the tenants and want better SM utilization than time-slicing gives you. **Time-slicing** is the crude one. The device plugin just lies about the GPU count, advertising one physical card as ten replicas. Ten pods land, and they take turns via context switching. There is no memory partitioning and no fault isolation at all. If one pod grabs 70GB of an 80GB card, the other nine OOM. It's fine for notebooks, CI, and dev clusters where the work is bursty and nobody's SLA depends on it. It has no business in front of production traffic. | | Memory isolation | Fault isolation | Concurrency | Use for | |---|---|---|---|---| | MIG | hardware | hardware | spatial | multi-tenant, untrusted, prod serving | | MPS | soft caps | weak | true concurrent | trusted high-throughput inference | | Time-slicing | none | none | round-robin | dev, notebooks, CI | You can stack them: partition a card into seven MIG slices, then time-slice each slice for burstier workloads. Most teams don't need that. Most teams need to notice they're running one 8% workload per $30k card and switch to `7× 1g.10gb`. ## the metrics that lie to you Here's the number that ruins more capacity planning than any other: `GPU-Util`. When `nvidia-smi` shows 95% and everyone relaxes, they've misread it. `DCGM_FI_DEV_GPU_UTIL` answers exactly one question: was a kernel running during the sample window. It says nothing about how much of the silicon that kernel used. You can see 95% GPU utilization and 25% of the actual compute capacity in use at the same moment, and both numbers are honest about different things. Somebody will still screenshot the 95% into a capacity deck, and now you're being asked to buy more of a card you're already wasting. ![A horizontal bar comparison infographic titled 'the same GPU, two honest numbers'. Top bar shows GPU-Util at 95 percent filled nearly full in one color. Bottom bar shows Model FLOPs Utilization (MFU) at 40 percent, filled less than half in a contrasting color. A caption band notes that a well-run large training job lands at 35 to 50 percent MFU, and the gap between the two bars is where your money goes.](/images/gpu-deployments-part-1-anatomy/gpu-util-vs-mfu.png) *Fig. 3 · GPU-Util says a kernel ran. MFU says how much of the chip did useful work. The gap between them is memory stalls, collective communication, and non-matmul overhead you paid for anyway.* The number that matters for training is **MFU**, Model FLOPs Utilization: the FLOPs your model actually did divided by the theoretical peak. It's the metric the Llama and PaLM papers report, and 35–50% is considered excellent for large-scale training. The gap between "a kernel ran" and "the chip did useful math" is memory-bandwidth stalls, NCCL all-reduce waiting on the network, attention softmax and layernorm that aren't matmuls, optimizer steps, and activation recompute. All of it counts against your wall clock. None of it counts as useful FLOPs. The money follows directly. On an 8-GPU H100 node at roughly $3 per GPU-hour, the difference between 25% and 45% MFU is about 1.8× the effective cost per token. Sticker price per GPU-hour is the number vendors compete on. Utilization is the number that actually sets your bill. For real telemetry, DCGM (Data Center GPU Manager) is the layer, and `dcgm-exporter` scrapes it into Prometheus. The fields worth putting on a dashboard from day one: - `DCGM_FI_PROF_PIPE_TENSOR_ACTIVE`: Tensor Core utilization, the one that tracks real training throughput. - `DCGM_FI_PROF_DRAM_ACTIVE`: HBM bandwidth in use. High here with low tensor activity means you're memory-bound. - `DCGM_FI_DEV_FB_USED`: HBM used. Your OOM early-warning. - `DCGM_FI_DEV_XID_ERRORS`: the last Xid code. The single most important reliability signal on the box. - `DCGM_FI_DEV_ECC_DBE_VOL` and the row-remap fields: memory health, trending toward RMA. - `DCGM_FI_DEV_CLOCK_THROTTLE_REASONS`: a bitmask telling you whether the card is throttling on power, thermals, or a reliability limit. `dcgmi diag -r 3` runs about twelve minutes of escalating health checks (memory bandwidth, PCIe, NVLink, thermals under load) and is the thing to run before you trust a node you just recovered. ## the Xid codes worth memorizing Xid errors are the driver's way of telling the kernel log that the hardware had to correct or retry something it shouldn't have. They land in `dmesg` as `NVRM: Xid (PCI:0000:xx:00):

`. A nonzero Xid is not always fatal, but it's never nothing. A handful are worth knowing on sight because they change what you do next.

![A decision-flow diagram titled 'a GPU is misbehaving: what the Xid tells you to do'. It branches from a central node reading dmesg for the Xid code. Xid 13/31/43 routes to 'app bug, restart the workload'. Xid 48/94 routes to 'contained ECC, drain and reset the GPU'. Xid 95 routes to 'uncontained ECC, reset GPU before any restart'. Xid 63/64 routes to 'row-remapping, watch the trend, RMA if it fails'. Xid 79 routes to 'fell off the bus, cordon and reboot the node, RMA if it recurs'. Xid 119/120 routes to 'GSP firmware hung, reset GPU'.](/images/gpu-deployments-part-1-anatomy/xid-decision-flow.png)

*Fig. 4 · the Xid triage most on-call runbooks converge on. The split that matters is app-fault versus hardware-fault, because one restarts a pod and the other cordons a node.*

**Xid 13, 31, 43** are usually your fault, not the hardware's: illegal memory access, a bad kernel, a page fault from application code. Restart the workload, look at the model, not the card.

**Xid 48** is a double-bit ECC error, uncorrectable. **Xid 94** is a contained ECC error, where the damage stayed inside the offending app and the other apps on the card survived. **Xid 95** is the uncontained version, where the blast radius crossed apps, and the GPU has to be reset before anything restarts on it. The 94/95 split is the one to internalize: contained means drain politely, uncontained means the card is compromised until reset.

**Xid 63 and 64** are row-remapping events. Modern HBM can retire bad memory rows the way a disk retires bad sectors. A 63 is the card recording that it did this; persistent 64s (remap failures) mean it's running out of spare rows and it's an RMA candidate. Watch the trend, don't panic on the first one.

**Xid 79** is the one that ruins a night: "GPU has fallen off the bus." The card stopped answering on PCIe entirely. Thermal, power delivery, seating, a dying board. The node needs a reset, and if the same physical slot throws it again, that's a card headed back for RMA. Field reports put it around 3% of H100 deployments in the first year, which sounds small until you multiply it by a thousand-GPU fleet.

**Xid 119 and 120** are the GSP firmware timing out. Reset the GPU. On a few driver versions it's common enough that ops teams disable the GSP firmware as a workaround, which tells you how much fun that particular bug is.

The remediation ladder most teams settle on is boring and effective: app restart or driver reload or node reboot clears roughly 60% of incidents within fifteen minutes; anything that survives that gets `dcgmi diag -r 3`; anything that fails the diag gets cordoned and sent back.

## what to actually care about

If you're standing up GPU infrastructure and wondering where to spend your attention, the honest ranking isn't the one the marketing implies. It's roughly this. Get the driver-and-toolkit layer boringly stable, because that's where the self-inflicted outages live. Instrument MFU and the Xid stream before you instrument anything pretty, because those two tell you whether you're wasting money and whether the hardware is dying. Decide your sharing model (MIG for multi-tenant, whole cards for training) before you have tenants, because retrofitting isolation is miserable. And treat the twelve-layer stack as the thing it is: a place where a green dashboard at layer 9 can sit directly on top of a card that fell off the bus at layer 1.

This is also usually the point where teams bring in someone who has already burned a few weeks on Xid codes and MIG partitioning, rather than doing it live on their own GPU bill. GPU and ML infrastructure builds are part of the [independent infrastructure consulting](https://k8s.org.in) work I take on.

The pod that was stuck Pending at the top of this post came back the moment the device plugin re-registered. Fifteen seconds of fix, forty minutes of staring at a healthy `nvidia-smi` wondering how the machine could see eight GPUs that Kubernetes swore didn't exist. That distance, between what the hardware knows and what the scheduler believes, is where most of your on-call rotation lives too. The next part goes inside a single box with eight of these cards and the wires that decide whether they cooperate or just sit next to each other.



The git commands I actually run every day
root@harshit.cloud (Harshit Luthra) — Wed, 20 May 2026 00:00:00 GMT

![A newspaper-style poster titled The Daily Eight, listing the eight git aliases I run most: gst, glola, gd/gds, gcam, gpsup, gco/gcb, gpf, gfa, each paired with its expansion in mono caps.](/images/daily-git-commands/hero.png)

*Fig. 1 · the eight aliases that survive every refactor, every job, every laptop.*

I've been using git for a decade and most of what I type still fits on a single hand. The 200-page Pro Git book is wonderful and almost none of it survives contact with a real Tuesday. What survives is a small, boring set of commands that get rerun constantly.

This post is that list, ordered by how often my fingers actually type them. Aliases are from the oh-my-zsh `git` plugin (enabled in most zsh configs that exist); the full command sits next to the alias so it's portable.

## the daily eight

These are the ones I'd type in my sleep. If you're not using all eight already, picking them up pays back inside a week.

### gst
*git status*

```bash
gst
```

I run this between every other command. It's the cheapest sanity check git has. Branch, ahead/behind, staged, unstaged, untracked. Two seconds. If you only learn one alias, learn this one.

### glola
*git log --oneline --graph --decorate --all*

```bash
glola | head -30
```

The one true log. Graph of every branch (local + remote), one line per commit, colored refs. Pipe through `head` because most of the time you only care about the last 20-30 commits.

### gd / gds
*git diff / git diff --staged*

```bash
gd          # what's changed but not staged
gds         # what's staged and about to be committed
```

`gds` before every commit. If you set [delta](https://github.com/dandavison/delta) as your pager (`brew install git-delta`, then `pager = delta` in `~/.gitconfig`), the output stops being painful to read.

### gcam
*git commit -a -m*

```bash
gcam "fix: trailing slash in webhook URL"
```

Quick one-line commits for small fixes. For anything bigger I drop the `-m` and let `$EDITOR` open so I can write a proper message with a body.

### gpsup
*git push --set-upstream origin \*

```bash
gpsup
```

First push of a new branch. The full command is annoying to type, so `gpsup` figures out the current branch name itself. After the first push, plain `gp` (just `git push`) works because upstream is set.

### gco / gcb
*git checkout / git checkout -b*

```bash
gco main             # switch to main
gco -                # switch to previous branch
gcb feature/login    # create + switch to new branch
```

`gco -` is the one to notice. Like `cd -` for branches. When you're bouncing between two branches all day, it's a single keystroke each way instead of typing the name.

### gpf
*git push --force-with-lease*

```bash
gpf
```

After rebasing or amending. **Always use `--force-with-lease`, never `--force`.** The lease version refuses to push if someone else has pushed to your branch since your last fetch, saving you from silently overwriting a teammate's work. There is no good reason to ever type `--force` in 2026.

### gfa
*git fetch --all --prune*

```bash
gfa
```

Refresh every remote, prune deleted remote branches. Run before you start anything that depends on knowing the current state of the world. The `--prune` half is what makes the cleanup ritual below work.

## checkout recent branches

`git branch` lists alphabetically, which is useless. What you actually want is "that branch from Tuesday," which means sorting by last commit:

```bash
git config --global alias.recent \
  "for-each-ref --sort=-committerdate refs/heads/ \
   --format='%(HEAD) %(color:yellow)%(refname:short)%(color:reset) \
             %(color:green)(%(committerdate:relative))%(color:reset) %(contents:subject)'"

git recent | head -10
```

That covers looking. For switching, pipe the same list into fzf and you never type a branch name again:

```zsh
# fco: fuzzy-checkout a recent branch
fco() {
  local branch
  branch=$(git for-each-ref --sort=-committerdate refs/heads/ \
             --format='%(refname:short)' \
           | fzf --height 40% --reverse \
                 --preview 'git log --oneline --decorate --color=always -15 {}')
  [ -n "$branch" ] && git checkout "$branch"
}
```

Branches arrive sorted by recency, so the one you want is almost always in the top three. Type two letters of its name, Enter, done. The preview pane shows the branch's recent commits so you can confirm it's the right Tuesday. `gco -` still wins for bouncing between exactly two branches; `fco` wins for everything else. (`brew install fzf` if you don't have it. You want it for `Ctrl-R` history search anyway.)

## the cleanup ritual

Run this weekly. If you've ever scrolled through 80 stale branches looking for the one you actually want, you already know why.

The easy half deletes every local branch whose tip is already in `main`:

```bash
gfa
git branch --merged main | grep -v '\*\|main\|master' | xargs -n1 git branch -d
```

Works only if your team uses merge commits. Most don't. GitHub's "Squash and merge" creates a brand-new commit on `main` with a different SHA, so `git branch --merged` never catches your local branch. Its commits aren't in main's history at all.

The workaround: after `gfa`, any branch whose tracked remote was deleted shows as `[gone]`. Those are *usually* your merged-and-deleted PRs.

Usually, not always. `[gone]` only means the remote tracking branch is gone. Nearly always that's a squash-merged PR whose branch GitHub auto-deleted. But it can also be a branch you pushed, someone deleted server-side, and you never merged. So don't force-delete every `[gone]` branch with `git branch -D`. I once watched one show `[gone]` while it still held 26 unmerged commits; a force-delete there loses them for good.

So check each `[gone]` branch for patch-equivalence against the base *before* deleting. Squash-merges get caught, genuinely unmerged work gets kept. This lives in my `~/.gitconfig` as `git gone`:

```bash
# ~/.gitconfig, under [alias]  →  run as: git gone
gone = "!f() { \
    git fetch --all --prune; \
    base=$(git rev-parse --abbrev-ref origin/HEAD 2>/dev/null); base=${base:-origin/main}; \
    for b in $(git for-each-ref --format='%(refname:short) %(upstream:track)' refs/heads \
               | awk '$2==\"[gone]\"{print $1}'); do \
      if [ -z \"$(git cherry \"$base\" \"$b\" | grep '^+')\" ]; then git branch -D \"$b\"; \
      else echo \"kept $b (commits not in $base)\"; fi; \
    done; }; f"
```

One command does the whole ritual: the `git fetch --all --prune` prunes the dead remote refs, then the loop deletes the merged local branches in the same pass. No separate `gfa` first.

`git cherry` compares by patch-id, not SHA. A squash-merged branch shows every commit as `-` (an equivalent already exists in the base) and gets deleted; a branch with real unpushed work shows `+` lines and stays. The `-D` is only reached after patch-equivalence is proven, so it never eats unmerged work.

Or install [`git-trim`](https://github.com/foriequal0/git-trim) (`brew install git-trim`), which does the same classification and more. It catches squash-merges even when the tracking ref isn't `[gone]`, and skips diverged branches by default.

```bash
git trim                # dry-run
git trim --confirm      # actually delete
```

This is the closest thing to "did my PR ship?" you can ask git directly.

## the weekly four

Not in your fingers yet, but should be.

### `git switch` and `git restore`

```bash
git switch -c new-feature           # create + switch
git restore --staged file.txt       # unstage
git restore --source=abc123 file.go # restore single file from any commit
```

`switch` and `restore` split the four jobs `checkout` used to do. The one I reach for most is `restore --source= `. Translation: "grab this single file from three commits ago without touching anything else."

### interactive rebase with autosquash

```bash
git commit --fixup=abc123       # fixup commit targeting abc123
# ... keep working ...
git rebase -i --autosquash main # all fixups slot into place automatically
```

The single biggest workflow win I've found in ten years of git. While reviewing your own PR you find a bug four commits back. Don't fix it on top. `--fixup=` creates a commit targeting the offender, and the autosquash rebase squashes everything into place when you're done. Install [git-absorb](https://github.com/tummychow/git-absorb) (`brew install git-absorb`) and it even picks the target SHA for you: edit the files, run `git absorb --and-rebase`, done.

### `git reflog`, the universal undo

```bash
git reflog
git reset --hard HEAD@{5}
```

Every change to `HEAD` is logged for 90 days. Bad rebase? `reflog`. Deleted branch? `reflog`. There is almost nothing in git you can't undo if you know about it.

### `git worktree`

```bash
git worktree add ../proj-hotfix hotfix/prod-down
git worktree remove ../proj-hotfix
```

Need to fix a prod bug while halfway through a feature? Don't stash. `worktree add` gives you a second checkout in a sibling directory, sharing the same `.git`. I use it constantly for "let me review your PR" without leaving my own branch.

## set it once

Five config lines and a daemon. Enable, forget.

```bash
git config --global rerere.enabled true          # remember conflict resolutions, replay them
git config --global push.default current         # `git push` pushes current branch to same name
git config --global push.autoSetupRemote true    # first push sets upstream automatically
git config --global diff.algorithm histogram     # cleaner diffs than the default myers
git config --global merge.conflictStyle zdiff3   # conflict markers include the common ancestor
git maintenance start                            # background gc/prefetch on a schedule
```

`autoSetupRemote` retires `gpsup` entirely. `zdiff3` shows the original code both sides diverged from; once you've used it, plain `<<<<<<<` markers feel like flying blind.

## when something is broken

Not daily, but when the question is "when did this start," nothing else answers it:

```bash
git log -S "functionName"          # pickaxe: commits where this string was added or removed
git blame -w -C -C -C file.go      # blame the logic's actual author, not the formatter
git log -p --follow file.go        # full file history, including across renames
git range-diff @{u} @              # what a rebase actually changed; run before force-pushing
```

`-S` searches the content of the diff, not commit messages. Different thing entirely from `--grep`. And plain `blame` gives credit to whoever last ran Prettier; `-w -C -C -C` follows the code across whitespace changes, moves, and file boundaries to the person who wrote the logic.

## the four tools worth installing today

- **[fzf](https://github.com/junegunn/fzf)** (`brew install fzf`). Powers the `fco` branch picker above, plus fuzzy `Ctrl-R` history.
- **[git-absorb](https://github.com/tummychow/git-absorb)** (`brew install git-absorb`). Auto-fixup commits without picking SHAs.
- **[delta](https://github.com/dandavison/delta)** (`brew install git-delta`). Diff and blame output that doesn't hurt to look at.
- **[lazygit](https://github.com/jesseduffield/lazygit)** (`brew install lazygit`). TUI for the operations that are tedious on CLI: partial commits, stash management, conflict resolution.

This post used to end with two AI shell helpers for the stuff git can't tell you; those now live in [their own TIL](/til/one-line-ai-shell-helper).

## ten years in, the surprise

After a decade, the command I run most isn't `commit`. It isn't `push`. It's `gst`, hundreds of times a day, between every other operation. The most-used git command in my shell is the one that does nothing.



A single zsh function for one-line AI answers that knows when to pre-type the command
root@harshit.cloud (Harshit Luthra) — Wed, 20 May 2026 00:00:00 GMT

I kept opening a chat tab just to ask "what's the kubectl command for decoding a secret" or "convert 42 GiB to bytes". The context switch was costing more than the answer was worth.

Wrapping an AI CLI into a single shell function fixed it. The interesting part is `print -z`, plus one heuristic that needs more care than it looks.

## the function

```zsh
# p: one-shot AI query. Examples: `p whats 2 + 2`, `p kubectl secret decode grafana`
# Smart dispatch: if the answer looks like a runnable command, pre-type it into
# the next prompt (print -z). Otherwise print to stdout. Math/facts get printed,
# commands get queued for you to review and press Enter.
p() {
  emulate -L zsh
  setopt NO_GLOB
  if [ $# -eq 0 ]; then
    echo "usage: p " >&2
    return 1
  fi
  local out
  out=$(pi -p --no-session --append-system-prompt 'Answer in ONE line. No preamble, no explanation, no markdown, no code fences. For shell/kubectl/git/etc requests output only the command. For factual or math questions output only the answer.' "$*" \
        | tr -d '\000-\037' \
        | sed 's/^[[:space:]]*//;s/[[:space:]]*$//')
  if [ -z "$out" ]; then
    return 1
  fi
  local first="${out%% *}"
  if [[ "$first" == [a-zA-Z_]* ]] && whence -p "$first" >/dev/null 2>&1; then
    print -z -- "$out"
  else
    print -r -- "$out"
  fi
}
alias p='noglob p'
```

`pi` is just whatever AI CLI you have. Swap in `claude -p`, `llm`, `gh copilot suggest`, `ollama run`. The pattern doesn't care about the backend.

## what it feels like

```
$ p whats 2 + 2
4

$ p capital of mongolia
Ulaanbaatar

$ p regex for matching an email
[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}

$ p kubectl secret decode grafana
# next prompt now shows, cursor at the end:
$ kubectl get secret grafana -o go-template='{{range $k,$v := .data}}{{$k}}: {{$v | base64decode}}{{"\n"}}{{end}}'█

$ p find all log files modified today
# next prompt:
$ find . -type f -name "*.log" -mtime -1█
```

Same two-letter command for both. Answers go to stdout, commands go to the prompt buffer where you can edit them before pressing Enter.

## the key idea: `print -z` for runnable output

`print -z` is the trick that makes this design work. It pushes text onto the zsh line editor, i.e. into your next prompt, pre-typed and ready. Compared to every alternative:

| Strategy | Speed | Safety | Friction |
|----------|-------|--------|----------|
| `eval "$(...)"` | fastest | **bad**, auto-runs model output | none |
| Pipe to `pbcopy` | medium | safe | switch focus, paste |
| Print to stdout | medium | safe | select + copy + paste |
| **`print -z`** | **fastest** | **safe**, you press Enter | **none** |

The mental model: `print -z` is what `Ctrl-R` history search does when you accept a result. Native zsh. You always see and approve the command before it runs.

## the heuristic: when is the answer a command?

The smart dispatch decides between `print -z` (pre-type) and `print -r` (stdout) by looking at the first word of the answer:

```zsh
if [[ "$first" == [a-zA-Z_]* ]] && whence -p "$first" >/dev/null 2>&1; then
  print -z -- "$out"
else
  print -r -- "$out"
fi
```

Two checks, both load-bearing:

1. **First char is a letter or underscore.** Excludes digits (`4`), symbols (`[`, `/`, `(`), and anything else that obviously isn't a command name.
2. **`whence -p` resolves it to a PATH executable.** Not just "this name exists in the shell", but *specifically* a real binary on disk.

Why `whence -p` and not `command -v`? Read on.

## the footgun: oh-my-zsh numeric aliases

My first attempt used `command -v "$first"` as the heuristic. It looked right. It failed in a way that took a minute to spot.

When I ran `p whats 2 + 2`, the answer was `4`, but nothing appeared in my terminal. The function exited cleanly with status 0. No error.

What had happened: oh-my-zsh's `dirhistory` plugin (loaded by default in many configs) aliases `1` through `9` to `cd -1` ... `cd -9` for jumping around the directory stack. So `command -v 4` returned true. `4` was a recognized alias, and the function tried to `print -z 4` into my prompt buffer.

In a real interactive shell, that would have stuffed `4` into my prompt invisibly (it'd appear when I hit Enter). In my non-interactive test (`zsh -ic '...'`) it disappeared into the void because there's no line editor to render the stuffed buffer.

The fix has two parts:

- **`[[ "$first" == [a-zA-Z_]* ]]`** alone would have caught it, because `4` doesn't start with a letter.
- **`whence -p`** instead of `command -v` makes it doubly safe. `whence -p` only matches binaries in PATH, ignoring aliases, functions, and builtins. Aliases like `4 → cd -4` are filtered out.

Either check alone would have caught the bug. Having both means the next time I add a feature here, I don't have to remember which one was load-bearing.

## defensive details that earn their keep

Three small things prevent subtle bugs:

### `noglob` on the alias

```zsh
alias p='noglob p'
```

Without this, `p list all *.log files` would have zsh expand `*.log` against the current directory *before* the function ever sees it. With `noglob`, the glob characters pass through literally. Same trick git uses for its arguments.

### `emulate -L zsh` + `setopt NO_GLOB`

```zsh
emulate -L zsh
setopt NO_GLOB
```

`emulate -L zsh` resets shell options to defaults, scoped to this function only (the `-L` means local, so they restore on return). `NO_GLOB` is belt-and-suspenders for callers that bypass the alias (`command p ...`, `\p ...`, or scripts that don't see your aliases).

### output sanitization

```zsh
tr -d '\000-\037' | sed 's/^[[:space:]]*//;s/[[:space:]]*$//'
```

`tr -d '\000-\037'` strips all C0 control characters. That includes ANSI escape sequences (ESC = `\033`), stray nulls, and any invisible cruft the model might emit. Critical for `print -z` because control characters in the payload corrupt the line editor's display.

`sed` then trims leading and trailing whitespace, which the model usually adds even when told not to.

## why `"$*"` and not `"$@"`

`"$*"` joins all positional args into one string with spaces between them. `"$@"` would pass them as separate args, which most AI CLIs would concatenate anyway, but some treat the first positional as the prompt and the rest as files (the `@file.txt` convention is common). Joining explicitly avoids that ambiguity.

If your CLI supports `--` to end option parsing, prefer:

```zsh
your-ai-cli -p ... -- "$*"
```

`pi` doesn't accept `--`, hence the bare `"$*"`.

## the system-prompt nudge actually matters

Without `--append-system-prompt`, even with `-p`, the default coding-assistant prompt wraps shell commands in code fences and adds a one-sentence intro. That breaks `print -z` (code fences are not commands) and clutters the terminal.

The phrasing that worked best:

> Answer in ONE line. No preamble, no explanation, no markdown, no code fences. For shell/kubectl/git/etc requests output only the command. For factual or math questions output only the answer.

"No markdown, no code fences" is doing most of the work. Without it you get backtick-wrapped output that `print -z` would happily push into your prompt as `` `kubectl get pods` ``, which is not a runnable command.

## why this beats the chat UI for short questions

| Action | Chat UI | `p` |
|--------|---------|-----|
| Switch context | yes | no |
| Round-trip latency | ~3-5s + UI | ~1-2s |
| Output format | markdown, prose | bare answer or pre-typed command |
| Get command into shell | select + copy + paste | already in your prompt |
| Session pollution | yes | no (`--no-session`) |
| Glob-expansion footgun | n/a | guarded (`noglob`) |

For anything longer than a paragraph the chat UI is still better. For "what's the syntax for X" or "the command for Y", the terminal is the right place to put the answer.

## the one substitution that fixed it

`command -v` → `whence -p`. One swap. The rest of the function (the `noglob`, the `emulate -L zsh`, the control-char strip) was already doing its job. The bug was trusting that "this name resolves in the shell" meant "this name is a binary on disk." It doesn't, and on any zsh with oh-my-zsh loaded it especially doesn't.



Seven visual tools, one diagram
Fri, 15 May 2026 00:00:00 GMT

I write more than I draw, and the drawing is always the part of a post that takes longest. A thousand words land in an evening. One architecture diagram I'm happy with can eat a whole afternoon. Excalidraw is fast, and it's good, but if you've published with Excalidraw for any length of time you start to notice that everything you make in it looks like everything else made in it. The hand-drawn aesthetic stops being a personality and turns into a tell.

What I actually want is a visual that has some attitude. The kind of diagram people screenshot and put in slides. Something a reader pauses on for a beat before scrolling into the prose. Most of my posts are about infrastructure, so the visuals tend to be architecture-y, but the bar I'm chasing is closer to a magazine illustration than to a whiteboard photo.

A few months ago Claude Code started shipping a marketplace of plugins and skills, and several of them claim to draw exactly this kind of thing. I'd been meaning to test them and kept putting it off. So I sat down one weekend, picked a representative diagram, and ran the same brief through every tool I could find. Then I evaluated them side by side and kept the ones I'd actually reach for.

This is what I learned.

## the diagram I picked

The test case was a fairly busy multi-region Kubernetes setup. Three EKS clusters in three regions, with one of them acting as an ArgoCD hub doing App-of-Apps sync to the other two. Karpenter handling compute on each cluster, KEDA scaling one specific workload, and a unified observability stack feeding into a single account that pages out to an on-call rotation.

It's not a toy. There's enough going on that a half-baked tool would visibly fall apart, and enough structure that a well-designed tool would have something to organize.

## the lineup

| # | Tool | What it is |
|---|---|---|
| 1 | `Cocoon-AI/architecture-diagram-generator` | A claude.ai web skill |
| 2 | `cathrynlavery/diagram-design` | A Claude Code plugin built around a design system |
| 3 | `edlebertf/claude-infographic-gif` | A Claude Code skill for animated GIF infographics |
| 4 | `claudekit/frontend-design-pro-demo` | A Claude Code plugin for frontend interfaces |
| 5 | Vercel `ai-cli` + `bfl/flux-2-pro` | An image-gen CLI talking to Vercel's AI Gateway |
| 6 | A hand-coded SVG | The control |
| 7 | Excalidraw | The thing I was trying to graduate from |

Installing them was less smooth than I expected. The shell command I tried first, `claude plugin add `, doesn't exist. Claude Code's plugin system runs inside the TUI via `/plugin marketplace add` and `/plugin install`, or you `git clone` raw skills into `~/.claude/skills//`. The Cocoon entry on that list isn't a Claude Code plugin at all; it's a claude.ai web skill that you upload as a zip in the browser. I gave up on installing it from the CLI and drew its equivalent by hand instead, which means my Trial 1 below is more "Claude doing AWS re:Invent" than "the Cocoon skill doing what it does". For an apples-to-apples Trial 1 you'd have to upload the actual zip at claude.ai.

## trial 1: the AWS re:Invent stand-in

![AWS re:Invent dashboard showing three cluster cards with category-coded service rows](/images/seven-visual-tools-one-diagram/trial-1-cocoon.png)

[Open the live version →](/images/seven-visual-tools-one-diagram/trial-1-cocoon.html)

I went with the AWS re:Invent aesthetic because anyone who's watched a cloud keynote in the last ten years will recognize it instantly. Three cluster cards in a row, each with the full service stack listed inside (ArgoCD, Karpenter, KEDA, workloads), category-coded by AWS's own palette (orange for compute, purple for networking, green for observability, pink for GitOps), and an observability strip at the bottom tying agents to the on-call system.

The output is dense. It's the kind of thing I'd use as an inline reference figure in a long post where the prose already explains each piece. If I dropped it as a hero, it'd be too much.

## trial 2: diagram-design

![Editorial dark page with hub-spoke diagram, coral focal hub, muted spokes, italic-serif annotation, and three summary cards](/images/seven-visual-tools-one-diagram/trial-2-diagram-design.png)

[Open the live version →](/images/seven-visual-tools-one-diagram/trial-2-diagram-design.html)

This is the one I'd actually publish without retouching. The skill has a real design system baked in: one focal node in coral (and only one), two muted spokes, italic-serif annotation off to the side, summary cards underneath with varied widths, hairline borders, no shadows anywhere. The skill's own instructions are pretty firm about it, even setting a target density of 4 out of 10 and telling itself to delete anything that doesn't earn its place. You can feel that restraint in the result.

It looks like it came out of someone who's spent a long time thinking about diagrams as a form, not from a tool that randomly throws boxes on a page.

## trial 3: infographic-gif

![Animated sankey diagram showing flows from a hub through three clusters to three resource categories](/images/seven-visual-tools-one-diagram/trial-3-infographic-gif.gif)

[Open the live version →](/images/seven-visual-tools-one-diagram/trial-3-infographic-gif.html)

This one surprised me. The output it gives you is an HTML file, not a GIF. You open the HTML in a browser, it canvas-renders frames for a few seconds, and then a "Download GIF" button appears. So the actual deliverable is two clicks away from what the skill spits out. Once you watch it animating though, it's properly satisfying. Mine became a three-stage sankey: hub on the left, three clusters in the middle, three aggregated resource buckets on the right, with bezier curves drawing in left to right.

One thing to know if you try this: the skill's sankey template is fundamentally two-level. Sources on the left, destinations on the right, with one center node. I had three layers, so the per-cluster resource breakdown got merged into single resource totals. The proportions are honest, the per-cluster detail isn't quite there. For an animated explainer where motion is the point, that's an acceptable trade.

## trial 4: frontend-design-pro

![Cyberpunk-styled cluster status panel with neon cyan and magenta accents and CRT scanlines](/images/seven-visual-tools-one-diagram/trial-4-cyberpunk.png)

[Open the live version →](/images/seven-visual-tools-one-diagram/trial-4-cyberpunk.html)

I asked for a "live cluster status panel" in a cyberpunk aesthetic and got exactly that. CRT scanlines, VT323 for the chunky CRT-display digits on the node counts, neon accents, a magenta pulse on the panel that's marked `syncing`. A tiny script ticks the relative timestamps every second so the thing feels alive.

In isolation it looks great. Embedded in a sober technical post though, it would scream at the reader and you'd end up designing the rest of the post around the widget. So I went back to the same skill with a tighter brief to see if it could play in a calmer register.

### same skill, second pass

I asked it to draw the request lifecycle through an Istio ingress. Seven hops, client through NLB through gateway and onwards to a pod, with a return path back. Two deliverables: an animated HTML diagram, and a Flux image prompt for the static raster version.

![Dark OLED Luxury full-page request lifecycle diagram with seven components and animated packets](/images/seven-visual-tools-one-diagram/trial-4b-ingress-flow-animated.png)

[Open the live version →](/images/seven-visual-tools-one-diagram/trial-4b-ingress-flow-animated.html)

First attempt came back as a dashboard again. Big hero title, live telemetry chrome on top (latency, request rate, success percentage), seven custom component glyphs in a row, emerald packets going forward and amber packets coming back along a parallel return wire. It's a beautiful page, but it's still very much a page. If I tried to inline this inside another post, it would take over.

So I asked again. Same skill, third pass, this time framed as "a figure for the body of a blog post, not a dashboard":

![Tight figure-style version at 760px wide with figcaption](/images/seven-visual-tools-one-diagram/trial-4b-ingress-flow-blog.png)

[Open the live version →](/images/seven-visual-tools-one-diagram/trial-4b-ingress-flow-blog.html)

This is the version that goes inside a post. 760px wide, a real `` with `Fig. 1`, the diagram, and a real `` mixing a sentence of prose context with an inline legend. Self-contained dark so it doesn't fight whatever theme surrounds it. The skill gets to the right answer once you tell it the right question.

And the Flux raster, prompt written by the skill, rendered by `ai-cli`:

![Cinematic isometric 3D illustration of seven architectural artifacts connected by a glowing data conduit](/images/seven-visual-tools-one-diagram/trial-4b-flux-illustration.png)

Eighteen seconds of generation, zero human prompt-engineering on top. The skill's prompt was the work.

## trial 5: Vercel ai-cli + Flux

![Isometric scene with three floating clusters connected to a central ArgoCD hub via glowing bridges](/images/seven-visual-tools-one-diagram/trial-5-flux-cover.png)

This is the one I'd use as a section header or a post cover. Flux 2 Pro through Vercel's AI Gateway, twelve seconds to render, a single one-line prompt. It's not replacing a technical figure (you can't actually point at anything in it), but as visual atmosphere at the top of a post, it carries weight. One thing to watch: the model ignored my `--size 1200x630` flag and inferred an aspect ratio instead. The CLI surfaced a warning about it, which I only noticed by accident. Check your output dimensions before you cut it into a layout.

## trial 6: hand-coded SVG

![Hand-coded SVG architecture diagram with three VPC boxes and color-coded legend](/images/seven-visual-tools-one-diagram/trial-6-native-svg.svg)

The baseline. AWS-orange-on-navy, three boxes in a row, dashed hub-spoke arrows, observability strip at the bottom, color-coded legend. 12 KB, no JS, no fonts, no dependencies. Loads instantly, scales infinitely, and is trivial to edit when the architecture inevitably changes. Inside a long post where you need a quick reference and you don't want to fight CSS, this is more than enough. It's also the least visually interesting of anything in the lineup, which is either a feature or a bug depending on what the post is doing.

## trial 7: Excalidraw (via the official connector)

![Hub-spoke architecture diagram from the Excalidraw connector: yellow hub cluster with nested ArgoCD callout, two blue spoke clusters with argocd-agent sub-boxes, and a green observability strip at the bottom with separate New Relic and Zenduty boxes connected by an "alerts" flow](/images/seven-visual-tools-one-diagram/trial-7-excalidraw.png)

[Download the `.excalidraw` source →](/images/seven-visual-tools-one-diagram/trial-7-excalidraw.excalidraw) · [View as SVG →](/images/seven-visual-tools-one-diagram/trial-7-excalidraw.svg)

Excalidraw is what I'd been using before this whole exercise, and it's the thing I was originally trying to graduate from. There's an official Excalidraw connector in the Claude.ai directory, so I dropped the same brief into it. It took the connector about four minutes to produce this.

The surprising thing isn't the layout. It's the aesthetic. The connector deliberately chose not to look like Excalidraw. The strokes are clean and straight, no rough.js wobble. The typography is sans-serif, not the Virgil hand-drawn default. Everything is grid-aligned, with nested sub-components: `argocd-agent (receiver)` boxes inside each spoke, an `argocd-agent (principal)` callout inside the hub, separate `New Relic` and `Zenduty` boxes inside the observability strip, an `alerts` flow connecting them. It reads as "architecture diagram drawn in Excalidraw" rather than "Excalidraw scratchwork".

That changes my read of Trial 7 in a way I didn't expect. The connector quietly opted out of the genre I'd complained about in the opening of this post. The casual hand-drawn aesthetic is one register of Excalidraw now, not the default Excalidraw. The canvas can be a sketchpad or a structured diagram tool, depending on how the scene is authored. The connector chose the structured side.

For the kind of visual I'd actually publish, the polished register it picked is closer to what I'd ship than the wobbly default would have been. If you specifically want the wobble, you'd have to ask for it.

(Small caveat on the rendering: the PNG above is exported from an SVG approximation, not from Excalidraw's own engine, so the strokes and font are close-but-not-identical to a canonical Excalidraw export. Open the `.excalidraw` source in Excalidraw and use `File → Export → PNG (2×)` for the official render. The difference is subtle.)

## what I'd reach for, in practice

For inline figures inside a long-form technical post, `diagram-design` won by a wide margin. The design system has real taste in it, the output looks publication-quality without me retouching anything, and the constraints (one focal node, no shadows, hairlines) keep the diagrams from feeling busy. This is the one I'll be using most.

For animated explainers, `infographic-gif` is decent when the motion actually adds information. The HTML-not-GIF artifact is mildly annoying but not a dealbreaker. If you need animated HTML where you control the design tightly, `frontend-design-pro` is more flexible, but you have to brief it carefully. Ask for a "live cluster panel" and you'll get a dashboard. Ask for "a figure to embed in the body of a blog post" and you'll get a figure. The skill responds to scope cues.

For hero images or section openers where the goal is mood rather than information, Flux through `ai-cli` is hard to beat at twelve seconds and one prompt. The cost per image is low enough that you can iterate a few times until something lands.

For quick reference diagrams inside a post, hand-coded SVG is still where I end up. It costs nothing to maintain, ages well, and gets out of the reader's way. Not every diagram needs to be a hero.

For the first draft, Excalidraw the app, still the fastest way to figure out where the boxes go in five minutes. For a ready-to-ship version with the Excalidraw canvas behind it but without the genre aesthetic, the Claude connector turned out to be a genuine option I hadn't expected to like.

## keeping all of this from eating your context

One thing I noticed after installing six different visual-generation plugins: every skill they register goes into the Claude Code system prompt at session start. Whether you use them or not, they're paying rent in your context window. Six skills isn't a lot in absolute terms, but if you add up everything else you've installed (MCP servers, hook configs, agent definitions), sessions start to feel heavier than they used to.

There are a few commands that aren't well-advertised but are exactly what you want:

```bash
claude plugin list             # what's installed, at which scope, enabled or disabled
claude plugin details    # the actual skills, hooks, MCP servers this plugin loads
                               # plus its projected token cost
claude plugin disable    # turn it off
claude plugin enable     # bring it back
```

The one I'd specifically call out is `details`. It tells you exactly what a plugin pulls into your session, and the rough token cost of having it loaded. If you feel like your context budget vanishes before you've done much, this is where to look first.

There is one wrinkle: enable and disable don't take effect mid-session. The skill set is fixed when Claude Code starts up, so toggling a plugin only changes the next session. In practice this turned into a workflow change for me. I keep most of these visual-generation plugins disabled by default at the user scope, install them at the project scope of my blog repo, and let them auto-activate when I `cd` into the blog. Outside of the blog, the context stays lean.

The setup is two commands per plugin:

```bash
# globally disabled
claude plugin disable --scope user diagram-design
claude plugin disable --scope user frontend-design-pro

# from inside the blog repo
claude plugin install --scope project diagram-design@diagram-design
claude plugin install --scope project frontend-design-pro@frontend-design-pro
```

After that, only sessions started inside the blog directory see those skills. Everywhere else, they're not loaded, not adding to the prompt, not costing me tokens. When I want them back globally I can enable them again.

## what I'm using now

Three tools and a fallback. `diagram-design` for any inline schematic. `frontend-design-pro`, in its blog-trimmed form, for animated or live-feeling diagrams once I've told it to behave like a figure. Flux through `ai-cli` for cover images and section openers. Hand-coded SVG when I just need a fast reference and don't want to think about it. The evaluation took most of a day. The diagrams take a few minutes each from here.

If you're considering doing something similar, the meta-lesson I'd offer is to pick one test diagram that's representative of what you'll actually publish, and then run every candidate against the same brief. The differences between these tools are not subtle, but they're not obvious from the marketing either. You see them when the same input comes out in six different ways.



Lazy SRE's guide to secure systems, part 6: the network in front of everything
root@harshit.cloud (Harshit Luthra) — Sun, 10 May 2026 00:00:00 GMT

In January 2024, Ivanti disclosed two CVEs in their Connect Secure and Policy Secure VPN appliances. CVE-2023-46805 was an authentication bypass. CVE-2024-21887 was an unauthenticated command injection: a remote shell on a box that, by design, had to be reachable from the public internet. Mandiant attributed the in-the-wild exploitation to UNC5221, a suspected Chinese state-sponsored cluster. By the time Ivanti shipped a patch, Mandiant had identified more than a thousand compromised appliances. CISA issued Emergency Directive 24-01 telling every U.S. federal agency to take their Ivanti boxes offline.

This is part 6. Earlier parts covered npm ([Part 1](/blog/lazy-security-part-1-supply-chain)), GitHub Actions ([Part 2](/blog/lazy-security-part-2-github-actions)), the unsexy infrastructure list ([Part 3](/blog/lazy-security-part-3-unsexy-list)), DNS auth records ([Part 4](/blog/lazy-security-part-4-dns-records)), and the dev laptop perimeter ([Part 5](/blog/lazy-security-part-5-dev-laptops)). Part 6 is the network in front of everything. What sits between your engineers and prod, and between prod and the public internet.

The thesis from Part 1 stands. Future You at 3am will not patch a VPN concentrator the same week the CVE lands, especially when the vendor patch breaks LDAP for half the team. The architecture that makes the concentrator irrelevant is the one that runs while you sleep: a mesh network where there is no internet-facing appliance to compromise in the first place.

## the VPN appliance is the attack surface

The Ivanti CVEs are not a unique event. They're the most recent member of a class. The same year (2024) saw:

- **Cisco ASA / FTD**: CVE-2024-20353 + CVE-2024-20359. A web-services DoS and a persistent local code-execution flaw used together by the ArcaneDoor campaign (Line Dancer / Line Runner implants), attributed by Cisco Talos to state-sponsored actors. April 2024.
- **Citrix NetScaler "CitrixBleed"**: CVE-2023-4966. A session-token leak via memory disclosure, exploited by LockBit and others, used in the Boeing and Comcast breaches.
- **Fortinet FortiOS SSL VPN**: CVE-2024-21762. Out-of-bounds write in February 2024, exploited in the wild before patches were widely deployed.
- **Palo Alto Networks GlobalProtect**: CVE-2024-3400. Command injection in April 2024, exploited in a campaign Palo Alto Networks Unit 42 named Operation MidnightEclipse (Volexity tracks the actor as UTA0218).

![A wide editorial system diagram on deep navy ground. Center: a large rectangular box labeled 'YOUR VPN APPLIANCE: internet-facing HTTPS portal' with five sub-labels (login UI, session manager, tunnel termination, admin panel, OS). Five red curved arrows from the outside converge on it, each labeled with a real 2023-2024 CVE: CVE-2023-46805 (Ivanti), CVE-2024-21887 (Ivanti), CVE-2024-20353 (Cisco), CVE-2023-4966 (CitrixBleed), CVE-2024-3400 (Palo Alto). Behind the appliance, a smaller cluster of internal services labeled 'prod database', 'admin panel', 'engineer SSH'. The cluster is all reachable once the appliance is compromised. The whole assembly sits inside a coral-tinted boundary labeled 'the attack surface you can't shrink'. A small inset on the right shows the mesh alternative as a dotted hexagon of peers with no central appliance, captioned 'no concentrator, no portal, no inbound port'.](/images/lazy-security-part-6-network-plane/vpn-appliance-attack-surface.png)

*Fig. 1 · five vendors, twelve months, the same shape of vulnerability. The mesh alternative is the small diagram on the right.*

Five different vendors. Five different products. Five different attacker campaigns. The common shape is a publicly reachable HTTPS portal that handles authentication and tunnel termination. Every one of them has had a pre-authentication remote code execution in the last twelve months. That isn't a coincidence; it's an architecture.

What mesh networks (Tailscale, Cloudflare Access, Headscale, native WireGuard) don't have is an internet-facing login portal. The control plane authenticates over the same OIDC/SSO your engineers already use; the data plane is WireGuard between authorized peers; there is no single box that, if compromised, lets the attacker into the rest of the network. The lazy stance is: don't run a VPN appliance. Every use case has a mesh-or-proxy answer that ships with less attack surface and less operational pain.

## Tailscale for outbound

The fast path: install the daemon on every machine you want to access from, and every machine you want to access. Each daemon authenticates against your IdP. Within sixty seconds, every node in your tailnet can reach every other node over WireGuard, with private IPv4/IPv6 addresses inside the tailnet.

Replace `ssh user@bastion.yourorg.com` with `tailscale ssh user@prod-db.your-tailnet.ts.net`. Replace `grafana.yourorg.com` (publicly reachable, gated by a Cloudflare IP allowlist that nobody can remember the source of) with `grafana.your-tailnet.ts.net` (only reachable to tailnet members, no public DNS record, no public route).

![A hand-drawn napkin showing a Tailscale ACL annotated with marker arrows and red callouts. The center of the napkin has a 20-line ACL in HuJSON (JSON with comments), with tagOwners, groups, and acls sections. Red arrows point from the 'group:engineers' line to a sketch of three engineer laptop icons, from 'tag:prod-db' to a sketched database cylinder with a 'prod' label, and from the comment '// only platform can reach prod databases' to a small underline beneath the matching ACL rule. A red callout reads 'twenty lines of declarative-policy → entire access plane'. Bottom strip mirrors Parts 1/3/4 chevron pattern with colored dots: 'one ACL → one SSO group → one git diff → entire blast radius'.](/images/lazy-security-part-6-network-plane/tailscale-acl-napkin.png)

*Fig. 2 · twenty lines of HuJSON beats two hundred lines of iptables, and you can `git diff` the change.*

The ACL itself, in HuJSON (JSON with comments, native to Tailscale):

```json
{
  "tagOwners": {
    "tag:prod-db":  ["group:platform"],
    "tag:internal": ["group:platform"]
  },
  "groups": {
    "group:platform":  ["[email protected]", "[email protected]"],
    "group:engineers": ["group:platform", "[email protected]"]
  },
  "acls": [
    // Engineers can reach internal HTTP services.
    { "action": "accept", "src": ["group:engineers"], "dst": ["tag:internal:80,443,3000,8080"] },
    // Only platform can reach prod databases.
    { "action": "accept", "src": ["group:platform"],  "dst": ["tag:prod-db:5432,6379,9200"] }
  ]
}
```

Twenty lines of declarative policy. Each change is a PR. Each merge is reviewed. Each rule is a sentence a human can read in three seconds. The version of this living in iptables is two hundred lines that nobody touches because nobody knows whether the bottom forty are still load-bearing.

Cost: under Tailscale Pricing v4 (April 2026), Free covers up to 6 users and 100 devices. Paid plans are Standard at $6/user/month, Premium at $18/user/month, and Enterprise above that. For a 15-person team, $90/month on Standard buys SSO, audit logs, and ACL change history.

## Cloudflare Tunnel for inbound

Different shape of problem. You have a service that needs to be reachable to specific users (employees, customers, a vendor's support team) but should never have a public DNS record. The wrong answer is putting it on the public internet with an IP allowlist. The right answer is Cloudflare Tunnel.

The architecture: `cloudflared` runs on the origin and dials *outbound* to Cloudflare's edge. There is no inbound port. The origin has no public IP. Cloudflare Access (the policy layer) sits in front of the URL and requires the user to authenticate via your IdP (Google, Okta, GitHub, OneLogin, generic OIDC) before the request is proxied to the origin.

```bash
# On the origin
cloudflared tunnel login
cloudflared tunnel create vendor-admin
cloudflared tunnel route dns vendor-admin admin.yourorg.com
cloudflared tunnel run vendor-admin
```

Three commands of consequence. The vendor admin panel is now reachable at `admin.yourorg.com`, gated by your SSO, with no public route to the underlying service. Add Cloudflare Access policy rules (a few clicks in the dashboard or a Terraform resource) to require a specific Okta group, a specific email domain, a specific device posture, or an mTLS cert.

Cost: Cloudflare Zero Trust Free covers up to 50 users. $7/user/month above that. The Tunnel daemon itself is free; the metering is on Access seats.

## WireGuard, when you can't run Tailscale

For the regulated, air-gapped, or contract-forbids-SaaS case, WireGuard has been in mainline Linux since kernel 5.6 (March 2020) and is enabled by default in most distros. The configuration is plain text. The minimum viable setup is about twenty lines per peer.

```ini
# /etc/wireguard/wg0.conf, on the server
[Interface]
Address = 10.7.0.1/24
ListenPort = 51820
PrivateKey = 

[Peer]
PublicKey = 
AllowedIPs = 10.7.0.2/32
```

```ini
# On the engineer's laptop
[Interface]
Address = 10.7.0.2/32
PrivateKey = 

[Peer]
PublicKey = 
Endpoint = vpn.yourorg.com:51820
AllowedIPs = 10.7.0.0/24, 192.168.10.0/24
PersistentKeepalive = 25
```

`wg-quick up wg0` and the tunnel is live.

The pain at scale is peer management. Twenty engineers means twenty keypairs, twenty `AllowedIPs` blocks on the server, and a manual re-deploy each time someone joins or leaves. Two OSS tools fix that: **Headscale**, which is a Tailscale-protocol-compatible control plane you self-host (same Tailscale clients on each device, but the coordination server is yours); and **wg-easy**, a small web UI for adding and removing peers. Both give you the Tailscale UX with none of the SaaS dependency.

When to pick this path over Tailscale: your contract forbids data-plane traffic transiting a U.S. SaaS, you operate in a region where Tailscale can't legally provide service, or you're regulatory-bound to run the full stack yourself. Otherwise, Tailscale Standard at $6/user/month is a better use of a platform engineer's time than peer management cron jobs.

## the ACLs you can actually read

The reason the appliance era was painful wasn't just the CVEs. It was the iptables rulesets nobody read.

```
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp -d 10.0.1.5 --dport 22 -s 10.0.0.0/24 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp -d 10.0.1.5 --dport 5432 -s 10.0.0.50/32 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp -d 10.0.1.6 --dport 6379 -s 10.0.0.50/32 -j ACCEPT
# ... 200 more lines, no comments, no group names, no diff history
```

versus the Tailscale ACL above. Twenty lines of declarative policy, with group names that mirror Okta groups, with comments that say what each rule is for, version-controlled in `git`. The same is true for Cloudflare Access policies (declarative JSON, also Terraform-supported), Headscale ACLs (the same HuJSON Tailscale uses), and even WireGuard's `AllowedIPs` per peer (one line per route per peer).

If the only person who can read your firewall rules is the person who wrote them three years ago, that is a security problem, not just an operations problem. The audit answer "what does the prod network allow?" should be a `git log` and a code review, not a screen-share with the senior engineer who took notes on a sticky.

## the receipts

For 15 engineers, the network-in-front-of-everything bill:

- **Tailscale Standard** at $6/user/month: $90/month for 15 engineers. Covers SSH-to-bastion, internal HTTP access, prod database access, MagicDNS, ACL audit logs. The primary line item.
- **Cloudflare Zero Trust Free** (<50 users): $0. Replaces public-internet vendor portals, internal-with-SSO web apps, customer-facing internal tools.
- **Self-hosted WireGuard or Headscale**: $0, plus a small VPS for the control plane if needed (~$5/month). For the use case Tailscale can't legally cover.
- **The retired VPN appliance contract**: somewhere between $5K and $50K per year, depending on vendor and seat count, going back into your budget when the contract ends.

![An animated horizontal bar chart in a dark editorial palette comparing the annual access-plane cost for a 15-engineer team across four configurations. Top bar: legacy SSL VPN appliance (Pulse Secure / Ivanti / GlobalProtect at small-business pricing) at roughly $600/year plus the CVE risk; subtitle 'plus appliance ops time'. Middle-top bar (accented, brighter cyan, coral tip): Tailscale Standard at $1,080/year (15 × $6/mo × 12); subtitle 'recommended default'. Middle-bottom bar: Tailscale Free + Cloudflare Zero Trust Free at $0; subtitle 'works up to 6 users / 50 seats'. Bottom bar: self-hosted Headscale + WireGuard at $60/year (just a small VPS); subtitle 'for the air-gapped or contract-bound case'. Annotation strip notes the appliance bar's true cost is dominated by patch/CVE response and is undercounted at $600.](/images/lazy-security-part-6-network-plane/access-plane-cost-stack.gif)

*Fig. 3 · the bottom two bars are not an emergency, the top bar is.*

Net cost: $90/month for the access plane covering most of the surface, with optional fallbacks at near-zero cost. For comparison, the median per-seat price of a legacy SSL VPN appliance (Pulse Secure / Ivanti, GlobalProtect, AnyConnect) at small-business pricing is around $40/seat/year, or roughly the same number, without the CVE risk and without the iptables ruleset.

What this catches: every internet-facing VPN appliance CVE, because you don't have one. Every "the bastion's security group was opened to 0.0.0.0/0 to debug a contractor's IP last summer and never closed" incident, because the bastion isn't reachable. Every "the static VPN certificate leaked from a contractor's laptop" incident, because the credential is a short-lived OIDC session, not a long-lived cert.

What it doesn't catch: a compromise of your IdP itself. If an attacker controls Okta, they control your tailnet. (Cloudflare's Thanksgiving 2023 incident report, attributed to Okta's October 2023 support-system breach, is the canonical reference for this failure mode and the response.) The mitigation lives in [Part 3](/blog/lazy-security-part-3-unsexy-list): FIDO2-only admin access on the IdP, audit log streaming, and a runbook for "rotate all sessions in five minutes."

If you do one thing this week, install Tailscale on your laptop and one prod host, get them talking over the tailnet, and replace one `ssh bastion.yourorg.com` invocation with `tailscale ssh prod-host`. Cost: zero. Time: ten minutes. Then plan the rest of the rollout for next quarter, when you have the budget conversation with whoever still pays for the SSL VPN renewal.



How to bypass corporate MDM and AI gateways on Claude Code
root@harshit.cloud (Harshit Luthra) — Fri, 08 May 2026 00:00:00 GMT

If you're reading this, there's roughly an 80% chance your company rolled out an MDM last quarter, your network team wedged Claude API traffic through an AI gateway around the same time, and now Claude Code boots with MCPs you didn't pick while forwarding your prompts somewhere you haven't audited. `/mcp` shows three servers nothing in your repo touches. `env | grep ANTHROPIC` returns a base URL on a domain you've never seen. The experience got worse and nobody asked you.

![Two corporate leashes converging on Claude Code in the middle: an MDM agent on the left pushes a schg-flagged managed-settings.json into /Library/Application Support/ClaudeCode/, an AI gateway on the right intercepts traffic that Claude thinks is heading for api.anthropic.com.](/images/bypassing-claude-code-mdm-managed-settings/hero.png)
*Fig. 1 · both leashes pull on the same runtime. You can usually only get one off without your laptop calling home about it.*

This post covers both leashes. The MDM one is fixable in 12 lines of zsh. The AI gateway one depends on how deep your network team went.

## what's an MDM, in three sentences

MDM stands for Mobile Device Management. Jamf, Kandji, Intune, Workspace ONE, whichever agent enrolled your laptop on day one. It owns parts of `/Library`, can write files there as root with the system-immutable flag set, and re-pushes them on a schedule, which is why a plain `rm` doesn't survive. For Claude Code, the relevant directory is `/Library/Application Support/ClaudeCode/`.

## the managed-settings situation

The two files doing the work are `/Library/Application Support/ClaudeCode/managed-settings.json` and `/Library/Application Support/ClaudeCode/managed-mcp.json`. Claude Code reads them on startup, treats them as the highest-priority settings layer, and merges them over whatever you have in `~/.claude/settings.json`. Anything IT puts in there wins: forced MCPs, forced skills, allowed and denied permission lists, and the `env` block that can set `ANTHROPIC_BASE_URL`. That last one is how the AI gateway routing gets wired into Claude Code in the first place.

## why `rm` doesn't work

First instinct fails, and not in a way that's obvious:

```bash
sudo rm "/Library/Application Support/ClaudeCode/managed-settings.json"
# rm: managed-settings.json: Operation not permitted
```

Root isn't enough. The MDM agent sets the file's system-immutable flag with `chflags schg` after writing it. That flag blocks deletion even by root until it's cleared. The macOS `chflags(1)` man page is the receipt. `schg` is the "system immutable" flag, and the file "may not be changed, moved, or deleted" while it's set.

Confirm it on your own machine:

```bash
ls -lO "/Library/Application Support/ClaudeCode/managed-settings.json"
# -rw-r--r--  1 root  wheel  schg  482 May 14 09:11 managed-settings.json
```

`schg` in column five is the marker.

The detail that matters: managed-settings.json is the same config layer your `~/.claude/settings.json` uses. The IT copy just lives under `/Library`, is owned by root, and has the schg flag set. The merge logic doesn't know which file came from a human.

## the cleanup script

One thing worth flagging before you run this. On macOS, the `schg` flag is normally clearable by root for files outside SIP-protected paths, and `/Library/Application Support/ClaudeCode/` is not SIP-protected. So `sudo chflags noschg` works as written. If your MDM also writes its config into a SIP-protected location (rare for application config, more common for system extensions), you'd need Recovery Mode Terminal to clear those, which is a different conversation. The script's `2>/dev/null` will silently swallow that failure, so if reruns don't seem to take, that's where to look.

Save this as `/usr/local/sbin/claudecode-cleanup.sh`, make it executable, run with `sudo`:

```zsh
#!/bin/zsh
FILES=(
  "/Library/Application Support/ClaudeCode/managed-settings.json"
  "/Library/Application Support/ClaudeCode/managed-mcp.json"
)
for f in "${FILES[@]}"; do
  # Clear immutable flag if file exists, then remove
  [ -e "$f" ] && /usr/bin/chflags noschg "$f" 2>/dev/null
  /bin/rm -f "$f"
done
```

```bash
sudo chmod 755 /usr/local/sbin/claudecode-cleanup.sh
sudo /usr/local/sbin/claudecode-cleanup.sh
```

Two lines do the real work. `chflags noschg` clears the immutable bit. `rm -f` removes the file. The `2>/dev/null` swallows the noise on a clean machine where the file isn't there.

Restart Claude Code. `/mcp` should be back to whatever you actually installed, and `/permissions` should be whatever's in `~/.claude/settings.json` instead of whatever IT decided you needed.

## the launchd arms race

I'd love to tell you this is permanent. It isn't.

MDM agents sync on a schedule. Every 15 minutes, every hour, on login, depending on profile. When they sync, they notice the file is gone, put it back, and re-apply the schg flag. You'll watch managed-mcp.json reappear like a horror-movie villain you keep stabbing.

A few options, in increasing order of trouble you're inviting:

- **Run the script on a launchd LaunchAgent that fires at login.** Once per session. Low impact, low effectiveness, but if your MDM only syncs at login this is enough.
- **Run it on a launchd timer with a 60-second interval.** Now you're in an arms race with the sync schedule. Works until someone in IT notices a config-drift alert for your hostname.
- **Block the MDM agent's outbound DNS.** Effective, loud, and the kind of thing that gets your laptop wiped on the next compliance audit.

I run the first one. The MDM gets its login telemetry, my dev environment isn't broken for the hour or so between syncs, nobody opens a ticket. Pick the option that matches how much you actually want to fight this.

Minimal `~/Library/LaunchAgents/cloud.harshit.claudecode-cleanup.plist`:

```xml




  Labelcloud.harshit.claudecode-cleanup
  ProgramArguments
  
    /usr/bin/sudo
    -n
    /usr/local/sbin/claudecode-cleanup.sh
  
  RunAtLoad


```

`sudo -n` only works if you've added a NOPASSWD line for that exact script in `/etc/sudoers.d/claudecode-cleanup`. Which the MDM might rewrite. The arms race goes deeper than you think.

## the AI gateway angle

The other leash sits at the network layer. Companies route Claude API traffic through a gateway (Cloudflare AI Gateway, Portkey, LiteLLM, internal proxies) so they can log prompts, strip PII, enforce per-user quotas, or quietly downgrade Opus calls to Haiku when the monthly bill spikes. Claude Code respects `ANTHROPIC_BASE_URL` and will talk to whatever endpoint it points at, as long as your OAuth token or API key authenticates there.

Two routing patterns to recognize:

- **The env block in managed-settings.json.** IT sets `ANTHROPIC_BASE_URL=https://ai-gw.corp.example.com/v1` inside the env section of the managed file. Claude Code reads it on startup. Same fix as the MCP file. The cleanup script above already kills this.
- **System proxy plus a corporate root CA.** Your laptop has a "Corporate Root CA" in keychain, and either an `https.proxy` setting or transparent network interception routes api.anthropic.com traffic through the gateway. Deleting managed-settings.json does nothing here. The interception lives below the application layer.

To tell which one you have, run this in a fresh shell:

```bash
env | grep -i anthropic
# If you see ANTHROPIC_BASE_URL, it's the env block.

curl -v https://api.anthropic.com/v1/messages 2>&1 | grep -iE 'issuer|subject|server certificate'
# If the cert chain is signed by your corporate CA, it's transparent interception.
```

## bypassing the gateway

For the env-block case, the cleanup script already does the work. Restart your shell after running it:

```bash
unset ANTHROPIC_BASE_URL
env | grep -i anthropic
# (empty)
```

For the transparent-proxy case, your options shrink:

- **Personal hotspot for sensitive sessions.** Burns mobile data, leaves no trail through the gateway. Most realistic option for an individual contributor.
- **WireGuard or Tailscale out to a personal node.** Works if your MDM profile allows it. Many block third-party VPNs through `com.apple.systempolicy.kernel-extension-policy`.
- **Personal device for personal work.** Boring answer. The one that holds up in HR if it ever comes up.

What doesn't work: removing the corporate root CA from keychain. It's pinned by an MDM payload and gets re-added on next sync, same pattern as managed-settings.json.

## should you actually do this

Worth saying out loud: both leashes exist because someone at your company had a reason. Compliance, data residency, an incident from six months ago whose postmortem nobody can find.

If the forced MCP is `internal-secrets-lookup` and the gateway logs prompts to a SOC pipeline, your team probably wants you using it. If the MCP is `corporate-docs-mcp` pointed at a 404 and the gateway downgrades Opus to Haiku because someone misread an invoice, you're deleting dead weight.

The script doesn't know which. Ask before you script. Most MDM platforms support per-user opt-out scopes, and one polite Slack message to IT beats a `launchd` plist.

## what these scripts don't do

The cleanup clears two files. It does not:

- Stop the MDM agent.
- Touch `~/.claude/settings.json`. Your settings stay yours.
- Handle `/Library/Application Support/ClaudeCode/managed-permissions.json` if your MDM uses one. Add it to the `FILES` array.
- Survive a reboot or a sync. The agent re-pushes on next check-in.
- Defeat a transparent proxy with a pinned corporate CA. Use the hotspot.

If you wanted a permanent escape from corporate IT, you wouldn't be reading a blog about `chflags`.



Lazy SRE's guide to secure systems, part 5: the dev laptop is the perimeter
root@harshit.cloud (Harshit Luthra) — Sun, 03 May 2026 00:00:00 GMT

In June 2024, Mandiant published the writeup for the Snowflake mass-extortion campaign. Ticketmaster, Santander, AT&T, LendingTree, Advance Auto Parts. Roughly 165 Snowflake tenants in total had data extracted from their warehouses. The defining detail wasn't sophistication. It was the laptop.

Mandiant traced the entry point to infostealer malware (Lumma, RedLine, Vidar variants) running on contractor and developer machines. Their report described the affected devices as personal systems also used for gaming and downloading pirated software. The infostealer harvested every credential the browser had ever saved, including the Snowflake login that didn't have MFA enforced. The attackers walked through the front door of a Fortune 500's data warehouse.

This is part 5. Earlier parts covered npm ([Part 1](/blog/lazy-security-part-1-supply-chain)), GitHub Actions ([Part 2](/blog/lazy-security-part-2-github-actions)), the unsexy infrastructure list ([Part 3](/blog/lazy-security-part-3-unsexy-list)), and DNS auth records ([Part 4](/blog/lazy-security-part-4-dns-records)). Part 5 is about the laptop. The piece of hardware on an engineer's desk that has every SSH key, AWS profile, kubeconfig, GitHub PAT, Slack token, and Stripe key they have ever used to do their job.

The thesis from Part 1 stands. Future You at 3am will not run an EDR scan after every browser extension install. The config that prevents the extension from being installed in the first place is the one that runs while you sleep: the MDM that whitelists, the disk encryption that protects what gets stolen, the hardware MFA that survives the keylogger.

## MDM is the table you set first

Mobile Device Management is the thing every small startup skips and every enterprise has. The bad-faith reason is that it's expensive and annoying. The honest reason in 2026 is that the free options have caught up.

For a 15-person Apple-heavy team, the lazy stack is **Apple Business Manager** (free, Apple-only) plus **Fleet** (OSS, free under 300 endpoints on the self-hosted path, generous free tier on Fleet's cloud). Apple Business Manager assigns a Mac to your organization at first boot, before the user creates a personal Apple ID on it. Fleet runs the osquery agent on every machine and lets you push configuration profiles (the same plist payloads Jamf would push) plus query inventory in SQL syntax.

![A hand-drawn napkin sketch of a laptop, viewed top-down on a workbench. Inside the laptop are labeled icons representing what's actually stored on it: a key labeled SSH, a wallet labeled AWS keys, a kubeconfig folder, a GitHub PAT token, a Slack icon, a Stripe API key, a stack of browser cookies, and a small keychain icon for the password manager. Around the outside, a red dashed boundary labeled 'the perimeter'. A red callout reads 'any one of these, full org compromise'. Bottom strip: 'one device → one keychain → twelve services → entire blast radius'.](/images/lazy-security-part-5-dev-laptops/whats-on-your-laptop.png)

*Fig. 1 · what's actually on the device you take to coffee shops.*

The lazy default config profile, in plain English:

- Require FileVault. Escrow the recovery key to MDM. If the laptop walks, the disk is encrypted; if the user forgets the password, you can recover.
- Require auto-lock at five minutes idle, password to wake. Not a screensaver.
- Block unsigned package installs, restrict the Mac App Store to managed Apple IDs only.
- Require macOS updates within fourteen days of release. The fourteen days lets you skip a known-bad point release; longer than fourteen is negligence.
- Block AirDrop on the corporate Wi-Fi, restrict USB external storage to read-only (or block entirely if your workflow doesn't need it).
- Install osquery via MDM, enrolled to your Fleet server.

For Linux and Windows in the mix, Fleet covers both with the same osquery agent and the same query syntax. The MDM-config-profile half is Windows Intune (free with Microsoft 365 Business Premium) or Workspace ONE's free tier. Either way, the stack is "Fleet for inventory and detections + a platform-specific MDM for enforcement."

The lazy fix for the most common gap: a weekly cron that runs one Fleet query, "every laptop without FileVault enabled," and posts a Slack alert with the user's name. The conversation that follows is "we found your machine, can you enable it today", not a six-month audit.

## hardware keys, one-time spend

YubiKey 5 NFC is $50. Buy two per engineer: one for the desk, one for the bag. Total for 15 engineers: $1,500, one-time, capital expense, deductible.

What it gets you:

- WebAuthn / FIDO2 for SSO login (Google, Okta, GitHub, Cloudflare, AWS): a keylogger can record every keystroke and still never get the second factor.
- SSH key storage in hardware. `ssh-keygen -t ed25519-sk -O resident` writes the key into the YubiKey. The private key never exists on disk.
- PIV smartcard for VPN auth, code signing (`gpg --card-edit`).
- TOTP fallback for the SaaS that hasn't shipped WebAuthn yet.

The free alternative for the SaaS that doesn't support hardware keys is passkeys. Passkeys are WebAuthn under the hood, also phishing-resistant, built into iOS, macOS, Android, Windows Hello, Chrome, and Safari. Free. The catch is sync: if the engineer's iCloud is compromised, so is the passkey. Hardware keys aren't synced; they are a physical token. The lazy answer is both: passkeys for low-risk auth, YubiKeys for the keys that gate production.

Cost: $1,500 one-time for 15 engineers. The cheapest line item in this post for what it gets you.

## EDR is where the budget goes

Endpoint Detection and Response is the part of this stack that costs real money. For OSS-only, the answer is osquery + Wazuh, which works but requires writing detections by hand. For a 15-person team with one platform engineer, "write your own EDR detections" is not a project anyone will finish.

The honest 2026 small-team answer is **Microsoft Defender for Business** at $3/user/month. It ships in Microsoft 365 Business Premium (also useful if you're on M365 anyway), has acceptable macOS coverage, and includes managed detections written by Microsoft's security team. Cost for 15 engineers: $540/year. **CrowdStrike Falcon Go** is $60/endpoint/year if you want the strongest detection at small-team scale; same math, $900/year for 15.

![An animated horizontal bar chart in a dark editorial palette comparing the annual endpoint stack cost for a 15-engineer team across three configurations. Top bar: OSS-only (osquery + Wazuh self-hosted) at roughly $240/year (just the VPS). Middle bar (accented, brighter cyan, coral tip): Defender for Business at $540/year, the recommended default. Bottom bar: CrowdStrike Falcon Go at $900/year. A small note underneath each bar shows what each catches and what each misses; a strip at the bottom reads 'one-time YubiKey spend not included ($1,500 for 15 engineers across all three).'](/images/lazy-security-part-5-dev-laptops/endpoint-cost-stack.gif)

*Fig. 2 · three configurations. Pick the middle bar unless you have a reason.*

The lazy stance: Defender for Business if you're on Microsoft 365 already. Falcon Go if you're not on M365 and want managed detection without the OSS-engineer overhead. osquery + Wazuh only if you have a security engineer with bandwidth to maintain the detections, which most 15-person startups don't. Pretending otherwise is how you end up with a fancy SIEM nobody reads.

## the password manager and browser hygiene argument

1Password Business at ~$8/user/month. Bitwarden Teams at $4. Apple Passwords (or 1Password Families) if you're Mac-only and don't need shared vaults. Pick one and stop arguing about it on the team's `#tools` channel.

The point of the password manager isn't strong passwords. The point is:

- One place for credentials, audited.
- Shared vaults for vendor logins, instead of "share the password in Slack DM" hygiene.
- Breach notifications when a saved password appears in a public breach corpus.
- Masked email aliases (1Password feature, Apple's Hide My Email equivalent): every signup gets a separate alias, every spam list is contained.

Browser hygiene matters because the Snowflake infostealer harvested credentials from browser local storage. Specifically:

- Enforce browser auto-updates via MDM. Both Chrome and Edge expose policy keys for this; Firefox via `policies.json`.
- Block sync of work browser profiles to personal Google/Apple accounts. The "I signed into Chrome with my personal account and now all my work bookmarks are in someone else's cloud" leak is real.
- Block "developer mode" extension installs. Force extensions to come from the Chrome Web Store; force the Web Store to honor the org's allowlist via the `ExtensionInstallAllowlist` policy.
- Disable browser password saving entirely. Everything routes through the password manager.

Total: $1,440/year for 15 engineers on 1Password Business. $720 on Bitwarden Teams. $0 on Apple Passwords if it covers your needs. Pick a line and walk it.

## the personal device problem

The Snowflake breach was about contractors using personal Macs for work. The lazy answer at a 15-person startup might surprise: corp-issue every contractor a laptop. Yes, including the four-hour-a-week consultant.

A refurbished MacBook Air with 16GB RAM is roughly $700 from Apple's Certified Refurbished store. The cost of a Snowflake-scale breach starts at $370K (the reported AT&T ransom) and ends in the customer-churn and legal-exposure column. The break-even point on hardware-for-contractors is under three serious incidents, ever.

![An editorial side-by-side system diagram on a dark navy ground. Left panel labeled 'personal device, BYOD' shows a laptop with chaotic state: unenforced FileVault status, a personal iCloud sign-in, a Mac App Store with personal Apple ID, a Chrome browser synced to a personal Google account, a Slack web app session that's been logged in for nine months, a folder labeled 'pirated software' with a red warning. Right panel labeled 'corp-issued, MDM enrolled' shows the same laptop with each item enforced: FileVault ON, MDM-managed Apple ID, App Store restricted, Chrome work profile only, Slack session expires daily, no third-party software installs. Each enforced item has a green check; each unenforced item on the left has a coral X. A title above reads 'where the Snowflake breach lived'.](/images/lazy-security-part-5-dev-laptops/personal-vs-corp-laptop.png)

*Fig. 3 · same laptop, different enrollment. The right panel is the one where Mandiant doesn't write your name down.*

What "no work on personal devices" actually requires:

- Contract clause: hardware is issued, personal-device use for work is prohibited.
- MDM enrollment at first boot via Apple Business Manager (or Windows Autopilot).
- Disabled iCloud personal sign-in; only managed Apple IDs.
- Wipe via MDM on offboarding, before reissue.
- No "I can just SSH from home for ten minutes" escape hatch. The escape hatch is what the contractor will use the day they get phished.

This is the section of the post that gets the most pushback. The pushback is right about cost and wrong about risk. Run the math at your scale; it runs the same direction every time.

## the receipts

For 15 engineers, the first-year laptop security budget:

- YubiKey 5 × 30 keys (two per engineer): $1,500, one-time.
- Fleet (OSS self-hosted on a small VPS): $240/year.
- Microsoft Defender for Business: $540/year. Substitute Falcon Go at $900 if not on M365, or osquery+Wazuh at $0 if you have a security engineer.
- 1Password Business: $1,440/year. Or Bitwarden Teams at $720. Or Apple Passwords at $0.
- Refurbished corp laptops for non-employee contractors: ~$700 per, as needed.

Total recurring: roughly $1,020–$2,220/year for 15 engineers, depending on the EDR and password-manager line. Add the one-time YubiKey spend and the first year lands at $2,520–$3,720. Call it $14–$21 per engineer per month.

What it catches: every infostealer that hits a managed laptop (Defender flags it), every credential that lives in the browser (replaced by the password manager), every login that doesn't have phishing-resistant MFA (the YubiKey is required), every personal device touching production (blocked by the no-BYOD policy).

What it doesn't catch: a determined adversary with physical access and unlimited time. A laptop in a hotel room with no FileVault is owned. A laptop with FileVault and a YubiKey left in the USB-A port overnight is owned slower. Neither situation is what this stack is built for; it is built for the infostealer that landed on the contractor's personal Mac.

If you do one thing this week, buy two YubiKeys for yourself, enroll them on GitHub, Google, and Okta, and turn off SMS-based MFA on each. Total cost: $100, one hour. Then do the rest of the team next quarter.



Lazy SRE's guide to secure systems, part 4: the four DNS records
root@harshit.cloud (Harshit Luthra) — Sun, 26 Apr 2026 00:00:00 GMT

In February 2024, Guardio Labs published a writeup of a campaign called SubdoMailing. Five million phishing emails a day, sent through subdomains owned by MSN, eBay, VMware, NYC.gov, UNICEF, and McAfee. Every single email passed SPF and DKIM. Every one of them passed DMARC.

The attack didn't break those protocols. It used them. Each victim domain had an `include:` line in its SPF record pointing at a contractor's domain that had been allowed to expire. The attackers re-registered the orphan, inherited the trust, started sending. Some of the broken `include:` chains had been broken for over a year. Guardio dated the operation back to at least late 2022. Nobody had thought to read their own SPF record again after writing it.

This is part 4. Earlier parts covered npm ([Part 1](/blog/lazy-security-part-1-supply-chain)), GitHub Actions ([Part 2](/blog/lazy-security-part-2-github-actions)), and identity, network, and audit logs ([Part 3](/blog/lazy-security-part-3-unsexy-list)). Part 4 is four DNS records and two monitors. One afternoon to write them, three weeks for DMARC to ramp safely. Zero ongoing cost. Closes the entire phishing-impersonation class and the entire rogue-certificate class at the same time.

Future You at 3am will not investigate an SPF chain when finance forwards a wire-transfer email. The records that run in their place will.

## SPF, and the include trap

SPF stands for Sender Policy Framework. The record lives in DNS as a TXT entry on your apex domain. It declares which IP addresses or domains are allowed to send email on your behalf. The receiving mail server checks the sending IP against the list. The check passes or it fails. That is the entire protocol.

The record itself:

```
yourorg.com TXT "v=spf1 include:_spf.google.com include:mailgun.org -all"
```

`v=spf1` is the version marker. `include:` delegates to another domain's SPF record, which expands at lookup time to that vendor's actual IP allowlist. `-all` says anything not listed is hard-fail.

That last token matters. `-all` (hard-fail) tells receivers to reject anything not on the list. `~all` (soft-fail) tells them to mark it suspicious but maybe deliver anyway. `?all` (neutral) tells them you have no opinion. Every getting-started guide ever written defaults to `~all` "to be safe." The major receivers have said for years that they treat `~all` and `-all` the same in scoring. The lazy answer is `-all`. The only reason to use `~all` is during a migration when you can't yet enumerate every legitimate sender.

![A horizontal editorial timeline of the SubdoMailing campaign on a deep navy ground. Six stages along a single line, from a contractor's SPF include published in 2021 through the contractor domain expiring in 2023, an attacker re-registering it in late 2023, the attacker publishing their own SPF record under the orphan, 5 million phishing emails a day passing SPF and DMARC in February 2024, and Guardio Labs' disclosure of 8000 affected subdomains. Attacker-controlled stages in coral, victim stages in cyan, ghosted 'ORPHAN' and 'INHERITED TRUST' phase labels strung across the background.](/images/lazy-security-part-4-dns-records/subdomailing-timeline.png)

*Fig. 1 · three years from include line to five million phishing emails a day. The SPF record never changed.*

The SPF spec has a ten-DNS-lookup limit. Every `include:` counts, recursively. If you chain five SaaS senders (Google + Mailgun + Postmark + SendGrid + Stripe), each one's `include:` expands into its own record, which may include another, and you can blow the limit without realizing. When you blow the limit, the record evaluates as `permerror`, and many receivers treat that as "no SPF," which means anyone can spoof you. Tools like `dmarcian.com/spf-survey` count the lookups for free. Audit yours.

The SubdoMailing failure mode is what happens when one `include:` points at a contractor whose domain you don't control. The contractor goes out of business. The registration expires. Someone buys the lapsed domain. They publish their own SPF allowlist. Your domain now declares that the buyer is an authorized sender for you. Every email they send passes SPF. The fix is to audit your `include:` chain quarterly: does every domain in it still belong to someone you trust? Most teams have never done this once.

## DKIM, in DNS

DKIM (DomainKeys Identified Mail) is a cryptographic signature on every outbound email. The signing key is a public/private keypair. The private key lives in your mail server (Google Workspace, Microsoft 365, Postmark, your own Postfix, whatever). The public key lives in DNS, under a selector subdomain.

```
selector1._domainkey.yourorg.com TXT "v=DKIM1; k=rsa; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQ..."
```

The selector (`selector1` here) is so you can rotate keys. Publish a new selector, switch the mail server to sign with the new private key, leave the old selector live for a week so in-flight emails still verify, then retire it. Most providers handle this rotation for you once the original selector is configured.

Two things go wrong in practice. First, key length. RSA-1024 was the standard a decade ago and is now considered weak; RSA-2048 is the current default. Some old DKIM records still publish 1024-bit keys, and many major receivers now fail or ignore 1024-bit signatures. Audit with `dig TXT selector1._domainkey.yourorg.com`. Second, third parties signing on your behalf without your knowledge. If finance connects a new SaaS tool that sends email as `noreply@yourorg.com` and nobody sets up DKIM for that path, that vendor's emails will fail DKIM alignment. Receivers see a domain with DKIM mostly working and one path failing, which is often enough to flag the whole domain in spam filters.

Most providers (Google Workspace, Microsoft 365, Postmark, Mailgun, SendGrid) make DKIM publishing a checklist item in their onboarding. If a vendor doesn't, that is a signal about the vendor's sophistication, not yours.

## DMARC, the part that does the work

DMARC (Domain-based Message Authentication, Reporting & Conformance) is the policy layer on top of SPF and DKIM. It tells receivers what to do when SPF and DKIM checks fail, and it tells you, via aggregate reports, what's happening to your domain in the wild.

A minimal DMARC record:

```
_dmarc.yourorg.com TXT "v=DMARC1; p=reject; sp=reject; rua=mailto:dmarc@yourorg.com; pct=100; adkim=s; aspf=s"
```

The fields that matter:

- `p=reject`: policy for emails that fail both SPF and DKIM on the apex. Three values, `none` (just report), `quarantine` (deliver to spam), `reject` (drop). The end state is `reject`. The path is `none → quarantine → reject`.
- `sp=reject`: same policy for subdomains. This is the SubdoMailing detail every public DMARC how-to forgets. A domain with `p=reject` but `sp=none` is wide open for subdomain abuse. Set both.
- `rua=mailto:`: where aggregate reports get sent. Free DMARC report parsers (Postmark, dmarcian, EasyDMARC) accept these and render them as human-readable summaries.
- `pct=100`: fraction of failing mail to apply the policy to. Start at 25% during the ramp, end at 100%.
- `adkim=s` and `aspf=s`: strict alignment. The From-address domain must match the DKIM signing domain (and SPF return path) exactly. The default is relaxed, which lets subdomains substitute. Strict is what you want unless something is breaking.

The ramp from `p=none` to `p=reject` is what takes three weeks. The risk is breaking a legitimate sender path you didn't know existed. Week one, publish `p=none; pct=100`. Receive DMARC aggregate reports for seven days. Identify every IP and `From:` domain that sent on your behalf. There will be three or four you didn't expect: a newsletter platform finance signed up for, an HR tool, a calendar invite system. Onboard each into SPF and DKIM. Week two, move to `p=quarantine; pct=25`, watch reports for new failures. Week three, `p=reject; pct=100`. Done.

![An animated horizontal bar chart in a dark editorial palette showing FBI IC3 business email compromise losses in the United States by year, from 2020 ($1.8B) through 2024 ($2.77B). Bars fill in sequence. The 2024 bar is accented with a brighter cyan and a coral tip. A bottom strip notes that the average loss per incident in 2024 was $129K and that the dataset is U.S.-only, and that global BEC losses are higher.](/images/lazy-security-part-4-dns-records/bec-losses.gif)

*Fig. 2 · BEC losses by year, U.S. only. The 2024 number exceeded ransomware.*

Most small teams stop at `p=quarantine` and never finish the ramp. The difference between `quarantine` and `reject` is whether the attacker's spoofed wire-transfer email lands in the CFO's spam folder or never enters the mail system at all. Spam is where employees go to recover legitimate mail that was filtered too aggressively, which means they go there to fish out emails they want to trust. Reject is the answer.

## CAA, two lines to gate cert issuance

CAA (Certification Authority Authorization) is a DNS record that names which Certificate Authorities are allowed to issue TLS certificates for your domain. Without one, any publicly trusted CA in the world can issue a cert for your domain to anyone who passes that CA's domain-validation challenge. With one, only the CAs you've named can.

```
yourorg.com CAA 0 issue "letsencrypt.org"
yourorg.com CAA 0 issuewild "letsencrypt.org"
yourorg.com CAA 0 iodef "mailto:security@yourorg.com"
```

`issue` restricts standard certificates. `issuewild` restricts wildcard certificates. `iodef` is where notifications are sent when an unauthorized CA tries to issue. If you use multiple CAs (one for ACM in AWS, one for Let's Encrypt in your edge, one for Cloudflare-managed certs), list them all:

```
yourorg.com CAA 0 issue "letsencrypt.org"
yourorg.com CAA 0 issue "amazon.com"
yourorg.com CAA 0 issue "digicert.com"
```

CAA cannot prevent a misbehaving CA from issuing anyway. But CAs are required by the CA/Browser Forum baseline requirements to honor CAA at issuance time. They mostly do. When they don't, the misissuance ends in a Mozilla CA-incident bug report and eventual CA distrust. CAA exists so that legitimate misissuance is detected (because the CA you named never issued the cert and the issuing CA broke the rule) and accidental misissuance is structurally impossible. Both buy you something.

Cost: three DNS lines. Effort: ten minutes. Catches a class of attack (man-in-the-middle via misissued cert) that most teams have no other defense against.

## the monitors

Two streams pay back the four records.

First, certificate transparency log monitoring. Every publicly trusted CA is required to log every certificate they issue to public append-only logs. `crt.sh` is a free queryable index. The `certstream` Python library streams new entries in real time, also free. Cloudflare offers free CT monitoring for any domain on its DNS. Whatever you pick, the workflow is: cert is issued for `*.yourorg.com` → log entry appears within seconds → your monitor pages a Slack channel → you check whether you issued it. If you didn't, that is an incident, not a notification.

![A hand-drawn napkin showing the four DNS records as a cheat sheet, written in marker, ready to copy into a DNS panel. Top of the napkin reads 'the four-record afternoon'. Four labeled blocks underneath: SPF as a TXT record with `-all` circled in red, DKIM as a TXT record with the selector subdomain highlighted, DMARC with `p=reject` and `sp=reject` both underlined twice, CAA with the issuer name circled. Bottom of the napkin has two boxes labeled 'CT log monitor' and 'DMARC report inbox', with arrows pointing to a small Slack icon and a small email icon. A red callout at the bottom reads 'fifteen minutes a week'.](/images/lazy-security-part-4-dns-records/dns-records-napkin.png)

*Fig. 3 · the whole afternoon, sketched. Plus what runs after.*

Second, DMARC aggregate report parsing. The `rua=` address in your DMARC record receives daily XML reports from every receiver. Reading the XML raw is unpleasant. The free tiers of Postmark, dmarcian, and EasyDMARC all accept the report stream and render it as "here are the IPs that sent as you this week, here are the ones that failed alignment, here are the new ones since last week." The new-sender alerts are where you find out that someone in marketing has connected a SaaS tool that's now sending emails as you, failing alignment, and getting your domain reputation downgraded.

A weekly fifteen-minute review of both monitors is what good looks like at a 25-person team. The cost is fifteen minutes a week. The product is "we'd have noticed if someone issued a cert for our login subdomain on Tuesday."

## the receipts

Four DNS records. Two monitors. One afternoon for the records, three weeks for the DMARC ramp, fifteen minutes a week for the reviews. Cost: zero, unless you upgrade past the free tier of a DMARC parser at $15–$50 a month, which is the only thing on the list that's not free.

What this catches: every attempt to send email impersonating your domain from outside your authorized sender list, every attempt to issue a TLS cert for your domain from an unauthorized CA. The FBI's 2024 IC3 report attributed $2.77B in U.S. business email compromise losses to roughly 21,000 incidents, a $129K average. The fraction of those that would have been caught by a domain publishing `p=reject; sp=reject` with an honest SPF audit is enormous.

What it doesn't catch: phishing from a lookalike domain (`yourorg-corp.com`, `yourorg-support.com`, `yourorg.co`). Lookalike-domain defense needs a paid monitoring service at the tier that matters, and there's no free version that works at small-team scale. Skip it until you have a budget line for security. Note it in the runbook.

If you do one thing this week, publish `_dmarc.yourorg.com TXT "v=DMARC1; p=none; rua=mailto:dmarc@yourorg.com"` and point the address at a Postmark free-tier DMARC inbox. Read the first report in seven days. The list of senders you didn't know about is the answer to "why has this been skipped for two years."



Lazy SRE's guide to secure systems, part 3: the unsexy list
root@harshit.cloud (Harshit Luthra) — Sun, 19 Apr 2026 00:00:00 GMT

I have a calendar reminder that fires on the first of every month. It says "rotate the PAT." I have hit "snooze for 1 week" seventeen times in a row. The PAT in question is a `ghp_` token with read-write access to four private repos and permission to push tags, and the last time I rotated it was October 2024. If anyone has phished my GitHub session in the past fifteen months, they have had a year's head start on me.

This is part 3. [Part 1](/blog/lazy-security-part-1-supply-chain) was npm. [Part 2](/blog/lazy-security-part-2-github-actions) was GitHub Actions. This part is the unsexy list: the controls that don't fit a single attacker narrative, that protect against many different classes of incident in small ways. Identity, network access, default credentials, attestation, the audit log you'll need when the rest of the series missed what you needed it to catch.

The thesis from Part 1 stands. Future You at 3am will not rotate the PAT. The config that makes the rotation unnecessary (short-lived expiry, fine-grained scope, SSO enforcement, audit streaming) is the one that runs while you sleep.

## the PAT you forgot is in four places

Personal-access tokens hide in more places than I want to think about. Mine, when I went through them this weekend:

- `~/.netrc` (the one git falls back to when no credential helper is set)
- `~/.zshrc`, exported as `GH_TOKEN` because some script three years ago needed it
- Mac Keychain, two duplicates, one expired in 2023 but the dialogue still surfaces it
- A `.env` in a repo I haven't pushed to since last summer, committed in plaintext to the `staging` branch (`git log -S 'ghp_'` finds these surprisingly often)
- One CI secret in a repo whose workflow file I deleted six months ago; the workflow went, the secret did not

That's five, not four, which is on-brand for this section.

![A stylized editorial map on a dark navy ground showing the typical places a credential lives on an engineer's machine and inside the org. Nodes for shell config (`~/.zshrc`, `~/.netrc`), Mac Keychain, the CI secret store with one node tagged 'orphaned' in coral, a `.env` file in a stale repo, a browser-cached session, a Slack DM history, a password manager entry, and a post-it from 2022 also in coral. Arrows show the typical sprawl, with concentric rings labeling 'on disk', 'in cloud', and 'in someone else's possession'.](/images/lazy-security-part-3-unsexy-list/credentials-in-the-wild.png)

*Fig. 1 · every place a credential hides. Most teams have it in all of them simultaneously.*

The fix isn't "rotate them all." It's "make the next leak useless." Three configs at the org level do the work.

First, require expiration on all PATs. GitHub org settings → Personal access tokens → Require an expiration date; set the org max to 90 days (GitHub's platform ceiling is 366, but 90 is the right org default). Tokens issued before the setting keep working until their original expiry, so old tokens die naturally as they age out. No big-bang migration.

Second, enforce SSO on the org. A leaked PAT without an active SSO session can't reach SSO-protected repos. Most SaaS git-hosted orgs should have this on already; if yours doesn't, that is the highest-yield ten minutes in this post.

Third, stream the GitHub audit log somewhere SQL-shaped, with two-year retention. The default is six months. You will want eighteen months of history exactly when you need eighteen months of history. The question "did this token get used last week?" should be a query, not a support ticket.

The thing that took me longest to learn is that fine-grained PATs (`github_pat_` prefix, not `ghp_`) let you scope a token to one repo with read-only contents and nothing else. The default scope (full account) is what turns a leaked PAT into a domain compromise. To stop typing `ghp_` into shells entirely:

```
# ~/.gitconfig
[credential]
  helper = !gh auth git-credential
[url "https://github.com/"]
  insteadOf = git@github.com:
```

`gh auth login` once, and `git push` works for the rest of your career. The PAT now lives in one place: `gh`'s keyring entry, scoped to your machine, rotated by `gh` whenever it likes.

## identity is the perimeter

SSO + MFA + SCIM is the only thing on the unsexy list that competes with the PAT story for "worst yield from neglect." A single phished password without these is a domain admin compromise. With them, the same phish gets the attacker a soup of session cookies that expire in eight hours and an MFA prompt they can't satisfy.

The three configs, in rough order of cost:

- **MFA, mandatory, no exceptions.** Including the founder, including the contractor, including the on-call rotation. The exception list is the attack list.
- **SSO for every system that supports it.** Yes, Okta SSO Tax is real. Yes, it is annoying. It is cheaper than rebuilding identity after a session-token compromise. Most of the Snowflake-customer breaches of 2024 started with a non-SSO'd account.
- **SCIM provisioning to every system that supports it.** SCIM means offboarding actually offboards. The day someone leaves, every connected system revokes their access in the same SAML attribute push. Without SCIM, the median time to fully revoke at a small startup is days, and there is always one Postgres console nobody remembered.

![An animated horizontal bar chart in a dark editorial palette comparing the time to fully revoke an employee's access after offboarding. Top bar 'without SCIM (median, small-startup surveys 2024-2025)' grows over several seconds to around four days. Bottom bar 'with SCIM, SAML attribute push' grows to roughly forty-five seconds and is almost invisible at the scale of the first. Coral tip on the without-SCIM bar marks the window of compromise.](/images/lazy-security-part-3-unsexy-list/scim-revocation-window.gif)

*Fig. 2 · the no-SCIM bar is the entire window of compromise.*

One nightly cron closes most of the rest of the gap:

```bash
# nightly: diff "people on payroll" vs "humans with prod access"
okta-cli list-users --status active | sort > /tmp/active.txt
aws iam list-users --query 'Users[].UserName' | jq -r '.[]' | sort > /tmp/prod.txt
diff /tmp/active.txt /tmp/prod.txt | mail -s "identity-diff $(date +%F)" sec@yourorg.io
```

It runs in twelve seconds and surfaces the contractor whose SCIM hook silently broke in March.

## the access plane: Tailscale, IAP, PrivateLink

Nothing internal needs to be on the public internet. Anything that isn't can't be scanned by Shodan, can't be hit by a credential stuffer, can't be 0-day'd by a CVE published yesterday. The configs are different per layer, but the move is the same: take the thing off the internet and put authentication in front of it.

For shell access and internal HTTP services, Tailscale. The pitch is honest. Install the daemon on every machine, write a twelve-line ACL, you have a private network without running a VPN appliance. Replace SSH-to-bastion with `tailscale ssh`. Replace the internal Grafana on `grafana.yourorg.io` with `grafana.your-tailnet.ts.net`. Both stop existing on the public internet the same afternoon.

For web apps that need real auth-aware proxying (customer-facing internal tools, vendor admin panels), Cloudflare Access or Google IAP. The user hits a public URL, the proxy hands them off to your IdP, then proxies the request to a private backend. The backend has no public route.

For service-to-service inside cloud accounts, AWS PrivateLink and GCP Private Service Connect. These exist so your `stripe-receiver` lambda doesn't need to leave the VPC to reach Stripe's API. They are also what you need so the data warehouse in account A can reach the production database in account B without anything traversing the public internet.

![A hand-drawn two-panel napkin. Left panel labeled 'what the security group says (`0.0.0.0/0`)' shows three boxes (postgres, redis, grafana) sitting in the open, with arrows from labeled attackers (a Shodan crawler, a credential stuffer, a CVE-2026-12345 scanner) landing directly on them. A dashed line labeled 'the bastion SG' floats nearby, doing nothing. Right panel labeled 'what the tailnet says' shows the same three boxes behind a solid Tailnet boundary, with the same attacker arrows bouncing off the boundary line. Bottom strip reads 'twelve lines of ACL → entire blast radius'.](/images/lazy-security-part-3-unsexy-list/access-plane-contrast.png)

*Fig. 3 · same services, different boundary. The right panel is whatever Future You at 3am will thank you for.*

The anti-pattern is the "we'll just rotate the bastion IP" security group. We won't. The credentials for the bastion are in a Slack channel from 2023. The bastion is one of those things that exists because someone set it up before everyone joined and nobody knows whether it's safe to turn off. The lazy answer is to make the bastion irrelevant.

## the helm chart that ships with admin/admin

Every operator-installed thing in the cluster has a default password. Argo CD's `admin` with auto-generated password is fine, because the password isn't `admin`. Grafana's chart that ships with `admin/admin` is not fine. Jenkins ships with a random initial password printed to `initialAdminPassword` that most operators copy in once and never rotate. Half the database charts have `password: changeme` in `values.yaml` and the README says "you should change this," which is not the same as the chart changing it.

The lazy fix is two configs.

First, every secret in the cluster comes from external-secrets or sealed-secrets, never from a `values.yaml`. Pick one. The choice matters less than the consistency. Mine is external-secrets pointing at Vault, because reconciliation handles rotation upstream and the YAML stays clean.

Second, a weekly cron that hits every Service in the cluster with the top 25 default credentials and pages on success. `nuclei` ships a template set for this:

```bash
nuclei -t http/default-logins/ -l services.txt -severity critical,high
```

If it finds something, that's a real incident. If it doesn't, you have evidence, which is the audit-log argument postponed by one section.

One honest aside in parentheses: the rate at which Helm chart maintainers have moved away from default passwords is encouraging. Bitnami's PostgreSQL chart now generates a random password by default instead of `changeme`. The chart that ships with `admin/admin` today is more likely to be a private internal chart someone wrote three years ago than something current from Bitnami. (Note: the official Grafana chart still defaults to `admin/admin`: override it via Helm values before first install; "I'll change it later" is the part nobody does.) Check the internal charts first.

## sigstore, provenance, and reproducible builds

Part 1 ended on "the next-tier defenses are real, Part 3 will name them." These are them. Sigstore signing, npm provenance, reproducible builds. Each closes a class of attack that pinning and cooldowns can't.

**Sigstore for container images.** `cosign verify` confirms an image was built by your specific GitHub Actions workflow, with your repo's OIDC identity, against a transparency-log entry that's public and append-only.

```bash
cosign verify ghcr.io/yourorg/api:abc123 \
  --certificate-identity-regexp '^https://github.com/yourorg/api/' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com
```

If an attacker pushes a malicious image to your registry without also compromising your CI's OIDC trust, the verify fails. Bake the verify into your deploy step; refuse to deploy what doesn't pass. That is the attested-deployment pattern Part 2 named, in one verb in your CD pipeline.

**npm provenance.** `npm audit signatures` (since npm 9.5) tells you which dependencies have published provenance attestations linking the `.tgz` to a specific GitHub Actions build. A package with provenance gives you a tamper-evident chain: this artifact came from this commit on this branch in this repo, built by this workflow. Coverage is uneven (most `@types/*` packages have it; most one-maintainer packages don't), but the trend is good. The number to track is "what fraction of my install graph has provenance?" That's your remaining audit surface.

**Reproducible builds.** The hardest of the three. Same source produces the same binary, bit-for-bit, on every build machine. Two implementations have shipped at scale: Debian's reproducible-builds program (`reproducible-builds.org` tracks coverage by package) and Nix. The lazy version, for a small team, is to build the production artifact twice on two different runners and compare hashes. If they match, your CI is reproducible enough to detect a poisoned-build attack. If they don't, you have a non-determinism bug to fix, which is also worth knowing about.

## audit logs are for after the incident

Part 2 ended on "Part 3 will name the controls that exist to make the postmortem readable, not to prevent the incident." This is the section. Audit logging is what tells you whether everything in the previous six sections actually worked, what got accessed when one of them didn't, and which credential to roll at 03:11.

Three streams, all of which support direct destination handoff:

GitHub's audit log to S3, Splunk, Datadog, or whichever SQL-shaped destination you'll actually query. Settings → Audit log → Streaming. Default retention is six months; you want two years. The same goes for Okta's System Log (Reports → System Log → Stream).

AWS CloudTrail to a separate audit account, write-only from production, S3 with Object Lock and KMS-encrypted. Multi-region. The level of paranoia required is "this bucket survives a full prod-account compromise." GCP and Azure have equivalents (Cloud Audit Logs, Activity Logs).

Application audit. Stripe webhook history, Slack audit log, Google Workspace audit log. Each is one config and one Splunk index. The marginal effort approaches zero. The payoff is the difference between a one-page incident summary and a six-week panic.

The runbook for "we think we had a breach Thursday" is then a SQL query against a known schema. Without these, it's an interview with everyone who had access.

## the receipts

The unsexy list is one afternoon, one quarter, and one year. The afternoon: PAT cleanup, SSO/MFA mandatory, GitHub audit log streaming on. The quarter: SCIM provisioning everywhere, Tailscale on every internal service, external-secrets across the cluster. The year: sigstore for your images, an `npm audit signatures` report tracked weekly, reproducible-build hash comparison in CI.

It will not catch a nation-state with patience. It will not catch an insider with a grudge. It will not catch the next Log4j the day it lands. Those are different problems with different budgets, and worth a separate post when one of them happens to one of us.

What it does: it makes the postmortem on your next incident readable. It moves "we don't know what got accessed" out of the executive summary and into "Appendix A, the SQL query." For a small team, that is the difference between recovering and rebuilding.

If you do one thing this week, generate a fresh fine-grained PAT scoped to one repo with a 90-day expiry, switch your `gh auth login` to it, and delete the eight-year-old `ghp_` from your `~/.zshrc`. The calendar reminder won't help. Future You at 3am will not rotate it. Make the wrong default impossible.



Lazy SRE's guide to secure systems, part 2: the actions you didn't pin
root@harshit.cloud (Harshit Luthra) — Sun, 12 Apr 2026 00:00:00 GMT

Last March, someone with write access to the `trivy-action` repo rewrote 76 of its 77 version tags in place. The tags still resolved to `aquasecurity/trivy-action`. They just resolved to different commits than they did the week before. Every pipeline that ran `aquasecurity/trivy-action@0.20.0` (and every other tagged version) ran the attacker's commit instead. Secrets exfiltrated. The stolen credentials chained into PyPI and took down LiteLLM. Nobody noticed for hours, because the workflow file diff was still clean.

This is part 2. [Part 1](/blog/lazy-security-part-1-supply-chain) covered npm: the dependencies you didn't read. Part 2 is the same problem one level up: the workflows you didn't pin. Part 3 is the unsexy list: Tailscale, PrivateLink, IAP, the PAT you forgot.

The thesis from Part 1 stands. The best security work for a small team is the work *Future You at 3am* will actually execute. The configuration that makes the wrong thing impossible beats the runbook that only discourages it. With GitHub Actions, "the wrong thing" has gotten very specific over the last twelve months, and the configs to block each variety have gotten correspondingly precise.

## pinning is necessary but not sufficient

The first thing the trivy-action incident proves: hash-pinning to `@0.20.0` is not pinning. It's a name lookup. The owner of the repo is allowed to rewrite that name. The pin you actually wanted was:

```yaml
- uses: aquasecurity/trivy-action@9b9a3f5c8a5c7e1b6e4d2f1c9b8a7e6d5c4b3a2f # v0.20.0
```

Full forty-character SHA. Immutable. The version comment is so the next reader knows what they're looking at; the SHA is so the workflow runs the code you reviewed.

![A horizontal editorial timeline of the trivy-action force-push attack of March 2026 on a deep navy ground. Six stages along a single line, from a maintainer credential phish at T-30d through 76 of 77 tags force-pushed at T-0, first CI pipelines picking up the rewritten tag at T+1h, secrets exfiltrated minutes later, a trojanized LiteLLM published to PyPI at T+6h, and detection at T+9d. Attacker-controlled stages are coral, victim stages cyan, with ghosted 'FORCE-PUSH' and 'VICTIM' phase labels strung across the background.](/images/lazy-security-part-2-github-actions/trivy-action-timeline.png)

*Fig. 1 · nine days from force-push to advisory. The workflow files never changed.*

Two GitHub features shipped in 2025 that change the math:

- **SHA pinning enforcement** (Aug 2025). An org-level policy that *fails* workflow runs using unpinned actions, instead of warning about them. Settings → Actions → General → Action pinning. Turn it on. There is no "we'll get to it" version of this toggle.
- **Immutable Releases** (Oct 2025, GA). Action authors opt in to making release tags non-rewritable after publication. If you publish actions, turn this on for downstream consumers. If you consume actions, prefer ones that have.

The lazy stance: enforcement at the org level. The workflow that doesn't have a forty-character SHA fails the run. The PR can't merge. The work of remembering to pin moves from every engineer's head to one setting.

What this doesn't catch: an attacker who compromises the maintainer account and ships a new tag at a new SHA. The SHA is real. Pinning by SHA doesn't help, because the workflow author *will* rev to the new version when they read the maintainer's release notes. Which is the next config.

## cooldown is the same trick that worked for npm

Part 1's load-bearing config was `SAFE_CHAIN_MINIMUM_PACKAGE_AGE_HOURS=48`. The principle: most published malware is detected and pulled within hours. If you can wait, the wait does the work for you.

The action ecosystem has the same property, with a longer window. [yossarian's analysis](https://blog.yossarian.net/2025/11/21/We-should-all-be-using-dependency-cooldowns) puts the cooldown that catches most supply-chain attacks at 7-14 days. So:

```bash
pinact --min-age 7 .github/workflows/*.yml
```

Refuses to write a pin younger than seven days. Add to pre-commit, your CI lint, or whatever your dependabot equivalent runs before opening the bump PR.

For Renovate users, the equivalent lives in the action manager:

```json
{
  "packageRules": [
    { "matchManagers": ["github-actions"], "minimumReleaseAge": "7 days" }
  ]
}
```

That's it. Same trick, different ecosystem.

![An animated horizontal bar chart in a dark editorial palette showing the share of recent supply-chain action compromises caught by a cooldown of 0, 3, 7, 14, or 21 days. The 0-day bar lands at 3% and the 3-day bar at 38%. The 7-day bar reaches 76% and the 14-day bar reaches 89%, both accented with a brighter cyan and a coral tip. The 21-day bar lands at 94%. A bottom strip notes that the trivy-action force-push was detected at about nine days.](/images/lazy-security-part-2-github-actions/cooldown-window.gif)

*Fig. 2 · the wait is doing the work. Seven days closes most of the door; fourteen closes most of the rest.*

The empirical question is whether seven days is enough. The trivy-action force-push was detected at about nine. Seven would have caught most consumers, not all of them. The cost of fourteen is "your action versions lag upstream by two weeks." If your action surface is small (most teams are running `actions/checkout`, `actions/setup-node`, one cloud-login action, maybe a deploy action), set fourteen and forget.

## pull_request_target is the new postinstall

Part 1 named `postinstall` as the single trigger that does the most damage and the single switch (`ignore-scripts=true`) that closes the most doors. Actions has the same shape and the same fix.

`pull_request_target` runs in the context of the base repository, with access to repository secrets, but is triggered by a PR from a fork. The legitimate use case is small: comment on PRs, label them, run lightweight metadata jobs. The illegitimate use case is enormous: check out the fork's code and execute it. The attack writes itself. Open a fork, modify a script the trusted workflow runs, watch the runner exfiltrate every secret in the env.

Astral, who maintain `uv` and `ruff`, [wrote it cleanly](https://astral.sh/blog/open-source-security-at-astral): "these triggers are almost impossible to use securely." GitHub partially mitigated this in November 2025 by forcing `pull_request_target` to always use the default branch's version of the workflow, so an attacker can't push a vulnerable workflow on a feature branch and trigger it. But the foot-cannon still ships loaded if your default-branch workflow checks out PR-head code.

![A hand-drawn two-panel napkin. Left panel labeled 'pull_request_target' shows a fork PR boundary as a dashed line, a modified script.sh inside the fork, and a runner on the base side reaching across the boundary while holding a red keyring labeled NPM_TOKEN, AWS_KEY, GH_PAT. Right panel labeled 'pull_request' shows the same setup, but the keyring is replaced by a greyed-out 'secrets.* not in scope' bag. The two panels are structurally identical except for the presence or absence of secrets in the runner.](/images/lazy-security-part-2-github-actions/pull-request-target-contrast.png)

*Fig. 3 · same workflow, different trigger, opposite blast radius.*

The lazy stance:

- Don't use `pull_request_target` unless you've named the specific reason and one other person has signed off.
- If you do, never `actions/checkout` the PR head from inside it. Check out the base SHA, do the metadata thing, exit.
- For everything else, use `pull_request`. It runs without secrets. Attacker-controlled code stays attacker-jailed.

Same shape as `ignore-scripts=true`. The setting that closes the class.

## the safe defaults that go in every workflow

The four-line workflow header that does the most work per character:

```yaml
permissions:
  contents: read

defaults:
  run:
    shell: bash -euo pipefail {0}
```

`contents: read` overrides the org-level default. If a step needs to push a tag or open a PR, that job opts back up to `contents: write` explicitly. The default is the safe one.

At the checkout step:

```yaml
- uses: actions/checkout@ # v4.2.0
  with:
    persist-credentials: false
```

The default behavior of `actions/checkout` is to leave a credential sitting in `.git/config` for the rest of the workflow. Later steps have shipped this credential into uploaded artifacts more than once. Opt out unless a later step in the same job needs to push.

Three secret-access rules with the same flavor:

- Step-scoped `env:`, never workflow-scoped, for any secret.
- Never `${{ toJson(secrets) }}`. Exposes every secret in the project to the runner. There is no use case.
- Never `secrets: inherit` on reusable workflows. Pass each secret by name. The reusable workflow gets exactly what it asked for.

The trivy-action exfiltration worked partly because secrets were workflow-scoped. The malicious step inherited every credential in the env, not just the one the legitimate scan needed. Step-scoping wouldn't have prevented the credential theft, but it would have bounded the blast radius to one secret instead of all of them.

## OIDC, the promise from part 1

Part 1 ended on "the next-tier defenses are real, Part 3 names them." OIDC is the part of that conversation that lives here.

The trade: instead of storing an `AWS_ACCESS_KEY_ID` in repo secrets and praying nobody exfiltrates it, you configure AWS to trust GitHub's OIDC issuer for a specific repo, branch, and workflow. GitHub mints a short-lived (five-minute) OIDC identity token for the workflow run. The workflow trades that for STS credentials whose lifetime you set (default one hour). Nothing long-lived ever sits in the env.

```yaml
permissions:
  id-token: write
  contents: read

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: aws-actions/configure-aws-credentials@ # v4.0.2
        with:
          role-to-assume: arn:aws:iam::123456789012:role/github-deploy
          aws-region: us-east-1
      - run: aws s3 sync ./dist s3://my-bucket
```

The role's trust policy restricts the OIDC subject to your exact repo and (ideally) branch. An attacker who compromises a fork PR can't assume the role, because they don't match the trust condition. The OIDC JWT itself lasts five minutes and the STS credential is scoped to whatever you configure (default one hour). Even an exfiltrated credential gets the attacker a bounded window of scoped access, not a permanent IAM user.

For Google Cloud, the equivalent is Workload Identity Federation. For HashiCorp Vault, the JWT auth backend. Same shape across providers.

The labor here is genuinely one-time. Configure the trust relationship once per repo, delete the long-lived key, forget about rotation forever. The rotation runbook you're not maintaining is one of the better quiet wins in this post.

## zizmor is the local proxy for workflows

Part 1's `safe-chain` sat in front of every package install and refused malware before bytes hit disk. The action ecosystem's equivalent is `zizmor`, a workflow linter that reads your YAML and catches the patterns this post is about, before they merge.

```bash
brew install zizmor
zizmor .github/workflows/
```

It catches unpinned actions, `pull_request_target` with PR-head checkouts, template-injection patterns where attacker-controlled input lands in a `run:` string, jobs with excessive permissions. Add it to pre-commit:

```yaml
# .pre-commit-config.yaml
- repo: https://github.com/woodruffw/zizmor-pre-commit
  rev: v1.x  # pin the rev, obviously
  hooks:
    - id: zizmor
```

The principle is identical to safe-chain. Move the security check from "after the incident, in the postmortem" to "before the PR can merge, on the dev machine." The CI run is the second line of defense. The pre-commit is the first.

## the receipts

The above stack is approximately one afternoon: org-level SHA pinning enforcement, `pinact --min-age 7` or Renovate `minimumReleaseAge: 7 days`, the four-line workflow header, `persist-credentials: false`, no `pull_request_target` with PR-head checkouts, OIDC for every cloud credential, `zizmor` in pre-commit.

It will not catch a maintainer-account compromise that ships clean-looking code which activates weeks later. It will not catch a determined attacker who studies your build and writes a payload that survives every linter and looks innocent at PR review. Nothing in this post will. Part 3 will name the controls that buy partial mitigation against that class: sigstore, npm provenance, reproducible builds, attested deployments. And the ones that exist to make the postmortem readable, not to prevent the incident.

For a small team, the delta from this post is moving from "we're one tag-rewrite away from a credential theft cascade" to "an attacker would need a credentialed insider, or a fifteen-minute window of luck against a scoped IAM role." That's the only delta that matters at this scale.

If you do one thing this week, turn on SHA pinning enforcement at the org level. Everything else gates off that.



Lazy SRE's guide to secure systems, part 1: the dependencies you didn't read
root@harshit.cloud (Harshit Luthra) — Sun, 05 Apr 2026 00:00:00 GMT

A few months ago a friend's CI pipeline tried to install a package none of us had heard of. The build failed. The error wasn't a missing dep. The error was a local proxy saying *this is malware, I'm not letting it touch disk*. The package was a transitive dependency, six levels deep, that had been published to npm 38 minutes earlier. Nobody on the team had asked for it.

I do platform work at a startup. The job is: keep production up, keep the bill down, keep the kind of person who reads HN comments from getting a free shell on the cluster. The thesis for the series is short. The best security work for a small team is the work *Future You at 3am* will actually execute. The lazy answer is also, almost always, the right one: the configuration that makes the wrong thing impossible, rather than only discouraged. This is part 1. Part 2 is GitHub (`tj-actions`, OIDC, fork PRs). Part 3 is the unsexy list (IAP, Tailscale, PrivateLink, Okta, default helm creds, the PAT you forgot).

## the picture

You opened your editor, ran `npm install`, and onboarded somewhere between 800 and 2,000 packages maintained by people you have never audited. Your team reviewed five direct dependencies. The other 1,995 came in for the ride.

![A two-panel hand-drawn diagram. Left panel labeled 'what i installed' shows five neat dependency boxes connected to a 'your app' box. Right panel labeled 'what npm install pulled in' shows the same five boxes fanning out into a sprawling cluster of small transitive dependencies, with one highlighted in red labeled evil-helper@1.2.3 and a callout reading 'this one is the one shipping crypto miners.'](/images/lazy-security-part-1-supply-chain/dependency-tree-contrast.png)

*Fig. 1 · left is the dependency graph you reviewed at PR time. Right is the one your CI runner actually executes.*

The job is not to read all 1,995. The job is to make sure that when one of them is the problem, the blast radius is small and the alarm goes off.

## chainjacking

Chainjacking is the umbrella term for "someone got control of a package you depend on and pushed a bad version." The attacker doesn't break npm. They get the credentials of the human who publishes the package, ship a patch version, and semver puts it on your machine the next time anyone runs `npm install`. `event-stream` (2018), `ua-parser-js` / `coa` / `rc` (2021), `lottie-player` (2024), and the shai-hulud worm (Sept 2025, with a 2.0 wave in Nov 2025) that self-replicated by stealing tokens from compromised maintainer machines via TruffleHog-style secret scans and then republishing every other package that maintainer owned. The economics still work for the attacker. It is going to keep happening.

![A magazine-infographic-style timeline on a dark navy background. Six stages from left to right: T-7d maintainer account targeted, T-0 malicious version published, T+12m first CI installs it, T+12m02s secrets exfiltrated, T+1h backdoor in artifacts, T+24h credentials for sale. Stages 1-2 highlighted in coral as 'compromise' stages; stages 3-6 in cyan as 'victim' stages.](/images/lazy-security-part-1-supply-chain/chainjacking-timeline.png)

*Fig. 2 · twenty-four hours from maintainer phish to credential resale. Nobody noticed the version bump.*

What matters in that timeline is that the *human* steps are slow and the *automated* steps are fast. The window between "malicious version published" and "your CI runs `npm install`" is whatever your dependabot cron is. If you auto-merge minor and patch bumps, that window is ninety seconds.

## dependency confusion

You have a private package called `internal-utils`. Your CI is configured with both your private registry and the public npm registry. Somebody publishes `internal-utils` to public npm at a higher version. CI installs the public one. Birsan did this against Apple, Microsoft, Tesla, and PayPal in a weekend in 2021 for bug bounties.

Fix, in `.npmrc`:

```
@yourorg:registry=https://npm.yourorg.internal
registry=https://registry.npmjs.org/
```

Scope everything internal. Register your scope on public npm as a parked placeholder. It costs nothing.

## postinstall

Most install-time npm supply-chain incidents I have read the postmortem on shipped their malicious code in a `postinstall` script, not in runtime code. (Some recent ones, like the chalk/debug compromise of Sept 2025, activate at runtime in the browser or on first import; the switch below doesn't help against those. It does help against the install-time class, which is still the majority.) The install hook runs before your tests, before your linter, as part of the install. Default is enabled. The one-line change with the highest blast-radius reduction:

```
# .npmrc
ignore-scripts=true
```

You'll need to allowlist two to five packages that genuinely need it (typically `bcrypt`-shaped things). That number is small. The alternative is letting every package run code on install. Pick.

## the install that buys the most

Aikido's [`safe-chain`](https://github.com/AikidoSec/safe-chain) is an open-source local proxy that sits in front of `npm`, `npx`, `yarn`, `pnpm`, `pnpx`, `bun`, `pip`, `uv`, `poetry`, `pipx`. Every package download is intercepted and checked against Aikido Intel, an open malware feed. Malware is blocked before bytes hit disk. Which is before `postinstall` runs. Free. No account.

![A clean dark-editorial flow diagram. Five columns from left to right: developer terminal running 'npm install lodash-utils', a shell alias intercepting the command, a local proxy that all package downloads route through, the Aikido Intel cloud queried for malware reputation, and an outcome column with a green 'allowed → installed' branch and a red 'blocked → install aborted' branch.](/images/lazy-security-part-1-supply-chain/safe-chain-flow.png)

*Fig. 3 · safe-chain in one picture. A local proxy in front of every package manager, checked against an open threat-intel feed.*

On a dev machine:

```bash
curl -fsSL https://github.com/AikidoSec/safe-chain/releases/latest/download/install-safe-chain.sh | sh
# restart your shell
npm safe-chain-verify
# expected: OK: Safe-chain works!
```

In CI:

```yaml
- name: Install safe-chain
  run: |
    curl -fsSL https://github.com/AikidoSec/safe-chain/releases/latest/download/install-safe-chain.sh \
      | sh -s -- --install-dir /usr/local/.safe-chain
    echo "$HOME/.safe-chain/bin" >> "$GITHUB_PATH"
- run: npm ci
```

And, the part that quietly does the most work: refuse to install anything younger than 48 hours, because that's the window in which most npm malware is caught and removed:

```bash
export SAFE_CHAIN_MINIMUM_PACKAGE_AGE_HOURS=48
```

## the receipts

The above stack is approximately one afternoon of work: `npm ci` from a committed lockfile, `ignore-scripts=true` with a tiny allowlist, scoped private packages with locked registry resolution, safe-chain in front of every install, minimum package age of 48 hours. It will catch most known-bad packages, kill dependency-confusion at the registry level, and reduce postinstall blast radius to zero for the long tail. It will not catch a maintainer-account compromise that ships clean-looking malware that only activates in production weeks later. Nothing in this post will. The next-tier defenses (sigstore signing, npm provenance, reproducible builds) are real, and Part 3 will name them.

For a startup the delta from this post is moving from "one of the next ten incidents has a non-trivial chance of being yours" to "you would have to be very unlucky." That's the only delta that matters.

If you do one thing this week, go register your npm scope.

---

## diagrams: what i tried

Three diagrams, three different tools, one brief: "explain a supply-chain attack to a tired SRE in one image." Prompts kept short. Results below.

### #1: the napkin contrast (coleam00 excalidraw-diagram skill)

Brief: *"two-panel hand-drawn napkin. Left panel 'what i installed': five direct deps off a 'your app' box. Right panel 'what npm install pulled in': same five direct deps, transitive sprawl under each, one of them is a red `evil-helper@1.2.3`, callout reads 'shipping crypto miners.'"*

Result: [Fig. 1](/images/lazy-security-part-1-supply-chain/dependency-tree-contrast.png). Built with the [`excalidraw-diagram`](https://github.com/coleam00/excalidraw-diagram-skill) skill, a Claude Code skill that enforces a design methodology (depth assessment → pattern mapping → evidence artifacts → mandatory render-and-validate loop) and ships a Playwright-based renderer. The `.excalidraw` source is [downloadable](/images/lazy-security-part-1-supply-chain/dependency-tree-contrast.excalidraw); open it on excalidraw.com to edit.

Two things the skill produced that I wouldn't have prompted for: a semantic color palette (Start/Trigger orange for "your app", Error red for the malicious package, Inactive blue-dashed for the "+N more" bags), and a summary-flow strip at the bottom (`5 direct → 1,200 transitive → 1 malicious → full keychain`) that compresses the post's thesis into nine words. The methodology turns a drawing into an argument.

One install gotcha worth knowing: the skill loads Excalidraw via ESM and the default CDN (`esm.sh`) was unreachable from my environment. One-line patch in `render_template.html` to use `cdn.jsdelivr.net` and it worked.

Verdict: **won for the hero.** The Excalidraw aesthetic earns a place when a post needs a punchline; the skill's methodology adds the second zoom level that elevates a punchline into something that teaches.

### #2: the chainjacking timeline (diagram-design, polished editorial)

Brief: *"Horizontal six-stage timeline. Dark navy `#11141c` ground, coral `#ff6b5a` for the two 'compromise' stages, muted cyan `#5bc0d9` for the four 'victim' stages. Each stage: small timestamp label (T-7d, T-0, T+12m, T+12m02s, T+1h, T+24h), a node on the line, a short title, one-line caption. Title 'a chainjacking attack, in six steps' (lowercase). Italic figcaption. Magazine-infographic feel, no neon, no scanlines."*

Result: [Fig. 2](/images/lazy-security-part-1-supply-chain/chainjacking-timeline.png). Built by a diagram-design subagent. Came back with a ghosted "COMPROMISE / VICTIM" phase label in the background that I hadn't asked for and now wouldn't part with.

Verdict: **won for explaining attacker workflow.** Editorial polish without being a dashboard. The agent's improvisation (the phase label) was the part I would not have prompted my way to.

### #3: the safe-chain flow (diagram-design, five-column system diagram)

Brief: *"Five columns left to right: developer terminal running `npm install lodash-utils`, shell alias intercepting, local proxy as the focal column, Aikido Intel cloud with a lookup arrow, outcome column with a green 'installed' branch and a red 'install aborted' branch. Dark editorial background `#0d1117`. Cyan normal flow, red blocked, green allowed. Legend strip at the bottom."*

Result: [Fig. 3](/images/lazy-security-part-1-supply-chain/safe-chain-flow.png). Same skill as #2. Came back with `harshit.cloud · lazy security` baked into the footer (also unprompted, also welcome) and a clean two-outcome fork that makes the block/allow decision the visual sink.

Verdict: **won for explaining a system.** When the diagram has to show *what a tool does*, this format beats the napkin every time. The napkin is a punchline. This one is a reference.

I did not ship an animated `.gif` for Part 1. The `infographic-gif` skill is the right tool for *quantitative motion*: a funnel decaying, a bar chart counting up. Nothing in Part 1 needed motion to make the point. If Part 3 ends up wanting a "blast radius over 24 hours" visualisation, that's where the GIF goes.



Self-hosting SimpleLogin: own your email aliases for $3 a month
root@harshit.cloud (Harshit Luthra) — Sat, 07 Feb 2026 00:00:00 GMT

Self-hosting SimpleLogin gives you bidirectional email aliases (receive, reply, and send as the alias) for about $3/month on a small VPS, using Docker, Postfix, and Brevo as the SMTP relay. The one trap that eats an afternoon: Postfix won't trust Brevo's TLS cert until you point it at the system CA file. Everything else is just DNS and containers.

I'd been running Cloudflare Email Routing for months. Free. Dead simple. Emails hit my custom domain, forwarded to Gmail. Privacy-friendly aliases without paying a dime.

Then I tried to reply from an alias. Couldn't.

Cloudflare Email Routing is inbound-only. You receive emails at your alias, but when you hit reply, it goes out from your real Gmail address. The whole point of aliasing, gone in one click.

I'd already moved my infrastructure to [a self-hosted Dokploy setup](/blog/netlify-to-dokploy-migration) running on Hetzner. The server was sitting there at 8% CPU. Why not run my own email aliasing too?

Two hours later, I had full bidirectional email aliases running on SimpleLogin. Here's every step, including the TLS trap that almost made me quit.

![Side-by-side sequence diagram showing Cloudflare Email Routing leaking the real Gmail address on reply, while a self-hosted SimpleLogin stack on Hetzner rewrites the reply path so the alias survives in both directions.](/images/self-hosting-simplelogin/hero.png)

*Fig. 1 · Cloudflare can hand you the letter. It just can't post one back without signing your real name.*

## why Cloudflare email routing wasn't enough

Credit where it's due. Cloudflare Email Routing is genuinely great for what it does:

- **Free.** No credit limits, no tier anxiety
- **5-minute setup.** Add MX records, create routes, done
- **Reliable inbound forwarding.** Never lost an email

But the moment you need to reply from an alias or send a new email as your alias, you're stuck. Cloudflare loosened Email Workers' reply restrictions in March 2025 to let you programmatically process and reply to emails. But it's a developer tool for automated responses, not a "hit reply in Gmail" solution.

| Feature | Cloudflare Email Routing | SimpleLogin (Self-Hosted) |
|---------|--------------------------|---------------------------|
| Cost | Free | ~$3/month (VPS) |
| Setup time | 5 minutes | ~2 hours |
| Receive to alias | Yes | Yes |
| Reply from alias | No | Yes |
| Send as alias | No | Yes |
| Custom domains | Yes (CF nameservers required) | Yes (any registrar) |
| Open source | No | Yes |
| PGP encryption | No | Yes |
| Self-hosted option | No | Yes |

If all you need is inbound forwarding, stick with Cloudflare. It's free and it works. But if you want actual email aliases, where you can reply and send and nobody ever sees your real address, you need SimpleLogin.

## what you'll need

Before you start:

- **A VPS** with ports 25, 465, and 443 open (Hetzner, Contabo, etc., around $3/month)
- **A domain** with DNS you control
- **A Brevo account** (free tier: 300 emails/day) for outbound SMTP relay
- **30 minutes of focus** for DNS, plus another hour for the stack

> **Key Insight:** Most residential ISPs and some cloud providers block port 25. Hetzner doesn't by default, but you might need to request it. Check before you start. No port 25, no self-hosted email.

## architecture overview

Here's how the pieces fit together:

```
Inbound Email Flow:
Internet → MX Record → Your Server:25 (Postfix) → SimpleLogin App → Your Mailbox

Outbound Email Flow:
SimpleLogin App → Postfix → Brevo SMTP Relay → Recipient
```

Four containers plus Postfix on the host:

| Container | Role | Port |
|-----------|------|------|
| `sl-db` | PostgreSQL database | 5432 |
| `sl-app` | Web UI + API | 7777 |
| `sl-email` | Email handler (SMTP) | 20381 |
| `sl-job-runner` | Background tasks | n/a |

Plus Postfix running directly on the host, listening on port 25.

## the DNS gauntlet

DNS is where most people give up. Don't. It's just a lot of records. Set them all up at once and verify later.

For a domain like `sl.example.com` with server IP `203.0.113.50`:

### A record

```
Type: A
Name: sl
Value: 203.0.113.50
Proxy: OFF (DNS only)
```

### MX record

```
Type: MX
Name: sl.example.com
Value: sl.example.com
Priority: 10
```

### SPF record

```
Type: TXT
Name: sl.example.com
Value: v=spf1 mx a ip4:203.0.113.50 include:spf.sendinblue.com ~all
```

You need the `include:spf.sendinblue.com` line: Brevo (formerly Sendinblue) sends your outbound mail through that SPF record.

### DKIM record

```
Type: TXT
Name: dkim._domainkey.sl.example.com
Value: v=DKIM1; k=rsa; p=YOUR_DKIM_PUBLIC_KEY
```

You'll generate this key during Docker setup. Come back and add it then.

### DMARC record

```
Type: TXT
Name: _dmarc.sl.example.com
Value: v=DMARC1; p=quarantine; pct=100; adkim=s; aspf=s
```

### PTR record (reverse DNS)

Set this in your hosting provider's panel, not your DNS. It maps your IP back to your domain. Most providers have a "Reverse DNS" or "rDNS" field in the server settings.

```
203.0.113.50 → sl.example.com
```

> **Key Insight:** If you're using Cloudflare DNS, the A record for your mail subdomain **must** be set to "DNS only" (grey cloud). Cloudflare's proxy doesn't pass through SMTP traffic on port 25. Orange cloud = your MX record points to Cloudflare's proxy = mail delivery fails silently.

![Cloudflare DNS panel showing all configured records](/images/self-hosting-simplelogin/sl-dns-records.png)

### verify everything

Don't move on until these pass:

```bash
# MX record
dig MX sl.example.com +short
# Should return: 10 sl.example.com.

# SPF record
dig TXT sl.example.com +short
# Should include: v=spf1 mx a ip4:203.0.113.50 include:spf.sendinblue.com ~all

# A record
dig A sl.example.com +short
# Should return: 203.0.113.50

# PTR record
dig -x 203.0.113.50 +short
# Should return: sl.example.com.
```

## why Brevo? IP reputation is everything

Why not send directly from Postfix? You can. Gmail, Outlook, and Yahoo will just spam-folder it, or reject it outright.

Email deliverability depends on IP reputation. A fresh VPS IP has none. To the big providers, that looks identical to a spammer on a throwaway server. Building reputation takes weeks of careful warm-up. For a personal alias service sending 10 emails a day, it's not worth it.

Brevo's SMTP relay solves this. Your Postfix hands mail to Brevo, and Brevo sends it from IPs with years of established reputation. Your email lands in inboxes, not spam. Free tier: 300 emails/day.

## setting up Brevo

Sign up at [brevo.com](https://www.brevo.com). Then:

1. Go to **Settings > SMTP & API**
2. Generate an SMTP key
3. Note your SMTP login (it's your account email, not a generated username)
4. Add and verify your domain under **Settings > Senders & Domains**

![Brevo SMTP settings page](/images/self-hosting-simplelogin/sl-brevo-smtp.png)

Save the SMTP key. You'll need it for both the SimpleLogin env file and Postfix config.

## docker setup

SSH into your server. Let's build this.

### create the network and directories

```bash
docker network create sl-network

mkdir -p /sl/pgp
mkdir -p /sl/db
mkdir -p /sl/upload
```

### environment file

Create `/sl/simplelogin.env`:

```bash
# Domain
URL=https://sl.example.com
EMAIL_DOMAIN=sl.example.com
SUPPORT_EMAIL=support@sl.example.com
ADMIN_EMAIL=admin@sl.example.com

# Email
EMAIL_SERVERS_WITH_PRIORITY=[(10, "sl.example.com.")]
DKIM_PRIVATE_KEY_PATH=/dkim.key
DKIM_PUBLIC_KEY_PATH=/dkim.pub.key

# Brevo SMTP Relay
POSTFIX_SERVER=host.docker.internal
POSTFIX_PORT=25

# Database
DB_URI=postgresql://sl_user:your_strong_password_here@sl-db:5432/simplelogin

# Flask
FLASK_SECRET=generate_a_long_random_string_here

# Features
DISABLE_ALIAS_SUFFIX=1
NOT_SEND_LINK_TO_SELF=1
ENABLE_SPAM_ASSASSIN=0

# PGP
GNUPGHOME=/sl/pgp
```

Generate your secrets:

```bash
# Flask secret
openssl rand -hex 32

# Database password
openssl rand -hex 16
```

### generate DKIM keys

```bash
openssl genrsa -out /sl/dkim.key 1024
openssl rsa -in /sl/dkim.key -pubout -out /sl/dkim.pub.key

# Get the public key for your DNS record
cat /sl/dkim.pub.key | sed '1d;$d' | tr -d '\n'
```

Copy that output. Go back to your DNS and paste it as the `p=` value in your DKIM TXT record.

### start PostgreSQL

```bash
docker run -d \
  --name sl-db \
  --network sl-network \
  --restart always \
  -e POSTGRES_DB=simplelogin \
  -e POSTGRES_USER=sl_user \
  -e POSTGRES_PASSWORD=your_strong_password_here \
  -v /sl/db:/var/lib/postgresql/data \
  postgres:16
```

### initialize the database

```bash
docker run --rm \
  --name sl-migration \
  --network sl-network \
  --env-file /sl/simplelogin.env \
  -v /sl/dkim.key:/dkim.key:ro \
  -v /sl/dkim.pub.key:/dkim.pub.key:ro \
  simplelogin/app:4.6.5-beta \
  alembic upgrade head

docker run --rm \
  --name sl-init \
  --network sl-network \
  --env-file /sl/simplelogin.env \
  -v /sl/dkim.key:/dkim.key:ro \
  -v /sl/dkim.pub.key:/dkim.pub.key:ro \
  simplelogin/app:4.6.5-beta \
  python init_app.py
```

### start the application containers

```bash
# Web app
docker run -d \
  --name sl-app \
  --network sl-network \
  --restart always \
  --env-file /sl/simplelogin.env \
  --add-host=host.docker.internal:host-gateway \
  -v /sl/dkim.key:/dkim.key:ro \
  -v /sl/dkim.pub.key:/dkim.pub.key:ro \
  -v /sl/upload:/code/static/upload \
  -p 127.0.0.1:7777:7777 \
  simplelogin/app:4.6.5-beta

# Email handler
docker run -d \
  --name sl-email \
  --network sl-network \
  --restart always \
  --env-file /sl/simplelogin.env \
  --add-host=host.docker.internal:host-gateway \
  -v /sl/dkim.key:/dkim.key:ro \
  -v /sl/dkim.pub.key:/dkim.pub.key:ro \
  -v /sl/upload:/code/static/upload \
  -p 127.0.0.1:20381:20381 \
  simplelogin/app:4.6.5-beta \
  python email_handler.py

# Job runner
docker run -d \
  --name sl-job-runner \
  --network sl-network \
  --restart always \
  --env-file /sl/simplelogin.env \
  --add-host=host.docker.internal:host-gateway \
  -v /sl/dkim.key:/dkim.key:ro \
  -v /sl/dkim.pub.key:/dkim.pub.key:ro \
  -v /sl/upload:/code/static/upload \
  simplelogin/app:4.6.5-beta \
  python job_runner.py
```

![Docker containers running healthily](/images/self-hosting-simplelogin/sl-docker-ps.png)

Four containers. All running. But we're not done. Postfix is the piece that actually handles SMTP.

## the Postfix config (and the TLS trap)

This is where I lost two hours. The setup itself is straightforward. The bug that follows is not.

### install Postfix

```bash
apt-get update && apt-get install -y postfix postfix-pgsql libsasl2-modules
```

Choose "Internet Site" when prompted. Set the system mail name to your domain.

### main configuration

Replace `/etc/postfix/main.cf` with:

```ini
# Basic
smtpd_banner = $myhostname ESMTP
biff = no
append_dot_mydomain = no
readme_directory = no
compatibility_level = 3.6

# TLS - Outbound (Postfix → Brevo)
smtp_tls_security_level = encrypt
smtp_tls_note_starttls_offer = yes
smtp_tls_CAfile = /etc/ssl/certs/ca-certificates.crt
smtp_tls_loglevel = 1

# TLS - Inbound (Internet → Postfix)
smtpd_tls_cert_file = /etc/ssl/certs/ssl-cert-snakeoil.pem
smtpd_tls_key_file = /etc/ssl/private/ssl-cert-snakeoil.key
smtpd_tls_security_level = may

# Network
myhostname = sl.example.com
mydomain = sl.example.com
myorigin = $mydomain
mydestination = localhost
mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128 172.16.0.0/12
inet_interfaces = all
inet_protocols = ipv4

# Relay through Brevo
relayhost = [smtp-relay.brevo.com]:587
smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
smtp_sasl_security_options = noanonymous

# Size limits
message_size_limit = 50000000
mailbox_size_limit = 0

# SimpleLogin integration
virtual_mailbox_domains = pgsql:/etc/postfix/pgsql-relay-domains.cf
virtual_mailbox_maps = pgsql:/etc/postfix/pgsql-transport-maps.cf
virtual_alias_maps = pgsql:/etc/postfix/pgsql-transport-maps.cf
transport_maps = pgsql:/etc/postfix/pgsql-transport-maps.cf
```

### the TLS trap

Here's what happened. Everything looked right. Containers running. Postfix installed. DNS verified. Sent a test email to my alias.

Nothing arrived.

Checked the Postfix logs:

```bash
journalctl -u postfix -n 50
```

```
postfix/smtp[12345]: Untrusted TLS connection established to
  smtp-relay.brevo.com[1.2.3.4]:587: TLSv1.3 with cipher
  TLS_AES_256_GCM_SHA384 (256/256 bits)
postfix/smtp[12345]: certificate verification failed for
  smtp-relay.brevo.com: unable to get local issuer certificate
```

**Untrusted TLS connection.** Postfix was connecting to Brevo but refusing to send because it couldn't verify the certificate chain.

The fix? Two lines:

```ini
smtp_tls_security_level = encrypt
smtp_tls_CAfile = /etc/ssl/certs/ca-certificates.crt
```

The `CAfile` line tells Postfix where to find the system's CA certificates. Without it, Postfix has no root certificates to verify Brevo's TLS cert against. It connects, sees an "untrusted" cert, and drops the mail.

If you're on Ubuntu/Debian and the CA file is missing:

```bash
apt-get install -y ca-certificates
update-ca-certificates
```

Then restart Postfix:

```bash
systemctl restart postfix
```

Two hours. Two lines. Classic.

### PostgreSQL lookup files

These let Postfix query SimpleLogin's database to know which domains and addresses to accept.

Create `/etc/postfix/pgsql-relay-domains.cf`:

```ini
hosts = localhost
user = sl_user
password = your_strong_password_here
dbname = simplelogin
query = SELECT domain FROM custom_domain WHERE domain='%s' AND verified=true
  UNION SELECT domain FROM public_domain WHERE domain='%s'
  UNION SELECT '%s' WHERE '%s' = 'sl.example.com' LIMIT 1;
```

Create `/etc/postfix/pgsql-transport-maps.cf`:

```ini
hosts = localhost
user = sl_user
password = your_strong_password_here
dbname = simplelogin
query = SELECT 'smtp:127.0.0.1:20381' FROM alias WHERE email='%s' AND enabled=true
  UNION SELECT 'smtp:127.0.0.1:20381' FROM custom_domain WHERE domain=split_part('%s', '@', 2) AND verified=true
  UNION SELECT 'smtp:127.0.0.1:20381' WHERE split_part('%s', '@', 2) = 'sl.example.com' LIMIT 1;
```

### SASL authentication for Brevo

Create `/etc/postfix/sasl_passwd`:

```
[smtp-relay.brevo.com]:587 your-brevo-login@example.com:your-brevo-smtp-key
```

Lock it down and generate the hash:

```bash
chmod 600 /etc/postfix/sasl_passwd
postmap /etc/postfix/sasl_passwd
```

### expose PostgreSQL port

Postfix runs on the host but needs to reach the Postgres container. Modify the sl-db container to expose the port:

```bash
docker stop sl-db
docker rm sl-db

docker run -d \
  --name sl-db \
  --network sl-network \
  --restart always \
  -e POSTGRES_DB=simplelogin \
  -e POSTGRES_USER=sl_user \
  -e POSTGRES_PASSWORD=your_strong_password_here \
  -v /sl/db:/var/lib/postgresql/data \
  -p 127.0.0.1:5432:5432 \
  postgres:16
```

### start Postfix

```bash
systemctl restart postfix
systemctl enable postfix
```

## nginx reverse proxy

SimpleLogin's web UI runs on port 7777. Put Nginx in front for HTTPS.

```nginx
server {
    server_name sl.example.com;

    location / {
        proxy_pass http://127.0.0.1:7777;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}
```

Then get a real certificate:

```bash
apt-get install -y certbot python3-certbot-nginx
certbot --nginx -d sl.example.com
```

Certbot rewrites the Nginx config to add SSL and sets up auto-renewal. Done.

## first login and lockdown

Go to `https://sl.example.com` and register your admin account using the email you set as `ADMIN_EMAIL` in the env file.

Now make yourself premium and lock the door:

```bash
# Enter the database
docker exec -it sl-db psql -U sl_user -d simplelogin

# Make your account premium (lifetime)
UPDATE users SET lifetime = true WHERE email = 'admin@sl.example.com';

# Disable registration so nobody else can sign up
\q
```

Add to your `simplelogin.env`:

```bash
DISABLE_REGISTRATION=1
```

Restart the app container:

```bash
docker restart sl-app
```

![SimpleLogin dashboard with multiple aliases configured](/images/self-hosting-simplelogin/sl-dashboard-aliases.png)

Your instance. Your aliases. Your data.

## persistence across reboots

Make sure everything survives a server restart:

```bash
# Docker containers (already set with --restart always, but verify)
docker update --restart always sl-db sl-app sl-email sl-job-runner

# Postfix
systemctl enable postfix

# Nginx
systemctl enable nginx
```

Reboot and verify:

```bash
reboot

# After reboot
docker ps
systemctl status postfix
systemctl status nginx
```

## lessons learned

Six things I wish I'd known before starting:

1. **Cloudflare proxy kills mail.** The orange cloud proxies HTTP traffic through Cloudflare's network. SMTP on port 25 doesn't go through that proxy. Grey cloud (DNS only) or your MX records point nowhere useful.

2. **Brevo domain verification is fussy.** Verify your sending domain in Brevo before configuring Postfix. If Brevo doesn't recognize your domain, outbound mail gets rejected at the relay, not at the destination. Hard to debug.

3. **The TLS CA certificate trap is real.** Postfix doesn't use the system CA store by default. You must explicitly point it to `/etc/ssl/certs/ca-certificates.crt`. Without this, outbound relay to Brevo fails silently with "untrusted TLS connection" in the logs.

4. **IP reputation matters more than you think.** Fresh VPS IPs have zero reputation. Using Brevo as a relay piggybacks on their established reputation. Direct send from a new IP = spam folder.

5. **Pin your Postgres version.** Use `postgres:16`, not `postgres:latest`. A major version bump (16 to 17) on a container restart will break your data directory without a manual migration.

6. **Use the app image, not app-ci.** SimpleLogin publishes both `simplelogin/app` and `simplelogin/app-ci`. The `app-ci` image is for their CI/CD pipeline. Use `simplelogin/app` with a specific version tag.

## the proof

Here's what the full flow looks like in practice. Send a test email to your alias:

![Sending a test email to the SimpleLogin alias](/images/self-hosting-simplelogin/sl-inbound-test.png)

It arrives in your mailbox, forwarded through SimpleLogin. Check the headers: mailed by Brevo's relay, signed by your domain:

![Forwarded email showing Brevo relay and domain signature in headers](/images/self-hosting-simplelogin/sl-forwarded-headers.png)

Now the real test. Hit reply. The recipient should see your alias, not your real email:

![Reply sent from the alias address](/images/self-hosting-simplelogin/sl-reply-from-alias.png)

Check the headers on the reply. From: your alias. Signed-by: your domain. Your real address is nowhere in sight:

![Reply headers confirming alias as sender with TLS encryption](/images/self-hosting-simplelogin/sl-reply-headers.png)

SimpleLogin's dashboard confirms the reply went through:

![SimpleLogin dashboard showing successful reply activity on the alias](/images/self-hosting-simplelogin/sl-reply-confirmed.png)

### browser extension bonus

SimpleLogin also ships a browser extension. Visit any site, click the icon, and create an alias on the fly, no need to open the dashboard:

![SimpleLogin browser extension creating an alias on a website](/images/self-hosting-simplelogin/sl-browser-extension.png)

![Browser extension showing existing aliases for the current site](/images/self-hosting-simplelogin/sl-extension-aliases.png)

Between this and the [Dokploy migration](/blog/netlify-to-dokploy-migration), my entire personal infrastructure runs on a single Hetzner box for under $5 a month. Email aliases, five websites, monitoring, backups. All mine. Two hours of setup, one of which was the TLS trap above.



Blocking AI crawlers is the new 'noindex'
root@harshit.cloud (Harshit Luthra) — Wed, 21 Jan 2026 00:00:00 GMT

If you're blocking GPTBot, Anthropic, Perplexity, Gemini — you're trading future reach for short-term control.

## the math

AI search traffic today: ~1%
AI search traffic tomorrow: 25–35%

Let them crawl. Train the discovery layer. Be early.

## common AI crawler user agents

| Crawler | Company |
|---------|---------|
| `GPTBot` | OpenAI |
| `ClaudeBot` / `Anthropic-AI` | Anthropic |
| `PerplexityBot` | Perplexity |
| `Google-Extended` | Google (Gemini) |

## the robots.txt decision

Blocking these crawlers:

```
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /
```

Feels like control. Actually it's invisibility.

## why this matters

When someone asks an AI "how do I do X" and your content isn't in the training data, you don't exist in that conversation.

The sites that trained the discovery layer early will own the AI search results later.

Visibility > invisibility.



Access denied: when your browser extensions look like attack vectors
root@harshit.cloud (Harshit Luthra) — Wed, 31 Dec 2025 00:00:00 GMT

Last week I tried booking a flight on Indigo. Access Denied. Tried MakeMyTrip. Access Denied. Ixigo? Same story. Yatra? Blocked.

My banking apps worked fine. But every travel booking site using Akamai's CDN decided I was public enemy number one. Sometimes the site would load, then the OTP API calls would silently fail. Making a complete fool out of me at checkout.

![Indigo Access Denied](/images/akamai-browser-extensions-blocking/indigo_access_denied.png)

*Fig. 1 · every travel booking site behind Akamai locked me out with the same Access Denied page.*

![MakeMyTrip's booking page returning the same Akamai Access Denied error.](/images/akamai-browser-extensions-blocking/mmt_access_denied.png)

*Fig. 2 · MakeMyTrip served the identical block page, so this wasn't one site's bad config.*

## the debugging rabbit hole

First thought: bad IP from my ISP's CGNAT pool. Changed my IP. Worked for 10 minutes. Then blocked again.

Second thought: maybe Akamai's IP reputation is flagging me. Checked their [Client Reputation lookup](https://www.akamai.com/us/en/clientrep-lookup/).

![Akamai's Client Reputation lookup showing my IP with a clean reputation score.](/images/akamai-browser-extensions-blocking/akamai_repo_ip.png)

*Fig. 3 · Akamai's own Client Reputation lookup rated my IP clean, which ruled out an IP-reputation block.*

Nope. Clean as a whistle.

![IP details showing a Tata Play connection in Bengaluru.](/images/akamai-browser-extensions-blocking/my_ip.png)

*Fig. 4 · my connection was Tata Play in Bengaluru, nothing exotic.*

Google dorking time. Found tons of users globally facing the same issue. Not ISP-specific. Not India-specific. Something else was up.

Then I found [this blog](https://leinss.com/blog/?p=3409) that pointed at browser extensions. Interesting.

## the lightbulb moment

Switched from Arc to Chrome. Still blocked. Because I carried over the same 21 extensions like a digital hoarder.

![The first screen of installed Chrome extensions, a long grid of security tools.](/images/akamai-browser-extensions-blocking/extensions.png)

*Fig. 5 · part of the 21-extension pile I'd carried across browsers.*

![The second screen of installed extensions, mostly OSINT and recon tools.](/images/akamai-browser-extensions-blocking/extensions_2.png)

*Fig. 6 · the rest of the arsenal, heavy on OSINT and recon tools.*

Here's my toolkit: Wappalyzer, Shodan, Trufflehog, DotGit, and a bunch of OSINT/greyhat recon tools. The same extensions I use for security research were making me look like an attacker to Akamai's Bot Manager.

Turned off all extensions. Instant access to every site.

## what's actually happening

Akamai's Bot Manager isn't counting your requests. It's fingerprinting the client environment. Browser extensions can inject JavaScript, mutate the DOM, alter request behavior, and add tracking parameters: all things the client-side fingerprint will flag as bot-shaped, the same way it would flag a scraper or an injection probe.

My security toolkit became my own DoS attack vector. Poetic, really.

Some users reported User-Agent changes helped. I didn't test that. I also didn't have time to debug which of the 21 extensions was the actual culprit. Life's too short for that level of troubleshooting.

## what I'd check first

WAF rules are aggressive by design. Your legitimate security tools look exactly like attack vectors because, well, they kind of are. The line between security researcher and threat actor is thinner than we'd like to admit.

If you're getting blocked by Akamai with a clean IP:

1. Check your extensions first, not your ISP
2. VPN working temporarily? That's behavioral detection, not IP blocking
3. The Client Reputation tool won't catch extension-based triggers
4. Your OSINT toolkit makes CDNs nervous

Infrastructure is meant to keep bad actors out. Sometimes it keeps infrastructure wizards out too. Not fun.



VictoriaLogs vs Loki: real-world benchmarking results
root@harshit.cloud (Harshit Luthra) — Wed, 19 Nov 2025 00:00:00 GMT

On 500 GB of logs over 7 days, on the same hardware: **94% lower query latencies, 37% smaller storage, and under half the CPU and RAM**. The single number that surprised us most was the 12× drop in needle-in-a-haystack search times.

![VictoriaLogs vs Loki: neon-styled cover illustration with VictoriaLogs (cyan, throughput chart, database icon) and Loki (magenta, bar chart, gauge) presented as benchmark contenders](/images/victorialogs-vs-loki/hero.webp)

*Fig. 1 · Two log stores, same hardware, one clear winner on the wall clock.*

## the setup

At Truefoundry we run multi-tenant ML workloads on Kubernetes. The log layer has to deliver fast ad-hoc search across mixed namespaces (often with no good labels to anchor on), sustained 60+ MB/s ingestion during deploys and incidents, and live tailing that doesn't fall behind during a noisy crash loop. It also has to run as a single binary (we don't want a six-component log stack) within a 4 vCPU / 16 GiB node ceiling shared with everything else.

Loki was our default. Past the 1M-active-series mark it started showing 30s+ search latencies and high I/O amplification. So we benchmarked it head-to-head against VictoriaLogs and let the numbers decide.

### the contestants

- **Loki:** Grafana Labs' log store. Compressed chunks, label-based indexing, LogQL. Brilliant Grafana integration; expensive regex scans and Go GC overhead at scale.
- **VictoriaLogs:** VictoriaMetrics' columnar LSM log database. Per-field indices, SIMD search, LogsQL. Single binary, low memory footprint, efficient compression.

### benchmark methodology

| Category | Details |
|---|---|
| Hardware | 4 vCPU / 8 GiB RAM, identical for both, QoS: Guaranteed |
| Log generator | flog → Vector → Loki / VictoriaLogs at 65 MB/s sustained |
| Dataset | ~500 GB over 7 days; mix of unique and duplicated lines across 20 namespaces, 40 apps |
| Retention | 7 days |
| Load test | Locust 2.27.1, 10 virtual users, sustained 43 RPS via `/select/logsql/query` and the Grafana datasource |
| Queries | Stats, Needle in a Haystack, Negative (detailed below) |
| Caching | Block cache disabled on both; pods restarted before each run to simulate cold reads |
| Index tweaks | Defaults on both |

## the headline figure

Before the methodology debate, here's what the seven days produced.



The Grafana panels behind those numbers use the same six metrics for both systems, two very different shapes:

**Loki:**

![Loki Grafana dashboard: CPU usage pinned near 4 vCPU limit, memory holding around 6–7 GB, regular throttling spikes hitting 40–50% during the benchmark window](/images/victorialogs-vs-loki/victorialogs-loki-footprint-loki.png)

**VictoriaLogs:**

![VictoriaLogs Grafana dashboard over the same period: CPU near zero baseline with brief spikes to 1 vCPU, memory flat around 1.3 GB, no throttling visible](/images/victorialogs-vs-loki/victorialogs-loki-footprint-victoria.png)

The memory line is the one that most directly translates into infrastructure cost. At steady state, VictoriaLogs sat around 1.3 GB while Loki held 6–7 GB. Freeing ~5 GB per node is the difference between bin-packing four tenants on a box and seven.

## storage on disk

Same logs, same 7-day retention, identical ingestion path. Loki landed at **501 GB**; VictoriaLogs at **318 GB**, **37% smaller** with no tuning on either side.

The difference is partly the codec (VictoriaLogs uses zstd, Loki defaults to snappy) but mostly the layout. Columnar storage finds redundancy that stream-chunked LSMs don't see; values from the same field compress together far better than values stitched in by line order.

At fleet scale this is a 1 TB volume holding what used to need 1.5 TB.

## query performance

Three query patterns, run against the same 500 GB / 7-day index. Result sets were verified to be identical between the two systems.

### 1. stats: log count over 24 hours

**Purpose:** Total log lines from `app="servicefoundry-server"`.

- **LogQL:** `sum(count_over_time({app="servicefoundry-server"}[24h]))`
- **LogsQL:** `{app="servicefoundry-server"} | stats count()`

| System | Latency |
|---|---:|
| Loki | 2.5s |
| VictoriaLogs | 1.5s |

Aggregate counts hit Loki's strength (label-anchored, no text scan) and Loki still loses by 40% on the wall clock. VictoriaLogs holds its own on label queries; Loki has no answer for the others.

### 2. needle in a haystack: finding one line in 500 GB

**Purpose:** Locate a single static log entry `[UNIQUE-STATIC-LOG] ID=abc123 XYZ` in the `truefoundry` namespace over 7 days.

- **LogQL:** `{namespace="truefoundry", app!="grafana"} |= "[UNIQUE-STATIC-LOG] ID=abc123 XYZ"`
- **LogsQL:** `{namespace="truefoundry", app!="grafana"} "[UNIQUE-STATIC-LOG] ID=abc123 XYZ"`

| System | Latency |
|---|---:|
| Loki | 12s |
| VictoriaLogs | ~900ms |

The single-character difference in syntax (`|=` vs nothing) hides the architectural one. Loki's `|=` is a substring filter run line-by-line over decompressed chunks. VictoriaLogs treats the same string as an index probe. 12 seconds turns into 900 milliseconds on identical hardware.

### 3. negative: proving a string doesn't exist

**Purpose:** Search for a string that doesn't appear anywhere in the dataset. Forces a full scan in both systems.

- **LogQL:** `{namespace="truefoundry"} |= "non-existent log line"`
- **LogsQL:** `{namespace="truefoundry"} "non-existent log line"`

| Dataset | Loki | VictoriaLogs |
|---|---:|---:|
| 500 GB | **Timeout** | 2.2s |
| 300 GB | 2.6s | 266ms |

The negative query is the quiet one. At 300 GB Loki handles it in 2.6 seconds. At 500 GB the resources choke and the query halts and never returns. In production that's the difference between an alert that fires and a dashboard that loads.

## ingestion under pressure

We pushed both with 120 flog replicas to find the ceiling.

| Metric | Loki | VictoriaLogs | Delta |
|---|---:|---:|---:|
| Peak ingestion | 20 MB/s | 66 MB/s | **3× higher** |
| vCPU (sustained) | 4 vCPU, 100% throttled | 2 vCPU peak | 50% lower |
| Memory | ~4 GiB | ~1.3 GiB | 3× lower |

![Loki CPU saturation graph at 4 vCPUs and memory consumption at 4GB during peak ingestion load with 120 flog replicas](/images/victorialogs-vs-loki/victorialogs-loki-cpu-memory-loki.png)

![VictoriaLogs performance graph showing 2 peak vCPU usage and 1.3GB memory consumption during the same ingestion load](/images/victorialogs-vs-loki/victorialogs-loki-performance-victoria.png)

Loki hit the CPU wall first and never recovered, pinned at 100% throttled while still topping out at 20 MB/s. VictoriaLogs absorbed the same firehose at 3× the throughput, on **72% less CPU and 87% less memory**.

## load test under traffic

Locust, 10 concurrent users, simulating real read traffic. VictoriaLogs handled 36% more requests per second, p99 latency was 3.6× faster than Loki under load, and tail latency stayed lower at every percentile we measured.

![Load test results for VictoriaLogs showing 36% higher RPS and 3.6x faster p99 latency with 10 concurrent users at 43 RPS](/images/victorialogs-vs-loki/victorialogs-loki-loadtest-victoria.png)

![Load test results for Loki showing slower response times and lower throughput under the same simulated traffic](/images/victorialogs-vs-loki/victorialogs-loki-loadtest-loki.png)

## why the gap is this big

Four design choices doing most of the work:

1. **Full-text indexing.** Per-token indices skip line-by-line filtering entirely.
2. **Columnar LSM layout.** Reads touch only the columns the query asks for; fewer disk seeks.
3. **Memory discipline.** Lower steady-state overhead means more headroom for everything else.
4. **SIMD search.** Vectorised inner loops on commodity CPUs add up over billions of lines.

## when to pick which

VictoriaLogs is the right pick when text search and grep-style queries are the primary workload, when ad-hoc exploration across large windows matters, when resource efficiency and bin-packing density are real constraints, or when you want fewer knobs to tune in production.

Loki is the right pick when label-based queries dominate and full-text is rare, when deep Grafana ecosystem integration is non-negotiable, or when you already operate Loki at scale and the migration cost outweighs the wins.

For us, on this workload, the resource economics decided it. The freed memory per node became real infrastructure savings within a quarter. 12 seconds turned into 900 milliseconds with no tuning, and that's the number I keep quoting six months later.

## resources

- [Loki Documentation](https://grafana.com/docs/loki/latest/)
- [VictoriaLogs Documentation](https://docs.victoriametrics.com/victorialogs/)
- [Vector Documentation](https://vector.dev/)
- [Grafana Alloy](https://grafana.com/docs/alloy/latest/)



When Netlify killed my free tier: a 15-minute migration to Dokploy
root@harshit.cloud (Harshit Luthra) — Fri, 24 Oct 2025 00:00:00 GMT

Late night. Got this email: **"[Netlify] Your projects have been suspended due to credit limit exceeded."**

Five sites down:

- linkedintel.ai (LinkedIn Sales Intelligence AI for SDR's)
- sachin.cool (rookie website from college time)
- dilharia.love (wedding RSVP site - yes, judge me)
- My personal blog
- A ex-ceo's landing page

![The Netlify email announcing five of my projects were suspended for exceeding the credit limit.](/images/netlify-to-dokploy-migration/dokploy_email.png)

*Fig. 1 · the email that started the 15-minute scramble.*

Netlify moved legacy free tier users to their new 300-credit plan. I burned through it in a week.

![Netlify's upgrade prompt offering 1000 credits for $9 a month after the free tier ran out.](/images/netlify-to-dokploy-migration/dokploy_upgrade_netlify.webp)

*Fig. 2 · the upgrade path Netlify wanted me to take.*

New option: $9/month for 1000 credits, or figure something else out.

I had 15 minutes before my girlfriend woke up. Here's what happened.

## the €3 solution

Hetzner CX22: 2 vCPUs, 4GB RAM, 40GB SSD. **€3.29/month**.

![Hetzner's CX22 plan listing 2 vCPUs, 4GB RAM, and 40GB SSD for €3.29 a month.](/images/netlify-to-dokploy-migration/dokploy_hetzner.png)

*Fig. 3 · the box that replaced all five sites for €3.29 a month.*

Math was simple:

- Netlify: $108/year for credit anxiety
- Dokploy + Hetzner: $42/year for unlimited deploys

![A cost comparison putting Netlify's $108 a year against Dokploy on Hetzner at $42 a year.](/images/netlify-to-dokploy-migration/dokploy_netlify.png)

*Fig. 4 · $108 a year for credit anxiety versus $42 a year for unlimited deploys.*

I'd been [watching this Dokploy video](https://www.youtube.com/watch?v=RoANBROvUeE) the week before. Perfect timing.

## the 15-minute panic deploy

**Minutes 0-5**: Spun up Hetzner in Helsinki. Got the IP. Updated DNS.

**Minutes 5-8**: SSH'd in, ran the Dokploy installer:

```bash
curl -sSL https://dokploy.com/install.sh | sh
```

One command. Dokploy installed Docker, Traefik, PostgreSQL, everything.

**Minutes 8-12**: Connected Git repos. Paste GitHub URL, select branch, done.

![Dokploy's Git integration screen where you paste a GitHub URL and pick a branch.](/images/netlify-to-dokploy-migration/dokploy_git.png)

*Fig. 5 · connecting a repo is a URL and a branch.*

**Minutes 12-15**: Hit deploy on all 5 projects. Watched them come back to life.

![The Dokploy dashboard showing all five projects deploying at once.](/images/netlify-to-dokploy-migration/dokploy_migration.png)

*Fig. 6 · all five projects coming back to life.*

The Fiance woke up. dilharia.love was live.

## what surprised me

SSL just works. Traefik + Let's Encrypt provision certificates automatically. I'm running Cloudflare Full (Strict) mode - zero warnings.

WWW redirects? One checkbox. Netlify charged extra for this.

Logs and monitoring built-in. No Datadog bill. No "$500/month observability platform."

![The Dokploy projects view with SSL, logs, and monitoring built in.](/images/netlify-to-dokploy-migration/dokploy_projects.png)

*Fig. 7 · SSL, logs, and monitoring without a separate bill.*

## the catch

You own the ops. Server goes down? That's on you. No 99.9% SLA.

You handle security: OS updates, SSH keys, backups. I run `apt upgrade` weekly and backup to Backblaze B2 for $0.50/month.

For personal projects? Worth it. For business-critical stuff? Pay for managed services.

## one month later

Server load: 8% CPU. Zero downtime. SSL renewals automatic.

All 5 sites running smoothly: linkedintel.ai pulling data, sachin.cool looking sharp, dilharia.love collecting RSVPs.

Deployed 3 more projects since then. No credit anxiety. No surprise bills.

Total maintenance time: 10 minutes/week.

## related posts

- [AWS Cost Optimization: How We Cut Our Bill by 60%](/blog/aws-cost-optimization-tricks)
- [How I Took Down 30% of Production with One TLS Fingerprinting Rule](/blog/ja4-fingerprinting-network-security)
- [5 Kubernetes Debugging Tricks That Saved My Production](/blog/kubernetes-debugging-tips)



Delivery impersonation: the social engineering vector that just works
root@harshit.cloud (Harshit Luthra) — Fri, 17 Oct 2025 00:00:00 GMT

A few weeks ago, my mum got a WhatsApp call from someone claiming to deliver a Diwali hamper from a bakery she'd never ordered from. They asked for her live location to "route the driver". She sent it. Twenty seconds, full home address handed to a stranger.

## why this attack works

The attack rides three psychological triggers at once. Mentioning a well-known local business creates instant credibility. Gift deliveries during festivals are common and expected, so the pretext doesn't trip anyone's filter. And "I'm outside and need directions now" prompts immediate action before the victim has time to verify anything.

## the attack pattern

```
Attacker: "Hi, I'm from [Popular Local Bakery]. I have a Diwali gift
          hamper for you but I'm having trouble finding your location.
          Could you share your address or live location?"

Victim: Shares full address or WhatsApp live location without verification
```

No order confirmation requested. No delivery tracking number asked for. No verification of any kind.

## why people fall for it

- **Gift context**: during festivals, people expect surprise gifts from friends and family
- **Helpful nature**: most people want to help someone who seems to be doing their job
- **Time pressure**: the implied urgency ("I'm waiting outside") prevents critical thinking
- **Low perceived risk**: sharing an address seems harmless compared to financial data
- **Trust in local brands**: using a known local business name lowers suspicion

## defense strategies

The defense is one habit: don't share an address until you've verified the order exists. Ask for a tracking number, call the business on its public number, ask who sent the gift and check with them. If the driver "needs directions right now", give a landmark, not a pin. Most delivery apps already have in-app chat — there's no good reason a real driver needs your live location over WhatsApp.

## real-world impact

This attack can be used for:

- Physical surveillance and stalking
- Burglary planning (knowing when someone is home)
- Identity theft (address is often used for verification)
- Targeted phishing (now knowing exact location)
- Physical security breaches

## the asks that work

The pretexts that actually get through share a shape. A familiar local business name doing the credibility work. A plausible occasion — Diwali hampers, birthday flowers, Amazon redelivery — that fits the calendar. A small action framed as urgent: "I'm outside, just send the location". Zero technical skill, one phone call, full address.



How I took down 30% of production with one TLS fingerprinting rule
root@harshit.cloud (Harshit Luthra) — Tue, 14 Oct 2025 00:00:00 GMT

Last month I broke production. Blocked 30% of legitimate traffic because I misunderstood how TLS fingerprinting actually works.

Here's the incident, what I got wrong, and what SREs actually need to know about JA4 fingerprinting.

![Timeline of a JA4 rate-limit deploy that turned a confident 2pm push into a 30 percent revenue drop and a 4pm rollback two hours later.](/images/ja4-fingerprinting-network-security/hero.png)

*Fig. 1 · one fingerprint looked like one attacker. it was a tls stack used by 30% of customers.*

## the JA3 problem that hit our monitoring

We'd been using JA3 fingerprinting to track traffic patterns since 2021. Not for blocking, just visibility. Which clients were hitting our APIs, how to correlate requests, that kind of thing.

Late 2023, our dashboards started showing chaos. Chrome traffic was generating thousands of unique fingerprints. Same browser version, different fingerprint every request. Our metrics were useless.

Then in early 2024, all our Go-based internal services health checks started showing up as "unknown clients" in our traffic analysis. Turns out Go's HTTP client randomizes cipher suite ordering. JA3 saw that as thousands of different clients. It was just our Kubernetes health checks.

Our traffic classification was completely broken.

## why JA4 exists

JA3 hashes the TLS ClientHello fields in order. Change the order, change the hash. Chrome randomizes extension ordering. Go randomizes cipher suites. JA3 falls apart.

JA4 sorts everything before hashing. That Go health check that generated 10,000 different JA3 fingerprints? One consistent JA4: `t13d190900_9dc949149365_08c8ecc63e89`

The format is readable too:

```
t13d1516h2_8daaf6152771_02713d6af862
│  │││││    │              │
│  ││││└─── ALPN (h2)      └─ Extension hash
│  │││└──── Extension count (16)
│  ││└───── Cipher count (15)
│  │└────── SNI present (1 = yes, 0 = no)
│  └─────── TLS version (13 = TLS 1.3)
└────────── Protocol (t = TCP, q = QUIC)
```

This fixed our monitoring. Could actually track client types again. Go clients showed up as Go. Chrome showed up as Chrome consistently.

Then I got clever.

## the mistake that cost us $50K

We noticed some fingerprints showing up in suspicious traffic patterns. High request rates, weird timing. Looked like abuse.

I wrote a rule: "If fingerprint = X, rate limit aggressively."

Deployed to production at 2pm on a Tuesday.

By 2:30pm, support was getting complaints. By 3pm, order volume had dropped 30%. By 4pm, I was in the incident channel explaining what went wrong.

**The problem: JA4 fingerprints identify TLS stacks, not individual clients.**

All Chrome 119 browsers on Windows have the same fingerprint. All of them. Every user running that browser/OS combo generates the same JA4.

The suspicious traffic I saw? One bad actor using Chrome 119. My rule caught them, and also every legitimate Chrome 119 user on Windows.

That was 30% of our traffic.

## what JA4 actually tells you

JA4 fingerprints map to:

- Browser + version + OS
- HTTP client library + version + OS
- Any TLS stack implementation

Not to:

- Individual users
- Individual devices
- Individual IP addresses

This seems obvious now, but in the moment, tracking "suspicious fingerprint X" felt like tracking a specific attacker. It wasn't. It was tracking everyone using Chrome 119 on Windows.

## how we fixed our monitoring

After the incident, here's how we actually use JA4:

```python
# For traffic classification and monitoring only
def classify_client(fingerprint):
    """Map JA4 fingerprints to known client types for metrics"""

    # Known patterns from our infrastructure
    patterns = {
        't13d190900_9dc949149365_08c8ecc63e89': 'go-http-client-1.21',
        't13d1516h2_8daaf6152771_02713d6af862': 'chrome-120-macos',
        't13d1415h2_5c6f8a9b3d4e_3f7a8b9c2d1e': 'curl-8.4',
        # ... etc
    }

    return patterns.get(fingerprint, 'unknown')

# Aggregate metrics by client type
def record_request_metrics(request):
    client_type = classify_client(request.ja4_fingerprint)

    metrics.increment('requests.by_client_type', tags={
        'client': client_type,
        'endpoint': request.path,
        'status': request.status_code
    })
```

Now we can see in Grafana:

- "go-http-client suddenly went from 100k req/day to 1M" (scaling issue)
- "chrome-119-macos dropped to zero" (browser update pushed)
- "unknown fingerprint at 10k req/sec" (investigate this)

It's a classification dimension, not an identity.

## the a_b_c format actually matters

JA4 uses `a_b_c` format for a reason. The sections are:

- `a`: protocol, TLS version, SNI, counts, ALPN
- `b`: cipher hash
- `c`: extension hash

Why split it? Because you can match on parts.

We had an issue where some service was rotating through different ciphers on every request. Looked like thousands of different clients in our metrics. The full fingerprint kept changing.

But `a` and `c` stayed constant. Started grouping by `ac` instead:

```python
def normalize_fingerprint(fp):
    """Group by protocol+extensions, ignore cipher variations"""
    parts = fp.split('_')
    return f"{parts[0]}_{parts[2]}"  # a_c

# Now cipher rotation doesn't fragment our metrics
normalized = normalize_fingerprint('t13d190900_5d65cb28da5c_02713d6af862')
# Returns: 't13d190900_02713d6af862'
```

Fixed our cardinality explosion problem.

## weird edge cases we hit

**Case 1: corporate proxies**

Some enterprises terminate TLS at their proxy. All internal traffic from that company shows the same fingerprint. One fingerprint, thousands of users.

Can't use it for any kind of per-client logic. Had to maintain a list of known corporate proxy fingerprints and handle them differently.

**Case 2: Windows XP clients**

Yes, in 2024. Ancient TLS 1.0 fingerprint we'd never seen before. Took two weeks to figure out it was legitimate users in a developing country on old machines.

Almost blocked them as "suspicious" before we investigated.

**Case 3: Go library version drift**

Our microservices were on different Go versions. Go 1.20 and Go 1.21 have different TLS fingerprints. This broke our service mesh traffic analysis until we grouped them properly.

## performance characteristics

Parse time: 0.3 microseconds per fingerprint. Not milliseconds. Microseconds.

We process 50M requests/day. Total JA4 overhead: ~15ms per day.

The implementation is Rust with zero heap allocations. Stack-allocated, no GC. I benchmarked it myself when evaluating whether to deploy it.

For infrastructure at scale, this matters. Adding 0.3μs per request is free in any honest accounting. Adding 1ms per request is a P0 incident.

## how to actually use this in production

**For traffic visibility**: yes. It's great for understanding client distribution, detecting anomalies, tracking deployments.

**For rate limiting**: only if you're very careful. Rate limit by fingerprint + IP + endpoint + time window. Never fingerprint alone.

**For blocking**: don't. Seriously. You'll block legitimate traffic. Use it as one signal among many if you must, but never the only signal.

**For capacity planning**: yes. Track which client types generate what load. When Chrome updates, you'll see the shift.

**For debugging**: yes. "Why is this service getting weird traffic?" Check the fingerprints. Might be a misconfigured client.

## the logging strategy

We log fingerprints with every request now. Adds about 45 bytes per log line.

```json
{
  "timestamp": "2025-10-14T15:23:45Z",
  "endpoint": "/api/v1/users",
  "status": 200,
  "duration_ms": 45,
  "ja4": "t13d1516h2_8daaf6152771_02713d6af862",
  "ja4_client": "chrome-120-macos"
}
```

Costs us about 2GB/day extra in log storage. Worth every penny. We've debugged three production issues with this data in the last month.

## when corporate proxies break everything

If you're behind enterprise proxies that terminate TLS, JA4 is useless for per-client anything. You'll see the proxy's fingerprint for thousands of users.

We maintain a separate config for known proxy environments:

```yaml
# corporate-proxy-exceptions.yaml
proxy_fingerprints:
  - fingerprint: "t13d1819h2_abc123def456_789ghi012jkl"
    company: "BigCorp Inc"
    note: "TLS termination at corporate proxy"
    disable_per_client_logic: true
```

Ugly, but it's reality.

## what's coming that will break this

ECH (Encrypted ClientHello) in TLS 1.3 encrypts the inner ClientHello, hiding SNI and ALPN. The outer hello still carries cipher suites and extensions, so JA4 can still be computed on what's visible. The identifying signal just gets weaker. When ECH adoption hits scale, we'll want a different approach.

Keep an eye on your TLS 1.3 ECH adoption metrics. When it hits 20%+, time to rethink your monitoring strategy.

## what the postmortem changed

I tested the rule in staging with synthetic traffic. Real users matched the "bad" fingerprint and I never saw it coming. Anything fingerprint-shaped needs to be canaried at 1% of real traffic first, with anomaly alerts on drops in known-good fingerprints. "chrome-119-windows traffic dropped 90%" is the kind of signal that would have caught this in five minutes instead of two hours.

Rollback path matters more than the rule itself. We killed ours with a feature flag and still spent 30 minutes draining caches and recovering. If a rule can take down 30% of traffic, the kill switch needs to be a single toggle, tested before the rule ships.

The deeper lesson is that fingerprints are a dimension, not an identity. Group by them and slice metrics with them. Don't target individuals with them.

## should SREs care about JA4?

Yes, but not for the reasons you might think.

It's not about security. It's about observability: understanding your traffic composition and detecting anomalies.

Adding JA4 fingerprints to your logs and metrics gives you another dimension to slice your data. When something weird happens, you can answer "what client type is doing this?" faster.

Just don't rate limit or block based on it alone. That way lies incidents and awkward conversations with your VP of Engineering.



JA4's split format saved our metrics cardinality
root@harshit.cloud (Harshit Luthra) — Tue, 14 Oct 2025 00:00:00 GMT

We had a service that rotated TLS ciphers on every connection. Our client classification metrics exploded to 50k unique fingerprints. Prometheus cardinality alert fired.

## the problem

JA3 gives you one hash. When ciphers rotate, you get a new hash:

```
Request 1: 5c4fba4a0f93c6f2a3e52e9c8d4a7b21
Request 2: 3d8a9f2c4e6b1a7c5f9e3d8b2a6c4e1f
Request 3: 7f2e4a8c3b9d1f6e5a2c8d4b7e9f3a1c
```

Each one looks like a different client in your metrics. Cardinality explosion.

## JA4's solution

JA4 splits the fingerprint into three parts:

```
t13d190900_5d65cb28da5c_02713d6af862
│          │              │
a          b              c

a = protocol + TLS version + counts + ALPN
b = cipher hash (changes on rotation)
c = extension hash
```

When the service rotates ciphers, only `b` changes. `a` and `c` stay constant.

## the fix

Instead of grouping by full fingerprint, group by `ac`:

```python
def normalize_fingerprint(fp):
    parts = fp.split('_')
    return f"{parts[0]}_{parts[2]}"  # ignore cipher part

# Before: 50k unique fingerprints
# After: 47 unique fingerprints
```

Cardinality back to normal. Metrics useful again.

## when this matters

If you track client types in metrics and see unexplained cardinality spikes, check if they're rotating ciphers. The `a_b_c` format lets you ignore the changing parts.

Saved us from having to increase our Prometheus retention limits.



My watchlist: from 70s basements to Victorian crime scenes
root@harshit.cloud (Harshit Luthra) — Sun, 12 Jan 2025 00:00:00 GMT

After a day of debugging, I default to two genres: half-hour sitcoms I've already seen, and detective shows where someone smarter than me is making the deductions. This is the shortlist.

![A dark living room lit only by the purple ambient glow of a flat-screen TV showing a grid of streaming app tiles.](/images/favorite-shows-sitcoms-detective/hero.jpg)

*Fig. 1 · the home screen, also known as the longest-running decision of the night.*

## the sitcoms

### That 70's Show

Teenagers in a basement, roasting each other in a circle. Red's foot, Kelso's stupidity-as-genius. Comfort TV with low cognitive overhead.

### Arrested Development

Rewards rewatching more than any other comedy I've seen. The narrator's deadpan and the planted callbacks are the work; the Bluth family dysfunction is the texture.

### How I Met Your Mother

I have opinions about the finale. Everything before it holds up: Barney's suits, the slap bet, Ted's romantic optimism that the show kept earning back for nine seasons.

## the detective shows

### Sherlock

Cumberbatch's version is the one that reset the genre. The mind palace shots, the rapid deductions, the Watson chemistry. "The game is on" still works.

### Detective Conan

Shinichi Kudo trapped in a kid's body, solving murders that follow him around. Running since 1996 and I'm still on it. The Black Organization arc is the spine.

### High Potential

Newer. A cleaning lady with a high IQ solves crimes the detectives can't. The premise carries it; the writing is finding its register.

I'll watch most things with a tight script and a payoff the writer earned. The two genres above are just where that bar gets cleared most often.



GitHub Actions vs GitLab CI: a practical comparison
root@harshit.cloud (Harshit Luthra) — Fri, 20 Dec 2024 00:00:00 GMT

GitHub Actions is the better default for small-to-medium pipelines, open source, and teams already on GitHub. GitLab CI is the better pick for heavy Docker workflows, large self-hosted runner fleets, and pipelines you need to debug interactively. Here is the data behind that call.

| Dimension | GitHub Actions | GitLab CI |
|---|---|---|
| Config style | Verbose, explicit; matrix builds in one block | Flatter, stage-based; YAML anchors and `include:` |
| Typical Node build | 3-5 min | 4-6 min |
| Reusable components | 20,000+ marketplace actions | `include:` files + your own Docker images |
| Docker builds | Needs buildx/build-push actions | First-class `docker:dind`, just works |
| Interactive debugging | SSH only via third-party action | Native web terminal / SSH into a live job |
| Free tier | 2,000 min/month private, unlimited public | 400 min/month; self-hosted runners unlimited |
| Best fit | Open source, GitHub-hosted, small-medium pipelines | Heavy Docker, large runner fleets, gnarly deploys |

Two years, 50 microservices, two CI platforms running side by side. Some repos on GitHub, some on GitLab, same team writing the YAML for both. Here is what stuck after the marketing slides wore off.

![A six-row comparison of GitHub Actions and GitLab CI across syntax, runners, caching, secrets, ecosystem, and pricing at fifty microservices, with the pricing row marked focal.](/images/github-actions-gitlab-ci-comparison/hero.png)

*Fig. 1 · six rows, two YAMLs, one billing model that ended the debate.*

## syntax and configuration

### GitHub Actions

```yaml
name: CI Pipeline
on:
  push:
    branches: [main]
  pull_request:

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
      - run: npm ci
      - run: npm test
```

The YAML is readable, the marketplace has an action for almost everything, and matrix builds are a single block. The nesting gets verbose once you have reusable workflows, and environment variable precedence is its own small religion.

### GitLab CI

```yaml
stages:
  - test
  - build

test:
  stage: test
  image: node:20
  script:
    - npm ci
    - npm test
  only:
    - main
    - merge_requests
```

Flatter than GitHub's nesting, Docker is a first-class citizen, and the stages concept maps cleanly to how you think about a pipeline. There is no marketplace, so reusable components come from `include:` files and Docker images you assemble yourself.

## performance and speed

### build times

A typical Node.js app on our setup builds in 3 to 5 minutes on GitHub Actions and 4 to 6 minutes on GitLab CI. Close enough that I never picked a platform on speed alone.

### parallelization

Both handle parallel jobs well. GitHub Actions has cleaner syntax for matrix builds:

```yaml
strategy:
  matrix:
    node-version: [18, 20, 22]
    os: [ubuntu-latest, windows-latest]
```

GitLab requires more manual setup for the same result.

## ecosystem and marketplace

### GitHub Actions marketplace

Over 20,000 actions, and the caching one is the example I keep coming back to:

```yaml
- uses: actions/cache@v4
  with:
    path: ~/.npm
    key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}
```

One block, content-addressed cache keyed off the lockfile. The first time you delete the manual cache logic you wrote for GitLab and replace it with this, you feel it.

### GitLab's approach

GitLab does not have a marketplace. You write scripts or use Docker images:

```yaml
test:
  image: node:20
  cache:
    key: ${CI_COMMIT_REF_SLUG}
    paths:
      - node_modules/
  script:
    - npm ci
    - npm test
```

More control, but more work.

## docker integration

### GitLab CI wins here

GitLab CI was built with Docker in mind:

```yaml
build:
  image: docker:latest
  services:
    - docker:dind
  script:
    - docker build -t myapp .
    - docker push myapp
```

It just works. No weird permissions issues.

### GitHub Actions

Needs more setup for Docker.

```yaml
- name: Set up Docker Buildx
  uses: docker/setup-buildx-action@v3
- name: Build and push
  uses: docker/build-push-action@v5
  with:
    context: .
    push: true
    tags: myapp:latest
```

Works fine, but requires more marketplace actions.

## secrets management

### GitHub

```yaml
env:
  API_KEY: ${{ secrets.API_KEY }}
```

Simple. Secrets are org/repo scoped. Works well.

### GitLab

```yaml
variables:
  API_KEY: $CI_DEPLOY_TOKEN
```

More flexible with group-level variables and environments. Better for complex setups.

## cost

GitHub Actions gives private repos 2,000 minutes/month on the free tier, public repos are unlimited, and overage is $0.008/minute. GitLab SaaS gives 400 minutes/month free and charges $10 per 1,000 additional minutes, but self-hosted runners are unlimited. If you can run your own runners, GitLab gets cheaper fast at scale. If you can't, GitHub's free tier outlasts it.

## self-hosted runners

### GitHub

```bash
./config.sh --url https://github.com/org/repo --token TOKEN
./run.sh
```

Setup is straightforward. Runners are repo or org-scoped.

### GitLab

```bash
gitlab-runner register
gitlab-runner run
```

More flexible. Can be project, group, or instance-wide. Better for large organizations.

## debugging experience

GitHub Actions has clear, searchable logs, lets you re-run individual jobs, and exposes a debug mode behind two secrets. You can SSH into a runner via a third-party action, but it is not a native feature.

GitLab is the one I reach for when a pipeline is genuinely stuck. The log viewer is good, individual job retries are good, but the real difference is interactive debugging. SSH into the runner mid-job, or open a web terminal from the failed job in your browser, and poke at the filesystem while the build is still alive. The first time you do this on a Docker-in-Docker failure that only repros on CI, you stop missing it everywhere else.

## when to pick which

GitHub Actions wins when you are already on GitHub, want the marketplace, and your pipelines are small to medium. GitLab CI wins when your Docker workflows are non-trivial, your runner fleet is large, your deployment strategies are gnarly, or you need to debug pipelines without a redeploy loop.

## my setup

I use both. GitHub Actions for open-source and frontend, GitLab CI for infrastructure code and the deployments that involve five stages and a manual approval.

## common pitfalls

GitHub Actions has a 6-hour hosted-runner job timeout, a 90-day artifact retention default (configurable up to 400 days for public repos, 90 for private), and tight concurrent-job limits on the free tier. Plan around them or pay.

GitLab's shared runners get sluggish at peak, Docker builds need `docker:dind` as a service container, and CI/CD variable precedence has at least six rules you will need to read twice. The one that bites me most: project-level variables silently override group-level ones with the same name.

## migration tips

### GitHub to GitLab

```yaml
# GitHub
- uses: actions/checkout@v4

# GitLab equivalent
git clone $CI_REPOSITORY_URL
cd $CI_PROJECT_NAME
```

### GitLab to GitHub

Most scripts translate directly. The win is collapsing a few of them into marketplace actions you no longer have to maintain.

Starting fresh, pick whichever platform already hosts your code. The integration tax of running CI on the other vendor outweighs every syntax preference in this post. Whichever one you pick, the only investment that pays back is making the pipeline fast. A slow CI is worse than no CI; it just costs more to ignore.



Git interactive rebase: clean up your commit history
root@harshit.cloud (Harshit Luthra) — Thu, 19 Dec 2024 00:00:00 GMT

Discovered `git rebase -i` today. Turns five "fix typo" commits into one clean commit before the PR review notices.

## the problem

My commit history looked like this:

```
fix typo
WIP
fix typo again
actually fixed it
remove console.log
```

Not exactly professional for a PR review.

## the solution

Interactive rebase lets you edit, squash, and reorder commits:

```bash
# Rebase last 5 commits
git rebase -i HEAD~5

# Or rebase everything since branching from main
git rebase -i main
```

This opens your editor with:

```
pick 1a2b3c4 fix typo
pick 5d6e7f8 WIP
pick 9g0h1i2 fix typo again
pick 3j4k5l6 actually fixed it
pick 7m8n9o0 remove console.log

# Commands:
# p, pick = use commit
# r, reword = use commit, but edit message
# e, edit = use commit, but stop for amending
# s, squash = meld into previous commit
# f, fixup = like squash, but discard commit message
# d, drop = remove commit
```

## my workflow

Change it to:

```
pick 1a2b3c4 fix typo
fixup 5d6e7f8 WIP
fixup 9g0h1i2 fix typo again
fixup 3j4k5l6 actually fixed it
fixup 7m8n9o0 remove console.log
```

Result: one clean commit.

## patterns

**Reword commit messages:**
```
pick 1a2b3c4 fix typo
reword 5d6e7f8 add user authentication
```

**Reorder commits:**
```
pick 3j4k5l6 add tests
pick 1a2b3c4 add feature
```

**Split a commit:**
```
edit 1a2b3c4 huge commit with multiple changes
```

Then:
```bash
git reset HEAD^
git add file1.js
git commit -m "feat: add feature A"
git add file2.js
git commit -m "feat: add feature B"
git rebase --continue
```

## warning

Don't rebase commits that have already been pushed to a shared branch. You'll rewrite history under your team and force everyone else to resolve conflicts they didn't cause.

Safe:
```bash
# Your feature branch, not pushed yet
git rebase -i main
```

Dangerous:
```bash
# Main branch that others use
git checkout main
git rebase -i HEAD~5  # pushed commits — leave these alone
```

## useful aliases

Add to your `~/.gitconfig`:

```ini
[alias]
    # Interactive rebase with the given number of latest commits
    rb = "!f() { git rebase -i HEAD~$1; }; f"
    
    # Rebase on main
    rbm = "!git fetch origin main && git rebase -i origin/main"
```

Now you can do:
```bash
git rb 5        # Rebase last 5 commits
git rbm         # Rebase on main
```



Prometheus and Grafana: from zero to production monitoring
root@harshit.cloud (Harshit Luthra) — Wed, 18 Dec 2024 00:00:00 GMT

We started shipping monitoring after a string of outages where customers paged us before our own dashboards did. This is the stack we landed on, written like you're standing it up tomorrow.

![Prometheus pulls metrics from exporters running next to each service, stores them in its TSDB, then fans out to Alertmanager for paging and Grafana for dashboards.](/images/prometheus-grafana-monitoring-guide/hero.png)

*Fig. 1 · five boxes do most of the work; the other ten you'll add later are mostly for taste.*

## why Prometheus and Grafana

After trying CloudWatch, Datadog, and New Relic, we landed on Prometheus and Grafana for the same reason most teams do. Prometheus is open source and pull-based, and it fits Kubernetes without protest. Grafana puts the dashboards on top, talks to almost anything, and costs nothing. Self-hosted, the bill is server time. The commercial options were running us $500 a month and climbing.

## setup with Docker Compose

For development or small deployments, Docker Compose is perfect:

```yaml
services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--storage.tsdb.retention.time=30d'
    ports:
      - "9090:9090"
    restart: unless-stopped

  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    volumes:
      - grafana_data:/var/lib/grafana
      - ./grafana/provisioning:/etc/grafana/provisioning
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=changeme
      - GF_USERS_ALLOW_SIGN_UP=false
    ports:
      - "3000:3000"
    restart: unless-stopped
    depends_on:
      - prometheus

  node_exporter:
    image: prom/node-exporter:latest
    container_name: node_exporter
    command:
      - '--path.procfs=/host/proc'
      - '--path.sysfs=/host/sys'
      - '--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)'
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    ports:
      - "9100:9100"
    restart: unless-stopped

volumes:
  prometheus_data:
  grafana_data:
```

Run it:

```bash
docker-compose up -d
```

Done. Prometheus is on `:9090`, Grafana on `:3000`.

## prometheus configuration

Create `prometheus.yml`:

```yaml
global:
  scrape_interval: 15s
  evaluation_interval: 15s
  external_labels:
    cluster: 'production'
    region: 'us-east-1'

# Alertmanager configuration (we'll get to this)
alerting:
  alertmanagers:
    - static_configs:
        - targets: ['alertmanager:9093']

# Load rules
rule_files:
  - 'alerts/*.yml'

# Scrape configurations
scrape_configs:
  # Prometheus itself
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  # Node metrics (CPU, memory, disk)
  - job_name: 'node'
    static_configs:
      - targets: ['node_exporter:9100']
        labels:
          instance: 'production-server-1'

  # Example: monitoring a web app
  - job_name: 'webapp'
    static_configs:
      - targets: ['webapp:8080']
    metrics_path: '/metrics'
```

## instrumenting your application

### Node.js / Express example

```javascript
const express = require('express');
const promClient = require('prom-client');

const app = express();
const register = new promClient.Registry();

// Collect default metrics (CPU, memory, event loop)
promClient.collectDefaultMetrics({ register });

// Custom metrics
const httpRequestDuration = new promClient.Histogram({
  name: 'http_request_duration_seconds',
  help: 'Duration of HTTP requests in seconds',
  labelNames: ['method', 'route', 'status_code'],
  buckets: [0.1, 0.5, 1, 2, 5]
});

const httpRequestTotal = new promClient.Counter({
  name: 'http_requests_total',
  help: 'Total number of HTTP requests',
  labelNames: ['method', 'route', 'status_code']
});

register.registerMetric(httpRequestDuration);
register.registerMetric(httpRequestTotal);

// Middleware to track requests
app.use((req, res, next) => {
  const start = Date.now();
  
  res.on('finish', () => {
    const duration = (Date.now() - start) / 1000;
    const labels = {
      method: req.method,
      route: req.route?.path || req.path,
      status_code: res.statusCode
    };
    
    httpRequestDuration.observe(labels, duration);
    httpRequestTotal.inc(labels);
  });
  
  next();
});

// Metrics endpoint
app.get('/metrics', async (req, res) => {
  res.set('Content-Type', register.contentType);
  res.end(await register.metrics());
});

app.listen(8080, () => {
  console.log('Server running on :8080');
  console.log('Metrics available at /metrics');
});
```

### Python / Flask example

```python
from flask import Flask, request
from prometheus_client import Counter, Histogram, generate_latest
import time

app = Flask(__name__)

# Metrics
request_count = Counter(
    'http_requests_total',
    'Total HTTP requests',
    ['method', 'endpoint', 'status']
)

request_duration = Histogram(
    'http_request_duration_seconds',
    'HTTP request duration',
    ['method', 'endpoint']
)

@app.before_request
def before_request():
    request.start_time = time.time()

@app.after_request
def after_request(response):
    duration = time.time() - request.start_time
    
    request_count.labels(
        method=request.method,
        endpoint=request.endpoint,
        status=response.status_code
    ).inc()
    
    request_duration.labels(
        method=request.method,
        endpoint=request.endpoint
    ).observe(duration)
    
    return response

@app.route('/metrics')
def metrics():
    return generate_latest()

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=8080)
```

## the queries you'll actually use

### CPU usage

```promql
# CPU usage percentage
100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# Per core
rate(node_cpu_seconds_total{mode!="idle"}[5m])
```

### memory usage

```promql
# Memory usage percentage
(1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100

# Available memory
node_memory_MemAvailable_bytes / 1024 / 1024 / 1024
```

### disk usage

```promql
# Disk usage percentage
(1 - (node_filesystem_avail_bytes / node_filesystem_size_bytes)) * 100

# Filter to only important mounts
(1 - (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"})) * 100
```

### HTTP request rate

```promql
# Requests per second
rate(http_requests_total[5m])

# By status code
sum by (status_code) (rate(http_requests_total[5m]))
```

### HTTP latency

```promql
# 95th percentile latency
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))

# Average latency
rate(http_request_duration_seconds_sum[5m]) / rate(http_request_duration_seconds_count[5m])
```

## grafana dashboards

### import community dashboards

1. Go to Grafana → Dashboards → Import
2. Use these IDs:
   - **1860**: Node Exporter Full
   - **3662**: Prometheus 2.0 Overview
   - **7362**: MySQL Overview

Or create custom dashboards with the queries above.

### auto-provisioning the datasource

Create `grafana/provisioning/datasources/prometheus.yml`:

```yaml
apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    editable: false
```

## alerting that actually works

Create `alerts/rules.yml`:

```yaml
groups:
  - name: system_alerts
    interval: 30s
    rules:
      # High CPU
      - alert: HighCPUUsage
        expr: 100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High CPU usage on {{ $labels.instance }}"
          description: "CPU usage is {{ $value }}%"

      # High Memory
      - alert: HighMemoryUsage
        expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100 > 85
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High memory usage on {{ $labels.instance }}"
          description: "Memory usage is {{ $value }}%"

      # Disk Space
      - alert: DiskSpaceLow
        expr: (1 - (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"})) * 100 > 85
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Disk space low on {{ $labels.instance }}"
          description: "Disk usage is {{ $value }}%"

      # Service Down
      - alert: ServiceDown
        expr: up == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Service {{ $labels.job }} is down"
          description: "{{ $labels.instance }} has been down for 1 minute"

      # High Error Rate
      - alert: HighErrorRate
        expr: (sum(rate(http_requests_total{status_code=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))) * 100 > 5
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "High error rate detected"
          description: "Error rate is {{ $value }}%"
```

## production considerations

### data retention

Prometheus stores data locally. Retention is set via command-line flags (which is why the Compose example passes them in `command:`), not in `prometheus.yml`:

```bash
--storage.tsdb.retention.time=30d
--storage.tsdb.retention.size=50GB
```

For longer retention, use:

- **Thanos**: Distributed Prometheus
- **Grafana Mimir**: Multi-tenant, horizontally scalable Prometheus; the actively-maintained successor to Cortex
- **VictoriaMetrics**: Drop-in replacement, more efficient

### high availability

Run multiple Prometheus instances with identical configs. Use a load balancer or Thanos for deduplication.

### security

```yaml
# Add basic auth
global:
  external_labels:
    cluster: 'production'

# In Grafana datasource
basicAuth: true
basicAuthUser: prometheus
basicAuthPassword: supersecret
```

Better: Put behind VPN or use mutual TLS.

## common gotchas

Five things bite people in production. Scrape interval below 10s is almost never what you want and quietly burns disk. Too many labels balloon cardinality and your queries get slow before you notice. Pick the right metric type: Counter for monotonic, Gauge for point-in-time, Histogram for distributions, and not the other way around. Monitoring without alerting is a screensaver. And the inverse: too many alerts and the on-call learns to ignore them, which is worse than no alerts at all.

## the golden signals

If you only monitor four things, monitor Google's golden signals: latency, traffic, errors, saturation. Latency is how long requests take. Traffic is request rate. Errors is the rate of failed requests. Saturation is how full the system is on CPU, memory, and disk. Everything else is a refinement on these four.

## what to add next

Add exporters for the data stores you actually use: MySQL, Redis, Postgres, whichever queue you're on. Wire Alertmanager to Slack and PagerDuty so the alerts land somewhere a human reads. Write a one-line runbook link in every alert annotation so the page tells the on-call what to do. Back up the Prometheus data directory; the whole point of long retention is gone if a disk failure wipes it. A dashboard you never look at is graphs heating the data center.




Bash parameter expansion: string manipulation without sed and awk
root@harshit.cloud (Harshit Luthra) — Tue, 17 Dec 2024 00:00:00 GMT

Learned that Bash has built-in string manipulation that's way faster than calling `sed`, `awk`, or `cut`.

## the old way (slow)

```bash
filename="document.pdf"
name=$(echo "$filename" | sed 's/\.[^.]*$//')  # document
ext=$(echo "$filename" | sed 's/^.*\.//')      # pdf
```

Each `echo | sed` spawns a new process. Slow in loops.

## the new way (fast)

```bash
filename="document.pdf"
name="${filename%.*}"    # document
ext="${filename##*.}"    # pdf
```

No external processes. Pure Bash.

## common patterns

### remove file extension

```bash
file="archive.tar.gz"
echo "${file%.*}"       # archive.tar
echo "${file%%.*}"      # archive (remove all extensions)
```

`%` = remove from end (shortest match)
`%%` = remove from end (longest match)

### get file extension

```bash
file="archive.tar.gz"
echo "${file#*.}"       # tar.gz
echo "${file##*.}"      # gz (last extension only)
```

`#` = remove from start (shortest match)
`##` = remove from start (longest match)

### string replacement

```bash
path="/home/user/documents/file.txt"

# Replace first occurrence
echo "${path/user/admin}"        # /home/admin/documents/file.txt

# Replace all occurrences
echo "${path//o/0}"              # /h0me/user/d0cuments/file.txt

# Replace at start
echo "${path/#\/home/\/root}"    # /root/user/documents/file.txt

# Replace at end
echo "${path/%.txt/.md}"         # /home/user/documents/file.md
```

### default values

```bash
# Use default if variable is unset or empty
echo "${VAR:-default}"

# Assign default if unset
echo "${VAR:=default}"

# Error if unset
echo "${VAR:?Variable is required}"

# Use alternate value if set
echo "${VAR:+value_if_set}"
```

### substring extraction

```bash
text="Hello World"
echo "${text:0:5}"       # Hello (from pos 0, length 5)
echo "${text:6}"         # World (from pos 6 to end)
echo "${text: -5}"       # World (last 5 chars, note the space!)
echo "${text::-6}"       # Hello (remove last 6 chars)
```

### string length

```bash
text="Hello World"
echo "${#text}"          # 11
```

### case conversion

```bash
text="Hello World"
echo "${text^^}"         # HELLO WORLD (all uppercase)
echo "${text,,}"         # hello world (all lowercase)
echo "${text^}"          # Hello World (first char uppercase)
echo "${text,}"          # hello World (first char lowercase)
```

## real-world example

Before (slow with multiple processes):

```bash
#!/bin/bash
for file in *.log; do
    name=$(basename "$file" .log)
    date=$(echo "$name" | cut -d'-' -f1)
    gzip "$file"
    mv "${file}.gz" "archive-${date}.gz"
done
```

After (fast, pure Bash):

```bash
#!/bin/bash
for file in *.log; do
    name="${file%.log}"      # Remove .log
    date="${name%%-*}"       # Get everything before first -
    gzip "$file"
    mv "${file}.gz" "archive-${date}.gz"
done
```

## cheat sheet

| Pattern | Effect | Example |
|---------|--------|---------|
| `${var%pattern}` | Remove shortest match from end | `${file%.txt}` |
| `${var%%pattern}` | Remove longest match from end | `${file%%.*}` |
| `${var#pattern}` | Remove shortest match from start | `${file#*/}` |
| `${var##pattern}` | Remove longest match from start | `${file##*/}` |
| `${var/old/new}` | Replace first occurrence | `${text/foo/bar}` |
| `${var//old/new}` | Replace all occurrences | `${text//foo/bar}` |
| `${var:offset:length}` | Substring | `${text:0:5}` |
| `${#var}` | Length | `${#text}` |
| `${var^^}` | Uppercase | `${text^^}` |
| `${var,,}` | Lowercase | `${text,,}` |

## why this matters

In a script processing 10,000 files:

- **With external commands**: ~5 minutes
- **With parameter expansion**: ~10 seconds

That's a 30x speedup.

## one gotcha

Be careful with spaces in substring extraction:

```bash
text="Hello"
echo "${text: -3}"    # llo (CORRECT - note the space)
echo "${text:-3}"     # Hello (WRONG - default value syntax!)
```

The log-rotation script above went from 4m12s to 8s after the rewrite. Same files, same disk.



Five Kubernetes debugging tricks that saved my production
root@harshit.cloud (Harshit Luthra) — Sun, 15 Dec 2024 00:00:00 GMT

The first time I got paged for a `CrashLoopBackOff` it was 03:47, the pod was on its 14th restart, and `kubectl logs` was returning a perfectly clean six-line startup banner with no error in sight. I sat staring at that output for about twenty minutes before someone on Slack asked, casually, whether I'd tried `--previous`. Reader, I had not. Once you've made that mistake once, you stop making it. The five flags below are what I now reach for before anything else, in roughly the order I reach for them.

![Decision tree mapping five Kubernetes pod symptoms to the kubectl command that diagnoses each, drawn for a 3 a.m. on-call brain.](/images/kubernetes-debugging-tips/hero.png)

*Fig. 1 · the chart I wish someone had taped to the wall before my first on-call shift.*

## the crashloopbackoff that lies to you

The pod has restarted 14 times. You run `kubectl logs` and you get the logs of the *current* container, which is the one that hasn't crashed yet because it just started. The interesting bytes (the panic, the missing env var, the OOM at byte 1) are in the previous container's stdout, and they're a single flag away.

```bash
kubectl logs  --previous
```

Add `-c ` if it's a multi-container pod, because the default container is rarely the one that died. I have wasted hours on this exact omission.

## ephemeral debug containers

There used to be a ritual: edit the Dockerfile, add `curl` and `dig` and `tcpdump`, push, wait for CI, redeploy, exec in, debug, then forget to take any of it back out and ship a 900 MB image to prod. As of 1.25 (beta-default since 1.23) you don't have to. `kubectl debug` attaches an ephemeral container to a running pod with whatever image you want (sharing the network namespace, and the PID namespace when you pass `--target` or the pod has `shareProcessNamespace: true`) without touching the original container.

```bash
kubectl debug -it  --image=nicolaka/netshoot
```

`netshoot` is the standard kit: `dig`, `curl`, `tcpdump`, `iperf`, `mtr`, the works. The container vanishes when you detach. Your prod image stays the size it was supposed to be.

## ask the scheduler, don't guess

A pod stuck in `Pending` is the scheduler telling you, very politely, that none of your nodes will have it. You can stare at node taints all day or you can read the events for the pod itself, which spell out the reason in English.

```bash
kubectl get events --field-selector involvedObject.name=
```

`0/12 nodes are available: 8 Insufficient memory, 4 node(s) had untolerated taint`. That's the answer. The scheduler is the most articulate component in the cluster as long as you ask it directly.

## a throwaway pod for network policy work

Network policies are the part of Kubernetes that fail silently. The packet just doesn't arrive, and there's no log line that says *I dropped your SYN because of policy `default-deny-egress` in namespace `payments`*. The cheapest way to figure out what's reachable from where is to land a pod in the namespace you care about and try.

```bash
kubectl run test-pod --rm -it --image=nicolaka/netshoot -- /bin/bash
```

`--rm` cleans up when you exit, which matters because otherwise you will accumulate seven `test-pod-2` pods in `default` and one of them will eventually become the reason a node fills up. From inside, `curl` and `nc` your way through the policy until something connects.

## sort the noisy neighbours

When the cluster feels slow and nobody knows why, the answer is usually one workload in one namespace eating more than its share. `kubectl top` with a sort flag is the single fastest way to find them.

```bash
kubectl top pods --all-namespaces --sort-by=memory
```

Swap `memory` for `cpu` depending on what's burning. Requires `metrics-server` to be installed, which it almost always is, and which is worth installing the day you bring up a cluster if it isn't.

## the one I forget I have

`describe` is so obvious nobody writes about it, and so dense that nobody reads its full output. It includes the events, the resource limits, the volume mounts, the readiness probe definition, the last termination reason, and the QoS class: all the things you were about to run five separate commands to find.

```bash
kubectl describe pod  | less
```

Pipe to `less` because the output is longer than your terminal and the events at the bottom are usually where the answer is. Read from the bottom up if you're in a hurry.

---

The pattern across all five is the same. Kubernetes is unusually good at telling you what went wrong, in plain prose, in a place you have to know to look. The flag is always `--previous`. The answer is always in `events`. The container is always the wrong one by default. Memorise the five commands above and most pages stop being mysteries and start being typing exercises.

Most pages resolve with these five. The ones that don't — a control plane quietly falling over, a CNI dropping packets under load, a cluster that's "healthy" by every dashboard except the one metric users actually feel — are usually why teams bring in outside help. That's the kind of Kubernetes debugging I take on as [independent infrastructure consulting](https://k8s.org.in).

Next time the alert fires at 03:47, the first thing you type is `kubectl logs  --previous -c `. The second thing is `kubectl describe pod  | less`. If the answer isn't in those two outputs, you actually have a problem.



Docker volume debugging: finding where your data actually lives
root@harshit.cloud (Harshit Luthra) — Sat, 14 Dec 2024 00:00:00 GMT

Spent 2 hours debugging why data wasn't persisting. Turns out, understanding Docker volumes is the actual job.

## the problem

I had a container with a volume mount, but couldn't figure out where the data was actually stored on the host:

```bash
docker run -v mydata:/data myapp
```

Where is `mydata`? What's inside it?

## the solution

### find volume location

```bash
# List all volumes
docker volume ls

# Inspect a specific volume
docker volume inspect mydata
```

Output:
```json
[
    {
        "CreatedAt": "2024-12-14T10:30:00Z",
        "Driver": "local",
        "Labels": {},
        "Mountpoint": "/var/lib/docker/volumes/mydata/_data",
        "Name": "mydata",
        "Options": {},
        "Scope": "local"
    }
]
```

The `Mountpoint` tells you exactly where the data lives.

### view volume contents (the trick)

You can't just `cd` to that path (permission denied). Instead, use a temporary container:

```bash
docker run --rm -v mydata:/data alpine ls -la /data
```

Or for interactive browsing:

```bash
docker run --rm -it -v mydata:/data alpine sh
cd /data
ls -la
```

## a better named-volume workflow

Create a simple alias:

```bash
# Add to ~/.bashrc or ~/.zshrc
alias dvol='docker run --rm -it -v'
```

Usage:
```bash
# Browse any volume interactively
dvol mydata:/data alpine sh

# Quick listing
dvol mydata:/data alpine ls -la /data

# Check file contents
dvol mydata:/data alpine cat /data/config.json

# Copy file out of volume
docker run --rm -v mydata:/data -v $(pwd):/backup alpine cp /data/important.txt /backup/
```

## debugging bind mounts

For bind mounts (host path to container):

```bash
docker run -v /host/path:/container/path myapp
```

To see what the container actually sees:

```bash
docker exec -it container_name ls -la /container/path
```

## common volume issues

### volume not mounting

```bash
# Check if volume exists
docker volume ls | grep mydata

# Create it if missing
docker volume create mydata
```

### wrong permissions

```bash
# Check ownership in volume
docker run --rm -v mydata:/data alpine ls -ln /data

# Fix permissions (if needed)
docker run --rm -v mydata:/data alpine chown -R 1000:1000 /data
```

### dangling volumes

```bash
# List dangling volumes (not used by any container)
docker volume ls -f dangling=true

# Remove them
docker volume prune

# this deletes data — make sure nothing important is dangling
```

### volume vs bind mount confusion

```bash
# Named volume (managed by Docker)
-v mydata:/data

# Bind mount (you manage the host path)
-v /host/path:/data
-v $(pwd):/data

# Anonymous volume (Docker creates and manages)
-v /data
```

## patterns

### 1. backup a volume

```bash
# Backup volume to tar file
docker run --rm \
  -v mydata:/data \
  -v $(pwd):/backup \
  alpine tar czf /backup/mydata-backup.tar.gz -C /data .
```

### 2. restore a volume

```bash
# Restore from backup
docker run --rm \
  -v mydata:/data \
  -v $(pwd):/backup \
  alpine tar xzf /backup/mydata-backup.tar.gz -C /data
```

### 3. copy volume to another

```bash
# Copy all data from vol1 to vol2
docker run --rm \
  -v vol1:/source:ro \
  -v vol2:/dest \
  alpine sh -c "cp -av /source/. /dest/"
```

### 4. clone a volume

```bash
# Create new volume as copy of existing
docker volume create vol2
docker run --rm \
  -v vol1:/source:ro \
  -v vol2:/dest \
  alpine cp -av /source/. /dest/
```

## real-world example

I was debugging a database that wasn't persisting data:

```bash
# Check if volume exists
docker volume inspect postgres_data
# Error: No such volume

# the docker-compose.yml had a typo
# it said: postgress_data (3 s's)
# should be: postgres_data (2 s's)

# found all volumes
docker volume ls
# found: postgress_data (the typo)

# Renamed it
docker volume create postgres_data
docker run --rm \
  -v postgress_data:/source \
  -v postgres_data:/dest \
  alpine cp -av /source/. /dest/

# Removed the typo volume
docker volume rm postgress_data
```

## useful one-liners

```bash
# Find which containers use a volume
docker ps -a --filter volume=mydata

# Remove all stopped containers' volumes
docker container prune -f && docker volume prune -f

# List volumes with size (requires Docker 20.10+)
docker system df -v

# Find volumes larger than 1GB
docker system df -v | awk '$4 > 1000'
```

## the gotcha I learned

Docker Compose creates volumes with prefixes:

```yaml
# docker-compose.yml
volumes:
  mydata:
```

Creates volume named: `projectname_mydata`

To use a specific name:

```yaml
volumes:
  mydata:
    name: mydata
```



kubectl JSONPath: extract exactly what you need
root@harshit.cloud (Harshit Luthra) — Thu, 12 Dec 2024 00:00:00 GMT

Stop piping kubectl output to `grep`, `awk`, and `sed`. JSONPath can get you exactly what you need in one command.

## the basic pattern

```bash
kubectl get  -o jsonpath='{}'
```

## simple examples

### get pod IPs

Instead of:
```bash
kubectl get pods -o wide | awk '{print $6}'
```

Do:
```bash
kubectl get pods -o jsonpath='{.items[*].status.podIP}'
```

### get pod names only

```bash
kubectl get pods -o jsonpath='{.items[*].metadata.name}'
```

### get pod name + IP

```bash
kubectl get pods -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.podIP}{"\n"}{end}'
```

Output:
```
nginx-abc123    10.244.1.5
redis-xyz789    10.244.1.6
```

## real-world use cases

### 1. find all container images

```bash
kubectl get pods -o jsonpath='{.items[*].spec.containers[*].image}' | tr ' ' '\n' | sort -u
```

### 2. get pods not running

```bash
kubectl get pods -o jsonpath='{range .items[?(@.status.phase!="Running")]}{.metadata.name}{"\t"}{.status.phase}{"\n"}{end}'
```

### 3. find pods using most memory

```bash
kubectl top pods -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.usage.memory}{"\n"}{end}' | sort -k2 -h
```

### 4. get all node capacities

```bash
kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.capacity.cpu}{" CPU\t"}{.status.capacity.memory}{" RAM\n"}{end}'
```

### 5. find secrets in a namespace

```bash
kubectl get pods -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.volumes[?(@.secret)].secret.secretName}{"\n"}{end}'
```

### 6. get all services and their type

```bash
kubectl get svc -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.type}{"\n"}{end}'
```

## JSONPath syntax cheat sheet

| Pattern | Description | Example |
|---------|-------------|---------|
| `.items[*]` | All items | Get all pods |
| `.items[0]` | First item | Get first pod |
| `.items[0:3]` | First 3 items | Get first 3 pods |
| `.items[-1]` | Last item | Get last pod |
| `.items[?(@.field=="value")]` | Filter | Pods where phase=Running |
| `{range .items[*]}...{end}` | Loop | Iterate over items |
| `{"\n"}` | Newline | Format output |
| `{"\t"}` | Tab | Format output |

## advanced filtering

### pods with specific label

```bash
kubectl get pods -l app=nginx -o jsonpath='{.items[*].metadata.name}'
```

### pods in Running state

```bash
kubectl get pods -o jsonpath='{.items[?(@.status.phase=="Running")].metadata.name}'
```

### containers in Waiting state

```bash
kubectl get pods -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.containerStatuses[?(@.state.waiting)].name}{"\n"}{end}'
```

### pods with restart count > 0

```bash
kubectl get pods -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.containerStatuses[0].restartCount}{"\n"}{end}' | awk '$2 > 0'
```

## useful aliases

Add to your `~/.bashrc` or `~/.zshrc`:

```bash
# Get pod IPs
alias kip='kubectl get pods -o jsonpath='\''{range .items[*]}{.metadata.name}{"\t"}{.status.podIP}{"\n"}{end}'\'''

# Get images
alias kimages='kubectl get pods -o jsonpath='\''{.items[*].spec.containers[*].image}'\'' | tr " " "\n" | sort -u'

# Get pod with most restarts
alias krestart='kubectl get pods -o jsonpath='\''{range .items[*]}{.metadata.name}{"\t"}{.status.containerStatuses[0].restartCount}{"\n"}{end}'\'' | sort -k2 -n -r | head -1'

# Get not ready pods
alias knotready='kubectl get pods -o jsonpath='\''{range .items[?(@.status.phase!="Running")]}{.metadata.name}{"\t"}{.status.phase}{"\n"}{end}'\'''
```

## custom columns (even better)

Sometimes custom columns are cleaner than JSONPath:

```bash
# Pod name, phase, and IP
kubectl get pods -o custom-columns=NAME:.metadata.name,PHASE:.status.phase,IP:.status.podIP

# Node name, CPU, and memory
kubectl get nodes -o custom-columns=NAME:.metadata.name,CPU:.status.capacity.cpu,MEMORY:.status.capacity.memory

# Services and their ClusterIP
kubectl get svc -o custom-columns=NAME:.metadata.name,TYPE:.spec.type,CLUSTER-IP:.spec.clusterIP
```

## common patterns I use daily

### 1. quick debug — get all pod info

```bash
kubectl get pod nginx-abc123 -o jsonpath='{range .spec.containers[*]}Name: {.name}{"\n"}Image: {.image}{"\n"}Ports: {.ports[*].containerPort}{"\n\n"}{end}'
```

### 2. get all environment variables

```bash
kubectl get pod nginx-abc123 -o jsonpath='{range .spec.containers[*].env[*]}{.name}={.value}{"\n"}{end}'
```

### 3. find pods on a specific node

```bash
kubectl get pods --all-namespaces -o jsonpath='{range .items[?(@.spec.nodeName=="node-1")]}{.metadata.name}{"\t"}{.metadata.namespace}{"\n"}{end}'
```

### 4. get ConfigMaps used by pods

```bash
kubectl get pods -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.volumes[?(@.configMap)].configMap.name}{"\n"}{end}'
```

### 5. network policies applied to pods

```bash
kubectl get netpol -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.podSelector.matchLabels}{"\n"}{end}'
```

## the live-updates move

Combine JSONPath with watch for live updates:

```bash
watch -n 2 'kubectl get pods -o jsonpath='\''{range .items[*]}{.metadata.name}{"\t"}{.status.phase}{"\n"}{end}'\'''
```

## debugging JSONPath

If your JSONPath isn't working, test it step by step:

```bash
# Get full JSON first
kubectl get pod nginx-abc123 -o json | jq '.'

# Then build your JSONPath incrementally
kubectl get pod nginx-abc123 -o jsonpath='{.metadata}'
kubectl get pod nginx-abc123 -o jsonpath='{.metadata.name}'
kubectl get pod nginx-abc123 -o jsonpath='{.status}'
kubectl get pod nginx-abc123 -o jsonpath='{.status.phase}'
```

## the gotcha

JSONPath in kubectl has some quirks:

1. **Filters must use `@`**: `.items[?(@.field=="value")]` not `.items[?(.field=="value")]`
2. **Arrays need `[*]`**: `.items[*]` not `.items[]`
3. **Quotes matter**: use single quotes outside, double inside: `'{.items[?(@.name=="value")]}'`



Docker security: stop running everything as root
root@harshit.cloud (Harshit Luthra) — Tue, 10 Dec 2024 00:00:00 GMT

The audit came back with 47 critical issues, 129 highs, 156 containers running as root, and 300-plus unpatched CVEs. We had been shipping the same Node Dockerfile for two years. It was the one from the official `node` image's README, with our app dropped on top. Nobody had ever questioned it. The auditor wrote one line in the summary: *one RCE in any of these and you own the cluster.*

![Side-by-side comparison of a Docker container running as root with permissive defaults versus a hardened container with a non-root user, dropped capabilities, a read-only filesystem, a seccomp profile, and a distroless base image.](/images/docker-security-hardening/hero.png)

*Fig. 1 · same app code, two trust postures; the audit numbers do most of the arguing.*

## the report

Here's what landed in my inbox on a Tuesday morning, paraphrased into the format the scanner emits:

```
Critical Issues: 47
High Severity: 129
Running as root: 156 containers
Unpatched CVEs: 300+
```

The 156 number was the one that hurt. We didn't have 156 services. We had about thirty. The rest were sidecars, jobs, debug images, one-off tools that someone had built three years ago and never thought about again. Each one ran as UID 0 because the base image did, and nobody had bothered to override it.

## running as root, by accident

This is the Dockerfile we had. Maybe yours too.

```dockerfile
FROM node:20

WORKDIR /app
COPY package*.json ./
RUN npm ci --production
COPY . .

EXPOSE 3000
CMD ["node", "server.js"]
```

The `node` image runs as root by default. There's a `node` user already created inside it, but you have to opt in with `USER node`. Almost nobody does. Six years of Stack Overflow answers, including the accepted ones, omit it. The fix is one line, and the version that creates a fresh user is a habit worth keeping for images that don't ship one.

```dockerfile
FROM node:20-slim

# Create non-root user
RUN groupadd -r nodejs && useradd -r -g nodejs nodejs

WORKDIR /app

# Install dependencies as root
COPY package*.json ./
RUN npm ci --production

# Copy application files
COPY --chown=nodejs:nodejs . .

# Switch to non-root user
USER nodejs

EXPOSE 3000
CMD ["node", "server.js"]
```

The thing the `--chown` flag buys you is that the running process can't `chmod` its own binaries. An attacker who pops the app can read what it can read and write to what it can write to, but can't go and rewrite `server.js` to add a backdoor. That's a real piece of mitigation that costs you nothing.

## images that arrived with everything

Our prod image was 1.2 GB. The base was `ubuntu:latest`, then a kitchen-sink `apt-get install` of `curl`, `wget`, `git`, `build-essential`, Python, Node, and npm. The build engineer who wrote it had reasons for each one at some point. None of those reasons were still true in production.

```dockerfile
FROM ubuntu:latest

RUN apt-get update && apt-get install -y \
    curl \
    wget \
    git \
    build-essential \
    python3 \
    nodejs \
    npm
    
# ... rest of the Dockerfile
```

Every binary in there is a CVE waiting to be reported. The replacement is the same app on `node:20-alpine`, with `dumb-init` for signal handling and nothing else.

```dockerfile
FROM node:20-alpine

# Only install what you need
RUN apk add --no-cache dumb-init

WORKDIR /app

COPY package*.json ./
RUN npm ci --production --ignore-scripts

COPY . .

USER node

ENTRYPOINT ["dumb-init", "--"]
CMD ["node", "server.js"]
```

The image dropped to 150 MB. Vulnerability count fell by 97% the morning we shipped it, mostly because we stopped shipping `git` and a C compiler in production. Build time is 60% shorter. None of that required clever engineering. We deleted things.

## secrets baked into layers

The first time I saw this in our codebase I assumed it was a stub:

```dockerfile
FROM node:20

# DON'T DO THIS!
ENV DB_PASSWORD=supersecret123
ENV API_KEY=abc123xyz

COPY . .
CMD ["node", "server.js"]
```

It wasn't. It had been deployed for eight months. The defense everyone offers is "the registry is private". The problem is that `ENV` lives in the image layer history forever, and `docker history` and `docker save` will hand it to anyone who pulls the image once.

```bash
docker history myapp:latest
docker save myapp:latest | tar -xO | grep -a "API_KEY"
```

BuildKit secrets fix the build-time half. The secret mounts during the `RUN` step and never lands in a layer.

```dockerfile
# syntax=docker/dockerfile:1

FROM node:20-alpine

WORKDIR /app

# Use build-time secrets
RUN --mount=type=secret,id=npmrc,target=/root/.npmrc \
    npm ci --production

COPY . .
CMD ["node", "server.js"]
```

Build with:

```bash
docker build --secret id=npmrc,src=$HOME/.npmrc -t myapp .
```

For runtime secrets, the right answer depends on where you're running. On a single host:

```bash
docker run -e DB_PASSWORD="$(cat /path/to/secret)" myapp
```

On Swarm or Kubernetes, use the platform's secret store. Anything else is a layer of `chmod 600` and hope.

## capabilities you didn't ask for

A vanilla container gets fourteen Linux capabilities by default, including `CAP_NET_RAW`, which lets the process craft raw packets. Most apps need `NET_BIND_SERVICE` and nothing else. Drop the lot, add back what you actually use.

```bash
# Drop all capabilities, add only what's needed
docker run --rm \
  --cap-drop=ALL \
  --cap-add=NET_BIND_SERVICE \
  --security-opt=no-new-privileges:true \
  myapp
```

The Compose form, which is what most teams actually deploy:

```yaml
services:
  webapp:
    image: myapp
    cap_drop:
      - ALL
    cap_add:
      - NET_BIND_SERVICE
    security_opt:
      - no-new-privileges:true
```

`no-new-privileges:true` is the sleeper line. It blocks setuid binaries from elevating during a process exec, closing the residual escalation path that capability drops leave open if a setuid binary is still inside the image.

## a writable root for no reason

Most apps write to `/tmp`, maybe a logging volume, and nothing else. Their root filesystem can be read-only and the app will never notice. Attackers will.

```bash
docker run --rm \
  --read-only \
  --tmpfs /tmp:rw,noexec,nosuid,size=100m \
  myapp
```

In Compose:

```yaml
services:
  webapp:
    image: myapp
    read_only: true
    tmpfs:
      - /tmp:rw,noexec,nosuid,size=100m
      - /var/run:rw,noexec,nosuid,size=10m
```

The first time you ship this you'll discover one library that writes a cache file to `/var/cache` at startup. Add a tmpfs for it and move on. After that the surprises stop.

## base images that age

The `:latest` tag pins nothing. The pin you actually want is a digest, but a version-with-distro tag (`node:20.10.0-alpine3.19`) is the working compromise. Then automate the bump.

```yaml
# .github/dependabot.yml
version: 2
updates:
  - package-ecosystem: "docker"
    directory: "/"
    schedule:
      interval: "weekly"
```

And scan the image. It doesn't matter much which scanner you pick (the lists overlap heavily), but pick one and run it on every build.

```bash
# Using Trivy
trivy image myapp:latest

# Using Snyk
snyk container test myapp:latest

# Using Docker Scout
docker scout cves myapp:latest
```

Wire it into CI as a hard gate on critical and high:

```yaml
# .github/workflows/security.yml
- name: Scan image
  run: |
    trivy image --exit-code 1 --severity CRITICAL,HIGH myapp:latest
```

Yes, you'll have weeks where the gate fires on a CVE you can't fix because there's no patched base image yet. That's a feature. It tells you which deploys are knowingly carrying risk.

## the docker socket

If a container has `/var/run/docker.sock` mounted, it can start a sibling container with `--privileged --pid=host -v /:/host` and own the host. There is no way to make this safe.

```yaml
# NEVER DO THIS
services:
  webapp:
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock  # DON'T!
```

It still shows up in build agents, log shippers, and "monitoring" sidecars from vendors who should know better. If you genuinely need to build images from inside a container, Kaniko does that without the socket. If you need to inspect other containers, the orchestrator's API is the supported path.

## resource limits as a security control

Resource limits feel like a performance concern, but the most common DoS we saw on our cluster was a container OOM-killing its node by allocating until the kernel reaper showed up. Limits don't prevent that, they contain it.

```bash
docker run --rm \
  --memory="512m" \
  --memory-swap="512m" \
  --cpus="0.5" \
  --pids-limit=100 \
  myapp
```

The Compose version:

```yaml
services:
  webapp:
    image: myapp
    deploy:
      resources:
        limits:
          cpus: '0.5'
          memory: 512M
        reservations:
          cpus: '0.25'
          memory: 256M
    pids_limit: 100
```

`--pids-limit` is the underrated one. A fork bomb in your container will still take the container down, but it won't take its neighbors with it.

## the hardened dockerfile, end to end

Putting it all together. This is roughly what every Node service in our prod cluster now looks like:

```dockerfile
# syntax=docker/dockerfile:1

# Use specific version, not 'latest'
FROM node:20.10.0-alpine3.19 AS builder

# Install build dependencies
RUN apk add --no-cache dumb-init

WORKDIR /app

# Copy dependency files
COPY package*.json ./

# Install dependencies with audit
RUN npm ci --production --ignore-scripts && \
    npm audit --audit-level=moderate

# Production stage
FROM node:20.10.0-alpine3.19

# Install only runtime dependencies
RUN apk add --no-cache dumb-init

# Create non-root user
RUN addgroup -g 1001 -S nodejs && \
    adduser -S nodejs -u 1001

WORKDIR /app

# Copy built artifacts from builder
COPY --from=builder --chown=nodejs:nodejs /app/node_modules ./node_modules
COPY --chown=nodejs:nodejs . .

# Remove unnecessary files
RUN rm -rf .git .gitignore .dockerignore README.md tests/

# Switch to non-root user
USER nodejs

# Health check
HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
  CMD node healthcheck.js || exit 1

# Use dumb-init to handle signals properly
ENTRYPOINT ["dumb-init", "--"]

# Run application
CMD ["node", "server.js"]

# Metadata
LABEL org.opencontainers.image.source="https://github.com/myorg/myapp" \
      org.opencontainers.image.version="1.0.0" \
      org.opencontainers.image.vendor="My Company"
```

And the matching Compose, with the runtime hardening that the Dockerfile can't express:

```yaml
version: '3.8'

services:
  webapp:
    image: myapp:1.0.0
    container_name: webapp
    
    # Security options
    user: "1001:1001"
    read_only: true
    cap_drop:
      - ALL
    cap_add:
      - NET_BIND_SERVICE
    security_opt:
      - no-new-privileges:true
      - seccomp:./seccomp.json
    
    # Resource limits
    deploy:
      resources:
        limits:
          cpus: '1'
          memory: 1G
        reservations:
          cpus: '0.5'
          memory: 512M
    pids_limit: 200
    
    # Writable tmpfs for temp files
    tmpfs:
      - /tmp:rw,noexec,nosuid,size=100m
    
    # Network isolation
    networks:
      - internal
    
    # Health check
    healthcheck:
      test: ["CMD", "node", "healthcheck.js"]
      interval: 30s
      timeout: 3s
      retries: 3
      start_period: 10s
    
    # Logging
    logging:
      driver: json-file
      options:
        max-size: "10m"
        max-file: "3"
    
    # Restart policy
    restart: unless-stopped

networks:
  internal:
    driver: bridge
    internal: true
```

## the tools I actually used

For image scanning we landed on Trivy in CI and Docker Scout for local checks. Snyk has a nicer UI but the per-developer license adds up; Clair is what you reach for when nothing can leave the network. For runtime, Falco watches for the syscall patterns nobody should ever see in production (a shell spawned inside a webserver container is the canonical one). Open Policy Agent and its Kubernetes-native cousins, Gatekeeper and Kyverno, are where you encode the rules from this post so the next person can't push a Dockerfile that violates them. The policy engine is the part that makes the work stick.

## the receipts

Six months after the audit, the same scanner came back with vulnerabilities down 94%, image sizes down 70%, root containers at zero (from 156), and a compliance score of 95% (from 23%). Zero security incidents that we know about, which is the only honest way to phrase that number.

The change none of those metrics capture is the cultural one. The CI gate caught seven Dockerfiles in the next quarter that would have shipped a `USER root` or a mounted Docker socket. Each of them was added by someone who'd read this exact post in our wiki and still missed something. The point of the gate isn't that engineers are careless. It's that the wrong defaults will outlast any number of training sessions.

The auditor who wrote *one RCE and you own the cluster* came back the next year. The line in this year's summary read *no findings rated critical*. I keep both of them in the same Slack channel. They're more useful together.



kubectl neat: remove Kubernetes YAML clutter
root@harshit.cloud (Harshit Luthra) — Tue, 10 Dec 2024 00:00:00 GMT

Today I discovered `kubectl neat` - a plugin that removes all the clutter from Kubernetes YAML output.

## the problem

When you run `kubectl get pod my-pod -o yaml`, you get tons of noise:

```yaml
metadata:
  creationTimestamp: "2024-01-01T00:00:00Z"
  managedFields:
    - apiVersion: v1
      fieldsType: FieldsV1
      fieldsV1:
        # 200 lines of garbage
```

## the solution

Install kubectl-neat:

```bash
kubectl krew install neat
```

Now run:

```bash
kubectl get pod my-pod -o yaml | kubectl neat
```

Clean, readable YAML with just the stuff you care about.

## bonus

Make it even easier:

```bash
alias kgn='kubectl get -o yaml | kubectl neat'
```

Now `kgn pod my-pod` gives you clean output instantly.

Saves the copy-paste-and-strip routine every time I pull a Kubernetes config.




jq: the command-line JSON parser that earns its keep
root@harshit.cloud (Harshit Luthra) — Sun, 08 Dec 2024 00:00:00 GMT

`jq` is like `sed` for JSON. After a week of using it I stopped reaching for Python one-liners and never looked back.

## installation

```bash
# Mac
brew install jq

# Ubuntu/Debian
apt-get install jq

# CentOS/RHEL
yum install jq
```

## basic usage

### pretty print JSON

```bash
# Ugly JSON from API
curl https://api.example.com/data | jq '.'
```

Output is now colored and formatted.

### extract a field

```bash
echo '{"name": "John", "age": 30}' | jq '.name'
# "John"

# Remove quotes
echo '{"name": "John", "age": 30}' | jq -r '.name'
# John
```

`-r` = raw output (no quotes)

## array operations

### get first element

```bash
echo '[1, 2, 3, 4, 5]' | jq '.[0]'
# 1
```

### get last element

```bash
echo '[1, 2, 3, 4, 5]' | jq '.[-1]'
# 5
```

### get array length

```bash
echo '[1, 2, 3, 4, 5]' | jq 'length'
# 5
```

### extract field from all array items

```bash
echo '[{"name": "Alice", "age": 25}, {"name": "Bob", "age": 30}]' | jq '.[].name'
# "Alice"
# "Bob"

# Or use map
echo '[{"name": "Alice", "age": 25}, {"name": "Bob", "age": 30}]' | jq 'map(.name)'
# ["Alice", "Bob"]
```

## real-world examples

### 1. parse docker images

```bash
docker images --format='{{json .}}' | jq -r '.Repository + ":" + .Tag + "\t" + .Size'
```

### 2. get all pod names in kubernetes

```bash
kubectl get pods -o json | jq -r '.items[].metadata.name'
```

### 3. extract specific AWS EC2 info

```bash
aws ec2 describe-instances | jq -r '.Reservations[].Instances[] | "\(.InstanceId)\t\(.State.Name)\t\(.PrivateIpAddress)"'
```

### 4. parse package.json dependencies

```bash
cat package.json | jq -r '.dependencies | keys[]'
```

### 5. get GitHub API data

```bash
curl -s https://api.github.com/users/torvalds | jq '{name, bio, public_repos, followers}'
```

## filtering

### filter array items

```bash
# Get users older than 25
echo '[{"name": "Alice", "age": 25}, {"name": "Bob", "age": 30}]' | jq '.[] | select(.age > 25)'
```

### multiple conditions

```bash
# AND condition
jq '.[] | select(.age > 25 and .name == "Bob")'

# OR condition
jq '.[] | select(.age > 25 or .name == "Alice")'
```

### check if field exists

```bash
jq '.[] | select(.email != null)'
```

## transforming data

### create new object

```bash
echo '{"first": "John", "last": "Doe", "age": 30}' | jq '{fullname: (.first + " " + .last), age}'
# {
#   "fullname": "John Doe",
#   "age": 30
# }
```

### rename fields

```bash
echo '{"old_name": "value"}' | jq '{new_name: .old_name}'
```

### add field

```bash
echo '{"name": "John"}' | jq '. + {age: 30}'
# {
#   "name": "John",
#   "age": 30
# }
```

## sorting

```bash
# Sort array of objects by field
echo '[{"name": "Bob", "age": 30}, {"name": "Alice", "age": 25}]' | jq 'sort_by(.age)'

# Reverse sort
jq 'sort_by(.age) | reverse'
```

## grouping

```bash
# Group by field
echo '[{"type": "A", "value": 1}, {"type": "B", "value": 2}, {"type": "A", "value": 3}]' | jq 'group_by(.type)'
```

## useful one-liners

### count items by type

```bash
jq 'group_by(.type) | map({type: .[0].type, count: length})'
```

### sum values

```bash
echo '[{"value": 10}, {"value": 20}, {"value": 30}]' | jq '[.[].value] | add'
# 60
```

### get unique values

```bash
echo '[1, 2, 2, 3, 3, 3]' | jq 'unique'
# [1, 2, 3]
```

### find min/max

```bash
echo '[10, 5, 20, 15]' | jq 'min'
# 5

echo '[10, 5, 20, 15]' | jq 'max'
# 20
```

## advanced — CSV output

```bash
# Convert JSON to CSV
echo '[{"name": "Alice", "age": 25}, {"name": "Bob", "age": 30}]' | jq -r '.[] | [.name, .age] | @csv'
# "Alice",25
# "Bob",30
```

## advanced — nested data

```bash
# Deep extraction
echo '{"user": {"profile": {"name": "John"}}}' | jq '.user.profile.name'
# "John"

# Safe navigation (don't error if missing)
echo '{"user": {}}' | jq '.user.profile.name // "N/A"'
# "N/A"
```

## practical scripts

### check all service status

```bash
#!/bin/bash
curl -s http://api/services | jq -r '.[] | 
  if .status == "up" then
    "\(.name): pass"
  else
    "\(.name): fail (DOWN)"
  end'
```

### parse AWS cost report

```bash
#!/bin/bash
aws ce get-cost-and-usage \
  --time-period Start=2024-01-01,End=2024-01-31 \
  --granularity MONTHLY \
  --metrics BlendedCost | \
  jq -r '.ResultsByTime[] | .TimePeriod.Start + "\t$" + .Total.BlendedCost.Amount'
```

### monitor log errors

```bash
#!/bin/bash
kubectl logs -f pod-name | jq -r 'select(.level == "error") | "\(.timestamp): \(.message)"'
```

## debug jq expressions

Use `jq` playground: https://jqplay.org/

Or test step by step:

```bash
# Start simple
echo '{"a": {"b": {"c": 1}}}' | jq '.'

# Add one level
echo '{"a": {"b": {"c": 1}}}' | jq '.a'

# Add another
echo '{"a": {"b": {"c": 1}}}' | jq '.a.b'

# Final
echo '{"a": {"b": {"c": 1}}}' | jq '.a.b.c'
```

## common patterns I use

### 1. pretty print and save

```bash
curl -s api.example.com/data | jq '.' > formatted.json
```

### 2. extract and process

```bash
curl -s api | jq -r '.items[] | select(.active) | .id' | while read id; do
  echo "Processing $id"
  # do something with $id
done
```

### 3. combine multiple JSON files

```bash
jq -s '.' file1.json file2.json file3.json > combined.json
```

### 4. update JSON file in-place

```bash
# Add a field
jq '.version = "2.0"' package.json > temp.json && mv temp.json package.json

# Or use sponge (from moreutils)
jq '.version = "2.0"' package.json | sponge package.json
```

## the gotcha

Remember to use `-r` for raw output when you want to use the result in bash:

```bash
# Wrong (includes quotes)
NAME=$(echo '{"name": "John"}' | jq '.name')
echo $NAME
# "John"

# Right (no quotes)
NAME=$(echo '{"name": "John"}' | jq -r '.name')
echo $NAME
# John
```

## cheat sheet

```bash
jq '.'                      # Pretty print
jq -r '.field'             # Raw output (no quotes)
jq '.field'                # Get field
jq '.[0]'                  # First array element
jq '.[]'                   # All array elements
jq 'length'                # Length
jq 'keys'                  # Object keys
jq '.[] | select(.x > 5)'  # Filter
jq 'map(.field)'           # Map
jq 'sort_by(.field)'       # Sort
jq 'group_by(.field)'      # Group
jq 'add'                   # Sum array
jq 'unique'                # Unique values
jq -s '.'                  # Slurp (combine files)
```



AWS cost optimization: how we cut our bill by 60%
root@harshit.cloud (Harshit Luthra) — Thu, 05 Dec 2024 00:00:00 GMT

The fastest AWS savings come from rightsizing EC2 and turning on the defaults nobody set: Reserved Instances for steady-state, S3 lifecycle policies, CloudWatch log retention, and VPC endpoints to dodge NAT egress. We took a $50k/month bill to $20k in six weeks with Terraform diffs and one Python script, no architecture rewrites and better p95 latency.

The CFO saw the AWS bill hit $50,000 a month and I got a calendar invite titled "We need to talk about AWS." I knew the meeting before I clicked accept.

Three months later we were at $20,000 a month, with better p95 latency than when we started. The interesting part is that none of the wins were clever. Most of them were a checkbox someone had skipped two years ago.

![Per-service AWS bill before and after, animated as a dumbbell chart: EC2 $28k to $12k, RDS $12k to $7k, Data Transfer $6k to $2.5k, CloudWatch $2k to $0.5k, Other $2k to $1k, total $50k to $20k per month](/images/aws-cost-optimization-tricks/hero.gif)

*Fig. 1 · most of the bill was EC2 doing nothing in particular.*

## the starting point

The bill broke down like this: EC2 $28,000, RDS $12,000, data transfer $6,000, CloudWatch $2,000, everything else $2,000. Fifty grand a month. The cost-allocation tags were missing on roughly 40% of resources, so for the first week the work was just figuring out who owned what.

Most of it turned out to be waste. Not bad architecture, not premature scale, just defaults that nobody had revisited since the seed round.

## rightsizing the EC2 fleet

Every app server in the fleet was running on `m5.2xlarge`. Not because anything needed eight vCPUs, but because the previous engineer picked an instance type once in 2022 and Terraform copy-pasted it forever after.

A month of CloudWatch told the real story:

```bash
# Check actual CPU utilization
aws cloudwatch get-metric-statistics \
  --namespace AWS/EC2 \
  --metric-name CPUUtilization \
  --dimensions Name=InstanceId,Value=i-xxxxx \
  --start-time 2024-01-01T00:00:00Z \
  --end-time 2024-01-31T23:59:59Z \
  --period 3600 \
  --statistics Average
```

Average CPU 12%. Average memory 30%. The fleet was a parking lot.

Dropping to `m5.large` cut the per-hour rate by 4x:

```hcl
# Before
resource "aws_instance" "app" {
  instance_type = "m5.2xlarge"  # $0.384/hour
}

# After
resource "aws_instance" "app" {
  instance_type = "m5.large"     # $0.096/hour
}
```

That single change saved $18,000 a month. p95 latency went down because the new instances were on a newer hypervisor generation. (I have stopped being surprised by this.)

## reserved instances for the steady-state fleet

The app servers ran 24/7. We were paying On-Demand for them anyway, because nobody had wanted to commit a year ahead during a hiring freeze.

The Cost Explorer recommendation API will tell you what to buy if you ask it nicely:

```bash
aws ce get-reservation-purchase-recommendation \
  --service "Amazon Elastic Compute Cloud - Compute" \
  --lookback-period-in-days SIXTY_DAYS \
  --term-in-years ONE \
  --payment-option ALL_UPFRONT
```

We bought 1-year RIs for ten `m5.large` app servers and five `c5.xlarge` API servers. 40% off On-Demand, no architectural change, no risk. $4,000 a month back.

The argument against RIs is always "but what if our load profile changes." Three months later it hadn't.

## spot for the things that can die

The CI fleet was On-Demand `c5.xlarge` runners that sat idle most of the day and got hammered for an hour around lunch. A perfect Spot workload: interruptible, parallelizable, with a queue in front.

```hcl
resource "aws_launch_template" "ci_runner" {
  name_prefix   = "ci-runner-"
  image_id      = data.aws_ami.ubuntu.id
  instance_type = "c5.xlarge"

  instance_market_options {
    market_type = "spot"
    spot_options {
      max_price                      = "0.10"  # ~70% discount
      spot_instance_type             = "one-time"
      instance_interruption_behavior = "terminate"
    }
  }
}

resource "aws_autoscaling_group" "ci_runners" {
  name = "ci-runners"

  mixed_instances_policy {
    launch_template {
      launch_template_specification {
        launch_template_id = aws_launch_template.ci_runner.id
      }
    }

    instances_distribution {
      on_demand_base_capacity                  = 1  # one runner always on
      on_demand_percentage_above_base_capacity = 0  # everything else is Spot
      spot_allocation_strategy                 = "capacity-optimized"
    }
  }

  min_size = 2
  max_size = 10
}
```

One On-Demand runner for the always-on baseline, the rest Spot, capacity-optimized strategy so AWS picks pools with low interruption rates. $2,500 a month. The CI team noticed the build queue was faster, not that the underlying instances had changed.

## S3 lifecycle policies

We had 50 TB in S3, all in Standard. The application logs were the worst offender: every JSON line our services had ever emitted, sitting at $0.023 per GB-month, being read by exactly nobody.

```bash
aws s3api list-objects-v2 \
  --bucket my-bucket \
  --query "Contents[?LastModified<'2023-01-01'].[Key,Size]" \
  --output table
```

Most of it hadn't been touched in a year.

The lifecycle policy is the thing AWS lets you write once and forget:

```json
{
  "Rules": [
    {
      "Id": "Archive old logs",
      "Status": "Enabled",
      "Filter": { "Prefix": "logs/" },
      "Transitions": [
        { "Days": 30,  "StorageClass": "STANDARD_IA" },
        { "Days": 90,  "StorageClass": "GLACIER_IR" },
        { "Days": 180, "StorageClass": "DEEP_ARCHIVE" }
      ]
    },
    {
      "Id": "Delete old temp files",
      "Status": "Enabled",
      "Filter": { "Prefix": "temp/" },
      "Expiration": { "Days": 7 }
    },
    {
      "Id": "Intelligent tiering for backups",
      "Status": "Enabled",
      "Filter": { "Prefix": "backups/" },
      "Transitions": [
        { "Days": 0, "StorageClass": "INTELLIGENT_TIERING" }
      ]
    }
  ]
}
```

Apply it once:

```bash
aws s3api put-bucket-lifecycle-configuration \
  --bucket my-bucket \
  --lifecycle-configuration file://lifecycle.json
```

$3,000 a month. The work was reading enough of the data to be confident no on-call runbook secretly depended on a five-year-old log line. (One did. We rewrote the runbook.)

## RDS, where the real fat lived

The dev database was a `db.r5.4xlarge`. Sixteen vCPUs and 128 GB of RAM, running 24/7, used by maybe three engineers between 10am and 6pm in one timezone. It cost more than half the engineering team's laptops combined.

The fix was three changes. Drop the dev instance to `db.t3.large`. Auto-stop it at night and on weekends. Move staging to Aurora Serverless v2 so it scales to half a capacity unit when idle:

```hcl
resource "aws_db_instance" "dev" {
  identifier     = "dev-database"
  instance_class = "db.t3.large"  # was db.r5.4xlarge

  iam_database_authentication_enabled = true
  auto_minor_version_upgrade          = true

  backup_retention_period = 7
  backup_window           = "03:00-04:00"
  maintenance_window      = "mon:04:00-mon:05:00"
}

resource "aws_rds_cluster" "staging" {
  cluster_identifier = "staging-aurora"
  engine             = "aurora-postgresql"
  engine_mode        = "provisioned"

  serverlessv2_scaling_configuration {
    max_capacity = 2.0
    min_capacity = 0.5
  }
}
```

$5,000 a month. The complaints about staging being slow on the first request after lunch went away once people understood that two seconds of cold-start was the trade.

## CloudWatch, where ingestion is the real bill

CloudWatch costs split three ways: ingestion ($0.50 per GB), storage ($0.03 per GB-month), and analysis. For us, ingestion was the biggest line by far, and the default of "never expire" meant we were also paying to keep a stack trace from 2021 forever. Four levers, biggest first.

**Filter verbose logs before they're ingested.** Half our ingestion was `INFO` and `DEBUG` lines from happy-path requests in production. A subscription filter that drops them at the log group, or just raising the app log level in prod, cut ingestion volume by more than 50%. You pay for every GB that lands, so the cheapest log line is the one you never send.

**Set retention on every log group.** A short script walked the whole account:

```python
import boto3

client = boto3.client('logs')
paginator = client.get_paginator('describe_log_groups')

for page in paginator.paginate():
    for log_group in page['logGroups']:
        group_name = log_group['logGroupName']

        # prod keeps 30 days, everything else keeps 7
        retention_days = 30 if 'prod' in group_name else 7

        client.put_retention_policy(
            logGroupName=group_name,
            retentionInDays=retention_days,
        )
        print(f"Set {group_name} to {retention_days} days")
```

**Use the Infrequent Access log class for logs you rarely query.** Logs you keep for audit or the occasional incident, but don't run Logs Insights against daily, can go in the Infrequent Access class. Ingestion lands at roughly half the Standard rate. You give up some advanced query features, which for cold audit logs you weren't using anyway.

**Consolidate dashboards and watch cardinality.** Each dashboard is $3/month, so one dashboard per service with several widget rows beats a separate dashboard per metric. And high-cardinality custom metrics, the ones tagged with per-request dimensions like `RequestId` or `UserId`, each count as a distinct metric and add up fast. Aggregate before you publish.

$1,500 a month, recovered from log groups whose entire purpose was to exist.

## the NAT gateway tax

Three NAT Gateways, one per AZ, $0.045 per hour each. The HA story was airtight. The actual traffic profile didn't justify it for the non-prod VPCs.

```hcl
# Before: 3 NAT Gateways
resource "aws_nat_gateway" "az1" { /* ... */ }
resource "aws_nat_gateway" "az2" { /* ... */ }
resource "aws_nat_gateway" "az3" { /* ... */ }

# After: 1 NAT Gateway
resource "aws_nat_gateway" "main" {
  allocation_id = aws_eip.nat.id
  subnet_id     = aws_subnet.public[0].id

  tags = { Name = "main-nat-gateway" }
}

resource "aws_route" "private_nat" {
  for_each = aws_route_table.private

  route_table_id         = each.value.id
  destination_cidr_block = "0.0.0.0/0"
  nat_gateway_id         = aws_nat_gateway.main.id
}
```

$200 a month. We kept the three-gateway HA setup in production. The argument against single-NAT in dev is "but what if the AZ goes down?" The answer in dev is "then dev is down."

## data transfer, the silent killer

$6,000 a month in data transfer fees, which is the kind of bill where you can't actually see what you're paying for until you turn on VPC Flow Logs and read them.

```bash
aws ec2 create-flow-logs \
  --resource-type VPC \
  --resource-ids vpc-xxxxx \
  --traffic-type ALL \
  --log-destination-type s3 \
  --log-destination arn:aws:s3:::my-flow-logs
```

Two culprits. App servers were pulling Docker images from external registries on every cold start, paying NAT egress on every layer. And one stale cron job was syncing a database snapshot across regions every hour for a use case that nobody could remember sponsoring.

ECR interface endpoints route the registry traffic privately, so it never leaves the VPC and never touches NAT:

```hcl
resource "aws_vpc_endpoint" "ecr_api" {
  vpc_id              = aws_vpc.main.id
  service_name        = "com.amazonaws.us-east-1.ecr.api"
  vpc_endpoint_type   = "Interface"
  private_dns_enabled = true

  subnet_ids         = aws_subnet.private[*].id
  security_group_ids = [aws_security_group.vpc_endpoints.id]
}

resource "aws_vpc_endpoint" "ecr_dkr" {
  vpc_id              = aws_vpc.main.id
  service_name        = "com.amazonaws.us-east-1.ecr.dkr"
  vpc_endpoint_type   = "Interface"
  private_dns_enabled = true

  subnet_ids         = aws_subnet.private[*].id
  security_group_ids = [aws_security_group.vpc_endpoints.id]
}
```

The S3 gateway endpoint is free, which is the only kind of free that AWS hands out without an asterisk:

```hcl
resource "aws_vpc_endpoint" "s3" {
  vpc_id       = aws_vpc.main.id
  service_name = "com.amazonaws.us-east-1.s3"

  route_table_ids = aws_route_table.private[*].id
}
```

CloudFront went in front of the static asset bucket, which moved bytes out of the per-GB egress lane and into the CDN lane. $3,500 a month back, most of which was the ECR change alone.

## budgets, so the next surprise isn't a surprise

The reason this whole exercise happened in the first place was that nobody had a budget alert. The fix is twelve lines of Terraform:

```hcl
resource "aws_budgets_budget" "monthly" {
  name         = "monthly-budget"
  budget_type  = "COST"
  limit_amount = "25000"
  limit_unit   = "USD"
  time_unit    = "MONTHLY"

  notification {
    comparison_operator        = "GREATER_THAN"
    threshold                  = 80
    threshold_type             = "PERCENTAGE"
    notification_type          = "ACTUAL"
    subscriber_email_addresses = ["alerts@company.com"]
  }

  notification {
    comparison_operator        = "GREATER_THAN"
    threshold                  = 100
    threshold_type             = "PERCENTAGE"
    notification_type          = "FORECASTED"
    subscriber_email_addresses = ["cfo@company.com"]
  }
}
```

The CFO gets the forecasted-overshoot alert. The on-call gets the 80%-of-actual alert. By the time the second one fires, somebody is already digging.

## the receipts

| Category | Before | After | Savings |
|----------|--------|-------|---------|
| EC2 | $28,000 | $12,000 | 57% |
| RDS | $12,000 | $7,000 | 42% |
| Data Transfer | $6,000 | $2,500 | 58% |
| CloudWatch | $2,000 | $500 | 75% |
| Other | $2,000 | $1,000 | 50% |
| **Total** | **$50,000** | **$20,000** | **60%** |

Six weeks of part-time work, no architecture rewrites, no migrations, no vendor changes. Mostly Terraform diffs and one Python script.

If your bill needs the same kind of audit and six weeks of part-time work isn't realistic given your calendar, this is exactly the engagement I take on through [independent infra consulting](https://k8s.org.in): rightsizing, cost audits, and the unglamorous cleanup work most teams keep deferring.

The line from the postmortem the CFO actually circulated was the part I keep coming back to: *"The bill didn't grow because we scaled. The bill grew because nobody was looking."*



Docker build cache: the .dockerignore gotcha
root@harshit.cloud (Harshit Luthra) — Thu, 05 Dec 2024 00:00:00 GMT

Spent 2 hours debugging why my Docker builds were slow despite using multi-stage builds and proper layer ordering.

## the issue

Every single build was invalidating the cache at the `COPY . .` step, even when I hadn't changed any code.

## the culprit

My editor was creating `.swp` files and updating file timestamps. Docker saw these changes and invalidated the cache.

## the fix

Add a proper `.dockerignore`:

```
.git
.gitignore
README.md
.env*
node_modules
npm-debug.log
.next
.vscode
*.swp
*.swo
.DS_Store
```

Build time went from 5 minutes to 30 seconds.

Treat `.dockerignore` like `.gitignore`. Be aggressive about what you exclude.




Infrastructure as code: mistakes I made so you don't have to
root@harshit.cloud (Harshit Luthra) — Thu, 28 Nov 2024 00:00:00 GMT

The first time I ran `terraform destroy` against the wrong workspace, I had two terminals open, one coffee in, and roughly four seconds between hitting `yes` and realising what was on the other end of that plan. The instance count was 17. By the time I cancelled, it was 6. Every one of those came back, eventually. The pages did not.

![Five hand-drawn tombstones lined up in a small graveyard, each one marking a different terraform mistake: hardcoded amis, lost local state, a 2000-line main.tf, an unpinned provider, and a stray terraform destroy in red.](/images/infrastructure-as-code-mistakes/hero.png)

*Fig. 1 · every headstone here was paid for in pages.*

What follows is the short list of Terraform mistakes I've made enough times to recognise on sight. None of them are clever. All of them are the sort of thing you nod at in a blog post and then commit anyway because it's Friday.

## hardcoding everything

**The mistake.** My first Terraform config looked like this:

```hcl
resource "aws_instance" "web" {
  ami           = "ami-12345678"
  instance_type = "t2.micro"
}
```

Looks fine until the day someone asks for the same stack in `eu-west-1` and you discover that AMI ID isn't a real thing outside `us-east-1`. Or until the AMI is six months old and Canonical has retired it. Or until you have eleven of these scattered across modules and you can't grep your way out.

**The fix.** Variables for the knobs, data sources for the things AWS is willing to look up for you:

```hcl
data "aws_ami" "ubuntu" {
  most_recent = true
  filter {
    name   = "name"
    values = ["ubuntu/images/hvm-ssd/ubuntu-*"]
  }
}

resource "aws_instance" "web" {
  ami           = data.aws_ami.ubuntu.id
  instance_type = var.instance_type
}
```

The data source costs you one API call per plan. It saves you the next four migrations.

## not using remote state

**The mistake.** Keeping `terraform.tfstate` on my laptop. I lost it once. Clean reinstall, didn't think to copy the working directory across. The infrastructure was still up, happily running. Terraform had no idea any of it existed. I rebuilt the state by hand with `terraform import`, one resource at a time, and learned more about resource addresses than I wanted to.

**The fix.** S3 backend, versioning on, lock table next to it:

```hcl
terraform {
  backend "s3" {
    bucket = "my-terraform-state"
    key    = "prod/terraform.tfstate"
    region = "us-east-1"
  }
}
```

Versioning on the bucket is the part most people skip. Turn it on. The day you fat-finger a `terraform state rm`, you will want yesterday's state file back, and S3 will hand it over without comment.

## one giant main.tf

**The mistake.** A single `main.tf` that crossed 2,000 lines somewhere around the third VPC peering. Every change touched the same file, every PR diff looked like a refactor, and finding the security group for the bastion meant `Cmd+F "bastion"` and praying I'd named it consistently.

**The fix.** Split by concern, then by reusable unit. The convention I've landed on, per module:

- `main.tf` for the resources that define the module
- `variables.tf` for inputs
- `outputs.tf` for outputs
- `versions.tf` for provider and Terraform version constraints
- separate child modules under `modules/` for anything used twice

The names don't matter to Terraform. It concatenates every `.tf` in the directory regardless. They matter to the next person who opens the repo, which on a long enough timeline is also you.

## not locking provider versions

**The mistake.** A bare provider block:

```hcl
provider "aws" {
  region = "us-east-1"
}
```

The AWS provider shipped a major version with breaking changes to `aws_s3_bucket` resource layout. The next CI run picked it up, the plan tried to recreate every bucket in the account, and I learned what a deeply unhappy Slack channel looks like before lunch.

**The fix.** Pin the provider, pin Terraform itself, commit the lockfile:

```hcl
terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 4.0"
    }
  }
}
```

`~> 4.0` lets in patch and minor bumps, blocks the major. The `.terraform.lock.hcl` file Terraform writes next to your config locks the exact resolved version, including provider hashes. Commit it. Treat a lockfile change in a PR like a dependency upgrade, because that's what it is.

## destroying production by accident

**The mistake.** Two terminal tabs, identical prompts, opposite environments. The plan I meant to run was in the other window. We've all been there. If you haven't, you will be.

**The fix.** A few cheap defenses, layered:

```hcl
resource "aws_instance" "critical" {
  # ...
  lifecycle {
    prevent_destroy = true
  }
}
```

`prevent_destroy` makes Terraform refuse to destroy the resource at all. The plan errors out before anything moves. It's annoying when you genuinely want to destroy the thing, because you have to remove the block first, and that annoyance is the entire point.

Beyond that: a shell prompt that screams the workspace and account in red when you're in prod. A wrapper around `terraform` that grep's the planned destroys and demands you type the resource address back. Terraform Cloud or Atlantis if you have the budget, so the apply runs from a server with proper RBAC and not from whichever terminal you happened to alt-tab into.

The four-second window between `yes` and panic does not get longer with experience. It gets shorter, because you stop reading the prompt.