DEV Community: DevOps Start

Docker TUI Dashboard: Manage Containers with Lazydocker

DevOps Start — Tue, 11 Aug 2026 08:15:42 +0000

If you keep three terminal tabs open just to run docker ps, docker logs -f,
and docker stats, replace all of them with one screen. Lazydocker
is a terminal UI that puts your containers, images, volumes, and networks in a
single dashboard, with live logs and resource graphs next to the list you are
scrolling. It is a single Go binary, it talks to the same Docker daemon your CLI
already uses, and it needs no config to be useful on the first run.

Install it

Pick whichever fits your setup. All three drop a lazydocker binary on your
PATH:

$ brew install jesseduffield/lazydocker/lazydocker   # macOS or Linuxbrew
$ go install github.com/jesseduffield/lazydocker@latest   # any Go 1.21+ toolchain
$ curl https://raw.githubusercontent.com/jesseduffield/lazydocker/master/scripts/install_update_linux.sh | bash

Then start it from any directory:

$ lazydocker

If you are inside a Compose project, run it there. Lazydocker reads the
docker-compose.yml in your working directory and adds a project view so you
can restart the whole stack, not just one container.

The layout

The left column is a stack of panels: Project, Containers, Images, Volumes, and
Networks. The right side is the main view, and it reacts to whatever you have
selected on the left. Highlight a container and the main view shows its logs.
Switch the main view's tabs and you get its config, its stats as an ASCII CPU and
memory graph, and its environment. Nothing to configure, no query to type.

Keys worth memorizing

You drive the whole thing from a handful of keys. These are the navigation keys
that never change:

Key	Action
`1`-`5`	Jump straight to Project, Containers, Images, Volumes, or Networks
`[` / `]`	Cycle the tabs in the main view (logs, stats, config, env)
`x`	Open the action menu for whatever is highlighted
`+` / `_`	Grow or shrink the focused panel
`Esc`	Go back one level
`q`	Quit

The one to lean on is x. Highlight a container, press x, and you get a menu
of everything you can do to it: restart, stop, remove, prune, attach, or exec a
shell. Because the menu is generated from your installed version, it is always
the authoritative list of shortcuts, so you never have to guess whether r
restarts or s stops on the build you have. The common defaults are r to
restart, s to stop, and d to remove, but let the x menu confirm them.

Three things it replaces on day one

Tailing logs across a multi-service app. Select each container and its live log
stream is already in the main view, no docker logs -f <id> and no copying
container IDs.

Cleaning up disk. Press x on the Images panel to prune dangling images, or on a
stopped container to remove it, instead of hunting for the right
docker image prune incantation.

Spotting the container that is eating your CPU. The stats tab draws a live graph
per container, so you see the offender without leaving the dashboard or parsing
docker stats columns.

Configuration, when you want it

Lazydocker writes a config file the first time it runs. On Linux it lands at
~/.config/jesseduffield/lazydocker/config.yml, and on macOS under
~/Library/Application Support. You can rebind keys, change the log tail length,
and add custom commands there, for example a one-key docker compose up -d for
the current project. The defaults are sensible, so treat the config as optional
polish rather than required setup.

When to reach for something else

Lazydocker is aimed at local development, where you are juggling a handful of
containers and want fast feedback. If you only need a live top-style table of
container metrics, ctop is lighter. If you are managing production hosts and
want a web UI with role-based access, that is Portainer's job, not a terminal
dashboard's. For local Docker work, though, lazydocker earns a permanent alias.

For the CLI commands lazydocker wraps, the Docker CLI reference
is the source of truth. If your daemon connection is broken before you even get
this far, start with this GitLab CI Docker daemon fix,
and when you are ready to slim the images you are inspecting, see
Docker multi-stage builds.
If you live on the Kubernetes side too, the same keyboard-first habit pays off in
the kubectl cheat sheet.

Jaeger 2.18 ClickHouse Backend Setup: 8.6x Compression

DevOps Start — Sat, 08 Aug 2026 08:01:23 +0000

The short version

Jaeger 2.18 ships a native ClickHouse storage backend, and you turn it on with a single storage stanza in your Jaeger v2 config file. No sidecar plugin, no gRPC storage process. You point Jaeger at a ClickHouse instance, set create_schema: true, and it builds the tables on startup. On the project's own 10 million span benchmark it hit 8.6x compression on the spans table while sustaining more than 50k spans per second of ingestion.

One caveat before you wire it into anything important: the ClickHouse backend is alpha in 2.18. Pin the exact version, keep it out of your primary production path for now, and treat it as a serious evaluation rather than a drop-in replacement for Elasticsearch or Cassandra.

Prerequisites

A reachable ClickHouse server (a local clickhouse-server container is fine for testing).
The Jaeger v2 binary or the jaegertracing/jaeger:2.18.0 image. Jaeger v2 is built on the OpenTelemetry Collector, so its config uses the collector's extension and pipeline model, not the old v1 flags.
An app already exporting OTLP traces, or anything that can send OTLP to port 4317/4318.

The config

Jaeger v2 declares storage as an extension and wires it into the traces pipeline through an exporter. Save this as config-clickhouse.yaml:

extensions:
  jaeger_storage:
    backends:
      clickhouse-storage:
        clickhouse:
          addresses:
            - localhost:9000
          database: jaeger
          auth:
            basic:
              username: default
              password: password
          create_schema: true
  jaeger_query:
    storage:
      traces: clickhouse-storage

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:

exporters:
  jaeger_storage_exporter:
    trace_storage: clickhouse-storage

service:
  extensions: [jaeger_storage, jaeger_query]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [jaeger_storage_exporter]

Three fields do the real work. addresses is the ClickHouse native protocol endpoint (port 9000, not the 8123 HTTP port). database is the target database, which should already exist. create_schema: true tells Jaeger to create the span and index tables inside that database on first start, so you do not hand-write DDL.

Run it

Start ClickHouse, then start Jaeger against the config. Both run as Docker containers, so a laptop is enough to kick the tires:

$ docker run -d --name clickhouse -p 9000:9000 clickhouse/clickhouse-server:latest
$ docker exec clickhouse clickhouse-client --query "CREATE DATABASE IF NOT EXISTS jaeger"
$ docker run --rm --name jaeger --network host -v "$(pwd)/config-clickhouse.yaml:/etc/jaeger/config.yaml" jaegertracing/jaeger:2.18.0 --config /etc/jaeger/config.yaml

The UI comes up on http://localhost:16686. Send a few OTLP spans, then confirm they landed in ClickHouse rather than trusting the UI alone:

$ docker exec clickhouse clickhouse-client --query "SELECT count() FROM jaeger.spans"

If that count climbs as traffic flows, the pipeline is healthy end to end.

What the compression actually buys you

The headline number comes from the Jaeger team's published benchmark on a 10 million span dataset. ClickHouse stores spans in columnar form and compresses each column independently, which is why trace data (highly repetitive service names, operation names, and attribute keys) shrinks so well.

Metric	Reported result
Spans table compression	8.6x
Sustained ingestion	50k+ spans/sec
Trace-by-ID retrieval	~100 ms
Typical search query	under 50 ms

Those are the project's figures on their hardware, not a promise for your cluster. Your ratio moves with attribute cardinality and how many unique tag values you carry per span. Still, an 8.6x reduction on the largest table is the kind of change that turns a storage line item you dread into one you stop thinking about, which is the same pressure driving teams toward real-time cost observability elsewhere in the stack.

Two things worth knowing

Service Performance Monitoring works without a second datastore. In v2.18 Jaeger can compute latency, call rate, and error rate directly from the spans stored in ClickHouse, so you do not need a separate Prometheus-backed metrics path just to light up the Monitor tab.

Set retention with ClickHouse TTL, not a Jaeger flag. Because the tables are plain ClickHouse tables, you control retention with a TTL clause on the spans table (for example TTL timestamp + INTERVAL 30 DAY). That keeps trace retention a database concern instead of an application one.

If you are running the collector tier that feeds this backend, the same OpenTelemetry foundations apply on the ingest side. See managing OTel collectors at scale for the fleet story, and setting up observability with OpenTelemetry if you are still standing up the tracing pipeline itself.

Start it on a staging Jaeger, point one service's traces at it, and watch the compression ratio on your own data before you commit. Alpha or not, the storage math is hard to argue with.

Manage Kubeflow AI Workloads with Headlamp Plugin

DevOps Start — Thu, 06 Aug 2026 10:00:42 +0000

When a Kubeflow notebook hangs on startup or a Katib trial dies without a clear reason, you usually end up in a terminal running kubectl describe, kubectl get events, and kubectl logs across three namespaces to reconstruct what happened. The Headlamp Kubeflow plugin removes most of that. It teaches the Headlamp dashboard to read Kubeflow's custom resources directly, so a stuck Notebook, a failed Pipeline run, or a bad AutoML suggestion shows up as a first-class object with its status conditions, owned pods, and configuration in one view.

This post walks through what the plugin actually surfaces, how to install it on a desktop or in-cluster Headlamp, and how it changes the way you debug ML workloads on Kubernetes. It was introduced on the Kubernetes blog in July 2026 and is maintained under Kubernetes SIG UI with an Apache 2.0 license.

The problem: Kubeflow hides behind CRDs

Kubeflow is not one application. It is a set of controllers that each register their own Custom Resource Definitions and reconcile them into ordinary Kubernetes objects. A Notebook becomes a StatefulSet and a Pod. A Katib Experiment fans out into Trial and Suggestion resources, each of which spawns Jobs. A Pipeline run creates a graph of Argo Workflow steps. The Kubeflow Central Dashboard gives data scientists a clean surface over all of this, but it is built for the person running experiments, not the person keeping the cluster healthy.

When something breaks, the operator's questions are Kubernetes questions. Which pod is pending, and why? What does the resource's status block say? Is the controller even reconciling this object? The Central Dashboard was not designed to answer those, so operators fall back to kubectl. That is fine for one broken notebook. It stops scaling the moment you are supporting a shared cluster with dozens of researchers and several Kubeflow components installed.

Headlamp already solves the general version of this problem. It is a lightweight, extensible Kubernetes UI that runs as a desktop app or in-cluster, and its plugin system lets anyone add first-class views for custom resources. The Kubeflow plugin is that idea applied to Kubeflow's CRDs.

What the plugin surfaces

The plugin adds operator-focused views for the resources that matter when you are troubleshooting AI/ML workloads. As of the current release it recognizes these component groups:

Component	Custom resources	API group
Notebooks	Notebook, Profile, PodDefault	`kubeflow.org/v1`
Pipelines	Pipeline, PipelineVersion	`pipelines.kubeflow.org/v2beta1`
Katib / AutoML	Experiment, Trial, Suggestion	`kubeflow.org/v1beta1`
Training	TrainJob, TrainingRuntime, ClusterTrainingRuntime	`kubeflow.org/v1`
Spark	SparkApplication, ScheduledSparkApplication	`sparkoperator.k8s.io`

For each object you get the same drill-down Headlamp gives any native resource: the live status conditions, the owned pods with their phases, the events, and the full YAML. That is the difference that matters during an incident. Instead of guessing that a stuck notebook is a scheduling problem, you open the Notebook, see its Pod stuck Pending, click through to the Pod, and read the FailedScheduling event that says there is no node with a free GPU.

Auto-detection: you install only what you run

Very few teams run all of Kubeflow. You might deploy only Katib for hyperparameter tuning, or only the Notebooks controller for a shared research environment. The plugin handles this by checking the cluster's API for which Kubeflow CRDs are actually registered, then showing sidebar sections only for the components it finds.

The practical effect is that the plugin is safe to install everywhere. If a cluster has no Katib CRDs, the Katib section simply does not appear. There is no configuration file listing which features to turn on, and no broken menu entries pointing at resources that do not exist. Install a new component later with Helm or Kustomize, refresh Headlamp, and its section shows up on its own.

Installing on a desktop Headlamp

The fastest way to try the plugin is the desktop app. Download and run Headlamp, point it at a cluster your kubeconfig can reach, and open the Plugin Catalog from the sidebar. Search for the Kubeflow plugin, install it, and reload. If the target cluster has any Kubeflow CRDs, the new sections appear in the left navigation.

This path is ideal for an operator who wants to inspect a cluster from a laptop without deploying anything into it. The plugin runs inside your local Headlamp process and talks to the cluster through the same API access your kubeconfig already grants.

Installing in-cluster

For a shared dashboard the whole team uses, run Headlamp in the cluster and load the plugin through the Helm chart's plugin manager. The chart supports declaring plugins in values.yaml, and a sidecar keeps them in sync:

config:
  pluginsDir: /headlamp/plugins
pluginsManager:
  enabled: true
  watchPlugins: true

There are two supported in-cluster patterns. The plugin manager shown above pulls and updates plugins for you, which is the recommended approach. The alternative is an initContainer that copies plugin files into a shared volume before Headlamp starts. Both end with the plugin's static assets sitting in the directory Headlamp reads at boot. Pick the plugin manager unless you have a reason to bake plugins into an image yourself.

Whichever you choose, remember that Headlamp respects the RBAC of whoever is logged in. The Kubeflow views do not grant new access. A user who cannot get notebooks in a namespace will not see them in the plugin either, which is exactly what you want on a multi-tenant research cluster.

Try it on a throwaway cluster

You do not need a real Kubeflow install to see how the plugin behaves. Because it keys off CRDs, applying the definitions alone is enough to light up the UI. Spin up a local cluster with kind:

$ kind create cluster --name headlamp-kubeflow
$ kubectl config use-context kind-headlamp-kubeflow

Apply a component's CRDs. For the Notebooks controller the upstream manifests install the Notebook definition among others:

$ kubectl apply -k "github.com/kubeflow/notebooks/notebook-controller/config/crd?ref=main"

Confirm the CRD registered:

$ kubectl get crd notebooks.kubeflow.org
NAME                     CREATED AT
notebooks.kubeflow.org   2026-08-05T09:14:22Z

Open Headlamp against this cluster and the Notebooks section appears. Create a sample Notebook object and you can watch the plugin render its status even before a real controller reconciles it, which is a fast way to learn the views without provisioning GPUs. When you are done, delete the cluster with $ kind delete cluster --name headlamp-kubeflow.

A realistic debugging pass

Here is how a shared-cluster incident looks with the plugin in place. A researcher reports that their training run "just stopped." The steps you would otherwise do by hand collapse into a short click path:

Open the Katib section and find the researcher's Experiment. Its status shows Failed with a condition message pointing at the last Trial.
Click into that Trial. The plugin shows the Job it created and the Pod that ran it, along with the Pod's phase.
Open the Pod. Its last state is Terminated with reason OOMKilled and exit code 137.
Read the container spec in the same view: the memory limit is 4Gi, well under what the model needs.

The fix is a bigger memory request in the trial template, but the point is the diagnosis. You went from a vague "it stopped" to OOMKilled without typing a single kubectl command or context-switching between namespaces. Memory pressure and eviction are the same failure modes you already know from general Kubernetes work, and if you want a deeper reference on reading pod-level failure states, our guide on fixing CrashLoopBackOff covers the status conditions the plugin puts in front of you.

Where it fits alongside the Central Dashboard

The Kubeflow Central Dashboard and the Headlamp plugin are not competitors. They serve different people. The Central Dashboard is where a data scientist launches notebooks, submits pipelines, and reviews experiment results. The Headlamp plugin is where an operator or SRE answers infrastructure questions about those same objects: scheduling, resource limits, controller health, and pod lifecycle.

On a small team one person wears both hats and might use both tools. On a larger platform team, the split is cleaner. Researchers live in the Central Dashboard. The people running the cluster live in Headlamp, and the Kubeflow plugin means they no longer have to translate every ML abstraction back into raw pods by hand. If your team is standing up this kind of shared ML platform, it pairs naturally with broader workload visibility work like LLM observability on Kubernetes and the capacity planning behind horizontal pod autoscaling for bursty training jobs.

What it does not do

Be clear about the plugin's scope so you do not expect the wrong thing from it.

It is read and inspect focused. It surfaces state and configuration for troubleshooting; it is not a control plane for editing experiment definitions or launching training runs. Do that through Kubeflow's own tooling.
It reflects only what the CRDs and controllers expose. If a controller writes a thin status block, the plugin can only show that thin block. It reads the cluster, it does not add telemetry the cluster is not already recording.
It does not replace metrics and logs pipelines. For GPU utilization trends, cost, or historical training throughput you still want Prometheus, your logging stack, and a proper observability setup. The plugin answers "what is the state of this object right now," not "how has this behaved over the last week."

None of that is a knock on the tool. It is a focused operator UI, and knowing the boundary keeps you from reaching for it when you actually need a time-series dashboard.

The takeaway

If you operate a cluster that runs Kubeflow, the Headlamp Kubeflow plugin is a low-cost addition that pays off the first time a notebook or trial breaks. It is open source under Kubernetes SIG UI, it detects which components you run so it is safe to install broadly, and it turns the CRD archaeology of an ML incident into a few clicks through resources you already understand. Install it on a desktop Headlamp to try it in minutes, then move it into an in-cluster deployment through the plugin manager once your team wants a shared view. The next time someone says their training run "just stopped," you will find out why without opening a terminal.

Azure SDK for Rust Migration Guide: REST to GA Crates

DevOps Start — Sat, 01 Aug 2026 09:23:29 +0000

If you have been hitting Azure REST endpoints from Rust with hand-rolled reqwest calls and a pile of header-signing code, you can delete most of it now. The Azure SDK for Rust reached general availability in mid-2026, shipping stable 1.0 crates for Core, Identity, Key Vault, and Storage. This guide walks you through replacing raw REST access with the official clients: adding the crates, wiring up DefaultAzureCredential, reading a Key Vault secret, downloading a blob, and turning on retries and tracing. Every step maps a piece of REST plumbing you can retire to the typed client that replaces it.

The migration is not a rewrite. The clients follow the same design patterns as the .NET, Python, Go, and Java SDKs, so the shapes are predictable. What changes is that authentication, retries, and pagination stop being your problem and become the SDK's.

What actually went GA

The GA wave promoted a specific set of crates to stable 1.0. Knowing which ones are production-ready keeps you from pinning a preview crate by accident:

Crate	Purpose
`azure_core`	Shared pipeline: HTTP, retries, auth traits, error types
`azure_identity`	Credential types, including `DefaultAzureCredential`
`azure_security_keyvault_secrets`	Key Vault secrets client
`azure_security_keyvault_keys`	Key Vault keys client
`azure_security_keyvault_certificates`	Key Vault certificates client
`azure_storage_blob`	Blob upload, download, and container operations
`azure_storage_queue`	Queue send and receive
`azure_core_opentelemetry`	Distributed tracing bridge for the pipeline

Two things did not make the GA cut, and you should plan around them. Event Hubs is slated for the next stable wave, and Cosmos DB support is in active development with a stable release expected later in 2026. If your service depends on either, keep your existing REST or preview code for those paths and migrate the rest now. Microsoft's Rust on Azure overview tracks the current crate status if you want to confirm before you pin a version.

Step 1: add the crates

Start with a clean dependency set. From your crate root, let Cargo resolve the latest stable versions rather than guessing patch numbers:

$ cargo add azure_identity azure_security_keyvault_secrets azure_storage_blob tokio

That pulls azure_core transitively, so you rarely add it by hand. If you prefer to pin versions explicitly, your Cargo.toml ends up looking like this:

[dependencies]
azure_identity = "1.0"
azure_security_keyvault_secrets = "1.0"
azure_storage_blob = "1.0"
azure_core = "1.0"
tokio = { version = "1", features = ["full"] }

The SDK is async-first and expects a Tokio runtime, which is why tokio is in the list with the full feature. If your service already runs on async-std or a custom runtime, you will need a compatibility shim, because the clients are built and tested against Tokio.

Verify the tree resolved cleanly before you write any client code:

$ cargo tree -p azure_core

You should see a single azure_core 1.x in the output. If two versions show up, one of your other Azure crates is still on a preview release, and mixing a 1.0 core with a 0.x client is the most common source of trait-mismatch errors during migration.

Step 2: replace your auth code with azure_identity

This is where you delete the most code. If your REST client was fetching tokens from the IMDS endpoint or juggling a client secret from an environment variable, DefaultAzureCredential replaces all of it with one type that tries a chain of sources in order: environment variables, workload identity, managed identity, and your local developer sign-in.

Here is the full pattern for constructing a credential and handing it to a client:

use azure_identity::DefaultAzureCredential;
use azure_security_keyvault_secrets::SecretClient;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let credential = DefaultAzureCredential::new()?;

    let client = SecretClient::new(
        "https://your-vault-name.vault.azure.net/",
        credential.clone(),
        None,
    )?;

    // client is now ready to make authenticated calls
    Ok(())
}

The credential.clone() is cheap. Credentials are reference-counted internally, so cloning shares the same token cache rather than re-authenticating. Build one credential at startup and clone it into every client you construct.

For local development, DefaultAzureCredential will pick up the identity you signed in with through az login. Confirm that works before you run anything:

$ az login
$ az account show --query user.name -o tsv

If you want to be explicit about using developer tooling and skip the managed-identity probes (which add latency and noisy log lines when you run locally), swap in DeveloperToolsCredential:

use azure_identity::DeveloperToolsCredential;

let credential = DeveloperToolsCredential::new(None)?;

A note on scope. In production, prefer managed identity so there is no secret to leak, and grant the identity only the specific Key Vault and Storage roles it needs. The same discipline that keeps secrets out of your CI pipelines applies here: the fewer long-lived credentials your Rust service holds, the smaller your blast radius. If you run across several Azure subscriptions, the multi-subscription patterns from the Terraform side carry over, because DefaultAzureCredential honors the same AZURE_SUBSCRIPTION_ID and tenant environment variables.

Step 3: migrate a Key Vault secret read

A typical pre-SDK secret fetch was a signed GET against https://your-vault.vault.azure.net/secrets/{name}?api-version=7.4, plus JSON parsing to dig the value out of the response envelope. Replace the whole thing with a typed call:

use azure_identity::DefaultAzureCredential;
use azure_security_keyvault_secrets::SecretClient;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let credential = DefaultAzureCredential::new()?;
    let client = SecretClient::new(
        "https://your-vault-name.vault.azure.net/",
        credential.clone(),
        None,
    )?;

    let secret = client
        .get_secret("database-password", "", None)
        .await?
        .into_body()
        .await?;

    if let Some(value) = secret.value {
        println!("secret length: {}", value.len());
    }

    Ok(())
}

Three details matter here. First, the empty string for the version argument means "current version," which is what you almost always want. Pass a specific version string only when you need to pin to a historical value. Second, the response is a two-stage unwrap: .await? gives you the HTTP response, and .into_body().await? deserializes it into the typed Secret model. Third, never log the secret value itself. Print its length or a hash if you need a sanity check, as the example does above.

Run it with your vault name substituted in:

$ cargo run

If you get a 403, the problem is almost always RBAC rather than code. Grant your identity the Key Vault Secrets User role:

$ az role assignment create \
    --role "Key Vault Secrets User" \
    --assignee "$(az ad signed-in-user show --query id -o tsv)" \
    --scope "/subscriptions/<sub-id>/resourceGroups/<rg>/providers/Microsoft.KeyVault/vaults/<vault-name>"

The authentication overview on Microsoft Learn documents the full credential chain and which environment variables each link reads.

Step 4: migrate blob storage access

Blob access follows the same construction pattern. You build a client against the account URL, then operate on containers and blobs. Here is a download that streams the blob body into memory:

use azure_identity::DefaultAzureCredential;
use azure_storage_blob::BlobClient;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let credential = DefaultAzureCredential::new()?;

    let blob_client = BlobClient::new(
        "https://yourstorageaccount.blob.core.windows.net/",
        "reports".to_string(),
        "2026-q3.json".to_string(),
        credential.clone(),
        None,
    )?;

    let response = blob_client.download(None).await?;
    let body = response.into_body().collect().await?;
    println!("downloaded {} bytes", body.len());

    Ok(())
}

The constructor arguments are the account URL, the container name, the blob name, the credential, and an options struct. Passing None for options accepts the defaults, which is the right starting point. The download call returns a response whose body you collect into bytes. For large blobs you would stream chunks rather than collecting the whole body, but collecting is fine for config files and small artifacts.

The same authorization rule applies: the identity needs the Storage Blob Data Reader role (or Contributor if you also write). Assign it the same way you did for Key Vault, swapping the role name and the scope to your storage account.

Uploading is the mirror image. You build the same client and hand it the bytes plus a length:

use azure_core::http::RequestContent;

let data = b"{\"generated_at\":\"2026-07-31\"}".to_vec();
let len = data.len() as u64;

blob_client
    .upload(RequestContent::from(data), true, len, None)
    .await?;

The boolean argument is the overwrite flag. Passing true replaces an existing blob of the same name, and false fails the call if the blob already exists, which is the safer default when you are writing an object that should be created exactly once. As with the download, the options struct is None until you need to set content type, metadata, or an access tier.

Step 5: retries and resilience are already on

One of the quieter wins in the GA release is that resilience is built into azure_core's pipeline and on by default. Transient failures (HTTP 429, 503, and connection resets) are retried automatically with exponential backoff. You do not write retry loops anymore, and you should delete any you carried over from your REST client, because doubling up on retries turns a brief throttle into a much longer stall.

When you do need to tune the behavior, you set it through the client options struct rather than wrapping calls yourself. The pattern looks like this:

use azure_core::http::policies::RetryOptions;

// Construct retry options and pass them through the client's options struct
let retry = RetryOptions::exponential(Default::default());

The GA release also added challenge-based authentication, so the clients work correctly in sovereign and private cloud environments where the token audience is discovered from a challenge response rather than assumed. If you previously special-cased Azure Government or a private cloud, that branch can likely go.

Start with the defaults. The built-in policy is tuned for the common case, and premature retry tuning is a frequent way to make throttling worse rather than better.

Step 6: turn on distributed tracing

If your service already emits OpenTelemetry spans, you can thread Azure SDK calls into the same traces using the azure_core_opentelemetry crate. It bridges the SDK's internal pipeline instrumentation to your OpenTelemetry tracer, so every Key Vault or Storage call shows up as a child span under your request.

Add the crate:

$ cargo add azure_core_opentelemetry

You wire it in by attaching the OpenTelemetry tracer provider to the client options, after which HTTP calls the SDK makes are recorded as spans with the operation name, target, and status. The HTTP logging layer sanitizes secrets by default, so authorization headers and secret values are redacted before anything reaches your log sink. That default matters: it means turning on verbose SDK logging during an incident will not accidentally dump a Key Vault secret into your log aggregator.

If you are new to wiring OpenTelemetry through an application, the mechanics of tracer setup and exporters are the same ones covered in this walkthrough on setting up observability with OpenTelemetry; the Azure crate simply feeds the SDK's own spans into that pipeline instead of you instrumenting each call by hand.

Handle errors with the typed error model

Your REST client probably branched on raw status codes pulled out of a response struct. The SDK gives you a typed azure_core::Error instead, and the useful move during migration is to inspect its HTTP status when you need to distinguish a genuine "not found" from a transient failure the pipeline already gave up retrying.

A common case is treating a missing secret as an expected outcome rather than a hard failure:

use azure_core::http::StatusCode;

match client.get_secret("optional-flag", "", None).await {
    Ok(response) => {
        let secret = response.into_body().await?;
        println!("found: {}", secret.value.unwrap_or_default().len());
    }
    Err(err) if err.http_status() == Some(StatusCode::NotFound) => {
        println!("secret not set, using default");
    }
    Err(err) => return Err(err.into()),
}

The pattern is worth internalizing because it is identical across every GA client. A Key Vault 404, a Storage 404, and an Identity failure all surface through the same azure_core::Error type with the same http_status() accessor. That consistency is a large part of why migrating the second and third service is faster than the first: once you have written error handling for one client, you have written it for all of them.

Resist the urge to match on error strings. The status accessor is stable across releases; the human-readable message is not, and matching on it will break the next time the wire format changes a word.

Deploying to AKS with workload identity

The payoff of DefaultAzureCredential shows up in production, where you want zero secrets in the container. On Azure Kubernetes Service, workload identity federates your pod's service account to an Azure managed identity, and the credential picks it up automatically through environment variables the workload-identity webhook injects. Your Rust code does not change at all between laptop and cluster, which is the point.

The cluster-side wiring is three annotations and a federated credential:

$ az identity federated-credential create \
    --name rust-app-federated \
    --identity-name rust-app-identity \
    --resource-group <rg> \
    --issuer "$(az aks show -g <rg> -n <cluster> --query oidcIssuerProfile.issuerUrl -o tsv)" \
    --subject "system:serviceaccount:<namespace>:<service-account>"

Once the federated credential exists and your pod's service account carries the azure.workload.identity/client-id annotation, the same binary you tested locally authenticates as the managed identity with no code path difference. Grant that identity the same Key Vault and Storage roles you used during local testing, scoped to production resources, and you have a service with no long-lived credential anywhere in the deployment.

A migration checklist

Work through your codebase in this order to keep the change reviewable:

Inventory every place you call an Azure REST endpoint from Rust. Group them by service (Key Vault, Storage, and anything not yet GA).
Add the GA crates for the services you found, and run cargo tree to confirm a single azure_core 1.x in the graph.
Replace token acquisition with a single DefaultAzureCredential built at startup and cloned into each client.
Convert one service at a time. Migrate Key Vault first, since it is usually the smallest surface, then Storage.
Delete your hand-written retry loops and header-signing helpers once the typed client covers that path.
Leave Event Hubs and Cosmos DB on your existing code until their stable crates ship, and tag those spots with a comment so you remember to revisit.
Add azure_core_opentelemetry last, once functionality is proven, so tracing reflects the new call paths.

Migrating incrementally like this is the same principle behind any staged platform move, including the Azure DevOps to GitHub playbook: change one bounded surface, verify it in production, then move to the next. Do not try to flip every service in a single pull request.

Common migration gotchas

A few things trip people up on the first pass:

Mixed crate versions. A preview 0.x client against a 1.0 azure_core produces confusing trait errors. Pin everything to the 1.0 line and re-run cargo tree.
Forgetting .into_body(). The first .await? gives you the response, not the parsed model. The typed value comes from the second .into_body().await?. Skipping it is a frequent compile-time confusion.
Rebuilding credentials per request. Constructing DefaultAzureCredential inside a request handler defeats the token cache and adds latency. Build once, clone many.
RBAC, not code. A 401 usually means the audience or tenant is wrong; a 403 almost always means a missing role assignment. Check az role assignment list before you suspect the SDK.

Where this leaves you

After this migration, your Azure access code in Rust is smaller, typed, and consistent with how the rest of your Azure fleet is written in other languages. Authentication is one credential built at startup. Retries and secret redaction are handled by the pipeline. Tracing plugs into the OpenTelemetry setup you already run. The two gaps to watch are Event Hubs and Cosmos DB, both of which have stable crates on the roadmap, so keep those integration points isolated and ready to swap.

Start with a single non-critical service, prove the pattern end to end in a staging environment, and use the checklist above to roll it out service by service. The crate status page on Microsoft Learn is worth a bookmark, because the GA surface is still expanding and the next wave will let you delete even more REST plumbing.

Manage OTel Collectors at Scale with OpAMP

DevOps Start — Tue, 28 Jul 2026 09:56:39 +0000

If you run more than a handful of OpenTelemetry Collectors, you already know the pain: a config change means SSHing into boxes, redeploying DaemonSets, or babysitting a Git pipeline per cluster, and you never quite trust that every agent is running the config you think it is. OpAMP fixes exactly that. It is a protocol that lets a central server push configuration to a fleet of Collectors, watch their health, and roll changes out in stages, without you touching each host. This post walks through how OpAMP works, the two ways a Collector can speak it, and the config you need to wire one up.

The problem OpAMP solves

A single Collector is easy. A hundred of them, spread across clusters, VMs, and edge nodes, is a fleet-management problem that has nothing to do with telemetry itself. Every observability team eventually builds some version of the same thing: a way to ship a new pipeline config, confirm it actually applied, and back it out when a processor starts dropping spans.

Without a management protocol you end up gluing that together from ConfigMaps, Ansible runs, and dashboards that only tell you an agent is alive, not what config it is actually running. Config drift creeps in. One node keeps an old sampling rate for months because its rollout quietly failed and nobody noticed.

OpAMP, the Open Agent Management Protocol, is the OpenTelemetry answer to this. Splunk donated it to the project in 2022, and it has since become the standard control channel for the Collector. It is worth pairing with a clear-eyed view of what a Collector actually is versus lighter agents; the OpenTelemetry Collector vs Grafana Alloy comparison covers that trade-off if you are still choosing a data plane.

What OpAMP actually is

OpAMP is a client/server network protocol for remote management of large fleets of data-collection agents. It is transport-flexible: agents connect to the server over either plain HTTP or a WebSocket, and the WebSocket path gives you a persistent bidirectional channel so the server can push a new config the moment you save it.

The protocol is deliberately narrow. It handles a specific set of jobs:

Remote configuration: the server sends a config, the agent applies it and reloads.
Health and status reporting: agents report whether they are healthy and what they are doing.
Effective config reporting: agents send back the config they are actually running, so you can detect drift.
Own-telemetry reporting: agents can stream their own metrics, logs, and traces about themselves.
Package and version management: the server can discover an agent's version and, optionally, push binary updates.

Notice what is not in that list: OpAMP does not define what your telemetry pipeline looks like. It carries an opaque config blob to the agent and lets the agent decide what to do with it. For a Collector, that blob is just your normal Collector YAML. OpAMP is the envelope, not the letter.

That separation is the whole design. The OpenTelemetry management docs describe OpAMP as the recommended path for fleet management, and the protocol spec is published at opentelemetry.io/docs/specs/opamp.

Two ways a Collector speaks OpAMP

There are two distinct integration points, and mixing them up is the most common early mistake.

Approach	What it is	What it manages
`opamp` extension	An extension compiled into the Collector	Reports health, effective config, and identity to the server
OpAMP Supervisor	A separate process that wraps the Collector	Full lifecycle: applies remote config, restarts, and reports on the Collector's behalf

The built-in opamp extension is the lightweight option. It lets a Collector announce itself to an OpAMP server and report status, but the extension alone cannot rewrite the Collector's config and restart it, because a running process cannot swap out the config file it booted from and cleanly reload every component.

The Supervisor closes that gap. It is a small parent process that launches the Collector as a child, holds the OpAMP connection itself, and owns the Collector's lifecycle. When a new config arrives, the Supervisor writes it to disk, restarts the Collector against it, and reports the result upstream. For actual remote-configuration-driven fleet management, the Supervisor is the path you want. It lives in the opentelemetry-collector-contrib repository under cmd/opampsupervisor.

Wiring up the Supervisor

The Supervisor takes its own config file, usually supervisor.yaml, which is separate from the Collector config it manages. Here is a representative example:

server:
  endpoint: wss://opamp.example.com:4320/v1/opamp
  tls:
    insecure_skip_verify: false
capabilities:
  accepts_remote_config: true
  reports_effective_config: true
  reports_own_metrics: true
  reports_own_logs: true
  reports_health: true
  reports_remote_config: true
agent:
  executable: /usr/local/bin/otelcol-contrib
storage:
  directory: /var/lib/otelcol-supervisor

Three blocks matter here.

The server block points at your OpAMP backend. The wss:// scheme selects the WebSocket transport, and 4320 is the conventional OpAMP port used across the project's examples. Keep insecure_skip_verify at false in anything real; you are opening a control channel that can change what runs on your hosts, so certificate verification is not optional.

The capabilities block is a set of explicit opt-ins. Nothing is implied. If you want the server to be able to push config, you must set accepts_remote_config: true. If you want drift detection, reports_effective_config: true is what sends the running config back. Turning these on individually means you can start conservative (health only) and add remote config later once you trust the setup.

The agent block tells the Supervisor which binary to run and manage, and storage is where it persists the last-known remote config so a restart does not lose it.

You start the Supervisor, not the Collector, and let it own the child process:

$ otelcol-supervisor --config /etc/otelcol/supervisor.yaml

From here on you never start the Collector directly. The Supervisor connects to the server, sends an AgentDescription that identifies this instance, and waits for config. When you push a new pipeline from the server, the Supervisor lands it on disk and cycles the Collector.

What a remote config flow looks like

Once an agent is connected with accepts_remote_config enabled, the loop is simple to reason about:

You edit a Collector config in the server's UI or API and target a set of agents.
The server sends the config over the open connection.
The Supervisor writes it to its storage directory and restarts the Collector against it.
The Collector boots, and the Supervisor reports back the new effective config and health.
The server marks the rollout applied for that agent, or surfaces an error if the Collector rejected the config.

Step 4 is the part teams underrate. Because the agent reports its effective config, you get a closed loop: the server does not just assume the push worked, it sees the config the Collector is genuinely running. That is how you catch the node that silently kept an old sampling rate. A dashboard built on reports_effective_config shows you real drift instead of a green checkmark that means nothing.

Health, identity, and self-telemetry

The reporting capabilities are worth turning on even before you trust remote config. With reports_health: true, each agent tells the server whether it is up and functioning, which beats inferring liveness from whether metrics happen to be flowing. Health here means the Collector's own view of itself, including whether its pipelines started cleanly.

Identity comes from the AgentDescription message. Every connecting agent sends a set of attributes about itself: hostname, OS, Collector version, and any custom labels you attach. Those labels are the backbone of fleet management, because they are how you target a subset of agents. You push a config to service.namespace=payments and only those Collectors receive it. Getting your labeling scheme right early is more important than the config content itself; without good labels, every rollout is all-or-nothing.

With reports_own_metrics: true, the Collector streams its internal metrics (queue sizes, dropped spans, export failures) as part of the same channel. Feed those into your existing metrics backend. If you scrape them with Prometheus, the Prometheus documentation covers the receiver side, and Grafana's own agent tooling documented at grafana.com shows how a similar management model looks in a different distribution.

Package management, and why to be careful

OpAMP can also push binary updates. The server can discover an agent's version and, with the right capability enabled, deliver a new package so the agent upgrades or downgrades itself. On paper this is the dream: patch a Collector CVE across a thousand hosts from one console.

In practice, treat auto-update as the most dangerous capability in the protocol and turn it on last. A bad config push restarts a Collector; a bad package push replaces the binary on every targeted host at once. Stage it the way you would any other production rollout: a canary group first, watch health and effective-config reporting, then widen. The protocol gives you the mechanism, not the judgment. Keep package management off until your health and config feedback loops are boringly reliable.

Rolling out across a real fleet

Scale is where the labeling discipline pays off. A sane rollout pattern looks like this:

Tag everything. Attach environment, region, and service labels to every agent via its identifying attributes.
Canary by label. Push a config change to a small, clearly labeled canary group first.
Watch effective config. Confirm the canary agents report the new config as their effective config, not just that they acknowledged the push.
Watch health. Give it long enough to catch a pipeline that starts fine but fails under load.
Widen in waves. Expand to the next label group, then the rest, with the same two checks each time.

This is the same staged-rollout thinking you would apply to a Kubernetes deployment, and it composes well with cost and reliability work already in your pipeline. If you are instrumenting application workloads at the same time, the collector fleet you manage with OpAMP is what those pipelines feed into; see How to Set Up LLM Observability with OpenTelemetry and, for cluster-scale patterns, LLM Observability on Kubernetes for the data-plane side of the same system.

Picking a server

OpAMP is only half a system; you also need a server that speaks it. You have two routes. You can run a managed or open-source OpAMP backend such as BindPlane, which grew out of the same observIQ work that seeded the protocol, and get a UI, agent inventory, and config management out of the box. Or you can build against opamp-go, the reference server and client implementation, if you want the control channel wired directly into your own platform.

For most teams, starting with an existing server is the right call. The value of OpAMP is the fleet view and the safe rollout mechanics, and rebuilding those from the reference libraries is a real engineering investment. Start managed, learn the operational patterns, and only build your own server if you have a platform reason the off-the-shelf options cannot meet.

Where this leaves you

OpAMP turns a pile of independently configured Collectors into a fleet you can actually operate. The mental model is small: the Supervisor owns the Collector's lifecycle, the capabilities block is a set of explicit opt-ins, and effective-config reporting is what makes the whole thing trustworthy instead of hopeful. Start with health and effective-config reporting so you can see your fleet, add remote config once you trust the feedback loop, and leave package auto-update for last.

If you are still deciding whether the full Collector is even the right data plane for your fleet, settle that first, then bring OpAMP in to manage whatever you land on. The protocol is agnostic about the pipeline; it just makes running a lot of them survivable.

Validate Kubernetes Manifests with Flux Schema

DevOps Start — Sun, 26 Jul 2026 09:26:07 +0000

If you run GitOps with Flux, a broken manifest usually gets caught the slow way: it merges, the reconciler chokes, and you find out from a failing Kustomization. Flux Schema, the plugin that shipped with Flux 2.9, moves that check left into CI. It validates every YAML document against JSON Schema and CEL rules using the same evaluation logic as the Kubernetes API server, so a bad field fails the pull request instead of the cluster.

Install and run it

Flux Schema is a CLI plugin, not part of the core binary. Install it through the plugin system:

$ flux plugin install schema
$ flux schema --help

Pin a version in CI so a new release never changes your gate's behavior mid-sprint:

$ flux plugin install schema@0.5.0

Point it at a directory of manifests and it validates each document:

$ flux schema validate ./manifests

It ships with built-in schemas for Kubernetes, OpenShift, Gateway API, and the Flux CRDs, so a fresh install already knows your HelmRelease and Kustomization kinds without any setup. Strict validation flags unknown fields, wrong types, and missing required properties as hard errors, which catches the typos kubectl apply --dry-run=client quietly ignores.

What CEL adds over plain schema checks

JSON Schema catches shape problems: a string where an int belongs, a misspelled key. CEL rules catch logic problems. Because Flux Schema runs the x-kubernetes-validations rules embedded in CRDs through the same CEL engine the API server uses, a manifest that violates a cross-field constraint (say, a replica count that must stay below a limit, or two mutually exclusive fields both set) fails in CI with the exact message the cluster would have returned. You are testing against the real admission logic, not a stale copy of it.

Wire it into a config file

Drop a .fluxschema.yml at your repo root to control what gets checked. The file uses the schema.plugin.fluxcd.io/v1beta1 API and a Config kind:

apiVersion: schema.plugin.fluxcd.io/v1beta1
kind: Config
skipKind:
  - Secret
skipJSONPath:
  - "$.metadata.annotations"

skipKind, skipFile, and skipJSONPath let you exclude the things that legitimately fail strict checks, like sealed secrets or generated fields. Then the command reads it automatically:

$ flux schema validate ./manifests --config .fluxschema.yml

Put it in the pull request gate

The real payoff is in CI. Flux Schema ships two composite GitHub Actions: setup installs the CLI on the runner, and validate auto-detects your kustomize overlays, renders them, and validates every rendered document. A minimal gate looks like this:

name: validate-manifests
on: [pull_request]
jobs:
  flux-schema:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: fluxcd/flux-schema/actions/validate@main
        with:
          config: .fluxschema.yml

Because the Ecosystem Catalog behind the plugin refreshes daily from upstream releases, your CI validates against the current API versions rather than whatever was frozen months ago. That matters most right after a Kubernetes minor bump, when a deprecated field you have used for a year suddenly needs to change.

One habit worth keeping: run flux schema validate locally before you push, not just in CI. The feedback loop is a second or two, and it saves a round trip through the runner. If you are still deciding between Flux and Argo CD for this kind of workflow, our Argo CD vs Flux guide compares them for multi-cluster setups, and our GitOps testing strategies piece covers where manifest validation fits in a broader test pyramid.

Read the Flux Schema announcement on fluxcd.io for the full catalog details, then add the action to one repo and watch the first bad manifest fail its PR instead of your cluster.

Fix GitLab CI "dial tcp: lookup docker" no such host error

DevOps Start — Sat, 25 Jul 2026 09:13:31 +0000

The fast fix

If your GitLab CI job dies with error during connect: ... dial tcp: lookup docker on 127.0.0.11:53: no such host, your docker client resolved DOCKER_HOST fine but the DNS name docker does not exist on the job's network. That name is the alias of the docker:dind service, and it only registers when the service container actually starts. The usual fix is to define the service with the -dind image tag and an explicit alias, and point DOCKER_HOST at TLS port 2376:

build:
  image: docker:28.3
  services:
    - name: docker:28.3-dind
      alias: docker
  variables:
    DOCKER_HOST: tcp://docker:2376
    DOCKER_TLS_CERTDIR: "/certs"
    DOCKER_CERT_PATH: "/certs/client"
    DOCKER_TLS_VERIFY: "1"
  script:
    - docker info
    - docker build -t my-app .

That covers the common case on the docker executor. If you are on the Kubernetes executor, or the service still refuses to resolve, keep reading. The name resolution is the whole game here, and there are three distinct reasons it fails.

Why this error is not the daemon-connection error

The address in the message tells you exactly how far the client got. This DNS variant is different from the two connection variants:

error during connect: Post "https://docker:2376/v1.44/info":
dial tcp: lookup docker on 127.0.0.11:53: no such host

The client tried to resolve the hostname docker through the container DNS resolver (127.0.0.11 on a Docker bridge network) and got back nothing. That is a name-resolution failure, not a refused connection. Compare it to the tcp://docker:2375 form, where the name resolves but the daemon is unreachable, covered in the Docker daemon connection error write-up, and the unix:///var/run/docker.sock form, where DOCKER_HOST was never set at all, covered in the socket-variant fix. If your message says no such host, the client never reached any daemon because the name pointed at nothing.

The three real causes

1. The dind service never started, so its alias never registered

This is the most common cause and the least obvious. GitLab registers the docker alias on the build network only when the docker:dind service container comes up. If that container fails to start, the alias is missing and every lookup returns no such host. Three things stop it from starting:

You used the plain image, not the -dind tag. docker:28.3 ships only the client. You need docker:28.3-dind, which bundles dockerd. A plain docker service starts, exits immediately (no daemon to run), and takes its alias down with it.
The runner is not in privileged mode. docker:dind runs its own daemon and needs privileged = true in the runner config.toml. Without it the container cannot start the daemon and dies during boot.
The image failed to pull. A registry rate limit or a typo in the tag means the service container never exists. Check the job log's Preparing the "docker" ... service lines near the top.

Pin the runner config and confirm privileged mode:

$ grep -A3 '\[runners.docker\]' /etc/gitlab-runner/config.toml

You want to see privileged = true. If it says false or is absent, that is your problem.

2. A renamed service image broke the derived alias

GitLab derives a service's default alias from its image name. docker:28.3-dind becomes the alias docker. But if you pull the image through a mirror or a private registry, the derived alias changes and docker stops resolving:

services:
  # WRONG: alias becomes "my-mirror.example.com__docker", not "docker"
  - name: my-mirror.example.com/library/docker:28.3-dind

The daemon is running, but under a name your DOCKER_HOST never asks for. Always set the alias explicitly when the image path is anything other than the bare docker name:

services:
  - name: my-mirror.example.com/library/docker:28.3-dind
    alias: docker

3. The Kubernetes executor puts services on localhost

On the docker executor, the service is a linked container with its own alias, so tcp://docker:2376 is correct. The Kubernetes executor is different: every service runs as a container in the same Pod as the build, sharing one network namespace. They all reach each other on localhost, and the docker alias may not resolve at all. If you copied a working docker-executor config onto a Kubernetes runner, this is why it broke.

Set the host to localhost for the Kubernetes executor:

variables:
  DOCKER_HOST: tcp://localhost:2376
  DOCKER_TLS_CERTDIR: "/certs"
  DOCKER_CERT_PATH: "/certs/client"
  DOCKER_TLS_VERIFY: "1"

Behavior here has shifted across runner versions, so verify against your own version rather than trusting a blog snippet. If tcp://localhost:2376 gives you connection refused instead of no such host, the name resolved and you are back to a daemon-startup problem (see cause 1). The two messages tell you which side of the wall you are on.

A checklist that isolates the cause in under a minute

Work through these in order the next time the job goes red:

Read the address in the error. no such host is DNS. connection refused is a live-but-unreachable daemon. Do not fix the wrong one.
Confirm the -dind tag. Grep your .gitlab-ci.yml for the service image. No -dind, no daemon, no alias.
Check privileged mode in the runner config.toml as shown above.
Match DOCKER_HOST to your executor. tcp://docker:2376 for the docker executor, tcp://localhost:2376 for Kubernetes.
Match the port to TLS. With DOCKER_TLS_CERTDIR set, the daemon listens on 2376. Blank it out and it listens on the plain 2375. A mismatch here surfaces as a connection error once the name resolves.
Scan the top of the job log for the service Preparing/Waiting lines. A service that logs an exit code never registered its alias.

Confirm the daemon is reachable before you build

Once the service is up and named correctly, prove the client can talk to it before your real build steps run. A one-line docker info at the start of script fails fast and loud instead of letting a ten-minute build collapse at the push step:

  script:
    - docker info
    - docker build -t my-app .
    - docker push my-app

The daemon's TCP listener and its 2375-versus-2376 split are documented in the Docker daemon reference; the TLS port only exists because DOCKER_TLS_CERTDIR generated certificates on boot. Keep the port and the TLS variables in sync and the no such host error stays gone. Once your images build cleanly, tightening them is the next job, and multi-stage builds are where that starts.

GitLab CI "Cannot connect to unix:///var/run/docker.sock"

DevOps Start — Thu, 23 Jul 2026 09:39:01 +0000

The fast fix

If your GitLab CI job fails with Cannot connect to the Docker daemon at unix:///var/run/docker.sock, your docker client is looking for a local socket that does not exist inside the job container, because DOCKER_HOST is not set. Point the client at the docker:dind service over TCP and the error goes away:

build:
  image: docker:28.3
  services:
    - name: docker:28.3-dind
      alias: docker
  variables:
    DOCKER_HOST: tcp://docker:2376
    DOCKER_TLS_CERTDIR: "/certs"
    DOCKER_CERT_PATH: "/certs/client"
    DOCKER_TLS_VERIFY: "1"
  script:
    - docker info
    - docker build -t my-app .

That is the whole fix for the common case. The rest of this page explains why the socket variant of the error is different from the tcp://docker:2375 variant, and covers the two other setups (socket-mounted runners and the Kubernetes executor) where the same message shows up for a different reason.

Why you get the unix socket variant specifically

This error is not the same as Cannot connect to the Docker daemon at tcp://docker:2375. The address in the message tells you exactly what the client tried:

Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

When DOCKER_HOST is empty, the Docker CLI falls back to its compiled-in default, the local unix socket at /var/run/docker.sock. Inside a GitLab CI job that uses the docker executor, that socket file simply is not there. The daemon runs in a separate docker:dind service container, not in your job container, so there is nothing listening on the local socket. The client connects, finds no socket, and prints the message above.

The tcp://docker:2375 form is the opposite problem: DOCKER_HOST is set correctly but the dind service is not reachable (missing service, no privileged mode, or a TLS mismatch). If you are seeing that address instead, read the companion write-up on the tcp://docker:2375 form of this error, which walks the service and privileged-mode causes in detail. This page is about the case where the client never even tried the network, because nothing told it to.

The three real causes

1. DOCKER_HOST is unset

This is the usual cause. You added services: - docker:dind but never set DOCKER_HOST, so the client ignores the service container and hits the local socket. Set it as a job or top-level variable:

variables:
  DOCKER_HOST: tcp://docker:2376

Use port 2376 (TLS) when DOCKER_TLS_CERTDIR is set, and 2375 (plain) when you disable TLS with DOCKER_TLS_CERTDIR: "". Mixing them is the second most common mistake, covered next.

2. TLS is half-configured

Docker Engine 19.03 and later turns on TLS between the client and the daemon by default. The dind service generates certificates into the path named by DOCKER_TLS_CERTDIR. If you set the certs directory but point DOCKER_HOST at the plain-text port 2375, or you set neither cert variable, the handshake fails and the client can end up falling back to the socket. Keep the three TLS variables consistent:

variables:
  DOCKER_HOST: tcp://docker:2376
  DOCKER_TLS_CERTDIR: "/certs"
  DOCKER_CERT_PATH: "/certs/client"
  DOCKER_TLS_VERIFY: "1"

If you would rather skip TLS for an internal runner, disable it cleanly and use port 2375:

variables:
  DOCKER_HOST: tcp://docker:2375
  DOCKER_TLS_CERTDIR: ""

Pick one style and set every variable it needs. Do not leave DOCKER_TLS_CERTDIR set while talking to 2375.

3. The runner cannot start dind at all

If DOCKER_HOST is right but the dind container never boots, the client still fails, sometimes reporting the socket address after a retry. The docker:dind service needs privileged mode in the runner's config.toml:

[[runners]]
  executor = "docker"
  [runners.docker]
    privileged = true

With privileged = false or the key absent, dind cannot create its own daemon and no address will work. Confirm this on the runner host before touching your pipeline file.

Socket-mounted runners are the exception

Some self-managed runners deliberately mount the host's Docker socket instead of running dind. In that setup /var/run/docker.sock is supposed to exist inside the job, and the same error means the mount is missing or the path is wrong. The runner's config.toml binds the host socket:

[[runners]]
  executor = "docker"
  [runners.docker]
    volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"]

With this style you do not set DOCKER_HOST at all, because the local socket is the daemon. If you get the unix socket error here, check that the host actually has Docker running and that the bind path in volumes matches the real socket location. This approach shares the host daemon with every job, so treat it as a security tradeoff, not a default. The dind service is the safer choice for untrusted pipelines.

The Kubernetes executor needs the same variables

Running GitLab Runner on Kubernetes does not change the fix, but it adds one gotcha. Each dind service runs as a sidecar container in the same Pod, so DOCKER_HOST: tcp://docker:2376 still resolves through the service alias. What breaks people is TLS cert sharing between containers in the Pod. Set an explicit shared volume for the certs directory in the runner's Helm values or pin DOCKER_TLS_CERTDIR to a path both containers can read. If certs land in a directory only the dind container sees, the client falls back to the socket and you get this exact error inside Kubernetes.

Verify the fix

Add a one-line probe to the top of your job and rerun the pipeline:

$ docker info --format '{{.ServerVersion}}'

If that prints a version string, the client reached the daemon and your docker build will work. If it still fails, echo the variable to confirm the pipeline actually applied it:

$ echo "DOCKER_HOST=$DOCKER_HOST"

An empty value here means your variables block is scoped wrong (defined under the wrong job, or shadowed by a group or project variable). Move it to the top level or the specific job that runs Docker.

Prevention checklist

Always set DOCKER_HOST explicitly when you use docker:dind. Never rely on the default socket in a CI job.
Keep the TLS variables consistent: 2376 with a cert dir, or 2375 with DOCKER_TLS_CERTDIR: "". Never mix them.
Pin the image and the dind service to the same tag, for example docker:28.3 and docker:28.3-dind, so client and daemon versions match.
Confirm privileged = true in the runner config.toml before pushing jobs that need dind.
On Kubernetes, give the certs directory a shared volume so both containers in the Pod can read it.

For the daemon flags behind all of this, including the -H host option and the default unix:///var/run/docker.sock binding, see the Docker daemon reference. Once your builds connect reliably, the multi-stage build guide is a good next step for shrinking the images those pipelines produce.

Fix Flux SOPS MAC mismatch in kustomize-controller

DevOps Start — Tue, 21 Jul 2026 09:41:48 +0000

A SOPS MAC mismatch in Flux almost always means one thing: the encrypted file was changed outside of sops. SOPS signs every file with a message authentication code computed over the plaintext at encrypt time. When kustomize-controller decrypts the file and recomputes that code, it no longer matches, so Flux refuses the Secret and stops reconciling. You cannot patch the ciphertext by hand to fix this. The reliable fix is to recover the real values and re-encrypt the file from scratch, which is what the rest of this page walks through.

What the error looks like

The failure shows up on the Kustomization, not the Secret. Check the object status and the controller logs:

$ flux get kustomizations --all-namespaces
$ kubectl -n flux-system logs deploy/kustomize-controller | grep -i "mac mismatch"

You will see a line similar to this:

Kustomization/flux-system/apps: Reconciliation failed after 1.2s:
failed to decrypt secret 'db-credentials': Error getting data key:
Error decrypting tree: MAC mismatch. Expected '9f8c...', got 'a71b...'

The two hashes are the point. "Expected" is the MAC that sops stored when the file was last encrypted correctly. "Got" is the MAC recomputed from the plaintext it just decrypted. They differ, so the content changed since the last clean encryption.

Why it happens

Four situations produce a MAC mismatch in practice.

Cause	Signal	Fix path
File hand-edited outside sops	Recent commit touched the `.enc.yaml` directly	Recover plaintext, re-encrypt
Git merge conflict resolved by hand	Merge commit on the encrypted file	Re-merge from plaintext
`encrypted_regex` / `mac_only_encrypted` changed	`.sops.yaml` edited since last encrypt	Align rules, re-encrypt
Copy-paste truncation or corruption	Value looks short or malformed	Restore from git, re-encrypt

The first row covers most incidents. Opening an encrypted YAML in a text editor and changing a value, a key name, or even reindenting it alters the plaintext that sops will recompute the MAC over. The same goes for resolving a Git merge conflict by editing the encrypted file directly: you end up with a document sops never signed.

The mac_only_encrypted case is subtler. When your .sops.yaml sets mac_only_encrypted: true, only the encrypted values feed the MAC. Flip that flag, or change which fields encrypted_regex selects, and the recomputed MAC covers a different set of values than the stored one, even though nothing looks wrong in the diff.

Fix it step by step

Step 1: Confirm the file is the problem, not the key

Decrypt the file locally with the same age key Flux uses. If you get a MAC mismatch here too, the file is corrupt and the cluster key secret is fine:

$ export SOPS_AGE_KEY_FILE=$HOME/.config/sops/age/keys.txt
$ sops --decrypt clusters/prod/db-credentials.enc.yaml

If instead you see no key could decrypt or a base64 error, that is a different failure. The related Flux SOPS illegal base64 error fix covers the key-encoding case.

Step 2: Recover the real plaintext

You need a trustworthy copy of the values. Pick whichever source you actually trust.

Restore the last known-good version from Git history:

$ git log --oneline -- clusters/prod/db-credentials.enc.yaml
$ git show <good-sha>:clusters/prod/db-credentials.enc.yaml > /tmp/recovered.enc.yaml
$ sops --decrypt /tmp/recovered.enc.yaml > /tmp/plain.yaml

If you trust the current values but only the MAC is stale (for example the file was reindented, not semantically changed), decrypt while skipping MAC verification:

$ sops --decrypt --ignore-mac clusters/prod/db-credentials.enc.yaml > /tmp/plain.yaml

One caveat: --ignore-mac does not work with --in-place, and it will not rescue a file whose ciphertext was actually modified. Use it only to pull known-good plaintext back out so you can re-encrypt it.

As a last resort, if the Secret already applied to the cluster at least once, read the live values back:

$ kubectl -n app get secret db-credentials -o jsonpath='{.data.password}' | base64 -d

Step 3: Re-encrypt from clean plaintext

Encrypt the recovered plaintext into a fresh file with a valid MAC:

$ sops --encrypt --age age1yourclusterpublickey /tmp/plain.yaml > clusters/prod/db-credentials.enc.yaml
$ rm /tmp/plain.yaml

If you keep a .sops.yaml with creation_rules (recommended, so everyone encrypts identically), sops picks up the key and regex automatically:

$ sops --encrypt /tmp/plain.yaml > clusters/prod/db-credentials.enc.yaml

Step 4: Verify before you push

Never push an encrypted file you have not decrypted at least once. This one command is the difference between a clean reconcile and another red Kustomization:

$ sops --decrypt clusters/prod/db-credentials.enc.yaml | head -5

If that returns plaintext with no MAC error, the file is good.

Step 5: Commit and reconcile

$ git add clusters/prod/db-credentials.enc.yaml
$ git commit -m "fix: re-encrypt db-credentials to repair sops mac"
$ git push
$ flux reconcile kustomization apps --with-source

Watch the Kustomization go Ready again with flux get kustomizations.

Can you tell Flux to ignore the MAC?

No. The kustomize-controller always verifies the MAC during decryption and exposes no ignore-mac option, by design: a Secret that fails integrity checks should not be applied to a cluster. Tested with Flux v2.4.0 and sops v3.9.4, there is no spec.decryption field that disables MAC verification. The only durable fix is a file that decrypts cleanly, so treat the mismatch as a signal to re-encrypt rather than something to bypass.

Prevention

Never open a .enc.yaml in a plain text editor. Run sops clusters/prod/db-credentials.enc.yaml, which decrypts into your editor and recomputes the MAC on save.
Add a CI check or pre-commit hook that runs sops --decrypt on every changed encrypted file, so a broken MAC fails the pull request instead of the cluster.
Commit a .sops.yaml with creation_rules so encrypted_regex and key groups are identical for everyone. Inconsistent rules are a quiet source of MAC drift.
Resolve merge conflicts on encrypted files by decrypting both sides, merging the plaintext, and re-encrypting. Do not hand-edit ciphertext to settle a conflict.

For the canonical setup, the official Flux SOPS guide documents the age and key-secret wiring end to end. If you are still deciding how to structure encrypted secrets across environments, the Argo CD vs Flux multi-cluster GitOps guide and the Argo CD GitOps setup tutorial walk through the surrounding reconciliation model that this Secret plugs into.

Fix vLLM CUDA OutOfMemoryError in Kubernetes

DevOps Start — Sun, 19 Jul 2026 09:18:51 +0000

If your vLLM pod dies at startup with torch.cuda.OutOfMemoryError: CUDA out of memory, the model plus its KV cache needs more VRAM than the GPU allocated to that pod can give. The fastest fix is to cap two things: pass --gpu-memory-utilization 0.85 and --max-model-len 4096 on the serve command, then redeploy. If you have more than one GPU in the pod, add --tensor-parallel-size N to shard the weights across them. Those three flags resolve most of these crashes. The rest of this guide explains when each one matters and the Kubernetes-specific traps that make the error worse than it looks.

This is a GPU error, not a pod OOMKill

The first thing to get straight: torch.cuda.OutOfMemoryError is not the same failure as an OOMKilled pod. An OOMKill happens when your container exceeds its host RAM limit and the kernel sends Exit Code 137. A CUDA OOM happens entirely inside the GPU's own memory, which the Linux OOM killer and your pod memory limit know nothing about. You can have gigabytes of free node RAM and still hit this. If you are chasing an Exit Code 137 instead, the diagnosis path is different and covered in debugging OOMKilled pods.

The full error usually looks like this:

torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 GiB.
GPU 0 has a total capacity of 39.38 GiB of which 224.00 MiB is free.

vLLM does a memory profiling run at startup. It loads the weights, runs a forward pass to measure peak activation memory, then claims the remaining GPU memory (up to gpu_memory_utilization) as a static KV cache pool. The crash happens when weights plus activations already exceed what is free, or when the KV cache target overcommits memory that another process on the GPU is holding.

Confirm what is actually on the GPU

Before changing flags, look at the GPU from inside the running or crash-looping pod. Guessing wastes redeploys.

$ kubectl exec -it deploy/vllm-server -- nvidia-smi

Read two numbers from the output: total GPU memory and current used memory. If used memory is already high before vLLM starts, something else is sharing the card. That is common on shared or MIG-partitioned GPUs, where gpu_memory_utilization of 0.9 (the vLLM default) is a fraction of the full physical card, not of your slice, so the target quietly overcommits.

Then confirm how many GPUs the pod was actually granted:

$ kubectl get pod -l app=vllm -o jsonpath='{.items[0].spec.containers[0].resources.limits}'

If this shows nvidia.com/gpu: "1" but you set --tensor-parallel-size 2, vLLM will try to place shards on GPUs that were never scheduled to the pod, and you get an OOM (or a hang) instead of a clean error. GPUs reach the pod through the Kubernetes device plugin, so the limit you request is the hard ceiling vLLM sees. The tensor parallel size must equal the GPU count in the resource limit.

The four flags that fix it

Each flag trades a different resource. Reach for them in this order.

Flag	Default	What it does
`--gpu-memory-utilization`	0.9	Fraction of GPU memory vLLM may claim for weights plus KV cache. Lower it to leave headroom on a shared card.
`--max-model-len`	model max	Caps context length. KV cache size scales with this, so a 128k model capped at 8k frees a large block.
`--max-num-seqs`	256	Max sequences batched at once. Fewer concurrent requests means a smaller KV cache.
`--tensor-parallel-size`	1	Shards model weights across N GPUs in the pod. The main lever for models too big for one card.

Start by trimming the KV cache, since that is where most waste lives:

$ vllm serve meta-llama/Llama-3.1-8B-Instruct \
    --gpu-memory-utilization 0.85 \
    --max-model-len 4096 \
    --max-num-seqs 64

If the model weights alone do not fit on one GPU, no amount of KV cache trimming helps. That is when you shard:

$ vllm serve meta-llama/Llama-3.1-70B-Instruct \
    --tensor-parallel-size 4 \
    --gpu-memory-utilization 0.90 \
    --max-model-len 8192

A rough sizing check: a model in FP16 needs about 2 GB of VRAM per billion parameters just for weights, before any KV cache. A 70B model is roughly 140 GB, so it will not fit on a single 80 GB A100 no matter how you tune the cache. Shard it across GPUs or quantize it with --quantization fp8 to halve the weight footprint. If you are still deciding which serving engine to run, the tradeoffs are compared in choosing an LLM serving engine.

Kubernetes-specific traps

The same flags behave differently under Kubernetes than on a bare workstation. Three traps account for most repeat incidents.

1. Shared memory is too small for tensor parallelism

vLLM uses shared memory for inter-GPU communication when --tensor-parallel-size is greater than 1. Containers default /dev/shm to 64 MB, which is far too small, and the symptom is often a confusing OOM or NCCL hang rather than a clear message. Mount a memory-backed volume:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-server
spec:
  template:
    spec:
      containers:
      - name: vllm
        image: vllm/vllm-openai:latest
        resources:
          limits:
            nvidia.com/gpu: "4"
        volumeMounts:
        - name: dshm
          mountPath: /dev/shm
      volumes:
      - name: dshm
        emptyDir:
          medium: Memory
          sizeLimit: 8Gi

2. Memory fragmentation on long-running pods

If the error says a large amount is reserved but unallocated, the allocator has fragmented the pool. Set the PyTorch allocator to use expandable segments so freed blocks can be reused:

        env:
        - name: PYTORCH_CUDA_ALLOC_CONF
          value: "expandable_segments:True"

3. CUDA graph capture spikes memory

vLLM captures CUDA graphs at startup for lower latency, and the capture itself needs extra memory. If the OOM lands during capture rather than during the profiling run, disable it with --enforce-eager. You lose some throughput but the pod starts:

$ vllm serve meta-llama/Llama-3.1-8B-Instruct \
    --gpu-memory-utilization 0.85 \
    --enforce-eager

Verify the fix

After redeploying, watch the startup logs for the KV cache report. A healthy start prints the number of GPU blocks it allocated:

$ kubectl logs -f deploy/vllm-server | grep -i "kv cache"

If that line appears and the readiness probe passes, the crash is resolved. Then send a real request so a full sequence actually fills the cache, since a crash can still surface under load rather than at boot:

$ kubectl port-forward deploy/vllm-server 8000:8000 &
$ curl http://localhost:8000/v1/completions \
    -H "Content-Type: application/json" \
    -d '{"model": "meta-llama/Llama-3.1-8B-Instruct", "prompt": "ping", "max_tokens": 16}'

Once serving is stable, keep an eye on GPU memory over time. A slow climb points to fragmentation or a KV cache sized too close to the limit, and wiring GPU metrics into your dashboards early makes that obvious before the next crash. See LLM observability on Kubernetes for the metrics worth tracking, and the top LLMOps tools for the wider serving stack.

The vLLM documentation on conserving memory lists the full set of flags and their interactions. For most Kubernetes deployments, though, the pattern is consistent: cap the KV cache first, shard the weights only when a single GPU genuinely cannot hold the model, and give tensor parallelism the shared memory it needs.

Fix "Resource not accessible by integration" in GitHub Actions

DevOps Start — Sat, 18 Jul 2026 09:00:34 +0000

Your workflow logs a red Error: Resource not accessible by integration and the job dies the moment it tries to write something back: a label, a comment, a commit, a release. The cause is almost always the same. The GITHUB_TOKEN your job runs with is read-only, so any API call that mutates the repository gets a 403. The fix is to grant that token the specific scope the failing step needs, using the permissions key in your workflow. There are three other situations where that fix alone is not enough, and knowing which one you are in saves you an hour of guessing.

What the error actually means

Every workflow run gets a short-lived GITHUB_TOKEN, generated per job and revoked when the job finishes. It authenticates as a bot identity (github-actions[bot]) against the GitHub API. When a step calls the API to change repository state and the token lacks the matching permission, the API answers 403 Resource not accessible by integration. The word "integration" is GitHub API language for the app behind the token, not a hint that some external integration is misconfigured.

So the message is really saying: this token is not allowed to do that. Two things decide what it is allowed to do: the repository or organization default, and any permissions block in your workflow.

Cause 1: the default token is read-only

Since 2023, new repositories default GITHUB_TOKEN to read-only. Many organizations also flip existing repos to read-only as a hardening step, which is the right call. You can confirm the setting under Settings, Actions, General, Workflow permissions. If it says "Read repository contents and packages permissions", the default token cannot write anything.

Do not fix this at the repository level by switching the default back to read/write. That grants every workflow in the repo broad access it does not need. Instead, grant the scope in the one workflow that needs it:

permissions:
  contents: write        # push commits or tags
  pull-requests: write   # comment on or label PRs

jobs:
  release:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: ./scripts/tag-release.sh

One rule trips people up constantly. The moment you add a permissions block, GitHub switches that scope from permissive defaults to explicit mode: every permission you do not list becomes none. If your job also needs to read packages or write to the Checks API, you have to list those too. A job that reads and writes typically needs a handful of scopes spelled out, not one.

You can also scope permissions per job, which is stricter and what I reach for by default. A build job gets contents: read, and only the publish job gets contents: write:

jobs:
  build:
    permissions:
      contents: read
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: make build

  publish:
    needs: build
    permissions:
      contents: write
      packages: write
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: make publish

The full list of scopes and their defaults lives in the GitHub Docs on automatic token authentication. Keep least privilege in mind here; over-scoping the token is one of the quiet ways CI becomes a security liability, a theme covered in GitHub Actions Security: How to Stop Secret Leaks in CI/CD.

Cause 2: the pull request came from a fork

This one catches teams with public repos. When a pull_request event fires from a forked repository, GitHub deliberately hands the workflow a read-only GITHUB_TOKEN and withholds secrets, no matter what your permissions block says. An attacker could otherwise open a PR that runs arbitrary code with write access to your repo. The read-only downgrade is a security boundary, not a bug, and you cannot override it with the permissions key.

If you need to write back on a fork PR (post a comment, apply a label), use the pull_request_target event instead. It runs in the context of the base repository, so the token can be granted write scopes:

on:
  pull_request_target:
    types: [opened, synchronize]

permissions:
  pull-requests: write

Handle pull_request_target carefully. It runs with repository secrets available, so never check out and execute untrusted PR code inside it. Check out the base branch, or only run trusted logic like labeling. The safer pattern for anything that needs the PR's build output is a two-workflow split: an untrusted pull_request job that builds and uploads an artifact, and a trusted workflow_run job that downloads it and writes results back.

Cause 3: the PR was opened by Dependabot

Dependabot PRs look like internal PRs, but since March 2021 GitHub treats workflow runs triggered by Dependabot as if they came from a fork. The GITHUB_TOKEN is read-only and repository secrets are unavailable. Since October 2021 those runs do respect the permissions key, so a workflow that labels or auto-merges Dependabot PRs can work, but you still have to know two things.

First, secrets your job expects are missing on a Dependabot event. Reference secrets.DEPENDABOT_* values from the separate Dependabot secrets store, not the Actions secrets store. Second, the token is still fork-grade, so write operations that GitHub blocks for forks stay blocked. GitHub documents the exact event matrix in Troubleshooting Dependabot on GitHub Actions.

A common working shape for auto-approving patch bumps:

permissions:
  contents: write
  pull-requests: write

jobs:
  automerge:
    runs-on: ubuntu-latest
    if: github.actor == 'dependabot[bot]'
    steps:
      - run: gh pr merge --auto --squash "$PR_URL"
        env:
          PR_URL: ${{ github.event.pull_request.html_url }}
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}

Cause 4: the action creates or approves a PR

Some steps hit a separate switch. Actions that open a pull request or approve one (for example, peter-evans/create-pull-request) need both pull-requests: write on the token and a repository setting that is off by default. Under Settings, Actions, General, look for "Allow GitHub Actions to create and approve pull requests" and enable it. Without that box checked, the API returns the same Resource not accessible by integration even when your permissions block looks correct. This is the one case where the scope is right and the error still fires, so check it early if the token clearly has pull-requests: write.

A diagnosis checklist

Work through this in order. The first match is almost always your fix:

Read the failing step. Which API call 403'd (contents, issues, pull-requests, packages, deployments)? That names the scope you are missing.
Is the trigger a fork pull_request or a Dependabot PR? If yes, the token is read-only by design. Move the write logic to pull_request_target or a workflow_run job (fork) or accept fork-grade limits (Dependabot).
Does the repo default to read-only? Add a permissions block granting only the scope from step 1.
Is the step creating or approving a PR? Enable the repository setting for it.
Re-run and confirm.

To see what the token actually carries at runtime, print the scopes near the top of the failing job:

$ echo "$GITHUB_TOKEN" | gh auth login --with-token
$ gh api rate_limit -i 2>&1 | grep -i 'x-oauth-scopes' || echo "using GITHUB_TOKEN, scopes set by permissions block"

If the workflow is a Terraform or infrastructure pipeline that comments plans back on PRs, the same permission model applies; a full working example lives in How to Automate Terraform Reviews with GitHub Actions. And if you are moving a mutating workflow behind a gate before it touches production, the patterns in Testing in Production: Guide to Progressive Delivery pair well with least-privilege tokens.

Verify the fix

After adding the scope, re-run the job and confirm the failing API call now succeeds. A green run is the real signal, but you can also confirm intent by reading the run's permissions in the logs: expand the "Set up job" step, and GitHub prints the resolved GITHUB_TOKEN permissions for that job. If the scope you added shows there and the call still 403s, you are in Cause 2, 3, or 4, not a missing scope. That distinction is the whole game with this error: decide whether the token could carry the permission at all, then whether you actually granted it.

k8s-aibom: Automated AI BOM for Kubernetes Workloads

DevOps Start — Wed, 15 Jul 2026 14:13:43 +0000

If you run a shared Kubernetes cluster, you already have AI workloads you don't know about. Someone shipped a vLLM inference service last sprint, a data team stood up a RAG pipeline behind a plain Deployment, and a contractor left an Ollama pod running in a namespace nobody audits. k8s-aibom, the controller Google Cloud open-sourced this month, exists to find exactly those. It watches your live workloads and writes a CycloneDX 1.6 ML-BOM for every AI system it can identify, so the answer to "what AI is running in this cluster right now" stops being a guess.

That "right now" is the whole point. A build-time SBOM tells you what your CI pipeline thought it was shipping. It says nothing about the pod a teammate kubectl apply'd by hand at 2am, or the image that pulled a new model layer since the last scan. Shadow AI is a runtime problem, and k8s-aibom is a runtime tool.

Why a runtime AI BOM is different

An AI Bill of Materials answers the same question a software BOM does, scoped to AI: which models, frameworks, and inference engines are in play, and where did they come from. The category matters now because regulators started asking. The EU AI Act's logging and transparency obligations, the NIST AI Risk Management Framework's "know what you deployed" controls, and ISO/IEC 42001's inventory clauses all assume you can produce a current, accurate list of your AI systems. You cannot produce that list from a spreadsheet someone updates quarterly.

Build-time and runtime BOMs solve different halves of the problem. Your build pipeline can attest to what it produced, and tools that live there are a good idea. But the pipeline never sees the workload that skipped it. If your supply-chain story stops at the CI system, read our take on why that is not enough in Supply Chain Security Proxy: Move Beyond Vulnerability Scanning. k8s-aibom fills the runtime gap: it reconciles against what the API server actually reports, not what a pipeline claims it built.

What the controller actually watches

k8s-aibom is a standard Kubernetes controller, not a DaemonSet or a privileged agent. It reconciles a set of workload kinds and emits a BOM per workload. The kinds it tracks:

Deployments
StatefulSets
DaemonSets
Jobs and CronJobs
KServe InferenceService resources

That list covers the shapes AI actually takes in a cluster. Inference services and agent stacks run as Deployments, batch training and evaluation runs as Jobs, and model servers packaged for KServe show up as InferenceService objects. Because it reconciles against the API server, a workload created outside your GitOps flow is just as visible as one that went through it. That is the property that makes it useful against shadow AI: you did not have to know the workload existed for the controller to catalog it.

Detection works by pattern-matching signals the workload already carries: container image references, command-line arguments, environment variables such as HF_MODEL_ID, mounted volumes, and workload annotations. From those signals it recognizes a broad set of AI software:

Category	Examples it identifies
Inference runtimes	vLLM, Hugging Face TGI, NVIDIA Triton, Ollama, Ray Serve, SGLang
Agent frameworks	LangChain, LangGraph, AutoGen, CrewAI, Langflow, Flowise
Vector databases	Milvus, Qdrant, Weaviate, Chroma, pgvector
Training frameworks	PyTorch, KubeRay, JAX, Hugging Face Accelerate
Evaluation tools	lm-evaluation-harness, Ragas, Trulens

If you are already running two inference engines and cannot decide whether that is a problem, our comparison Choosing an LLM Serving Engine: vLLM vs TGI covers the tradeoffs k8s-aibom will happily inventory for you.

What detection looks like on a real workload

Concretely, picture a RAG API someone shipped as a plain Deployment. Nothing about the object name says "AI", but the pod spec gives it away:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: rag-api
  namespace: team-ml
spec:
  template:
    spec:
      containers:
        - name: server
          image: vllm/vllm-openai:v0.6.3
          args: ["--model", "meta-llama/Llama-3.1-8B-Instruct"]
          env:
            - name: HF_MODEL_ID
              value: meta-llama/Llama-3.1-8B-Instruct

The controller does not need a label saying this is AI. The vllm/vllm-openai image matches its inference-runtime catalog, the --model arg and HF_MODEL_ID env var name the model, and both get recorded on the BOM. The image tag lands as declared because it is read straight from the spec; the runtime identification lands as inferred because it came from a heuristic. A workload that went out of its way to hide, say a custom image with the model passed through a mounted config file, would still surface the pieces the controller can see and mark the rest unresolved. That gap is visible in the BOM, which is the honest behavior: you get told what the tool is unsure about instead of a confident lie.

Deploying it

Deployment is a Helm install into its own namespace. You build and push the image, then install the chart:

$ git clone https://github.com/GoogleCloudPlatform/k8s-aibom.git
$ cd k8s-aibom
$ export IMG=my-registry.example.com/k8s-aibom:v1.0.0
$ make image
$ make docker-push
$ helm install k8s-aibom ./charts/k8s-aibom \
    --namespace k8s-aibom-system \
    --create-namespace \
    --set image.repository=my-registry.example.com/k8s-aibom \
    --set image.tag=v1.0.0

The controller does not scan every namespace by default, which is the right call on a busy cluster. You opt a namespace in with a label:

$ kubectl label namespace team-ml aibom.k8saibom.dev/enabled=true

This opt-in model is deliberate. On a large platform you probably want to start with the namespaces where AI is likely, confirm the BOMs look right, then widen the net. Rolling it cluster-wide on day one buries you in output before you have tuned anything.

The custom resources live under the aibom.k8saibom.dev/v1alpha1 API group. There are two kinds: AIBOM, a namespace-scoped resource holding the BOM for one workload, and AIBOMControllerConfig, a cluster-scoped singleton that configures where BOMs are sent.

Reading a BOM

Once a namespace is enabled, the controller starts producing AIBOM resources. You read them like any other object:

$ kubectl get aibom -A
$ kubectl describe aibom -n team-ml deployment-rag-api

The output is a CycloneDX 1.6 ML-BOM. If the document is small it lives inline in the resource status; if it is large the status carries a reference to the externalized copy instead, so you are not stuffing megabytes into etcd.

The detail worth understanding is the confidence model. k8s-aibom does not pretend every field is a hard fact. Each attribute is tagged as one of three states:

declared: taken straight from the workload spec or an explicit annotation, so it is authoritative.
inferred: derived from a heuristic, such as recognizing an inference runtime from its image and args.
unresolved: the controller saw a signal but could not pin it down with confidence.

That grading is what makes the output auditable rather than a pile of guesses. When a reviewer asks why a BOM claims a workload runs a particular model, the answer is a field-level provenance tag, not a shrug. If you have been burned by AI tools that state everything with false certainty, our writeup AI Agent Risks: Lessons from Snyk's 10,000 Environment Audit is a good reminder of why that provenance matters.

Sending BOMs somewhere durable

A BOM that only lives in a cluster resource disappears when the workload does, which is useless for an audit trail. AIBOMControllerConfig defines sinks that push each BOM out. Three sink types exist: the always-on CR status (no external egress), a Google Cloud Storage bucket, and a generic webhook. A config with both external sinks looks like this:

apiVersion: aibom.k8saibom.dev/v1alpha1
kind: AIBOMControllerConfig
metadata:
  name: default
spec:
  sinks:
    - name: audit-archive
      type: GCS
      gcs:
        bucket: my-aibom-archive
        pathTemplate: "aibom/{namespace}/{kind}-{name}/{timestamp}.json"
        workloadIdentity: k8s-aibom-controller@my-project.iam.gserviceaccount.com
    - name: graph-ingest
      type: Webhook
      webhook:
        endpoint: https://guac.internal.example.com/ingest
        auth:
          bearerToken:
            secretRef:
              name: graph-ingest-creds
              key: token

The GCS sink has a property worth calling out: writes use a DoesNotExist precondition, so a stored BOM cannot be overwritten once created. That turns the bucket into an append-only historical record. For anyone who has ever tried to reconstruct "what was running when the incident happened" from mutable logs, an immutable, timestamped BOM per workload is a real upgrade. Pair the pathTemplate above with a bucket retention policy and you have a compliance artifact that survives the workload that produced it.

The webhook sink is how you feed a graph database or an SBOM platform. A common pattern is pushing into a supply-chain graph so AI components sit alongside your other software inventory instead of in a separate silo.

Where it fits, and where it doesn't

k8s-aibom is narrow on purpose, and that is a strength. It does not scan for vulnerabilities, enforce policy, or block anything. It builds an accurate inventory of AI workloads and gets it somewhere durable. Everything downstream, such as CVE correlation, policy gates, and drift alerts, is a separate tool consuming the BOM. Trying to make one controller do all of that is how you end up with a privileged agent nobody trusts.

Keep three limitations in mind before you lean on it:

Detection is pattern-based, so it has a coverage frontier. A homegrown inference server with no recognizable image, args, or environment signals may land as unresolved or be missed. The v1alpha1 API group is a fair signal that the detection catalog is still moving. Treat the BOM as a strong lead, not a guarantee of completeness, and watch what shows up unresolved.
It reports, it does not enforce. Finding a shadow workload and doing something about it are different jobs. You still need policy tooling, whether that is an admission controller or a governance layer, to act on what the BOM reveals. If you are building that layer, Governing AI Agents in CI/CD with OPA and MCP covers the policy side.
The GCS sink is Google Cloud native. The webhook sink is portable and works anywhere, but the tightest integration, immutable object writes via Workload Identity, assumes GKE. On other platforms you wire the webhook into your own durable store.

The bottom line

Shadow AI is not going away, and "we think we know what's running" is not an answer an auditor accepts. k8s-aibom gives you a runtime, provenance-tagged inventory of the AI workloads actually live in your cluster, written to an immutable store you can hand to a compliance review. It is early software with a moving detection catalog, so verify its output rather than trusting it blindly. But as a way to turn shadow AI from an unknown into a tracked list, it is a genuinely useful addition to a Kubernetes security stack.

Start small: install it, enable one namespace where you suspect unmanaged AI, and read the first few BOMs. The controller runs as a lightweight reconciler with no privileged access, so there is little downside to letting it watch. For the broader context on how Google frames this problem, their software supply chain security guidance and the how GKE powers AI innovation writeup are worth reading. For the Kubernetes primitives the controller builds on, the upstream controllers documentation covers the reconcile loop it uses.