Skaffold – Guides

Docs: Getting Started With Your Project

Mon, 01 Jan 0001 00:00:00 +0000

Skaffold requires a skaffold.yaml, but - for supported projects - Skaffold can generate a simple config for you that you can get started with. To configure Skaffold for your application you can run skaffold init.

Running skaffold init at the root of your project directory will walk you through a wizard and create a skaffold.yaml that defines how your project is built and deployed.

skaffold init

What’s next

You can further set up File Sync for source files that do not need a rebuild in dev mode.

Skaffold automatically forwards Kubernetes Services in dev mode if you run it with --port-forward. If your project contains resources other than services, you can set-up port-forwarding to port-forward these resources in dev or debug mode.

For more understanding on how init works, see skaffold init.

Docs: skaffold dev

Mon, 01 Jan 0001 00:00:00 +0000

skaffold dev enables continuous local development on an application. While in dev mode, Skaffold will watch an application’s source files, and when it detects changes, will rebuild your images (or sync files to your running containers), push any new images, test built images, and redeploy the application to your cluster.

skaffold dev is considered Skaffold’s main mode of operation, as it allows you to leverage all of the features of Skaffold in a continuous way while iterating on your application.

💡 Tip

Running skaffold dev is equivalent to running the IDE command Run on Kubernetes if you’re using Skaffold with the Cloud Code IDE extensions. In addition to this guide you should also look at the corresponding guides for VSCode, IntelliJ and Cloud Shell.

Dev loop

When skaffold dev is run, Skaffold will first do a full build, test, render and deploy of all artifacts specified in the skaffold.yaml, similar to skaffold run. By default, Skaffold will exit if any of these stages fails. Upon successful build, test, render and deploy, Skaffold will start watching all source file dependencies for all artifacts specified in the project. As changes are made to these source files, Skaffold will rebuild and retest the associated artifacts, and redeploy the new changes to your cluster.

The --keep-running-on-failure flag can be used to keep Skaffold running by suspending the Skaffold session while encountering errors in the first build, test, render, deploy. The user can attempt to fix the errors and then restore the session by pressing any key or exit the session with Ctrl+C. The session will be suspended again unless the errors are fixed. Using this flag should result in an better experience for dev sessions where errors occur that would otherwise cause Skaffold to exit

The dev loop will run until the user cancels the Skaffold process with Ctrl+C. Upon receiving this signal, Skaffold will clean up all deployed artifacts on the active cluster, meaning that Skaffold won’t abandon any Kubernetes resources that it created throughout the lifecycle of the run. This can be optionally disabled by using the --no-prune flag.

Precedence of Actions

The actions performed by Skaffold during the dev loop have precedence over one another, so that behavior is always predictable. The order of actions is:

File Watcher and Watch Modes

Skaffold computes the dependencies for each artifact based on the builder being used, and the root directory of the artifact. Once all source file dependencies are computed, in dev mode, Skaffold will continuously watch these files for changes in the background, and conditionally re-run the loop when changes are detected.

By default, Skaffold uses filesystem notifications of your OS to monitor changes on the local filesystem and re-runs the loop on every change.

Skaffold also supports a polling mode where the filesystem is checked for changes on a configurable interval, or a manual mode, where Skaffold waits for user input to check for file changes. These watch modes can be configured through the --trigger flag.

Controlling the Dev Loop with API

Note

This section is intended for developers who build tooling on top of Skaffold.

By default, the dev loop will carry out all actions (as needed) each time a file is changed locally, with the exception of operating in manual trigger mode. However, individual actions can be gated off by user input through the Skaffold API.

With this API, users can selectively turn off the automatic dev loop and can tell Skaffold to wait for user input before performing any of these actions, even if the requisite files were changed on the filesystem. By doing so, users can “queue up” changes while they are iterating locally, and then have Skaffold rebuild and redeploy only when asked. This can be very useful when builds are happening more frequently than desired, when builds or deploys take a long time or are otherwise very costly, or when users want to integrate other tools with skaffold dev.

For more documentation, see the Skaffold API Docs.

Docs: Debugging With Skaffold

Mon, 01 Jan 0001 00:00:00 +0000

Skaffold lets you set breakpoints and step through your application, even when deployed to remote Kubernetes clusters, as if the code were running locally on your machine. Skaffold detects the language runtimes of your container images, reconfigures the pods for debugging, forwards debugging ports, and then monitors for when debuggable containers come online. IDE integrations like Cloud Code leverage Skaffold’s events to automatically set up debugging sessions.

Debugging is currently supported for five language runtimes.

Go 1.13+ (runtime ID: go) using Delve
NodeJS (runtime ID: nodejs) using the NodeJS Inspector (Chrome DevTools)
Java and JVM languages (runtime ID: jvm) using JDWP
Python 3.5-3.10 runtimes (runtime ID: python) using debugpy (Debug Adapter Protocol) or pydevd
- NOTE: Python 3.11 support is planned and coming soon but is currently still in progress
.NET Core (runtime ID: netcore) using vsdbg (only for VS Code)

Skaffold can usually detect the correct language runtime if present. However if you encounter difficulties then checkout the Supported Language Runtimes section for the exact heuristics that Skaffold uses and you can modify your application accordingly, or read about how you can manually configure your container image.

(Recommended) Debugging using Cloud Code

The easiest way to debug using Skaffold is by using the Cloud Code IDE extension for Visual Studio Code, JetBrains IDEs, and Cloud Shell. Cloud Code will automatically configure container images for debugging so you can debug Kubernetes services just like how you would debug a local service in your IDE.

For more information, see the corresponding documentation for VS Code, IntelliJ and Cloud Shell.

Detailed debugger configuration and setup

This section describes how Skaffold configures the container image for debugging for the supported language runtimes, and how to setup with certain IDEs.

Note that many debuggers may require additional information for the location of source files. We are looking for ways to identify this information and to pass it back if found.

Go (runtime: `go`, protocols: `dlv`)

Go-based applications are configured to run under Delve in its headless-server mode.

On recognizing a Go-based container image, debug rewrites the container image’s entrypoint to invoke your application using dlv:

dlv exec --headless --continue --accept-multiclient --listen=:56268 --api-version=2 <app> -- <args> ...

Skaffold debug using the JetBrains GoLand and IntelliJ Ultimate

Debugging is only supported in JetBrains IDEs for Go applications built using Go Modules. The IDE settings must also be explicitly configured to use Go Modules. Errors like cannot find debugger path indicate misconfiguration.

Skaffold debug using the VS Code Go extension

If you use the debug functionality of the VS Code Go extension to connect to dlv, you may need to configure local and remote source paths, via either the cwd and remotePath properties, or the substitutePath array property (reference).

The cwd or from properties should be the absolute path of the top-level directory of your source files and should generally match the artifact’s context directory in skaffold.yaml. You can represent this using the ${workspaceFolder} variable.

The remotePath or to properties should be set to the absolute path of the source location during compilation. For example, the golang images, which are often used in multi-stage builds, have a default working directory of /go. The following configuration works in this case:

{
    "name": "Skaffold Debug",
    "type": "go",
    "request": "attach",
    "mode": "remote",
    "host": "localhost",
    "port": 56268,
    "cwd": "${workspaceFolder}",
    "remotePath": "/go"
}

You can use the substitutePath property instead of cwd and remotePath:

{
    "name": "Skaffold Debug",
    "type": "go",
    "request": "attach",
    "mode": "remote",
    "host": "localhost",
    "port": 56268,
    "substitutePath": [
        {
            "from": "${workspaceFolder}",
            "to": "/go",
        },
    ],
}

If you use the ko local builder, the source path during compilation matches the source path in your IDE. In this case, the value for remotePath will match cwd. If the values of both cwd and remotePath are ${workspaceFolder}, you can omit these properties.

NodeJS (runtime: `nodejs`, protocols: `devtools`)

NodeJS applications are configured to use the Chrome DevTools inspector via the --inspect argument.

On recognizing a NodeJS-based container image, debug rewrites the container image’s entrypoint to invoke your application with --inspect:

node --inspect=9229 <app.js>

Note

Many applications use NodeJS-based tools as part of their launch, like npm, rather than invoke node directly. These intermediate node instances may interpret the –inspect arguments. Skaffold introduces a node wrapper that only invokes the real node with –inspect if running an application script, and skips scripts located in node_modules. For more details see the associated PR.

A debugging client must first obtain the inspector UUID.

Java and Other JVM Languages (runtime: `jvm`, protocols: `jdwp`)

Java/JVM applications are configured to expose the JDWP agent using the JAVA_TOOL_OPTIONS environment variable. Note that the use of JAVA_TOOL_OPTIONS causes extra debugging output from the JVM on launch.

On recognizing a JVM-based container image, debug rewrites the container image’s environment to set:

JAVA_TOOL_OPTIONS=-agentlib:jdwp=transport=dt_socket,server=y,address=5005,suspend=n,quiet=y

Python (runtime: `python`, protocols: `dap` or `pydevd`)

Python applications are configured to use either debugpy or pydevd. debugpy uses the debug adapter protocol (DAP) which is supported by Visual Studio Code, Eclipse LSP4e, and other editors and IDEs.

On recognizing a Python-based container image, debug rewrites the container image’s entrypoint to invoke Python using either the pydevd or debugpy modules:

python -m debugpy --listen 5678 <app>

python -m pydevd --server --port 5678 <app.py>

Note

As many Python web frameworks use launcher scripts, like gunicorn, Skaffold now uses a debug launcher that examines the app command-line.

.NET Core (runtime: `dotnet`, protocols: `vsdbg`)

.NET Core applications are configured to be deployed along with vsdbg for VS Code.

JetBrains Rider

This set up does not yet work automatically with Cloud Code for IntelliJ in JetBrains Rider. There is a manual workaround.

Omnisharp for VS Code

For users of VS Code’s debug adapter for C#:** the following configuration can be used to debug a container. It assumes that your code is deployed in /app or /src folder in the container. If that is not the case, the sourceFileMap property should be changed to match the correct folder. processId is usually 1 but might be different if you have an unusual entrypoint. You can also use "${command:pickRemoteProcess}" instead if supported by your base image. (// comments must be stripped.)

{
    "name": "Skaffold Debug",
    "type": "coreclr",
    "request": "attach",
    "processId" : 1,
    "justMyCode": true, // set to `true` in debug configuration and `false` in release configuration
    "pipeTransport": {
        "pipeProgram": "kubectl",
        "pipeArgs": [
            "exec",
            "-i",
            "<NAME OF YOUR POD>", // name of the pod you debug.
            "--"
        ],
        "pipeCwd": "${workspaceFolder}",
        "debuggerPath": "/dbg/netcore/vsdbg", // location where vsdbg binary installed.
        "quoteArgs": false
    },
    "sourceFileMap": {
        // Change this mapping if your app in not deployed in /src or /app in your docker image
        "/src": "${workspaceFolder}",
        "/app": "${workspaceFolder}"
        // May also be like this, depending of your repository layout
        // "/src": "${workspaceFolder}/src",
        // "/app": "${workspaceFolder}/src/<YOUR PROJECT TO DEBUG>"
    }
}

Supported Language Runtimes

The language runtime for the artifacts can be specified in the skaffold.yaml file.

build:
  artifacts:
  - image: gcr.io/k8s-skaffold/example
    runtimeType: go # specify one of `go`, `nodejs`, `jvm`, `python`, or `netcore`
  local: {}

Otherwise Skaffold debug recognizes the language runtime used in a container image for specific language runtimes using standard heuristics described below.

Go (runtime: `go`)

Go-based container images are recognized by:

the presence of one of the standard Go runtime environment variables: GODEBUG, GOGC, GOMAXPROCS, or GOTRACEBACK, or
the presence of the KO_DATA_PATH environment variable in container images built by ko, or
is launching using dlv.

Unless you built your container image using ko, set one of the standard Go environment variables to allow Skaffold to detect your container as Go. GOTRACEBACK=single is the default setting for Go, so you can add this environment variable without changing behavior. Alternatively, GOTRACEBACK=all is a useful setting in many situtions.

Your application should be built with the -gcflags='all=-N -l' flag to disable optimizations and inlining. If you do not add this flag, debugging can becoming confusing due to seemingly-random execution jumps that happen due to statement reordering and inlining. Skaffold configures Docker builds with a SKAFFOLD_GO_GCFLAGS build argument that you should use in your Dockerfile:

FROM golang
ENV GOTRACEBACK=all
COPY . .
ARG SKAFFOLD_GO_GCFLAGS
RUN go build -gcflags="${SKAFFOLD_GO_GCFLAGS}" -o /app .

Note that the Alpine Linux-based golang container images do not include a C compiler which is required to build with -gcflags='all=-N -l'.

The ko builder adds -gcflags='all=-N -l' automatically when you use skaffold debug.

We recommend that you do not add the -trimpath flag when debugging, as this flag removes information from the resulting binary that aids debugging.

Java and Other JVM Languages (runtime: `jvm`)

JVM application are recognized by:

the presence of a JAVA_VERSION or JAVA_TOOL_OPTIONS environment variable, or
the container command-line invokes java.

NodeJS (runtime: `nodejs`)

NodeJS images are recognized by:

the presence of a NODE_VERSION, NODEJS_VERSION, or NODE_ENV environment variable, or
the container command-line invokes node or npm.

Python (runtime: `python`)

Python application are recognized by:

the presence of a standard Python environment variable: PYTHON_VERSION, PYTHONVERBOSE, PYTHONINSPECT, PYTHONOPTIMIZE, PYTHONUSERSITE, PYTHONUNBUFFERED, PYTHONPATH, PYTHONUSERBASE, PYTHONWARNINGS, PYTHONHOME, PYTHONCASEOK, PYTHONIOENCODING, PYTHONHASHSEED, PYTHONDONTWRITEBYTECODE, or
the container command-line invokes python, python2, or python3.

.NET Core (runtime: `dotnet`)

.NET Core application are recognized by:

the presence of a standard .NET environment variable: ASPNETCORE_URLS, DOTNET_RUNNING_IN_CONTAINER, DOTNET_SYSTEM_GLOBALIZATION_INVARIANT, or
the container command-line invokes the dotnet cli

Furthermore, your app must be built with the --configuration Debug options to disable optimizations.

How It works

skaffold debug acts like skaffold dev, but it configures containers in pods for debugging as required for each container’s runtime technology. The associated debugging ports are exposed and labelled so that they can be port-forwarded to the local machine. One notable difference from skaffold dev is that debug disables image rebuilding and syncing as it leads to users accidentally terminating debugging sessions by saving file changes. These behaviours can be re-enabled with the --auto-build, --auto-deploy, and --auto-sync flags, or triggering a devloop iteration using the Skaffold API.

Enabling debugging has two phases:

Configuring: Skaffold automatically examines each built container image and attempts to recognize the underlying language runtime. Container images can be explicitly configured too.
Monitoring: Skaffold watches the cluster to detect when debuggable containers start execution.

Configuring container images for debugging

skaffold debug examines the built artifacts to determine the underlying language runtime technology. Kubernetes manifests that reference these artifacts are transformed on-the-fly to enable the language runtime’s debugging functionality. These transforms add or alter environment variables and entrypoints, and more.

Some language runtimes require additional support files to enable debugging. For these languages, a special set of runtime-specific images are configured as init-containers to populate a shared-volume that is mounted into each of the appropriate containers. These images are hosted at gcr.io/k8s-skaffold/skaffold-debug-support; alternative locations can be specified in Skaffold’s global configuration.

For images that are successfully recognized, Skaffold adds a debug.cloud.google.com/config annotation to the corresponding Kubernetes pod-spec that encode the debugging parameters.

Monitoring for debuggable containers

Once the application is deployed, debug monitors the cluster looking for debuggable pods with a debug.cloud.google.com/config annotation. For each new debuggable pod, Skaffold emits an event that can be used by tools like IDEs to establish a debug session.

Additional changes

debug makes some other adjustments to simplify the debug experience:

Replica Counts: debug rewrites the replica counts to 1 for deployments, replica sets, and stateful sets. This results in requests being serialized so that one request is processed at a time.
Kubernetes Probes: debug changes the timeouts on HTTP-based liveness, readiness, and startup probes to 600 seconds (10 minutes) from the default of 1 second. This change allows probes to be debugged, and avoids negative consequences from blocked probes when the app is already suspended during a debugging session. Failed liveness probes in particular result in the container being terminated and restarted.

The probe timeout value can be set on a per-podspec basis by setting a debug.cloud.google.com/probe-timeouts annotation on the podspec’s metadata with a valid duration (see Go’s time.ParseDuration()). This probe timeout-rewriting can be skipped entirely by using skip. For example:
```
metadata:
  annotations:
    debug.cloud.google.com/probe-timeouts: skip
spec: ...
```

Troubleshooting

My container is not being made debuggable?

Was this image built by Skaffold? debug only works for images that were built by Skaffold so as to avoid affecting system- or infrastructure-level containers such as proxy sidecars.

Was Skaffold able to recognize the image? debug emits a warning when it is unable to configure an image for debugging:

WARN[0005] Image "image-name" not configured for debugging: unable to determine runtime for "image-name"

See the language runtime section details on how container images are recognized.

Is your container command-line using shell constructs?

Automatic debugging of Go and Python applications requires that Skaffold adjust the command-line to use a specific launcher (dlv for Go, or our custom Python launcher to configure the appropriate debugging module). These launchers require that the container command-line invokes the binary or launch script (i.e., a pip-generated launcher script such as gunicorn). Skaffold cannot debug Go and Python applications that use user-created shell scripts, or that use shell constructs like exec or eval. Either rewrite your container image command-line or manually configure your container for debugging. For example:

❌ CMD launch.sh
❌ CMD exec gunicorn app:app
✅ CMD gunicorn app:app
✅ CMD [gunicorn, app:app]
✅ CMD [python, -m, gunicorn, app:app]

Why aren’t my breakpoints being hit?

Missing required language extension. Go, Python, and JavaScript debugging in IDEA Ultimate requires the corresponding language plugin to be installed. If it is not installed, the debugger will not attach and breakpoints will not be hit.

Breakpoints are not hit during program startup. Skaffold configures the containers to run-on-start (sometimes called continue mode), such that the containers do not wait for the debugger to connect. The container will have likely finished its startup by the time the debugger is able to connect and configure the breakpoints, and so breakpoints in the normal startup path are unlikely to be hit.

File mapping misconfiguration between local and container filesystems. Some language runtimes require configuring the IDE with a source map to map files in the container to their corresponding local files. If this mapping is incorrect, then the remote debugging runtime will not be able to find the source file for the breakpoint. The Cloud Code plugins generally prompt the user when required.

Some builds use multi-stage dockerfiles to compile the binary using a build-time image and then copy the binary into a smaller image intended for deployment. In these cases, the source mapping must use the build-time paths. For example, many Go builds use a golang image for building, where the source files are typically copied into /go.
JetBrains GoLand and IDEA Ultimate only support remote breakpoints for Go applications built using Go Modules. The IDE settings must also be explicitly configured to use Go Modules.
JVM languages do not require the mapping as the JVM uses class names rather than file paths.

Can images be debugged without the runtime support images?

The special runtime-support images are provided as a convenience for automatic configuration. You can manually configure your images for debugging by one of the following means:

skaffold debug usually recognizes already-configured container images that use the appropriate debugger. For example, you could use a multi-stage Dockerfile with a debug stage that invokes the application using the debugger.
Use a debug.cloud.google.com/config workload annotation on the pod-spec to describe the debug configuration of each container image in the pod, as described in Workload Annotations.

Limitations

skaffold debug has some limitations.

Unsupported Container Entrypoints

skaffold debug requires being able to examine and alter the command-line used in the container entrypoint. This transformation will not work with images that use intermediate launch scripts or binaries.

Supported Deployers

skaffold debug is only supported with the kubectl, kustomize, and helm deployers.

Note

Helm support requires using Helm v3.1.0 or greater.

Deprecated Workload API Objects

skaffold debug does not support deprecated versions of Workload API objects:

extensions/v1beta1 and apps/v1beta1 was deprecated in Kubernetes 1.8 and removed in Kubernetes 1.16.
apps/v1beta2 was deprecated in Kubernetes 1.9 and removed in Kubernetes 1.16.

Applications should transition to the apps/v1 APIs, introduced in Kubernetes 1.9.

Appendix: IDE Support via Events and Metadata

debug provides additional support for IDEs to detect the debuggable containers and to determine appropriate configuration parameters.

Workload Annotations

Each transformed workload object carries a debug.cloud.google.com/config annotation with a JSON object describing the debug configurations for the pod’s containers (linebreaks for readability):

	debug.cloud.google.com/config={
		"<containerName>":{"runtime":"<runtimeId>",...},
		"<containerName>":{"runtime":"<runtimeId>",...},
		}

For example the following annotation indicates that the container named web is a Go application that is being debugged by a headless Delve session on port 56268 (linebreaks for readability):

debug.cloud.google.com/config={
  "web":{
    "artifact":"gcr.io/random/image",
    "runtime":"go",
    "ports":{"dlv":56268},
    "workingDir":"/some/path"}}

artifact is the corresponding artifact’s image name in the skaffold.yaml. runtime is the language runtime detected (one of: go, jvm, nodejs, python). ports is a list of debug ports keyed by the language runtime debugging protocol. workingDir is the working directory (if not an empty string).

API: Events

Each debuggable container being started or stopped raises a debug-container-event through Skaffold’s event mechanism (gRPC, REST).

`/v1/events` stream of `skaffold debug` within `examples/jib`

In this example, we do a skaffold debug, and then kill the deployed pod. The deployment starts a new pod. We get a terminated event for the container for the killed pod.

{
  "result": {
    "timestamp": "2020-02-05T03:27:30.114354Z",
    "event": {
      "debuggingContainerEvent": {
        "status": "Started",
        "podName": "web-f6d56bcc5-6csgs",
        "containerName": "web",
        "namespace": "default",
        "artifact": "skaffold-jib",
        "runtime": "jvm",
        "debugPorts": {
          "jdwp": 5005
        }
      }
    },
    "entry": "Debuggable container started pod/web-f6d56bcc5-6csgs:web (default)"
  }
}

API: State

The API’s state (gRPC, REST) also includes a list of debuggable containers.

The `/v1/state` listing debugging containers

{
  "buildState": {
    "artifacts": {
      "skaffold-jib": "Complete"
    }
  },
  "deployState": {
    "status": "Complete"
  },
  "forwardedPorts": {
    "5005": {
      "localPort": 5005,
      "remotePort": 5005,
      "podName": "web-f6d56bcc5-6csgs",
      "containerName": "web",
      "namespace": "default",
      "portName": "jdwp",
      "resourceType": "pod",
      "resourceName": "web-f6d56bcc5-6csgs",
      "address": "127.0.0.1"
    },
    "8080": {
      "localPort": 8080,
      "remotePort": 8080,
      "namespace": "default",
      "resourceType": "service",
      "resourceName": "web",
      "address": "127.0.0.1"
    },
    "8081": {
      "localPort": 8081,
      "remotePort": 8080,
      "podName": "web-f6d56bcc5-6csgs",
      "containerName": "web",
      "namespace": "default",
      "resourceType": "pod",
      "resourceName": "web-f6d56bcc5-6csgs",
      "address": "127.0.0.1"
    }
  },
  "statusCheckState": {
    "status": "Not Started"
  },
  "fileSyncState": {
    "status": "Not Started"
  },
  "debuggingContainers": [
    {
      "status": "Started",
      "podName": "web-f6d56bcc5-6csgs",
      "containerName": "web",
      "namespace": "default",
      "artifact": "skaffold-jib",
      "runtime": "jvm",
      "debugPorts": {
        "jdwp": 5005
      }
    }
  ]
}

Docs: Continuous Delivery

Mon, 01 Jan 0001 00:00:00 +0000

Note

Skaffold is used under the hood to power Google Cloud Platform’s Cloud Deploy API. See the docs here for more information on how to use Skaffold + Cloud Deploy for CI/CD

Skaffold provides several features and sub-command “building blocks” that make it very useful for integrating with (or creating entirely new) CI/CD pipelines. The ability to use the same skaffold.yaml for iterative development and continuous delivery eases handing off an application from a development team to an ops team.

Let’s start with the simplest use case: a single, full deployment of your application.

Run entire pipeline end-to-end

skaffold run is a single command for a one-off deployment. It runs through every major phase of the Skaffold lifecycle: building your application images, tagging these images (and optionally pushing them to a remote registry), deploying your application to the target cluster, and monitoring the created resources for readiness.

We recommend skaffold run for the simplest Continuous Delivery setup, where it is sufficient to have a single step that deploys from version control to a cluster.

For more sophisticated Continuous Delivery pipelines, Skaffold offers building blocks:

status-check - wait for deployments to stabilize and succeed only if all deployments are successful
skaffold build - build, tag and push artifacts to a registry
skaffold deploy - deploy built artifacts to a cluster
skaffold render - export the transformed Kubernetes manifests
skaffold apply - send hydrated Kubernetes manifests to the API server to create resources on the target cluster

Traditional continuous delivery

skaffold build will build your project’s artifacts, and push the build images to the specified registry. If your project is already configured to run with Skaffold, skaffold build can be a very lightweight way of setting up builds for your CI pipeline. Passing the --file-output flag to Skaffold build will also write out your built artifacts in JSON format to a file on disk, which can then by passed to skaffold deploy later on. This is a great way of “committing” your artifacts when they have reached a state that you’re comfortable with, especially for projects with multiple artifacts for multiple services.

Example using the current git state as a unique file ID to “commit” build state:

Storing the build result in a commit specific JSON file:

export STATE=$(git rev-list -1 HEAD --abbrev-commit)
skaffold build --file-output build-$STATE.json

outputs the tag generation and cache output from Skaffold:

Generating tags...
 - gcr.io/k8s-skaffold/skaffold-example:v0.41.0-17-g3ad238db
Checking cache...
 - gcr.io/k8s-skaffold/skaffold-example: Found. Tagging

The content of the JSON file

cat build-$STATE.json

looks like:

{"builds":[{"imageName":"gcr.io/k8s-skaffold/skaffold-example","tag":"gcr.io/k8s-skaffold/skaffold-example:v0.41.0-17-g3ad238db@sha256:eeffb639f53368c4039b02a4d337bde44e3acc728b309a84353d4857ee95c369"}]}

We can then use this build result file to deploy with Skaffold:

skaffold deploy -a build-$STATE.json

and as we’d expect, we see a bit of deploy-related output from Skaffold:

Tags used in deployment:
 - gcr.io/k8s-skaffold/skaffold-example -> gcr.io/k8s-skaffold/skaffold-example:v0.41.0-17-g3ad238db@sha256:eeffb639f53368c4039b02a4d337bde44e3acc728b309a84353d4857ee95c369
Starting deploy...
 - pod/getting-started configured

Separation of rendering and deployment

beta

Skaffold allows separating the generation of fully-hydrated Kubernetes manifests from the actual deployment of those manifests, using the skaffold render and skaffold apply commands.

skaffold render builds all application images from your artifacts, templates the newly-generated image tags into your Kubernetes manifests (based on your project’s deployment configuration), and then prints out the final hydrated manifests to a file or your terminal. This allows you to capture the full, declarative state of your application in configuration, such that applying the changes to your cluster can be done as a separate step.

skaffold apply consumes one or more fully-hydrated Kubernetes manifests, and then sends the results directly to the Kubernetes control plane via kubectl to create resources on the target cluster. After creating the resources on your cluster, skaffold apply uses Skaffold’s built-in health checking to monitor the created resources for readiness. See resource health checks for more information on how Skaffold’s resource health checking works.

Note: skaffold apply always uses kubectl to deploy resources to a target cluster, regardless of deployment configuration in the provided skaffold.yaml. Only a small subset of deploy configuration is honored when running skaffold apply:

deploy.statusCheckDeadlineSeconds
deploy.kubeContext
deploy.logs.prefix
deploy.kubectl.flags
deploy.kubectl.defaultNamespace
deploy.kustomize.flags
deploy.kustomize.defaultNamespace

Note

skaffold apply attempts to honor the deployment configuration mentioned above. But when conflicting configuration is detected in a multi-configuration project, skaffold apply will not work.

skaffold apply works with any arbitrary Kubernetes YAML, whether it was generated by Skaffold or not, making it an ideal counterpart to skaffold render.

GitOps-style continuous delivery

You can use this separation of rendering and deploying to enable GitOps CD pipelines.

GitOps-based CD pipelines traditionally see fully-hydrated Kubernetes manifests committed to a configuration Git repository (separate from the application source), which triggers a deployment pipeline that applies the changes to resources on the cluster.

Example: Hydrate manifests then deploy/apply to cluster

First, use skaffold render to hydrate the Kubernetes resource file with a newly-built image tag:

$ skaffold render --output render.yaml

# render.yaml
apiVersion: v1
kind: Pod
metadata:
  name: getting-started
  namespace: default
spec:
  containers:
  - image: gcr.io/k8s-skaffold/skaffold-example:v1.19.0-89-gdbedd2a20-dirty
    name: getting-started

Then, we can apply this output directly to the cluster:

$ skaffold apply render.yaml

Starting deploy...
 - pod/getting-started created
Waiting for deployments to stabilize...
Deployments stabilized in 49.277055ms

Docs: Managing ARM workloads [NEW]

Mon, 01 Jan 0001 00:00:00 +0000

Skaffold has a lot of intelligence built-in to simplify working with ARM workloads. Whether developing on an Apple Silicon Macbook that uses an ARM based chip, or deploying to a GKE Kubernetes cluster having ARM nodes, Skaffold can take away the complexities that arise when the architecture of your development machine and Kubernetes cluster don’t match.

Why is image architecture important?

Container images are built targeting specific Instruction Set Architectures like amd64, arm64, etc. You must use container images that are compatible with the architecture of the node where you intend to run the workloads. For example, to deploy to a GKE cluster running ARM nodes, the image needs to be built for linux/arm64 platform.

All image builders build for different default architecture and not all support cross-architecture builds. For instance Docker will build the image for the same architecture as the host machine, whereas Buildpacks will always build it for amd64.

Additionally, the following combination of development machine and cluster node architectures can make it difficult to build and deploy images correctly:

Dev machine architecture is amd64 while the target cluster runs arm64 nodes.
Dev machine architecture is arm64 (say Apple Silicon Macbooks) while the target cluster runs amd64 nodes.
The target cluster runs both arm64 and amd64 nodes (mixed node pools).

🎉 Skaffold provides an opionated way to handle all these cases effectively. 🎉

Skaffold can set the image architecture automatically

When running Skaffold in an interactive mode like skaffold dev, skaffold debug or skaffold run where the intention is to build an image from the application code, and immediately deploy it to a Kubernetes cluster, Skaffold will check the active Kubernetes cluster node architecture and provide that as an argument to the respective image builder. If the cluster has multiple architecture nodes, then Skaffold will also create appropriate Kubernetes affinity rules so that the Kubernetes Pods with these images are assigned to matching architecture nodes.

Note

Skaffold will create platform node affinity rules only for clusters having multiple architecture nodes. You can also force this using the flag --enable-platform-node-affinity=true to always create these affinity rules in the Kubernetes manifests for built images.

Let’s test this in a sample Golang project:

The skaffold.yaml file defines a single Docker build artifact and deploys it in a Kubernetes Pod.

First set the active Kubernetes context to a cluster having only linux/amd64 nodes, and run:

skaffold dev --default-repo=your/container/registy

Skaffold will detect the cluster node platform linux/amd64 and build the image for this platform:

skaffold dev --default-repo=gcr.io/k8s-skaffold
Listing files to watch...
- skaffold-example
Generating tags...
- skaffold-example -> gcr.io/k8s-skaffold/skaffold-example:latest
Starting build...
Building [skaffold-example]...
Target platforms: [linux/amd64]
...
Build [skaffold-example] succeeded
Starting deploy...
- pod/getting-started created
Waiting for deployments to stabilize...
- pods is ready.
Deployments stabilized in 7.42 seconds
Press Ctrl+C to exit
Watching for changes...
[getting-started] Hello world! Running on linux/amd64

Now set the active Kubernetes context to a cluster containing only linux/arm64 nodes. See here to know how you can create an ARM GKE cluster.

Re-running the dev command will now build a linux/arm64 image.
```
skaffold dev --default-repo=gcr.io/k8s-skaffold
...
...
[getting-started] Hello world! Running on linux/arm64
```
Now set the active Kubernetes context to a cluster containing both linux/arm64 and linux/amd64 nodes. You can create a GKE cluster with 2 node pools, one having linux/amd64 nodes, and the other having linux/arm64 nodes.

Re-run the dev command but with an explicit platform target this time via the --platform flag. If we don’t provide the target platform explicitly then Skaffold will choose one between linux/amd64 and linux/arm64, trying to match your local dev machine architecture.
```
skaffold dev --default-repo=your/container/registy --platform=linux/amd64
```
Skaffold will build a linux/amd64 image and insert a nodeAffinity definition to the Pod so that it gets scheduled on the matching architecture node.
```
skaffold dev --default-repo=gcr.io/k8s-skaffold --platform=linux/amd64
...
...
[getting-started] Hello world! Running on linux/amd64
```

Validate that the nodeAffinity was applied by running the command (skip | jq if you don’t have jq installed):

kubectl get pod getting-started  -o=jsonpath='{.spec.affinity}' | jq
{
  "nodeAffinity": {
    "requiredDuringSchedulingIgnoredDuringExecution": {
      "nodeSelectorTerms": [
        {
          "matchExpressions": [
            {
              "key": "kubernetes.io/os",
              "operator": "In",
              "values": [
                "linux"
              ]
            },
            {
              "key": "kubernetes.io/arch",
              "operator": "In",
              "values": [
                "amd64"
              ]
            }
          ]
        }
      ]
    }
  }
}

This example will run the same whether you’re using an arm64 machine (say an Apple Silicon Macbook) or an amd64 machine.

Skaffold also supports cross-architecture builds on Google Cloud Build. You can rerun this example, with the additional flag --profile cloudbuild to all the dev commands to build on Google Cloud Build instead of the local Docker daemon.

What about multi-arch images?

A multi-arch image is an image that can support multiple architectures. It looks like a single image with a single tag, but is actually a list of images targeting multiple architectures organized by an image index. When you deploy a multi-arch image to a cluster, the container runtime automatically chooses the right image that is compatible with the architecture of the node to which it is being deployed. This simplifies targeting multiple clusters of different architecture nodes, and/or mixed-architecture nodes.

Skaffold supports building multi-platform images natively using the jib builder, the ko builder and the custom builder. For other builders that support building cross-architecture images, Skaffold will iteratively build a single platform image for each target architecture and stitch them together into a multi-platform image, and push it to the registry.

Let’s test this in the same sample Golang project as before:

Run this command to build for the target architectures linux/amd64 and linux/arm64:

skaffold build -t latest --default-repo=your/container/registy --platform=linux/amd64,linux/arm64
...
Building [skaffold-example]...
Target platforms: [linux/amd64,linux/arm64]
...
[+] Building 0.3s (13/13) FINISHED
...
=> => writing image sha256:10af3142e460566f5791c48758f0040cef6932cbcb0766082dcbb0d8db7653e7
=> => naming to gcr.io/k8s-skaffold/skaffold-example:latest_linux_amd64
...
latest_linux_amd64: digest: sha256:15bd4f2380e99b3563f8add1aba9691e414d4cc5701363d9c74960a20fb276c4 size: 739
...
[+] Building 52.8s (13/13) FINISHED
...
=> => writing image sha256:68866691e2d6f079b116e097ae4e67a53eaf89e825b52d6f31f2e9cc566974de
=> => naming to gcr.io/k8s-skaffold/skaffold-example:latest_linux_arm64
...
4e0c2525c370: Pushed
latest_linux_arm64: digest: sha256:868d0aec1cc7d2ed1fa1e840f38ff1aa50c3cc3d3232ea17a065618eaec4e82b size: 739
Build [skaffold-example] succeeded

Validate that the image just built was multi-arch, by running the following docker command:

docker manifest inspect your/container/registry/skaffold-example:latest | grep -A 3 "platform"

Outputs:

  "platform": {
      "architecture": "amd64",
      "os": "linux"
  }
--
  "platform": {
      "architecture": "arm64",
      "os": "linux"
  }

Now if we render the Kubernetes Pod manifest for this multi-arch image, then it’ll have platform affinity definition targeting both linux/amd64 and linux/arm64 architectures.

skaffold render --default-repo=your/container/registry --enable-platform-node-affinity

Outputs:

apiVersion: v1
kind: Pod
metadata:
  name: getting-started
  namespace: default
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: kubernetes.io/os
            operator: In
            values:
            - linux
          - key: kubernetes.io/arch
            operator: In
            values:
            - amd64
        - matchExpressions:
          - key: kubernetes.io/os
            operator: In
            values:
            - linux
          - key: kubernetes.io/arch
            operator: In
            values:
            - arm64
  containers:
  - image: gcr.io/k8s-skaffold/skaffold-example:latest@sha256:9ecf4e52f7ff64b35deacf9d6eedc03f35d69e0b4bf3679b97ba492f4389f784
    name: getting-started

Note

Multi-arch images need to be pushed to a container registry, as the local Docker deamon doesn’t yet support storing multi-arch images.
For interactive modes like skaffold dev and skaffold debug requiring fast and repeated build-render-deploy iterations, Skaffold will choose only one build architecture and build a single-platform image, even if you specify multiple target platforms.
If you need to build a multi-arch image with an interactive mode then use skaffold run. This will build the multi-arch image and deploy it to the active Kubernetes cluster.

Skaffold – Guides

Docs: Getting Started With Your Project

What’s next

Docs: skaffold dev

💡 Tip

Dev loop

Precedence of Actions

File Watcher and Watch Modes

Controlling the Dev Loop with API

Note

Docs: Debugging With Skaffold

(Recommended) Debugging using Cloud Code

Detailed debugger configuration and setup

Go (runtime: go, protocols: dlv)

Skaffold debug using the JetBrains GoLand and IntelliJ Ultimate

Skaffold debug using the VS Code Go extension

NodeJS (runtime: nodejs, protocols: devtools)

Note

Java and Other JVM Languages (runtime: jvm, protocols: jdwp)

Python (runtime: python, protocols: dap or pydevd)

Note

.NET Core (runtime: dotnet, protocols: vsdbg)

JetBrains Rider

Omnisharp for VS Code

Supported Language Runtimes

Go (runtime: go)

Java and Other JVM Languages (runtime: jvm)

NodeJS (runtime: nodejs)

Python (runtime: python)

.NET Core (runtime: dotnet)

How It works

Configuring container images for debugging

Monitoring for debuggable containers

Additional changes

Troubleshooting

My container is not being made debuggable?

Why aren’t my breakpoints being hit?

Can images be debugged without the runtime support images?

Limitations

Unsupported Container Entrypoints

Supported Deployers

Note

Deprecated Workload API Objects

Appendix: IDE Support via Events and Metadata

Workload Annotations

API: Events

API: State

Docs: Continuous Delivery

Note

Run entire pipeline end-to-end

Traditional continuous delivery

Separation of rendering and deployment

Note

GitOps-style continuous delivery

Example: Hydrate manifests then deploy/apply to cluster

Docs: Managing ARM workloads [NEW]

Why is image architecture important?

Skaffold can set the image architecture automatically

Note

What about multi-arch images?

Note

Go (runtime: `go`, protocols: `dlv`)

NodeJS (runtime: `nodejs`, protocols: `devtools`)

Java and Other JVM Languages (runtime: `jvm`, protocols: `jdwp`)

Python (runtime: `python`, protocols: `dap` or `pydevd`)

.NET Core (runtime: `dotnet`, protocols: `vsdbg`)

Go (runtime: `go`)

Java and Other JVM Languages (runtime: `jvm`)

NodeJS (runtime: `nodejs`)

Python (runtime: `python`)

.NET Core (runtime: `dotnet`)