com.docker.build repeatedly spawns ssh threads that never terminate

[HollisTech]

Description

After upgrading to 4.20.1 my Windows 11 system was observed to be spawning lots of ssh processes from the com.docker.build.exe application. They all were idle but not terminating. The command line for ssh was like this "ssh -l xxxx -- xxx.yyy.zzz.aaa docker system dial-stdio". On investigation the targets were all 'contexts' in docker for windows configured for ssh rather than npipe access. As an experiment I removed all the ssh based contexts. The process generation problem stopped, although it left 135 ssh processes.

Reproduce

1. configure ssh based contexts for several other hosts (3 in my case.)
2. upgrade to 4.20.1 or restart docker if already at 4.20.1
3. monitor processes using taskmanager or procexplorer, or powershell, e.g.
   get-process -name ssh | measure-object
   and track the number of such processes
4. observe that the count increases periodically and there can be hundreds of processes.

Expected behavior

There should be only a small, non incrementing number of ssh processes.

docker version

Client:
 Cloud integration: v1.0.33
 Version:           24.0.2
 API version:       1.43
 Go version:        go1.20.4
 Git commit:        cb74dfc
 Built:             Thu May 25 21:53:15 2023
 OS/Arch:           windows/amd64
 Context:           default
Server: Docker Desktop 4.20.1 (110738)
 Engine:
  Version:          24.0.2
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.4
  Git commit:       659604f
  Built:            Thu May 25 21:52:17 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.21
  GitCommit:        3dce8eb055cbb6872793272b4f20ed16117344f8
 runc:
  Version:          1.1.7
  GitCommit:        v1.1.7-0-g860f061
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Version:    24.0.2
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.10.5
    Path:     C:\Program Files\Docker\cli-plugins\docker-buildx.exe
  compose: Docker Compose (Docker Inc.)
    Version:  v2.18.1
    Path:     C:\Program Files\Docker\cli-plugins\docker-compose.exe
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-dev.exe
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.19
    Path:     C:\Program Files\Docker\cli-plugins\docker-extension.exe
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v0.1.0-beta.4
    Path:     C:\Program Files\Docker\cli-plugins\docker-init.exe
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-sbom.exe
  scan: Docker Scan (Docker Inc.)
    Version:  v0.26.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-scan.exe
  scout: Command line tool for Docker Scout (Docker Inc.)
    Version:  v0.12.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-scout.exe

Server:
 Containers: 5
  Running: 3
  Paused: 0
  Stopped: 2
 Images: 8
 Server Version: 24.0.2
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 3dce8eb055cbb6872793272b4f20ed16117344f8
 runc version: v1.1.7-0-g860f061
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
 Kernel Version: 5.15.90.1-microsoft-standard-WSL2
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 15.58GiB
 Name: docker-desktop
 ID: 85ee50f6-8745-4730-80d1-f3999ac3a397
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support

Diagnostics ID

B9C4D290-D432-44C2-8003-48CBB992C71D/20230622224344

Additional Info

No response

[HollisTech]

Also on inspection the targets were also flooded with non-terminating sshd processes. This was like a DOS attack on my lan. I had to reboot everyone.

[crazy-max]

This is related to moby/moby#45539. This has been patched in com.docker.build and will be available in next 4.21.

[crazy-max]

@HollisTech Would you mind showing the output of the following commands please?

• docker context ls
• docker buildx ls

Thanks

[HollisTech]

@crazy-max Of course I deleted the ssh contexts as they are causing the problem.
I've restored one of the ssh contexts and ssh processes are piling up :-)

NAME	TYPE	DESCRIPTION	DOCKER ENDPOINT	KUBERNETES ENDPOINT	ORCHESTRATOR
default *	moby	Current DOCKER_HOST based configuration	npipe:////./pipe/docker_engine
desktop-linux	moby	Docker Desktop	npipe:////./pipe/dockerDesktopLinuxEngine
kant	moby		ssh://xxxxx@kant.zzzz.aaaa.bbb

NAME/NODE	DRIVER/ENDPOINT	STATUS	BUILDKIT	PLATFORMS
default *	docker
default	default	running	v0.11.7-0.20230525183624-798ad6b0ce9f	linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/mips64le, linux/mips64, linux/arm/v7, linux/arm/v6
desktop-linux	docker
desktop-linux	desktop-linux	running	v0.11.7-0.20230525183624-798ad6b0ce9f	linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/mips64le, linux/mips64, linux/arm/v7, linux/arm/v6
kant	docker
kant	kant	running	20.10.9	windows/amd64, linux/amd64

[crazy-max]

20.10.9

Are also other SSH contexts against a Docker engine 20.10? Have you tried with an up to date Docker engine? (Docker 24.0.2 atm).

[HollisTech]

The other ssh contexts were linux systems (ubuntu 20.04) running Version: 20.10.16.
Same results: ssh sessions pile up on both ends.
No I am not interested in updating the docker hosts as they provide my local infrastructure. The ssh bug started on updating docker for windows. The contexts were just a convenience in case I wanted to manage docker from my windows session, I have many other ways to do that.

[crazy-max]

@HollisTech This will be fixed in next Docker Desktop 4.21.1

[lorenrh]

We have released Docker Desktop 4.21.1, containing a fix for this issue! Please see: https://docs.docker.com/desktop/release-notes/#4211