[gentoo-user] Docker installation issues

Discussion:

(too old to reply)

Mansour Al Akeel

2022-12-10 17:40:02 UTC

I am using Openrc

This was my initial /etc/conf.d/docker
DOCKER_OPTS="--storage-driver overlay2 --data-root /srv/var/lib/docker"

when I try:

$ docker pull hello-world

Error response from daemon: Get "https://registry-1.docker.io/v2/":
net/http: request canceled while waiting for connection (Client.Timeout
exceeded while awaiting headers)

Trying to troubleshoot:

localhost /home/mansour # tail -n 20 /var/log/docker.log
time="2022-12-10T12:17:03.473550705-05:00" level=info msg="scheme \"unix\"
not registered, fallback to default scheme" module=grpc
time="2022-12-10T12:17:03.473566413-05:00" level=info
msg="ccResolverWrapper: sending update to cc:
{[{unix:///run/containerd/containerd.sock <nil> 0 <nil>}] <nil> <nil>}"
module=grpc
time="2022-12-10T12:17:03.473573787-05:00" level=info msg="ClientConn
switching balancer to \"pick_first\"" module=grpc
time="2022-12-10T12:17:03.474530993-05:00" level=info msg="parsed scheme:
\"unix\"" module=grpc
time="2022-12-10T12:17:03.474545549-05:00" level=info msg="scheme \"unix\"
not registered, fallback to default scheme" module=grpc
time="2022-12-10T12:17:03.474563752-05:00" level=info
msg="ccResolverWrapper: sending update to cc:
{[{unix:///run/containerd/containerd.sock <nil> 0 <nil>}] <nil> <nil>}"
module=grpc
time="2022-12-10T12:17:03.474571186-05:00" level=info msg="ClientConn
switching balancer to \"pick_first\"" module=grpc
time="2022-12-10T12:17:03.478908716-05:00" level=warning msg="Your kernel
does not support cgroup blkio weight"
time="2022-12-10T12:17:03.478927115-05:00" level=warning msg="Your kernel
does not support cgroup blkio weight_device"
time="2022-12-10T12:17:03.479037897-05:00" level=info msg="Loading
containers: start."
time="2022-12-10T12:17:03.495743563-05:00" level=info msg="failed to read
ipv6 net.ipv6.conf.<bridge>.accept_ra" bridge=docker0
syspath=/proc/sys/net/ipv6/conf/docker0/accept_ra
time="2022-12-10T12:17:03.518761943-05:00" level=info msg="Default bridge
(docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip
can be used to set a preferred IP address"
time="2022-12-10T12:17:03.518886881-05:00" level=info msg="failed to read
ipv6 net.ipv6.conf.<bridge>.accept_ra" bridge=docker0
syspath=/proc/sys/net/ipv6/conf/docker0/accept_ra
time="2022-12-10T12:17:03.534616741-05:00" level=info msg="Loading
containers: done."
time="2022-12-10T12:17:03.541080189-05:00" level=info msg="Docker daemon"
commit=a89b84221c graphdriver(s)=overlay2 version=20.10.17
time="2022-12-10T12:17:03.541122352-05:00" level=info msg="Daemon has
completed initialization"
time="2022-12-10T12:17:03.549888103-05:00" level=info msg="API listen on
/var/run/docker.sock"
time="2022-12-10T12:17:27.025622231-05:00" level=warning msg="Error getting
v2 registry: Get \"https://registry-1.docker.io/v2/\": net/http: request
canceled while waiting for connection (Client.Timeout exceeded while
awaiting headers)"
time="2022-12-10T12:17:27.025667054-05:00" level=info msg="Attempting next
endpoint for pull after error: Get \"https://registry-1.docker.io/v2/\":
net/http: request canceled while waiting for connection (Client.Timeout
exceeded while awaiting headers)"
time="2022-12-10T12:17:27.026851821-05:00" level=error msg="Handler for
POST /v1.41/images/create returned error: Get \"
https://registry-1.docker.io/v2/\": net/http: request canceled while
waiting for connection (Client.Timeout exceeded while awaiting headers)"

time="2022-12-10T12:15:42.036053086-05:00" level=info msg="loading plugin
\"io.containerd.internal.v1.tracing\"..." type=io.containerd.internal.v1
time="2022-12-10T12:15:42.036068412-05:00" level=error msg="failed to
initialize a tracing processor \"otlp\"" error="no OpenTelemetry endpoint:
skip plugin"
time="2022-12-10T12:15:42.036100189-05:00" level=info msg="loading plugin
\"io.containerd.grpc.v1.cri\"..." type=io.containerd.grpc.v1
time="2022-12-10T12:15:42.036390695-05:00" level=info msg="Start cri plugin
with config {PluginConfig:{ContainerdConfig:{Snapshotter:overlayfs
DefaultRuntimeName:runc DefaultRuntime:{Type: Path: Engine:
PodAnnotations:[] ContainerAnnotations:[] Root: Options:map[]
PrivilegedWithoutHostDevices:false BaseRuntimeSpec: NetworkPluginConfDir:
NetworkPluginMaxConfNum:0} UntrustedWorkloadRuntime:{Type: Path: Engine:
PodAnnotations:[] ContainerAnnotations:[] Root: Options:map[]
PrivilegedWithoutHostDevices:false BaseRuntimeSpec: NetworkPluginConfDir:
NetworkPluginMaxConfNum:0} Runtimes:map[runc:{Type:io.containerd.runc.v2
Path: Engine: PodAnnotations:[] ContainerAnnotations:[] Root:
Options:map[BinaryName: CriuImagePath: CriuPath: CriuWorkPath: IoGid:0
IoUid:0 NoNewKeyring:false NoPivotRoot:false Root: ShimCgroup:
SystemdCgroup:false] PrivilegedWithoutHostDevices:false BaseRuntimeSpec:
NetworkPluginConfDir: NetworkPluginMaxConfNum:0}] NoPivot:false
DisableSnapshotAnnotations:true DiscardUnpackedLayers:false
IgnoreRdtNotEnabledErrors:false}
CniConfig:{NetworkPluginBinDir:/opt/cni/bin
NetworkPluginConfDir:/etc/cni/net.d NetworkPluginMaxConfNum:1
NetworkPluginConfTemplate: IPPreference:} Registry:{ConfigPath:
Mirrors:map[] Configs:map[] Auths:map[] Headers:map[]}
ImageDecryption:{KeyModel:node} DisableTCPService:true
StreamServerAddress:127.0.0.1 StreamServerPort:0 StreamIdleTimeout:4h0m0s
EnableSelinux:false SelinuxCategoryRange:1024 SandboxImage:
k8s.gcr.io/pause:3.6 StatsCollectPeriod:10 SystemdCgroup:false
EnableTLSStreaming:false X509KeyPairStreaming:{TLSCertFile: TLSKeyFile:}
MaxContainerLogLineSize:16384 DisableCgroup:false DisableApparmor:false
RestrictOOMScoreAdj:false MaxConcurrentDownloads:3 DisableProcMount:false
UnsetSeccompProfile: TolerateMissingHugetlbController:true
DisableHugetlbController:true DeviceOwnershipFromSecurityContext:false
IgnoreImageDefinedVolumes:false NetNSMountsUnderStateDir:false
EnableUnprivilegedPorts:false EnableUnprivilegedICMP:false}
ContainerdRootDir:/var/lib/containerd
ContainerdEndpoint:/run/containerd/containerd.sock
RootDir:/var/lib/containerd/io.containerd.grpc.v1.cri
StateDir:/run/containerd/io.containerd.grpc.v1.cri}"
time="2022-12-10T12:15:42.036530498-05:00" level=info msg="Connect
containerd service"
time="2022-12-10T12:15:42.036627681-05:00" level=info msg="Get image
filesystem path
\"/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs\""
time="2022-12-10T12:15:42.036896077-05:00" level=error msg="failed to load
cni during init, please check CRI plugin status before setting up network
for pods" error="cni config load failed: no network config found in
/etc/cni/net.d: cni plugin not initialized: failed to load cni config"
time="2022-12-10T12:15:42.036986623-05:00" level=info msg="Start
subscribing containerd event"
time="2022-12-10T12:15:42.037028034-05:00" level=info msg="Start recovering
state"
time="2022-12-10T12:15:42.037035874-05:00" level=info msg=serving...
address=/run/containerd/containerd.sock.ttrpc
time="2022-12-10T12:15:42.037068181-05:00" level=info msg=serving...
address=/run/containerd/containerd.sock
time="2022-12-10T12:15:42.037085346-05:00" level=info msg="containerd
successfully booted in 0.015889s"
time="2022-12-10T12:15:42.037072303-05:00" level=info msg="Start event
monitor"
time="2022-12-10T12:15:42.037107987-05:00" level=info msg="Start snapshots
syncer"
time="2022-12-10T12:15:42.037119126-05:00" level=info msg="Start cni
network conf syncer for default"
time="2022-12-10T12:15:42.037127266-05:00" level=info msg="Start streaming
server"

Looking at resolv.conf

localhost /home/mansour # cat /etc/resolv.conf
# Generated by dhcpcd from wlp3s0.dhcp
# /etc/resolv.conf.head can replace this line
domain Home
nameserver 192.168.1.1
# /etc/resolv.conf.tail can replace this line

So, I added the following to my /etc/dhcpcd.conf
static domain_name_servers=8.8.8.8 8.8.4.4 192.168.1.1

And restarted dhcpcd:

localhost /home/mansour # cat /etc/resolv.conf
# Generated by dhcpcd from wlp3s0.dhcp
# /etc/resolv.conf.head can replace this line
domain Home
nameserver 8.8.8.8
nameserver 8.8.4.4
nameserver 192.168.1.1
# /etc/resolv.conf.tail can replace this line

docker.log still shows the same error:
time="2022-12-10T12:28:16.519126445-05:00" level=warning msg="Error getting
v2 registry: Get \"https://registry-1.docker.io/v2/\": net/http: request
canceled while waiting for connection (Client.Timeout exceeded while
awaiting headers)"
time="2022-12-10T12:28:16.519173944-05:00" level=info msg="Attempting next
endpoint for pull after error: Get \"https://registry-1.docker.io/v2/\":
net/http: request canceled while waiting for connection (Client.Timeout
exceeded while awaiting headers)"
time="2022-12-10T12:28:16.520489790-05:00" level=error msg="Handler for
POST /v1.41/images/create returned error: Get \"
https://registry-1.docker.io/v2/\": net/http: request canceled while
waiting for connection (Client.Timeout exceeded while awaiting headers)"

I am not able to login with "docker login"

Where do I look further for additional troubleshooting hints? Google didn't
help a lot.

Thank you.

Andreas Fink

2022-12-10 18:40:01 UTC

Permalink

On Sat, 10 Dec 2022 12:30:40 -0500

Post by Mansour Al Akeel
I am using Openrc
This was my initial /etc/conf.d/docker
DOCKER_OPTS="--storage-driver overlay2 --data-root /srv/var/lib/docker"
$ docker pull hello-world
net/http: request canceled while waiting for connection (Client.Timeout
exceeded while awaiting headers)
localhost /home/mansour # tail -n 20 /var/log/docker.log
time="2022-12-10T12:17:03.473550705-05:00" level=info msg="scheme \"unix\"
not registered, fallback to default scheme" module=grpc
time="2022-12-10T12:17:03.473566413-05:00" level=info
{[{unix:///run/containerd/containerd.sock <nil> 0 <nil>}] <nil> <nil>}"
module=grpc
time="2022-12-10T12:17:03.473573787-05:00" level=info msg="ClientConn
switching balancer to \"pick_first\"" module=grpc
\"unix\"" module=grpc
time="2022-12-10T12:17:03.474545549-05:00" level=info msg="scheme \"unix\"
not registered, fallback to default scheme" module=grpc
time="2022-12-10T12:17:03.474563752-05:00" level=info
{[{unix:///run/containerd/containerd.sock <nil> 0 <nil>}] <nil> <nil>}"
module=grpc
time="2022-12-10T12:17:03.474571186-05:00" level=info msg="ClientConn
switching balancer to \"pick_first\"" module=grpc
time="2022-12-10T12:17:03.478908716-05:00" level=warning msg="Your kernel
does not support cgroup blkio weight"
time="2022-12-10T12:17:03.478927115-05:00" level=warning msg="Your kernel
does not support cgroup blkio weight_device"
time="2022-12-10T12:17:03.479037897-05:00" level=info msg="Loading
containers: start."
time="2022-12-10T12:17:03.495743563-05:00" level=info msg="failed to read
ipv6 net.ipv6.conf.<bridge>.accept_ra" bridge=docker0
syspath=/proc/sys/net/ipv6/conf/docker0/accept_ra
time="2022-12-10T12:17:03.518761943-05:00" level=info msg="Default bridge
(docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip
can be used to set a preferred IP address"
time="2022-12-10T12:17:03.518886881-05:00" level=info msg="failed to read
ipv6 net.ipv6.conf.<bridge>.accept_ra" bridge=docker0
syspath=/proc/sys/net/ipv6/conf/docker0/accept_ra
time="2022-12-10T12:17:03.534616741-05:00" level=info msg="Loading
containers: done."
time="2022-12-10T12:17:03.541080189-05:00" level=info msg="Docker daemon"
commit=a89b84221c graphdriver(s)=overlay2 version=20.10.17
time="2022-12-10T12:17:03.541122352-05:00" level=info msg="Daemon has
completed initialization"
time="2022-12-10T12:17:03.549888103-05:00" level=info msg="API listen on
/var/run/docker.sock"
time="2022-12-10T12:17:27.025622231-05:00" level=warning msg="Error getting
v2 registry: Get \"https://registry-1.docker.io/v2/\": net/http: request
canceled while waiting for connection (Client.Timeout exceeded while
awaiting headers)"
time="2022-12-10T12:17:27.025667054-05:00" level=info msg="Attempting next
net/http: request canceled while waiting for connection (Client.Timeout
exceeded while awaiting headers)"
time="2022-12-10T12:17:27.026851821-05:00" level=error msg="Handler for
POST /v1.41/images/create returned error: Get \"
https://registry-1.docker.io/v2/\": net/http: request canceled while
waiting for connection (Client.Timeout exceeded while awaiting headers)"
time="2022-12-10T12:15:42.036053086-05:00" level=info msg="loading plugin
\"io.containerd.internal.v1.tracing\"..." type=io.containerd.internal.v1
time="2022-12-10T12:15:42.036068412-05:00" level=error msg="failed to
skip plugin"
time="2022-12-10T12:15:42.036100189-05:00" level=info msg="loading plugin
\"io.containerd.grpc.v1.cri\"..." type=io.containerd.grpc.v1
time="2022-12-10T12:15:42.036390695-05:00" level=info msg="Start cri plugin
with config {PluginConfig:{ContainerdConfig:{Snapshotter:overlayfs
PodAnnotations:[] ContainerAnnotations:[] Root: Options:map[]
PodAnnotations:[] ContainerAnnotations:[] Root: Options:map[]
NetworkPluginMaxConfNum:0} Runtimes:map[runc:{Type:io.containerd.runc.v2
Options:map[BinaryName: CriuImagePath: CriuPath: CriuWorkPath: IoGid:0
NetworkPluginConfDir: NetworkPluginMaxConfNum:0}] NoPivot:false
DisableSnapshotAnnotations:true DiscardUnpackedLayers:false
IgnoreRdtNotEnabledErrors:false}
CniConfig:{NetworkPluginBinDir:/opt/cni/bin
NetworkPluginConfDir:/etc/cni/net.d NetworkPluginMaxConfNum:1
Mirrors:map[] Configs:map[] Auths:map[] Headers:map[]}
ImageDecryption:{KeyModel:node} DisableTCPService:true
StreamServerAddress:127.0.0.1 StreamServerPort:0 StreamIdleTimeout:4h0m0s
k8s.gcr.io/pause:3.6 StatsCollectPeriod:10 SystemdCgroup:false
EnableTLSStreaming:false X509KeyPairStreaming:{TLSCertFile: TLSKeyFile:}
MaxContainerLogLineSize:16384 DisableCgroup:false DisableApparmor:false
RestrictOOMScoreAdj:false MaxConcurrentDownloads:3 DisableProcMount:false
UnsetSeccompProfile: TolerateMissingHugetlbController:true
DisableHugetlbController:true DeviceOwnershipFromSecurityContext:false
IgnoreImageDefinedVolumes:false NetNSMountsUnderStateDir:false
EnableUnprivilegedPorts:false EnableUnprivilegedICMP:false}
ContainerdRootDir:/var/lib/containerd
ContainerdEndpoint:/run/containerd/containerd.sock
RootDir:/var/lib/containerd/io.containerd.grpc.v1.cri
StateDir:/run/containerd/io.containerd.grpc.v1.cri}"
time="2022-12-10T12:15:42.036530498-05:00" level=info msg="Connect
containerd service"
time="2022-12-10T12:15:42.036627681-05:00" level=info msg="Get image
filesystem path
\"/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs\""
time="2022-12-10T12:15:42.036896077-05:00" level=error msg="failed to load
cni during init, please check CRI plugin status before setting up network
for pods" error="cni config load failed: no network config found in
/etc/cni/net.d: cni plugin not initialized: failed to load cni config"
time="2022-12-10T12:15:42.036986623-05:00" level=info msg="Start
subscribing containerd event"
time="2022-12-10T12:15:42.037028034-05:00" level=info msg="Start recovering
state"
time="2022-12-10T12:15:42.037035874-05:00" level=info msg=serving...
address=/run/containerd/containerd.sock.ttrpc
time="2022-12-10T12:15:42.037068181-05:00" level=info msg=serving...
address=/run/containerd/containerd.sock
time="2022-12-10T12:15:42.037085346-05:00" level=info msg="containerd
successfully booted in 0.015889s"
time="2022-12-10T12:15:42.037072303-05:00" level=info msg="Start event
monitor"
time="2022-12-10T12:15:42.037107987-05:00" level=info msg="Start snapshots
syncer"
time="2022-12-10T12:15:42.037119126-05:00" level=info msg="Start cni
network conf syncer for default"
time="2022-12-10T12:15:42.037127266-05:00" level=info msg="Start streaming
server"
Looking at resolv.conf
localhost /home/mansour # cat /etc/resolv.conf
# Generated by dhcpcd from wlp3s0.dhcp
# /etc/resolv.conf.head can replace this line
domain Home
nameserver 192.168.1.1
# /etc/resolv.conf.tail can replace this line
So, I added the following to my /etc/dhcpcd.conf
static domain_name_servers=8.8.8.8 8.8.4.4 192.168.1.1
localhost /home/mansour # cat /etc/resolv.conf
# Generated by dhcpcd from wlp3s0.dhcp
# /etc/resolv.conf.head can replace this line
domain Home
nameserver 8.8.8.8
nameserver 8.8.4.4
nameserver 192.168.1.1
# /etc/resolv.conf.tail can replace this line
time="2022-12-10T12:28:16.519126445-05:00" level=warning msg="Error getting
v2 registry: Get \"https://registry-1.docker.io/v2/\": net/http: request
canceled while waiting for connection (Client.Timeout exceeded while
awaiting headers)"
time="2022-12-10T12:28:16.519173944-05:00" level=info msg="Attempting next
net/http: request canceled while waiting for connection (Client.Timeout
exceeded while awaiting headers)"
time="2022-12-10T12:28:16.520489790-05:00" level=error msg="Handler for
POST /v1.41/images/create returned error: Get \"
https://registry-1.docker.io/v2/\": net/http: request canceled while
waiting for connection (Client.Timeout exceeded while awaiting headers)"
I am not able to login with "docker login"
Where do I look further for additional troubleshooting hints? Google didn't
help a lot.
Thank you.

Hello,
my first step would be checking if there is some inherent network
problem reaching registry-1.docker.io/v2, e.g. via
curl -v https://registry-1.docker.io/v2/

If that runs successfully, you could try if there is an MTU problem. I
remember that I had to set MTU to 1450 in some virtualized environment,
otherwise docker would not run correctly, i.e. I had in
/etc/conf.d/docer
DOCKER_OPTS="--mtu=1450"

Hope that gives you an idea what to look for.

If both does not help, there is wireshark, to watch your network, very
low-level, but might give some hints.

Cheers
Andreas

Mansour Al Akeel

2022-12-10 21:00:02 UTC

Permalink

Andreas,

Thank you very much. In fact I didn't go that deep yet, and not sure if I
should. I just found that the url is not accessible even from a browser.
Googling a bit, tells me there is no clear URL for docker-registry and
possibly this one is outdated. I will continue looking around to confirm
what the current default repo should be. If you have any suggestions,
please let me know.

localhost in ~
â â curl -k -v https://registry-1.docker.io/v2/
* Trying 34.228.211.243:443...

* connect to 34.228.211.243 port 443 failed: Connection timed out
* Failed to connect to registry-1.docker.io port 443 after 129401 ms:
Couldn't connect to server
* Closing connection 0
curl: (28) Failed to connect to registry-1.docker.io port 443 after 129401
ms: Couldn't connect to server

Post by Andreas Fink
On Sat, 10 Dec 2022 12:30:40 -0500

\"unix\"

Post by Mansour Al Akeel
not registered, fallback to default scheme" module=grpc
time="2022-12-10T12:17:03.473566413-05:00" level=info
{[{unix:///run/containerd/containerd.sock <nil> 0 <nil>}] <nil> <nil>}"
module=grpc
time="2022-12-10T12:17:03.473573787-05:00" level=info msg="ClientConn
switching balancer to \"pick_first\"" module=grpc
\"unix\"" module=grpc
time="2022-12-10T12:17:03.474545549-05:00" level=info msg="scheme

\"unix\"

Post by Mansour Al Akeel
not registered, fallback to default scheme" module=grpc
time="2022-12-10T12:17:03.474563752-05:00" level=info
{[{unix:///run/containerd/containerd.sock <nil> 0 <nil>}] <nil> <nil>}"
module=grpc
time="2022-12-10T12:17:03.474571186-05:00" level=info msg="ClientConn
switching balancer to \"pick_first\"" module=grpc
time="2022-12-10T12:17:03.478908716-05:00" level=warning msg="Your kernel
does not support cgroup blkio weight"
time="2022-12-10T12:17:03.478927115-05:00" level=warning msg="Your kernel
does not support cgroup blkio weight_device"
time="2022-12-10T12:17:03.479037897-05:00" level=info msg="Loading
containers: start."
time="2022-12-10T12:17:03.495743563-05:00" level=info msg="failed to read
ipv6 net.ipv6.conf.<bridge>.accept_ra" bridge=docker0
syspath=/proc/sys/net/ipv6/conf/docker0/accept_ra
time="2022-12-10T12:17:03.518761943-05:00" level=info msg="Default bridge
(docker0) is assigned with an IP address 172.17.0.0/16. Daemon option

--bip

Post by Mansour Al Akeel
can be used to set a preferred IP address"
time="2022-12-10T12:17:03.518886881-05:00" level=info msg="failed to read
ipv6 net.ipv6.conf.<bridge>.accept_ra" bridge=docker0
syspath=/proc/sys/net/ipv6/conf/docker0/accept_ra
time="2022-12-10T12:17:03.534616741-05:00" level=info msg="Loading
containers: done."
time="2022-12-10T12:17:03.541080189-05:00" level=info msg="Docker daemon"
commit=a89b84221c graphdriver(s)=overlay2 version=20.10.17
time="2022-12-10T12:17:03.541122352-05:00" level=info msg="Daemon has
completed initialization"
time="2022-12-10T12:17:03.549888103-05:00" level=info msg="API listen on
/var/run/docker.sock"
time="2022-12-10T12:17:27.025622231-05:00" level=warning msg="Error

getting

Post by Mansour Al Akeel
v2 registry: Get \"https://registry-1.docker.io/v2/\": net/http: request
canceled while waiting for connection (Client.Timeout exceeded while
awaiting headers)"
time="2022-12-10T12:17:27.025667054-05:00" level=info msg="Attempting

Post by Mansour Al Akeel
net/http: request canceled while waiting for connection (Client.Timeout
exceeded while awaiting headers)"
time="2022-12-10T12:17:27.026851821-05:00" level=error msg="Handler for
POST /v1.41/images/create returned error: Get \"
https://registry-1.docker.io/v2/\": net/http: request canceled while
waiting for connection (Client.Timeout exceeded while awaiting headers)"
time="2022-12-10T12:15:42.036053086-05:00" level=info msg="loading plugin
\"io.containerd.internal.v1.tracing\"..." type=io.containerd.internal.v1
time="2022-12-10T12:15:42.036068412-05:00" level=error msg="failed to
initialize a tracing processor \"otlp\"" error="no OpenTelemetry
skip plugin"
time="2022-12-10T12:15:42.036100189-05:00" level=info msg="loading plugin
\"io.containerd.grpc.v1.cri\"..." type=io.containerd.grpc.v1
time="2022-12-10T12:15:42.036390695-05:00" level=info msg="Start cri

plugin

Post by Mansour Al Akeel
with config {PluginConfig:{ContainerdConfig:{Snapshotter:overlayfs
PodAnnotations:[] ContainerAnnotations:[] Root: Options:map[]
PodAnnotations:[] ContainerAnnotations:[] Root: Options:map[]
NetworkPluginMaxConfNum:0} Runtimes:map[runc:{Type:io.containerd.runc.v2
Options:map[BinaryName: CriuImagePath: CriuPath: CriuWorkPath: IoGid:0
NetworkPluginConfDir: NetworkPluginMaxConfNum:0}] NoPivot:false
DisableSnapshotAnnotations:true DiscardUnpackedLayers:false
IgnoreRdtNotEnabledErrors:false}
CniConfig:{NetworkPluginBinDir:/opt/cni/bin
NetworkPluginConfDir:/etc/cni/net.d NetworkPluginMaxConfNum:1
Mirrors:map[] Configs:map[] Auths:map[] Headers:map[]}
ImageDecryption:{KeyModel:node} DisableTCPService:true
StreamServerAddress:127.0.0.1 StreamServerPort:0 StreamIdleTimeout:4h0m0s
k8s.gcr.io/pause:3.6 StatsCollectPeriod:10 SystemdCgroup:false
EnableTLSStreaming:false X509KeyPairStreaming:{TLSCertFile: TLSKeyFile:}
MaxContainerLogLineSize:16384 DisableCgroup:false DisableApparmor:false
RestrictOOMScoreAdj:false MaxConcurrentDownloads:3 DisableProcMount:false
UnsetSeccompProfile: TolerateMissingHugetlbController:true
DisableHugetlbController:true DeviceOwnershipFromSecurityContext:false
IgnoreImageDefinedVolumes:false NetNSMountsUnderStateDir:false
EnableUnprivilegedPorts:false EnableUnprivilegedICMP:false}
ContainerdRootDir:/var/lib/containerd
ContainerdEndpoint:/run/containerd/containerd.sock
RootDir:/var/lib/containerd/io.containerd.grpc.v1.cri
StateDir:/run/containerd/io.containerd.grpc.v1.cri}"
time="2022-12-10T12:15:42.036530498-05:00" level=info msg="Connect
containerd service"
time="2022-12-10T12:15:42.036627681-05:00" level=info msg="Get image
filesystem path
\"/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs\""
time="2022-12-10T12:15:42.036896077-05:00" level=error msg="failed to

load

Post by Mansour Al Akeel
cni during init, please check CRI plugin status before setting up network
for pods" error="cni config load failed: no network config found in
/etc/cni/net.d: cni plugin not initialized: failed to load cni config"
time="2022-12-10T12:15:42.036986623-05:00" level=info msg="Start
subscribing containerd event"
time="2022-12-10T12:15:42.037028034-05:00" level=info msg="Start

recovering

Post by Mansour Al Akeel
state"
time="2022-12-10T12:15:42.037035874-05:00" level=info msg=serving...
address=/run/containerd/containerd.sock.ttrpc
time="2022-12-10T12:15:42.037068181-05:00" level=info msg=serving...
address=/run/containerd/containerd.sock
time="2022-12-10T12:15:42.037085346-05:00" level=info msg="containerd
successfully booted in 0.015889s"
time="2022-12-10T12:15:42.037072303-05:00" level=info msg="Start event
monitor"
time="2022-12-10T12:15:42.037107987-05:00" level=info msg="Start

snapshots

Post by Mansour Al Akeel
syncer"
time="2022-12-10T12:15:42.037119126-05:00" level=info msg="Start cni
network conf syncer for default"
time="2022-12-10T12:15:42.037127266-05:00" level=info msg="Start

streaming

Post by Mansour Al Akeel
server"
Looking at resolv.conf
localhost /home/mansour # cat /etc/resolv.conf
# Generated by dhcpcd from wlp3s0.dhcp
# /etc/resolv.conf.head can replace this line
domain Home
nameserver 192.168.1.1
# /etc/resolv.conf.tail can replace this line
So, I added the following to my /etc/dhcpcd.conf
static domain_name_servers=8.8.8.8 8.8.4.4 192.168.1.1
localhost /home/mansour # cat /etc/resolv.conf
# Generated by dhcpcd from wlp3s0.dhcp
# /etc/resolv.conf.head can replace this line
domain Home
nameserver 8.8.8.8
nameserver 8.8.4.4
nameserver 192.168.1.1
# /etc/resolv.conf.tail can replace this line
time="2022-12-10T12:28:16.519126445-05:00" level=warning msg="Error

getting

Post by Mansour Al Akeel
net/http: request canceled while waiting for connection (Client.Timeout
exceeded while awaiting headers)"
time="2022-12-10T12:28:16.520489790-05:00" level=error msg="Handler for
POST /v1.41/images/create returned error: Get \"
https://registry-1.docker.io/v2/\": net/http: request canceled while
waiting for connection (Client.Timeout exceeded while awaiting headers)"
I am not able to login with "docker login"
Where do I look further for additional troubleshooting hints? Google

didn't

Post by Mansour Al Akeel
help a lot.
Thank you.

Hello,
my first step would be checking if there is some inherent network
problem reaching registry-1.docker.io/v2, e.g. via
curl -v https://registry-1.docker.io/v2/
If that runs successfully, you could try if there is an MTU problem. I
remember that I had to set MTU to 1450 in some virtualized environment,
otherwise docker would not run correctly, i.e. I had in
/etc/conf.d/docer
DOCKER_OPTS="--mtu=1450"
Hope that gives you an idea what to look for.
If both does not help, there is wireshark, to watch your network, very
low-level, but might give some hints.
Cheers
Andreas

Andreas Fink

2022-12-11 06:00:01 UTC

Permalink

On Sat, 10 Dec 2022 15:51:17 -0500

Post by Mansour Al Akeel
Andreas,
Thank you very much. In fact I didn't go that deep yet, and not sure if I
should. I just found that the url is not accessible even from a browser.
Googling a bit, tells me there is no clear URL for docker-registry and
possibly this one is outdated. I will continue looking around to confirm
what the current default repo should be. If you have any suggestions,
please let me know.
localhost in ~
○ → curl -k -v https://registry-1.docker.io/v2/
* Trying 34.228.211.243:443...
* connect to 34.228.211.243 port 443 failed: Connection timed out
Couldn't connect to server
* Closing connection 0
curl: (28) Failed to connect to registry-1.docker.io port 443 after 129401
ms: Couldn't connect to server

You have a a strange DNS resolution. The IP address 34.227.211.243
seems wrong. Here is what I see when I look at the DNS records:
***@localhost ~$ dig registry-1.docker.io

; <<>> DiG 9.16.33 <<>> registry-1.docker.io
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 11419
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;registry-1.docker.io. IN A

;; ANSWER SECTION:
registry-1.docker.io. 13 IN A 34.205.13.154
registry-1.docker.io. 13 IN A 44.205.64.79
registry-1.docker.io. 13 IN A 3.216.34.172

;; Query time: 10 msec
;; SERVER: 79.143.183.251#53(79.143.183.251)
;; WHEN: Sun Dec 11 06:56:50 CET 2022
;; MSG SIZE rcvd: 97

I am not sure
Not sure where you get the wrong IP from, but it is a DNS issue.

Mansour Al Akeel

2022-12-11 16:20:01 UTC

Permalink

This is what I have on my machine. I can see the ANSWER SECTION in reversed
order

localhost ~ # dig registry-1.docker.io

; <<>> DiG 9.16.33 <<>> registry-1.docker.io
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 10324
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 4, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;registry-1.docker.io. IN A

;; ANSWER SECTION:
registry-1.docker.io. 51 IN A 3.216.34.172
registry-1.docker.io. 51 IN A 44.205.64.79
registry-1.docker.io. 51 IN A 34.205.13.154

;; AUTHORITY SECTION:
docker.io. 171984 IN NS ns-1168.awsdns-18.org.
docker.io. 171984 IN NS ns-1827.awsdns-36.co.uk.
docker.io. 171984 IN NS ns-421.awsdns-52.com.
docker.io. 171984 IN NS ns-513.awsdns-00.net.

;; Query time: 15 msec
;; SERVER: 192.168.1.1#53(192.168.1.1)
;; WHEN: Sun Dec 11 11:06:33 EST 2022
;; MSG SIZE rcvd: 237

Post by Andreas Fink
On Sat, 10 Dec 2022 15:51:17 -0500

Post by Mansour Al Akeel
Andreas,
Thank you very much. In fact I didn't go that deep yet, and not sure if I
should. I just found that the url is not accessible even from a browser.
Googling a bit, tells me there is no clear URL for docker-registry and
possibly this one is outdated. I will continue looking around to confirm
what the current default repo should be. If you have any suggestions,
please let me know.
localhost in ~
â â curl -k -v https://registry-1.docker.io/v2/
* Trying 34.228.211.243:443...
* connect to 34.228.211.243 port 443 failed: Connection timed out
Couldn't connect to server
* Closing connection 0
curl: (28) Failed to connect to registry-1.docker.io port 443 after

129401

Post by Mansour Al Akeel
ms: Couldn't connect to server

You have a a strange DNS resolution. The IP address 34.227.211.243
; <<>> DiG 9.16.33 <<>> registry-1.docker.io
;; global options: +cmd
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 11419
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1
; EDNS: version: 0, flags:; udp: 4096
;registry-1.docker.io. IN A
registry-1.docker.io. 13 IN A 34.205.13.154
registry-1.docker.io. 13 IN A 44.205.64.79
registry-1.docker.io. 13 IN A 3.216.34.172
;; Query time: 10 msec
;; SERVER: 79.143.183.251#53(79.143.183.251)
;; WHEN: Sun Dec 11 06:56:50 CET 2022
;; MSG SIZE rcvd: 97
I am not sure
Not sure where you get the wrong IP from, but it is a DNS issue.

David Rosenbaum

2022-12-16 19:00:01 UTC

Permalink

Thank u

Dave

Post by Mansour Al Akeel
Andreas,
Thank you very much. In fact I didn't go that deep yet, and not sure if I
should. I just found that the url is not accessible even from a browser.
Googling a bit, tells me there is no clear URL for docker-registry and
possibly this one is outdated. I will continue looking around to confirm
what the current default repo should be. If you have any suggestions,
please let me know.
localhost in ~
â â curl -k -v https://registry-1.docker.io/v2/
* Trying 34.228.211.243:443...
* connect to 34.228.211.243 port 443 failed: Connection timed out
Couldn't connect to server
* Closing connection 0
curl: (28) Failed to connect to registry-1.docker.io port 443 after
129401 ms: Couldn't connect to server

Post by Andreas Fink
On Sat, 10 Dec 2022 12:30:40 -0500

\"unix\"

Post by Mansour Al Akeel
not registered, fallback to default scheme" module=grpc
time="2022-12-10T12:17:03.473566413-05:00" level=info
{[{unix:///run/containerd/containerd.sock <nil> 0 <nil>}] <nil> <nil>}"
module=grpc
time="2022-12-10T12:17:03.473573787-05:00" level=info msg="ClientConn
switching balancer to \"pick_first\"" module=grpc
time="2022-12-10T12:17:03.474530993-05:00" level=info msg="parsed
\"unix\"" module=grpc
time="2022-12-10T12:17:03.474545549-05:00" level=info msg="scheme

\"unix\"

kernel

Post by Mansour Al Akeel
does not support cgroup blkio weight"
time="2022-12-10T12:17:03.478927115-05:00" level=warning msg="Your

kernel

Post by Mansour Al Akeel
does not support cgroup blkio weight_device"
time="2022-12-10T12:17:03.479037897-05:00" level=info msg="Loading
containers: start."
time="2022-12-10T12:17:03.495743563-05:00" level=info msg="failed to

read

Post by Mansour Al Akeel
ipv6 net.ipv6.conf.<bridge>.accept_ra" bridge=docker0
syspath=/proc/sys/net/ipv6/conf/docker0/accept_ra
time="2022-12-10T12:17:03.518761943-05:00" level=info msg="Default

bridge

Post by Mansour Al Akeel
(docker0) is assigned with an IP address 172.17.0.0/16. Daemon option

--bip

Post by Mansour Al Akeel
can be used to set a preferred IP address"
time="2022-12-10T12:17:03.518886881-05:00" level=info msg="failed to

read

Post by Mansour Al Akeel
ipv6 net.ipv6.conf.<bridge>.accept_ra" bridge=docker0
syspath=/proc/sys/net/ipv6/conf/docker0/accept_ra
time="2022-12-10T12:17:03.534616741-05:00" level=info msg="Loading
containers: done."
time="2022-12-10T12:17:03.541080189-05:00" level=info msg="Docker

daemon"

Post by Mansour Al Akeel
commit=a89b84221c graphdriver(s)=overlay2 version=20.10.17
time="2022-12-10T12:17:03.541122352-05:00" level=info msg="Daemon has
completed initialization"
time="2022-12-10T12:17:03.549888103-05:00" level=info msg="API listen on
/var/run/docker.sock"
time="2022-12-10T12:17:27.025622231-05:00" level=warning msg="Error

getting
request

Post by Mansour Al Akeel
canceled while waiting for connection (Client.Timeout exceeded while
awaiting headers)"
time="2022-12-10T12:17:27.025667054-05:00" level=info msg="Attempting

Post by Mansour Al Akeel
endpoint for pull after error: Get \"https://registry-1.docker.io/v2/\
net/http: request canceled while waiting for connection (Client.Timeout
exceeded while awaiting headers)"
time="2022-12-10T12:17:27.026851821-05:00" level=error msg="Handler for
POST /v1.41/images/create returned error: Get \"
https://registry-1.docker.io/v2/\": net/http: request canceled while
waiting for connection (Client.Timeout exceeded while awaiting headers)"
time="2022-12-10T12:15:42.036053086-05:00" level=info msg="loading

plugin

Post by Mansour Al Akeel
\"io.containerd.internal.v1.tracing\"..." type=io.containerd.internal.v1
time="2022-12-10T12:15:42.036068412-05:00" level=error msg="failed to
initialize a tracing processor \"otlp\"" error="no OpenTelemetry
skip plugin"
time="2022-12-10T12:15:42.036100189-05:00" level=info msg="loading

plugin

Post by Mansour Al Akeel
\"io.containerd.grpc.v1.cri\"..." type=io.containerd.grpc.v1
time="2022-12-10T12:15:42.036390695-05:00" level=info msg="Start cri

plugin

StreamIdleTimeout:4h0m0s

Post by Mansour Al Akeel
k8s.gcr.io/pause:3.6 StatsCollectPeriod:10 SystemdCgroup:false
EnableTLSStreaming:false X509KeyPairStreaming:{TLSCertFile: TLSKeyFile:}
MaxContainerLogLineSize:16384 DisableCgroup:false DisableApparmor:false
RestrictOOMScoreAdj:false MaxConcurrentDownloads:3

DisableProcMount:false

Post by Mansour Al Akeel
UnsetSeccompProfile: TolerateMissingHugetlbController:true
DisableHugetlbController:true DeviceOwnershipFromSecurityContext:false
IgnoreImageDefinedVolumes:false NetNSMountsUnderStateDir:false
EnableUnprivilegedPorts:false EnableUnprivilegedICMP:false}
ContainerdRootDir:/var/lib/containerd
ContainerdEndpoint:/run/containerd/containerd.sock
RootDir:/var/lib/containerd/io.containerd.grpc.v1.cri
StateDir:/run/containerd/io.containerd.grpc.v1.cri}"
time="2022-12-10T12:15:42.036530498-05:00" level=info msg="Connect
containerd service"
time="2022-12-10T12:15:42.036627681-05:00" level=info msg="Get image
filesystem path
\"/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs\""
time="2022-12-10T12:15:42.036896077-05:00" level=error msg="failed to

load

Post by Mansour Al Akeel
cni during init, please check CRI plugin status before setting up

network

Post by Mansour Al Akeel
for pods" error="cni config load failed: no network config found in
/etc/cni/net.d: cni plugin not initialized: failed to load cni config"
time="2022-12-10T12:15:42.036986623-05:00" level=info msg="Start
subscribing containerd event"
time="2022-12-10T12:15:42.037028034-05:00" level=info msg="Start

recovering

snapshots

streaming

getting
request

Post by Mansour Al Akeel
canceled while waiting for connection (Client.Timeout exceeded while
awaiting headers)"
time="2022-12-10T12:28:16.519173944-05:00" level=info msg="Attempting

Post by Mansour Al Akeel
endpoint for pull after error: Get \"https://registry-1.docker.io/v2/\
net/http: request canceled while waiting for connection (Client.Timeout
exceeded while awaiting headers)"
time="2022-12-10T12:28:16.520489790-05:00" level=error msg="Handler for
POST /v1.41/images/create returned error: Get \"
https://registry-1.docker.io/v2/\": net/http: request canceled while
waiting for connection (Client.Timeout exceeded while awaiting headers)"
I am not able to login with "docker login"
Where do I look further for additional troubleshooting hints? Google

didn't

Post by Mansour Al Akeel
help a lot.
Thank you.

Hello,
my first step would be checking if there is some inherent network
problem reaching registry-1.docker.io/v2, e.g. via
curl -v https://registry-1.docker.io/v2/
If that runs successfully, you could try if there is an MTU problem. I
remember that I had to set MTU to 1450 in some virtualized environment,
otherwise docker would not run correctly, i.e. I had in
/etc/conf.d/docer
DOCKER_OPTS="--mtu=1450"
Hope that gives you an idea what to look for.
If both does not help, there is wireshark, to watch your network, very
low-level, but might give some hints.
Cheers
Andreas