Getting logs from worker node in multi node cluster / multi node set up

[dMARLAN]

I'm trying to set up a multi node cluster, so far just one server, one agent.

Air-gap in quotations because for development I'm still connected to the internet fully, but I am installing k3s the airgap .tar.gz, and am importing my images from .tars.

I launched the k3s server:

    INSTALL_K3S_BIN_DIR=/usr/bin \
    INSTALL_K3S_SKIP_DOWNLOAD=true \
    INSTALL_K3S_EXEC="--write-kubeconfig-mode 644 --kubelet-arg=eviction-hard=nodefs.available<3%" \
    INSTALL_K3S_SKIP_SELINUX_RPM=true \
    INSTALL_K3S_SELINUX_WARN=true \
    ./k3s_install.sh

Imported my airgap images on the server
Applied my helm chart (don't think this is relevant, but for completion) on the server

Moved my CA certs to the worker node / updated CA.

Then launched my k3s agent:

sudo k3s agent -t foo::server:bar -s https://<ip>:6443

and if I try to get logs I get:

root@dvchad/home# k3s kubectl logs dewey-taskworker-deployment-5dfbdbd9d7-76fgn -n dewey-system
Error from server: Get "https://192.168.0.164:10250/containerLogs/dewey-system/dewey-taskworker-deployment-5dfbdbd9d7-76fgn/dewey-taskworker-deployment": proxy error from 127.0.0.1:6443 while dialing 192.168.0.164:10250, code 503: 503 Service Unavailable

Any help is appreciated! Thank you

[brandond]

What do you see in the k3s service logs on the server node?

[dMARLAN]

I uninstalled everything last night for a fresh start, currently stuck on Getting list of apiserver endpoints from server

dewey-ci@dewey-ci:~/Development$ sudo k3s agent -t foo::server:bar -s https://100.94.157.87:6443
INFO[0000] Starting k3s agent v1.30.0+k3s1 (14549535)   
INFO[0000] Adding server to load balancer k3s-agent-load-balancer: 100.94.157.87:6443 
INFO[0000] Adding server to load balancer k3s-agent-load-balancer: 10.0.2.15:6443 
INFO[0000] Removing server from load balancer k3s-agent-load-balancer: 100.94.157.87:6443 
INFO[0000] Running load balancer k3s-agent-load-balancer 127.0.0.1:6444 -> [10.0.2.15:6443] [default: 100.94.157.87:6443] 
ERRO[0020] failed to get CA certs: Get "https://127.0.0.1:6444/cacerts": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 
ERRO[0042] failed to get CA certs: Get "https://127.0.0.1:6444/cacerts": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 
ERRO[0064] failed to get CA certs: Get "https://127.0.0.1:6444/cacerts": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 
ERRO[0086] failed to get CA certs: Get "https://127.0.0.1:6444/cacerts": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 
ERRO[0108] failed to get CA certs: Get "https://127.0.0.1:6444/cacerts": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 
ERRO[0130] failed to get CA certs: Get "https://127.0.0.1:6444/cacerts": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 
ERRO[0152] failed to get CA certs: Get "https://127.0.0.1:6444/cacerts": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 
ERRO[0174] CA cert validation failed: Get "https://127.0.0.1:6444/cacerts": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 
WARN[0176] Cluster CA certificate is trusted by the host CA bundle. Token CA hash will not be validated. 
INFO[0176] Module overlay was already loaded            
INFO[0176] Module nf_conntrack was already loaded       
INFO[0176] Module br_netfilter was already loaded       
INFO[0176] Module iptable_nat was already loaded        
INFO[0176] Module iptable_filter was already loaded     
INFO[0176] Found nvidia container runtime at /usr/bin/nvidia-container-runtime 
INFO[0176] Logging containerd to /var/lib/rancher/k3s/agent/containerd/containerd.log 
INFO[0176] Running containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd 
INFO[0176] containerd is now running                    
INFO[0176] Importing images from /var/lib/rancher/k3s/agent/images/k3s-airgap-images-amd64.tar 
INFO[0182] Imported docker.io/rancher/klipper-helm:v0.8.3-build20240228 
INFO[0182] Imported docker.io/rancher/klipper-lb:v0.4.7 
INFO[0182] Imported docker.io/rancher/local-path-provisioner:v0.0.26 
INFO[0182] Imported docker.io/rancher/mirrored-coredns-coredns:1.10.1 
INFO[0182] Imported docker.io/rancher/mirrored-library-busybox:1.36.1 
INFO[0182] Imported docker.io/rancher/mirrored-library-traefik:2.10.7 
INFO[0182] Imported docker.io/rancher/mirrored-metrics-server:v0.7.0 
INFO[0182] Imported docker.io/rancher/mirrored-pause:3.6 
INFO[0182] Imported images from /var/lib/rancher/k3s/agent/images/k3s-airgap-images-amd64.tar in 6.077697952s 
INFO[0182] Getting list of apiserver endpoints from server
(stuck)

Going to try to figure this out first and I'll reply again once I'm back to my original state. My company is using Tailscale so I'm thinking maybe this is the source of some of my problems as I am assuming k3s isn't using signing the certs with the right IP?

curl -k foo:6443/cacerts returns just fine, but without -k doesn't: curl: (60) SSL: no alternative certificate subject name matches target host name '100.94.157.87' ...

Edit: Might be working after adding --advertise-address, testing now

[brandond]

Yeah, I think that's it, given the messages in the log about changing the server address from one IP to another - I'm assuming the server is reachable at the 10.94 address but not the 10.0 address that it is advertising by default?

[dMARLAN]

Yeah, that was the problem regarding logs.

I am now having an issue with the pods themselves being able to reach each other though, or reaching external (running apt-get update inside a pod hangs) but that's a different thing than this thread, so going to close it, thanks for the help!