CoreDNS // External Queries from Pods

[fzebib]

Hi all,

recently installed a k3s stack using the playbook, and it seemed to be working fine - but any external routing breaks - simple queries like curling a page, or pinging a page breaks. I noticed when I remove the ndots option, or the search line in /etc/resolv.conf within a pod, the queries + pinging of websites seem to work.

Any thoughts? These are Ubuntu 20.04.6 LTS VM's running on proxmox.

Here is the the resolv.conf:

search default.svc.cluster.local svc.cluster.local cluster.local center
nameserver 10.43.0.10
options ndots:5

Attempting in a broken state.

/ # ping google.com
PING google.com (3.64.163.50): 56 data bytes
--- google.com ping statistics ---
2 packets transmitted, 0 packets received, 100% packet loss

I edit the resolv.conf to not use ndots, and the result then becomes:

PING google.com (142.250.72.110): 56 data bytes
64 bytes from 142.250.72.110: seq=0 ttl=116 time=6.909 ms
64 bytes from 142.250.72.110: seq=1 ttl=116 time=5.480 ms
64 bytes from 142.250.72.110: seq=2 ttl=116 time=3.719 ms
--- google.com ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 3.719/5.369/6.909 ms

I'll continue to look, but any help would be appreciated, thanks!

[timothystewart6]

I'm assuming your request are appending the search domain (although you didn't post that part) If you have a search domain set on your dhcp scope, remove it.

[fzebib]

The proxmox host had the DNS domain set. Without editing the cloudinit images, it assumes the DNS domain/DNS servers of the host. That was a pretty deep rabbithole. It's always DNS.

[timothystewart6]

Nice work figuring it out!

[fzebib]

How would I double check the appending of the search domains? I should have been more clear, that's what the ndots option is doing. Re: DHCP Scope, I haven't really touched any of it - my router handles the DHCP element - for some internal DNS, my 1st route within the router is a pihole instance that has a couple of internal A records, with secondary as 1.1.1. in-case the pi-hole instance goes down it doesn't break DNS.

Is search domain being set on the DHCP scope a default setting usually? Thanks a ton for the response - I'm digging into it now, as well. Have a great Sunday!

[fzebib]

Yep. I added some logging to the CoreDNS Yaml.

[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[INFO] 10.42.2.8:49009 - 5570 "A IN github.com.flux-system.svc.cluster.local. udp 69 false 1232" NXDOMAIN qr,aa,rd 151 0.000985451s
[INFO] 10.42.2.8:45902 - 56392 "AAAA IN github.com.flux-system.svc.cluster.local. udp 69 false 1232" NXDOMAIN qr,aa,rd 151 0.001136616s
[INFO] 10.42.2.8:49690 - 36778 "AAAA IN github.com.svc.cluster.local. udp 57 false 1232" NXDOMAIN qr,aa,rd 139 0.000128744s
[INFO] 10.42.2.8:47978 - 18543 "A IN github.com.svc.cluster.local. udp 57 false 1232" NXDOMAIN qr,aa,rd 139 0.000190427s
[INFO] 10.42.2.8:56596 - 9933 "AAAA IN github.com.cluster.local. udp 53 false 1232" NXDOMAIN qr,aa,rd 135 0.000144992s
[INFO] 10.42.2.8:52585 - 61703 "A IN github.com.cluster.local. udp 53 false 1232" NXDOMAIN qr,aa,rd 135 0.000226283s
[INFO] 10.42.2.8:53834 - 14516 "AAAA IN github.com.center. udp 46 false 1232" NOERROR qr,rd,ra 112 0.007824999s
[INFO] 10.42.2.8:38417 - 61528 "A IN github.com.center. udp 46 false 1232" NOERROR qr,rd,ra 68 0.009225153s

Recognized this because Flux won't install properly.

[exodious]

Can anyone explain why having the extra search domain is causing these problems?