I spent a month working on nixos in a vm thought I was ready to put it on a ssd and of course right before I was making the last changes such as a few important sops changes and script fixes. DNS stopped resolving, so when I went through my whole install, which was git cloned it was all out of date and had no access to all my changes. It took a while to realise this and then I had to do all them last important changes again but in tty and was still trying to figure out why it didn't backup.
The annoying thing is that ping github.com would work fine then a git pull succeeds then the git push would fail two secs later with Could not resolve hostname github.com: Name or service not known. Nix builds would also fetch half their dependencies then sometimes they would die on a fixed-output derivation with curl: (6) Could not resolve host. Then you could go and retry the same command and then it would work.
I thought the network was just being flakey or something my brother was having issues before with internet as well.
for i in (seq 1 10); getent hosts github.com; end
Instead of just using one lookup I used a loop for 10 and only got 1.
Why was this happening /etc/resolv.conf had this,
search home
nameserver 192.168.20.1
nameserver 2403:5800:100:1::142
nameserver 2403:5800:1:5::242
options edns0
Basically router and two resolvers, what would happen is that glibc tries nameservers in order with short and limited retries where every lookup hit the router first, and glibc would give up rather than reliably falling through to the others. Adding good nameservers to the end did nothing because it would because they would never get reached.
I then tested using this,
for i in {1..20}; do dig +short @192.168.20.1 github.com; done
the first result was correct but everyone after that produced,
;; Warning: Message parser reports malformed message packet.
10 8 15UPR+w3nxU=
Whilst the same loop with 1.1.1.1 were all correct. So in the end it was not rate limiting or timeouts the router was responding but with malformed packets full of base64 garbage.
Solution
networking.networkmanager.dns = "none";
networking.resolvconf.enable = false;
networking.nameservers = [ "1.1.1.1" "1.0.0.1" ];
environment.etc."resolv.conf".text = ''
nameserver 1.1.1.1
nameserver 1.0.0.1
options edns0
'';
Obviously here you will want to probably add back you router to the bottom of the list.
Why did it not warn me on the VM? My script ran push inside a if ! git diff --cached --quiet block so once a commit did existed locally and the push had failed newer runs had nothing new to commit and skipped the push. So the router caused the first failed push but bad scripting stopped it from fixing itself.
