[BUG] Can't deploy from aarch64-darwin to x86_64-linux

[notthebee]

Field	Description
Plugin	N/A
Nixpkgs	24.05
Home Manager	24.05

• I have read the FAQ and my bug is not listed there.

Description

Trying to deploy a flake which includex nix-community/nixvim from an M1 Mac to an x86-64 NixOS machine using deploy-rs produces the following error:

error: a 'x86_64-linux' with features {} is required to build '/nix/store/dknd53ikkkjvk5jgw0hcwmby9ddpj9v1-init.lua.drv', but I am a 'aarch64-darwin' with features {apple-virt, benchmark, big-parallel, nixos-test}

#1637 seemed to fix a similar error in the file nixpkgs-nixvim-doc.drv, but the issue with init.lua.drv remains.

It seems to be related to the IFDs mentioned in this issue: #1154

Minimal, Reproducible Example (MRE)

1. Include the nix-community/nixvim flake in flake.nix inputs
2. Run nix flake update
3. Attempt to deploy your flake from an aarch64 Mac to a x86-64 NixOS machine using deploy-rs, for instance:

deploy -s --remote-build ".?submodules=1#spencer" 

[notthebee]

Additionally, same error happens when deploying with plain nixos-rebuild:

nixos-rebuild switch --fast -j auto --use-remote-sudo --impure --flake .\?submodules=1#emily --target-host emily --build-host emily
building the system configuration...
error:
       … while calling the 'head' builtin

         at /nix/store/l9ghpp1wk6hyng0g56vqzc923mdmwrsn-source/lib/attrsets.nix:1575:11:

         1574|         || pred here (elemAt values 1) (head values) then
         1575|           head values
             |           ^
         1576|         else

       … while evaluating the attribute 'value'

         at /nix/store/l9ghpp1wk6hyng0g56vqzc923mdmwrsn-source/lib/modules.nix:809:9:

          808|     in warnDeprecation opt //
          809|       { value = builtins.addErrorContext "while evaluating the option `${showOption loc}':" value;
             |         ^
          810|         inherit (res.defsFinal') highestPrio;

       (stack trace truncated; use '--show-trace' to show the full trace)

       error: a 'x86_64-linux' with features {} is required to build '/nix/store/dknd53ikkkjvk5jgw0hcwmby9ddpj9v1-init.lua.drv', but I am a 'aarch64-darwin' with features {apple-virt, benchmark, big-parallel, nixos-test}

[MattSturgeon]

Just to confirm: removing nixvim from your config resolves the issue?

I have a feeling this is related to how flakes have one "system" variable, while nixpkgs actually needs a distinct host and target system. See https://nix.dev/tutorials/cross-compilation.html and https://nixos.wiki/wiki/Cross_Compiling

Flakes usually just assume system is the same for both host & target.

[notthebee]

Yes, removing nixvim from the config resolves the issue:

Without nixvim:

deploy --dry-activate  -s --remote-build ".?submodules=1#emily"                                                                                                                                                                                                                                       ~~~
🚀 ℹ️ [deploy] [INFO] Building profile `system` for node `emily` on remote host
would activate the configuration...
would restart systemd
would reload the following units: dbus.service
would restart the following units: home-manager-notthebee.service
🚀 ℹ️ [deploy] [INFO] Completed dry-activate!

With nixvim:

deploy --dry-activate  -s --remote-build ".?submodules=1#emily"                                                                                                                                                        
🚀 ℹ️ [deploy] [INFO] Evaluating flake in .?submodules=1
error:
       … while evaluating attribute 'nodes'

         at «string»:4:25:

            3|                       (deploy // {
            4|                         nodes = {
             |                         ^
            5|                           inherit (deploy.nodes) "emily";

       … while evaluating attribute 'emily'

         at «string»:5:49:

            4|                         nodes = {
            5|                           inherit (deploy.nodes) "emily";
             |                                                 ^
            6|                         };

       (stack trace truncated; use '--show-trace' to show the full trace)

       error: a 'x86_64-linux' with features {} is required to build '/nix/store/dknd53ikkkjvk5jgw0hcwmby9ddpj9v1-init.lua.drv', but I am a 'aarch64-darwin' with features {apple-virt, benchmark, big-parallel, nixos-test}

[jameslounds]

The way I found around this was to specify builders in the deploying machine's /etc/nix/machines as specified here. You can find the builder's features by running nix show-config | grep "system-features". As a bonus, this lets you omit the --remote-build flag as nix now knows about a machine of the correct architecture on which it can build the derivations.

My /etc/nix/machines looks something like this:

$ cat /etc/nix/machines
ssh://root@my-arm-machine aarch64-linux /Users/james/.ssh/my-privkey 4 1 benchmark,big-parallel,gccarch-armv8-a,kvm,nixos-test
ssh://root@my-intel-machine x86_64-linux /Users/james/.ssh/my-privkey 4 1 benchmark,big-parallel,kvm,nixos-test

I am unsure which of the features I'm actually using, so I just stuck them all in there.

One thing that caught me out was that all the ssh config is from root's perspective. So you need to add the host to root's known_hosts.

I won't pretend to know enough about nix to understand why this happens, and I would love to find out why if anyone can enlighten me!

[notthebee]

I came up with a simple workaround. This shell script runs nix flake update, rsyncs the current folder to the machine's /etc/nixos, and runs nixos-rebuild switch --upgrade. It takes the machine's hostname as an argument

#!/usr/bin/env zsh

nix flake update
rsync -q -avx --delete ./ $1:/etc/nixos/
ssh -t $1 'cd /etc/nixos && sudo nixos-rebuild switch --impure --upgrade --flake .\?submodules=1#$1'

While not a solution to the failing check, this lets me deploy NixOS configurations to my target machines until the issue is fully resolved

[GaetanLepage]

Thank you @notthebee for sharing this workaround.
I am having trouble seeing if nixvim really is the problem here...
I have never tried doing cross-architecture deployment.
The only thing I know is that our entire test suite succeeds on all four main architectures.

[notthebee]

@GaetanLepage as I mentioned above, the flake can be deployed with no issues after I remove Nixvim from my configuration.

[MattSturgeon]

Does this commit help at all?

nix flake lock --override-input nixvim github:MattSturgeon/nixvim/d6e4c98fa767fd1eb6032ca1b0e8d2552590e014

EDIT: I think that line was a red-herring, the pkgs used there should be the pkgs from the host module system.

[notthebee]

Unfortunately it doesn't

error: attribute 'pkgs' missing

       at /nix/store/6p95y71c4baahvhf13pj7nyjdn0q9g1z-source/lib/modules.nix:508:28:

          507|         builtins.addErrorContext (context name)
          508|           (args.${name} or config._module.args.${name})
             |                            ^
          509|       ) (lib.functionArgs f);