Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Can't mount writable /nix/store overlay #210

Closed
onsails opened this issue Mar 5, 2024 · 4 comments
Closed

Can't mount writable /nix/store overlay #210

onsails opened this issue Mar 5, 2024 · 4 comments

Comments

@onsails
Copy link
Sponsor

onsails commented Mar 5, 2024 •
edited
Loading

I'm trying to mount a writeable overlay with exact options suggested by docs:

        microvm = {
            mem = 16 * 1024;
            vcpu = 16;

            hypervisor = "qemu";
            interfaces = [{
              type = "tap";
              id = "vm-1";
              mac = "02:00:00:01:01:01";
            }];

            writableStoreOverlay = "/nix/.rw-store";

            volumes = [{
              image = "nix-store-overlay.img";
              mountPoint = config.microvm.writableStoreOverlay;
              size = 2048;
            }];
          };
        };

It fails to mount overlay with the following log:

Mar 05 20:05:43 my-github-runner microvm@nix-runner-1[587697]:          Mounting /sysroot...
Mar 05 20:05:43 my-github-runner microvm@nix-runner-1[587697]:          Starting Rule-based Manager for Device Events and Files...
Mar 05 20:05:43 my-github-runner microvm@nix-runner-1[587697]: [  OK  ] Mounted /sysroot.
Mar 05 20:05:43 my-github-runner microvm@nix-runner-1[587697]: [  OK  ] Reached target Initrd Root File System.
Mar 05 20:05:43 my-github-runner microvm@nix-runner-1[587697]:          Starting Mountpoints Configured in the Real Root...
Mar 05 20:05:43 my-github-runner microvm@nix-runner-1[587697]: [  OK  ] Started Journal Service.
Mar 05 20:05:43 my-github-runner microvm@nix-runner-1[587697]:          Starting Create Volatile Files and Directories...
Mar 05 20:05:43 my-github-runner microvm@nix-runner-1[587697]: [  OK  ] Found device /dev/disk/by-label/nix-store.
Mar 05 20:05:43 my-github-runner microvm@nix-runner-1[587697]: [  OK  ] Found device /dev/vdb.
Mar 05 20:05:43 my-github-runner microvm@nix-runner-1[587697]: [  OK  ] Started Rule-based Manager for Device Events and Files.
Mar 05 20:05:43 my-github-runner microvm@nix-runner-1[587697]: [  OK  ] Finished Load Kernel Modules.
Mar 05 20:05:43 my-github-runner microvm@nix-runner-1[587697]: [  OK  ] Finished Coldplug All udev Devices.
Mar 05 20:05:43 my-github-runner microvm@nix-runner-1[587697]: [  OK  ] Finished Create Volatile Files and Directories.
Mar 05 20:05:43 my-github-runner microvm@nix-runner-1[587697]: [  OK  ] Finished Mountpoints Configured in the Real Root.
Mar 05 20:05:43 my-github-runner microvm@nix-runner-1[587697]:          Starting File System Check on /dev/vdb...
Mar 05 20:05:43 my-github-runner microvm@nix-runner-1[587697]:          Starting Apply Kernel Variables...
Mar 05 20:05:43 my-github-runner microvm@nix-runner-1[587697]:          Starting Virtual Console Setup...
Mar 05 20:05:43 my-github-runner microvm@nix-runner-1[587697]: [  OK  ] Finished Apply Kernel Variables.
Mar 05 20:05:43 my-github-runner microvm@nix-runner-1[587697]: [  OK  ] Finished File System Check on /dev/vdb.
Mar 05 20:05:43 my-github-runner microvm@nix-runner-1[587697]: [  OK  ] Finished Virtual Console Setup.
Mar 05 20:05:43 my-github-runner microvm@nix-runner-1[587697]: [  OK  ] Reached target System Initialization.
Mar 05 20:05:43 my-github-runner microvm@nix-runner-1[587697]: [  OK  ] Reached target Basic System.
Mar 05 20:05:44 my-github-runner microvm@nix-runner-1[587697]:          Mounting /sysroot/nix/.ro-store...
Mar 05 20:05:44 my-github-runner microvm@nix-runner-1[587697]:          Mounting /sysroot/nix/.rw-store...
Mar 05 20:05:44 my-github-runner microvm@nix-runner-1[587697]: [  OK  ] Mounted /sysroot/nix/.ro-store.
Mar 05 20:05:44 my-github-runner microvm@nix-runner-1[587697]: [  OK  ] Mounted /sysroot/nix/.rw-store.
Mar 05 20:05:44 my-github-runner microvm@nix-runner-1[587697]:          Starting rw-store.service...
Mar 05 20:05:44 my-github-runner microvm@nix-runner-1[587697]: [  OK  ] Finished rw-store.service.
Mar 05 20:05:44 my-github-runner microvm@nix-runner-1[587697]:          Mounting /sysroot/nix/store...
Mar 05 20:05:44 my-github-runner microvm@nix-runner-1[587697]: [FAILED] Failed to mount /sysroot/nix/store.
Mar 05 20:05:44 my-github-runner microvm@nix-runner-1[587697]: See 'systemctl status sysroot-nix-store.mount' for details.
Mar 05 20:05:44 my-github-runner microvm@nix-runner-1[587697]: [DEPEND] Dependency failed for Initrd File Systems.
Mar 05 20:05:44 my-github-runner microvm@nix-runner-1[587697]: [  OK  ] Stopped Dispatch Password Requests to Console Directory Watch.
Mar 05 20:05:44 my-github-runner microvm@nix-runner-1[587697]:          Starting Cleaning Up and Shutting Down Daemons...
Mar 05 20:05:44 my-github-runner microvm@nix-runner-1[587697]: [  OK  ] Stopped target Basic System.
Mar 05 20:05:44 my-github-runner microvm@nix-runner-1[587697]: [  OK  ] Stopped target System Initialization.
Mar 05 20:05:44 my-github-runner microvm@nix-runner-1[587697]: [  OK  ] Started Emergency Shell.
Mar 05 20:05:44 my-github-runner microvm@nix-runner-1[587697]: [  OK  ] Reached target Emergency Mode.
Mar 05 20:05:44 my-github-runner microvm@nix-runner-1[587697]: [  OK  ] Stopped target Emergency Mode.
Mar 05 20:05:44 my-github-runner microvm@nix-runner-1[587697]: [  OK  ] Stopped target Initrd Root Device.
Mar 05 20:05:44 my-github-runner microvm@nix-runner-1[587697]: [  OK  ] Stopped target Path Units.
Mar 05 20:05:44 my-github-runner microvm@nix-runner-1[587697]: [  OK  ] Stopped target Slice Units.
Mar 05 20:05:44 my-github-runner microvm@nix-runner-1[587697]: [  OK  ] Stopped target Socket Units.
Mar 05 20:05:44 my-github-runner microvm@nix-runner-1[587697]: [  OK  ] Stopped target Swaps.
Mar 05 20:05:44 my-github-runner microvm@nix-runner-1[587697]: [  OK  ] Stopped target Timer Units.
Mar 05 20:05:44 my-github-runner microvm@nix-runner-1[587697]: [  OK  ] Stopped Create List of Static Device Nodes.
Mar 05 20:05:44 my-github-runner microvm@nix-runner-1[587697]:          Starting rw-store.service...
Mar 05 20:05:44 my-github-runner microvm@nix-runner-1[587697]: [  OK  ] Stopped Apply Kernel Variables.
Mar 05 20:05:44 my-github-runner microvm@nix-runner-1[587697]: [  OK  ] Stopped Create Volatile Files and Directories.
Mar 05 20:05:44 my-github-runner microvm@nix-runner-1[587697]: [  OK  ] Stopped target Local File Systems.
Mar 05 20:05:44 my-github-runner microvm@nix-runner-1[587697]: [  OK  ] Stopped target Preparation for Local File Systems.
Mar 05 20:05:44 my-github-runner microvm@nix-runner-1[587697]: [  OK  ] Stopped Coldplug All udev Devices.
Mar 05 20:05:44 my-github-runner microvm@nix-runner-1[587697]:          Stopping Rule-based Manager for Device Events and Files...
Mar 05 20:05:44 my-github-runner microvm@nix-runner-1[587697]: [  OK  ] Stopped Virtual Console Setup.
Mar 05 20:05:44 my-github-runner microvm@nix-runner-1[587697]: [  OK  ] Stopped Emergency Shell.
Mar 05 20:05:44 my-github-runner microvm@nix-runner-1[587697]: [  OK  ] Finished rw-store.service.
Mar 05 20:05:44 my-github-runner microvm@nix-runner-1[587697]: [  OK  ] Finished Cleaning Up and Shutting Down Daemons.
Mar 05 20:05:44 my-github-runner microvm@nix-runner-1[587697]:          Mounting /sysroot/nix/store...
Mar 05 20:05:44 my-github-runner microvm@nix-runner-1[587697]: [FAILED] Failed to mount /sysroot/nix/store.
Mar 05 20:05:44 my-github-runner microvm@nix-runner-1[587697]: See 'systemctl status sysroot-nix-store.mount' for details.
Mar 05 20:05:44 my-github-runner microvm@nix-runner-1[587697]: [DEPEND] Dependency failed for Initrd File Systems.
Mar 05 20:05:44 my-github-runner microvm@nix-runner-1[587697]: [  OK  ] Started Emergency Shell.
Mar 05 20:05:44 my-github-runner microvm@nix-runner-1[587697]: [  OK  ] Reached target Emergency Mode.
Mar 05 20:05:44 my-github-runner microvm@nix-runner-1[587697]: [  OK  ] Stopped Rule-based Manager for Device Events and Files.
Mar 05 20:05:44 my-github-runner microvm@nix-runner-1[587697]: [  OK  ] Closed udev Control Socket.
Mar 05 20:05:44 my-github-runner microvm@nix-runner-1[587697]: [  OK  ] Closed udev Kernel Socket.
Mar 05 20:05:44 my-github-runner microvm@nix-runner-1[587697]:          Starting Cleanup udev Database...
Mar 05 20:05:44 my-github-runner microvm@nix-runner-1[587697]: [  OK  ] Stopped Create Static Device Nodes in /dev.
Mar 05 20:05:44 my-github-runner microvm@nix-runner-1[587697]: [  OK  ] Finished Cleanup udev Database.
Mar 05 20:05:44 my-github-runner microvm@nix-runner-1[587697]: [  OK  ] Reached target Switch Root.
Mar 05 20:05:44 my-github-runner microvm@nix-runner-1[587697]:          Starting Switch Root...
Mar 05 20:05:44 my-github-runner microvm@nix-runner-1[587697]: [FAILED] Failed to start Switch Root.
Mar 05 20:05:44 my-github-runner microvm@nix-runner-1[587697]: See 'systemctl status initrd-switch-root.service' for details.
Mar 05 20:05:44 my-github-runner microvm@nix-runner-1[587697]: You are in emergency mode. After logging in, type "journalctl -xb" to view
Mar 05 20:05:44 my-github-runner microvm@nix-runner-1[587697]: system logs, "systemctl reboot" to reboot, or "exit"
Mar 05 20:05:44 my-github-runner microvm@nix-runner-1[587697]: to continue bootup.
Mar 05 20:05:44 my-github-runner microvm@nix-runner-1[587697]:
Mar 05 20:05:44 my-github-runner microvm@nix-runner-1[587697]: Cannot open access to console, the root account is locked.
Mar 05 20:05:44 my-github-runner microvm@nix-runner-1[587697]: See sulogin(8) man page for more details.
Mar 05 20:05:44 my-github-runner microvm@nix-runner-1[587697]:
Mar 05 20:05:44 my-github-runner microvm@nix-runner-1[587697]: Press Enter to continue.

Similar failure happens on cloud-hypervisor and crosvm, I can share logs if it matters.
VM is defined declaratively via nixosModule.

Not sure if it's possible to connect to VM console to look at exact systemd logs, couldn't find a way to do so considering microvms are headless.
@onsails
Copy link
Sponsor Author

onsails commented Mar 5, 2024 •
edited
Loading

Managed to get some details with boot.initrd.systemd.enable = false:

Mar 05 21:07:48 my-github-runner microvm@nix-runner-1[678035]: <<< NixOS Stage 1 >>>
Mar 05 21:07:48 my-github-runner microvm@nix-runner-1[678035]:
Mar 05 21:07:48 my-github-runner microvm@nix-runner-1[678035]: loading module virtio_mmio...
Mar 05 21:07:48 my-github-runner microvm@nix-runner-1[678035]: loading module virtio_pci...
Mar 05 21:07:48 my-github-runner microvm@nix-runner-1[678035]: loading module virtio_blk...
Mar 05 21:07:48 my-github-runner microvm@nix-runner-1[678035]: loading module 9pnet_virtio...
Mar 05 21:07:49 my-github-runner microvm@nix-runner-1[678035]: loading module 9p...
Mar 05 21:07:49 my-github-runner microvm@nix-runner-1[678035]: loading module virtiofs...
Mar 05 21:07:49 my-github-runner microvm@nix-runner-1[678035]: loading module overlay...
Mar 05 21:07:49 my-github-runner microvm@nix-runner-1[678035]: loading module dm_mod...
Mar 05 21:07:49 my-github-runner microvm@nix-runner-1[678035]: running udev...
Mar 05 21:07:49 my-github-runner microvm@nix-runner-1[678035]: Starting systemd-udevd version 254.6
Mar 05 21:07:50 my-github-runner microvm@nix-runner-1[678035]: kbd_mode: KDSKBMODE: Inappropriate ioctl for device
Mar 05 21:07:50 my-github-runner microvm@nix-runner-1[678035]: [36B blob data]
Mar 05 21:07:50 my-github-runner microvm@nix-runner-1[678035]: mounting rootfs on /...
Mar 05 21:07:50 my-github-runner microvm@nix-runner-1[678035]: mounting /dev/disk/by-label/nix-store on /nix/.ro-store...
Mar 05 21:07:50 my-github-runner microvm@nix-runner-1[678035]: [    1.630476] /dev/disk/by-label/nix-store: Can't open blockdev
Mar 05 21:07:50 my-github-runner microvm@nix-runner-1[678035]: checking /dev/vdb...
Mar 05 21:07:50 my-github-runner microvm@nix-runner-1[678035]: fsck (busybox 1.36.1)
Mar 05 21:07:50 my-github-runner microvm@nix-runner-1[678035]: [fsck.ext4 (1) -- /mnt-root/nix/.rw-store] fsck.ext4 -a /dev/vdb
Mar 05 21:07:50 my-github-runner microvm@nix-runner-1[678035]: /dev/vdb: clean, 131072/131072 files, 216453/524288 blocks
Mar 05 21:07:50 my-github-runner microvm@nix-runner-1[678035]: mounting /dev/vdb on /nix/.rw-store...
Mar 05 21:07:50 my-github-runner microvm@nix-runner-1[678035]: mounting overlay on /nix/store...
Mar 05 21:07:50 my-github-runner microvm@nix-runner-1[678035]: mount: mounting overlay on /mnt-root/nix/store failed: No space left on device
Mar 05 21:07:50 my-github-runner microvm@nix-runner-1[678035]:
Mar 05 21:07:50 my-github-runner microvm@nix-runner-1[678035]: An error occurred in stage 1 of the boot process, which must mount the
Mar 05 21:07:50 my-github-runner microvm@nix-runner-1[678035]: root filesystem on `/mnt-root' and then start stage 2.  Press one
Mar 05 21:07:50 my-github-runner microvm@nix-runner-1[678035]: of the following keys:
Mar 05 21:07:50 my-github-runner microvm@nix-runner-1[678035]:
Mar 05 21:07:50 my-github-runner microvm@nix-runner-1[678035]:   r) to reboot immediately
Mar 05 21:07:50 my-github-runner microvm@nix-runner-1[678035]:   *) to ignore the error and continue

I'm sure it's enough space on host root volume (it's a single ext4 volume), it has something to do with space on VM itself.

@onsails
Copy link
Sponsor Author

onsails commented Mar 5, 2024 •
edited
Loading

It started working after I changed image = "nix-store-overlay.img" to image = "nix-store-overlay1.img".
Apparently when I launched my first microvm I used the same name with less size and it's now being reused. Shouldn't we add an option to resize image if it's less than size required by a configuration?

@onsails
Copy link
Sponsor Author

onsails commented Mar 5, 2024 •
edited
Loading

Hotfixed with:

systemd.services = {
  "rm-image-${name}" = {
      serviceConfig = {
        Type = "oneshot";
        ExecStart = pkgs.writeScript "rm-image" ''
          #!${pkgs.runtimeShell}
          rm -f /var/lib/microvms/${name}/nix-store-overlay.img
        '';
      };
    };

    "microvm@${name}" = {
      after = [ "rm-image.service" ];
      requires = [ "rm-image.service" ];
    };
};

@astro
Copy link
Owner

astro commented Mar 6, 2024

Duplicate of #134

@astro astro marked this as a duplicate of #134 Mar 6, 2024
@astro astro closed this as completed Mar 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@astro @onsails