Initramfs: Dropping to a Shell

As usual, I was heading off to a meeting, and my laptop decided to avoid booting. When it’s Linux, then it’s always at the exact moment you need to do something. I’ve always found the XKCD on Success to best detail the process.

Obviously my favorite words to see on the screen dropping to a shell. And usually it’s caused by your own fault because you wanted to complete no less than one of these:

• bitcoin mining before scripts existed
• add a second monitor to your setup
• remove X.org
• revert nouveau graphics driver (because its freezing your screen)
• Something else graphics card related.

However this time was different.

To be fair, it was booting, but I was being faced with this:

Gave up waiting for root device. Common problems:
 - Boot args (cat /proc/cmdline)
   - Check rootdelay= (did the system wait long enough?)
   - Check root= (did the system wait for the right device?)
 - Missing modules (cat /proc/modules; ls /dev)
ALERT! /dev/disk/by-uuid/xxxxxxxx-xxxx-xxxx-xxxx does not exist. Dropping to a shell!

BusyBox v1.21.1 (Ubuntu 1:1.21.0-1ubuntu1) built-in shell (ash)
Enter 'help' for a lost of built-in commands.
(initramfs)

As usual I was then met with realization of what I would be doing Thursday night, and probably for then next week as well. What is the world was going on. And par for the course, of course it was working perfectly the day before, and I hadn’t touched anything.

Lucky for me, I knew what the problem was, sort of. When I had first got my XPS 13. I decided to try my hand at using UEFI instead of BIOS, and was originally met with this same screen. In this cause the RAID controller was set to use RAID mode instead of AHCI, which was its own adventure to initially resolve. So the drive should have been loaded correctly but it wasn’t.

There is no shortage of posts on stack overflow mentioning this problem, but most of them have nothing to do with my problem (obviously). In this case the problem, I knew was crypt related. Because it’s a laptop, I opted for full drive encryption, and it was that encrypted drive that wasn’t mounting.

At 4 hours, I was getting the hang of the grub configuration and initramfs commands, I hadn’t used either of these specifically in a long time. There isn’t much you can do in either one, and neither of them of helpful in any way, because what you really need to do is fix both of them, which of course you can’t from broken mount screen. But that didn’t stop me from pouring over the endless Grub Modules to see if enabling one of them was the missing piece. Spoiler Alert: They weren’t.

At 8 hours, I thought I had found the holy grail, a post about this exact problem. And it was equally unhelpful. But it was indeed the holy grail, I just didn’t know it yet. Part of that was poor instructions (which for everyone following along I’ll include at the end.)

On Friday, I was told that I should give up and just install a new OS. But I wasn’t convinced it wouldn’t have the same problem.

On Saturday, I eventually found my way to Boot-Repair which put me on the path of perhaps Grub install itself was the problem. For the amount Googling I did for “Reinstall Grub”, Boot-Repair promised to be the solution as well. While it does quite a lot to try to help out, it couldn’t fix the problem I was having. However before the day was over, I found something else to try and this changed everything.

I had inadvertently stumbled across a somewhat related stack exchange post, while the post itself didn’t point to the answer, it mentioned something interesting.

The package cryptsetup has to be installed because it has startup scripts which provide support for automounting encrypted volumes at boot.

And so…

dpkg --get-selections | grep cryptsetup
> cryptsetup                              deinstall

Yay?!

So that was first up on my radar, and it significantly changed the result. Now instead of going directly to the initramfs shell, I got to stare at this screen for 30 seconds:

Different error = progress, as every great software engineer knows.

From there it occurred to me that update-initramfs needed an update before everything worked, and that was all it took. Sort of.

To fix anything on here, I needed to use another OS, as the current one was locked away in an encrypted setup. So I decided to mount my live usb linux installation, however it turned out that, thiswasn’t a good idea. initramfs needed to know the value of my /etc/crypttab mount point name, and the live usb was dynamically generating one for the live usb installation, not one that would work with my already existing /boot setup. Sigh, that was a few hours down the drain. Luckily I was clued into this issue:

cryptsetup: WARNING: The initramfs image may not contain cryptsetup binaries nor crypto modules. If that's on purpose, you may want to uninstall the 'cryptsetup-initramfs' package in order to disable the cryptsetup initramfs.

So I knew something was going wrong…

There are endless articles talking about how to set up an encrypted partition, but it is worth mentioning what the setup was for me:

# cat /etc/crypttab (cryptsetup will mount this)
nvme0n1p7_crypt UUID=727fa348-8804-4773-ae3d-f3e176d12dac   none luks

# cat /etc/fstab
/dev/mapper/nvme0n1p7_crypt       /               ext4

The initial warning message was a failure to find the nvme0n1p7_crypt because cryptsetup was never run. Installing cryptsetup fixed the first part, then all I needed to do was, chroot into the fs and update.

# decrypt and mount the drive
cryptsetup open /dev/nvme0n1p7 nvme0n1p7_cryptmount /dev/mapper/nvme0n1p7_crypt /mnt
mount /dev/nvme0n1p5 /mnt/boot
mount /dev/nvme0n1p1 /mnt/boot/efi
# necessary for chroot
mount --bind /proc /mnt/proc
mount --bind /dev /mnt/dev
mount --bind /sys /mnt/sys
chroot /mnt
# actually run the fix
# (and this time it will use the correct location)
apt install cryptsetup
update-initramfs -u -k all # update all images
# and for good measure
grub-install 
exit
reboot