Alright I’m at the end of the rope with my Linux knowledge. So I need your help.
I’m on EndeavourOS and just recently upgraded to Linux 7.1.5, all upgrades thus far have been without issue. I am dual-booting but have Linux on its own SSD, and grub handles booting into Linux or Windows.
Now, after booting into Windows and back into Linux, Linux won’t start.
The error:
A start job is running for /dev/disks/by-uuid/<CORRECT UUID>
Let me tell you what I’ve already done:
- Regenerate mkinitrd with dracut
- Regenerate grub.cfg and verified via diff it is identical to the current one
- On a live usb verified that the UUIDs in the grub and fstab files are correct using the KDE partition manager (they are)
- Verify on a live usb the SSD still works (data and everything is still there)
- Changed the grub boot option on boot to check that the UUID is not hardcoded somewhere and can be changed (this is true, inputting a garbage UUID changes the boot error to A start job is running for /dev/disks/by-uuid/garbage-uuid
One last idea I had was to drop myself into the dracut emergency shell and actually check what’s actually under /dev/disks/by-uuid and lo and behold: it’s actually missing the drive.
Here’s whats nuts tho: grub loads the drive by UUID as well, loads the initrd, and all that works. When I regenerated the initrd with dracut I could also tell that grub was now loading the newer initrd.
WHAT THE FUCK???
Help?
At this point I REALLY don’t know wtf to do. Why doesnt the initramfs detect the drive? The only useful info I can add here is that all drives listed under /dev/disks/by-uuid had that shorter UUID with no dashes like ABCD1234, whereas the UUID for the correct drive/partition is the longer type UUID like abc1234-1234-0000-defg
Afaik, those by-* paths are symlinks created by udev during boot (before switching to the real root fs) The kernel creates the actual device files directly in /dev. My guess is that Windows left an NTFS volume in an unmountable state or something and udev got stuck processing this disk before processing rules for your boot disk. This probably requires you to have a fstab entry for mounting your windows disk, but I’m not sure.
When you boot into a live distro again (I know you’re done, but maybe someone else has this problem) check if the disk has a block device file directly in /dev. Maybe check the other disk, see if you can mount it or remove references to it in fstab or unplug the device entirely.
Yeah from what I found that whole “udev create the drives” thing holds true, but I gotta say, that’s above my paygrade :D
I couldn’t find out how I could tell udev to mount the drives already. I only found one thing where someone said his problem was fixed by telling dracut not to include udev, but dracut immediately was like “im not gonna build u a initrd without udev cause all those other modules rely on it”… fair enough.
This sounds like a rather weird issue. I agree with your take that grub is not the problem. It seems that there is some issue with the kernel, or the initramfs systemd… Downgrading the kernel didn’t resolve it, so maybe it is a corrupted disk after all? Some suggestions:
- Run
fsckon the partition from a live USB if you haven’t already - Enable more verbose logs from the kernel and the initramfs systemd. The LLM suggests removing
quiet splashand addingrd.debug systemd.log_level=debug systemd.log_target=console ignore_loglevel(best check the actual parameters yourself). - If possible for you: Copy the install to another drive, change the UUIDs, and try again with that.
Ehh I’ll just back up my stuff and pack my bags for now. A reinstall is just gonna be faster at this point.
- Run
Smarter people than me can answer, but I had two similar issues before. 1) I added nofail to fstab because in my situation it wasn’t finding a certain drive and getting stuck, and 2) windows fastboot or hybernate had somehow made drive locked or appear dirty to linux
Its not the fstab, I mean what am I gonna do, not mount the root drive? idk if its windows fastboot but this setup has been working for a couple months and survived windows and linux updates so this seems unlikely
Are your windows drives mounted in linux? If so comment them out in fstab or nofail them as a minimum, although it wouldn’t normally manifest as a root drive error (unless you’ve set up a strange mount point for the windows drives outside /mnt on root or even in home folder? or do you have them listed above your root drive in fstab so they are trying to mount first and that fails?).
Also you said elsewhere your linux drives can be “seen” in windows. Just to be clear, you’ve not mounted the drives in Windows right using a tool like Ext2Fsd? Because if so, go into windows and unmount the drive.
And have you disabled fastboot? Not just rebooted but disabled fastboot completely?
Yes, I mount my NTFS drives in linux, and they’re all nofail already, it’s literally the root drive it can’t find which makes no sense.
No, I don’t mount my linux drive on windows, I am not insane :D
Just checked, fast startup is disabled
No I realize that, for me it was stuck on a windows partition , which was irrelevant to Linux running. It sounds like it found some corruption and wants to do a full disk check. Or maybe their is a hardware fault arising.
Others may have better answers for you.
Just for testing purposes, have you removed every device except the root device and sys/dev/tmp from fstab? I know Linux can get a bit upset when a NTFS filesysyem is marked as needing chkdsk’d and might be blocking other disks from mounting.
- all drives besides root are marked as nofail in my fstab, 2) I’ve already confirmed this issue happens way before the fstab file is ever read. The kernel can’t even find the drive that the fstab file is on.
Update: I’m giving up. I tried every solution I could find. Tried the LTS kernel, fucking around with dracut options. I don’t know what the issue is or how to fix it.
I’ll be backing up my shit now and will move back to windows until I can find the brain power necessary for a reinstall.
Don’t know yet if I’ll stick to EndeavourOS or try nix or something, EOS worked out of the box with my NVIDIA card.
I want to thank everyone for your support. I would not have guessed for this issue to be so deep that it couldn’t be fixed, it is what it is I guess.
If you do switch back to Linux I’ll note that Endeavour and other Arch forks have moved away from older NVIDIA cards because NVIDIA stopped supporting open source drivers. There’s community-maintained solutions out there though. I forget exactly how old, but my 1080 got the boot, and then I had to gety Grover’s from AUR for a bit, right before that security breach. That contributed to my decision to upgrade to a new AMD card.
Just install a distro that doesn’t fail like that by overwriting system files on a whim. You’re saying that you’re going from an arch based system (which are prone to get weird errors every few weeks), to NixOS, which has at the very least a steep learning curve. My assessment is that you are asking for trouble yourself. I know that Debian-stable sounds boring, but it never breaks like that. If you’re truly burnt out as you say, you would go with the safe, unglamorous solution. I used to tinker with Linux all the time (I started on slackware in 1998), now I just run Mint on laptops, and Debian on my PCs. I just don’t want bad surprises anymore, I’m at an age that the OS needs to work as expected.
Thing is every time I read about Nix I get a bit excited about it and I just read smth about it before posting this. Tho I do have bad experience with breaking a franken debian install once way back when :D
I’ll obvs do research and maybe some distro hopping before deciding on the next thing, but imma chill for a while now. Specially cause I slept kinda bad because my brain kept trying to fix the distro in bed ><
That suicide did a number on me
Hey I encountered a similar issue. The boot drive switched from mounting to /efi/ to /boot/ on the off chance that this may be causing the same issue for you check fstabs and you kernel cmdline
Thanks but I’m not gonna try that out now. Just finished backing everything up and am kinda burnt out now trying to fix this issue. Honestly I’d rather just to a full reinstall.
Did the problem begin right after a kernel upgrade? Slim chance, but only thing I can think of is maybe some module your boot process depends on is no longer built-in the kernel and needs to be added to initrd manually. I always have an LTS kernel installed as backup to be able to exclude these kind of fucky issues.
If you don’t have an LTS kernel installed, you could try to boot a live usb, chroot into your EndeavourOS installation (don’t forget to do a
mount -ainside the chroot environment to mount boot and efi partitions) and install the LTS kernel. If you’re able to reboot with an LTS kernel that makes debugging a whole lot easier. Best of luck.(The ‘shorter uuids’ are probably the vfat boot and efi partitions btw. Maybe you have access to
lsblkorblkidin rescue mode to give some hints? You could compare them to live usb outputs of those commands)If the previous kernel is still available, can you boot that? That should help determine if the kernel is at fault.
Yeah I up upgraded the kernel, finished my song in windows, then Linux go boom.
Tried downgrading to 7.1.4 the previous version no luck, will try lts kernel tomorrowThe specific grub command wasn’t listed in the OP, I don’t think?
grub-mkconfig -o /boot/grub/grub.cfgThat’s Gentoo syntax anyway, and your grub config may live elsewhere, but you should get the idea. It’s supposed to find the new kernel.
I’ve never used an initrd, so I can’t comment on that.
I did do that with a new cfg file then diff’d the current and newly generated cfg files and they were identical
Fascinating.
I’d like to recommend rEFInd once you figure this out.
Windows has never been able to squash it.
thanks but since my boot manager already doesn’t live on the same drive as windows, it can’t squash it anyway hehe
I took a picture of the grub screen maybe that helps idk
Have you tried to boot from the grub shell manually? Here’s a guide if trying that is something that interests you. It seems like you’ve tried most everything else.
thanks, I might try that tho I don’t have high hopes since grub doesn’t seem to be the issue. You never know tho
Yeah, one more thing to try before you nuke and reroll. Good luck! I hope you’re able to find a satisfactory conclusion to your problem.
Yeah so grub can see the disk but then the kernel can’t. Efi and other tricks aren’t involved.
I think your route is to find out why the kernel hates your ass. Probably an upd pulled a driver … But who removes the SATA or nvme stuff?
Either way, it’s in the kernel and not the boot or the core apps. I don’t use endeavour, where I could provide better help, but I think the kernel is your best suspect for this crime.
well since endeavour is very close to arch I hope I’ll find some solutions in that realm. But yeah I’m thinking something must’ve gone awry with the kernel… hm I should downgrade that one too, only downgraded systemd but maybe 7.1.5 has some fuckery
since endeavour is very close to arch I hope I’ll find some solutions in that realm.
Smart thinking. I don’t use arch (btw) either, so I’m merely a cheerleader at this point. Yes, I’d downgrade the kernel for the sake of Science. Find a popular stable kernel, jump to that and see if it behaves better.
(That may be my bias: I only really do enterprise linux, so after 28 years since I moved from slackware I’m really risk-averse!)
You could install
linux-lts, I sometimes use that as a sanity check in case there is some kernel fuckery
can windows see the ssd? did you shut windows down the normal way or the hard way? because there’s one way to shut down windows that is basically a hard hibernate, where it keeps control of the drives. i don’t remember exactly how it works but it can lock other oses out.
yes windows sees the ssd. no I shut it down the normal way i.e. no hibernation.
The “normal” Windows shutdown is not a real shutdown. I don’t know if it will change anything for your situation, but for a full power off, you have to hold left shift while you click shutdown from the start menu, otherwise it is basically just hibernating.
Can you boot windows and shut it down normally again just to see if it was setting a dirty flag or something.
It’s not too hard to fix, but risky.
This sounds like the bit locker function that windows usea. I had this issue with it locking my extra ntfs drive when I installed Linux over my whole primary partition. However, your Windows Drive should not be able to read your Linux drive.
If Windows bit locker function still caused the trouble, there is a way to fix it but it will continue to happen until you remove Windows.
no drive encryption here, no bitlocker, no luks
Just a passing thought, but have you checked if the root partition has any space left?
all partitions have space left
Holy shit, are you me?
One minor update I have from this experience, after building a new rig and restoring from a backup of the original system, I tried plugging in my SSD to my new computer and it isn’t even recognized in BIOS. I was under the impression my drive failed.
hmm sad to see there’s no solution in that thread. tho its weird a reinstall didn’t fix the issue for you.
Agreed. I was under the impression mine was a graphics card issue, but the apparent failure of my hard drive is puzzling.
that’s the only thing I can say for certain: drive’s not failing…
Did you also update systemd at the same time? It might be worth reverting that update to see whether it’s that causing the issues.
To me, this sounds more like an fstab/systemd-automount issue rather than a bootloader/initrd issue
Maybe? But thanks for pointing me to systemd. Tho I don’t know how to downgrade a package from a live usb. I can find that out tho
@SavvyBeardedFish@reddthat.com downgraded systemd from 261.2 to 261.1, did not fix the issue sadly :/
Is the Linux disk still visible in your UEFI firmware?
Yeah it has to be, since the grub bootloader I use to boot either linux or windows is on the same disk as linux. If the UEFI firmware could not see the linux disk it couldn’t load grub
