Tag: fsck

How to repair an LVM disk using fsck

How to repair an LVM disk

If, due to errors on the disk, the system cannot boot, then usually in the emergency mode console you need to check the disk partitions, approximately as follows (you need to specify your disk name and partition number):

umount /dev/sda2
fsck -y /dev/sda2

But if we are talking about LVM, or LVM with encryption, then the situation becomes more complicated.

See also:

You can determine that the LVM or LVM technology with encryption is used by the entries /dev/mapper/hostname--vg-root and /dev/mapper/hostname--vg-home, which are indicated instead of the partition names in the command output

mount

1. Checking and repairing an unencrypted LVM disk using fdisk

Use the following steps to repair LVM.

First, we check the partitioning of disks with lsblk:

lsblk

Output example:

NAME                 MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sda                    8:0    0    20G  0 disk  
└─sda1                 8:1    0    20G  0 part  
  └─xubuntu--vg-root 253:0    0    19G  0 lvm   / 
sr0                   11:0    1  1024M  0 rom   

As you can see, LVM is named xubuntu--vg-root, but we cannot run fsck on that name because the command will not find it. We need to get the full name, for this we need to run the lvm lvscan command to get the LV name with which we can run fsck on LVM.

The following command should be run with elevated privileges (with sudo or as root):

lvscan

Output example:

  inactive          '/dev/xubuntu-vg/root' [<19.04 GiB] inherit
  inactive          '/dev/xubuntu-vg/swap_1' [980.00 MiB] inherit

As you can see, the drive name to check for errors is /dev/xubuntu-vg/root, it should be good enough to run fsck on that name.

If /dev/xubuntu-vg/root root is not ACTIVE, we need to make it active so that we can start checking it.

lvchange -ay /dev/xubuntu-vg/root

It should now look something like this:

sudo lvscan
  ACTIVE            '/dev/xubuntu-vg/root' [<19.04 GiB] inherit
  inactive          '/dev/xubuntu-vg/swap_1' [980.00 MiB] inherit

Now we can run fsck to check the LVM volume:

fsck /dev/xubuntu-vg/root

or run a forced check with automatic error correction:

fsck -fy /dev/xubuntu-vg/root

2. Fix LVM with encryption

It may be that the actions from the previous example are possible only during normal system boot, when all possible utilities are available. Accordingly, in Emergency mode, these operations will fail.

Therefore, let's consider an example of recovering a system from a Live image. The reason for the analyzed problem is that “apt autoremove” command removed the “cryptsetup” package and other utilities important for decryption and normal operation of the partition. This caused the system to stop booting (because the root partition could not be mounted and decrypted using LVM).

If you are not using LVM and full disk encryption, then the probably following is not for you.

In this example, it was possible to fix the system and reinstall cryptsetup and lvm2 in a chroot environment: for this you needed to boot from the Live USB stick, run the following commands in the terminal and reboot

Finding the root partition:

sudo fdisk -l

We decrypt the partition.

sudo cryptsetup open --type luks /dev/nvme0n1p3 nvme0n1p3_crypt

Attention:

  • replace /dev/nvme0n1p3 with your own drive
  • replace “nvme0n1p3_crypt” with the correct partition name for your computer, you can find it by running the following in the chroot:
cat /etc/crypttab | cut -f1 -d " "

Output example:

nvme0n1p3_crypt

Mounting the root partition

sudo vgscan
sudo vgchange -ay
sudo mount /dev/mapper/xubuntu--vg-root /mnt

Preparing the chroot environment:

sudo mount /dev/nvme0n1p2 /mnt/boot/ # replace nvme0n1p2 with your boot partition!
sudo mount -o rbind /dev/ /mnt/dev/
sudo mount -t proc proc /mnt/proc/
sudo mount -t sysfs sys /mnt/sys/

Make DNS service available in chroot:

sudo cp /etc/resolv.conf /mnt/etc/resolv.conf

Enter the chroot:

sudo chroot /mnt /bin/bash

We reinstall the missing packages:

apt install cryptsetup lvm2

Re-generate (this can be done with the “apt” command in the previous step – if it was already done, then skip):

update-initramfs -u -k all

Leaving the chroot environment:

exit

Let’s write the buffer to disk:

sudo sync

Unmount the file systems:

sudo umount /mnt/sys
sudo umount /mnt/proc
sudo umount /mnt/boot

How to determine why Linux boots into Emergency mode

How to determine the exact reason why Systemd falls in emergency mode

A Linux system can go into an emergency mode shell if it encounters problems during boot.

The screen prompts you to execute the command

journalctl -xb

to find the causes of system problems.

It is also proposed to execute

systemctl default

or

exit

so that the system tries to boot normally.

You can try “systemctl default” – sometimes it really helps, but sometimes first you need to fix the problem that caused the Emergency mode.

The output of “journalctl -xb” is quite extensive, and examining it without filters does not always give a clue why it is booted into an emergency shell. Let's look at ways that can help you find the problem.

1. Finding problems with mounting

There are not so many reasons why the system goes into Emergency mode, usually these are problems with mounting disks and partitions. See what the following commands tell you?

systemctl status local-fs.target
journalctl -xb | grep -i -E 'local-fs.target'

2. Finding errors

What's do “journalctl -xb” tell? Try looking for mount and error related strings – maybe there is an answer.

journalctl -xb | grep -i mount
journalctl -xb | grep -i -E '(error|fail|warn|\(EE\))'

3. Unsuccessful fsck launch

Check fsck related entries:

journalctl -xb | grep -i -E 'fsck'
systemctl status systemd-fsck*

4. Unsuccessful start of any services

The following commands (they are identical) will list the services that failed to start:

systemctl --state=failed
systemctl --failed

5. Log search in Emergency mode and Maintenance mode

You can search for errors in the journald log without using commands – you may find it more convenient for you. Since journalctl uses the “less” command for multi-page browsing, you can use all of the keyboard shortcuts for this utility for your searches.

Output the log:

journalctl -xb

If you are relying on the search (/) function and are looking for something like “error”, “warning”, or “fail”, use -i to make sure the search is case insensitive.

List of commands and keys to search through journalctl (and less generally):

  • -i (case insensitive)
  • g (go to start)
  • /error (find “error”)
  • nnnn (skip nnnn results)
  • g (go to start)
  • /fail (find “fail”)
  • nnnn (skip nnnn results)
  • g (go to start)
  • /warn (find “warn”
  • nnnn (skip nnnn results)
Loading...
X