Tag: system recovery

“Initramfs unpacking failed: invalid magic at start of compressed archive” error (SOLVED)

This article will show you how to fix the Linux won't boot error. In this case, the solution is shown using Kali Linux as an example, but the steps are also applicable to Debian, Ubuntu, Linux Mint, and derivative distributions.

An error occurred while booting Linux:

Initramfs unpacking failed: invalid magic at start of compressed archive
Kernel panic — not syncing: VFS: Unable to mount root fs on unknown-block(0,0)

The bottom line is that it was not possible to unpack the Initramfs due to archive corruption. Without this, the loading and operation of the operating system is impossible.

However, you can try to solve this problem without booting from a rescue disk (for example, a Live CD image).

First, try booting into Recovery Mode/Root Access on your current kernel.

To do this, select “Advanced options” in the bootloader.

Then select the item with “recovery mode” – usually the second line.

If you managed to do this, then run the following command:

sudo update-initramfs -c -k $(uname -r)

In my case, the error message changed, but the system was never booted.

If it was not possible to boot into recovery mode, then boot with a previous version of the kernel.

My system booted successfully with the previous version:

Now we need to recreate the ramdisk file. This can be done with a special command, but you need to know the kernel version number. Also, the ramdisk file is recreated each time a Linux kernel package is installed. Let's consider both options.

1. Reinstalling the Linux kernel

To reinstall the kernel, run the command:

sudo apt install linux-image-amd64

This command is suitable for 64-bit systems, if you have a 32-bit or ARM computer, then use the appropriate kernel package name linux-image-*

The previous command didn't work, and I got an error:

E: dpkg was interrupted, you must manually run 'sudo dpkg --configure -a' to correct the problem.

To fix the error, it is recommended to run the following command:

sudo dpkg --configure -a

It also didn't work and generated an error:

dpkg: error: fgets gave an empty string from ‘/var/lib/dpkg/triggers/Unincorp’

To fix it:

sudo rm /var/lib/dpkg/triggers/Unincorp
sudo touch /var/lib/dpkg/triggers/Unincorp

See also: dpkg: error: fgets gave an empty string from ‘/var/lib/dpkg/triggers/Unincorp’ (SOLVED)

Then the command to fix problems with packages was successfully executed:

sudo dpkg --configure -a

During its execution, the ramdisk file was re-created (as a step in setting up one of the packages), that is, reinstalling the Linux kernel was not required. After that I rebooted Linux and the operating system booted up as usual.

2. Generate ramdisk file

To generate an initramfs (ramdisk), you need to run a command like:

sudo update-initramfs -c -k VERSION

Instead of NUMBER, you need to specify the latest version of the Linux kernel installed on your system. That is, this is the version that causes problems.

You can view the available kernels with the command:

ls -al /boot

Execution result:

итого 149988
drwxr-xr-x  4 root root     4096 ноя 27 06:02 .
drwxr-xr-x 20 root root     4096 ноя 15 03:40 ..
-rw-r--r--  1 root root   254811 окт 10 16:05 config-5.19.0-kali2-amd64
-rw-r--r--  1 root root   257010 ноя  7 16:51 config-6.0.0-kali3-amd64
drwx------  3 root root     4096 янв  1  1970 efi
drwxr-xr-x  6 root root     4096 ноя 27 06:00 grub
-rw-r--r--  1 root root 67132798 ноя  9 03:32 initrd.img-5.19.0-kali2-amd64
-rw-r--r--  1 root root 70411394 ноя 27 06:02 initrd.img-6.0.0-kali3-amd64
-rw-r--r--  1 root root       83 окт 10 16:05 System.map-5.19.0-kali2-amd64
-rw-r--r--  1 root root       83 ноя  7 16:51 System.map-6.0.0-kali3-amd64
-rw-r--r--  1 root root  7703168 окт 10 16:05 vmlinuz-5.19.0-kali2-amd64
-rw-r--r--  1 root root  7788992 ноя  7 16:51 vmlinuz-6.0.0-kali3-amd64

The kernel version is numbers followed by words. For example, on this system, the latest kernel version is “6.0.0-kali3-amd64”. Then the command to create a new initramfs file is:

sudo update-initramfs -c -k 6.0.0-kali3-amd64

After that restart your computer:

reboot

dpkg: error: fgets gave an empty string from ‘/var/lib/dpkg/triggers/Unincorp’ (SOLVED)

When trying to use the apt package manager, for example:

sudo apt install linux-image-amd64

An error occurred:

E: dpkg was interrupted, you must manually run 'sudo dpkg --configure -a' to correct the problem.

The error is caused by disk problems or package upgrade failure.

When trying to use the recommended command:

sudo dpkg --configure -a

There was another error:

dpkg: error: fgets gave an empty string from '/var/lib/dpkg/triggers/Unincorp'
E: Sub-process /usr/bin/dpkg returned an error code (2)

To fix the issue, run the following commands:

sudo rm /var/lib/dpkg/triggers/Unincorp
sudo touch /var/lib/dpkg/triggers/Unincorp
sudo dpkg --configure -a

After that, run

sudo dpkg --configure -a

How to repair an LVM disk using fsck

How to repair an LVM disk

If, due to errors on the disk, the system cannot boot, then usually in the emergency mode console you need to check the disk partitions, approximately as follows (you need to specify your disk name and partition number):

umount /dev/sda2
fsck -y /dev/sda2

But if we are talking about LVM, or LVM with encryption, then the situation becomes more complicated.

See also:

You can determine that the LVM or LVM technology with encryption is used by the entries /dev/mapper/hostname--vg-root and /dev/mapper/hostname--vg-home, which are indicated instead of the partition names in the command output

mount

1. Checking and repairing an unencrypted LVM disk using fdisk

Use the following steps to repair LVM.

First, we check the partitioning of disks with lsblk:

lsblk

Output example:

NAME                 MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sda                    8:0    0    20G  0 disk  
└─sda1                 8:1    0    20G  0 part  
  └─xubuntu--vg-root 253:0    0    19G  0 lvm   / 
sr0                   11:0    1  1024M  0 rom   

As you can see, LVM is named xubuntu--vg-root, but we cannot run fsck on that name because the command will not find it. We need to get the full name, for this we need to run the lvm lvscan command to get the LV name with which we can run fsck on LVM.

The following command should be run with elevated privileges (with sudo or as root):

lvscan

Output example:

  inactive          '/dev/xubuntu-vg/root' [<19.04 GiB] inherit
  inactive          '/dev/xubuntu-vg/swap_1' [980.00 MiB] inherit

As you can see, the drive name to check for errors is /dev/xubuntu-vg/root, it should be good enough to run fsck on that name.

If /dev/xubuntu-vg/root root is not ACTIVE, we need to make it active so that we can start checking it.

lvchange -ay /dev/xubuntu-vg/root

It should now look something like this:

sudo lvscan
  ACTIVE            '/dev/xubuntu-vg/root' [<19.04 GiB] inherit
  inactive          '/dev/xubuntu-vg/swap_1' [980.00 MiB] inherit

Now we can run fsck to check the LVM volume:

fsck /dev/xubuntu-vg/root

or run a forced check with automatic error correction:

fsck -fy /dev/xubuntu-vg/root

2. Fix LVM with encryption

It may be that the actions from the previous example are possible only during normal system boot, when all possible utilities are available. Accordingly, in Emergency mode, these operations will fail.

Therefore, let's consider an example of recovering a system from a Live image. The reason for the analyzed problem is that “apt autoremove” command removed the “cryptsetup” package and other utilities important for decryption and normal operation of the partition. This caused the system to stop booting (because the root partition could not be mounted and decrypted using LVM).

If you are not using LVM and full disk encryption, then the probably following is not for you.

In this example, it was possible to fix the system and reinstall cryptsetup and lvm2 in a chroot environment: for this you needed to boot from the Live USB stick, run the following commands in the terminal and reboot

Finding the root partition:

sudo fdisk -l

We decrypt the partition.

sudo cryptsetup open --type luks /dev/nvme0n1p3 nvme0n1p3_crypt

Attention:

  • replace /dev/nvme0n1p3 with your own drive
  • replace “nvme0n1p3_crypt” with the correct partition name for your computer, you can find it by running the following in the chroot:
cat /etc/crypttab | cut -f1 -d " "

Output example:

nvme0n1p3_crypt

Mounting the root partition

sudo vgscan
sudo vgchange -ay
sudo mount /dev/mapper/xubuntu--vg-root /mnt

Preparing the chroot environment:

sudo mount /dev/nvme0n1p2 /mnt/boot/ # replace nvme0n1p2 with your boot partition!
sudo mount -o rbind /dev/ /mnt/dev/
sudo mount -t proc proc /mnt/proc/
sudo mount -t sysfs sys /mnt/sys/

Make DNS service available in chroot:

sudo cp /etc/resolv.conf /mnt/etc/resolv.conf

Enter the chroot:

sudo chroot /mnt /bin/bash

We reinstall the missing packages:

apt install cryptsetup lvm2

Re-generate (this can be done with the “apt” command in the previous step – if it was already done, then skip):

update-initramfs -u -k all

Leaving the chroot environment:

exit

Let’s write the buffer to disk:

sudo sync

Unmount the file systems:

sudo umount /mnt/sys
sudo umount /mnt/proc
sudo umount /mnt/boot

How to determine why Linux boots into Emergency mode

How to determine the exact reason why Systemd falls in emergency mode

A Linux system can go into an emergency mode shell if it encounters problems during boot.

The screen prompts you to execute the command

journalctl -xb

to find the causes of system problems.

It is also proposed to execute

systemctl default

or

exit

so that the system tries to boot normally.

You can try “systemctl default” – sometimes it really helps, but sometimes first you need to fix the problem that caused the Emergency mode.

The output of “journalctl -xb” is quite extensive, and examining it without filters does not always give a clue why it is booted into an emergency shell. Let's look at ways that can help you find the problem.

1. Finding problems with mounting

There are not so many reasons why the system goes into Emergency mode, usually these are problems with mounting disks and partitions. See what the following commands tell you?

systemctl status local-fs.target
journalctl -xb | grep -i -E 'local-fs.target'

2. Finding errors

What's do “journalctl -xb” tell? Try looking for mount and error related strings – maybe there is an answer.

journalctl -xb | grep -i mount
journalctl -xb | grep -i -E '(error|fail|warn|\(EE\))'

3. Unsuccessful fsck launch

Check fsck related entries:

journalctl -xb | grep -i -E 'fsck'
systemctl status systemd-fsck*

4. Unsuccessful start of any services

The following commands (they are identical) will list the services that failed to start:

systemctl --state=failed
systemctl --failed

5. Log search in Emergency mode and Maintenance mode

You can search for errors in the journald log without using commands – you may find it more convenient for you. Since journalctl uses the “less” command for multi-page browsing, you can use all of the keyboard shortcuts for this utility for your searches.

Output the log:

journalctl -xb

If you are relying on the search (/) function and are looking for something like “error”, “warning”, or “fail”, use -i to make sure the search is case insensitive.

List of commands and keys to search through journalctl (and less generally):

  • -i (case insensitive)
  • g (go to start)
  • /error (find “error”)
  • nnnn (skip nnnn results)
  • g (go to start)
  • /fail (find “fail”)
  • nnnn (skip nnnn results)
  • g (go to start)
  • /warn (find “warn”
  • nnnn (skip nnnn results)

Error “Cannot open access to console, the root account is locked” (SOLVED)

After a sudden power outage, unsuccessful update or adding a new disk to /etc/fstab, you may face the problem that your system does not boot, or rather, boots into the console or into a black screen. Sometimes the problem is compounded by the fact that the system administrator cannot even get into the emergency console. Let's see how to solve the following error:

You are in emergency mode. After logging in, type 'journalctl -xb' to view system logs, 'systemctl reboot'
'systemctl default' or "exit" to boot into default mode.

Cannot open access to console, the root account is locked.

See sulogin(8) man page for more details.

Press Enter to continue.

Reloading system manager configuration

After pressing Enter, everything can be repeated.

This message states that the system booted in emergency mode. In fact, this is not so bad – sometimes a system administrator can deliberately boot into emergency mode to restore the OS.

The real problem is that the root account is locked (this is indicated by the message “Cannot open access to console, the root account is locked”) and you cannot get into the console to start solving problems.

The situation becomes stalemate – the system will not let you go anywhere except in the root console, and it will not let you in the root console either, since this user is locked…

How to unlock the root user in emergency mode

Nevertheless, there is a way out of this situation. Start by booting into single user mode – this is the same mode used to reset the Linux password. Below is a generalized instruction, if something does not work out for you, then separate instructions for different distributions for booting into single-user mode can be found here.

Stop the booting by holding down the SHIFT key while starting the computer, you will see:

Press the “e” key and you will proceed to editing the boot settings:

Find the line starting with “linux”.

Go to the end of this line, insert a space, and add:

single init=/bin/bash

It should look something like this (the kernel number may differ):

When everything is ready, press Ctrl+x or F10 to continue the booting with the set options.

You will see a shell prompt, also note that we are logged in as root i.e. we have elevated privileges, including the use of the passwd command:

Command

passwd

will set the password for the root user.

If the passwd command fails:

passwd: Authentication token manipulation error
passwd: password unchanged

the filesystem is most likely mounted read-only. To verify this, enter the command:

mount

The “ro” indicate that the filesystem is mounted read-only and therefore the changes made cannot be saved. Remount the file system:

mount -rw -o remount /

After that, the password change should be successful.

Remove the password lock for the root user:

passwd -u root

If the root user is locked, then this may not be enough. Check which shell is set for root:

less /etc/passwd

If “/usr/sbin/nologin” is specified as the shell for root, run one of the following commands.

  • To assign a Bash shell to the root user:
sudo usermod -s /usr/bin/bash root
  • To assign a ZSH shell to the root user:
sudo usermod -s /usr/bin/zsh root

See also: How to change the login shell in Linux. chsh instruction

To exit, type:

sync
umount /

To turn off your computer run:

poweroff -f

Or restart your computer with the command:

reboot -f

If after the reboot you see “Give root password for maintenance”, then this means that the first stage of recovery was successful – we activated the root user and can now start restoring the system.

How to recover a computer in emergency mode

Now we have the opportunity to restore the system. If you have no idea what exactly caused the error, then run the command

journalctl -xb

and try to find the cause of the problem there.

Checking disks for errors

If you think the error is caused by hard disk problems, then use the following commands to check the partitions:

umount /dev/sda2
fsck -y /dev/sda2

Partition number and disk name may differ from “sda2”, to find out the exact name, use the command

fdisk -l

or

mount

Failed update

If the system does not boot due to an interrupted update, then try the following commands:

apt install -f -y
dpkg --configure -a

If you think that the problem was caused by an unsuccessful update of a specific package, then use a command like this:

dpkg-reconfigure PACKAGE

This command will reconfigure an already installed package.

For example, the following command will re-configure the Linux kernel:

dpkg-reconfigure linux-image-`uname -r`

System does not boot due to incorrect entry in /etc/fstab

In the event of an unsuccessful mount (this can happen if you made an incorrect entry in the /etc/fstab file, the system will not be able to boot, it will go into emergency mode and the following message will be displayed:

You are in emergency mode. After logging in, type "journalctl -xb" to view system logs, "systemctl reboot" to reboot, "systemctl default" or "exit" to boot into default mode.
Give root password for maintenance
(or press Control-D to continue):

To fix the problem, enter the root password and open the /etc/fstab file for editing:

nano /etc/fstab

Comment out or delete the problematic line. Save the file (Ctrl+o), close it (Ctrl+x) and reboot:

reboot
Loading...
X