Tag: file systems

How to check if a computer is using BIOS or UEFI; GRUB or systemd-boot bootloader; MBR or GPT partition table

How to know if UEFI or GRUB is being used?

The system booting process consists of the following 4 steps:

  1. System initialization: UEFI or BIOS (POST)
  2. Starting the bootloader (GRUB2 or systemd-boot)
  3. Kernel initialization
  4. Starting systemd, the parent of all processes

Related:

If the BIOS is used in the first step, then the GRUB bootloader is most likely involved in the second step.

If UEFI is used in the first step, then both the GRUB bootloader and the systemd-boot bootloader can be used in the second step.

If you're wondering if your computer is running UEFI or GRUB, then this question is the wrong one because UEFI and GRUB are not mutually exclusive.

Modern GRUB2 can work with both BIOS and UEFI (using efibootmgr). On Arch Linux, BIOS and UEFI support is bundled into a single grub package. In Debian and derivative distributions, GRUB comes in two versions:

  • grub-pc (BIOS version)
  • grub-efi (UEFI version)

How to find out if your computer is using BIOS or UEFI

The easiest way is to check if the computer is using UEFI. If the answer is negative, then this means that the BIOS is used on the computer.

To check if your computer supports UEFI, run the command:

ls -l /sys/firmware/efi

If a list of directories is displayed, for example this:

This means that this computer supports UEFI.

If an error message is shown:

ls: cannot access '/sys/firmware/efi': No such file or directory

This means that this computer is working with BIOS.

Also you can check for UEFI variables:

efivar -l

If a large list of variables is shown, then the computer was booted using UEFI.

If an error message is displayed

-bash: efivar: command not found

That means the computer is running on BIOS.

You can also use the efibootmgr utility to check:

efibootmgr

If it displays information about the UEFI Boot Manager, then UEFI is being used.

Please note that the latter method is not reliable, since the efibootmgr utility is not required even for UEFI computers. That is, if, when using UEFI, you receive a message that the command was not found, then the computer may support UEFI, but the efibootmgr package is not installed on it.

Another way is the bootctl utility:

sudo bootctl

Couldn't find EFI system partition. It is recommended to mount it to /boot or /efi.
Alternatively, use --esp-path= to specify path to mount point.
System:
    Not booted with EFI

Pay attention to the message “Not booted with EFI”, that is, the computer uses BIOS.

Finally, the presence of an EFI partition can be checked with fdisk:

sudo fdisk -l

If the boot disk has an “EFI System” partition, then this computer supports UEFI.

Since UEFI requires a separate “EFI System” partition, it can be argued that if this partition is not on the boot disk, then this computer does not support EFI, but works with BIOS.

How to find out if a computer is running GRUB or systemd-boot

To find out if the systemd-boot bootloader is being used, run the command:

sudo bootctl is-installed

The answer is rather concise: “yes” or “no”.

You will only get an answer if your computer uses UEFI. Otherwise, it will output:

Couldn't find EFI system partition. It is recommended to mount it to /boot or /efi.
Alternatively, use --esp-path= to specify path to mount point.

By exclusion, you can determine that if the systemd-boot bootloader is not used, then the GRUB bootloader is most likely used. But how to reliably confirm that it is the GRUB bootloader that is being used?

A system with GRUB installed must have a /boot/grub folder and a /boot/grub/grub.cfg file. To confirm that the GRUB bootloader is being used, you can check for their presence:

ls -l /boot/grub/grub.cfg

You can find other ways to identify GRUB, such as looking for the string “GRUB” in the boot area of the disk, but this method only applies to GRUB-PC! This method does not work with GRUB-EFI.

Since Debian currently has two different versions of GRUB-EFI and GRUB-PC, which are quite different, let's look at how to identify each of them.

How to know if GRUB-EFI or GRUB-PC is being used

As already mentioned, there are two versions of GRUB in Debian and derivatives:

  • grub-pc (BIOS version)
  • grub-efi (UEFI version)

They are quite different, and the methods for determining them also differ.

On Arch Linux and its derivatives, the GRUB-EFI and GRUB-PC versions are bundled into one package.

Using the previous information, you can logically determine whether GRUB-EFI or GRUB-PC is being used:

  1. If the computer uses EFI but does not use the systemd-boot bootloader, then the GRUB-EFI bootloader is used
  2. If the computer does not use EFI, then the GRUB-PC bootloader is probably used

There are also more reliable ways to check which version of the GRUB bootloader your computer is using.

First of all, check the contents of the /boot/grub directory:

ls -l /boot/grub

The presence in this directory of the subdirectory “x86_64-efi” indicates that this operating system uses GRUB-EFI:

The presence of the “i386-pc” directory says that GRUB-PC is being used:

If you receive an error message:

ls: cannot access '/boot/grub': No such file or directory

It means that this OS does not use GRUB.

In the Debian and derivatives repositories, the GRUB-EFI and GRUB-PC installation packages are marked as conflicting, meaning that only one of them can be installed. Therefore, we can check the installed packages to determine which version of GRUB is being used:

dpkg -l | grep grub | grep ii

The installed packages “grub-efi-amd64” and “grub-efi-amd64-bin” indicate that GRUB-EFI is being used:

The installed packages “grub-pc” and “grub-pc-bin” say that GRUB-PC is being used:

Another way to confirm that the operating system is using the GRUB-PC bootloader is the following command:

dd bs=512 count=1 if=/dev/sda 2>/dev/null | strings | grep GRUB

If any output is shown, GRUB-PC is being used.

In the previous command, you need to correctly specify the disk name, you can see the disk names with the command:

fdisk -l

An example command if the operating system is installed on the /dev/vda drive:

dd bs=512 count=1 if=/dev/vda 2>/dev/null | strings | grep GRUB

This screenshot shows that the OS uses GRUB-PC:

There is no similar command for GRUB-EFI.

How to find out if your computer is using an MBR or GPT partition table

Typically, EFI computers use the GPT partition table, and BIOS computers use the MBR, although exceptions are possible.

The following command will show the partition table type for a particular drive:

lsblk -dno PTTYPE /dev/DISK

For example:

lsblk -dno NAME,PTTYPE /dev/nvme0n1

And the following command will display the partition table types for all disks in the operating system:

lsblk -dno NAME,PTTYPE

To check which partition table is on the computer using fdisk, run the command:

sudo fdisk -l

Pay attention to the “Disklabel type” value:

  • gpt stands for GPT
  • dos stands for MBR

The same information can be obtained using the cfdisk utility:

cfdisk /dev/nvme0n1

Pay attention to the value of “Label”:

  • gpt stands for GPT
  • dos stands for MBR

Persistent names for block devices

Table of contents

1. How to make drives and USB sticks have the same name

2. Persistent block device naming

2.1 by-label

2.2 by-uuid

2.3 by-id

2.4 by-path

2.5 by-partlabel

2.6 by-partuuid

3. Using persistent naming

3.1 Permanent block device names can be used as aliases for /dev/sdX

3.2 Using persistent names in the /etc/fstab filesystem table

Conclusion


How to make drives and USB sticks have the same name

Block devices include hard drives, SSDs, USB sticks and disks, as well as memory cards.

In Linux, there are only block devices, that is, data in them is read and written in blocks, as opposed to character devices, in which data can be read or written in single characters. Character device files are used for unbuffered data exchange. On Linux, you must use a “raw” driver to get a character device for a disk, although you can get the same effect as opening a character device by opening a block device with the Linux-specific O_DIRECT flag.

The previous information is provided only for your understanding that “disks” and “block devices” are the same for the purposes of this article.

Internal and external disks, as well as USB flash drives have names like /dev/sdX, where the letter a, b, c and so on is substituted for X, for example: /dev/sda and /dev/sdb.

NVM Express (NVMe) device names start with nvme, for example /dev/nvme0n1.

If your computer has multiple SATA, SCSI, or IDE drive controllers, the order of adding the corresponding device nodes is arbitrary. This can cause device names such as /dev/sda and /dev/sdb to change with each other on every boot, making the system unable to boot. Also, assigning the same name to the same disk can be important for handling them in scripts or other uses when it is important that the block device has the same name.

Block device overview:

lsblk -f

In fact, Linux “out of the box” provides the ability to customize disks – no additional configuration or installation of utilities is required. The problem is that distinguishing one disk from another (especially after formatting) is not always an easy task, therefore, on the one hand, there is no unambiguous single solution, but on the other hand, several options are offered at once, among which you can choose the most suitable one.

The bottom line is the following, in addition to the names of the form /dev/sd* which most users are used to, there are four more permanent naming schemes, all of them are presented as links in directories:

  • /dev/disk/by-label
  • /dev/disk/by-uuid
  • /dev/disk/by-id
  • /dev/disk/by-path

Directories and files in /dev/disk/ are created and destroyed dynamically, depending on whether they have devices or not. That is, this means that some of these directories may not exist in your system, or be present only after connecting certain devices (disks).

By the way, this article has nothing to do with LVM logical volumes, since device paths of the form /dev/VolumeGroupName/LogicalVolumeName are persistent.

Persistent block device naming

by-label

Almost every type of filesystem can have a label. All your volumes that have it are listed in the /dev/disk/by-label directory:

ls -l /dev/disk/by-label

Output example:

total 0
lrwxrwxrwx 1 root root 10 May 27 23:31 Data -> ../../sda3
lrwxrwxrwx 1 root root 10 May 27 23:31 Arch\x20Linux -> ../../sda2

Advantages of this method of uniqueizing a disk and obtaining a permanent name:

  • you yourself choose the name by which you can access the disk

Disadvantages:

  • the label is lost when formatting and creating a new filesystem
  • by default, if not specified when the filesystem was created, most disks and partitions are unlabeled
  • label refers to the file system, i.e. it is not possible to create a label for an entire disk if it is partitioned
  • it is impossible to create a label for a disk without a file system

Most filesystems support customizing the label during filesystem creation, see the man page for the corresponding mkfs.* utilities for details. For some filesystems it is possible to change labels, below are several methods for changing labels on popular filesystems:

  • swap

Package: util-linux. Command to change disk or partition label:

swaplabel -L "new label" /dev/XXX
  • ext2/3/4

Package: e2fsprogs. Command to change disk or partition label:

e2label /dev/XXX "new label"
  • btrfs

Package: btrfs-progs. Command to change disk or partition label:

btrfs filesystem label /dev/XXX "new label"
  • reiserfs

Package: reiserfsprogs. Command to change disk or partition label:

reiserfstune -l "new label" /dev/XXX
  • jfs

Package: jfsutils. Command to change disk or partition label:

jfs_tune -L "new label" /dev/XXX
  • xfs

Package: xfsprogs. Command to change disk or partition label:

xfs_admin -L "new label" /dev/XXX
  • fat/vfat

Package: dosfstools. Command to change disk or partition label:

fatlabel /dev/XXX "new label"

Package: mtools. Command to change disk or partition label:

mlabel -i /dev/XXX ::"new label"
  • exfat

Package: exfatprogs. Command to change disk or partition label:

tune.exfat -L "new label" /dev/XXX

Package: exfatprogs or exfat-utils. Command to change disk or partition label:

exfatlabel /dev/XXX "new label"
  • ntfs

Package: ntfs-3g. Command to change disk or partition label:

ntfslabel /dev/XXX "new label"
  • udf

Package: udftools. Command to change disk or partition label:

udflabel /dev/XXX "new label"
  • crypto_LUKS (LUKS2 only)

Package: cryptsetup. Command to change disk or partition label:

cryptsetup config --label="new label" /dev/XXX

The device label can be viewed with lsblk:

lsblk -dno LABEL /dev/sda2

Or with blkid:

sudo blkid -s LABEL -o value /dev/sda2

Notes:

  • The filesystem must not be mounted to change the label. For the root file system, this can be achieved by booting from a different volume (from the Live system, for example).
  • Labels should be different for all partitions to avoid possible conflicts.
  • Labels can be up to 16 characters long.
  • Since the label is a property of the file system, it is not suitable for permanently addressing a single device in a RAID array.
  • When using encrypted containers with dm-crypt, the filesystem labels inside the containers are not available while the container is locked/encrypted.

by-uuid

UUID is a mechanism that allows each file system to be assigned a unique identifier. These identifiers are generated by filesystem utilities (such as mkfs.*) When the device is formatted and are designed so that collisions are unlikely. All GNU/Linux filesystems (including the swap and LUKS headers of raw encrypted devices) support UUIDs. FAT, exFAT and NTFS file systems do not support UUIDs, but are still listed in /dev/disk/by-uuid/ with a shorter UID (unique identifier):

ls -l /dev/disk/by-uuid/

The UUID of the disk (if there is only one partition) can be obtained using lsblk:

lsblk -dno UUID /dev/sda
aad9d8d7-b1b9-435a-9e41-fc8159d2c484

If there are several partitions, then you need to specify the partition number:

lsblk -dno UUID /dev/nvme0n1p1
7327-B4E4

There is no UUID for the disk as a whole if the disk contains multiple partitions. Also there is no UUID for disks and partitions without file systems.

The advantage of using the UUID method is that name collisions are much less likely than with labels. It is also generated automatically when the file system is created. For example, it will remain unique even if the device is connected to another system (which could possibly have a device with the same lable).

The downside is that UUIDs are difficult for humans to read and break formatting in many configuration files (like fstab or crypttab). In addition, every time the volume is reformatted, a new UUID is created and the configuration files must be manually changed.

Tip:

  • if your swap does not have a UUID assigned, you will need to reset it using the mkswap utility.

by-id

by-id creates a unique name based on the serial number of the hardware. It contains a string indicating which subsystem the device belongs to (for example, ata-, nvme-, usb-, or wwn- for by-id), so it is associated with the hardware that controls the device.

An example of a by-id for a USB flash drive: /dev/disk/by-id/usb-Samsung_Flash_Drive_0325121050019952-0:0

This identifier will be constant even if the filesystem is formatted. Thus, it can be used to access RAW file devices.

by-id also creates World Wide Name links for storage devices that support it. Unlike other links by ID, WWNs are completely persistent and do not change depending on the subsystem used.

The by-id will change when the device is connected to a hardware controller port that is subordinate to another subsystem. It means that if the device is connected as USB or as SATA, then its by-id will change.

by-path

by-path creates a unique name based on the shortest physical path (according to sysfs).

The by-path value for a file device will change if you switch it from one slot to another. In the case of a single socket, the by-path value remains constant.

by-partlabel

This method only applies to disks with a GUID Partition Table (GPT).

GPT Partition Labels can be defined in the header of a partition entry on GPT disks.

This method is very similar to file system labels, except that the partition labels do not change when the file system on the partition changes.

All partitions with partition labels are listed in the /dev/disk/by-partlabel/ directory.

ls -l /dev/disk/by-partlabel/
total 0
lrwxrwxrwx 1 root root 10 May 27 23:31 EFI\x20system\x20partition -> ../../sda1
lrwxrwxrwx 1 root root 10 May 27 23:31 GNU\x2fLinux -> ../../sda2
lrwxrwxrwx 1 root root 10 May 27 23:31 Home -> ../../sda3
lrwxrwxrwx 1 root root 10 May 27 23:31 Swap -> ../../sda4

The device partition label can be obtained using lsblk:

lsblk -dno PARTLABEL /dev/sda1
EFI system partition

Or with blkid:

sudo blkid -s PARTLABEL -o value /dev/sda1
EFI system partition

Notes:

  • GPT partition labels should also be different to avoid conflicts. To change the label of a partition, you can use gdisk or the ncurses version of cgdisk. Both are available in the gptfdisk package.
  • According to the specification, GPT partition labels can be up to 72 characters long.
  • The by-partlabel will be cleared when the GPT scheme is removed.

by-partuuid

Like GPT partition labels, the UUIDs of GPT partitions are defined in the partition entry on GPT disks.

MBR does not support UUID partitions, but Linux and software using libblkid (like udev) are capable of generating a pseudo PARTUUID for MBR partitions. The format is SSSSSSSS-PP, where SSSSSSSS is the 32-bit zero padded MBR disk signature and PP is the zero padded partition number in hexadecimal form. Unlike the usual PARTUUID of a GPT partition, the pseudo PARTUUID of the MBR can change when the partition number changes.

ls -l /dev/disk/by-partuuid/
total 0
lrwxrwxrwx 1 root root 10 May 27 23:31 0003e1e5-01 -> ../../mmcblk0p1
lrwxrwxrwx 1 root root 10 May 27 23:31 039b6c1c-7553-4455-9537-1befbc9fbc5b -> ../../sda4
lrwxrwxrwx 1 root root 10 May 27 23:31 7280201c-fc5d-40f2-a9b2-466611d3d49e -> ../../sda3
lrwxrwxrwx 1 root root 10 May 27 23:31 98a81274-10f7-40db-872a-03df048df366 -> ../../sda2
lrwxrwxrwx 1 root root 10 May 27 23:31 d0d0d110-0a71-4ed6-936a-304969ea36af -> ../../sda1

The UUID of the device partition can be obtained using lsblk:

lsblk -dno PARTUUID /dev/sda1
d0d0d110-0a71-4ed6-936a-304969ea36af

Or with blkid:

sudo blkid -s PARTUUID -o value /dev/sda1
d0d0d110-0a71-4ed6-936a-304969ea36af

Using persistent naming

Permanent block device names can be used as aliases for /dev/sdX

You can use persistent names in the same way as /dev/sdX. That is, in programs for working with block devices, such as mount, umount, fdisk, gdisk, VBoxManage and others, you can use one of the constant names instead of the name /dev/sdX.

An example of unmounting a disk by its by-id:

sudo umount '/dev/disk/by-id/usb-Samsung_Flash_Drive_0325121050019952-0:0-part2'

An example of getting information about disk partitions by its by-id:

fdisk -l '/dev/disk/by-id/usb-Samsung_Flash_Drive_0325121050019952-0:0'

An example of creating a virtual disk file pointing to a real disk by its by-id:

VBoxManage internalcommands createrawvmdk -filename flash.vmdk -rawdisk '/dev/disk/by-id/usb-Samsung_Flash_Drive_0325121050019952-0:0'

Related: How to boot into VirtualBox from USB

Please note that it is not enough to specify only the identifier, for example usb-Samsung_Flash_Drive_0325121050019952-0:0, you must specify the full name like /dev/disk/by-*, for example '/dev/disk/by-id/usb-Samsung_Flash_Drive_0325121050019952-0:0'.

Using persistent names in the /etc/fstab filesystem table

You can also use persistent disk device names in the /etc/fstab file, but you must explicitly specify which disk alias to use.

LABEL=<label> or UUID=<uuid> may be given instead of a device name. This is the recommended method, as device names are often a coincidence of hardware detection order, and can change when other disks are added or removed. For example, 'LABEL=Boot' or 'UUID=3e6be9de-8139-11d1-9106-a43f08d823a6'. (Use a filesystem-specific tool like e2label, xfs_admin, or fatlabel to set LABELs on filesystems).

It’s also possible to use PARTUUID= and PARTLABEL=. These partitions identifiers are supported for example for GUID Partition Table (GPT).

The string representation of the UUID should be based on lower case characters. But when specifying the volume ID of FAT or NTFS file systems upper case characters are used (e.g UUID="A40D-85E7" or UUID="61DB7756DB7779B3").

An example /etc/fstab file using UUIDs (common names of disk devices are indicated in the comments):

# /dev/nvme0n1p2
UUID=01e4d4f5-698f-4dc2-987b-270499457f48	/         	ext4      	rw,relatime	0 1

# /dev/nvme0n1p1
UUID=7327-B4E4      	/boot     	vfat      	rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,utf8,errors=remount-ro	0 2
/dev/sda   /mnt/disk_d  ext4    rw,relatime 0   2

# /dev/sdc1 1.5 TB external USB
UUID=26FC3023FC2FEC2D	/mnt/disk_e	ntfs	noauto,nofail,rw,utf8	0	0

Note that you do not need to specify a fully qualified name like /dev/disk/by-*.

Conclusion

So, for most applications that do not require formatting and creating new file systems, constant names (aliases) from the /dev/disk/by-label and /dev/disk/by-uuid directories are suitable.

If you need to use the disk as a RAW device, that is, without relying on its file systems, then aliases from the /dev/disk/by-id directory will do the job.

How to repair an LVM disk using fsck

How to repair an LVM disk

If, due to errors on the disk, the system cannot boot, then usually in the emergency mode console you need to check the disk partitions, approximately as follows (you need to specify your disk name and partition number):

umount /dev/sda2
fsck -y /dev/sda2

But if we are talking about LVM, or LVM with encryption, then the situation becomes more complicated.

See also:

You can determine that the LVM or LVM technology with encryption is used by the entries /dev/mapper/hostname--vg-root and /dev/mapper/hostname--vg-home, which are indicated instead of the partition names in the command output

mount

1. Checking and repairing an unencrypted LVM disk using fdisk

Use the following steps to repair LVM.

First, we check the partitioning of disks with lsblk:

lsblk

Output example:

NAME                 MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sda                    8:0    0    20G  0 disk  
└─sda1                 8:1    0    20G  0 part  
  └─xubuntu--vg-root 253:0    0    19G  0 lvm   / 
sr0                   11:0    1  1024M  0 rom   

As you can see, LVM is named xubuntu--vg-root, but we cannot run fsck on that name because the command will not find it. We need to get the full name, for this we need to run the lvm lvscan command to get the LV name with which we can run fsck on LVM.

The following command should be run with elevated privileges (with sudo or as root):

lvscan

Output example:

  inactive          '/dev/xubuntu-vg/root' [<19.04 GiB] inherit
  inactive          '/dev/xubuntu-vg/swap_1' [980.00 MiB] inherit

As you can see, the drive name to check for errors is /dev/xubuntu-vg/root, it should be good enough to run fsck on that name.

If /dev/xubuntu-vg/root root is not ACTIVE, we need to make it active so that we can start checking it.

lvchange -ay /dev/xubuntu-vg/root

It should now look something like this:

sudo lvscan
  ACTIVE            '/dev/xubuntu-vg/root' [<19.04 GiB] inherit
  inactive          '/dev/xubuntu-vg/swap_1' [980.00 MiB] inherit

Now we can run fsck to check the LVM volume:

fsck /dev/xubuntu-vg/root

or run a forced check with automatic error correction:

fsck -fy /dev/xubuntu-vg/root

2. Fix LVM with encryption

It may be that the actions from the previous example are possible only during normal system boot, when all possible utilities are available. Accordingly, in Emergency mode, these operations will fail.

Therefore, let's consider an example of recovering a system from a Live image. The reason for the analyzed problem is that “apt autoremove” command removed the “cryptsetup” package and other utilities important for decryption and normal operation of the partition. This caused the system to stop booting (because the root partition could not be mounted and decrypted using LVM).

If you are not using LVM and full disk encryption, then the probably following is not for you.

In this example, it was possible to fix the system and reinstall cryptsetup and lvm2 in a chroot environment: for this you needed to boot from the Live USB stick, run the following commands in the terminal and reboot

Finding the root partition:

sudo fdisk -l

We decrypt the partition.

sudo cryptsetup open --type luks /dev/nvme0n1p3 nvme0n1p3_crypt

Attention:

  • replace /dev/nvme0n1p3 with your own drive
  • replace “nvme0n1p3_crypt” with the correct partition name for your computer, you can find it by running the following in the chroot:
cat /etc/crypttab | cut -f1 -d " "

Output example:

nvme0n1p3_crypt

Mounting the root partition

sudo vgscan
sudo vgchange -ay
sudo mount /dev/mapper/xubuntu--vg-root /mnt

Preparing the chroot environment:

sudo mount /dev/nvme0n1p2 /mnt/boot/ # replace nvme0n1p2 with your boot partition!
sudo mount -o rbind /dev/ /mnt/dev/
sudo mount -t proc proc /mnt/proc/
sudo mount -t sysfs sys /mnt/sys/

Make DNS service available in chroot:

sudo cp /etc/resolv.conf /mnt/etc/resolv.conf

Enter the chroot:

sudo chroot /mnt /bin/bash

We reinstall the missing packages:

apt install cryptsetup lvm2

Re-generate (this can be done with the “apt” command in the previous step – if it was already done, then skip):

update-initramfs -u -k all

Leaving the chroot environment:

exit

Let’s write the buffer to disk:

sudo sync

Unmount the file systems:

sudo umount /mnt/sys
sudo umount /mnt/proc
sudo umount /mnt/boot

How to determine why Linux boots into Emergency mode

How to determine the exact reason why Systemd falls in emergency mode

A Linux system can go into an emergency mode shell if it encounters problems during boot.

The screen prompts you to execute the command

journalctl -xb

to find the causes of system problems.

It is also proposed to execute

systemctl default

or

exit

so that the system tries to boot normally.

You can try “systemctl default” – sometimes it really helps, but sometimes first you need to fix the problem that caused the Emergency mode.

The output of “journalctl -xb” is quite extensive, and examining it without filters does not always give a clue why it is booted into an emergency shell. Let's look at ways that can help you find the problem.

1. Finding problems with mounting

There are not so many reasons why the system goes into Emergency mode, usually these are problems with mounting disks and partitions. See what the following commands tell you?

systemctl status local-fs.target
journalctl -xb | grep -i -E 'local-fs.target'

2. Finding errors

What's do “journalctl -xb” tell? Try looking for mount and error related strings – maybe there is an answer.

journalctl -xb | grep -i mount
journalctl -xb | grep -i -E '(error|fail|warn|\(EE\))'

3. Unsuccessful fsck launch

Check fsck related entries:

journalctl -xb | grep -i -E 'fsck'
systemctl status systemd-fsck*

4. Unsuccessful start of any services

The following commands (they are identical) will list the services that failed to start:

systemctl --state=failed
systemctl --failed

5. Log search in Emergency mode and Maintenance mode

You can search for errors in the journald log without using commands – you may find it more convenient for you. Since journalctl uses the “less” command for multi-page browsing, you can use all of the keyboard shortcuts for this utility for your searches.

Output the log:

journalctl -xb

If you are relying on the search (/) function and are looking for something like “error”, “warning”, or “fail”, use -i to make sure the search is case insensitive.

List of commands and keys to search through journalctl (and less generally):

  • -i (case insensitive)
  • g (go to start)
  • /error (find “error”)
  • nnnn (skip nnnn results)
  • g (go to start)
  • /fail (find “fail”)
  • nnnn (skip nnnn results)
  • g (go to start)
  • /warn (find “warn”
  • nnnn (skip nnnn results)

Error “Cannot open access to console, the root account is locked” (SOLVED)

After a sudden power outage, unsuccessful update or adding a new disk to /etc/fstab, you may face the problem that your system does not boot, or rather, boots into the console or into a black screen. Sometimes the problem is compounded by the fact that the system administrator cannot even get into the emergency console. Let's see how to solve the following error:

You are in emergency mode. After logging in, type 'journalctl -xb' to view system logs, 'systemctl reboot'
'systemctl default' or "exit" to boot into default mode.

Cannot open access to console, the root account is locked.

See sulogin(8) man page for more details.

Press Enter to continue.

Reloading system manager configuration

After pressing Enter, everything can be repeated.

This message states that the system booted in emergency mode. In fact, this is not so bad – sometimes a system administrator can deliberately boot into emergency mode to restore the OS.

The real problem is that the root account is locked (this is indicated by the message “Cannot open access to console, the root account is locked”) and you cannot get into the console to start solving problems.

The situation becomes stalemate – the system will not let you go anywhere except in the root console, and it will not let you in the root console either, since this user is locked…

How to unlock the root user in emergency mode

Nevertheless, there is a way out of this situation. Start by booting into single user mode – this is the same mode used to reset the Linux password. Below is a generalized instruction, if something does not work out for you, then separate instructions for different distributions for booting into single-user mode can be found here.

Stop the booting by holding down the SHIFT key while starting the computer, you will see:

Press the “e” key and you will proceed to editing the boot settings:

Find the line starting with “linux”.

Go to the end of this line, insert a space, and add:

single init=/bin/bash

It should look something like this (the kernel number may differ):

When everything is ready, press Ctrl+x or F10 to continue the booting with the set options.

You will see a shell prompt, also note that we are logged in as root i.e. we have elevated privileges, including the use of the passwd command:

Command

passwd

will set the password for the root user.

If the passwd command fails:

passwd: Authentication token manipulation error
passwd: password unchanged

the filesystem is most likely mounted read-only. To verify this, enter the command:

mount

The “ro” indicate that the filesystem is mounted read-only and therefore the changes made cannot be saved. Remount the file system:

mount -rw -o remount /

After that, the password change should be successful.

Remove the password lock for the root user:

passwd -u root

If the root user is locked, then this may not be enough. Check which shell is set for root:

less /etc/passwd

If “/usr/sbin/nologin” is specified as the shell for root, run one of the following commands.

  • To assign a Bash shell to the root user:
sudo usermod -s /usr/bin/bash root
  • To assign a ZSH shell to the root user:
sudo usermod -s /usr/bin/zsh root

See also: How to change the login shell in Linux. chsh instruction

To exit, type:

sync
umount /

To turn off your computer run:

poweroff -f

Or restart your computer with the command:

reboot -f

If after the reboot you see “Give root password for maintenance”, then this means that the first stage of recovery was successful – we activated the root user and can now start restoring the system.

How to recover a computer in emergency mode

Now we have the opportunity to restore the system. If you have no idea what exactly caused the error, then run the command

journalctl -xb

and try to find the cause of the problem there.

Checking disks for errors

If you think the error is caused by hard disk problems, then use the following commands to check the partitions:

umount /dev/sda2
fsck -y /dev/sda2

Partition number and disk name may differ from “sda2”, to find out the exact name, use the command

fdisk -l

or

mount

Failed update

If the system does not boot due to an interrupted update, then try the following commands:

apt install -f -y
dpkg --configure -a

If you think that the problem was caused by an unsuccessful update of a specific package, then use a command like this:

dpkg-reconfigure PACKAGE

This command will reconfigure an already installed package.

For example, the following command will re-configure the Linux kernel:

dpkg-reconfigure linux-image-`uname -r`

System does not boot due to incorrect entry in /etc/fstab

In the event of an unsuccessful mount (this can happen if you made an incorrect entry in the /etc/fstab file, the system will not be able to boot, it will go into emergency mode and the following message will be displayed:

You are in emergency mode. After logging in, type "journalctl -xb" to view system logs, "systemctl reboot" to reboot, "systemctl default" or "exit" to boot into default mode.
Give root password for maintenance
(or press Control-D to continue):

To fix the problem, enter the root password and open the /etc/fstab file for editing:

nano /etc/fstab

Comment out or delete the problematic line. Save the file (Ctrl+o), close it (Ctrl+x) and reboot:

reboot

How to use lsof to view open files (on Linux everything is files)

If everything in Linux is a file, then there must be more to this operating system than just files on your hard drive. In this tutorial, you will learn how to use lsof to view all other devices and processes that are being treated as files.

Everything in Linux is files

The oft-quoted phrase that everything in Linux is a file is kind of true. A file is a collection of bytes. Programs and devices (printer, screen, keyboard, mouse, terminal, network card) can generate or receive streams of bytes, and in this capacity they themselves appear as files. Many system components such as keyboards, sockets, printers, and communications processes receive or generate streams of bytes. Because they accept, generate, or receive and generate streams of bytes, these devices can be treated – at a very low level – like files. In addition to ordinary files, files are also directories (folders), network sockets (network connections), devices (disks, terminals, keyboard, mouse, etc. – these are all files), pipe (a technology for exchanging data between processes).

This device concept has simplified the implementation of the Unix operating system. This meant that a small set of handlers, tools, and APIs could be built to handle a wide variety of different resources.

The data and program files that reside on your hard drive are regular files in the file system. We can use the ls command to list them and find out some details about them. But how do we know about all other processes and devices that are treated as files? The lsof command is used for this. It lists the open files on the system. That is, it lists everything that is processed as if it were a file.

lsof command

lsof stands for “LiSt Open Files”. This program is used to find out which files are open and by which process.

Why do you need to know which files are open? This information will help you learn about a lot that is happening in the system, about the device and operation of Linux, and also solve problems, for example, when you cannot unmount the disk due to the device being in use, but you cannot find which program.

Many processes or devices that lsof can report are owned by root or were started by the root user, so you will need to use the sudo with lsof.

And since this list will be very long, you can pipe it to the less command:

sudo lsof | less

Before lsof output appears, users might see a warning message in a terminal window.

lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/1000/gvfs
      Output information may be incomplete.

lsof tries to process all mounted filesystems. This warning message appears because lsof has detected the GNOME Virtual File System (GVFS). This is a special case of a user space filesystem (FUSE). It acts as a bridge between GNOME, its API, and the kernel. No one, not even root, can access one of these filesystems other than the owner who mounted it (in this case, the current user). You can ignore this warning. If you want to access this filesystem, then run lsof without sudo.

The lsof output header:

lsof column meaning

There are many types of open files, and not all columns apply to all of them. Therefore, if for some files some columns are not filled with data, then this is normal.

  • COMMAND contains the first nine characters of the name of the UNIX command associated with the process.
  • PID is the Process IDentification number of the process.
  • TID is the task (thread) IDentification number, if task (thread) reporting is supported by the dialect and a task (thread) is being listed. A blank TID column in Linux indicates a process – i.e., a non-task.
  • TASKCMD is the task command name. Generally this will be the same as the process named in the COMMAND column, but some task implementations (e.g., Linux) permit a task to change its command name.
  • USER is the user ID number or login name of the user to whom the process belongs, usually the same as reported by ps. However, on Linux USER is the user ID number or login that owns the directory in /proc where lsof finds information about the process. Usually that is the same value reported by ps, but may differ when the process has changed its effective user ID.
  • FD is the File Descriptor number of the file. File descriptors are described below.
  • TYPE is the type of node associated with the file. The types of these types are described below.
  • DEVICE contains the device numbers, separated by commas, for a character special, block special, regular, directory or NFS file.
  • SIZE/OFF is the size of the file or the file offset in bytes.
  • NODE is the node number of a local file; or the inode number of an NFS file in the server host; or the Internet protocol type - e.g, “TCP”; or “STR” for a stream; or “CCITT” for an HP-UX x.25 socket; or the IRQ or inode number of a Linux AX.25 socket device.
  • NAME is the name of the mount point and file system on which the file resides. For other types of files, specific data is indicated here, details will be given below.

Column FD

The file descriptor in the FD column can be one of many; the man page lists them all:

man lsof

An entry in the FD column can have three parts: a file descriptor, a mode symbol, and a lock symbol.

Some popular file descriptors are:

  • cwd: current working directory;
  • err FD: information error (see NAME column);
  • ltx: shared library text (code and data);
  • m86: DOS Merge mapped file;
  • mem: memory-mapped file;
  • mmap: memory-mapped device;
  • pd: parent directory;
  • rtd: root directory;
  • txt: program text (code and data);

The mode symbol can be one of the following:

  • r: for read access;
  • w: for write access;
  • u: for read and write access;
  •  : [space character] if mode unknown and no lock character follows;
  • -: if mode unknown and lock character follows.

The lock symbol can be one of:

  • r: for read lock on part of the file;
  • R: for a read lock on the entire file;
  • w: for a write lock on part of the file;
  • W: for a write lock on the entire file;
  • u: for a read and write lock of any length;
  • U: for a lock of unknown type;
  •  : [space character] if there is no lock.

Column TYPE

More than 70 entries can be displayed in the TYPE column. The following are just a few of the commonly used entries:

  • REG: Regular file system file.
  • DIR: Directory.
  • FIFO: for a FIFO (First In First Out) special file.
  • CHR: Character special file.
  • BLK: Block special file.
  • INET: Internet socket.
  • unix: UNIX domain socket.
  • IPv4: IPv4 socket.
  • IPv6: IPv6 files on the network – even if its IPv4 address is mapped to an IPv6 address.
  • sock: Unknown domain socket.
  • DEL: Linux pointer to remote file.
  • LINK: A symbolic link file.
  • PIPE: A pipe is a way of exchanging data between processes.

Column NAME

This field can be:

  • the name of the mount point and file system on which the file resides;
  • or the name of a file specified in the names option (after any symbolic links have been resolved);
  • or the name of a character special or block special device;
  • or the local and remote Internet addresses of a network file; the local host name or IP number is followed by a colon (':'), the port, “->”, and the two-part remote address; IP addresses may be reported as numbers or names, depending on the +|-M, -n, and -P options; colon-separated IPv6 numbers are enclosed in square brackets; IPv4 INADDR_ANY and IPv6 IN6_IS_ADDR_UNSPECIFIED addresses, and zero port numbers are represented by an asterisk ('*'); a UDP destination address may be followed by the amount of time elapsed since the last packet was sent to the destination; TCP, UDP and UDPLITE remote addresses may be followed by TCP/TPI information in parentheses - state (e.g., “(ESTABLISHED)”, “(Unbound)”), queue sizes, and window sizes (not all dialects) – in a fashion similar to what netstat reports; see the -T option description or the description of the TCP/TPI field in OUTPUT FOR OTHER PROGRAMS for more information on state, queue size, and window size;
  • or the address or name of a UNIX domain socket, possibly including a stream clone device name, a file system object's path name, local and foreign kernel addresses, socket pair information, and a bound vnode address;
  • or the local and remote mount point names of an NFS file;
  • or “STR”, followed by the stream name;
  • or a stream character device name, followed by “->” and the stream name or a list of stream module names, separated by “→”;
  • or another value that is usually typical for certain operating systems.

How to see the processes that opened the file

To see the processes that opened a specific file, specify the filename as a parameter to lsof. For example, to see the processes that opened the file /dev/sda, use this command:

lsof /dev/sda

Output example:

COMMAND    PID USER   FD   TYPE DEVICE SIZE/OFF     NODE NAME
doublecmd 1982 mial  cwd    DIR    8,0     4096 32768001 /mnt/disk_d/Share

As you may know (and if you don’t, see the article “Linux directory structure. Important Linux files”), the file /dev/sda is a hard disk. The above command will help you find the process that prevents you from unmounting (detaching) the disk.

You can check which command is used to open regular files, for example:

lsof 'Documents/Linux.odt'

Output:

COMMAND    PID USER   FD   TYPE DEVICE SIZE/OFF    NODE NAME
soffice.b 3686 mial   92uW  REG  259,2    19826 5390035 Documents/Linux.odt

You can see that the file is opened by the user mial, it is a regular file (REG) and that it is opened by the soffice.b application (actually the soffice.bin application, but the name was not written in full).

To display the full name, use the +c option, for example:

lsof +c 15 /home/mial/Вирус.odt

Similarly, you can check if executable files are open (running), for example, to check if the file /bin/bash is running:

sudo lsof /bin/bash

How to view all files opened in a specific directory

To see the files that were opened from a directory and the processes that opened them, pass the directory to lsof as a parameter. To do this, use the +D DIRECTORY option.

To see all files that are open in the /var/log/ directory, use this command:

sudo lsof +D /var/log/

lsof responds with a list of all open files in that directory.

To see all files that were opened from the /home directory, use the following command:

sudo lsof +D /home

Files that were opened from the /home directory. Please note that some columns do not fully fit the data.

How to show only regular files

To show regular files and directories and exclude all other kinds of files (devices, sockets, etc.), run the following command:

sudo lsof /

How to see files that are open with a specified program or command

The COMMAND column contains the name of the executable file that opened the file. To see the files that were opened by a particular process, use the -c COMMAND option. Note that you can specify more than one search query for lsof at the same time.

sudo lsof -c ssh -c systemd

lsof will list files that were opened by any of the processes provided on the command line.

How to view files opened by a specific user

To limit the display to files that were opened by a specific user, use the -u USER option. In this example, we will look at files that were opened by processes that are owned or run as mial.

sudo lsof -u mial

All files that were opened as user mial will be listed. They include files that were opened, for example, by the desktop environment, or simply as a result of mial login.

How to exclude files opened by a specific user

To exclude files that were opened by the user, use the ^ operator. Removing users from the list makes it easier to find the information you are interested in. You must use the -u option as before and add a ^ at the beginning of the username.

sudo lsof -u ^mial

Negation (searching for any values other than the ones listed) can be used with any options for which a value can be specified, that is, with usernames, command names, process IDs, folders, etc.

For example, to show all files that were open in the /home directory, but which are opened by processes not owned by mial:

sudo lsof +D /home -u ^mial

How to find files that are open by a specified process

To list the files that have been opened by a specific process, use the -p PROCESS parameter and specify the process ID as the parameter.

sudo lsof -p 9099

All files that were opened by the process ID you specified are listed for you.

How to find the IDs of the processes that opened the file

To see the IDs of the processes that opened a particular file, use the -t FILE option.

sudo lsof -t /usr/share/mime/mime.cache

The IDs of the process that opened the file are displayed in a simple list.

How to display the size of open files

Use the -s option to display the size:

lsof -s

To sort by size of regular files, run the following command:

lsof / -s | sort -rnk 7

How to find out which processes and how much memory they consume in the Buff/Cache area

Buffer/cache is used by virtual file systems and also to optimize disk read/write, that is, files that have already been read from disk are placed there for faster access.

The following command sorts open files by size – they are the ones that take up space in buffer/cache:

lsof -s | sort -rnk 7 | less

Buffer/cache consumes not so much the kernel as all sorts of browsers, office programs, graphical environments, programs with large logs, etc. That is, all programs that open many files and/or large files and keep them open.

How to combine lsof search terms (logical AND and OR)

Let's list the files that were opened by user mial and which are associated with systemd processes. We know we can provide more than one search item on the command line, so this should be easy.

sudo lsof -u mial -c systemd

Now let's take a look at the output of lsof. It doesn't look right; the output contains entries that were started by the root user.

This is not what we expected. What happened?

If you provide multiple search terms, lsof will return any file that matches the first search term or the second search term, and so on. In other words, it performs an OR lookup.

To have lsof perform an AND search, use the -a option. This means that only those files will be listed that match the first search query, the second search query, and so on.

Let's try this again and use the -a option.

sudo lsof -u mial -c systemd -a

Now each file in the list is a file that was opened by or on behalf of mial and associated with systemd.

Note that two conditions, one of which contains a negation (^), are treated as logical AND:

sudo lsof +D /home -u ^mial

But if a logical AND is required, then it is better to make it a rule to always use the -a option, since this makes the command more unambiguous:

sudo lsof +D /home -u ^mial -a

and in the case of adding a third search term, it will work as expected:

sudo lsof -u mial +D /home/mial -c^chromium -a

Updating lsof output automatically

To put lsof into repeat mode, we can use the +r SECONDS option or its -r SECONDS option. The repetition option can be applied in two ways: +r or -r. We also need to add the number of seconds that we want lsof to wait before updating the display.

Using the repeat option in any format causes lsof to display the results as usual, but adds a dotted line at the bottom of the screen. The program waits for the number of seconds specified on the command line and then updates the display with a new set of results.

With the -r option, this will continue until you press Ctrl+c. In +r format, the program will continue until an empty result is received, or until you press Ctrl+c.

sudo lsof -u mial -c ssh -a -r5

Notice the dotted line (=======) at the bottom of the list. It decouples each new display of data when the output is refreshed.

Displaying files associated with Internet connections

The -i option allows you to view files opened by processes associated with network and Internet connections.

sudo lsof -i

All files related to network and Internet connections are displayed.

If you don't want port numbers to be replaced with names (for example, http instead of 80), then add the -P option:

sudo lsof -i -P

Display files associated with Internet connections by process ID

To see files opened by Internet connections that are associated with a specific process ID, add the -p PID parameter and the -a parameter.

Here we are looking for files opened over the Internet or a network connection using process ID 606.

sudo lsof -i -a -p 606

Displaying files related to Internet connections and commands

We can use the -c COMMAND option to find files opened by specific processes. To find files that were opened over the Internet or network connections associated with the ssh process, use the following command:

lsof -i -a -c ssh

Displaying files related to Internet connections and ports

We can make lsof report on files that were opened over the Internet or network connections on a specific port. To do this, we use the symbol “:” followed by the port number.

Here we ask lsof to list files that were opened by a network or internet connection using port 22.

lsof -i :22

How to show sockets only for a specific protocol

We can ask lsof to show files that have been opened by processes associated with network and Internet connections that use a particular protocol. We can choose from TCP, UDP and SMTP. Let's use the TCP protocol and see what we get. If you are only interested in network connections on the TCP protocol or UDP protocol, then you can use the following commands:

lsof -i TCP
# or:
lsof -i UDP

Protocols can be written in both uppercase and lowercase letters:

sudo lsof -i tcp

How to find out which process or service is listening on a port

To find out which program is listening on a particular port, run a command like this:

lsof -i :PORT

For example, to see the process that opened port 80:

lsof -i :80

If you want to check the busyness of a port on a specific protocol (TCP or UDP), then specify the protocol of interest before the port number:

lsof -i TCP:22
# or:
lsof -i UDP:53

How to view only IPv4 or IPv6 connections

IPv6 is the sixth and newest version of the IP protocol.

To list open files of network connections for IPv4 only, run the command:

sudo lsof -i 4

To show only open files of type IPv6 run:

sudo lsof -i 6

How to view network connections in a port range

If you need to see all running processes that opened TCP port files in the range 1-1024, then run the command:

lsof -i TCP:1-1024

In addition to the range of ports separated by a hyphen, you can specify several ports or their ranges separated by commas:

lsof -i TCP:1-1024,8000-9000,10000

How to see the network connections of a specific host

Each network connection has a source node and a destination, to see connections that refer only to a specific host, run a command like this:

lsof -i @HOST

For example, I want to view only network connections for a local machine named HackWare, then the command is as follows:

lsof -i @HackWare

If I want to see connections to the suip.biz site, then the command is:

lsof -i @suip.biz

As a host, you can specify not only the names of sites or hosts, but also the IP address.

Combination of network connection filters

You can use all filters at once in a command of the form:

lsof -i PROTOCOL@HOST:PORT

Example:

lsof -i TCP@suip.biz:1-4000

How to stop all activity of a specific user

Sometimes you need to kill all processes of a specific user. Below is the command that will close all processes of the user mial:

sudo kill -9 `lsof -t -u mial`

Search using regular expressions of programs that opened files

You can use regular expressions in lsof filters.

The following example will find all files opened by a process whose name consists of exactly four characters and the third character in the name is the letter “o”.

lsof -c /^..o.$/i

This example will find all files opened by a process whose name consists of exactly four characters:

lsof -c /^....$/i

lsof command basics

We have covered only the most popular examples of using the lsof command. You can find a huge documentation on this program with a description of all options in its man page:

man lsof

File attributes on Linux

File attributes on Linux file systems

Each file has a specific set of properties in the file system. For example, these are access permissions, owner, name, timestamps. In Linux, each file has quite a few properties, for example, access permissions are set three times (for the owner, group and all others), timestamps can also be of three different types (creation time, change and access time).

See also: How to see and change timestamps in Linux. How to perform timestamps-based searching

Some of the properties of files in the current directory can be viewed with the command:

ls -l

An example of the properties of one of the files:

-rw-rw-r-- 1 mial users 262144 авг 18 15:04 custom-x.cramfs.img

However, file properties should not be confused with metadata. Metadata is information that is stored in the file itself, regardless of the file system. And the file properties are specific to the file system and can be lost, for example, when transferring a file from the EXT4 file system to NTFS, some file properties (for example, access rights or timestamps) will be lost because the NTFS file system does not support them.

See also:

Linux users are usually aware of file access modes. But files and directories can be set to attributes that are not remembered by all users. This article is devoted to the file attributes, as well as utilities for setting and reading file attributes.

File attributes can be used by administrators and users to protect files from accidental deletions and changes, and they can also be used by attackers to make it impossible to delete a malicious file.

List of file attributes in Linux

There are the following types of extended attributes.

a

A file with the 'a' attribute set can only be opened in append mode for writing. Only the superuser or a process possessing the CAP_LINUX_IMMUTABLE capability can set or clear this attribute.

A

When a file with the 'A' attribute set is accessed, its atime record is not modified. This avoids a certain amount of disk I/O for laptop systems.

c

A file with the 'c' attribute set is automatically compressed on the disk by the kernel. A read from this file returns uncompressed data. A write to this file compresses data before storing them on the disk. Note: please make sure to read the bugs and limitations section at the end of this document. (Note: For btrfs, If the 'c' flag is set, then the 'C' flag cannot be set. Also conflicts with btrfs mount option 'nodatasum')

C

A file with the 'C' attribute set will not be subject to copy-on-write updates. This flag is only supported on file systems which perform copy-on-write. (Note: For btrfs, the 'C' flag should be set on new or empty files. If it is set on a file which already has data blocks, it is undefined when the blocks assigned to the file will be fully stable. If the 'C' flag is set on a directory, it will have no effect on the directory, but new files created in that directory will have the No_COW attribute set. If the 'C' flag is set, then the 'c' flag cannot be set.)

d

A file with the 'd' attribute set is not a candidate for backup when the dump program is run.

D

When a directory with the 'D' attribute set is modified, the changes are written synchronously to the disk; this is equivalent to the 'dirsync' mount option applied to a subset of the files.

e

The 'e' attribute indicates that the file is using extents for mapping the blocks on disk. It may not be removed using chattr.

E

A file, directory, or symlink with the 'E' attribute set is encrypted by the file system. This attribute may not be set or cleared using chattr, although it can be displayed by lsattr.

F

A directory with the 'F' attribute set indicates that all the path lookups inside that directory are made in a case-insensitive fashion. This attribute can only be changed in empty directories on file systems with the casefold feature enabled.

i

A file with the 'i' attribute cannot be modified: it cannot be deleted or renamed, no link can be created to this file, most of the file's metadata can not be modified, and the file can not be opened in write mode. Only the superuser or a process possessing the CAP_LINUX_IMMUTABLE capability can set or clear this attribute.

I

The 'I' attribute is used by the htree code to indicate that a directory is being indexed using hashed trees. It may not be set or cleared using chattr, although it can be displayed by lsattr.

j

A file with the 'j' attribute has all of its data written to the ext3 or ext4 journal before being written to the file itself, if the file system is mounted with the "data=ordered" or "data=writeback" options and the file system has a journal. When the file system is mounted with the "data=journal" option all file data is already journalled and this attribute has no effect. Only the superuser or a process possessing the CAP_SYS_RESOURCE capability can set or clear this attribute.

m

A file with the 'm' attribute is excluded from compression on file systems that support per-file compression.

N

A file with the 'N' attribute set indicates that the file has data stored inline, within the inode itself. It may not be set or cleared using chattr, although it can be displayed by lsattr.

P

A directory with the 'P' attribute set will enforce a hierarchical structure for project id's. This means that files and directories created in the directory will inherit the project id of the directory, rename operations are constrained so when a file or directory is moved into another directory, that the project ids must match. In addition, a hard link to file can only be created when the project id for the file and the destination directory match.

s

When a file with the 's' attribute set is deleted, its blocks are zeroed and written back to the disk. Note: please make sure to read the bugs and limitations section at the end of this document.

S

When a file with the 'S' attribute set is modified, the changes are written synchronously to the disk; this is equivalent to the 'sync' mount option applied to a subset of the files.

t

A file with the 't' attribute will not have a partial block fragment at the end of the file merged with other files (for those file systems which support tail-merging). This is necessary for applications such as LILO which read the file system directly, and which don't understand tail-merged files. Note: As of this writing, the ext2, ext3, and ext4 file systems do not support tail-merging.

T

A directory with the 'T' attribute will be deemed to be the top of directory hierarchies for the purposes of the Orlov block allocator. This is a hint to the block allocator used by ext3 and ext4 that the subdirectories under this directory are not related, and thus should be spread apart for allocation purposes. For example it is a very good idea to set the 'T' attribute on the /home directory, so that /home/john and /home/mary are placed into separate block groups. For directories where this attribute is not set, the Orlov block allocator will try to group subdirectories closer together where possible.

u

When a file with the 'u' attribute set is deleted, its contents are saved. This allows the user to ask for its undeletion. Note: please make sure to read the bugs and limitations section at the end of this document.

x

The 'x' attribute can be set on a directory or file. If the attribute is set on an existing directory, it will be inherited by all files and subdirectories that are subsequently created in the directory. If an existing directory has contained some files and subdirectories, modifying the attribute on the parent directory doesn't change the attributes on these files and subdirectories.

V

A file with the 'V' attribute set has fs-verity enabled. It cannot be written to, and the file system will automatically verify all data read from it against a cryptographic hash that covers the entire file's contents, e.g. via a Merkle tree. This makes it possible to efficiently authenticate the file. This attribute may not be set or cleared using chattr, although it can be displayed by lsattr.

You can also remember the sticky bit, the essence of which is that a file with a given bit can only be deleted by the user who owns the file. The sticky bit is set using the chmod program:

chmod +t test.txt

But the sticky bit has nothing to do with the file attributes in question.

Bugs and limitations

The 'c', 's', and 'u' attributes are not honored by the ext2, ext3, and ext4 file systems as implemented in the current mainline Linux kernels. Setting 'a' and 'i' attributes will not affect the ability to write to already existing file descriptors.

The 'j' option is only useful for ext3 and ext4 file systems.

The 'D' option is only useful on Linux kernel 2.5.19 and later.

chattr is a program for setting and changing file attributes

chattr changes the file attributes on a Linux file system.

The format of a symbolic mode is +-=[aAcCdDeFijmPsStTux].

The operator '+' causes the selected attributes to be added to the existing attributes of the files; '-' causes them to be removed; and '=' causes them to be the only attributes that the files have.

The letters 'aAcCdDeFijmPsStTux' select the new attributes for the files:

  • append only (a),
  • no atime updates (A),
  • compressed (c),
  • no copy on write (C),
  • no dump (d),
  • synchronous directory updates (D),
  • extent format (e),
  • case-insensitive directory lookups (F),
  • immutable (i),
  • data journaling (j),
  • don't compress (m),
  • project hierarchy (P),
  • secure deletion (s),
  • synchronous updates (S),
  • no tail-merging (t),
  • top of directory hierarchy (T),
  • undeletable (u),
  • and direct access for files (x).

The following attributes are read-only, and may be listed by lsattr but not modified by chattr:

  • encrypted (E),
  • indexed directory (I),
  • inline data (N),
  • and verity (V).

Not all flags are supported or utilized by all file systems; refer to file system-specific man pages such as btrfs, ext4, and xfs for more file system-specific details.

The following command will make the test.txt file locked for deletion, modification, move (i):

sudo chattr +i test.txt

Moreover, even adding sudo to the delete and move commands will not help, this file will not be able to change or delete until the “i” attribute is removed.

The following command removes the “i” attribute:

sudo chattr -i test.txt

If you selected directories as the target for modifying parameters, the -R option will cause the attributes to be recursively changed for the contents of the directories as well.

chattr is part of the e2fsprogs package and is available at http://e2fsprogs.sourceforge.net, although most Linux distributions have this package preinstalled by default.

lsattr is a program to list file attributes in the Linux filesystem

lsattr lists the file attributes on a second extended file system.

To view file attributes, specify the file name:

lsattr test.txt

Options:

-R

Recursively list attributes of directories and their contents.

-a

List all files in directories, including files that start with “.”.

-d

List directories like other files, rather than listing their contents.

-l

Print the options using long names instead of single character abbreviations.

-p

List the file's project number.

-v

List the file's version/generation number.

lsattr is part of the e2fsprogs package and is available at http://e2fsprogs.sourceforge.net, although most Linux distributions have this package preinstalled by default.

An unplugged disk causes a delay in system boot

The fstab file contains descriptive information about the file systems that the system can mount. In the /etc/fstab file, you can specify the parameters for mounting a disk, which can be either plugged to the system or unplugged from it. An example of such a disk is an external USB drive. In the latter case, you must additionally specify the nofail option.

An example of such an entry:

UUID=26FC3023FC2FEC2D	/mnt/disk_e	ntfs	nofail,rw,utf8	0	0

Thanks to the nofail option, the system boots normally and shows no errors even if the specified disk is missing.

But starting with systemd 249-3, the system began to try to mount this disk even if it was not connected. As a result, when the computer was turned on, it waited for the timeout of this operation to complete, that is, the boot freeze for 90 seconds (the default timeout).

The following messages appeared in the system log:

HackWare systemd[1]: dev-disk-by\x2duuid-26FC3023FC2FEC2D.device: Job dev-disk-by\x2duuid-26FC3023FC2FEC2D.device/start timed out.
HackWare systemd[1]: Timed out waiting for device /dev/disk/by-uuid/26FC3023FC2FEC2D.
HackWare systemd[1]: dev-disk-by\x2duuid-26FC3023FC2FEC2D.device: Job dev-disk-by\x2duuid-26FC3023FC2FEC2D.device/start failed with result 'timeout'.

Most likely, this problem is a temporary bug that will be fixed in future versions of systemd. But now you can use the x-systemd.device-timeout mount option so that the computer does not wait a minute and a half.

In the following example, the timeout value is set to 1 second:

 

UUID=26FC3023FC2FEC2D	/mnt/disk_e	ntfs	nofail,rw,utf8,x-systemd.device-timeout=1	0	0

As a result, the computer began to turn on almost immediately, without delay.

If the specified disk is connected to the computer after turning on, then the disk will be mounted correctly, taking into account the options from the /etc/fstab file. The minimum timeout time is not an obstacle for mounting after turning on the computer.

If the disk is usually unplugged, then the noauto option is more appropriate:

UUID=26FC3023FC2FEC2D	/mnt/disk_e	ntfs	noauto,nofail,rw,utf8	0	0

This option makes it so that the system will not try to mount the disk when you turn on the computer. However, the next time you connect the disk, it will be mounted according to the options in the /etc/fstab file.

How to find out when Linux was installed

This article will look at several ways to find out the date of installation of the Linux operating system. But let's start by thinking about what is considered the date of installation of the OS?

What is considered the date of installation of Linux

It would seem that the question and the answer to it are quite simple – the installation date is the day when the operating system files were copied from the installation disk or flash drive.

But if a major OS update was made, that is, a transition between major versions, can this be considered the date of installation of the current OS? This update can be done either by updating packages or using the installation media (for example, Tails is updated only this way).

If we take the age of the oldest file in the OS for the installation date, then you may encounter the following situation: the user's home folder, created and used much earlier with other (previous versions) of the OS, is mounted on the freshly installed system, - whether to consider the installation date according to the oldest part of the OS (home user folder) or by the date the files of the new OS were copied?

It can be assumed that the key element is the root file system and you need to look at the date of its creation. In general, usually many agree that it is the creation date of the root file system that should be considered the date of the OS installation, but the following situations are possible:

  • the file system was changed or converted after the OS was installed
  • the cloned file system may have been migrated to another disk

In general, there are nuances that you need to remember, but they do not apply to most users.

1. Date of creation of the file system

To find out the creation date of the file system, run the command:

tune2fs -l `df / | grep '/' | awk '{ print $1 }'` | grep 'Filesystem created:'

Output example:

Filesystem created:       Wed Aug  1 19:40:20 2018

That is, the operating system was installed on August 1, 2018.

Another command that will show the creation date of the file system in Linux:

dumpe2fs $(mount | grep 'on \/ ' | awk '{print $1}') | grep 'Filesystem created:'

It is also recommended to read the article “How to know when the ext4/ext3/ext2 filesystem was created and when it was last mounted”, in it you will learn how to find out how many times the file system was mounted, how much data was written to it , the date of the last entry and other interesting information.

2. Age of the oldest files in the OS

The most file system and distribution neutral solution is to use the oldest file found with the “ls -lact /etc” command, which checks the metadata of each file at the time of creation. This method is not affected by the touch command or files created by extracting archives (eg tar -p to preserve timestamps).

I think it is better to look at files rather than directories, since directories change their creation time metadata when their contents change (maybe someone can shed some light on why this is so?)

ls -lact --full-time /etc | tail

The results of this method are consistent with previous data – August 1, 2018.

If you only want to get the creation time of the oldest file in /etc:

ls -lact --full-time /etc | awk 'END {print $6,$7,$8}'

How to find out which process is modifying a file

Linux has an auditing platform that lets you know when files or/and folders is being accessed, modified, or executed. You can also monitor changes in entire directories.

How to install auditd (auditctl)

On Debian, Linux Mint, Kali Linux, Ubuntu and their derivatives, to install, run the command:

sudo apt install auditd

In Arch Linux, Manjaro, BlackArch and their derivatives, this package is called audit and is included in the core repository, therefore, it is preinstalled by default.

On CentOS to install, run the command:

yum install audit

How to start the file access and change monitor

You need to start by adding rules. The following command adds an access and change monitor to the /etc/resolv.conf file:

sudo auditctl -w /etc/resolv.conf -p wa

This is an example of a command with a different notation, but it performs the same action – it monitors all changes and access to the /etc/resolv.conf file:

sudo auditctl -a always,exit -F path=/etc/resolv.conf -F perm=wa

You can check which rules have been added with the following command:

sudo auditctl -l

Although the rule has been added, the audit service has not been started yet. To run it, run the command:

sudo systemctl start auditd.service

If you want to add this service to startup, then run:

sudo systemctl enable auditd.service

How to leave the audit daemon in the foreground

The previous command will start auditd as a daemon, that is, a service in the background. If you don't need this and want to run auditd in the foreground, then instead of using systemctl, run the following command:

sudo auditd -f

Option -f leaves the audit daemon in the foreground for debugging. Messages also go to stderr rather than the audit log.

In this case, all events with tracked files or folders will be displayed on standard output. In this case, the log file will not be kept.

This is useful when debugging rules, or if you need to track events in a short amount of time.

How to view the auditd log

The auditd log is stored in the /var/log/audit/audit.log file. But instead of viewing it directly, you can use the ausearch utility, for example:

sudo ausearch -f /etc/resolv.conf

If the output is

<no matches>

then this file has not yet been touched by any program.

If the events happened, there will be approximately the following entries:

----
time->Sun May 16 07:02:16 2021
type=PROCTITLE msg=audit(1621137736.023:543): proctitle=6765646974002F6574632F7265736F6C762E636F6E66
type=PATH msg=audit(1621137736.023:543): item=1 name="/etc/resolv.conf" inode=160660 dev=103:02 mode=0100644 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(1621137736.023:543): item=0 name="/etc/" inode=131073 dev=103:02 mode=040755 ouid=0 ogid=0 rdev=00:00 nametype=PARENT cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=CWD msg=audit(1621137736.023:543): cwd="/home/mial"
type=SYSCALL msg=audit(1621137736.023:543): arch=c000003e syscall=257 success=no exit=-17 a0=ffffff9c a1=55da2dd00770 a2=800c1 a3=1b6 items=2 ppid=78750 pid=78751 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts1 ses=2 comm="pool-gedit" exe="/usr/bin/gedit" key=(null)
----
time->Sun May 16 07:02:16 2021
type=PROCTITLE msg=audit(1621137736.023:544): proctitle=6765646974002F6574632F7265736F6C762E636F6E66
type=PATH msg=audit(1621137736.023:544): item=1 name="/etc/resolv.conf" inode=160660 dev=103:02 mode=0100644 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(1621137736.023:544): item=0 name="/etc/" inode=131073 dev=103:02 mode=040755 ouid=0 ogid=0 rdev=00:00 nametype=PARENT cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=CWD msg=audit(1621137736.023:544): cwd="/home/mial"
type=SYSCALL msg=audit(1621137736.023:544): arch=c000003e syscall=257 success=yes exit=11 a0=ffffff9c a1=55da2dd00770 a2=20041 a3=1b6 items=2 ppid=78750 pid=78751 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts1 ses=2 comm="pool-gedit" exe="/usr/bin/gedit" key=(null)
----
time->Sun May 16 07:02:16 2021
type=PROCTITLE msg=audit(1621137736.029:545): proctitle=6765646974002F6574632F7265736F6C762E636F6E66
type=PATH msg=audit(1621137736.029:545): item=4 name="/etc/resolv.conf" inode=163335 dev=103:02 mode=0100644 ouid=0 ogid=0 rdev=00:00 nametype=CREATE cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(1621137736.029:545): item=3 name="/etc/resolv.conf" inode=160660 dev=103:02 mode=0100644 ouid=0 ogid=0 rdev=00:00 nametype=DELETE cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(1621137736.029:545): item=2 name="/etc/.goutputstream-VQ4G30" inode=163335 dev=103:02 mode=0100644 ouid=0 ogid=0 rdev=00:00 nametype=DELETE cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(1621137736.029:545): item=1 name="/etc/" inode=131073 dev=103:02 mode=040755 ouid=0 ogid=0 rdev=00:00 nametype=PARENT cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(1621137736.029:545): item=0 name="/etc/" inode=131073 dev=103:02 mode=040755 ouid=0 ogid=0 rdev=00:00 nametype=PARENT cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=CWD msg=audit(1621137736.029:545): cwd="/home/mial"
type=SYSCALL msg=audit(1621137736.029:545): arch=c000003e syscall=82 success=yes exit=0 a0=7ff2a409a5a0 a1=7ff2a40a2da0 a2=0 a3=11 items=5 ppid=78750 pid=78751 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts1 ses=2 comm="pool-gedit" exe="/usr/bin/gedit" key=(null)

See the “exe=” line to find out which program performed the action.

How to stop auditd service

To remove a service from startup, run the command:

sudo systemctl disable auditd.service

If you try to stop the service with the following command:

sudo systemctl stop auditd.service

Then you will receive a message that it failed because the operation was rejected:

Failed to stop auditd.service: Operation refused, unit auditd.service may be requested by dependency only (it is configured to refuse manual start/stop).
See system logs and 'systemctl status auditd.service' for details.

To stop the service, run the command:

sudo auditctl --signal TERM

How to remove all change tracking rules for folders and files

To remove all rules at once, run the command:

sudo auditctl -D

It is possible to delete individual rules (both by the tracked event and by the associated identifier).

Error opening /var/log/audit/audit.log (No such file or directory)

If you get an error

Error opening /var/log/audit/audit.log (No such file or directory)

It means that the audit service was not started (you forgot to start it, it did not start due to an error, or you started it in the foreground).

Auditd Configuration Examples

To view all system calls made by a specific program:

sudo auditctl -a always,exit -S all -F pid=1005

To see files opened by a specific user:

sudo auditctl -a always,exit -S openat -F auid=510

To see failed openat calls:

sudo auditctl -a always,exit -S openat -F success=0

To track file changes (two ways of expression):

sudo auditctl -w /etc/shadow -p wa
sudo auditctl -a always,exit -F path=/etc/shadow -F perm=wa

To recursively monitor a directory for changes (two ways of expression):

sudo auditctl -w /etc/ -p wa
sudo auditctl -a always,exit -F dir=/etc/ -F perm=wa

To see if an administrator has access to a user's files:

sudo auditctl -a always,exit -F dir=/home/ -F uid=0 -C auid!=obj_uid

Auditd files

  • /etc/audit/auditd.conf – configuration file for audit daemon
  • /etc/audit/audit.rules – audit rules to be loaded at startup
  • /etc/audit/rules.d/ – directory holding individual sets of rules to be compiled into one file by augenrules.
  • /etc/audit/plugins.d/ – directory holding individual plugin configuration files.
  • /var/run/auditd.state – report about internal state.

Auditd documentation

This article shows you how to get started using auditd to track file changes and track file access.

The capabilities of auditd are not limited to the examples shown, and there are several utilities with many settings and options that allow you to very flexibly configure the rules for monitoring what is happening in the file system, as well as perform other related actions.

Using man, you can check out the following documentation:

  • auditd.conf
  • auditd-plugins
  • ausearch
  • aureport
  • auditctl
  • augenrules
  • audit.rules
Loading...
X