Tag: file systems

How to find out which process is modifying a file

Posted by Alex On May 16, 2021

Linux has an auditing platform that lets you know when files or/and folders is being accessed, modified, or executed. You can also monitor changes in entire directories.

How to install auditd (auditctl)

On Debian, Linux Mint, Kali Linux, Ubuntu and their derivatives, to install, run the command:

sudo apt install auditd

In Arch Linux, Manjaro, BlackArch and their derivatives, this package is called audit and is included in the core repository, therefore, it is preinstalled by default.

On CentOS to install, run the command:

yum install audit

How to start the file access and change monitor

You need to start by adding rules. The following command adds an access and change monitor to the /etc/resolv.conf file:

sudo auditctl -w /etc/resolv.conf -p wa

This is an example of a command with a different notation, but it performs the same action – it monitors all changes and access to the /etc/resolv.conf file:

sudo auditctl -a always,exit -F path=/etc/resolv.conf -F perm=wa

You can check which rules have been added with the following command:

sudo auditctl -l

Although the rule has been added, the audit service has not been started yet. To run it, run the command:

sudo systemctl start auditd.service

If you want to add this service to startup, then run:

sudo systemctl enable auditd.service

How to leave the audit daemon in the foreground

The previous command will start auditd as a daemon, that is, a service in the background. If you don't need this and want to run auditd in the foreground, then instead of using systemctl, run the following command:

sudo auditd -f

Option -f leaves the audit daemon in the foreground for debugging. Messages also go to stderr rather than the audit log.

In this case, all events with tracked files or folders will be displayed on standard output. In this case, the log file will not be kept.

This is useful when debugging rules, or if you need to track events in a short amount of time.

How to view the auditd log

The auditd log is stored in the /var/log/audit/audit.log file. But instead of viewing it directly, you can use the ausearch utility, for example:

sudo ausearch -f /etc/resolv.conf

If the output is

<no matches>

then this file has not yet been touched by any program.

If the events happened, there will be approximately the following entries:

----
time->Sun May 16 07:02:16 2021
type=PROCTITLE msg=audit(1621137736.023:543): proctitle=6765646974002F6574632F7265736F6C762E636F6E66
type=PATH msg=audit(1621137736.023:543): item=1 name="/etc/resolv.conf" inode=160660 dev=103:02 mode=0100644 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(1621137736.023:543): item=0 name="/etc/" inode=131073 dev=103:02 mode=040755 ouid=0 ogid=0 rdev=00:00 nametype=PARENT cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=CWD msg=audit(1621137736.023:543): cwd="/home/mial"
type=SYSCALL msg=audit(1621137736.023:543): arch=c000003e syscall=257 success=no exit=-17 a0=ffffff9c a1=55da2dd00770 a2=800c1 a3=1b6 items=2 ppid=78750 pid=78751 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts1 ses=2 comm="pool-gedit" exe="/usr/bin/gedit" key=(null)
----
time->Sun May 16 07:02:16 2021
type=PROCTITLE msg=audit(1621137736.023:544): proctitle=6765646974002F6574632F7265736F6C762E636F6E66
type=PATH msg=audit(1621137736.023:544): item=1 name="/etc/resolv.conf" inode=160660 dev=103:02 mode=0100644 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(1621137736.023:544): item=0 name="/etc/" inode=131073 dev=103:02 mode=040755 ouid=0 ogid=0 rdev=00:00 nametype=PARENT cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=CWD msg=audit(1621137736.023:544): cwd="/home/mial"
type=SYSCALL msg=audit(1621137736.023:544): arch=c000003e syscall=257 success=yes exit=11 a0=ffffff9c a1=55da2dd00770 a2=20041 a3=1b6 items=2 ppid=78750 pid=78751 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts1 ses=2 comm="pool-gedit" exe="/usr/bin/gedit" key=(null)
----
time->Sun May 16 07:02:16 2021
type=PROCTITLE msg=audit(1621137736.029:545): proctitle=6765646974002F6574632F7265736F6C762E636F6E66
type=PATH msg=audit(1621137736.029:545): item=4 name="/etc/resolv.conf" inode=163335 dev=103:02 mode=0100644 ouid=0 ogid=0 rdev=00:00 nametype=CREATE cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(1621137736.029:545): item=3 name="/etc/resolv.conf" inode=160660 dev=103:02 mode=0100644 ouid=0 ogid=0 rdev=00:00 nametype=DELETE cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(1621137736.029:545): item=2 name="/etc/.goutputstream-VQ4G30" inode=163335 dev=103:02 mode=0100644 ouid=0 ogid=0 rdev=00:00 nametype=DELETE cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(1621137736.029:545): item=1 name="/etc/" inode=131073 dev=103:02 mode=040755 ouid=0 ogid=0 rdev=00:00 nametype=PARENT cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(1621137736.029:545): item=0 name="/etc/" inode=131073 dev=103:02 mode=040755 ouid=0 ogid=0 rdev=00:00 nametype=PARENT cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=CWD msg=audit(1621137736.029:545): cwd="/home/mial"
type=SYSCALL msg=audit(1621137736.029:545): arch=c000003e syscall=82 success=yes exit=0 a0=7ff2a409a5a0 a1=7ff2a40a2da0 a2=0 a3=11 items=5 ppid=78750 pid=78751 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts1 ses=2 comm="pool-gedit" exe="/usr/bin/gedit" key=(null)

See the “exe=” line to find out which program performed the action.

How to stop auditd service

To remove a service from startup, run the command:

sudo systemctl disable auditd.service

If you try to stop the service with the following command:

sudo systemctl stop auditd.service

Then you will receive a message that it failed because the operation was rejected:

Failed to stop auditd.service: Operation refused, unit auditd.service may be requested by dependency only (it is configured to refuse manual start/stop).
See system logs and 'systemctl status auditd.service' for details.

To stop the service, run the command:

sudo auditctl --signal TERM

How to remove all change tracking rules for folders and files

To remove all rules at once, run the command:

sudo auditctl -D

It is possible to delete individual rules (both by the tracked event and by the associated identifier).

Error opening /var/log/audit/audit.log (No such file or directory)

If you get an error

Error opening /var/log/audit/audit.log (No such file or directory)

It means that the audit service was not started (you forgot to start it, it did not start due to an error, or you started it in the foreground).

Auditd Configuration Examples

To view all system calls made by a specific program:

sudo auditctl -a always,exit -S all -F pid=1005

To see files opened by a specific user:

sudo auditctl -a always,exit -S openat -F auid=510

To see failed openat calls:

sudo auditctl -a always,exit -S openat -F success=0

To track file changes (two ways of expression):

sudo auditctl -w /etc/shadow -p wa
sudo auditctl -a always,exit -F path=/etc/shadow -F perm=wa

To recursively monitor a directory for changes (two ways of expression):

sudo auditctl -w /etc/ -p wa
sudo auditctl -a always,exit -F dir=/etc/ -F perm=wa

To see if an administrator has access to a user's files:

sudo auditctl -a always,exit -F dir=/home/ -F uid=0 -C auid!=obj_uid

Auditd files

/etc/audit/auditd.conf – configuration file for audit daemon
/etc/audit/audit.rules – audit rules to be loaded at startup
/etc/audit/rules.d/ – directory holding individual sets of rules to be compiled into one file by augenrules.
/etc/audit/plugins.d/ – directory holding individual plugin configuration files.
/var/run/auditd.state – report about internal state.

Auditd documentation

This article shows you how to get started using auditd to track file changes and track file access.

The capabilities of auditd are not limited to the examples shown, and there are several utilities with many settings and options that allow you to very flexibly configure the rules for monitoring what is happening in the file system, as well as perform other related actions.

Using man, you can check out the following documentation:

auditd.conf
auditd-plugins
ausearch
aureport
auditctl
augenrules
audit.rules

How to open a Linux drive on Windows

Posted by Alex On March 11, 2021

The Linux operating system uses its own file systems. Unlike Windows, for which the usual file systems are NTFS, FAT32, and exFAT, the popular file systems on Linux are ext4, ext3, and ext2.

If you connect a Linux drive to Windows, or if you have Linux installed as a second system, then Windows simply will not be able to open the contents of the ext4, ext3 and ext2 drives. It is possible to install additional drivers for EXT, but there is a much easier way.

If you need to view or copy files from Linux disks from Windows, then this instruction will help you.

Start by installing the 7-Zip program, as described in detail in the article “Free alternative to WinRAR”.

Easy way to open Linux drives (ext4, ext3 and ext2) in Windows

Open 7-Zip by entering “7-Zip” in the system-wide search and launch the program.

Now use the “Up” icon to go to the very top of the file system hierarchy until you see the “\\.” Symbol.

Click on “\\.” and you will be taken to the list of physical disks.

To figure out which drive you want, use their size or just navigate to each one to find a Linux drive.

An example of a Linux disk with two partitions: 0.img is the boot partition and 1.img is the main data partition.

An example of the contents of the 0.img section - it contains the files necessary for loading, namely UEFI.

It takes a certain amount of time to enter the section. A 100 GB partition took about a minute to parse the data and list the files, a 1 TB partition took less than three minutes to parse.

When “unpacking” is complete, a list of files in the root file system will be shown.

Navigating through files and even launching files, including large ones, is performed without delay.

You can navigate to any folder and open any file - Linux file and access restrictions do not work.

You can view absolutely any files, even those to which users usually never have access rights.

7-Zip allows you to open Linux partitions directly from Windows without installing additional drivers. You can view, save files and folders, but you cannot make changes to the file system. That is, you won't mess up anything on Linux disks.

See also another option how to copy files from a Linux disk in the article “How in Windows view the contents of a Linux disk and copy files from it”.

Instructions for using the file command

Posted by Alex On February 20, 2021

How to find out the type of a file without an extension or with the wrong extension

The file command can determine the file type for regular files and the file system used for disks.

The file utility uses magic bytes. The bottom line is that files of the same type have the same bytes in certain places in the file, the file program uses an extensive database in which the types of files are described and which sequences of bytes are located in which places. Thanks to this, using file utility, you can determine the type of files without an extension or with the wrong extension.

Usage is very simple - specify one or more file names, that is:

file FILE

file FILE1 FILE2 FILE3 FILE4

To find out the type of a file without an extension named “file1”, run the command:

file file1

Output example:

file1: JPEG image data, Exif standard: [TIFF image data, little-endian, direntries=11, manufacturer=samsung, model=SM-J710F, orientation=upper-left, xresolution=164, yresolution=172, resolutionunit=2, software=J710FXXU6CSE1, datetime=2019:11:10 16:31:24, GPS-Data], baseline, precision 8, 4128x3096, components 3

First there is the file name, and then information about it is shown separated by colons. In this example, it is JPEG, that is, an image. Also, meta information is partially shown for this file.

If you don't want the filename to be shown first, use the -b option:

file -b FILE

How to check the type of multiple files at once

As you can see in the screenshot, there are several files in the current folder without extensions. Let's find out the type of each of them.

You can specify the names of all files or use * (asterisk) to check all files in the current directory:

file *

We managed to determine the type of each file:

JPEG image
Python script
PDF document
PE32 executable (console) Intel 80386 Mono/.Net assembly, for MS Windows - executable file for MS OS
7-zip archive
Bourne-Again shell script - Bash script
DOS/MBR boot sector, code offset 0x52+2, OEM-ID "NTFS - disk image with NTFS file system
Microsoft Word 2007+ - Office Word Document
OpenDocument Text - LibreOffice office document

How to find the extension by file type

To find out the correct extension for a specific file type, use the --extension option:

For instance:

file --extension *

How to make sure that the access date of a file does not change when using file

The file program reads data from a file and therefore changes the date it was last accessed. The -p, --preserve-date option on systems that support utime or utimes will try to preserve the access time of the parsed files to pretend that the file has never been read.

How to find out the file system of a disk

For file system image files, the file program will show the file system type. For disks, this utility can also detect the file system type, but two additional options are required because the block device file is a special file, and sometimes it is just a link to a special file.

The -L, --dereference option forces symbolic links to be followed, like a similarly named option in ls (on systems that support symbolic links). This is the default if the POSIXLY_CORRECT environment variable is defined.

Typically, file utility only tries to read and determine the type of files passed as arguments, which stat reports as regular files. This prevents problems, as reading special files can have specific consequences. Specifying the -s, --special-files option causes the file to also read files that are special block or character files. This is useful for identifying the file system types of data in raw disk partitions, which are special block files. This option also causes the file to ignore the file size reported by stat, since on some systems it reports zero size for raw disk partitions.

The following information will display information about the file system on the /dev/sda drive:

file -s -L /dev/sda

Output example:

/dev/sda: Linux rev 1.0 ext4 filesystem data, UUID=aad9d8d7-b1b9-435a-9e41-fc8159d2c484 (needs journal recovery) (extents) (64bit) (large files) (huge files)

As the output suggests, this is an ext4 file system. Some of its characteristics and supported features are also shown.

Without the -s option, that is, with the command

file /dev/sda

we would get the following result:

/dev/sda: block special (8/0)

How to look inside compressed files in file

With the -z, --uncompress option, you can try to look inside the compressed files. This option does not always work and sometimes it causes file utility to fail.

The in-memory file system - how to use tmpfs

Posted by Alex On February 19, 2021

The tmpfs file system is quite useful, as it is incredibly fast and can help reduce the load on your persistent storage (especially useful for those who have Linux installed on a flash drive or memory card).

tmpfs is an in-memory virtual file system.

The tmpfs tool allows you to create file systems whose content resides in virtual memory. Because the files on these file systems are usually in RAM, the files are accessed very quickly.

The file system is created automatically when a file system of type tmpfs is mounted using the following command:

sudo mount -t tmpfs -o size=10M tmpfs /mnt/mytmpfs

The tmpfs file system has the following properties:

The file system can use swap when the physical load on memory requires it.
The file system consumes as much physical memory and swap as is required to store the current contents of the file system.
During a remount operation (mount -o remount), the file system can be resized (without losing the existing contents of the file system).

If the tmpfs filesystem is unmounted, its contents are lost (deleted).

You can copy files to tmpfs for the fastest possible access. These can be database files or web server files.

Another use case is to reduce wear and tear on persistent storage. This is not particularly true for a hard drive or solid-state drive - modern models will outlive us for any type of home use. But this may be relevant if the system is installed on a memory card. You can place an application in RAM that constantly uses storage (often accesses files or continuously saves files), thereby speeding up the operation of this application, as well as the entire system by reducing the load on the memory card.

Another possible reason for its use is stealth, when working in tmpfs, everything will happen in RAM, and there will be no traces on persistent storages.

Consider an example of copying files - how much faster will it be in tmpfs compared to hard disks.

Let's create a mount point:

mkdir /tmp/mytmpfs

Let's create a virtual file system of 20 Gigabytes in RAM:

sudo mount -t tmpfs -o size=20g tmpfs /tmp/mytmpfs

Let's copy a file of several Gigabytes there:

cp /mnt/disk_d/Vuse/Space.Cop.2016.L2.BDRip.720p.mkv /tmp/mytmpfs

Let's check how long it will take to create a copy of this file in RAM:

time cp /tmp/mytmpfs/Space.Cop.2016.L2.BDRip.720p.mkv /tmp/mytmpfs/copy.mkv

Result:

real	0m1,403s
user	0m0,020s
sys	0m1,381s

It took quite a bit of time - about one and a half seconds.

Now let's make a copy of the same file on the hard disk:

time cp /mnt/disk_d/Vuse/Space.Cop.2016.L2.BDRip.720p.mkv /mnt/disk_d/Vuse/copy.mkv

Result:

real	0m14,463s
user	0m0,065s
sys	0m4,041s

It took 14 seconds - 10 times the time.

So, using tmpfs, you can achieve the maximum speed of access to files.

Life of a webmaster