How to archive a folder in parts

How to archive a directory in parts in Linux

Let's consider a situation: there is a very large folder (directory) on the server and it needs to be downloaded or moved to another server. In this case, you need to split this directory into several separate archives. Archiving the folder in parts is required, among other reasons, because there is not enough space on the server for one large archive.

That is, the straightforward idea of creating a directory archive is not suitable, since there is not enough space on the server.

Creating a multi-volume archive is not suitable for the same reason – archive volumes are created simultaneously and take up the same amount of space as a single archive.

Is it possible to find a way out of this situation?

Yes, you can archive individual directory folders. Let's look at exactly how to implement this.

How to archive part of the folders of one directory

Consider the following set of commands:

zip -r archive_af FOLDER/[a-f]* > /dev/null
zip -r archive_gl FOLDER/[g-l]* > /dev/null
zip -r archive_ms FOLDER/[m-s]* > /dev/null
zip -r archive_tz FOLDER/[t-z]* > /dev/null
zip -r archive_09 FOLDER/[0-9]* > /dev/null

For example, the first command finds in the FOLDER directory all subfolders whose names begin with letters from the range a-f and archives them into the archive_af.zip archive. Once the archive is created, you can move it (for example, download or upload to a new server), and then move on to creating the next archive.

The second command will create an archive_gl.zip archive, which will contain subfolders whose names begin with letters from the range g-l and which are located in the FOLDER directory.

All these commands will one by one place the entire contents of the FOLDER directory into different archives.

The advantage of this approach is that archives can be created one by one and when downloading the previous archive, the next one can be created. The created files are not parts of a multi-volume archive and can be worked with separately.

Note: You need to replace “FOLDER” with the name of the directory you want to archive in parts.

Note 2: The “> /dev/null” part of the command suppresses the command's extensive output, which lists all the files being archived. If you wish, you can remove this part of the command and monitor the process of creating archives.

How to check that all subfolders are included in the archives

The method shown above assumes that all subfolders have names that begin with letters of the English alphabet or with numbers. You can make sure that there are no folders that do not match this pattern:

ls FOLDER/[^a-z0-9]*

If the previous command showed the presence of subfolders that do not match the template, then you can archive them using the following command:

zip -r archive_the_rest FOLDER/[^a-z0-9]* > /dev/null

Another way to check is to count the number of subfolders. This command will print the number of subfolders without any conditions:

ls FOLDER/* | wc -l

And this command will display the number of subfolders matching the specified pattern:

ls FOLDER/[a-z0-9]* | wc -l

If the numbers are the same, then you can use the above method of archiving a directory in parts without the danger of missing any subfolders.

To find out the size of the “FOLDER” directory, run the following command:

du -sh FOLDER/

How to move large files from one server (or hosting) to another

Instead of first downloading files to your computer and then uploading them to another server or hosting, you can upload them directly to the new server. If your files are located in the web server directory, then they are available for download via URL.

https://your-site.net/archive_af.zip
https://your-site.net/archive_gl.zip
https://your-site.net/archive_ms.zip
https://your-site.net/archive_tz.zip
https://your-site.net/archive_09.zip