RHCSA: Working with Archives and Compressed Files

Extracted from Professor Nikolai Bezroukov unpublished lecture notes.

Copyright 2010-2018, Dr. Nikolai Bezroukov. This is a fragment of the copyrighted unpublished work. All rights reserved.

The most common archive format in Linux is compressed tarball.  To create such an archive you need to know ins and outs of the tar command . This command was originally designed to stream files to a tape.  By default compression is with  gzip. You need to specify bzip2 explicitly.  You should be able perform three important tasks:

• Create an archive
• List the contents of an archive
• Extract an archive
• Compress and uncompress archives

There are four major types are compressed archives in Linux

• gzip (parallel version is pigz) -- the most common type, standard de facto, pigz is provided by CentOS repositories. It is a reimplementation of zip by GNU project (see below). 
• bzip2 (parallel version is pbzip2) -- provides better compression on text files. Slightly slower in non-parallel version.  Parallel version is blazingly fast. It is ideal for long distance transfer via WAN from remote computers as the compression ration is 20-30% higher then with gzip.  Unfortunately pbzip2 is not provided by standard CentOS  or RHEL repositories. You need EPEL repository added to yum or you can download RPMs separately  (How to Enable EPEL Repository )
• xz (parallel version  is 7za) -- provides the best compression for text files. Used by Red Hat for distribution patches. Compression is approximately 100 times slower then gzip is unacceptably slow for large files, unless you use parallel version. Parallel version makes the speed acceptable but that's about it. It never achieve speed comparable with pigz and bzip2. Compression ratio is only 10-20% better then bzip2 and ~30% better then gzip (compressed file is close to half size of gzipped file). Still this is the best compression format for small collections of source code and for the long term storage of data, as it minimizes space and a standardized enough to survive the next 100 years.  7za also can be installed form EPEL repository.
• zip -- the oldest compression program derived from the famous DOS compressor pkzip by Phil Katz. While tar achieves compressed with  gzip are atomic -- you need to uncompress the whole archive in order to get to any single file, zip archive is like virtual filesystem -- all file can be  accesses individually and modified individually as well (updated deleted added) too.  Zip format is standard de-facto for Windows and even is supported by the OS -- you can view zip archives as directories.  

Tar archiver

Tar is  a very old (dates back to Version 6 of AT&T UNIX, circa 1975) zero-compression archiver. But it is not easy to replace it with any archiver that has a zero-compression option (for example zip) as with time it got some unique capabilities and became a standard de-facto for zero compression archiver.

Unlike most archivers, tar can be used as the head or tail of a pipeline. tar  is very convenient for moving hierarchies directories hierarchies while preserving all attributes and symbolic and hard links between servers.

Archive created by tar is usually called tarball.  A tarball historically was a magnetic tape, but now it's usually a disk file. The default device, /dev/rmt0, is seldom used today, the most common is to archive into file that is often processed additionally by gzip, or, for text files,  xz (option -J).

The tar command can specify a list of files or directories and can include name substitution characters. The basic form is

tar  keystring options -f tarball filenames... .

The keystring is a string of characters starting with one function letter (c, r, t  , u, or x) and zero or more function modifiers (letters or digits), depending on the function letter used.  You can also specify all of them as options. tarball is specified after option -f. Omitting -f  is  the most common  mistake for novice users of tar.

tar options files_to_include

Three examples:

• tar -cf archive.tar foo bar # Create archive.tar from files foo and bar.
• tar -tvf archive.tar # List all files in archive.tar verbosely.
• tar -xf archive.tar # Extract all files from archive.tar.

Main options:

1. -A, --catenate, --concatenate append tar files to an archive
2. -c, --create create a new archive
3. -d, --diff, --compare find differences between archive and file system
4. --delete delete from the archive (not on mag tapes!)
5. -r, --append append files to the end of an archive
6. -t, --list list the contents of an archive
7. --test-label test the archive volume label and exit
8. -u, --update only append files newer than copy in archive
9. -x, --extract, --get extract files from an archive
10. -C, --directory=DIR change to directory DIR
11. -f, --file=ARCHIVE use archive file or device ARCHIVE
12. -p, --preserve-permissions extract information about file permissions (default for superuser)
13. -v, --verbose verbosely list files processed

there are also three options that specify compression program used:

1. -z, --gzip filter the archive through gzip
2. -j, --bzip2 filter the archive through bzip2
3. -J, --xz filter the archive through xz

Creating Archives with tar

To create an archive, you use the command.

tar -cvf archivename.tar  directory_or_list_of_files_or_regular_expression

Always use  option -v to see what is happening. To put files in an archive, you need at least read permissions to the file. For example to compress your home directory and pup the resulting archive in  /tmp you can use the command

tar -cvf /tmp/joeuser.tar /home/joeuser

Notice the options that are used; the order in these options is important -- the last one should be option f  because it has parameter which specifies where to put the archive we are creating.

Similarly you can create archive of etc

tar -cvf -C /etc /root/etc181006.tar . 

cd /etc && tar cvf 

TIPS:

• It makes sense to use -C option as in this case you can extract files directly without tar creating some direction as undesirable side effect.
• It is good practice to create archive files with an extension such as .tar or .tgz so that they can be easily recognized, but not everyone is doing that. If you think that a file is a tar archive, but you are not sure, use the file command to check it out

Originally, tar did not use the dash (-) in front of its options. Modern tar implementations use that dash, as do all other Linux programs, but they still allow the old usage without a dash for backward compatibility.

While managing archives with tar, it is also possible to add a file to an existing archive, or to update an archive. To add a file to an archive, you use the -r options. For example: 

tar -rvf /root/etc181006.tar /root/*.cfg /root/.bash*

To update a currently existing archive file, you can use the -u option: 

tar -uvf /root/etc181006.tar /root/*.cfg /root/.bash*

write newer versions of specified files all files in /home to the end of archive and correct the directory of the archive to to new entries. Older files are not deleted, only directory is corrected,  so archive increases in size.

Extracting files from a tarball

Before extracting a file, it is good to check if file exists and what are its attributes. The option -t can be used to find out. Type, for instance,

tar -tvf /root/etc181006.tar 

to see the contents of the tar archive. 

TIPS:

• For a single file it is better to use mc to extract the file.
• When specifying a file for extraction use exact copy of how this file is listed in tar -tvf listing.  Here is one relevant comment from  tecmint.com blog.
  • Bhasker Reddy says: November 4, 2015 at 3:58 am

    While extracting a particular file from the tarball, just giving a file name doesn’t work. You need to give the path of the file within the tarball. For example: I have taken a backup of my home directory called “/home/aziz” and it contains test1 and test2 files.

    $ tar -xvf /tmp/aziz.tar test1 — This command does not work while extracting a particular called test1
    $ tar -xvf /tmp/aziz.tar home/aziz/test1 — This command works while extracting a file called test1
    OR
    $ tar –extract –file=/tmp/aziz.tar home/aziz/test1

To extract the contents of an archive, use tar -xvf /archivename. This extracts all the content of the archive into the current directory. That means that if you are in /root when typing tar -xvf /root/homes.tar, and the file contains a directory /home, after extracting you’ll have a new directory /root/home that contains the entire contents of the file. This might not be what you wanted to accomplish.

It is also possible to extract one file out of the archive. If the file is in root directory of the archive it is put into current directory. If it in one of the subdirectories subdirectories are created:

tar -xvf /archivename.tar exact_path_in_listing_of_tar_archive_for_that_file

There are two solutions to put the extracted contents right where you want to have them:

TIPS

• You can use option -C when extracting files from the archive too. In this case you specify where to put extracted files. Use the option -C /targetdir to specify the target directory where you want to extract the file to. If you want to put the contents of the file /root/homes.tar in the directory /tmp, for instance, you can use tar -xvf homes.tar -C /tmp.
• Alternatively you can cd to the directory where you want to extract the file(s) and then run extraction.  This  is the default behavior of tar

If your archive etc181012.tar lists the file hosts as e ./hosts that you want to extract, for instance, use

tar -xvf /root/etc181012.tar ./hosts

This is a very useful capability, if you mangled the file and want to recover the previous version. Of course, you need a fresh archive to that. That's why it makes sense to create etc archive ach morning when you login to the system, but before any changes. 

As etc is the nerve center of the Linux OS and  contains all system related configuration files in here or in its sub-directories a good practice is to create etc archive in you  on your first login each day. You can do do putting appropriate command in your .bash_profile. You also can use GIT to maintain files in etc181012.  There is a package  called etckeeper that automatically adds changed file to git database, but still you need to have a tarball to recover git database in case of SNAFU.

 A "configuration file" is defined as a local file used to control the operation of a program; it must be static and cannot be an executable binary. For this reason, it's a good idea to backup this directory regularly. It will definitely save you a lot of re-configuration later if you re-install or lose your current installation. Normally, no binaries should be or are located here.

Using Compression

For large files compression should be done via parallel version of the compression program and this typically done via pipe. For small archives such as software you develop you can use option in tar.  

tar -cv  /etc |  pbzip2 > /root/etc181006.tar.bz2

on /etc directory bzip2 provides ~12% better compression then gzip, while xz provides ~30% better compression.  So you save one archive space each ten days if you do it daily from cron.  if files that contitute the archive are very large (say around 500MB and larger) is makes sense to compress them individually first and only then create tar archive without compression.

Usage  pigz or bzip2 is a must in this case.  Especially the last, as it provides better compression ration then gzip, while having adequate (although slightly slower then pigz) speed.

For small archives compression is specified via one of tar option -z for gzip, -j for bzip and -J for xz. The result is the same.  For example

tar -cvzf  /root/etc181006.tgz /etc

Please note that compressed tarball is an atomic entity: you can't extract individual files without decompressing it first. That's a big difference with zip archive format, which is more like virtual file system and allow to extract individual files from the archive without decompression the whole archive.

To decompress files that have been compressed with gzip or bzip2, you can use the gunzip and bunzip2 utilities. Parallel version in this case are not much faster: decompression speed is approximately the same. Of course, if archive was compressed better the decompression speed might be higher and in this sense xz might beat alternatives as it provides the highest compression ration. 

Here are compressed sizes of default content of /etc directory in RHEL 7 using those three compression programs:

[root@test01 Archive]# ll
total 24600
-rw-r--r--. 1 root root 8580949 Oct  7 17:56 etc181006.tbz
-rw-r--r--. 1 root root 9795626 Oct  7 17:47 etc181006.tgz
-rw-r--r--. 1 root root 6808940 Oct  7 18:04 etc181006.txz

TIPS:

• For large directories of files it does not make any sense to use built-in option  and you are better off compressing the tarball via pipe using parallel version of archivers. for xz it makes sense to do it in all cases as non-parallel version in excruciatingly slow even on /etc directory. 
• You can skip specifies compression option while extracting. The tar utility recognizes the type of compressed contents and automatically decompress it using proper compression utility.

Using tar to backup etc directory and home partition

One of the important tips for aspiring Linux system administrators is to create backup of etc directory each morning on the first login. This can be done automatically you work with the tar command to manage archives.

It is also important to backup home directory. In both cases the simplest way to accomplish those tasks is to use tar.

1. Open a root shell on your server. By logging in, the home directory of user root will become the current directory, so all relative filenames used in this exercise refer to /root/.
2. Type export timestamp=`date +"%y%m%d"`
3. Type tar -cvf etc$timestamp.tar /etc to archive the contents of the /etc directory.
4. Type file etc$timestamp.tar and read the information that is provided by the command. This should look like the following:
5. Type pigz etc$timestamp.tar.gz
6. Type tar tvf etc$timestamp.tar.gz. Notice that the tar command has no issues reading from a gzip compressed file. Also notice that the archive content consists of all relative filenames.
7. Type tar xvf etc$timestamp.tar.gz etc/hosts.
8. Type ls -R. Notice that a subdirectory etc has been created in the current directory. In this subdirectory, the file hosts has been restored.
9. Type gunzip etc$timestamp.tar.gz. This decompresses the compressed file but does not change anything else with regard to the tar command.
10. Type tar xvf etc$timestamp.tar -C /tmp etc/passwd. This extracts the password file to the /tmp directory.
11. Type tar cjvf /tmp/home$timestamp /home. This creates a dated compressed archive of the home directory to the home directory of user root.
12. Type rm /root/etc$timestamp* /root/home$timestamp*  to remove all files

Working with archive using Midnight Commander

Midnight Commander presents archive as "virtual file system" -- that means that if you click Enter on an archive it will display its content in panel and all standard operations are instantly available.

Moreover  you can program additional operation using so called "User menu" (available if you hit F2).  By default it contains seven archive related operations (in bold): 

• 3 Compress the current subdirectory (tar.gz)
• 4 Compress the current subdirectory (tar_bz2)
• 5 Compress the current subdirectory (tar.7z)
• 6 Compress the current subdirectory (tar.xz)
• ... ... ...
• y Gzip or gunzip current file
• b Bzip2 or bunzip2 current file

Operation 3-6 work the following way. they compress the current directory content and put the resilting tarball into the parent directory.  by default mc user menu uses non parallel version of cursors. You can edit it and change that to parallel version.

You can also add you own operations. For example instead the parent directory you can put archive into the directory defined by passive panel (%D)

NEWS CONTENTS

• 201213 : 18 Tar Command Examples in Linux ( 18 Tar Command Examples in Linux, )

Old News ;-)

18 Tar Command Examples in Linux

This is a weak article with multiple mistakes and typos. Most examples are trivial. But a couple is useful..

## Untar files in Current Directory ##
# tar -xvf public_html-14-09-12.tar

## Untar files in specified Directory ##
# tar -xvf public_html-14-09-12.tar -C /home/public_html/videos/

/home/public_html/videos/
/home/public_html/videos/views.php
/home/public_html/videos/index.php
/home/public_html/videos/logout.php
/home/public_html/videos/all_categories.php
/home/public_html/videos/feeds.xml

Extract Group of Files using Wildcard

To extract a group of files we use wildcard based extracting. For example, to extract a group of all files whose pattern begins with .php from a tar, tar.gz and tar.bz2 archive file.

# tar -xvf Phpfiles-org.tar --wildcards '*.php'

# tar -zxvf Phpfiles-org.tar.gz --wildcards '*.php'

# tar -jxvf Phpfiles-org.tar.bz2 --wildcards '*.php'

/home/php/iframe_ew.php
/home/php/videos_all.php
/home/php/rss.php
/home/php/index.php
/home/php/vendor.php
/home/php/video_title.php
/home/php/report.php
/home/php/video.php

Bhasker Reddy, November 4, 2015 at 3:58 am

While extracting a particular file from the tarball, just giving a file name doesn't work. You need to give the path of the file within the tarball.

For example: I have taken a backup of my home directory called "/home/aziz" and it contains test1 and test2 files.

$ tar -xvf /tmp/aziz.tar test1 - This command does not work while extracting a particular called test1
$ tar -xvf /tmp/aziz.tar home/aziz/test1 - This command works while extracting a file called test1

OR

$ tar --extract --file=/tmp/aziz.tar home/aziz/test1

Recommended Links

Softpanorama Recommended

https://www.certdepot.net/

Red Hat Certified System Administrator (RHCSA EX200) – Study Guide

...

Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotes :  Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce :  Bernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS :  Programming Languages History : PL/1 : Simula 67 : C : History of GCC development :  Scripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month :  How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: December 13, 2020