|Contents||Bulletin||Scripting in shell and Perl||Network troubleshooting||History||Humor|
|News||Books||See also||Recommended Links||Reference||Tips|
While fist widely used compression program was created for Unix (tar and compress) the field was stale and it was DOS which provided strong impetus for the development of sophisticated compression programs. Many talented authors competed with each other until the dust settles and winners emerged.
The first widely popular DOS compression program was pkzip by late Phil Katz who create the company PKWARE. Starting from pkzip 2.0 (released in 1993) it used so called "deflating" a lossless data compression algorithm based on a combination of the LZ77 algorithm and Huffman coding. The resulting file format has since become ubiquitous in DOS and later Windows as well as on BBS and later the Internet -- almost all files with the .ZIP (or .zip) extension are in PKZIP 2.x format. Utilities to read and write these files are available on all common platforms. It was later specified in RFC 1951 and several OSes like Windows 2000 and XP are able to work with such files natively. On April 24, 2007, PKWARE announced the release of SecureZIP Standard Version 11 as freeware, available on www.securezip.com. Competing programs included Rahul Dhesi's ZOO, Dean W. Cooper's DWC, and LHarc by Haruhiko Okomura and Haruyasu Yoshizaki. Gzip is an attempt to replicate part of functionality of pkzip in Unix environment. gzip uses Lempel-Ziv coding (LZ77).
GZIP competes with bzip2 (slower and more CPU hungry, but with better compression), xz, Info zip package (zip and unzip) and rar. Gzip is weaker as the for compression ratio but has other advantages like high speed and wide availability. It is still it is more or less adequate and despite being obsolete from the compression ration standpoint is widely used.
By default applying gzip to the file leads to replacing the original file fo the archive with the extension .gz. it keeps while keeping the same ownership modes, access and modification times. If no files are specified or if a file name is "-", the standard input is compressed to the standard output. gzip will only attempt to compress regular files. In particular, it will ignore symbolic links.
If the new file name is too long for its file system, gzip truncates it. gzip attempts to truncate only the parts of the file name longer than 3 characters. (A part is delimited by dots.) If the name consists of small parts only, the longest parts are truncated. For example, if file names are limited to 14 characters, gzip.msdos.exe is compressed to gzi.msd.exe.gz. Names are not truncated on systems which do not have a limit on file name length.
By default, gzip keeps the original file name and timestamp in the compressed file. These are used when decompressing the file with the `-N' option. This is useful when the compressed file name was truncated or when the time stamp was not preserved after a file transfer. However, due to limitations in the current gzip file format, fractional seconds are discarded. Also, time stamps must fall within the range 1970-01-01 00:00:00 through 2106-02-07 06:28:15 UTC, and hosts whose operating systems use 32-bit time stamps are further restricted to time stamps no later than 2038-01-19 03:14:07 UTC. The upper bounds assume the typical case where leap seconds are ignored.
Compressed files can be restored to their original form using `gzip -d' or gunzip or zcat. If the original name saved in the compressed file is not suitable for its file system, a new name is constructed from the original one to make it legal.
gunzip takes a list of files on its command line and replaces each file whose name ends with `.gz', `.z', `.Z', `-gz', `-z' or `_z' and which begins with the correct magic number with an uncompressed file without the original extension. gunzip also recognizes the special extensions `.tgz' and `.taz' as shorthands for `.tar.gz' and `.tar.Z' respectively. When compressing, gzip uses the `.tgz' extension if necessary instead of truncating a file with a `.tar' extension.
gunzip can currently decompress files created by gzip, zip, compress or pack. The detection of the input format is automatic. When using the first two formats, gunzip checks a 32 bit CRC (cyclic redundancy check). For pack, gunzip checks the uncompressed length. The compress format was not designed to allow consistency checks. However gunzip is sometimes able to detect a bad `.Z' file. If you get an error when uncompressing a `.Z' file, do not assume that the `.Z' file is correct simply because the standard uncompress does not complain. This generally means that the standard uncompress does not check its input, and happily generates garbage output. The SCO `compress -H' format (lzh compression method) does not include a CRC but also allows some consistency checks.
Files created by zip can be uncompressed by gzip only if they have a single member compressed with the 'deflation' method. This feature is only intended to help conversion of tar.zip files to the tar.gz format. To extract a zip file with a single member, use a command like `gunzip <foo.zip' or `gunzip -S .zip foo.zip'. To extract zip files with several members, use unzip instead of gunzip.
zcat is identical to `gunzip -c'. zcat uncompresses either a list of files on the command line or its standard input and writes the uncompressed data on standard output. zcat will uncompress files that have the correct magic number whether they have a `.gz' suffix or not.
gzip uses the Lempel-Ziv algorithm used in zip and PKZIP. The amount of compression obtained depends on the size of the input and the distribution of common substrings. Typically, text such as source code or English is reduced by 60-70%. Compression is generally much better than that achieved by LZW (as used in compress), Huffman coding (as used in pack), or adaptive Huffman coding (compact).
Compression is always performed, even if the compressed file is slightly larger than the original. The worst case expansion is a few bytes for the gzip file header, plus 5 bytes every 32K block, or an expansion ratio of 0.015% for large files. Note that the actual number of used disk blocks almost never increases. gzip normally preserves the mode, ownership and time stamps of files when compressing or decompressing.
The gzip file format is specified in P. Deutsch, gzip file format specification version 4.3, Internet RFC 1952 (May 1996). The zip deflation format is specified in P. Deutsch, deflate Compressed Data Format Specification version 1.3, Internet RFC 1951 (May 1996).
Here are some realistic examples of running gzip.
Compress the ISO content into gz file:
gzip -rc /mnt > /tmp/iso.gzYou can time this operation is the speed of comression of ISO can serve as a poor man test of performace of the computer
time gzip -rc /mnt > /tmp/iso.gz
From 11 Simple Gzip Examples RootUsers
- 1. Compress a single file
This will compress file.txt and create file.txt.gz, note that this will remove the original file.txt file.
- 2. Compress multiple files at once
This will compress all files specified in the command, note again that this will remove the original files specified by turning file1.txt, file2.txt and file3.txt into file1.txt.gz, file2.txt.gz and file3.txt.gz
gzip file1.txt file2.txt file3.txt
To instead compress all files within a directory, see example 8 below.
- 3. Compress a single file and keep the original
You can instead keep the original file and create a compressed copy.
gzip -c file.txt > file.txt.gz
The -c flag outputs the compressed copy of file.txt to stdout, this is then sent to file.txt.gz, keeping the original file.txt file in place. Newer versions of gzip may also have -k or –keep available, which could be used instead with “gzip -k file.txt”.
- 4. Compress all files recursively
All files within the directory and all sub directories can be compressed recursively with the -r flag
[root@centos test]# ls -laR .: drwxr-xr-x. 2 root root 24 Jul 28 18:05 example -rw-r--r--. 1 root root 8 Jul 28 17:09 file1.txt -rw-r--r--. 1 root root 3 Jul 28 17:54 file2.txt -rw-r--r--. 1 root root 5 Jul 28 17:54 file3.txt ./example: -rw-r--r--. 1 root root 5 Jul 28 18:00 example.txt [root@centos test]# gzip -r * [root@centos test]# ls -laR .: drwxr-xr-x. 2 root root 27 Jul 28 18:07 example -rw-r--r--. 1 root root 38 Jul 28 17:09 file1.txt.gz -rw-r--r--. 1 root root 33 Jul 28 17:54 file2.txt.gz -rw-r--r--. 1 root root 35 Jul 28 17:54 file3.txt.gz ./example: -rw-r--r--. 1 root root 37 Jul 28 18:00 example.txt.gz
In the above example there are 3 .txt files in the test directory which is our current working directory, there is also an example sub directory which contains example.txt. Upon running gzip with the -r flag over everything, all files were recursively compressed.
This can be reversed by running “gzip -dr *”, where -d is used to decompress and -r performs this on all of the files recursively.
- 5. Decompress a gzip compressed file
To reverse the compression process and get the original file back that you have compressed, you can use the gzip command itself or gunzip which is also part of the gzip package.
gzip -d file.txt.gz
Both of these commands will produce the same result, decompressing file.txt.gz to file.txt, removing the compressed file.txt.gz file.
Similar to example 3, it is possible to decompress a file and keep the original .gz file as below.gunzip -c file.txt.gz > file.txt
As mentioned in step 4, -d can be combined with -r to decompress all files recursively.
- 6. List compression information
With the -l or --list flag we can see useful information regarding a compressed .gz file such as the compressed and uncompressed size of the file as well as the compression ratio, which shows us how much space our compression is saving.
[root@centos ~]# gzip -l linux-3.18.19.tar.gz compressed uncompressed ratio uncompressed_name 126117045 580761600 78.3% linux-3.18.19.tar [root@centos ~]# ls -lah -rw-r--r--. 1 root root 554M Jul 28 17:24 linux-3.18.19.tar -rw-r--r--. 1 root root 121M Jul 28 17:25 linux-3.18.19.tar.gz
In this example, a gzipped copy of the Linux kernel has compressed to 78.3% of its original size, taking up 121MB of space rather than 554MB.
- 7. Adjust compression level
The level of compression applied to a file using gzip can be specified as a value between 1 (less compression) and 9 (best compression). Using option 1 will complete faster, but space saved from the compression will not be optimal. Using option 9 will take longer to complete, however you will have the largest amount of space saved.
The below example compares the differences between -1 and -9, as shown while -1 finishes much faster it compresses around 5% less (approximately 30mb more space required).[root@centos ~]# time gzip -1 linux-3.18.19.tar real 0m13.602s user 0m12.908s sys 0m0.662s [root@mirror1 ~]# gzip -l linux-3.18.19.tar.gz compressed uncompressed ratio uncompressed_name 156001021 580761600 73.1% linux-3.18.19.tar [root@centos ~]# time gzip -9 linux-3.18.19.tar real 0m58.129s user 0m57.193s sys 0m0.735s [root@centos ~]# gzip -l linux-3.18.19.tar.gz compressed uncompressed ratio uncompressed_name 125064095 580761600 78.5% linux-3.18.19.tar
-1 can also be specified with the flag --fast, while option -9 can also be specified with the flag --best. By default gzip uses a compression level of -6, which is slightly biased towards higher compression at the expense of speed. When selecting a value between 1 and 9 it is important to consider what is more important to you, the amount of space saved or the amount of time spent compressing, the default -6 option provides a fair trade off.
- 8. Compress a directory
With the help of the tar command, we can create a tar file of a whole directory and gzip the result. We can perform the whole lot in one step, as the tar command allows us to specify a compression method to use.
tar czvf etc.tar.gz /etc/
This example creates a compressed etc.tar.gz file of the entire /etc/ directory. The tar flags are as follows, ‘c’ creates a new tar archive, ‘z’ specifies that we want to compress with gzip, ‘v’ provides verbose information, and ‘f’ specifies the file to create. The resulting etc.tar.gz file contains all files within /etc/ compressed using gzip.
- 9. Integrity test
The -t or --test flag can be used to check the integrity of a compressed file.
On a normal file, the result will be listed as OK, shown below.[root@centos test]# gzip -tv file1.txt.gz file1.txt.gz: OK
I have now manually modified this file with a text editor and added a random value, essentially introducing corruption and it is now no longer valid.[root@centos test]# gzip -tv file1.txt.gz file1.txt.gz: gzip: file1.txt.gz: invalid compressed data--crc error gzip: file1.txt.gz: invalid compressed data--length error
The compressed .gz file makes use of cyclic redundancy check (CRC) in order to detect errors. The CRC value can be viewed by running gzip with the -l and -v flags, as shown below.[root@centos test]# gzip -lv file1.txt.gz method crc date time compressed uncompressed ratio uncompressed_name defla 08db5c50 Jul 28 18:15 40 167772160 100.0% file1.txt
- 10. Concatenate multiple files
Multiple files can be concatenated into a single .gz file.
gzip -c file1.txt > files.gz gzip -c file2.txt >> files.gz
The files.gz now contains the contents of both file1.txt and file2.txt, if you decompress files.gz you will get a file named ‘files’ which contains the content of both .txt files. The output is similar to running ‘cat file1.txt file2.txt’. If instead you want to create a single file that contains multiple files you can use the tar command which supports gzip compression, as covered above in example 8.
- 11. Additional commands included with gzip
The gzip package provides some very useful commands for working with compressed files, such as zcat, zgrep and zless/zmore.
As you can probably tell by the names of the commands, these are essentially the cat, grep, and less/more commands, however they work directly on compressed data. This means that you can easily view or search the contents of a compressed file without having to decompress it and then view or search it in a second step.[root@centos test]# zcat test.txt.gz test example text [root@centos test]# zgrep exa test.txt.gz example
This is especially useful when searching through or reviewing log files which have been compressed during log rotation.
This is the output of the command `gzip -h':
gzip version-number usage: gzip [-cdfhlLnNrtvV19] [-S suffix] [file ...] -c --stdout write on standard output, keep original files unchanged -d --decompress decompress -f --force force overwrite of output file and compress links -h --help give this help -l --list list compressed file contents -L --license display software license -n --no-name do not save or restore the original name and time stamp -N --name save or restore the original name and time stamp -q --quiet suppress all warnings -r --recursive operate recursively on directories -S .suf --suffix .suf use suffix .suf on compressed files -t --test test compressed file integrity -v --verbose verbose mode -V --version display version number -1 --fast compress faster -9 --best compress better file... files to (de)compress. If none given, use standard input. Report bugs to <firstname.lastname@example.org>.
This is the output of the command `gzip -v texinfo.tex':
texinfo.tex: 69.7% -- replaced with texinfo.tex.gz
The following command will find all gzip files in the current directory and subdirectories, and extract them in place without destroying the original:
find . -name '*.gz' -print | sed 's/^\(.*\)[.]gz$/gunzip < "&" > "\1"/' | sh
The format for running the gzip program is:
gzip option ...
gzip supports the following options:
compressed size: size of the compressed file uncompressed size: size of the uncompressed file ratio: compression ratio (0.0% if unknown) uncompressed_name: name of the uncompressed file
The uncompressed size is given as `-1' for files not in gzip format, such as compressed `.Z' files. To get the uncompressed size for such a file, you can use:
zcat file.Z | wc -c
In combination with the `--verbose' option, the following fields are also displayed:
method: compression method (deflate,compress,lzh,pack) crc: the 32-bit CRC of the uncompressed data date & time: time stamp for the uncompressed file
The crc is given as ffffffff for a file not in gzip format.
With `--verbose', the size totals and compression ratio for all files is also displayed, unless some sizes are unknown. With `--quiet', the title and totals lines are not displayed.
The gzip format represents the input size modulo 2^32, so the uncompressed size and compression ratio are listed incorrectly for uncompressed files 4 GB and larger. To work around this problem, you can use the following command to discover a large uncompressed file's true size:
zcat file.gz | wc -c
gunzip -S "" * (*.* for MSDOS)
Previous versions of gzip used the `.z' suffix. This was changed to avoid a conflict
Multiple compressed files can be concatenated. In this case, gunzip will extract all members at once. If one member is damaged, other members might still be recovered after removal of the damaged member. Better compression can be usually obtained if all members are decompressed and then recompressed in a single step.
This is an example of concatenating gzip files:
gzip -c file1 > foo.gz gzip -c file2 >> foo.gz
gunzip -c foo
is equivalent to
cat file1 file2
In case of damage to one member of a `.gz' file, other members can still be recovered (if the damaged member is removed). However, you can get better compression by compressing all members at once:
cat file1 file2 | gzip > foo.gz
compresses better than
gzip -c file1 file2 > foo.gz
If you want to recompress concatenated files to get better compression, do:
zcat old.gz | gzip > new.gz
If a compressed file consists of several members, the uncompressed size and CRC reported by the `--list' option applies to the last member only. If you need the uncompressed size for all members, you can use:
zcat file.gz | wc -c
If you wish to create a single archive file with multiple members so that members can later be extracted independently, use an archiver such as tar or zip. GNU tar supports the `-z' option to invoke gzip transparently. gzip is designed as a complement to tar, not as a replacement.
The environment variable GZIP can hold a set of default options for gzip. These options are interpreted first and can be overwritten by explicit command line parameters. For example:
for sh: GZIP="-8v --name"; export GZIP for csh: setenv GZIP "-8v --name" for MSDOS: set GZIP=-8v --name
When writing compressed data to a tape, it is generally necessary to pad the output with zeroes up to a block boundary. When the data is read and the whole block is passed to gunzip for decompression, gunzip detects that there is extra trailing garbage after the compressed data and emits a warning by default if the garbage contains nonzero bytes. You have to use the `--quiet' option to suppress the warning. This option can be set in the GZIP environment variable, as in:
for sh: GZIP="-q" tar -xfz --block-compress /dev/rst0 for csh: (setenv GZIP "-q"; tar -xfz --block-compress /dev/rst0)
In the above example, gzip is invoked implicitly by the `-z' option of GNU tar. Make sure that the same block size (`-b' option of tar) is used for reading and writing compressed data on tapes. (This example assumes you are using the GNU version of tar.)
FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available in our efforts to advance understanding of environmental, political, human rights, economic, democracy, scientific, and social justice issues, etc. We believe this constitutes a 'fair use' of any such copyrighted material as provided for in section 107 of the US Copyright Law. In accordance with Title 17 U.S.C. Section 107, the material on this site is distributed without profit exclusivly for research and educational purposes. If you wish to use copyrighted material from this site for purposes of your own that go beyond 'fair use', you must obtain permission from the copyright owner.
ABUSE: IPs or network segments from which we detect a stream of probes might be blocked for no less then 90 days. Multiple types of probes increase this period.
Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers : Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism : The Iron Law of Oligarchy : Libertarian Philosophy
War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda : SE quotes : Language Design and Programming Quotes : Random IT-related quotes : Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce : Bernard Shaw : Mark Twain Quotes
Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 : Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law
Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds : Larry Wall : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS : Programming Languages History : PL/1 : Simula 67 : C : History of GCC development : Scripting Languages : Perl history : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history
The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month : How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite
Most popular humor pages:
Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor
The Last but not Least
Copyright © 1996-2016 by Dr. Nikolai Bezroukov. www.softpanorama.org was created as a service to the UN Sustainable Development Networking Programme (SDNP) in the author free time. This document is an industrial compilation designed and created exclusively for educational use and is distributed under the Softpanorama Content License.
Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.
FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available to advance understanding of computer science, IT technology, economic, scientific, and social issues. We believe this constitutes a 'fair use' of any such copyrighted material as provided by section 107 of the US Copyright Law according to which such material can be distributed without profit exclusively for research and educational purposes.
This is a Spartan WHYFF (We Help You For Free) site written by people for whom English is not a native language. Grammar and spelling errors should be expected. The site contain some broken links as it develops like a living tree...
|You can use PayPal to make a contribution, supporting development of this site and speed up access. In case softpanorama.org is down you can use the at softpanorama.info|
The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the author present and former employers, SDNP or any other organization the author may be associated with. We do not warrant the correctness of the information provided or its fitness for any purpose.
Last modified: October 03, 2016