Softpanorama

May the source be with you, but remember the KISS principle ;-)
Home Switchboard Unix Administration Red Hat TCP/IP Networks Neoliberalism Toxic Managers
(slightly skeptical) Educational society promoting "Back to basics" movement against IT overcomplexity and  bastardization of classic Unix

xz and 7za compression programs

xz is a lossless data compression program and file format which incorporates the LZMA compression algorithm. xz is used now for distributing patched in many Linux flavors. You can specify xz for compression in tar with option GNU tar.  For 7za you usually need to pipe tar output into the program.

7za is compatible and is much (often 10 times) faster then xz. For some reason xz does not multithread well. Decompression speeds are close as decompression is almost never multithreaded. 

xz format has almost better level of compression then dominant Gzip format and in some cases is able to produce files that half of the size of files compressed with gzip.

Recently it became standard for packages of  Debian family of systems deb (file format), openSUSE, Fedora, Arch Linux,Slackware,[FreeBSD, Gentoo, GNOME,and TeX Live, as well as being an option to compress a compiled Linux kernel.

In December 2013, the Linux kernel maintainers kernel.org announced that they would use xz instead of bzip2 as their compression tool from 2014 on.

One can think of both xz and 7za as a stripped-down version of the 7-Zip program.

xz compresses single files as input, and does not bundle multiple files into a single archive. It is therefore common to compress a file that is itself an archive, such as those created by the tar or cpio Unix programs.

xz options are close to gzip so immediate "transfer of skills" is possible. 7za options are closer to original 7zip program and RAR archiver. 

Format of files used

Although the original 7-Zip program, which implements LZMA2 compression, is able to produce small files at the cost of speed, it also created its own unique archive format which was made primarily for Windows and did not support Unix functionality. xz and 7za uses thier own Linux based format as Windows 7-Zip format does not support directly Unix file system metadata). Xz and 7za are compatible with each other: each can decompress files compressed with the other program. 

Implementation

An implementation of the xz file format is freely available online as XZ Utils. It is licensed under the terms of the GNU LGPL and GNU GPL, with the bulk of the software (e.g., liblzma) in the public domain. Version 1.22 of GNU tar supports using this software to handle xz files transparently.

FreeBSD tar has supported xz transparently since r191190 (released on April 17, 2009).

7-Zip has supported xz since version 9.04 beta (stable since 9.20).

 

7za man page

7za - A file archiver with highest compression ratio

7za [adeltux] [-] [SWITCH] <ARCHIVE_NAME>< ARGUMENTS>...

7-Zip is a file archiver with the highest compression ratio. The program supports 7z (that implements LZMA compression algorithm), ZIP, CAB, ARJ, GZIP, BZIP2, TAR, CPIO, RPM and DEB formats. Compression ratio in the new 7z format is 30-50% better than ratio in ZIP format.

7za is a stand-alone executable. 7za handles less archive formats than 7z, but does not need any others.

Function Letters

a
Add
d
Delete
e
Extract
l
List
t
Test
u
Update
x
eXtract with full paths

Switches

-ai[r[-|0]]{@listfile|!wildcard}
Include archives
-ax[r[-|0]]{@listfile|!wildcard}
eXclude archives
-bd
Disable percentage indicator
-i[r[-|0]]{@listfile|!wildcard}
Include filenames
-l
don't store symlinks; store the files/directories they point to (CAUTION : the scanning stage can never end because of recursive symlinks like 'ln -s .. ldir')
-m{Parameters}
-mhe=on|off
7z format only : enables or disables archive header encryption (Default : off)
-o{Directory}
Set Output directory
-p{Password}
Set Password
-r[-|0]
Recurse subdirectories (CAUTION: this flag does not do what you think, avoid using it)
-sfx[{name}]
Create SFX archive
-si
Read data from StdIn (eg: tar cf - directory | 7za a -si directory.tar.7z)
-so
Write data to StdOut (eg: % echo foo | 7z a dummy -tgzip -si -so > /dev/null)
-slt
Sets technical mode for l (list) command
-t{Type}
Type of archive (7z, zip, gzip, bzip2 or tar. 7z format is default)
-v{Size}[b|k|m|g]
Create volumes
-u[-][p#][q#][r#][x#][y#][z#][!newArchiveName]
Update options
-w[path]
Set Working directory
-x[r[-|0]]]{@listfile|!wildcard}
Exclude filenames
-y
Assume Yes on all queries

Diagnostics

7-Zip returns the following exit codes:
0
Normal (no errors or warnings detected)
1
Warning (Non fatal error(s)). For example, some files cannot be read during compressing. So they were not compressed
2
Fatal error
7
Bad command line parameters
8
Not enough memory for operation
255
User stopped the process with control-C (or similar)

Backup and limitations

DO NOT USE the 7-zip format for backup purpose on Linux/Unix because : - 7-zip does not store the owner/group of the file.

On Linux/Unix, in order to backup directories you must use tar : - to backup a directory : tar cf - directory | 7za a -si directory.tar.7z
- to restore your backup : 7za x -so directory.tar.7z | tar xf -

If you want to send files and directories (not the owner of file) to others Unix/MacOS/Windows users, you can use the 7-zip format.

example : 7za a directory.7z directory

Do not use "-r" because this flag does not do what you think.

Do not use directory/* because of ".*" files (example : "directory/*" does not match "directory/.profile")

Example 1

7za a -t7z -m0=lzma -mx=9 -mfb=64 -md=32m -ms=on archive.7z dir1
adds all files from directory "dir1" to archive archive.7z using "ultra settings"
-t7z
7z archive
-m0=lzma
lzma method
-mx=9
level of compression = 9 (Ultra)
-mfb=64
number of fast bytes for LZMA = 64
-md=32m
dictionary size = 32 megabytes
-ms=on
solid archive = on

Example 2

7za a -sfx archive.exe dir1
add all files from directory "dir1" to SFX archive archive.exe (Remark : SFX archive MUST end with ".exe")

Example 3

7za a -mhe=on -pmy_password archive.7z a_directory
add all files from directory "a_directory" to the archive "archive.7z" (with data and header archive encryption on)

 


Top Visited
Switchboard
Latest
Past week
Past month

NEWS CONTENTS

Old News ;-)

[May 28, 2018] TIP 7-zip s XZ compression on a multiprocessor system is often faster and compresses better than gzip linuxadmin

May 28, 2018 | www.reddit.com

TIP: 7-zip's XZ compression on a multiprocessor system is often faster and compresses better than gzip ( self.linuxadmin )


TyIzaeL line"> [–] kristopolous 4 years ago (4 children)

I did this a while back also. Here's a graph: http://i.imgur.com/gPOQBfG.png

X axis is compression level (min to max) Y is the size of the file that was compressed

I forget what the file was.

TyIzaeL 4 years ago (3 children)
That is a great start (probably better than what I am doing). Do you have time comparisons as well?
kristopolous 4 years ago (1 child)
http://www.reddit.com/r/linuxquestions/comments/1gdvnc/best_file_compression_format/caje4hm there's the post
TyIzaeL 4 years ago (0 children)
Very nice. I might work on something similar to this soon next time I'm bored.
kristopolous 4 years ago (0 children)
nope.
TyIzaeL 4 years ago (0 children)
That's a great point to consider among all of this. Compression is always a tradeoff between how much CPU and memory you want to throw at something and how much space you would like to save. In my case, hammering the server for 3 minutes in order to take a backup is necessary because the uncompressed data would bottleneck at the LAN speed.
randomfrequency 4 years ago (0 children)
You might want to play with 'pigz' - it's gzip, multi-threaded. You can 'pv' to restrict the rate of the output, and it accepts signals to control the rate limiting.
rrohbeck 4 years ago (1 child)
Also pbzip2 -1 to -9 and pigz -1 to -9.

With -9 you can surely make backup CPU bound. I've given up on compression though: rsync is much faster than straight backup and I use btrfs compression/deduplication/snapshotting on the backup server.

TyIzaeL 4 years ago (0 children)
pigz -9 is already on the chart as pigz --best. I'm working on adding the others though.
TyIzaeL 4 years ago (0 children)
I'm running gzip, bzip2, and pbzip2 now (not at the same time, of course) and will add results soon. But in my case the compression keeps my db dumps from being IO bound by the 100mbit LAN connection. For example, lzop in the results above puts out 6041.632 megabits in 53.82 seconds for a total compressed data rate of 112 megabits per second, which would make the transfer IO bound. Whereas the pigz example puts out 3339.872 megabits in 81.892 seconds, for an output data rate of 40.8 megabits per second. This is just on my dual-core box with a static file, on the 8-core server I see the transfer takes a total of about three minutes. It's probably being limited more by the rate at which the MySQL server can dump text from the database, but if there was no compression it'd be limited by the LAN speed. If we were dumping 2.7GB over the LAN directly, we would need 122mbit/s of real throughput to complete it in three minutes.
Shammyhealz 4 years ago (2 children)
I thought the best compression was supposed to be LZMA? Which is what the .7z archives are. I have no idea of the relative speed of LZMA and gzip
TyIzaeL 4 years ago (1 child)
xz archives use the LZMA2 format (which is also used in 7z archives). LZMA2 speed seems to range from a little slower than gzip to much slower than bzip2, but results in better compression all around.
primitive_screwhead 4 years ago (0 children)
However LZMA2 decompression speed is generally much faster than bzip2, in my experience, though not as fast as gzip. This is why we use it, as we decompress our data much more often than we compress it, and the space saving/decompression speed tradeoff is much more favorable for us than either gzip of bzip2.
crustang 4 years ago (2 children)
I mentioned how 7zip was superior to all other zip programs in /r/osx a few days ago and my comment was burried in favor of the the osx circlejerk .. it feels good seeing this data.

I love 7zip

RTFMorGTFO 4 years ago (1 child)
Why... Tar supports xz, lzma, lzop, lzip, and any other kernel based compression algorithms. Its also much more likely to be preinstalled on your given distro.
crustang 4 years ago (0 children)
I've used 7zip at my old job for a backup of our business software's database. We needed speed, high level of compression, and encryption. Portability wasn't high on the list since only a handful of machines needed access to the data. All machines were multi-processor and 7zip gave us the best of everything given the requirements. I haven't really looked at anything deeply - including tar, which my old boss didn't care for.

[May 28, 2018] RPM RedHat EL 6 p7zip 9.20.1 x86_64 rpm

May 28, 2018 | rpm.pbone.net
p7zip rpm build for : RedHat EL 6 . For other distributions click p7zip .
Name : p7zip
Version : 9.20.1 Vendor : Dag Apt Repository, http://dag_wieers_com/apt/
Release : 1.el6.rf Date : 2011-04-20 15:23:34
Group : Applications/Archiving Source RPM : p7zip-9.20.1-1.el6.rf.src.rpm
Size : 14.84 MB
Packager : Dag Wieers < dag_wieers_com>
Summary : Very high compression ratio file archiver
Description :
p7zip is a port of 7za.exe for Unix. 7-Zip is a file archiver with a very high
compression ratio. The original version can be found at http://www.7-zip.org/.

RPM found in directory: /mirror/apt.sw.be/redhat/el6/en/x86_64/rpmforge/RPMS

Content of RPM Changelog Provides Requires
Download
ftp.univie.ac.at p7zip-9.20.1-1.el6.rf.x86_64.rpm
ftp.rediris.es p7zip-9.20.1-1.el6.rf.x86_64.rpm
ftp.icm.edu.pl p7zip-9.20.1-1.el6.rf.x86_64.rpm
ftp.pbone.net p7zip-9.20.1-1.el6.rf.x86_64.rpm
ftp.pbone.net p7zip-9.20.1-1.el6.rf.x86_64.rpm
ftp.pbone.net p7zip-9.20.1-1.el6.rf.x86_64.rpm
ftp.is.co.za p7zip-9.20.1-1.el6.rf.x86_64.rpm

Recommended Links

Google matched content

Softpanorama Recommended

Top articles

Sites

Free software portal
  1. XZ Utils Release Notes (Git), Tukaaani
  2. Lindholm, Linux Gazette .
  3. XZ Utils Web site
  4. GNU tar Web site: References
  5. Changelog for Tar 1.22
  6. "release history", 7-Zip .
  7. FTP), Coreutils, GNU (see version 7.1 and newer files ending in .tar.xz).
  8. "openSUSE has moved off of LZMA to xz", News, openSUSE .
  9. "XZ RPM payloads" (wiki), Features (12 ed.), Fedora .
  10. "Switching to xz compression for new packages", News, Archlinux .
  11. Entry (FTP) (changelog) (13.0 ed.), Slackware, Fri May 8 18:49:03 CDT 2009 Check date values in: |date= (help).
  12. Stable (mailing list post), Free BSD, Jan 2011 .
  13. "Remove .lzma in favor of .xz portage snapshots" (mailing list post), Dev (RFC), Gentoo .
  14. "Important: Switch of GNOME tarball compression format" (mailing list post), Devel, Gnome, Apr 2011 .
  15. tex-archive/systems/texlive/tlnet/archive (direc­tory), CTAN .
  16. xz embedded (Git), Tukaani .
  17. https://www.kernel.org/happy-new-year-and-good-bye-bzip2.html


Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotesSomerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose BierceBernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOSProgramming Languages History : PL/1 : Simula 67 : C : History of GCC developmentScripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-MonthHow to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D


Copyright © 1996-2021 by Softpanorama Society. www.softpanorama.org was initially created as a service to the (now defunct) UN Sustainable Development Networking Programme (SDNP) without any remuneration. This document is an industrial compilation designed and created exclusively for educational use and is distributed under the Softpanorama Content License. Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.

FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available to advance understanding of computer science, IT technology, economic, scientific, and social issues. We believe this constitutes a 'fair use' of any such copyrighted material as provided by section 107 of the US Copyright Law according to which such material can be distributed without profit exclusively for research and educational purposes.

This is a Spartan WHYFF (We Help You For Free) site written by people for whom English is not a native language. Grammar and spelling errors should be expected. The site contain some broken links as it develops like a living tree...

You can use PayPal to to buy a cup of coffee for authors of this site

Disclaimer:

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: May, 31, 2018