Softpanorama

Home Switchboard Unix Administration Red Hat TCP/IP Networks Neoliberalism Toxic Managers
May the source be with you, but remember the KISS principle ;-)
Skepticism and critical thinking is not panacea, but can help to understand the world better

pdcp - copy files to groups of hosts in parallel

Exists in two variabts:

Synopsis

pdcp [options]... src [src2...] dest
rpdcp [options]... src [src2...] dir

Description

pdcp is a variant of the rcp(1) command. Unlike rcp(1), which copies files to a single remote host, pdcp can copy files to multiple remote hosts in parallel. However, pdcp does not recognize files in the format ''rname@rhost:path,'' therefore all source files must be on the local host machine. Destination nodes must be listed on the pdcp command line using a suitable target nodelist option (See the OPTIONS section below). Each destination node listed must have pdcp installed for the copy to succeed.

When pdcp receives SIGINT (ctrl-C), it lists the status of current threads. A second SIGINT within one second terminates the program. Pending threads may be canceled by issuing ctrl-Z within one second of ctrl-C. Pending threads are those that have not yet been initiated, or are still in the process of connecting to the remote host.

Like pdsh(1), the functionality of pdcp may be supplemented by dynamically loadable modules. In pdcp, the modules may provide a new connect protocol (replacing the standard rsh(1) protocol), filtering options (e.g. excluding hosts that are down), and/or host selection options (e.g. -a selects all nodes from a local config file). By default, pdcp requires at least one "rcmd" module to be loaded (to provide the channel for remote copy).

Reverse Pdcp

rpdcp performs a reverse parallel copy. Rather than copying files to remote hosts, files are retrieved from remote hosts and stored locally. All directories or files retrieved will be stored with their remote hostname appended to the filename. The destination file must be a directory when this option is used.

In other respects, rpdcp is exactly like pdcp, and further statements regarding pdcp in this manual also apply to rpdcp.

Options

The list of available pdcp options is determined at runtime by supplementing the list of standard pdcp options with any options provided by loaded rcmd and misc modules. In some cases, options provided by modules may conflict with each other. In these cases, the modules are incompatible and the first module loaded wins.

Standard target nodelist options

-w TARGETS,...
Target and or filter the specified list of hosts. Do not use with any other node selection options (e.g. -a-g, if they are available). No spaces are allowed in the comma-separated list. Arguments in the TARGETS list may include normal host names, a range of hosts in hostlist format (See HOSTLIST EXPRESSIONS), or a single '-' character to read the list of hosts on stdin.

If a host or hostlist is preceded by a '-' character, this causes those hosts to be explicitly excluded. If the argument is preceded by a single '^' character, it is taken to be the path to file containing a list of hosts, one per line. If the item begins with a '/' character, it is taken as a regular expression on which to filter the list of hosts (a regex argument may also be optionally trailed by another '/', e.g. /node.*/). A regex or file name argument may also be preceeded by a minus '-' to exclude instead of include thoses hosts.

A list of hosts may also be preceded by "user@" to specify a remote username other than the default, or "rcmd_type:" to specify an alternate rcmd connection type for these hosts. When used together, the rcmd type must be specified first, e.g. "ssh:user1@host0" would use ssh to connect to host0 as user "user1."

-x host,host,...
Exclude the specified hosts. May be specified in conjunction with other target node list options such as -a and -g (when available). Hostlists may also be specified to the -x option (see the HOSTLIST EXPRESSIONS section below). Arguments to -x may also be preceeded by the filename ('^') and regex ('/') characters as described above, in which case the resulting hosts are excluded as if they had been given to -w and preceeded with the minus '-' character.

Standard pdcp options

-h
Output usage menu and quit. A list of available rcmd modules will be printed at the end of the usage message.
-q
List option values and the target nodelist and exit without action.
-b
Disable ctrl-C status feature so that a single ctrl-C kills parallel copy. (Batch Mode)
-r
Copy directories recursively.
-p
Preserve modification time and modes.
-e PATH
Explicitly specify path to remote pdcp binary instead of using the locally executed path. Can also be set via the environment variable PDSH_REMOTE_PDCP_PATH.
-l user
This option may be used to copy files as another user, subject to authorization. For BSD rcmd, this means the invoking user and system must be listed in the user's .rhosts file (even for root).
-t seconds
Set the connect timeout. Default is 10 seconds.
-f number
Set the maximum number of simultaneous remote copies to number. The default is 32.
-R name
Set rcmd module to name. This option may also be set via the PDSH_RCMD_TYPE environment variable. A list of available rcmd modules may be obtained via either the -h or -L options.
-M name,...
When multiple misc modules provide the same options to pdsh, the first module initialized "wins" and subsequent modules are not loaded. The -M option allows a list of modules to be specified that will be force-initialized before all others, in-effect ensuring that they load without conflict (unless they conflict with eachother). This option may also be set via the PDSH_MISC_MODULES environment variable.
-L
List info on all loaded pdcp modules and quit.
-d
Include more complete thread status when SIGINT is received, and display connect and command time statistics on stderr when done.
-V
Output pdcp version information, along with list of currently loaded modules, and exit.

Hostlist Expressions

As noted in sections above, pdcp accepts ranges of hostnames in the general form: prefix[n-m,l-k,...], where n < m and l < k, etc., as an alternative to explicit lists of hosts. This form should not be confused with regular expression character classes (also denoted by ''[]''). For example, foo[19] does not represent foo1 or foo9, but rather represents a degenerate range: foo19.

This range syntax is meant only as a convenience on clusters with a prefixNN naming convention and specification of ranges should not be considered necessary -- the list foo1,foo9 could be specified as such, or by the range foo[1,9].

Some examples of range usage follow:

Copy /etc/hosts to foo01,foo02,...,foo05
    pdcp -w foo[01-05] /etc/hosts /etc
Copy /etc/hosts to foo7,foo9,foo10
    pdcp -w foo[7,9-10] /etc/hosts /etc
Copy /etc/hosts to foo0,foo4,foo5
    pdcp -w foo[0-5] -x foo[1-3] /etc/hosts /etc
As a reminder to the reader, some shells will interpret brackets ('[' and ']') for pattern matching. Depending on your shell, it may be necessary to enclose ranged lists within quotes. For example, in tcsh, the first example above should be executed as:

pdcp -w "foo[01-05]" /etc/hosts /etc
 

Origin

Pdsh/pdcp was originally a rewrite of IBM dsh(1) by Jim Garlick <garlick@llnl.gov> on LLNL's ASCI Blue-Pacific IBM SP system. It is now also used on Linux clusters at LLNL.

Limitations

When using ssh for remote execution, stderr of ssh to be folded in with that of the remote command. When invoked by pdcp, it is not possible for ssh to prompt for confirmation if a host key changes, prompt for passwords if RSA keys are not configured properly, etc.. Finally, the connect timeout is only adjustable with ssh when the underlying ssh implementation supports it, and pdsh has been built to use the correct option.

See Also


Top Visited
Switchboard
Latest
Past week
Past month

NEWS CONTENTS

Old News ;-)

Recommended Links

Google matched content

Softpanorama Recommended

Top articles

Sites



Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotesSomerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose BierceBernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOSProgramming Languages History : PL/1 : Simula 67 : C : History of GCC developmentScripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-MonthHow to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Haterís Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D


Copyright © 1996-2020 by Softpanorama Society. www.softpanorama.org was initially created as a service to the (now defunct) UN Sustainable Development Networking Programme (SDNP) without any remuneration. This document is an industrial compilation designed and created exclusively for educational use and is distributed under the Softpanorama Content License. Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.

FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available to advance understanding of computer science, IT technology, economic, scientific, and social issues. We believe this constitutes a 'fair use' of any such copyrighted material as provided by section 107 of the US Copyright Law according to which such material can be distributed without profit exclusively for research and educational purposes.

This is a Spartan WHYFF (We Help You For Free) site written by people for whom English is not a native language. Grammar and spelling errors should be expected. The site contain some broken links as it develops like a living tree...

You can use PayPal to to buy a cup of coffee for authors of this site

Disclaimer:

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: February 27, 2021