GNU parallel - xargs on steroids

The write-up below is based on parallel man page · super

xargs
 Getopt::Long
IPC::Open3
Symbol
IO::File
POSIX
File::Temp
rsync
ssh
xargs
tee
xargs
--sshlogin
-S
--transfer
parallel -S .. --transfer gzip ‘< {} | wc -c’ ::: *.txt
parallel -S .. --transfer --cleanup gzip ‘< {} | wc -c’ ::: *.txt
parallel -S .. --transfer --return {.}.bz2 --cleanup zcat ‘< {} | bzip2 >{.}.bz2’ ::: *.gz
parallel -S .. --trc {.}.bz2 zcat ‘< {} | bzip2 >{.}.bz2’ ::: *.gz
parallel [options] [command [arguments]] < 
list_of_arguments

parallel [options] [command [arguments]] (
::: arguments | :::: argfile(s) ) ...
-I.
  (echo file1; echo file2) | parallel gzip
  parallel gzip ::: file1 file2
  parallel gzip {} ::: file1 file2
  parallel --arg-sep ,, gzip {} ,, file1 file2
  parallel --arg-sep ,, gzip ,, file1 file2
  parallel ::: "gzip file1" "gzip file2"
  parallel echo {1} {2} {3} ::: 6 7 ::: 4 5 ::: 1 2 3
  parallel echo {1} {2} {3} :::: <(seq 6 7) <(seq 4 5) :::: <(seq 1 3)
  parallel -a <(seq 6 7) echo {1} {2} {3} :::: <(seq 4 5) :::: <(seq 1 3)
  parallel -a <(seq 6 7) -a <(seq 4 5) echo {1} {2} {3} ::: 1 2 3
  seq 6 7 | parallel -a - -a <(seq 4 5) echo {1} {2} {3} ::: 1 2 3
  seq 4 5 | parallel echo {1} {2} {3} :::: <(seq 6 7) - ::: 1 2 3
  find log -name '*gz' | parallel \
    --sshlogin server.example.com --transfer --return {.}.bz2 \
    --cleanup "zcat {} | bzip -9 >{.}.bz2"
  parallel -j4 sleep {}\; echo {} ::: 2 1 4 3
  parallel -j4 -k sleep {}\; echo {} ::: 2 1 4 3
  echo foo/bar.txt | parallel \
    --sshlogin server.example.com --return {.}.out touch {.}.out
  echo /tmp/foo/bar.txt | parallel \
    --sshlogin server.example.com --return {.}.out touch {.}.out
  echo /tmp/foo/bar.txt | \
    parallel --sshlogin server.example.com \
    --return {.}.out --return {.}.out2 touch {.}.out {.}.out2
  [sshcommand [options]][username@]hostname
  server.example.com
  [email protected]
  8/my-8-core-server.example.com
  2/[email protected]
  # This server has SSH running on port 2222
  ssh -p 2222 server.example.net
  4/ssh -p 2222 quadserver.example.net
  # Use a different ssh program
  myssh -p 2222 -l myusername hexacpu.example.net
  # Use a different ssh program with default number of cores
  //usr/local/bin/myssh -p 2222 -l myusername hexacpu.example.net
  # Use a different ssh program with 6 cores
  6//usr/local/bin/myssh -p 2222 -l myusername hexacpu.example.net
  # Assume 16 cores on the local computer
  16/:
  echo foo/bar.txt | parallel \
    --sshlogin server.example.com --transfer wc
  echo /tmp/foo/bar.txt | parallel \
    --sshlogin server.example.com --transfer wc
  parallel echo {1} {2} ::: 1 2 3 ::: a b c
  parallel --xapply echo {1} {2} ::: 1 2 3 ::: a b c
  #!/usr/bin/parallel -Yr traceroute
  foss.org.my
  debian.org
  freenetproject.org
     http://www.example.com/path/to/YYYYMMDD_##.jpg
  (for x in `cat list` ; do
    do_something $x
  done) | process_output
  cat list | (while read x ; do
    do_something $x
  done) | process_output
 (for x in `cat list` ; do
   no_extension=${x%.*};
   do_something $x scale $no_extension.jpg
   do_step2 <$x $no_extension
 done) | process_output
 cat list | (while read x ; do
   no_extension=${x%.*};
   do_something $x scale $no_extension.jpg
   do_step2 <$x $no_extension
 done) | process_output
  (for x in `cat xlist` ; do
    for y in `cat ylist` ; do
      do_something $x $y
    done
  done) | process_output
  seq 10 | parallel --sshlogin server.example.com echo
  seq 10 | parallel --sshlogin server.example.com,server2.example.net echo
  seq 10 | parallel --sshlogin server.example.com \
    --sshlogin server2.example.net echo
  seq 10 | parallel --sshlogin server.example.com \
    --sshlogin [email protected] echo
  server.example.com
  [email protected]
  server3.example.com
  seq 10 | parallel --sshloginfile mycomputers echo
  server.example.com
  [email protected]
  server3.example.com
  :
  seq 10 | parallel --sshlogin 8/server.example.com echo
  find logs/ -name '*.gz' | \
    parallel --sshlogin server.example.com \
    --transfer "zcat {} | bzip2 -9 >{.}.bz2"
  find logs/ -name '*.gz' | \
    parallel --sshlogin server.example.com \
    --transfer --return {.}.bz2 "zcat {} | bzip2 -9 >{.}.bz2"
  find logs/ -name '*.gz' | \
    parallel --sshlogin server.example.com \
    --transfer --return {.}.bz2 --cleanup "zcat {} | bzip2 -9 >{.}.bz2"
  find logs/ -name '*.gz' | \
    parallel --sshlogin server.example.com,server2.example.com \
    --sshlogin server3.example.com \
    --transfer --return {.}.bz2 --cleanup "zcat {} | bzip2 -9 >{.}.bz2"
  find logs/ -name '*.gz' | \
    parallel --sshlogin server.example.com,server2.example.com \
    --sshlogin server3.example.com \
    --sshlogin : \
    --transfer --return {.}.bz2 --cleanup "zcat {} | bzip2 -9 >{.}.bz2"
  find logs/ -name '*.gz' | \
    parallel --sshlogin server.example.com,server2.example.com \
    --sshlogin server3.example.com \
    --sshlogin : \
    --trc {.}.bz2 "zcat {} | bzip2 -9 >{.}.bz2"
  find logs/ -name '*.gz' | parallel --sshloginfile mycomputers \
    --trc {.}.bz2 "zcat {} | bzip2 -9 >{.}.bz2"
  find logs/ -name '*.gz' | parallel -S .. \
    --trc {.}.bz2 "zcat {} | bzip2 -9 >{.}.bz2"
  parallel --trc {.}.ogg -S server2,: \
  'mpg321 -w - {} | oggenc -q0 - -o {.}.ogg' ::: *.mp3
  foo<TAB>bar
  baz <TAB> quux
  cmd -o bar -i foo
  cmd -o quux -i baz
  ping -c 1 10.0.0.1
  wget http://example.com/status.cgi?ip=10.0.0.1
  ping -c 1 10.0.0.2
  wget http://example.com/status.cgi?ip=10.0.0.2
  ...
  ping -c 1 10.0.0.255
  wget http://example.com/status.cgi?ip=10.0.0.255
  for i in `ls *.log` ; do
    echo $i
    sem -j10 gzip $i ";" echo done
  done
  sem --wait
  seq 3 | parallel sem sed -i -e 'i{}' myfile
  seq 3 | parallel sem --id mymutex sed -i -e 'i{}' myfile
  parallel sudo echo ::: This is a bad idea
  sudo echo This
  parallel sudo echo ::: is a good idea
  sudo parallel echo ::: This is a good idea
  sh: -c: line 0: syntax error near unexpected token
  sh: Syntax error: Unterminated quoted string
  sh: -c: line 0: unexpected EOF while looking for matching `''
  sh: -c: line 1: syntax error: unexpected end of file
  echo -j-1 nice > ~/.parallel/nice_profile
  parallel -J nice_profile bzip2 -9 ::: *
  echo "perl -e '\$a=\$\$; print \$a,\" \",'\$PARALLEL_SEQ',\" \";';" > ~/.parallel/pre_perl
  parallel -J pre_perl echo ::: *
  echo -S .. nice > ~/.parallel/dist
  parallel -J dist --trc {.}.bz2 bzip2 -9 ::: *
pexec
pexec
/bin/sh
for ~ in ~ done
bash
pexec
1.0rc8
#!/bin/sh

/usr/bin/my-process-1 --args1 &
/usr/bin/my-process-2 --args2 &
/usr/bin/my-process-3 --args3 &

wait
echo all processes complete

#!/bin/sh

parallel /usr/bin/my-process-{} --args{} ::: 1 2 3
echo all processes complete

(wget -O - pi.dk/3 || curl pi.dk/3/ || fetch -o - http://pi.dk/3) | bash

#!/bin/sh

printf "1\n2\n3" | xargs -n1 -P3 -I{} /usr/bin/my-process-{} --args{}
echo all processes complete

function time_a_url ()
{
     oneurltime=$($time_command -p wget -p $1 -O /dev/null 2>&1 1>/dev/null | grep real | cut -d" " -f2)
     echo "Fetching $1 took $oneurltime seconds."
}

$ f() { echo "[$@]"; }
$ export -f f
$ echo -e "b 1\nc 2\nd 3 4" | xargs -P 0 -n 1 -I{} bash -c f\ \{\}
[b 1]
[d 3 4]
[c 2]

f() { echo "[$@]"; }
export -f f

echo -e "b 1\nc 2\nd 3 4" | sed 's/ /\\ /g' | xargs -P 10 -n 1 -I{} bash -c f\ \{\}

echo -e "b 1\nc 2\nd 3 4" | xargs -P 10 -I '{}' bash -c 'f "$@"' arg0 '{}'

  find . -name '*.html' | parallel gzip --best
  find . -type f -print0 | parallel -q0 perl -i -pe 's/FOO BAR/FUBAR/g'
  parallel gzip --best ::: *.html
  parallel lame {} -o {.}.mp3 ::: *.wav
  bash: /bin/mv: Argument list too long
  ls | grep -E '\.log$' | parallel mv {} destdir
  ls | grep -E '\.log$' | parallel -m mv {} destdir
  printf '%s\0' *.log | parallel -0 -m mv {} destdir
  seq -w 0 9999 | parallel rm pict{}.jpg
  seq -w 0 9999 | perl -pe 's/(.*)/pict$1.jpg/' | parallel -m rm
  seq -w 0 9999 | parallel -X rm pict{}.jpg
  convert -geometry 120 foo.jpg thumb_foo.jpg
  ls *.jpg | parallel convert -geometry 120 {} thumb_{}
  find . -name '*.jpg' | parallel convert -geometry 120 {} {}_thumb.jpg
  find . -name '*.jpg' | parallel convert -geometry 120 {} {.}_thumb.jpg
  parallel zcat {} ">"{.} ::: *.gz
  parallel "zcat {} >{.}" ::: *.gz
  ls | parallel 'echo -n {}" "; ls {}|wc -l'
  ls | parallel '(echo -n {}" "; ls {}|wc -l) >{}.dir'
  find . | parallel 'a={}; name=${a##*/};' \
    'upper=$(echo "$name" | tr "[:lower:]" "[:upper:]");'\
    'echo "$name - $upper"'

  ls | parallel 'mv {} "$(echo {} | tr "[:upper:]" "[:lower:]")"'
  cat urlfile | parallel "wget {} 2>/dev/null || grep -n {} urlfile"
  cp -rs /the/source/dir mirror_dir
  find mirror_dir -type l | parallel -m rm {} '&&' touch {}
  cat file_list | parallel 'if [ ! -e {} ] ; then echo {}; fi'
  parallel [ -f {1}:{2} ] "||" echo {1}:{2} does not exist \
    ::: {00..23} ::: {00..55..5}
  doit() {
    echo Doing it for $1
    sleep 2
    echo Done with $1
  }
  export -f doit
  parallel doit ::: 1 2 3

  doubleit() {
    echo Doing it for $1 $2
    sleep 2
    echo Done with $1 $2
  }
  export -f doubleit
  parallel doubleit ::: 1 2 3 ::: a b
  parallel --env doit -S server doit ::: 1 2 3
  parallel --env doubleit -S server doubleit ::: 1 2 3 ::: a b
  tester() {
    if (eval "$@") >&/dev/null; then
      perl -e 'printf "\033[30;102m[ OK ]\033[0m @ARGV\n"' "$@"
    else
      perl -e 'printf "\033[30;101m[FAIL]\033[0m @ARGV\n"' "$@"
    fi
  }
  export -f tester
  parallel tester my_program ::: arg1 arg2
  parallel tester exit ::: 1 0 2 0
  seq 9 -1 1 | parallel -j1 mv log.{} log.'{= $_++ =}'
  mv log log.1
  parallel 'mkdir {.}; cd {.}; unzip ../{}' ::: *.zip
  parallel "zcat {} | bzip2 >{.}.bz2 && rm {}" ::: *.gz
  find sounddir -type f -name '*.wav' | parallel lame {} -o {.}.mp3
  find sounddir -type f -name '*.wav' | \
    parallel lame {} -o mydir/{/.}.mp3
  parallel --plus 'mkdir {..}; tar -C {..} -xf {}' ::: *.tar.gz
  parallel --plus echo {%_demo} ::: mycode_demo keep_demo_here
  parallel --plus echo {#demo_} ::: demo_mycode keep_demo_here
  parallel --plus echo {/demo_/} ::: demo_mycode remove_demo_here
  http://www.example.com/path/to/YYYYMMDD_##.jpg
  getit() {
    date=$(date -d "today -$1 days" +%Y%m%d)
    num=$2
    echo wget http://www.example.com/path/to/${date}_${num}.jpg
  }
  export -f getit
  
  parallel getit ::: $(seq 30) ::: $(seq -w 24)
  base=https://map1a.vis.earthdata.nasa.gov/wmts-geo/wmts.cgi
  service="SERVICE=WMTS&REQUEST=GetTile&VERSION=1.0.0"
  layer="LAYER=BlueMarble_ShadedRelief_Bathymetry"
  set="STYLE=&TILEMATRIXSET=EPSG4326_500m&TILEMATRIX=5"
  tile="TILEROW={1}&TILECOL={2}"
  format="FORMAT=image%2Fjpeg"
  url="$base?$service&$layer&$set&$tile&$format"

  parallel -j0 -q wget "$url" -O {1}_{2}.jpg ::: {0..19} ::: {0..39}
  parallel eval convert +append {}_{0..39}.jpg line{}.jpg ::: {0..19}
  convert -append line{0..19}.jpg world.jpg
  base="https://images-api.nasa.gov/search"
  q="q=apollo 11"
  description="description=moon landing"
  media_type="media_type=image"
  wget -O - "$base?$q&$description&$media_type" |
    jq -r .collection.items[].href |
    parallel wget -O - |
    jq -r .[] |
    grep large |
    parallel wget
  find . | parallel cp {} '../destdir/{= $a=int(10000*rand); $_=pQ($_);
    $_=`date -r "$_" +%FT%T"$a"`; chomp; =}'
  parallel -k echo {1}'{=3 $_=$_%2?":":" "=}'{2}{3} \
    ::: {0..12} ::: {0..5} ::: {0..9}
  parallel --header : echo x{X}y{Y}z{Z} \> x{X}y{Y}z{Z} \
  ::: X {1..5} ::: Y {01..10} ::: Z {1..5}
  parallel eval 'cat {=s/y01/y*/=} > {=s/y01//=}' ::: *y01*
  cat x1y*z1 > x1z1
  PARALLEL=-j100 ./parallel-crawl http://gatt.org.yeslab.org/
  #!/bin/bash

  # E.g. http://gatt.org.yeslab.org/
  URL=$1
  # Stay inside the start dir
  BASEURL=$(echo $URL | perl -pe 's:#.*::; s:(//.*/)[^/]*:$1:')
  URLLIST=$(mktemp urllist.XXXX)
  URLLIST2=$(mktemp urllist.XXXX)
  SEEN=$(mktemp seen.XXXX)

  # Spider to get the URLs
  echo $URL >$URLLIST
  cp $URLLIST $SEEN

  while [ -s $URLLIST ] ; do
    cat $URLLIST |
      parallel lynx -listonly -image_links -dump {} \; \
        wget -qm -l1 -Q1 {} \; echo Spidered: {} \>\&2 |
        perl -ne 's/#.*//; s/\s+\d+.\s(\S+)$/$1/ and
          do { $seen{$1}++ or print }' |
      grep -F $BASEURL |
      grep -v -x -F -f $SEEN | tee -a $SEEN > $URLLIST2
    mv $URLLIST2 $URLLIST
  done

  rm -f $URLLIST $URLLIST2 $SEEN
  tar xvf foo.tgz | perl -ne 'print $l;$l=$_;END{print $l}' | \
    parallel echo
  (for x in `cat list` ; do
    do_something $x
  done) | process_output
  cat list | (while read x ; do
    do_something $x
  done) | process_output
  cat list | parallel do_something | process_output
  cat hosts.txt | parallel -P 100 host | grep 1.2.3.4
  (for x in `cat list` ; do
    no_extension=${x%.*};
    do_step1 $x scale $no_extension.jpg
    do_step2 <$x $no_extension
  done) | process_output
  cat list | (while read x ; do
    no_extension=${x%.*};
    do_step1 $x scale $no_extension.jpg
    do_step2 <$x $no_extension
  done) | process_output
  cat list | parallel "do_step1 {} scale {.}.jpg ; do_step2 <{} {.}" |\
    process_output
  (for x in `cat list` ; do
    do_something $x
    [... 100 lines that do something with $x ...]
  done) | process_output

  cat list | (while read x ; do
    do_something $x
    [... 100 lines that do something with $x ...]
  done) | process_output
  doit() {
    x=$1
    do_something $x
    [... 100 lines that do something with $x ...]
  }
  export -f doit
  cat list | parallel doit
  (for x in `cat xlist` ; do
    for y in `cat ylist` ; do
      do_something $x $y
    done
  done) | process_output
  parallel do_something {1} {2} :::: xlist ylist | process_output
  (for colour in red green blue ; do
    for size in S M L XL XXL ; do
      echo $colour $size
    done
  done) | sort
  parallel echo {1} {2} ::: red green blue ::: S M L XL XXL | sort
  parallel --tag 'diff {1} {2} | wc -l' ::: * ::: * | sort -nk3
  parallel --header : echo {colour} {size} \
    ::: colour red green blue ::: size S M L XL XXL
  cat addressbook.tsv | \
    parallel --colsep '\t' --header : echo {Name} {E-mail address}
  parallel echo {= 'if($arg[1] ge $arg[2]) { skip() }' =} ::: A B C D ::: A B C D
  parallel echo \
    '{= for $t (2..$#arg){ if($arg[$t-1] ge $arg[$t]) { skip() } } =}' \
    ::: A B C D ::: A B C D ::: A B C D
  aardvark
  babble
  cab
  dab
  each
  aardvark babble
  babble cab
  cab dab
  dab each
  parallel echo {1} - {2} ::::+ <(head -n -1 in.txt) <(tail -n +2 in.txt)
  seq $((${#a[@]}-1)) | env_parallel --env a echo '${a[{=$_--=}]} - ${a[{}]}'
  parallel echo {1} - {2} ::: "${a[@]::${#a[@]}-1}" :::+ "${a[@]:1}"
  parallel --results /tmp/diffcount "diff -U 0 {1} {2} | \
    tail -n +3 |grep -v '^@'|wc -l" ::: * ::: *
  seq -w 0 9999 | parallel touch pict{}.jpg
  seq -w 0 9999 | parallel -X touch pict{}.jpg
  seq -w 0 9999999 |
    parallel -j10 -q -I,, --pipe parallel -j0 touch pict{}.jpg
  ARR=("My brother's 12\" records are worth <\$\$\$>"'!' Foo Bar)
  parallel echo ::: ${ARR[@]} # This is probably not what you want
  ARR=("My brother's 12\" records are worth <\$\$\$>"'!' Foo Bar)
  parallel echo ::: "${ARR[@]}"
  VAR="My brother's 12\" records are worth <\$\$\$>"
  parallel -q echo "$VAR" ::: '!'
  export VAR
  parallel echo '"$VAR"' ::: '!'
  VAR="My 12\" records are worth <\$\$\$>"
  parallel echo "'$VAR'" ::: '!'
  VAR="My brother's 12\" records are worth <\$\$\$>"
  export VAR
  myfunc() { echo "$VAR" "$1"; }
  export -f myfunc
  parallel myfunc ::: '!'
  parallel wget --limit-rate=100k \
    https://ftpmirror.gnu.org/parallel/parallel-20{}0822.tar.bz2 \
    ::: {12..16}
  parallel --line-buffer wget --limit-rate=100k \
    https://ftpmirror.gnu.org/parallel/parallel-20{}0822.tar.bz2 \
    ::: {12..16}
  parallel -u wget --limit-rate=100k \
    https://ftpmirror.gnu.org/parallel/parallel-20{}0822.tar.bz2 \
    ::: {12..16}
  parallel --tag wget --limit-rate=100k \
    https://ftpmirror.gnu.org/parallel/parallel-20{}0822.tar.bz2 \
    ::: {12..16}
  parallel --tag --line-buffer wget --limit-rate=100k \
    https://ftpmirror.gnu.org/parallel/parallel-20{}0822.tar.bz2 \
    ::: {12..16}
  parallel --tag -S .. --nonall uptime
    parallel --tagstring '\033[30;3{=$_=++$::color%8=}m' seq {} ::: {1..10}
    parallel --rpl '{color} $_="\033[30;3".(++$::color%8)."m"' \
      --tagstring {color} seq {} ::: {1..10}
    ... | perl -pe 's/\t//'
  cat textfile | parallel -k echo {} append_string
  parallel traceroute ::: qubes-os.org debian.org freenetproject.org
  parallel -k traceroute ::: qubes-os.org debian.org freenetproject.org
  curl -r 10000000-19999999 http://example.com/the/big/file >file.part
  seq 0 99 | parallel -k curl -r \
    {}0000000-{}9999999 http://example.com/the/big/file > file
  find . -type f | parallel -k -j150% -n 1000 -m grep -H -n STRING {}
  grep -f regexps.txt bigfile
  grep -F -f regexps.txt bigfile
  ID1 foo bar baz Identifier1 quux
  fubar ID2 foo bar baz Identifier2
  ID1.*Identifier1
  ID2.*Identifier2
  ID1 foo bar baz Identifier1
  ID2 foo bar baz Identifier2
  parallel --pipepart -a regexps.txt --block 1M grep -Ff - -n bigfile |
    sort -un | perl -pe 's/^\d+://'
  free=$(awk '/^((Swap)?Cached|MemFree|Buffers):/ { sum += $2 }
              END { print sum }' /proc/meminfo)
  percpu=$((free / 200 / $(parallel --number-of-cores)))k

  parallel --pipepart -a regexps.txt --block $percpu --compress \
    grep -F -f - -n bigfile |
    sort -un | perl -pe 's/^\d+://'
  parallel --pipepart -a regexps.txt --block $percpu --compress \
    grep -F -f - bigfile
  cat regexp.txt | parallel --pipe -L1000 --round-robin --compress \
    grep -f - -n bigfile |
    sort -un | perl -pe 's/^\d+://'
  parallel --pipepart --block 100M -a bigfile -k --compress \
    grep -f regexp.txt
  parallel --pipepart --block 100M -a bigfile --fifo cat regexp.txt \
    \| parallel --pipe -L1000 --round-robin grep -f - {}
  # Add the servers to the sshloginfile
  (echo servera; echo serverb) > .parallel/my_cluster
  # Make sure .ssh/config exist
  touch .ssh/config
  cp .ssh/config .ssh/config.backup
  # Disable StrictHostKeyChecking temporarily
  (echo 'Host *'; echo StrictHostKeyChecking no) >> .ssh/config
  parallel --slf my_cluster --nonall true
  # Remove the disabling of StrictHostKeyChecking
  mv .ssh/config.backup .ssh/config
  seq 10 | parallel --sshlogin server.example.com echo
  seq 10 | parallel --sshlogin server.example.com,server2.example.net echo
  seq 10 | parallel --sshlogin server.example.com \
    --sshlogin server2.example.net echo
  seq 10 | parallel --sshlogin server.example.com \
    --sshlogin [email protected] echo
  seq 10 | parallel -Sfoo@server{1..88}.example.net echo
  server.example.com
  [email protected]
  server3.example.com
  seq 10 | parallel --sshloginfile mycomputers echo
  server.example.com
  [email protected]
  server3.example.com
  :
  seq 10 | parallel --sshlogin 8/server.example.com echo
  find logs/ -name '*.gz' | \
    parallel --sshlogin server.example.com \
    --transfer "zcat {} | bzip2 -9 >{.}.bz2"
  find logs/ -name '*.gz' | \
    parallel --sshlogin server.example.com \
    --transfer --return {.}.bz2 "zcat {} | bzip2 -9 >{.}.bz2"
  find logs/ -name '*.gz' | \
    parallel --sshlogin server.example.com \
    --transfer --return {.}.bz2 --cleanup "zcat {} | bzip2 -9 >{.}.bz2"
  find logs/ -name '*.gz' | \
    parallel --sshlogin server.example.com,server2.example.com \
    --sshlogin server3.example.com \
    --transfer --return {.}.bz2 --cleanup "zcat {} | bzip2 -9 >{.}.bz2"
  find logs/ -name '*.gz' | \
    parallel --sshlogin server.example.com,server2.example.com \
    --sshlogin server3.example.com \
    --sshlogin : \
    --transfer --return {.}.bz2 --cleanup "zcat {} | bzip2 -9 >{.}.bz2"
  find logs/ -name '*.gz' | \
    parallel --sshlogin server.example.com,server2.example.com \
    --sshlogin server3.example.com \
    --sshlogin : \
    --trc {.}.bz2 "zcat {} | bzip2 -9 >{.}.bz2"
  find logs/ -name '*.gz' | parallel --sshloginfile mycomputers \
    --trc {.}.bz2 "zcat {} | bzip2 -9 >{.}.bz2"
  find logs/ -name '*.gz' | parallel -S .. \
    --trc {.}.bz2 "zcat {} | bzip2 -9 >{.}.bz2"
  parallel --trc {.}.ogg -S server2,: \
    'mpg321 -w - {} | oggenc -q0 - -o {.}.ogg' ::: *.mp3
  parallel --tag --nonall -S server1,server2 uptime
  parallel --tag --onall -S server1,server2 echo ::: 1 2 3
  parallel --ssh 'ssh jumphost ssh' -S host1 echo ::: DOES NOT WORK
  Host host1 host2 host3
    ProxyCommand ssh jumphost.domain nc -w 1 %h 22
  parallel -S host1,host2,host3 echo ::: This does work
  Host host1.v
    Port 22001
  Host host2.v
    Port 22002
  Host host3.v
    Port 22003
  Host *.v
    Hostname firewall
  parallel -S host1.v,host2.v,host3.v echo ::: a b c
  HiddenServiceDir /var/lib/tor/hidden_service/
  HiddenServicePort 22 127.0.0.1:22
  parallel --ssh 'torsocks ssh' -S izjafdceobowklhz.onion \
    -S zfcdaeiojoklbwhz.onion,auclucjzobowklhi.onion echo ::: a b c
  parallel -S 'torsocks ssh izjafdceobowklhz.onion,host2,host3' \
    echo ::: a b c
  cd src-dir; find . -type f -size +100000 | \
    parallel -v ssh fooserver mkdir -p /dest-dir/{//}\; \
      rsync -s -Havessh {} fooserver:/dest-dir/{}
  rsync -Havessh src-dir/ fooserver:/dest-dir/
  seq -w 0 99 | parallel rsync -Havessh fooserver:src/*{}.png destdir/
  ls *.es.* | perl -pe 'print; s/\.es//' | parallel -N2 cp {1} {2}
  parallel -k echo ::: 0 1 ::: 0 1 ::: 0 1 ::: 0 1 ::: 0 1 ::: 0 1
  parallel --link -a <(seq 6) -a <(seq 6 -1 1) echo
  parallel --link echo :::: <(seq 6) <(seq 6 -1 1)
  parallel --link -a <(find . -type f | sort) \
    -a <(seq $(find . -type f|wc -l)) convert {1} {2}.png
  find . -type f | sort | parallel convert {} {#}.png
  foo<TAB>bar
  baz <TAB> quux
  cmd -o bar -i foo
  cmd -o quux -i baz
  parallel -a table_file.tsv --colsep '\t' cmd -o {2} -i {1}
  parallel -a table_file.tsv --trim n --colsep '\t' cmd -o {2} -i {1}
  DBURL=csv:///%2Ftmp%2Fmy.csv
  DBTABLEURL=$DBURL/mytable
  parallel --sqlandworker $DBTABLEURL seq ::: {1..10}
  DBURL=sqlite3:///%2Ftmp%2Fmy.sqlite
  DBTABLEURL=$DBURL/mytable
  parallel --sqlandworker $DBTABLEURL seq ::: {1..10}
  sql $DBURL '.headers on' '.mode csv' 'SELECT * FROM mytable;'
  DBURL=pg://user:pass@host/mydb
  DBTABLEURL=$DBURL/mytable
  parallel --sqlandworker $DBTABLEURL seq ::: {1..10}
  sql $DBURL "COPY (SELECT * FROM mytable) TO stdout DELIMITER ',' CSV HEADER;"
  DBURL=mysql://user:pass@host/mydb
  DBTABLEURL=$DBURL/mytable
  parallel --sqlandworker $DBTABLEURL seq ::: {1..10}
  sql -p -B $DBURL "SELECT * FROM mytable;" > mytable.tsv
  perl -pe 's/"/""/g; s/\t/","/g; s/^/"/; s/$/"/; s/\\\\/\\/g;
    s/\\t/\t/g; s/\\n/\n/g;' mytable.tsv
  parallel --results my.csv seq ::: 10 20 30
  R
  > mydf <- read.csv("my.csv");
  > print(mydf[2,])
  > write(as.character(mydf[2,c("Stdout")]),'')
  wget -O - http://arkiv.radio24syv.dk/audiopodcast/channel/4466232 |
    xpath -e "//ancestor::pubDate[contains(text(),'2016')]/../enclosure/@url" |
    parallel -u wget '{= s/ url="//; s/"//; =}'
  seq 10 | parallel -n0 my_command my_args
  ping -c 1 10.0.0.1
  wget http://example.com/status.cgi?ip=10.0.0.1
  ping -c 1 10.0.0.2
  wget http://example.com/status.cgi?ip=10.0.0.2
  ...
  ping -c 1 10.0.0.255
  wget http://example.com/status.cgi?ip=10.0.0.255
  parallel -j 100 < jobs_to_run
  cat bigfile | parallel --pipe --recend '' -k gzip -9 > bigfile.gz
  cat bigfile | parallel --pipe --files sort |\
    parallel -Xj1 sort -m {} ';' rm {} >bigfile.sort
  parallel --pipepart --block 100m -a bigfile --files sort |\
    parallel -Xj1 sort -m {} ';' rm {} >bigfile.sort
   Transaction Customer Item
        1       a       53
        2       b       65
        3       b       82
        4       c       96
        5       c       67
        6       c       13
        7       d       90
        8       d       43
        9       d       91
        10      d       84
        11      e       72
        12      e       102
        13      e       63
        14      e       56
        15      e       74
  sep=`perl -e 'print map { ("a".."z","A".."Z")[rand(52)] } (1..50);'`
  cat my.csv | perl -ape '$F[1] ne $l and print "'$sep'"; $l = $F[1]' |
     parallel --recend $sep --rrs --pipe -N1 wc
  cat myinput |\
    parallel --pipe -N 50 --round-robin -j50 parallel -j50 your_prg
  cat myinput |\
    parallel --pipe -N 250 --round-robin -j250 parallel -j250 your_prg
  for i in *.log ; do
    echo $i
    sem -j10 gzip $i ";" echo done
  done
  sem --wait
  seq 3 | parallel sem sed -i -e '1i{}' myfile
  seq 3 | parallel sem --id mymutex sed -i -e '1i{}' myfile
  #!/usr/bin/sem --shebang-wrap -u --id $0 --fg /bin/bash
  
  echo This will run
  sleep 5
  echo exclusively
  #!/usr/bin/sem --shebang-wrap -u --id $0 --fg /usr/bin/perl
  
  print "This will run ";
  sleep 5;
  print "exclusively\n";
  #!/usr/local/bin/sem --shebang-wrap -u --id $0 --fg /usr/bin/python
  
  import time
  print "This will run ";
  time.sleep(5)
  print "exclusively";
  cat filelist | parallel --tty -X emacs
  cat filelist | parallel --tty -X vi
  parallel sudo echo ::: This is a bad idea
  sudo echo This
  parallel sudo echo ::: is a good idea
  sudo parallel echo ::: This is a good idea
  true >jobqueue; tail -n+0 -f jobqueue | parallel
  echo my_command my_arg >> jobqueue
  true >jobqueue; tail -n+0 -f jobqueue | parallel -S ..
  true >jobqueue;
  while true; do
    tail -n+0 -f jobqueue |
      (parallel -E StOpHeRe -S ..; echo GNU Parallel is now done;
       perl -e 'while(<>){/StOpHeRe/ and last};print <>' jobqueue > j2;
       (seq 1000 >> jobqueue &);
       echo Done appending dummy data forcing tail to exit)
    echo tail exited;
    mv j2 jobqueue
  done
  # Day time
  echo 50% > jobfile
  cp day_server_list ~/.parallel/sshloginfile
  # Night time
  echo 100% > jobfile
  cp night_server_list ~/.parallel/sshloginfile
  tail -n+0 -f jobqueue | parallel --jobs jobfile -S ..
  inotifywait -qmre MOVED_TO -e CLOSE_WRITE --format %w%f my_dir |\
    parallel -u echo
  inotifywait -qmre MOVED_TO -e CLOSE_WRITE --format %w%f my_dir |\
    parallel -S ..  -u echo
  $ ./configure
  [...]
  checking for something.h... no
  configure: error: "libsomething not found"
  $ tracefile -un ./configure | tail | parallel -j0 apt-file search
  ( ) $ ` ' " < > ; | \
  ~ & # ! ? space * {
  perl -ne '/^\S+\s+\S+$/ and print $ARGV,"\n"' file
  parallel perl -ne \''/^\S+\s+\S+$/ and print $ARGV,"\n"'\' ::: file
  seq 10 | parallel -N2 echo seq:\$PARALLEL_SEQ arg1:{1} arg2:{2}
  VAR=this_is_set_before_starting
  echo test | parallel echo {} $VAR
  VAR="two  spaces  between  each  word"
  echo test | parallel echo {} \'"$VAR"\'
  echo test | parallel VAR=this_is_set_after_starting \; echo {} \$VAR
  echo test |\
    parallel VAR='"two  spaces  between  each  word"' echo {} \'"$VAR"\'
  seq 10 | parallel echo $$
  seq 10 | parallel echo \$\$
  echo test | parallel echo {} \\\$VAR
  echo test | parallel -q echo {} \$VAR
  perl -ne '/^\S+\s+\S+$/ and print $ARGV,"\n"' file
  ls | parallel perl -ne '/^\\S+\\s+\\S+\$/\ and\ print\ \$ARGV,\"\\n\"'
  ls | parallel perl -ne \''/^\S+\s+\S+$/ and print $ARGV,"\n"'\'
  ls | parallel -q  perl -ne '/^\S+\s+\S+$/ and print $ARGV,"\n"'
  ls *.gz | parallel -q "zcat {} >{.}"
  ls *.gz | parallel -q "zcat {} | bzip2 >{.}.bz2"
  sh: -c: line 0: syntax error near unexpected token
  sh: Syntax error: Unterminated quoted string
  sh: -c: line 0: unexpected EOF while looking for matching `''
  sh: -c: line 1: syntax error: unexpected end of file
  zsh:1: no matches found:
  ls | parallel -q bash -c 'wc -c <(echo {})'
  ls | parallel -q bash -c \
    'tar c {} | tee >(gzip >{}.tar.gz) | bzip2 >{}.tar.bz2'
  killall -USR1 parallel
  killall -TERM parallel
  seq 10 | parallel -N2 \
    echo seq:'$'PARALLEL_SEQ arg1:{1} arg2:{2}
  cat list | parallel -j1 -k -v ls
  cat list | parallel -j1 -k -v -S"myssh user@server" ls
  cat list | PARALLEL="-kvj1" parallel ls
  cat list | PARALLEL='-kvj1 -S myssh\ user@server' \
    parallel echo
  echo --tag -S .. --nonall > ~/.parallel/n
  parallel -Jn uptime
  echo -j-1 nice > ~/.parallel/nice_profile
  parallel -J nice_profile bzip2 -9 ::: *
  echo "perl -e '\$a=\$\$; print \$a,\" \",'\$PARALLEL_SEQ',\" \";';" \
    > ~/.parallel/pre_perl
  parallel -J pre_perl echo ::: *
  echo -S .. nice > ~/.parallel/dist
  parallel -J dist --trc {.}.bz2 bzip2 -9 ::: *
parallel --sshlogin yourserver.example.com hostname';' echo {} ::: 1 2 3
--sshlogin 
 -S
parallel -S yourserver.example.com,server2.example.net 
	hostname';' echo {} ::: 1 2 3
parallel -S yourserver.example.com -S [email protected] 
	hostname';' echo {} ::: 1 2 3
parallel -S yourserver.example.com -S [email protected] -S : 
	hostname';' echo {} ::: 1 2 3
parallel --sshloginfile mylistofsshlogins hostname';' echo {} ::: 1 2 3
parallel -S .. hostname';' echo {} ::: 1 2 3
parallel -S .. --transfer gzip '< {} | wc -c' ::: *.txt
parallel -S .. --transfer --cleanup gzip '< {} | wc -c' ::: *.txt
parallel -S .. --transfer --return {.}.bz2 --cleanup zcat '< {} | bzip2 >{.}.bz2' ::: *.gz
parallel -S .. --trc {.}.bz2 zcat '< {} | bzip2 >{.}.bz2' ::: *.gz
parallel --help
man parallel
man env_parallel
man parallel_tutorial
man parallel_design

Softpanorama May the source be with you, but remember the KISS principle ;-)	Home	Switchboard	Unix Administration	Red Hat	TCP/IP Networks	Neoliberalism	Toxic Managers
	(slightly skeptical) Educational society promoting "Back to basics" movement against IT overcomplexity and bastardization of classic Unix

News	Parallel command execution	Recommended Links	Options	Examples	Reference	Unix Xargs
pssh	Slurping	Shells	Pipes	AWK	Perl	Regex
find	uniq	cut	tee	History	Humor	Etc

Top Visited <p>Your browser does not support iframes.</p>					Switchboard
					Latest
					Past week
					Past month

GNU parallel - xargs on steroids

[Nov 08, 2018] pexec utility is similar to parallel

Nov 08, 2018 | www.gnu.org

[Jun 24, 2018] Three Ways to Script Processes in Parallel by Rudis Muiznieks

Sep 02, 2015 | www.codeword.xyz

[Jun 23, 2018] bash - Shell Scripting Using xargs to execute parallel instances of a shell function

Jun 23, 2018 | stackoverflow.com

[Jun 02, 2018] How to run Linux commands simultaneously with GNU Parallel

Jun 02, 2018 | www.techrepublic.com

[Jun 02, 2018] GNU Parallel Examples

Limiting factor: RAM

Limiting factor: CPU

Bigger problem

No jumphost, but port forwards

No jumphost, no port forwards

[Apr 30, 2013] GNU Parallel The Command-Line Power Tool USENIX by Ole Tange

Google matched content

Softpanorama Recommended

Downloading GNU Parallel

Documentation