SUMMARY LEGEND

The following features are in some of the comparable tools:

GNU Parallel

Summary (see legend above):

• I1 I2 I3 I4 I5 I6 I7

• M1 M2 M3 M4 M5 M6

• O1 O2 O3 O4 O5 O6 O7 O8 O9 O10

• E1 E2 E3 E4 E5 E6 E7 E8 E9

• R1 R2 R3 R4 R5 R6 R7 R8 R9

• S1 S2

DIFFERENCES BETWEEN xargs AND GNU Parallel

Summary (see legend above):

• I1 I2 - - - - -

• - M2 M3 - - -

• - O2 O3 - O5 O6

• E1 - - - - - - E8 E9

• - - - - - x - - -

• - -

xargs offers some of the same possibilities as GNU parallel.

xargs deals badly with special characters (such as space, ,円 ' and "). To see the problem try this:

touchimportant_file
touch'not important_file'
lsnot*|xargsrm
mkdir-p"My brother's 12\" records"
ls|xargsrmdir
touch'c:\windows\system32\clfs.sys'
echo'c:\windows\system32\clfs.sys'|xargsls-l

You can specify -0, but many input generators are not optimized for using NUL as separator but are optimized for newline as separator. E.g. awk, ls, echo, tar -v, head (requires using -z), tail (requires using -z), sed (requires using -z), perl (-0 and 0円 instead of \n), locate (requires using -0), find (requires using -print0), grep (requires using -z or -Z), sort (requires using -z).

GNU parallel's newline separation can be emulated with:

cat|xargs-d"\n"-n1command

xargs can run a given number of jobs in parallel, but has no support for running number-of-cpu-cores jobs in parallel.

xargs has no support for grouping the output, therefore output may run together, e.g. the first half of a line is from one process and the last half of the line is from another process. The example Parallel grep cannot be done reliably with xargs because of this. To see this in action try:

parallelperl-e"'"'$a="1"."{}"x100000000;print $a,"\n"'"'"\
'>'{}:::abcdefgh
# Serial = no mixing = the wanted result
# 'tr -s a-z' squeezes repeating letters into a single letter
echoabcdefgh|xargs-P1-n1grep1|tr-sa-z
# Compare to 8 jobs in parallel
parallel-kP8-n1grep1:::abcdefgh|tr-sa-z
echoabcdefgh|xargs-P8-n1grep1|tr-sa-z
echoabcdefgh|xargs-P8-n1grep--line-buffered1|\
tr-sa-z

Or try this:

slow_seq(){
echoCountto"$@"
seq"$@"|
perl-ne'$|=1; for(split//){ print; select($a,$a,$a,0.100);}'
}
export-fslow_seq
# Serial = no mixing = the wanted result
seq8|xargs-n1-P1-I{}bash-c'slow_seq {}'
# Compare to 8 jobs in parallel
seq8|parallel-P8slow_seq{}
seq8|xargs-n1-P8-I{}bash-c'slow_seq {}'

xargs has no support for keeping the order of the output, therefore if running jobs in parallel using xargs the output of the second job cannot be postponed till the first job is done.

xargs has no support for running jobs on remote computers.

xargs has no support for context replace, so you will have to create the arguments.

If you use a replace string in xargs (-I) you can not force xargs to use more than one argument.

Quoting in xargs works like -q in GNU parallel. This means composed commands and redirection require using bash -c.

ls|parallel"wc {} >{}.wc"
ls|parallel"echo {}; ls {}|wc"

becomes (assuming you have 8 cores and that none of the filenames contain space, " or ').

ls|xargs-d"\n"-P8-I{}bash-c"wc {} >{}.wc"
ls|xargs-d"\n"-P8-I{}bash-c"echo {}; ls {}|wc"

A more extreme example can be found on: https://unix.stackexchange.com/q/405552/

https://www.gnu.org/software/findutils/

DIFFERENCES BETWEEN find -exec AND GNU Parallel

Summary (see legend above):

• - - - x - x -

• - M2 M3 - - - -

• - O2 O3 O4 O5 O6

• - - - - - - -

• - - - - - - - - -

• x x

find -exec offers some of the same possibilities as GNU parallel.

find -exec only works on files. Processing other input (such as hosts or URLs) will require creating these inputs as files. find -exec has no support for running commands in parallel.

https://www.gnu.org/software/findutils/ (Last checked: 2019-01)

DIFFERENCES BETWEEN make -j AND GNU Parallel

Summary (see legend above):

• - - - - - - -

• - - - - - -

• O1 O2 O3 - x O6

• E1 - - - E5 -

• - - - - - - - - -

• - -

make -j can run jobs in parallel, but requires a crafted Makefile to do this. That results in extra quoting to get filenames containing newlines to work correctly.

make -j computes a dependency graph before running jobs. Jobs run by GNU parallel does not depend on each other.

(Very early versions of GNU parallel were coincidentally implemented using make -j).

https://www.gnu.org/software/make/ (Last checked: 2019-01)

DIFFERENCES BETWEEN ppss AND GNU Parallel

Summary (see legend above):

• I1 I2 - - - - I7

• M1 - M3 - - M6

• O1 - - x - -

• E1 E2 ?E3 E4 - - -

• R1 R2 R3 R4 - - ?R7 ? ?

• - -

ppss is a tool for running jobs in parallel.

The output of ppss is status information and thus not useful for using as input for another command. The output from the jobs are put into files.

The argument replace string ($ITEM) cannot be changed. Arguments must be quoted - thus arguments containing special characters (space '"&!*) may cause problems. More than one argument is not supported. Filenames containing newlines are not processed correctly. When reading input from a file null cannot be used as a terminator. ppss needs to read the whole input file before starting any jobs.

Output and status information is stored in ppss_dir and thus requires cleanup when completed. If the dir is not removed before running ppss again it may cause nothing to happen as ppss thinks the task is already done. GNU parallel will normally not need cleaning up if running locally and will only need cleaning up if stopped abnormally and running remote (--cleanup may not complete if stopped abnormally). The example Parallel grep would require extra postprocessing if written using ppss.

For remote systems PPSS requires 3 steps: config, deploy, and start. GNU parallel only requires one step.

1$./ppss.shstandalone-d/path/to/files-c'gzip '
1$find/path/to/files-typef|parallelgzip
2$./ppss.shstandalone-d/path/to/files\
-c'cp "$ITEM" /destination/dir '
2$find/path/to/files-typef|parallelcp{}/destination/dir
3$./ppss.shstandalone-flist-of-urls.txt-c'wget -q '
3$parallel-alist-of-urls.txtwget-q
4$./ppss.shstandalone-flist-of-urls.txt-c'wget -q "$ITEM"'
4$parallel-alist-of-urls.txtwget-q{}
5$./ppssconfig-Cconfig.cfg-c'encode.sh '-d/source/dir\
-m192.168.1.100-uppss-kppss-key.key-S./encode.sh\
-nnodes.txt-o/some/output/dir--upload--download;
./ppssdeploy-Cconfig.cfg
./ppssstart-Cconfig
5$# parallel does not use configs. If you want
# a different username put it in nodes.txt: user@hostname
findsource/dir-typef|
parallel--sshloginfilenodes.txt--trc{.}.mp3\
lame-a{}-o{.}.mp3--presetstandard--quiet
6$./ppssstop-Cconfig.cfg
6$killall-TERMparallel
7$./ppsspause-Cconfig.cfg
7$Press:CTRL-Zorkillall-SIGTSTPparallel
8$./ppsscontinue-Cconfig.cfg
8$Enter:fgorkillall-SIGCONTparallel
9$./ppss.shstatus-Cconfig.cfg
9$killall-SIGUSR2parallel

DIFFERENCES BETWEEN pexec AND GNU Parallel

Summary (see legend above):

• I1 I2 - I4 I5 - -

• M1 - M3 - - M6

• O1 O2 O3 - O5 O6

• E1 - - E4 - E6 -

• R1 - - - - R6 - - -

• S1 -

pexec is also a tool for running jobs in parallel.

1$pexec-osqrt-%s.dat-p"$(seq10)"-eNUM-n4-c--\
'echo "scale=10000;sqrt($NUM)" | bc'
1$seq10|parallel-j4'echo "scale=10000;sqrt({})" | \
 bc > sqrt-{}.dat'
2$pexec-p"$(lsmyfiles*.ext)"-i%s-o%s.sort--sort
2$lsmyfiles*.ext|parallelsort{}">{}.sort"
3$pexec-fimage.list-nauto-eB-ustar.log-c--\
'fistar $B.fits -f 100 -F id,x,y,flux -o $B.star'
3$parallel-aimage.list\
'fistar {}.fits -f 100 -F id,x,y,flux -o {}.star'2>star.log
4$pexec-r*.png-eIMG-c-o---\
'convert $IMG ${IMG%.png}.jpeg ; "echo $IMG: done"'
4$ls*.png|parallel'convert {} {.}.jpeg; echo {}: done'
5$pexec-r*.png-i%s-o%s.jpg-c'pngtopnm | pnmtojpeg'
5$ls*.png|parallel'pngtopnm < {} | pnmtojpeg > {}.jpg'
6$forpin*.png;doecho${p%.png};done|\
pexec-f--i%s.png-o%s.jpg-c'pngtopnm | pnmtojpeg'
6$ls*.png|parallel'pngtopnm < {} | pnmtojpeg > {.}.jpg'
7$LIST=$(forpin*.png;doecho${p%.png};done)
pexec-r$LIST-i%s.png-o%s.jpg-c'pngtopnm | pnmtojpeg'
7$ls*.png|parallel'pngtopnm < {} | pnmtojpeg > {.}.jpg'
8$pexec-n8-r*.jpg-yunix-eIMG-c\
'pexec -j -m blockread -d $IMG | \
 jpegtopnm | pnmscale 0.5 | pnmtojpeg | \
 pexec -j -m blockwrite -s th_$IMG'
8$# Combining GNU B<parallel> and GNU B<sem>.
ls*jpg|parallel-j8'sem --id blockread cat {} | jpegtopnm |'\
'pnmscale 0.5 | pnmtojpeg | sem --id blockwrite cat > th_{}'
# If reading and writing is done to the same disk, this may be
# faster as only one process will be either reading or writing:
ls*jpg|parallel-j8'sem --id diskio cat {} | jpegtopnm |'\
'pnmscale 0.5 | pnmtojpeg | sem --id diskio cat > th_{}'

DIFFERENCES BETWEEN xjobs AND GNU Parallel

xjobs is also a tool for running jobs in parallel. It only supports running jobs on your local computer.

xjobs deals badly with special characters just like xargs. See the section DIFFERENCES BETWEEN xargs AND GNU Parallel.

1$ls-1*.zip|xjobsunzip
1$ls*.zip|parallelunzip
2$ls-1*.zip|xjobs-nunzip
2$ls*.zip|parallelunzip>/dev/null
3$find.-name'*.bak'|xjobsgzip
3$find.-name'*.bak'|parallelgzip
4$ls-1*.jar|sed's/\(.*\)/1円 > 1円.idx/'|xjobsjartf
4$ls*.jar|paralleljartf{}'>'{}.idx
5$xjobs-sscript
5$catscript|parallel
6$mkfifo/var/run/my_named_pipe;
xjobs-s/var/run/my_named_pipe&
echounzip1.zip>>/var/run/my_named_pipe;
echotarcf/backup/myhome.tar/home/me>>/var/run/my_named_pipe
6$mkfifo/var/run/my_named_pipe;
cat/var/run/my_named_pipe|parallel&
echounzip1.zip>>/var/run/my_named_pipe;
echotarcf/backup/myhome.tar/home/me>>/var/run/my_named_pipe

DIFFERENCES BETWEEN prll AND GNU Parallel

prll is also a tool for running jobs in parallel. It does not support running jobs on remote computers.

prll encourages using BASH aliases and BASH functions instead of scripts. GNU parallel supports scripts directly, functions if they are exported using export -f, and aliases if using env_parallel.

prll generates a lot of status information on stderr (standard error) which makes it harder to use the stderr (standard error) output of the job directly as input for another program.

1$prll-s'mogrify -flip 1ドル'*.jpg
1$parallelmogrify-flip:::*.jpg

DIFFERENCES BETWEEN dxargs AND GNU Parallel

dxargs is also a tool for running jobs in parallel.

dxargs does not deal well with more simultaneous jobs than SSHD's MaxStartups. dxargs is only built for remote run jobs, but does not support transferring of files.

https://web.archive.org/web/20120518070250/http://www. semicomplete.com/blog/geekery/distributed-xargs.html (Last checked: 2019-01)

DIFFERENCES BETWEEN mdm/middleman AND GNU Parallel

middleman(mdm) is also a tool for running jobs in parallel.

1$seq19|parallelbuffon-o-|sort-n>result
catfiles|parallelcmd
finddir-execdirsemcmd{}\;

DIFFERENCES BETWEEN xapply AND GNU Parallel

xapply can run jobs in parallel on the local computer.

1$xapply'(cd %1 && make all)'*/
1$parallel'cd {} && make all':::*/
2$xapply-f'diff %1 ../version5/%1'manifest|more
2$paralleldiff{}../version5/{}<manifest|more
3$xapply-p/dev/null-f'diff %1 %2'manifest1checklist1
3$parallel--linkdiff{1}{2}::::manifest1checklist1
4$xapply'indent'*.c
4$parallelindent:::*.c
5$find~ksb/bin-typef!-perm-111-print|\
xapply-f-v'chmod a+x'-
5$find~ksb/bin-typef!-perm-111-print|\
parallel-vchmoda+x
6$find*/-...|fmt9601024|xapply-f-i/dev/tty'vi'-
6$sh<(find*/-...|parallel-s1024echovi)
6$find*/-...|parallel-s1024-Xuj1vi
7$find...|xapply-f-5-i/dev/tty'vi'-----
7$sh<(find...|parallel-n5echovi)
7$find...|parallel-n5-uj1vi
8$xapply-fn""/etc/passwd
8$parallel-kecho</etc/passwd
9$tr':''012円'</etc/passwd|\
xapply-7-nf'chown %1 %6'-------
9$tr':''012円'</etc/passwd|parallel-N7chown{1}{6}
10$xapply'[ -d %1/RCS ] || echo %1'*/
10$parallel'[ -d {}/RCS ] || echo {}':::*/
11$xapply-f'[ -f %1 ] && echo %1'List|...
11$parallel'[ -f {} ] && echo {}'<List|...

DIFFERENCES BETWEEN AIX apply AND GNU Parallel

apply can build command lines based on a template and arguments - very much like GNU parallel. apply does not run jobs in parallel. apply does not use an argument separator (like :::); instead the template must be the first argument.

1$applyecho*
1$parallelecho:::*

2$apply-2cmpa1b1a2b2
2$parallel-N2cmp:::a1b1a2b2

3$apply-0who12345
3$parallel-N0who:::12345

4$apply'ln %1 /usr/joe'*
4$parallelln{}/usr/joe:::*

DIFFERENCES BETWEEN paexec AND GNU Parallel

Summary (see legend above):

• I1 - - - - - I7

• - - M3 - - -

• (O1) O2 O3 (O4) (O5) O6 - O8 x -

• E1 - - - (E5) - - -

• R1 - - - x R6 - R8 R9

• - -

paexec can run jobs in parallel on both the local and remote computers.

paexec has a job dependency facility so a job can depend on another job to be executed successfully. Sort of a poor-man's make. This can partly be emulated in GNU parallel with tsort.

paexec fails if output of a single line is > 2 GB. Output of a 2 GB line requires 6 GB RAM. Lines of standard output is interleaved (but there is no half line mixing), and output of standard error mixes. Combined with paexec_reorder output order can be the same as input order. In certain situations paexec will eat the last newline of standard output.

There seems to be no way to have the number og jobs depend on the number of CPU threads in a mixed server setup: E.g run 4 jobs on a remote server with 4 cores and 16 jobs on a remote server with 16 cores.

1$paexec-t'/usr/bin/ssh -x'-n'host1 host2 host3'\
-le-g-ccalculate-me<tasks.txt|
paexec_reorder-Mf-Sl
# GNU Parallel cannot stop processing jobs that depend on another.
# It can either try all:
1$tsorttasks.txt|
parallel--ssh'/usr/bin/ssh -x'-S"host1,host2,host3"\
--tagstring{#} --pipe -N1 --log my.log calculate-me
# Or it can stop at the first failing:
1$tsorttasks.txt|
parallel--ssh'/usr/bin/ssh -x'-S"host1,host2,host3"\
--tagstring{#} --halt now,fail=1 --pipe -N1 --log my.log calculate-me
# To retry the the failed and missing tasks:
1$tsorttasks.txt|
parallel--ssh'/usr/bin/ssh -x'-S"host1,host2,host3"\
--tagstring{#} --halt now,fail=1 --pipe -N1 --joblog my.log \
--resume-failedcalculate-me
2$ls-1*.wav|paexec-x-n+4-c'oggenc -Q'
2$ls-1*.wav|parallel-j4oggenc-Q
3$ls-1*.wav|paexec-xCil-n+4flac-f--silent
3$ls-1*.wav|parallel--tagstring{#} -j4 'echo {}; flac -f --silent {}'
4${uname-s;uname-r;uname-m;}|
paexec-x-lp-n+2-cbanner|
paexec_reorder-l
4${uname-s;uname-r;uname-m;}|
parallel--tagstring'{#}'-k\
'banner {} | perl -pe "s/^/getppid().\" \"/e"'
5$find.-name'*.dat'-print0|
paexec-0-n+10-C-J//scp//remoteserver:/remote/path
5$find.-name'*.dat'-print0|
parallel-0-j10-I//scp//remoteserver:/remote/path
6$ls-1*.txt|paexec-n+10-J%%-c'awk "BEGIN {print toupper(\"%%\")}"'
6$ls-1*.txt|parallel-j10-I%%'awk "BEGIN {print toupper(\"%%\")}"'

7$ls-1*.wav|\
paexec-x-c'flac -s'-n+4>/dev/null
7$ls-1*.wav|\
parallel-j4flac-s>/dev/null
8$cat~/bin/toupper
#!/usr/bin/awk -f
{
print" ",toupper(0ドル)
print""# empty line -- end-of-task marker!
fflush()# We must flush stdout!
}
cattasks
apple
bananas
orange
paexec-tssh-c~/bin/toupper-n'server1 server2'<tasks
8$parallel--pipe-n1-Sserver1,server2~/bin/toupper<tasks
9$paexec-lr-tssh-c~/bin/toupper-n'server1 server2'<tasks
9$# GNU Parallel has no easy way to prepend the server
parallel--tagstring{#} --pipe -n1 -S server1,server2 ~/bin/toupper < tasks
cattasks|parallel--tagstring{#} --pipe -n1 -S server1,server2 --plus \
~/bin/toupper'| perl -pe "s/^/{sshlogin}/"'<tasks
10$paexec-n+4-c~/bin/toupper<tasks
10$parallel-j4--pipe-n1~/bin/toupper<tasks
11$paexec-x-tssh-n'server1 server2'\
-c"awk 'BEGIN {print toupper(ARGV[1])}' "<tasks
11$parallel-S'server1,server2'\
"awk 'BEGIN {print toupper(ARGV[1])}'"<tasks
12$paexec-x-C-tssh-n'server1 server2'\
awk'BEGIN {print toupper(ARGV[1])}'<tasks
12$parallel-S'server1,server2'-q\
awk'BEGIN {print toupper(ARGV[1])}'<tasks
13$paexec-Z240-x-tssh-n'server1 badhostname server2'\
-c"awk 'BEGIN {print toupper(ARGV[1])}' "<tasks
13$parallel--filter-hosts-S'server1,badhostname,server2'\
"awk 'BEGIN {print toupper(ARGV[1])}' "<tasks
14$cat~/bin/pbanner
#!/usr/bin/env sh
whilereadtask;do
banner-fM"$task"|pv-qL300
echo"$PAEXEC_EOT"# end-of-task marker
done
cattasks
pae
xec
paexec-l-mt='SE@X-L0S0!&'-c~/bin/pbanner-n+2<tasks|
paexec_reorder-mt='SE@X-L0S0!&'
14$paexec-y-lc~/bin/pbanner-n+2<tasks|paexec_reorder-y
14$paexec-l-x-cbanner-n+2<tasks
14$parallel--pipe-n1-j2~/bin/pbanner<tasks
16$cat~/tmp/packages_to_build
audio/cd-discidaudio/abcde
textproc/gsedaudio/abcde
audio/cdparanoiaaudio/abcde
audio/id3v2audio/abcde
audio/id3audio/abcde
misc/mkcueaudio/abcde
shells/bashaudio/abcde
devel/libtool-baseaudio/cdparanoia
devel/gmakeaudio/cdparanoia
devel/libtool-baseaudio/id3lib
devel/gmakeaudio/id3v2
audio/id3libaudio/id3v2
devel/m4devel/bison
lang/f2cdevel/libtool-base
devel/gmakemisc/mkcue
devel/bisonshells/bash
cat~/bin/pkg_builder
#!/usr/bin/awk -f
{
print"build "0ドル
print"success"# build succeeded! (paexec -ms=)
print""# end-of-task marker
fflush()# we must flush stdout
}
paexec-g-l-c~/bin/pkg_builder-n'server2 server1'\
-tssh<~/tmp/packages_to_build|paexec_reorder
# GNU Parallel cannot postpone jobs that depend on another.
# In some cases this will work
16$tsort~/tmp/packages_to_build|parallel-Sserver2,server1\
--pipe-n1~/bin/pkg_builder
17$cat~/bin/pkg_builder
#!/usr/bin/awk -f
{
print"build "0ドル
if(0ドル=="devel/gmake")
print"failure"# Oh no...
exit255# Exit value needed for GNU Parallel
else
print"success"# build succeeded!
print""# end-of-task marker
fflush()# we must flush stdout
}
paexec-gl-c~/bin/pkg_builder-n'server2 server1'\
-tssh<~/tmp/packages_to_build|paexec_reorder
# GNU Parallel cannot refrain from starting jobs, that depend on others
# In some cases this will work
17$tsort~/tmp/packages_to_build|parallel-Sserver2,server1\
--haltnow,fail=1--pipe-n1~/bin/pkg_builder
18$cat~/bin/pkg_builder
#!/usr/bin/awk -f
{
"hostname -s"|getlinehostname
print"build "0ドル" on "hostname
if(hostname=="server1"&&0ドル=="textproc/gsed")
exit139
# Damn it, I'm dying...
# Exit value is needed by GNU Parallel
else
print"success"# Yes! :-)
print""# end-of-task marker
fflush()# we must flush stdout
}
paexec-gl-Z300-tssh-c~/bin/pkg_builder\
-n'server2 server1'<~/tmp/packages_to_build|
paexec_reorder>result
# GNU Parallel retries a job on another server, if --retries > 1
17$tsort~/tmp/packages_to_build|parallel-Sserver2,server1\
--haltnow,fail=1--retries2--pipe-n1~/bin/pkg_builder
18$ls-1*.wav|paexec-x-c'flac -s'-n+3>/dev/null
18$ls-1*.wav|parallel-j3flac-s>/dev/null
19$ls-1*.wav|paexec-ixC-n+3oggenc-Q|grep.
19$ls-1*.wav|parallel-j3'echo {}; oggenc -Q {}'|grep.
20$catcalc
#!/bin/sh
# 1ドル -- task given on input
iftest1ドル=huge;then
sleep6
else
sleep1
fi
echo"task 1ドル done"
printf'small1\nsmall2\nsmall3\nsmall4\nsmall5\nhuge\n'|
time-ppaexec-c~/bin/calc-n+2-xg|grep-vsuccess
20$printf'small1\nsmall2\nsmall3\nsmall4\nsmall5\nhuge\n'|
time-pparallel-j2~/bin/calc|grep-vsuccess
21$printf'small1\nsmall2\nsmall3\nsmall4\nweight: huge 6\n'|
time-ppaexec-c~/bin/calc-n+2-x-W1|grep-vsuccess
21$# GNU Parallel does not support weighted jobs.
# It can be simulated by sorting:
printf'small1\nsmall2\nsmall3\nsmall4\nweight: huge 6\n'|
perl-pe's/^weight: (.*) (\d+)/2ドル 1ドル/ or s/^/1 /'|
sort-nr|timeparallel~/bin/calc'{=s/^\d* //=}'|
grep-vsuccess

$paexec-lpe-c"`pwd`/cmd"-n+3<<EOF
$ parallel -j3 --pipe -n1 --tagstring {#} \
 './cmd | perl -pe "s/^/getppid().\" \"/e"' <<EOF

$paexec-c"env CC=gcc CFLAGS=-O2 `pwd`/cmd"\
-n'host1 host2'\
-t'/usr/bin/ssh -x'<<EOF
$ parallel --pipe -n1 -S 'host1,host2' \
 "env CC=gcc CFLAGS=-O2 `pwd`/cmd" <<EOF
# This is not exactly the same, but avoids the wrapper
$parallel-Shost1,host2gcc-O2-c-o{.}.o{}<<EOF

$ls-1$PWD/*.c|paexec-c"env $CC$CFLAGS -c "-n+4-x
$ls-1$PWD/*.c|parallel-j4"env $CC$CFLAGS -c"

$paexec-gx-l-c'test -d'-md=';'-n+3<tasks
# GNU Parallel cannot refrain from starting jobs, that depend on others
$parallel-j3--tagtest-d'{= s/.*;// =}; echo $?'<tasks

$paexec-s-l-ccmd_divide-n+3<<EOF
# GNU Parallel cannot refrain from starting jobs, that depend on others
$ parallel -j3 --pipe -n1 cmd_divide <<EOF

1$paexec-g-le-c"`pwd`/cmd"-n+3<tasks|paexec_reorder-g-Ms
# GNU Parallel cannot refrain from starting jobs, that depend on others
1$cat<tasks|parallel--pipe-n1-j3"`pwd`/cmd"
2$paexec-g-le-c"`pwd`/cmd"-n+3<tasks_cycle
2$tsort<tasks_cycle|parallel--pipe-n1-j3"`pwd`/cmd"

$input|paexec-c"`pwd`/cmd"-n+2|cut-b2-
$input|parallel--pipe-n1-j2"`pwd`/cmd"|cut-b2-
$# Without the wrapper:
input|parallel--pipe-n1-j2'awk {print\ toupper\(\0ドル\)}'

$ls-1"$dir"/*.wav|paexec-x-c'flac --silent'-n+"$num"
$ls-1"$dir"/*.wav|parallel-j$numflac--silent

DIFFERENCES BETWEEN map(sitaramc) AND GNU Parallel

Summary (see legend above):

• I1 - - I4 - - (I7)

• M1 (M2) M3 (M4) M5 M6

• - O2 O3 - O5 - - x x O10

• E1 - - - - - -

• - - - - - - - - -

• - -

(I7): Only under special circumstances. See below.

(M2+M4): Only if there is a single replacement string.

map rejects input with special characters:

echo"The Cure">My\ brother\'s\ 12\"\ records
ls|map'echo %; wc %'

It works with GNU parallel:

ls|parallel'echo {}; wc {}'

Under some circumstances it also works with map:

ls|map'echo % works %'

But tiny changes make it reject the input with special characters:

ls|map'echo % does not work "%"'

This means that many UTF-8 characters will be rejected. This is by design. From the web page: "As such, programs that quietly handle them, with no warnings at all, are doing their users a disservice."

map delays each job by 0.01 s. This can be emulated by using parallel --delay 0.01.

map prints '+' on stderr when a job starts, and '-' when a job finishes. This cannot be disabled. parallel has --bar if you need to see progress.

map's replacement strings (% %D %B %E) can be simulated in GNU parallel by putting this in ~/.parallel/config:

--rpl'%'
--rpl'%D $_=Q(::dirname($_));'
--rpl'%B s:.*/::;s:\.[^/.]+$::;'
--rpl'%E s:.*\.::'

map does not have an argument separator on the command line, but uses the first argument as command. This makes quoting harder which again may affect readability. Compare:

map-p2'perl -ne '"'"'/^\S+\s+\S+$/ and print $ARGV,"\n"'"'"*
parallel-qperl-ne'/^\S+\s+\S+$/ and print $ARGV,"\n"':::*

map can do multiple arguments with context replace, but not without context replace:

parallel--xargsecho'BEGIN{'{}'}END':::123
map"echo 'BEGIN{'%'}END'"123

map has no support for grouping. So this gives the wrong results:

parallelperl-e'\$a=\"1{}\"x100000000\;print\ \$a,\"\\n\"''>'{}\
:::abcdef
ls-labcdef
parallel-kP4-n1grep1:::abcdef>out.par
map-n1-p4'grep 1'abcdef>out.map-unbuf
map-n1-p4'grep --line-buffered 1'abcdef>out.map-linebuf
map-n1-p1'grep --line-buffered 1'abcdef>out.map-serial
ls-lout*
md5sumout*

1$ls*.gif|mapconvert%%B.png# default max-args: 1
1$ls*.gif|parallelconvert{}{.}.png
2$map"mkdir %B; tar -C %B -xf %"*.tgz# default max-args: 1
2$parallel'mkdir {.}; tar -C {.} -xf {}':::*.tgz
3$ls*.gif|mapcp%/tmp# default max-args: 100
3$ls*.gif|parallel-Xcp{}/tmp
4$ls*.tar|map-n1tar-xf%
4$ls*.tar|paralleltar-xf
5$map"cp % /tmp"*.tgz
5$parallelcp{}/tmp:::*.tgz
6$map"du -sm /home/%/mail"alicebobcarol
6$parallel"du -sm /home/{}/mail":::alicebobcarol
orifyoupreferrunningasinglejobwithmultipleargs:
6$parallel-Xj1"du -sm /home/{}/mail":::alicebobcarol
7$cat/etc/passwd|map-d:'echo user %1 has shell %7'
7$cat/etc/passwd|parallel--colsep:'echo user {1} has shell {7}'
8$exportMAP_MAX_PROCS=$((`nproc`/2))
8$exportPARALLEL=-j50%

DIFFERENCES BETWEEN ladon AND GNU Parallel

ladon can run multiple jobs on files in parallel.

ladon only works on files and the only way to specify files is using a quoted glob string (such as \*.jpg). It is not possible to list the files manually.

As replacement strings it uses FULLPATH DIRNAME BASENAME EXT RELDIR RELPATH

These can be simulated using GNU parallel by putting this in ~/.parallel/config:

--rpl'FULLPATH $_=Q($_);chomp($_=qx{readlink -f $_});'
--rpl'DIRNAME $_=Q(::dirname($_));chomp($_=qx{readlink -f $_});'
--rpl'BASENAME s:.*/::;s:\.[^/.]+$::;'
--rpl'EXT s:.*\.::'
--rpl'RELDIR $_=Q($_);chomp(($_,$c)=qx{readlink -f $_;pwd});
 s:\Q$c/\E::;$_=::dirname($_);'
--rpl'RELPATH $_=Q($_);chomp(($_,$c)=qx{readlink -f $_;pwd});
 s:\Q$c/\E::;'

ladon deals badly with filenames containing " and newline, and it fails for output larger than 200k:

ladon'*'--seq36000|wc

1$ladon"**/*.txt"--echoRELPATH
1$parallelechoRELPATH:::**/*.txt
2$ladon"~/Documents/**/*.pdf"--shasumFULLPATH>hashes.txt
2$parallelshasumFULLPATH:::~/Documents/**/*.pdf>hashes.txt
3$ladon-mthumbs/RELDIR"**/*.jpg"--convertFULLPATH\
-thumbnail100x100^-gravitycenter-extent100x100\
thumbs/RELPATH
3$parallelmkdir-pthumbs/RELDIR\;convertFULLPATH
-thumbnail100x100^-gravitycenter-extent100x100\
thumbs/RELPATH:::**/*.jpg
4$ladon"~/Music/*.wav"--lame-V2FULLPATHDIRNAME/BASENAME.mp3
4$parallellame-V2FULLPATHDIRNAME/BASENAME.mp3:::~/Music/*.wav

DIFFERENCES BETWEEN jobflow AND GNU Parallel

Summary (see legend above):

• I1 - - - - - I7

• - - M3 - - (M6)

• O1 O2 O3 - O5 O6 (O7) - - O10

• E1 - - - - E6 -

• - - - - - - - - -

• - -

jobflow can run multiple jobs in parallel.

Just like xargs output from jobflow jobs running in parallel mix together by default. jobflow can buffer into files with -buffered (placed in /run/shm), but these are not cleaned up if jobflow dies unexpectedly (e.g. by Ctrl-C). If the total output is big (in the order of RAM+swap) it can cause the system to slow to a crawl and eventually run out of memory.

Just like xargs redirection and composed commands require wrapping with bash -c.

Input lines can at most be 4096 bytes.

jobflow is faster than GNU parallel but around 6 times slower than parallel-bash.

jobflow has no equivalent for --pipe, or --sshlogin.

jobflow makes it possible to set resource limits on the running jobs. This can be emulated by GNU parallel using bash's ulimit:

jobflow-limits=mem=100M,cpu=3,fsize=20M,nofiles=300myjob
parallel'ulimit -v 102400 -t 3 -f 204800 -n 300 myjob'

1$catthings.list|jobflow-threads=8-exec./mytask{}
1$catthings.list|parallel-j8./mytask{}
2$seq100|jobflow-threads=100-exececho{}
2$seq100|parallel-j100echo{}
3$caturls.txt|jobflow-threads=32-execwget{}
3$caturls.txt|parallel-j32wget{}
4$find.-name'*.bmp'|\
jobflow-threads=8-execbmp2jpeg{.}.bmp{.}.jpg
4$find.-name'*.bmp'|\
parallel-j8bmp2jpeg{.}.bmp{.}.jpg
5$seq100|jobflow-skip10-count10
5$seq100|parallel--filter'{1} > 10 and {1} <= 20'echo
5$seq100|parallelecho'{= $_>10 and $_<=20 or skip() =}'

DIFFERENCES BETWEEN gargs AND GNU Parallel

gargs can run multiple jobs in parallel.

Older versions cache output in memory. This causes it to be extremely slow when the output is larger than the physical RAM, and can cause the system to run out of memory.

See more details on this in man parallel_design.

Newer versions cache output in files, but leave files in $TMPDIR if it is killed.

Output to stderr (standard error) is changed if the command fails.

1$seq12-11|gargs-p4-n3"sleep {0}; echo {1} {2}"
1$seq12-11|parallel-P4-n3"sleep {1}; echo {2} {3}"
2$catt.txt|gargs--sep"\s+"\
-p2"echo '{0}:{1}-{2}' full-line: \'{}\'"
2$catt.txt|parallel--colsep"\\s+"\
-P2"echo '{1}:{2}-{3}' full-line: \'{}\'"

DIFFERENCES BETWEEN orgalorg AND GNU Parallel

orgalorg can run the same job on multiple machines. This is related to --onall and --nonall.

orgalorg supports entering the SSH password - provided it is the same for all servers. GNU parallel advocates using ssh-agent instead, but it is possible to emulate orgalorg's behavior by setting SSHPASS and by using --ssh "sshpass ssh".

To make the emulation easier, make a simple alias:

aliaspar_emul="parallel -j0 --ssh 'sshpass ssh' --nonall --tag --lb"

If you want to supply a password run:

SSHPASS=`ssh-askpass`

or set the password directly:

SSHPASS=P4$$w0rd!

If the above is set up you can then do:

orgalorg-ofrontend1-ofrontend2-p-Cuptime
par_emul-Sfrontend1-Sfrontend2uptime
orgalorg-ofrontend1-ofrontend2-p-Ctop-bid1
par_emul-Sfrontend1-Sfrontend2top-bid1
orgalorg-ofrontend1-ofrontend2-p-er/tmp-n\
'md5sum /tmp/bigfile'-Sbigfile
par_emul-Sfrontend1-Sfrontend2--basefilebigfile\
--workdir/tmpmd5sum/tmp/bigfile

orgalorg has a progress indicator for the transferring of a file. GNU parallel does not.

https://github.com/reconquest/orgalorg (Last checked: 2016-08)

DIFFERENCES BETWEEN Rust parallel(mmstick) AND GNU Parallel

Rust parallel focuses on speed. It is almost as fast as xargs, but not as fast as parallel-bash. It implements a few features from GNU parallel, but lacks many functions. All these fail:

# Read arguments from file
parallel-afileecho
# Changing the delimiter
parallel-d_echo:::a_b_c_

These do something different from GNU parallel

# -q to protect quoted $ and space
parallel-qperl-e'$a=shift; print "$a"x100000000':::abc
# Generation of combination of inputs
parallelecho{1}{2}:::redgreenblue:::SMLXLXXL
# {= perl expression =} replacement string
parallelecho'{= s/new/old/ =}':::my.newyour.new
# --pipe
seq100000|parallel--pipewc
# linked arguments
parallelecho:::SML:::+smlmedlrg:::RGB:::+redgrnblu
# Run different shell dialects
zsh-c'parallel echo \={} ::: zsh && true'
csh-c'parallel echo \$\{\} ::: shell && true'
bash-c'parallel echo \$\({}\) ::: pwd && true'
# Rust parallel does not start before the last argument is read
(seq10;sleep5;echo2)|timeparallel-j2'sleep 2; echo'
tail-f/var/log/syslog|parallelecho

Most of the examples from the book GNU Parallel 2018 do not work, thus Rust parallel is not close to being a compatible replacement.

Rust parallel has no remote facilities.

It uses /tmp/parallel for tmp files and does not clean up if terminated abruptly. If another user on the system uses Rust parallel, then /tmp/parallel will have the wrong permissions and Rust parallel will fail. A malicious user can setup the right permissions and symlink the output file to one of the user's files and next time the user uses Rust parallel it will overwrite this file.

attacker$mkdir/tmp/parallel
attacker$chmoda+rwX/tmp/parallel
# Symlink to the file the attacker wants to zero out
attacker$ln-s~victim/.important-file/tmp/parallel/stderr_1
victim$seq1000|parallelecho
# This file is now overwritten with stderr from 'echo'
victim$cat~victim/.important-file

If /tmp/parallel runs full during the run, Rust parallel does not report this, but finishes with success - thereby risking data loss.

https://github.com/mmstick/parallel (Last checked: 2016-08)

DIFFERENCES BETWEEN parallelion AND GNU Parallel

Summary (see legend above):

• - (I2) - I4 - - -

• M1 - M3 - - M6

• - O2 O3 - O5 (O6) - x x

• E1 - - (E4) E5 - - E8 ?

• - - - - - - - - -

• - -

I2: I was unable to cannot get parallelion to read from a file.

O6: There is extra output if a job fails.

E4: The default number of parallel jobs is the number of cpu threads.

-- is needed to force args not be parsed as options:

parallelion'echo {}'--Runswithout-v
parallelion'echo {}'Runswith-v

The commands are run through ion shell.

Ctrl-C does not stop processing.

The --log is similar to syslog - not a table.

The progressbar is nice.

parallelion is fast: 0.1 ms/job. Similar to parallel-bash.

1$parallelion--progress'echo {}'{1..1000}
1$parallel--barecho{}:::{1..1000}
2$parallelion--progress'echo {}'$(seq1999)
2$seq1999|parallel--barecho

DIFFERENCES BETWEEN Rush AND GNU Parallel

rush (https://github.com/shenwei356/rush) is written in Go and based on gargs.

Just like GNU parallel rush buffers in temporary files. But opposite GNU parallel rush does not clean up, if the process dies abnormally.

rush has some string manipulations that can be emulated by putting this into ~/.parallel/config (/ is used instead of %, and % is used instead of ^ as that is closer to bash's ${var%postfix}):

--rpl '{:} s:(\.[^/]+)*$::'
--rpl '{:%([^}]+?)} s:$1ドル(\.[^/]+)*$::'
--rpl '{/:%([^}]*?)} s:.*/(.*)$1ドル(\.[^/]+)*$:1ドル:'
--rpl '{/:} s:(.*/)?([^/.]+)(\.[^/]+)*$:2ドル:'
--rpl '{@(.*?)} /$1ドル/ and $_=1ドル;'

1$seq13|rushecho{}
1$seq13|parallelecho{}

2$rushecho{}-idata1.txt-idata2.txt
2$catdata1.txtdata2.txt|parallelecho{}

3$seq13|rush'echo {}'-k
3$seq13|parallel-kecho{}

4$timeseq1|rush'sleep 2; echo {}'-t1
4$timeseq1|parallel--timeout1'sleep 2; echo {}'

5$seq1|rush'python unexisted_script.py'-r1
5$seq1|parallel--retries2'python unexisted_script.py'

5$seq1|parallel-u--retries2'python unexisted_script.py'

6$echodir/file_1.txt.gz|rush'echo {/} {%} {^_1.txt.gz}'
6$echodir/file_1.txt.gz|
parallel--plusecho{//}{/}{%_1.txt.gz}

7$echodir.d/file.txt.gz|rush'echo {.} {:} {%.} {%:}'
7$echodir.d/file.txt.gz|parallel'echo {.} {:} {/.} {/:}'

8$echo12file.txtdir/s_1.fq.gz|
rush'echo job {#}: {2} {2.} {3%:^_1}'
8$echo12file.txtdir/s_1.fq.gz|
parallel--colsep' ''echo job {#}: {2} {2.} {3/:%_1}'

9$echoread_1.fq.gz|rush'echo {@(.+)_\d}'
9$echoread_1.fq.gz|parallel'echo {@(.+)_\d}'

10$echoa=b=c|rush'echo {1} {2} {3}'-d=
10$echoa=b=c|parallel-d=echo{1}{2}{3}

11$seq5|rush-n2-k'echo "{}"; echo'
11$seq5|
parallel-n2-k\
'echo {=-1 $_=join"\n",@arg[1..$#arg] =}; echo'
11$seq5|rush-n2-k'echo "{}"; echo'-J' '
11$seq5|parallel-n2-k'echo {}; echo'

12$echoabcd|rush-D" "-k'echo {}'
12$echoabcd|parallel-d" "-k'echo {}'
12$echoabcd|rush-D""-k'echo {}'
CannotbedonebyGNUParallel
12$catfasta.fa
>seq1
tag
>seq2
cat
gat
>seq3
attac
a
cat
12$catfasta.fa|rush-D">"\
'echo FASTA record {#}: name: {1} sequence: {2}'-k-d"\n"
# rush fails to join the multiline sequences
12$catfasta.fa|(read-n1ignore_first_char;
parallel-d'>'--colsep'\n'echoFASTArecord{#}: \
name:{1}sequence:'{=2 $_=join"",@arg[2..$#arg]=}'
)

13$seq1|
rush'echo Hello, {fname} {lname}!'-vfname=Wei-vlname=Shen
13$seq1|
parallel-N0\
'fname=Wei; lname=Shen; echo Hello, ${fname} ${lname}!'
13$forvarinab;do\
13$seq13|rush-k-vvar=$var'echo var: {var}, data: {}';\
13$done

13$seq13|parallel-kechovar:{1},data:{2}:::ab::::-

13$seq13|
parallel-kvar={1}';echo var: $var, data: {}':::ab::::-

13$forvarinab;do
exportvar;
seq13|parallel-k'echo var: $var, data: {}';
done

My brother's 12" records

14$# naive way
echoread_1.fq.gz|rush'echo {:^_1} {:^_1}_2.fq.gz'
14$echoread_1.fq.gz|parallel'echo {:%_1} {:%_1}_2.fq.gz'
14$# macro + removing suffix
echoread_1.fq.gz|
rush-vp='{:^_1}''echo {p} {p}_2.fq.gz'
14$echoread_1.fq.gz|
parallel'p={:%_1}; echo $p ${p}_2.fq.gz'
14$# macro + regular expression
echoread_1.fq.gz|rush-vp='{@(.+?)_\d}''echo {p} {p}_2.fq.gz'
14$echoread_1.fq.gz|parallel'p={@(.+?)_\d}; echo $p ${p}_2.fq.gz'

14$echo"My brother's 12\"read_1.fq.gz"|
parallel'p={@(.+?)_\d}; echo $p ${p}_2.fq.gz'

15$seq120|rush'sleep 1; echo {}'
^C
15$seq120|parallel'sleep 1; echo {}'
^C

16$seq13|rush'sleep {}; echo {}'-t3-c
catsuccessful_cmds.rush
seq13|rush'sleep {}; echo {}'-t3-c
16$seq13|parallel--joblogmylog--timeout2\
'sleep {}; echo {}'
catmylog
seq13|parallel--joblogmylog--retry-failed\
'sleep {}; echo {}'

16$seq13|rush'sleep {}; echo {}; \
 echo finish {}'-t3-c-Cfinished.rush
catfinished.rush
seq13|rush'sleep {}; echo {}; \
 echo finish {}'-t3-c-Cfinished.rush
16$seq13|
parallel--joblogmylog--timeout2'sleep {}; echo {}; \
 echo finish {}'
catmylog
seq13|
parallel--joblogmylog--retry-failed'sleep {}; echo {}; \
 echo finish {}'

17$forfin$(seq20142016);do\
/bin/rm-rf$f;mkdir-p$f;\
cat$f.html.txt|rush-vd=$f-d=\
'phantomjs save_page.js "{}" > {d}/{3}.html'\
-j20-t60-r3-c;\
done

17$rmmylog
forfin$(seq20142016);do
/bin/rm-rf$f;mkdir-p$f;
cat$f.html.txt|
parallel-j20--timeout60--retries4--joblog+mylog\
--colsep=\
phantomjssave_page.js{1}={2}={3}'>'$f/{3}.html
done

18$ref=ref/xxx.fa
threads=25
ls-draw.cluster.clean.mapping/*\
|rush-vref=$ref-vj=$threads-vp='{}/{%}'\
'bwa mem -t {j} -M -a {ref} {p}_1.fq.gz {p}_2.fq.gz >{p}.sam;\
 samtools view -bS {p}.sam > {p}.bam; \
 samtools sort -T {p}.tmp -@ {j} {p}.bam -o {p}.sorted.bam; \
 samtools index {p}.sorted.bam; \
 samtools flagstat {p}.sorted.bam > {p}.sorted.bam.flagstat; \
 /bin/rm {p}.bam {p}.sam;'\
-j2--verbose-c-Cmapping.rush

18$ref=ref/xxx.fa
exportref
thr=25
exportthr
bwa_sam(){
p="1ドル"
bam="$p".bam
sam="$p".sam
sortbam="$p".sorted.bam
bwamem-t$thr-M-a$ref${p}_1.fq.gz${p}_2.fq.gz>"$sam"
samtoolsview-bS"$sam">"$bam"
samtoolssort-T${p}.tmp-@$thr"$bam"-o"$sortbam"
samtoolsindex"$sortbam"
samtoolsflagstat"$sortbam">"$sortbam".flagstat
/bin/rm"$bam""$sam"
}
export-fbwa_sam
ls-draw.cluster.clean.mapping/*|
parallel-j2--verbose--joblogmylogbwa_sam

DIFFERENCES BETWEEN ClusterSSH AND GNU Parallel

ClusterSSH solves a different problem than GNU parallel.

ClusterSSH opens a terminal window for each computer and using a master window you can run the same command on all the computers. This is typically used for administrating several computers that are almost identical.

GNU parallel runs the same (or different) commands with different arguments in parallel possibly using remote computers to help computing. If more than one computer is listed in -S GNU parallel may only use one of these (e.g. if there are 8 jobs to be run and one computer has 8 cores).

GNU parallel can be used as a poor-man's version of ClusterSSH:

parallel--nonall-Sserver-a,server-bdo_stufffoobar

https://github.com/duncs/clusterssh (Last checked: 2024-06)

DIFFERENCES BETWEEN coshell AND GNU Parallel

coshell only accepts full commands on standard input. Any quoting needs to be done by the user.

Commands are run in sh so any bash/tcsh/zsh specific syntax will not work.

Output can be buffered by using -d. Output is buffered in memory, so big output can cause swapping and therefore be terrible slow or even cause out of memory.

https://github.com/gdm85/coshell (Last checked: 2019-01)

DIFFERENCES BETWEEN spread AND GNU Parallel

Summary (see legend above):

• - - - I4 - - I7

• M1 - - - - -

• O1 O2 O3 O4 O5 O6 - O8 - O10

• - - - - - - -

• - - - - - - - - -

• - -

spread runs commands on all directories. It does not run jobs in parallel.

It can be emulated with GNU parallel using this Bash function:

spread(){
_cmds(){
perl-e'$"=" && ";print "@ARGV"'"cd {}""$@"
}
parallel$(_cmds"$@")'|| echo exit status $?':::*/
}

https://github.com/tfogo/spread (Last checked: 2024-06)

DIFFERENCES BETWEEN pyargs AND GNU Parallel

pyargs deals badly with input containing spaces. It buffers stdout, but not stderr. It buffers in RAM. {} does not work as replacement string. It does not support running functions.

pyargs does not support composed commands if run with --lines, and fails on pyargs traceroute gnu.org fsf.org.

seq5|pyargs-P50-Lseq
seq5|parallel-P50--lbseq
seq5|pyargs-P50--mark-Lseq
seq5|parallel-P50--lb\
--tagstringOUTPUT'[{= $_=$job->replaced() =}]'seq
# Similar, but not precisely the same
seq5|parallel-P50--lb--tagseq
seq5|pyargs-P50--markcommand
# Somewhat longer with GNU Parallel due to the special
# --mark formatting
cmd="$(echo"command"|parallel--shellquote)"
wrap_cmd(){
echo"MARK $cmd$@================================">&3
echo"OUTPUT START[$cmd$@]:"
eval$cmd"$@"
echo"OUTPUT END[$cmd$@]"
}
(seq5|env_parallel-P2wrap_cmd)3>&1
# Similar, but not exactly the same
seq5|parallel-t--tagcommand
(echo'1 2 3';echo456)|pyargs--streamseq
(echo'1 2 3';echo456)|perl-pe's/\n/ /'|
parallel-r-d' 'seq
# Similar, but not exactly the same
parallelseq:::123456

DIFFERENCES BETWEEN concurrently AND GNU Parallel

concurrently runs jobs in parallel.

The output is prepended with the job number, and may be incomplete:

$concurrently'seq 100000'|(sleep3;wc-l)
7165

When pretty printing it caches output in memory. Output mixes by using test MIX below whether or not output is cached.

There seems to be no way of making a template command and have concurrently fill that with different args. The full commands must be given on the command line.

There is also no way of controlling how many jobs should be run in parallel at a time - i.e. "number of jobslots". Instead all jobs are simply started in parallel.

https://github.com/kimmobrunfeldt/concurrently (Last checked: 2019-01)

DIFFERENCES BETWEEN map(soveran) AND GNU Parallel

map does not run jobs in parallel by default. The README suggests using:

...|mapt'sleep $t && say done &'

But this fails if more jobs are run in parallel than the number of available processes. Since there is no support for parallelization in map itself, the output also mixes:

seq10|mapi'echo start-$i && sleep 0.$i && echo end-$i &'

The major difference is that GNU parallel is built for parallelization and map is not. So GNU parallel has lots of ways of dealing with the issues that parallelization raises:

• Keep the number of processes manageable

• Make sure output does not mix

• Make Ctrl-C kill all running processes

1$ls*.c|mapf'foo $f'
1$ls*.c|parallelfoo
2$ls*.c|mapf'foo $f; bar $f'
2$ls*.c|parallel'foo {}; bar {}'
3$caturls|mapu'curl -O $u'
3$caturls|parallelcurl-O
4$printf"1\n1\n1\n"|mapt'sleep $t && say done'
4$printf"1\n1\n1\n"|parallel'sleep {} && say done'
4$parallel'sleep {} && say done':::111
5$printf"1\n1\n1\n"|mapt'sleep $t && say done &'
5$printf"1\n1\n1\n"|parallel-j0'sleep {} && say done'
5$parallel-j0'sleep {} && say done':::111

DIFFERENCES BETWEEN loop AND GNU Parallel

loop mixes stdout and stderr:

loop'ls /no-such-file'>/dev/null

loop's replacement string $ITEM does not quote strings:

echo'two spaces'|loop'echo $ITEM'

loop cannot run functions:

myfunc(){echojoe;}
export-fmyfunc
loop'myfunc this fails'

# A couple of functions will make the code easier to read
$loopy(){
yes|parallel-uN0-j1"$@"
}
$export-floopy
$time_out(){
parallel-uN0-q--timeout"$@":::1
}
$match(){
perl-0777-ne'grep /'"1ドル"'/,$_ and print or exit 1'
}
$export-fmatch
$loop'ls'--every10s
$loopy--delay10sls
$loop'touch $COUNT.txt'--count-by5
$loopytouch'{= $_=seq()*5 =}'.txt
$loop--until-contains200--\
./get_response_code.sh--sitemysite.biz
$loopy--haltnow,success=1\
'./get_response_code.sh --site mysite.biz | match 200'
$loop'./poke_server'--for-duration8h
$time_out8hloopy./poke_server
$loop'./poke_server'--until-success
$loopy--haltnow,success=1./poke_server
$catfiles_to_create.txt|loop'touch $ITEM'
$catfiles_to_create.txt|paralleltouch{}
$loop'ls'--for-duration10min--summary
# --joblog is somewhat more verbose than --summary
$time_out10mloopy--joblogmy.log./poke_server;catmy.log
$loop'echo hello'
$loopyechohello
$loop'echo $COUNT'
# GNU Parallel counts from 1
$loopyecho{#}
# Counting from 0 can be forced
$loopyecho'{= $_=seq()-1 =}'
$loop'echo $COUNT'--count-by2
$loopyecho'{= $_=2*(seq()-1) =}'
$loop'echo $COUNT'--count-by2--offset10
$loopyecho'{= $_=10+2*(seq()-1) =}'
$loop'echo $COUNT'--count-by1.1
# GNU Parallel rounds 3.3000000000000003 to 3.3
$loopyecho'{= $_=1.1*(seq()-1) =}'
$loop'echo $COUNT $ACTUALCOUNT'--count-by2
$loopyecho'{= $_=2*(seq()-1) =} {#}'
$loop'echo $COUNT'--num3--summary
# --joblog is somewhat more verbose than --summary
$seq3|parallel--joblogmy.logecho;catmy.log
$loop'ls -foobarbatz'--num3--summary
# --joblog is somewhat more verbose than --summary
$seq3|parallel--joblogmy.log-N0ls-foobarbatz;catmy.log
$loop'echo $COUNT'--count-by2--num50--only-last
# Can be emulated by running 2 jobs
$seq49|parallelecho'{= $_=2*(seq()-1) =}'>/dev/null
$echo50|parallelecho'{= $_=2*(seq()-1) =}'
$loop'date'--every5s
$loopy--delay5sdate
$loop'date'--for-duration8s--every2s
$time_out8sloopy--delay2sdate
$loop'date -u'--until-time'2018年05月25日 20:50:00'--every5s
$seconds=$((`date-d2019-05-25T20:50:00+%s`-`date+%s`))s
$time_out$secondsloopy--delay5sdate-u
$loop'echo $RANDOM'--until-contains"666"
$loopy--haltnow,success=1'echo $RANDOM | match 666'
$loop'if (( RANDOM % 2 )); then
 (echo "TRUE"; true);
 else
 (echo "FALSE"; false);
 fi'--until-success
$loopy--haltnow,success=1'if (( $RANDOM % 2 )); then
 (echo "TRUE"; true);
 else
 (echo "FALSE"; false);
 fi'
$loop'if (( RANDOM % 2 )); then
 (echo "TRUE"; true);
 else
 (echo "FALSE"; false);
 fi'--until-error
$loopy--haltnow,fail=1'if (( $RANDOM % 2 )); then
 (echo "TRUE"; true);
 else
 (echo "FALSE"; false);
 fi'
$loop'date'--until-match"(\d{4})"
$loopy--haltnow,success=1'date | match [0-9][0-9][0-9][0-9]'
$loop'echo $ITEM'--forred,green,blue
$parallelecho:::redgreenblue
$cat/tmp/my-list-of-files-to-create.txt|loop'touch $ITEM'
$cat/tmp/my-list-of-files-to-create.txt|paralleltouch
$ls|loop'cp $ITEM $ITEM.bak';ls
$ls|parallelcp{}{}.bak;ls
$loop'echo $ITEM | tr a-z A-Z'-i
$parallel'echo {} | tr a-z A-Z'
# Or more efficiently:
$parallel--pipetra-zA-Z
$loop'echo $ITEM'--for"`ls`"
$parallelecho{}:::"`ls`"
$ls|loop'./my_program $ITEM'--until-success;
$ls|parallel--haltnow,success=1./my_program{}
$ls|loop'./my_program $ITEM'--until-fail;
$ls|parallel--haltnow,fail=1./my_program{}
$./deploy.sh;
loop'curl -sw "%{http_code}" http://coolwebsite.biz'\
--every5s--until-contains200;
./announce_to_slack.sh
$./deploy.sh;
loopy--delay5s--haltnow,success=1\
'curl -sw "%{http_code}" http://coolwebsite.biz | match 200';
./announce_to_slack.sh
$loop"ping -c 1 mysite.com"--until-success;./do_next_thing
$loopy--haltnow,success=1ping-c1mysite.com;./do_next_thing
$./create_big_file-omy_big_file.bin;
loop'ls'--until-contains'my_big_file.bin';
./upload_big_filemy_big_file.bin
# inotifywait is a better tool to detect file system changes.
# It can even make sure the file is complete
# so you are not uploading an incomplete file
$inotifywait-qmreMOVED_TO-eCLOSE_WRITE--format%w%f.|
grepmy_big_file.bin
$ls|loop'cp $ITEM $ITEM.bak'
$ls|parallelcp{}{}.bak
$loop'./do_thing.sh'--every15s--until-success--num5
$parallel--retries5--delay15s:::./do_thing.sh

DIFFERENCES BETWEEN lorikeet AND GNU Parallel

lorikeet can run jobs in parallel. It does this based on a dependency graph described in a file, so this is similar to make.

https://github.com/cetra3/lorikeet (Last checked: 2018-10)

DIFFERENCES BETWEEN spp AND GNU Parallel

spp can run jobs in parallel. spp does not use a command template to generate the jobs, but requires jobs to be in a file. Output from the jobs mix.

https://github.com/john01dav/spp (Last checked: 2024-06)

DIFFERENCES BETWEEN paral AND GNU Parallel

paral prints a lot of status information and stores the output from the commands run into files. This means it cannot be used the middle of a pipe like this

paral"echo this""echo does not""echo work"|wc

Instead it puts the output into files named like out_#_*command*.out.log. To get a very similar behaviour with GNU parallel use --results 'out_{#}_{=s/[^\sa-z_0-9]//g;s/\s+/_/g=}.log' --eta

paral only takes arguments on the command line and each argument should be a full command. Thus it does not use command templates.

This limits how many jobs it can run in total, because they all need to fit on a single command line.

paral has no support for running jobs remotely.

1$paral"command 1""command 2 --flag""command arg1 arg2"
1$parallel:::"command 1""command 2 --flag""command arg1 arg2"
2$paral"sleep 1 && echo c1""sleep 2 && echo c2"\
"sleep 3 && echo c3""sleep 4 && echo c4""sleep 5 && echo c5"
2$parallel:::"sleep 1 && echo c1""sleep 2 && echo c2"\
"sleep 3 && echo c3""sleep 4 && echo c4""sleep 5 && echo c5"
# Or shorter:
parallel"sleep {} && echo c{}":::{1..5}
3$paral-n=0"sleep 5 && echo c5""sleep 4 && echo c4"\
"sleep 3 && echo c3""sleep 2 && echo c2""sleep 1 && echo c1"
3$parallel:::"sleep 5 && echo c5""sleep 4 && echo c4"\
"sleep 3 && echo c3""sleep 2 && echo c2""sleep 1 && echo c1"
# Or shorter:
parallel-j0"sleep {} && echo c{}":::54321
4$paral-n=1"sleep 5 && echo c5""sleep 4 && echo c4"\
"sleep 3 && echo c3""sleep 2 && echo c2""sleep 1 && echo c1"
4$parallel-j1"sleep {} && echo c{}":::54321
5$paral-n=2"sleep 5 && echo c5""sleep 4 && echo c4"\
"sleep 3 && echo c3""sleep 2 && echo c2""sleep 1 && echo c1"
5$parallel-j2"sleep {} && echo c{}":::54321
6$paral-n=5"sleep 5 && echo c5""sleep 4 && echo c4"\
"sleep 3 && echo c3""sleep 2 && echo c2""sleep 1 && echo c1"
6$parallel-j5"sleep {} && echo c{}":::54321
7$paral-n=1"echo a && sleep 0.5 && echo b && sleep 0.5 && \
 echo c && sleep 0.5 && echo d && sleep 0.5 && \
 echo e && sleep 0.5 && echo f && sleep 0.5 && \
 echo g && sleep 0.5 && echo h"
7$parallel:::"echo a && sleep 0.5 && echo b && sleep 0.5 && \
 echo c && sleep 0.5 && echo d && sleep 0.5 && \
 echo e && sleep 0.5 && echo f && sleep 0.5 && \
 echo g && sleep 0.5 && echo h"

DIFFERENCES BETWEEN concurr AND GNU Parallel

concurr is built to run jobs in parallel using a client/server model.

1$concurr'echo job {#} on slot {%}: {}':arg1arg2arg3arg4
1$parallel'echo job {#} on slot {%}: {}':::arg1arg2arg3arg4
2$concurr'echo job {#} on slot {%}: {}'::file1file2file3
2$parallel'echo job {#} on slot {%}: {}'::::file1file2file3
3$concurr'echo {}'<input_file
3$parallel'echo {}'<input_file
4$catfile|concurr'echo {}'
4$catfile|parallel'echo {}'

DIFFERENCES BETWEEN lesser-parallel AND GNU Parallel

lesser-parallel is the inspiration for parallel --embed. Both lesser-parallel and parallel --embed define bash functions that can be included as part of a bash script to run jobs in parallel.

lesser-parallel implements a few of the replacement strings, but hardly any options, whereas parallel --embed gives you the full GNU parallel experience.

https://github.com/kou1okada/lesser-parallel (Last checked: 2024-06)

DIFFERENCES BETWEEN npm-parallel AND GNU Parallel

npm-parallel can run npm tasks in parallel.

There are no examples and very little documentation, so it is hard to compare to GNU parallel.

https://github.com/spion/npm-parallel (Last checked: 2024-06)

DIFFERENCES BETWEEN machma AND GNU Parallel

machma runs tasks in parallel. It gives time stamped output. It buffers in RAM.

1$# Put shorthand for timestamp in config for the examples
echo'--rpl '\
\''{time} $_=::strftime("%Y-%m-%d %H:%M:%S",localtime())'\'\
>~/.parallel/machma
echo'--line-buffer --tagstring "{#} {time} {}"'\
>>~/.parallel/machma
2$find.-iname'*.jpg'|
machma--mogrify-resize1200x1200-filterLanczos{}
find.-iname'*.jpg'|
parallel--bar-Jmachmamogrify-resize1200x1200\
-filterLanczos{}
3$cat/tmp/ips|machma-p2--ping-c2-q{}
3$cat/tmp/ips|parallel-j2-Jmachmaping-c2-q{}
4$cat/tmp/ips|
machma--sh-c'ping -c 2 -q 0ドル > /dev/null && echo alive'{}
4$cat/tmp/ips|
parallel-Jmachma'ping -c 2 -q {} > /dev/null && echo alive'
5$find.-iname'*.jpg'|
machma--timeout5s--mogrify-resize1200x1200\
-filterLanczos{}
5$find.-iname'*.jpg'|
parallel--timeout5s--barmogrify-resize1200x1200\
-filterLanczos{}
6$find.-iname'*.jpg'-print0|
machma--null--mogrify-resize1200x1200-filterLanczos{}
6$find.-iname'*.jpg'-print0|
parallel--null--barmogrify-resize1200x1200\
-filterLanczos{}

DIFFERENCES BETWEEN interlace AND GNU Parallel

Summary (see legend above):

• - I2 I3 I4 - - -

• M1 - M3 - - M6

• - O2 O3 - - - - x x

• E1 E2 - - - - -

• - - - - - - - - -

• - -

interlace is built for network analysis to run network tools in parallel.

interface does not buffer output, so output from different jobs mixes.

The overhead for each target is O(n*n), so with 1000 targets it becomes very slow with an overhead in the order of 500ms/target.

commands.txt:
mkdir-p_output_/_target_/scans/
_blocker_
nmap_target_-oA_output_/_target_/scans/_target_-nmap
interlace-tL./targets.txt-cLcommands.txt-o$output
parallel-atargets.txt\
mkdir-p$output/{}/scans/\;nmap{}-oA$output/{}/scans/{}-nmap

commands.txt:
_block:nmap_
mkdir-p_target_/output/scans/
nmap_target_-oN_target_/output/scans/_target_-nmap
_block:nmap_
nikto--host_target_
interlace-tL./targets.txt-cLcommands.txt
_nmap(){
mkdir-p1ドル/output/scans/
nmap1ドル-oN1ドル/output/scans/1ドル-nmap
}
export-f_nmap
parallel:::_nmap"nikto --host"::::targets.txt

interlace-tL./targets.txt-threads5\
-c"nikto --host _target_ > ./_target_-nikto.txt"-v
parallel-atargets.txt-P5nikto--host{}\>./{}_-nikto.txt

interlace-tL./targets.txt-threads5-c\
"nikto --host _target_:_port_ > ./_target_-_port_-nikto.txt"\
-p80,443-v
parallel-P5nikto--host{1}:{2}\>./{1}-{2}-nikto.txt\
::::targets.txt:::80443

commands.txt:
nikto--host_target_:_port_>_output_/_target_-nikto.txt
sslscan_target_:_port_>_output_/_target_-sslscan.txt
testssl.sh_target_:_port_>_output_/_target_-testssl.txt
interlace-texample.com-o~/Engagements/example/\
-cL./commands.txt-p80,443
parallel--results~/Engagements/example/{2}:{3}{1}{1}{2}:{3}\
:::"nikto --host"sslscantestssl.sh:::example.com:::80443

interlace-t192.168.12.0/24-c"vhostscan _target_ \
 -oN _output_/_target_-vhosts.txt"-o~/scans/-threads50
prips192.168.12.0/24|
parallel-P50vhostscan{}-oN~/scans/{}-vhosts.txt

interlace-t192.168.12.*-c"vhostscan _target_ \
 -oN _output_/_target_-vhosts.txt"-o~/scans/-threads50
# Glob is not supported in prips
prips192.168.12.0/24|
parallel-P50vhostscan{}-oN~/scans/{}-vhosts.txt

interlace-t192.168.12.1-15-c\
"vhostscan _target_ -oN _output_/_target_-vhosts.txt"\
-o~/scans/-threads50
# Dash notation is not supported in prips
prips192.168.12.1192.168.12.15|
parallel-P50vhostscan{}-oN~/scans/{}-vhosts.txt

interlace-tL./target-list.txt-c\
"vhostscan -t _target_ -oN _output_/_target_-vhosts.txt"\
-o~/scans/-threads50
cat./target-list.txt|
parallel-P50vhostscan-t{}-oN~/scans/{}-vhosts.txt

./vhosts-commands.txt:
vhostscan-t$target-oN_output_/_target_-vhosts.txt
interlace-cL./vhosts-commands.txt-tL./target-list.txt\
-threads50-o~/scans
./vhosts-commands.txt:
vhostscan-t"1ドル"-oN"2ドル"
parallel-P50./vhosts-commands.txt{}~/scans/{}-vhosts.txt\
::::./target-list.txt

interlace-t192.168.12.0/24-e192.168.12.0/26-c\
"vhostscan _target_ -oN _output_/_target_-vhosts.txt"\
-o~/scans/-threads50
prips192.168.12.0/24|grep-xv-Ff<(prips192.168.12.0/26)|
parallel-P50vhostscan{}-oN~/scans/{}-vhosts.txt

interlace-tL./targets.txt-pL./proxies.txt-threads5-c\
"nikto --host _target_:_port_ -useproxy _proxy_ > \
 ./_target_-_port_-nikto.txt"-p80,443-v
parallel-j5\
"nikto --host {1}:{2} -useproxy {3} > ./{1}-{2}-nikto.txt"\
::::./targets.txt:::80443::::./proxies.txt

DIFFERENCES BETWEEN otonvm Parallel AND GNU Parallel

I have been unable to get the code to run at all. It seems unfinished.

https://github.com/otonvm/Parallel (Last checked: 2024-06)

DIFFERENCES BETWEEN k-bx par AND GNU Parallel

par requires Haskell to work. This limits the number of platforms this can work on.

par does line buffering in memory. The memory usage is 3x the longest line (compared to 1x for parallel --lb). Commands must be given as arguments. There is no template.

These are the examples from https://github.com/k-bx/par with the corresponding GNU parallel command.

par"echo foo; sleep 1; echo foo; sleep 1; echo foo"\
"echo bar; sleep 1; echo bar; sleep 1; echo bar"&&echo"success"
parallel--lb:::"echo foo; sleep 1; echo foo; sleep 1; echo foo"\
"echo bar; sleep 1; echo bar; sleep 1; echo bar"&&echo"success"
par"echo foo; sleep 1; foofoo"\
"echo bar; sleep 1; echo bar; sleep 1; echo bar"&&echo"success"
parallel--lb--halt1:::"echo foo; sleep 1; foofoo"\
"echo bar; sleep 1; echo bar; sleep 1; echo bar"&&echo"success"
par"PARPREFIX=[fooechoer] echo foo""PARPREFIX=[bar] echo bar"
parallel--lb--colsep,--tagstring{1}{2}\
:::"[fooechoer],echo foo""[bar],echo bar"
par--succeed"foo""bar"&&echo'wow'
parallel"foo""bar";true&&echo'wow'

https://github.com/k-bx/par (Last checked: 2019-02)

DIFFERENCES BETWEEN parallelshell AND GNU Parallel

parallelshell does not allow for composed commands:

# This does not work
parallelshell'echo foo;echo bar''echo baz;echo quuz'

Instead you have to wrap that in a shell:

parallelshell'sh -c "echo foo;echo bar"''sh -c "echo baz;echo quuz"'

It buffers output in RAM. All commands must be given on the command line and all commands are started in parallel at the same time. This will cause the system to freeze if there are so many jobs that there is not enough memory to run them all at the same time.

https://github.com/keithamus/parallelshell (Last checked: 2019-02)

https://github.com/darkguy2008/parallelshell (Last checked: 2019-03)

DIFFERENCES BETWEEN shell-executor AND GNU Parallel

shell-executor does not allow for composed commands:

# This does not work
sx'echo foo;echo bar''echo baz;echo quuz'

Instead you have to wrap that in a shell:

sx'sh -c "echo foo;echo bar"''sh -c "echo baz;echo quuz"'

It buffers output in RAM. All commands must be given on the command line and all commands are started in parallel at the same time. This will cause the system to freeze if there are so many jobs that there is not enough memory to run them all at the same time.

https://github.com/royriojas/shell-executor (Last checked: 2024-06)

DIFFERENCES BETWEEN non-GNU par AND GNU Parallel

par buffers in memory to avoid mixing of jobs. It takes 1s per 1 million output lines.

par needs to have all commands before starting the first job. The jobs are read from stdin (standard input) so any quoting will have to be done by the user.

Stdout (standard output) is prepended with o:. Stderr (standard error) is sendt to stdout (standard output) and prepended with e:.

For short jobs with little output par is 20% faster than GNU parallel and 60% slower than xargs.

https://github.com/UnixJunkie/PAR

https://savannah.nongnu.org/projects/par (Last checked: 2019-02)

DIFFERENCES BETWEEN fd AND GNU Parallel

fd does not support composed commands, so commands must be wrapped in sh -c.

It buffers output in RAM.

It only takes file names from the filesystem as input (similar to find).

https://github.com/sharkdp/fd (Last checked: 2019-02)

DIFFERENCES BETWEEN lateral AND GNU Parallel

lateral is very similar to sem: It takes a single command and runs it in the background. The design means that output from parallel running jobs may mix. If it dies unexpectly it leaves a socket in ~/.lateral/socket.PID.

lateral deals badly with too long command lines. This makes the lateral server crash:

lateralrunecho`seq100000|head-c1000k`

Any options will be read by lateral so this does not work (lateral interprets the -l):

lateralrunls-l

Composed commands do not work:

lateralrunpwd';'ls

Functions do not work:

myfunc(){echoa;}
export-fmyfunc
lateralrunmyfunc

Running emacs in the terminal causes the parent shell to die:

echo'#!/bin/bash'>mycmd
echoemacs-nw>>mycmd
chmod+xmycmd
lateralstart
lateralrun./mycmd

Here are the examples from https://github.com/akramer/lateral with the corresponding GNU sem and GNU parallel commands:

1$lateralstart
foriin$(cat/tmp/names);do
lateralrun--some_command$i
done
lateralwait
1$foriin$(cat/tmp/names);do
semsome_command$i
done
sem--wait
1$parallelsome_command::::/tmp/names
2$lateralstart
foriin$(seq1100);do
lateralrun--my_slow_command<workfile$i>/tmp/logfile$i
done
lateralwait
2$foriin$(seq1100);do
semmy_slow_command<workfile$i>/tmp/logfile$i
done
sem--wait
2$parallel'my_slow_command < workfile{} > /tmp/logfile{}'\
:::{1..100}
3$lateralstart-p0# yup, it will just queue tasks
foriin$(seq1100);do
lateralrun--command_still_outputs_but_wont_spaminputfile$i
done
# command output spam can commence
lateralconfig-p10;lateralwait
3$foriin$(seq1100);do
echo"command inputfile$i">>joblist
done
parallel-j10::::joblist
3$echo1>/tmp/njobs
parallel-j/tmp/njobscommandinputfile{}\
:::{1..100}&
echo10>/tmp/njobs
wait

https://github.com/akramer/lateral (Last checked: 2024-06)

DIFFERENCES BETWEEN with-this AND GNU Parallel

The examples from https://github.com/amritb/with-this.git and the corresponding GNU parallel command:

with-v"$(catmyurls.txt)""curl -L this"
parallelcurl-L:::myurls.txt
with-v"$(catmyregions.txt)"\
"aws --region=this ec2 describe-instance-status"
parallelaws--region={}ec2describe-instance-status\
::::myregions.txt
with-v"$(ls)""kubectl --kubeconfig=this get pods"
ls|parallelkubectl--kubeconfig={}getpods
with-v"$(ls|grepconfig)""kubectl --kubeconfig=this get pods"
ls|grepconfig|parallelkubectl--kubeconfig={}getpods
with-v"$(echo{1..10})""echo 123"
parallel-N0echo123:::{1..10}

Stderr is merged with stdout. with-this buffers in RAM. It uses 3x the output size, so you cannot have output larger than 1/3rd the amount of RAM. The input values cannot contain spaces. Composed commands do not work.

with-this gives some additional information, so the output has to be cleaned before piping it to the next command.

https://github.com/amritb/with-this.git (Last checked: 2024-06)

DIFFERENCES BETWEEN Tollef's parallel (moreutils) AND GNU Parallel

Summary (see legend above):

• - - - I4 - - I7

• - - M3 - - M6

• - O2 O3 - O5 O6 - x x

• E1 - - - - - E7

• - x x x x x x x x

• - -

DIFFERENCES BETWEEN rargs AND GNU Parallel

Summary (see legend above):

• I1 - - - - - I7

• - - M3 M4 - -

• - O2 O3 - O5 O6 - O8 -

• E1 - - E4 - - -

• - - - - - - - - -

• - -

rargs has elegant ways of doing named regexp capture and field ranges.

With GNU parallel you can use --rpl to get a similar functionality as regexp capture gives, and use join and @arg to get the field ranges. But the syntax is longer. This:

--rpl'{r(\d+)\.\.(\d+)} $_=join"$opt::colsep",@arg[$1ドル..$2ドル]'

would make it possible to use:

{1r3..6}

for field 3..6.

For full support of {n..m:s} including negative numbers use a dynamic replacement string like this:

PARALLEL=--rpl\ \''{r((-?\d+)?)\.\.((-?\d+)?)((:([^}]*))?)}
 $a = defined $2ドル ? $2ドル < 0 ? 1+$#arg+$2ドル : $2ドル : 1;
 $b = defined $4ドル ? $4ドル < 0 ? 1+$#arg+$4ドル : $4ドル : $#arg+1;
 $s = defined $6ドル ? $7ドル : " ";
 $_ = join $s,@arg[$a..$b]'\'
exportPARALLEL

You can then do:

head/etc/passwd|parallel--colsep:echo..={1r..}..3={1r..3}\
4..={1r4..}2..4={1r2..4}3..3={1r3..3}..3:-={1r..3:-}\
..3:/={1r..3:/}-1={-1}-5={-5}-6={-6}-3..={1r-3..}

1$ls*.bak|rargs-p'(.*)\.bak'mv{0}{1}
1$ls*.bak|parallelmv{}{.}
2$catdownload-list.csv|
rargs-p'(?P<url>.*),(?P<filename>.*)'wget{url}-O{filename}
2$catdownload-list.csv|
parallel--csvwget{1}-O{2}
# or use regexps:
2$catdownload-list.csv|
parallel--rpl'{url} s/,.*//'--rpl'{filename} s/.*?,//'\
wget{url}-O{filename}
3$cat/etc/passwd|
rargs-d:echo-e'id: "{1}"\t name: "{5}"\t rest: "{6..::}"'
3$cat/etc/passwd|
parallel-q--colsep:\
echo-e'id: "{1}"\t name: "{5}"\t rest: "{=6 $_=join":",@arg[6..$#arg]=}"'

DIFFERENCES BETWEEN threader AND GNU Parallel

Summary (see legend above):

• I1 - - - - - -

• M1 - M3 - - M6

• O1 - O3 - O5 - - x x

• E1 - - E4 - - -

• - - - - - - - - -

• - -

Newline separates arguments, but newline at the end of file is treated as an empty argument. So this runs 2 jobs:

echotwo_jobs|threader-run'echo "$THREADID"'

threader ignores stderr, so any output to stderr is lost. threader buffers in RAM, so output bigger than the machine's virtual memory will cause the machine to crash.

https://github.com/voodooEntity/threader (Last checked: 2024-06)

DIFFERENCES BETWEEN runp AND GNU Parallel

Summary (see legend above):

• I1 I2 - - - - -

• M1 - (M3) - - M6

• O1 O2 O3 - O5 O6 - x x -

• E1 - - - - - -

• - - - - - - - - -

• - -

(M3): You can add a prefix and a postfix to the input, so it means you can only insert the argument on the command line once.

runp runs 10 jobs in parallel by default. runp blocks if output of a command is > 64 Kbytes. Quoting of input is needed. It adds output to stderr (this can be prevented with -q)

base='https://images-api.nasa.gov/search'
query='jupiter'
desc='planet'
type='image'
url="$base?q=$query&description=$desc&media_type=$type"
# Download the images in parallel using runp
curl-s$url|jq-r.collection.items[].href|\
runp-p'curl -s'|jq-r.[]|greplarge|\
runp-p'curl -s -L -O'
timecurl-s$url|jq-r.collection.items[].href|\
runp-g1-q-p'curl -s'|jq-r.[]|greplarge|\
runp-g1-q-p'curl -s -L -O'
# Download the images in parallel
curl-s$url|jq-r.collection.items[].href|\
parallelcurl-s|jq-r.[]|greplarge|\
parallelcurl-s-L-O
timecurl-s$url|jq-r.collection.items[].href|\
parallel-j1curl-s|jq-r.[]|greplarge|\
parallel-j1curl-s-L-O

# Create a file containing commands to run in parallel.
cat<< EOF > /tmp/test-commands.txt
sleep 5
sleep 3
blah # this will fail
ls $PWD # PWD shell variable is used here
EOF
# Run commands from the file.
runp/tmp/test-commands.txt>/dev/null
parallel-a/tmp/test-commands.txt>/dev/null

# First copy this line and press Enter
runp-p'ping -c 5 -W 2'-s'| grep loss'
localhost
1.1.1.1
8.8.8.8
# Press Enter and Ctrl-D when done entering the hosts
# First copy this line and press Enter
parallelping-c5-W2{}'| grep loss'
localhost
1.1.1.1
8.8.8.8
# Press Enter and Ctrl-D when done entering the hosts

echo-e"$HOME\n/etc\n/tmp"|runp-q-p'sudo du -sh'
echo-e"$HOME\n/etc\n/tmp"|parallelsudodu-sh
# or:
parallelsudodu-sh:::"$HOME"/etc/tmp

find.-iname'*.txt'|runp-p'gzip --best'
find.-iname'*.txt'|parallelgzip--best

exportCURL="curl -w 'time_total: %{time_total}\n'"
CURL="$CURL -o /dev/null -s https://golang.org/"
perl-wE'for (1..10) { say $ENV{CURL} }'|
runp-q# Make 10 requests
perl-wE'for (1..10) { say $ENV{CURL} }'|parallel
# or:
parallel-N0"$CURL":::{1..10}

cat<< EOF > /tmp/host-port.txt
localhost 22
localhost 80
localhost 81
127.0.0.1 443
127.0.0.1 444
scanme.nmap.org 22
scanme.nmap.org 23
scanme.nmap.org 443
EOF
1$cat/tmp/host-port.txt|
runp-q-p'netcat -v -w2 -z'2>&1|egrep'(succeeded!|open)$'
# --colsep is needed to split the line
1$cat/tmp/host-port.txt|
parallel--colsep' 'netcat-v-w2-z2>&1|
egrep'(succeeded!|open)$'
# or use uq for unquoted:
1$cat/tmp/host-port.txt|
parallelnetcat-v-w2-z{=uq=}2>&1|
egrep'(succeeded!|open)$'

DIFFERENCES BETWEEN papply AND GNU Parallel

Summary (see legend above):

• - - - I4 - - -

• M1 - M3 - - M6

• - - O3 - O5 - - x x O10

• E1 - - E4 - - -

• - - - - - - - - -

• - -

papply does not print the output if the command fails:

$papply'echo %F; false'foo
"echo foo; false"didnotsucceed

papply's replacement strings (%F %d %f %n %e %z) can be simulated in GNU parallel by putting this in ~/.parallel/config:

--rpl'%F'
--rpl'%d $_=Q(::dirname($_));'
--rpl'%f s:.*/::;'
--rpl'%n s:.*/::;s:\.[^/.]+$::;'
--rpl'%e s:.*\.:.:'
--rpl'%z $_=""'

papply buffers in RAM, and uses twice the amount of output. So output of 5 GB takes 10 GB RAM.

The buffering is very CPU intensive: Buffering a line of 5 GB takes 40 seconds (compared to 10 seconds with GNU parallel).

1$papplygzip*.txt
1$parallelgzip:::*.txt
2$papply"convert %F %n.jpg"*.png
2$parallelconvert{}{.}.jpg:::*.png

DIFFERENCES BETWEEN async AND GNU Parallel

Summary (see legend above):

• - - - I4 - - I7

• - - - - - M6

• - O2 O3 - O5 O6 - x x O10

• E1 - - E4 - E6 -

• - - - - - - - - -

• S1 S2

async is very similary to GNU parallel's --semaphore mode (aka sem). async requires the user to start a server process.

The input is quoted like -q so you need bash -c "...;..." to run composed commands.

1$S="/tmp/example_socket"
1$ID=myid
2$async-s="$S"server--start
2$# GNU Parallel does not need a server to run
3$foriin{1..20};do
# prints command output to stdout
async-s="$S"cmd--bash-c"sleep 1 && echo test $i"
done
3$foriin{1..20};do
# prints command output to stdout
sem--id"$ID"-j100%"sleep 1 && echo test $i"
# GNU Parallel will only print job when it is done
# If you need output from different jobs to mix
# use -u or --line-buffer
sem--id"$ID"-j100%--line-buffer"sleep 1 && echo test $i"
done
4$# wait until all commands are finished
async-s="$S"wait
4$sem--id"$ID"--wait
5$# configure the server to run four commands in parallel
async-s="$S"server-j4
5$exportPARALLEL=-j4
6$mkdir"/tmp/ex_dir"
foriin{21..40};do
# redirects command output to /tmp/ex_dir/file*
async-s="$S"cmd-o"/tmp/ex_dir/file$i"--\
bash-c"sleep 1 && echo test $i"
done
6$mkdir"/tmp/ex_dir"
foriin{21..40};do
# redirects command output to /tmp/ex_dir/file*
sem--id"$ID"--result'/tmp/my-ex/file-{=$_=""=}'"$i"\
"sleep 1 && echo test $i"
done
7$sem--id"$ID"--wait
7$async-s="$S"wait
8$# stops server
async-s="$S"server--stop
8$# GNU Parallel does not need to stop a server

DIFFERENCES BETWEEN pardi AND GNU Parallel

Summary (see legend above):

• I1 I2 - - - - I7

• M1 - - - - M6

• O1 O2 O3 O4 O5 - O7 - - O10

• E1 - - E4 - - -

• - - - - - - - - -

• - -

pardi is very similar to parallel --pipe --cat: It reads blocks of data and not arguments. So it cannot insert an argument in the command line. It puts the block into a temporary file, and this file name (%IN) can be put in the command line. You can only use %IN once.

It can also run full command lines in parallel (like: cat file | parallel).

1$timepardi-v-c100-idata/decoys.smi-ie.smi-oe.smi\
-odata/decoys_std_pardi.smi\
-w'(standardiser -i %IN -o %OUT 2>&1) > /dev/null'
1$catdata/decoys.smi|
timeparallel-N100--pipe--cat\
'(standardiser -i {} -o {#} 2>&1) > /dev/null; cat {#}; rm {#}'\
>data/decoys_std_pardi.smi
2$pardi-n1-idata/test_in.types-odata/test_out.types\
-d'r:^#atoms:'-w'cat %IN > %OUT'
2$catdata/test_in.types|
parallel-n1-k--pipe--cat--regexp--recstart'^#atoms'\
'cat {}'>data/test_out.types
3$pardi-c6-idata/test_in.types-odata/test_out.types\
-d'r:^#atoms:'-w'cat %IN > %OUT'
3$catdata/test_in.types|
parallel-n6-k--pipe--cat--regexp--recstart'^#atoms'\
'cat {}'>data/test_out.types
4$pardi-idata/decoys.mol2-odata/still_decoys.mol2\
-d's:@<TRIPOS>MOLECULE'-w'cp %IN %OUT'
4$catdata/decoys.mol2|
parallel-n1--pipe--cat--recstart'@<TRIPOS>MOLECULE'\
'cp {} {#}; cat {#}; rm {#}'>data/still_decoys.mol2
5$pardi-idata/decoys.mol2-odata/decoys2.mol2\
-db:10000-w'cp %IN %OUT'--preserve
5$catdata/decoys.mol2|
parallel-k--pipe--block10k--recend''--cat\
'cat {} > {#}; cat {#}; rm {#}'>data/decoys2.mol2

DIFFERENCES BETWEEN bthread AND GNU Parallel

Summary (see legend above):

• - - - I4 - - -

• - - - - - M6

• O1 - O3 - - - O7 O8 - -

• E1 - - - - - -

• - - - - - - - - -

• - -

bthread takes around 1 sec per MB of output. The maximal output line length is 1073741759.

You cannot quote space in the command, so you cannot run composed commands like sh -c "echo a; echo b".

https://gitlab.com/netikras/bthread (Last checked: 2024-06)

DIFFERENCES BETWEEN simple_gpu_scheduler AND GNU Parallel

Summary (see legend above):

• I1 - - - - - I7

• M1 - - - - M6

• - O2 O3 - - O6 - x x O10

• E1 - - - - - -

• - - - - - - - - -

• - -

1$simple_gpu_scheduler--gpus012<gpu_commands.txt
1$parallel-j3--shuf\
CUDA_VISIBLE_DEVICES='{=1 $_=slot()-1 =} {=uq;=}'\
<gpu_commands.txt
2$simple_hypersearch\
"python3 train_dnn.py --lr {lr} --batch_size {bs}"\
-plr0.0010.00050.0001-pbs3264128|
simple_gpu_scheduler--gpus0,1,2
2$parallel--header:--shuf-j3-v\
CUDA_VISIBLE_DEVICES='{=1 $_=slot()-1 =}'\
python3train_dnn.py--lr{lr}--batch_size{bs}\
:::lr0.0010.00050.0001:::bs3264128
3$simple_hypersearch\
"python3 train_dnn.py --lr {lr} --batch_size {bs}"\
--n-samples5-plr0.0010.00050.0001-pbs3264128|
simple_gpu_scheduler--gpus0,1,2
3$parallel--header:--shuf\
CUDA_VISIBLE_DEVICES='{=1 $_=slot()-1; seq()>5 and skip() =}'\
python3train_dnn.py--lr{lr}--batch_size{bs}\
:::lr0.0010.00050.0001:::bs3264128
4$touchgpu.queue
tail-f-n0gpu.queue|simple_gpu_scheduler--gpus0,1,2&
echo"my_command_with | and stuff > logfile">>gpu.queue
4$touchgpu.queue
tail-f-n0gpu.queue|
parallel-j3CUDA_VISIBLE_DEVICES='{=1 $_=slot()-1 =} {=uq;=}'&
# Needed to fill job slots once
seq3|parallelechotrue>>gpu.queue
# Add jobs
echo"my_command_with | and stuff > logfile">>gpu.queue
# Needed to flush output from completed jobs
seq3|parallelechotrue>>gpu.queue

DIFFERENCES BETWEEN parasweep AND GNU Parallel

parasweep is a Python module for facilitating parallel parameter sweeps.

A parasweep job will normally take a text file as input. The text file contains arguments for the job. Some of these arguments will be fixed and some of them will be changed by parasweep.

It does this by having a template file such as template.txt:

Xval:{x}
Yval:{y}
FixedValue:9
# x with 2 decimals
DecimalX:{x:.2f}
TenX:${x*10}
RandomVal:{r}

and from this template it generates the file to be used by the job by replacing the replacement strings.

Being a Python module parasweep integrates tighter with Python than GNU parallel. You get the parameters directly in a Python data structure. With GNU parallel you can use the JSON or CSV output format to get something similar, but you would have to read the output.

parasweep has a filtering method to ignore parameter combinations you do not need.

Instead of calling the jobs directly, parasweep can use Python's Distributed Resource Management Application API to make jobs run with different cluster software.

GNU parallel --tmpl supports templates with replacement strings. Such as:

Xval:{x}
Yval:{y}
FixedValue:9
# x with 2 decimals
DecimalX:{=x$_=sprintf("%.2f",$_)=}
TenX:{=x$_=$_*10=}
RandomVal:{=1$_=rand()=}

that can be used like:

parallel--header:--tmplmy.tmpl={#}.t myprog {#}.t \
:::x123:::y123

Filtering is supported as:

parallel--filter'{1} > {2}'echo:::123:::123

https://github.com/eviatarbach/parasweep (Last checked: 2021-01)

DIFFERENCES BETWEEN parallel-bash(2021) AND GNU Parallel

Summary (see legend above):

• I1 I2 - - - - -

• - - M3 - - M6

• - O2 O3 - O5 O6 - O8 x O10

• E1 - - - - - -

• - - - - - - - - -

• - -

parallel-bash is written in pure bash. It is really fast (overhead of ~0.05 ms/job compared to GNU parallel's 3-10 ms/job). So if your jobs are extremely short lived, and you can live with the quite limited command, this may be useful.

It works by making a queue for each process. Then the jobs are distributed to the queues in a round robin fashion. Finally the queues are started in parallel. This works fine, if you are lucky, but if not, all the long jobs may end up in the same queue, so you may see:

$printf"%b\n"111411141114|
timeparallel-P4sleep{}
(7seconds)
$printf"%b\n"111411141114|
timeparallel-bash-p4-csleep{}
(12seconds)

Because it uses bash lists, the total number of jobs is limited to 167000..265000 depending on your environment. You get a segmentation fault, when you reach the limit.

Ctrl-C does not stop spawning new jobs. Ctrl-Z does not suspend running jobs.

1$some_input|parallel-bash-p5-cecho
1$some_input|parallel-j5echo
2$parallel-bash-p5-cecho<some_file
2$parallel-j5echo<some_file
3$parallel-bash-p5-cecho<<<'some string'
3$parallel-j5-cecho<<<'some string'
4$something|parallel-bash-p5-cecho{}{}
4$something|parallel-j5echo{}{}

DIFFERENCES BETWEEN parallel-bash(2024) AND GNU Parallel

Summary (see legend above):

• I1 I2 - - - - -

• - - M3 - - M6

• - O2 O3 - O5 O6 - O8 x O10

• E1 - - - - - -

• - - - - - - - - -

• - -

parallel-bash is written in pure bash. It is really fast (overhead of ~0.05 ms/job compared to GNU parallel's 3-10 ms/job). So if your jobs are extremely short lived, and you can live with the quite limited command, this may be useful.

It seems the number of jobs must be divisible by -p, so it sometimes does not run the jobs:

# Does nothing
$seq3|parallel-bash-p4bash-c'touch myfile-{}'

This should create myfile-1..3, but creates nothing.

It splits the input into queues. Each queue is of length -p. So this will make 250 queues and run all 250 processes in parallel:

$seq1000|parallel-bash-p4bash-c'sleep {}'

This is quite different from parallel-bash(2021) where -p is the number of workers - similar to --jobs in GNU parallel.

In other words: parallel-bash does not quarantee that only 4 jobs will be run in parallel. This can overload your machine:

# Warning: This will start 25000 processes - not just 4
$seq100000|parallel-bash-p4sleep{}

If you are unlucky all long jobs may end up in the same queue:

$printf"%b\n"111155551111|
timeparallel-P4sleep{}
(7seconds)
$printf"%b\n"111155551111|
time./parallel-bash.bash-p4-csleep{}
(20seconds)

Ctrl-C kills the jobs (as expected). Ctrl-Z does not suspend running jobs.

1$main(){echo"${1}";}
export-fmain
1$printf"%b\n"{1..1000}|./parallel-bash-p10main{}
1$printf"%b\n"{1..1000}|parallel-j100main{}
2$# Number of inputs must be divisible by 5
some_input|parallel-bash-p5echo
2$some_input|parallel-j5echo
3$# Number of inputs must be divisible by 5
parallel-bash-p5echo<some_file
3$parallel-j5echo<some_file
4$# Number of lines in 'some string' must be divisible by 5
parallel-bash-p5echo<<<'some string'
4$parallel-j5-cecho<<<'some string'
5$something|parallel-bash-p5echo{}
5$something|parallel-j5echo{}

DIFFERENCES BETWEEN bash-concurrent AND GNU Parallel

bash-concurrent is more an alternative to make than to GNU parallel. Its input is very similar to a Makefile, where jobs depend on other jobs.

It has a nice progress indicator where you can see which jobs completed successfully, which jobs are currently running, which jobs failed, and which jobs were skipped due to a depending job failed. The indicator does not deal well with resizing the window.

Output is cached in tempfiles on disk, but is only shown if there is an error, so it is not meant to be part of a UNIX pipeline. If bash-concurrent crashes these tempfiles are not removed.

It uses an O(n*n) algorithm, so if you have 1000 independent jobs it takes 22 seconds to start it.

https://github.com/themattrix/bash-concurrent (Last checked: 2024-06)

DIFFERENCES BETWEEN spawntool AND GNU Parallel

Summary (see legend above):

• I1 - - - - - -

• M1 - - - - M6

• - O2 O3 - O5 O6 - x x O10

• E1 - - - - - -

• - - - - - - - - -

• - -

spawn reads a full command line from stdin which it executes in parallel.

http://code.google.com/p/spawntool/ (Last checked: 2021-07)

DIFFERENCES BETWEEN go-pssh AND GNU Parallel

Summary (see legend above):

• - - - - - - -

• M1 - - - - -

• O1 - - - - - - x x O10

• E1 - - - - - -

• R1 R2 - - - R6 - - -

• - -

go-pssh does ssh in parallel to multiple machines. It runs the same command on multiple machines similar to --nonall.

The hostnames must be given as IP-addresses (not as hostnames).

Output is sent to stdout (standard output) if command is successful, and to stderr (standard error) if the command fails.

1$go-pssh-l<ip>,<ip>-u<user>-p<port>-P<passwd>-c"<command>"
1$parallel-S'sshpass -p <passwd> ssh -p <port> <user>@<ip>'\
--nonall"<command>"
2$go-psshscp-fhost.txt-u<user>-p<port>-P<password>\
-s/local/file_or_directory-d/remote/directory
2$parallel--nonall--slfhost.txt\
--basefile/local/file_or_directory/./--wd/remote/directory
--ssh'sshpass -p <password> ssh -p <port> -l <user>'true
3$go-psshscp-l<ip>,<ip>-u<user>-p<port>-P<password>\
-s/local/file_or_directory-d/remote/directory
3$parallel--nonall-S<ip>,<ip>\
--basefile/local/file_or_directory/./--wd/remote/directory
--ssh'sshpass -p <password> ssh -p <port> -l <user>'true

DIFFERENCES BETWEEN go-parallel AND GNU Parallel

Summary (see legend above):

• I1 I2 - - - - (I7)

• - - M3 - - M6

• - O2 O3 - O5 - - x x - O10

• E1 - - E4 - - -

• - - - - - - - - -

• - -

go-parallel uses Go templates for replacement strings. Quite similar to the {= perl expr =} replacement string.

The basic replacement strings can be emulated by putting this into ~/.parallel/config:

--rpl'{{.Input}} '
--rpl'{{.Time}} use DateTime; $_= DateTime->from_epoch(time);'
--rpl'{{.Start}} use DateTime; $_= DateTime->from_epoch($^T);'

Then you can do:

seq10|parallelsleep{{.Input}}';'echo{{.Start}}{{.Time}}
seq10|go-parallel-t'bash -c "sleep {{.Input}}; echo \"{{.Start}}\" \"{{.Time}}\""'

If the input is too long (64K), you get no error:

perl-e'print "works."."x"x100'|parallel.go-t'echo {{noExt .Input}} '
perl-e'print "fails."."x"x100_000_000'|parallel.go-t'echo {{noExt .Input}} '

Special chars are quoted:

echo'"&+<>'|go-parallelecho
"&+<>

but not shell quoted when using replacement strings:

echo'"&+<>'|go-parallel-t'echo {{.Input}}'
&#34;&amp;&#43;&lt;&gt;

1$go-parallel-a./files.txt-t'cp {{.Input}} {{.Input | dirname | dirname}}'
1$parallel-a./files.txtcp{}'{= $_=::dirname(::dirname($_)) =}'
2$go-parallel-a./files.txt-t'mkdir -p {{.Input}} {{noExt .Input}}'
2$parallel-a./files.txtechomkdir-p{}{.}
3$go-parallel-a./files.txt-t'mkdir -p {{.Input}} {{.Input | basename | noExt}}'
3$parallel-a./files.txtechomkdir-p{}{/.}
4$timefind~/src/go-typef|go-parallelmd5sum>/dev/null
4$timefind~/src/go-typef|parallelmd5sum>/dev/null
# Though you would probably do this instead:
timefind~/src/go-typef|parallel-Xmd5sum>/dev/null

DIFFERENCES BETWEEN p AND GNU Parallel

Summary (see legend above):

• - - - I4 - - x

• - - - - - M6

• - O2 O3 - O5 O6 - x x - O10

• E1 - - - - - -

• - - - - - - - - -

• - -

p is a tiny shell script. It can color output with some predefined colors, but is otherwise quite limited.

It maxes out at around 116000 jobs (probably due to limitations in Bash).

1$p-bcblue"ping 127.0.0.1"-ucred"ping 192.168.0.1"\
-rcyellow"ping 192.168.1.1"-texample"ping example.com"
1$parallel--lb-j0--color--tagping\
:::127.0.0.1192.168.0.1192.168.1.1example.com
2$p"tail -f /var/log/httpd/access_log"\
-bcred"tail -f /var/log/httpd/error_log"
2$cd/var/log/httpd;
parallel--lb--color--tagtail-f:::access_logerror_log
3$ptail-f"some file"\&ptail-f"other file with space.txt"
3$parallel--lbtail-f:::'some file'"other file with space.txt"
4$p-tproject1"hg pull project1"-tproject2\
"hg pull project2"-tproject3"hg pull project3"
4$parallel--lbhgpull:::project{1..3}

DIFFERENCES BETWEEN senechal AND GNU Parallel

Summary (see legend above):

• I1 - - - - - -

• M1 - M3 - - M6

• O1 - O3 O4 - - - x x -

• E1 - - - - - -

• - - - - - - - - -

• - -

seneschal only starts the first job after reading the last job, and output from the first job is only printed after the last job finishes.

1 byte of output requites 3.5 bytes of RAM.

This makes it impossible to have a total output bigger than the virtual memory.

Even though output is kept in RAM outputing is quite slow: 30 MB/s.

Output larger than 4 GB causes random problems - it looks like a race condition.

This:

echo1|seneschal--prefix='yes `seq 1000`|head -c 1G'>/dev/null

takes 4100(!) CPU seconds to run on a 64C64T server, but only 140 CPU seconds on a 4C8T laptop. So it looks like seneschal wastes a lot of CPU time coordinating the CPUs.

Compare this to:

echo1|time-vparallel-N0'yes `seq 1000`|head -c 1G'>/dev/null

which takes 3-8 CPU seconds.

1$echo$REPOS|seneschal--prefix="cd {} && git pull"
# If $REPOS is newline separated
1$echo"$REPOS"|parallel-k"cd {} && git pull"
# If $REPOS is space separated
1$echo-n"$REPOS"|parallel-d' '-k"cd {} && git pull"
COMMANDS="pwd
sleep 5 && echo boom
echo Howdy
whoami"
2$echo"$COMMANDS"|seneschal--debug
2$echo"$COMMANDS"|parallel-k-v
3$ls-1|seneschal--prefix="pushd {}; git pull; popd;"
3$ls-1|parallel-k"pushd {}; git pull; popd;"
# Or if current dir also contains files:
3$parallel-k"pushd {}; git pull; popd;":::*/

DIFFERENCES BETWEEN async AND GNU Parallel

Summary (see legend above):

• x x x x x x x

• - x x x x x

• x O2 O3 O4 O5 O6 - x x O10

• E1 - - E4 - - -

• - - - - - - - - -

• S1 S2

async works like sem.

1$S="/tmp/example_socket"
async-s="$S"server--start
foriin{1..20};do
# prints command output to stdout
async-s="$S"cmd--bash-c"sleep 1 && echo test $i"
done
# wait until all commands are finished
async-s="$S"wait
1$S="example_id"
# server not needed
foriin{1..20};do
# prints command output to stdout
sem--bg--id"$S"-j100%"sleep 1 && echo test $i"
done
# wait until all commands are finished
sem--fg--id"$S"--wait
2$# configure the server to run four commands in parallel
async-s="$S"server-j4
mkdir"/tmp/ex_dir"
foriin{21..40};do
# redirects command output to /tmp/ex_dir/file*
async-s="$S"cmd-o"/tmp/ex_dir/file$i"--\
bash-c"sleep 1 && echo test $i"
done
async-s="$S"wait
# stops server
async-s="$S"server--stop
2$# starting server not needed
mkdir"/tmp/ex_dir"
foriin{21..40};do
# redirects command output to /tmp/ex_dir/file*
sem--bg--id"$S"--results"/tmp/ex_dir/file$i{}"\
"sleep 1 && echo test $i"
done
sem--fg--id"$S"--wait
# there is no server to stop

DIFFERENCES BETWEEN tandem AND GNU Parallel

Summary (see legend above):

• - - - I4 - - x

• M1 - - - - M6

• - - O3 - - - - x - -

• E1 - E3 - E5 - -

• - - - - - - - - -

• - -

tandem runs full commands in parallel. It is made for starting a "server", running a job against the server, and when the job is done, the server is killed.

More generally: it kills all jobs when the first job completes - similar to '--halt now,done=1'.

tandem silently discards some output. It is unclear exactly when this happens. It looks like a race condition, because it varies for each run.

$tandem"seq 10000"|wc-l
6731<-Thisshouldalwaysbe10002

tandem\
'php -S localhost:8000'\
'esbuild src/*.ts --bundle --outdir=dist --watch'\
'tailwind -i src/index.css -o dist/index.css --watch'
# Emulate tandem's behaviour
PARALLEL='--color --lb --halt now,done=1 --tagstring '
PARALLEL="$PARALLEL'"'{=s/ .*//; $_.=".".$app{$_}++;=}'"'"
exportPARALLEL
parallel:::\
'php -S localhost:8000'\
'esbuild src/*.ts --bundle --outdir=dist --watch'\
'tailwind -i src/index.css -o dist/index.css --watch'

# Emulate tandem's behaviour
PARALLEL='--color --lb --halt now,done=1 --tagstring '
PARALLEL="$PARALLEL'"'{=s/ .*//; $_.=".".$app{$_}++;=}'"'"
exportPARALLEL
1$tandem'sleep 5 && echo "hello"''sleep 2 && echo "world"'
1$parallel:::'sleep 5 && echo "hello"''sleep 2 && echo "world"'
# '-t 0' fails. But '--timeout 0 works'
2$tandem--timeout0'sleep 5 && echo "hello"'\
'sleep 2 && echo "world"'
2$parallel--timeout0:::'sleep 5 && echo "hello"'\
'sleep 2 && echo "world"'

# Emulate tandem's behaviour
PARALLEL='--color --lb --halt now,done=1 --tagstring '
PARALLEL="$PARALLEL'"'{=s/ .*//; $_.=".".$app{$_}++;=}'"'"
exportPARALLEL
1$tandem'next dev''nodemon --quiet ./server.js'
1$parallel:::'next dev''nodemon --quiet ./server.js'
2$catpackage.json
{
"scripts":{
"dev:php":"...",
"dev:js":"...",
"dev:css":"..."
}
}
tandem'npm:dev:php''npm:dev:js''npm:dev:css'
# GNU Parallel uses bash functions instead
2$catpackage.sh
dev:php(){...;}
dev:js(){...;}
dev:css(){...;}
export-fdev:phpdev:jsdev:css
.package.sh
parallel:::dev:phpdev:jsdev:css
3$tandem'npm:dev:*'
3$compgen-Afunction|grep^dev:|parallel

DIFFERENCES BETWEEN rust-parallel(aaronriekenberg) AND GNU Parallel

Summary (see legend above):

• I1 I2 I3 - - - -

• - - - - - M6

• O1 O2 O3 - O5 O6 - x - O10

• E1 - - E4 - - -

• - - - - - - - - -

• - -

rust-parallel has a goal of only using Rust. It seems it is impossible to call bash functions from the command line. You would need to put these in a script.

Calling a script that misses the shebang line (#! as first line) fails.

$cat>./test<<EOL
echo hi
echo there
echo how
echo are
echo you
EOL
1$cattest|rust-parallel-j5
1$cattest|parallel-j5
2$cattest|rust-parallel-j1
2$cattest|parallel-j1
3$head-100/usr/share/dict/words|rust-parallelmd5-s
3$head-100/usr/share/dict/words|parallelmd5-s
4$find.-typef-print0|rust-parallel-0gzip-f-k
4$find.-typef-print0|parallel-0gzip-f-k
5$head-100/usr/share/dict/words|
awk'{printf "md5 -s %s\n", 1ドル}'|rust-parallel
5$head-100/usr/share/dict/words|
awk'{printf "md5 -s %s\n", 1ドル}'|parallel
6$head-100/usr/share/dict/words|rust-parallelmd5-s|
grep-iabba
6$head-100/usr/share/dict/words|parallelmd5-s|
grep-iabba

DIFFERENCES BETWEEN parallelium AND GNU Parallel

Summary (see legend above):

• - I2 - - - - -

• M1 - - - - M6

• O1 - O3 - - - - x - -

• E1 - - E4 - - -

• - - - - - - - - -

• - -

parallelium merges standard output (stdout) and standard error (stderr). The maximal output of a command is 8192 bytes. Bigger output makes parallelium go into an infinite loop.

In the input file for parallelium you can define a tag, so that you can select to run only these commands. A bit like a target in a Makefile.

Progress is printed on standard output (stdout) prepended with '#' with similar information as GNU parallel's --bar.

$cattestjobs.txt
#tag common sleeps classA
(sleep4.495;echo"job 000")
:
(sleep2.587;echo"job 016")
#tag common sleeps classB
(sleep0.218;echo"job 017")
:
(sleep2.269;echo"job 040")
#tag common sleeps classC
(sleep2.586;echo"job 041")
:
(sleep1.626;echo"job 099")
#tag lasthalf, sleeps, classB
(sleep1.540;echo"job 100")
:
(sleep2.001;echo"job 199")
1$parallelium-ftestjobs.txt-llogdir-tclassB,classC
1$cattestjobs.txt|
parallel--plus--resultslogdir/testjobs.txt_{0#}.output \
'{= if(/^#tag /) { @tag = split/,|\s+/ }
 (grep /^(classB|classC)$/, @tag) or skip =}'

DIFFERENCES BETWEEN forkrun AND GNU Parallel

Summary (see legend above):

• I1 - - - - - I7

• - - - - - -

• - O2 O3 - O5 - - - - O10

• E1 - - E4 - - -

• - - - - - - - - -

• - -

forkrun blocks if it receives fewer jobs than slots:

echo|forkrun-p2echo

or when it gets some specific commands e.g.:

f(){seq"$@"|pv-qL3;}
seq10|forkrunf

It is not clear why.

It is faster than GNU parallel (overhead: 1.2 ms/job vs 3 ms/job), but way slower than parallel-bash (0.059 ms/job).

Running jobs cannot be stopped by pressing CTRL-C.

-k is supposed to keep the order but fails on the MIX testing example below. If used with -k it caches output in RAM.

If forkrun is killed, it leaves temporary files in /tmp/.forkrun.* that has to be cleaned up manually.

1$timefind./-typef|
forkrun-l512--sha256sum2>/dev/null|wc-l
1$timefind./-typef|
parallel-j28-m--sha256sum2>/dev/null|wc-l
2$timefind./-typef|
forkrun-l512-k--sha256sum2>/dev/null|wc-l
2$timefind./-typef|
parallel-j28-k-m--sha256sum2>/dev/null|wc-l

DIFFERENCES BETWEEN parallel-sh AND GNU Parallel

Summary (see legend above):

• I1 I2 - I4 - - -

• M1 - - - - M6

• O1 O2 O3 - O5 O6 - - - O10

• E1 - - E4 - - -

• - - - - - - - - -

• - -

parallel-sh buffers in RAM. The buffering data takes O(n^1.5) time:

2MB=0.107s 4MB=0.175s 8MB=0.342s 16MB=0.766s 32MB=2.2s 64MB=6.7s 128MB=20s 256MB=64s 512MB=248s 1024MB=998s 2048MB=3756s

It limits the practical usability to jobs outputting < 256 MB. GNU parallel buffers on disk, yet is faster for jobs with outputs > 16 MB and is only limited by the free space in $TMPDIR.

parallel-sh can kill running jobs if a job fails (Similar to --halt now,fail=1).

1$parallel-sh"sleep 2 && echo first""sleep 1 && echo second"
1$parallel:::"sleep 2 && echo first""sleep 1 && echo second"
2$cat/tmp/commands
sleep2&&echofirst
sleep1&&echosecond
2$parallel-sh-f/tmp/commands
2$parallel-a/tmp/commands
3$echo-e'sleep 2 && echo first\nsleep 1 && echo second'|
parallel-sh
3$echo-e'sleep 2 && echo first\nsleep 1 && echo second'|
parallel

DIFFERENCES BETWEEN bash-parallel AND GNU Parallel

Summary (see legend above):

• - I2 - - - - I7

• M1 - M3 - M5 M6

• - O2 O3 - - O6 - O8 - O10

• E1 - - - - - -

• - - - - - - - - -

• - -

bash-parallel is not as much a command as it is a shell script that you have to alter. It requires you to change the shell function process_job that runs the job, and set $MAX_POOL_SIZE to the number of jobs to run in parallel.

It is half as fast as GNU parallel for short jobs.

https://github.com/thilinaba/bash-parallel (Last checked: 2024-06)

DIFFERENCES BETWEEN PaSH AND GNU Parallel

Summary (see legend above): N/A

pash is quite different from GNU parallel. It is not a general parallelizer. It takes a shell script and analyses it and parallelizes parts of it by replacing the parts with commands that will give the same result.

This will replace sort with a command that does pretty much the same as parsort --parallel=8 (except somewhat slower):

pa.sh--width8-c'cat bigfile | sort'

However, even a simple change will confuse pash and you will get no parallelization:

pa.sh--width8-c'mysort() { sort; }; cat bigfile | mysort'
pa.sh--width8-c'cat bigfile | sort | md5sum'

From the source it seems pash only looks at: awk cat col comm cut diff grep head mkfifo mv rm sed seq sort tail tee tr uniq wc xargs

For pipelines where these commands are bottlenecks, it might be worth testing if pash is faster than GNU parallel.

pash does not respect $TMPDIR but always uses /tmp. If pash dies unexpectantly it does not clean up.

https://github.com/binpash/pash (Last checked: 2023-05)

DIFFERENCES BETWEEN korovkin-parallel AND GNU Parallel

Summary (see legend above):

• I1 - - - - - -

• M1 - - - - M6

• - - O3 - - - - x x -

• E1 - - - - - -

• R1 - - - - R6 x x -

• - -

korovkin-parallel prepends all lines with some info.

The output is colored with 6 color combinations, so job 1 and 7 will get the same color.

You can get similar output with:

(echo...)|
parallel--color-j10--lb--tagstring\
'[l:{#}:{=$_=sprintf("%7.03f",::now()-$^T)=} {=$_=hh_mm_ss($^T)=} {%}]'

Lines longer than 8192 chars are broken into lines shorter than 8192. korovkin-parallel loses the last char for lines exactly 8193 chars long.

Short lines from different jobs do not mix, but long lines do:

fun(){
perl-e'$a="'1ドル'"x1000000; for(1..'2ドル') { print $a };';
echo;
}
export-ffun
(echofuna100;echofunb100)|korovkin-parallel|tr-sabcdef
# Compare to:
(echofuna100;echofunb100)|parallel|tr-sabcdef

There should be only one line of a's and one line of b's.

Just like GNU parallel korovkin-parallel offers a master/slave model, so workers on other servers can do some of the tasks. But contrary to GNU parallel you must manually start workers on these servers. The communication is neither authenticated nor encrypted.

It caches output in RAM: a 1GB line uses ~2.5GB RAM

https://github.com/korovkin/parallel (Last checked: 2023-07)

DIFFERENCES BETWEEN xe AND GNU Parallel

Summary (see legend above):

• I1 I2 - I4 - - I7

• M1 - M3 M4 - M6

• - O2 O3 - O5 O6 - O8 - O10

• E1 - - E4 - - -

• - - - - - - - - -

• - -

xe has a peculiar limitation:

echo/bin/echo|xe{}OK
echoecho|xe/bin/{}fails

1$xe-a-j0gzip--*.c
1$parallelgzip:::*.c

2$lr-U-t'size == 0'|xe-N0rm
2$lr-U-t'size == 0'|parallel-Xrm

3$xe-a-j0-s'ffmpeg -i "${1}" "${1%.mp3}.ogg"'--*.mp3
3$parallelffmpeg-i{}{.}.ogg:::*.mp3

4$xe-a-j0-p%.mp3ffmpeg-i%.mp3%.ogg--*.mp3
4$parallel--rpl'% s/\.mp3// or skip'ffmpeg-i%.mp3%.ogg:::*.mp3

5$xe-ap-j0-vvq'%.{m4a,ogg,opus}'ffmpeg-y-i{}out/%.mp3--*
5$parallel-v--rpl'% s/\.(m4a|ogg|opus)// or skip'\
ffmpeg-y-i{}out/%.mp3'2>/dev/null':::*
5$parallel-vffmpeg-y-i{}out/{.}.mp3'2>/dev/null':::*

DIFFERENCES BETWEEN sp AND GNU Parallel

Summary (see legend above):

• - - - I4 - - -

• M1 - M3 - - M6

• - O2 O3 - O5 (O6) - x x O10

• E1 - - - - - -

• - - - - - - - - -

• - -

sp has very few options.

It can either be used like:

spcommand{}option::arg1arg2arg3

which is similar to:

parallelcommand{}option:::arg1arg2arg3

Or:

spcommand1::"command2 -option"::"command3 foo bar"

which is similar to:

parallel:::command1"command2 -option""command3 foo bar"

sp deals badly with too many commands: This causes sp to run out of file handles and gives data loss.

For each command that fails, sp will print an error message on stderr (standard error).

You cannot used exported shell functions as commands.

1$specho{}::123
1$parallelecho{}:::123
2$specho{}{}::123
2$parallelecho{}{}::123
3$specho1::echo2::echo3
3$parallel:::'echo 1''echo 2''echo 3'
4$spafoobar::"b 'baz bar'"::c
4$parallel:::'a foo bar'"b 'baz bar'"::c

DIFFERENCES BETWEEN repeater AND GNU Parallel

Summary (see legend above):

• - - - - - - -

• - - - - - -

• - O2 O3 x - O6 - x x ?O10

• E1 - - - E5 - -

• - - - - - - - - -

• - -

repeater runs the same job repeatedly. In other words: It does not read arguments, thus is it an alternative for GNU parallel for only quite limited applications.

repeater has an overhead of around 0.23 ms/job. Compared to GNU parallel's 2-3 ms this is fast. Compared to bash-parallel's 0.05 ms/job it is slow.

# Run time = 15 sec
# Memory use = 20 MB
# Output = 1 GB per job
\time-vparallel-j1seq:::120000000120000000>/dev/null
# Run time = 4.7 sec
# Memory use = 95 MB
# Output = 8 MB per job
\time-vrepeater-w1-n2-reportFile./run_outputseq1200000>/dev/null
# Run time = 42 sec
# Memory use = 277 MB
# Output = 27 MB per job
\time-vrepeater-w1-n2-reportFile./run_outputseq3600000>/dev/null
# Run time = 530 sec
# Memory use = 1000 MB
# Output = 97 MB per job
\time-vrepeater-w1-n2-reportFile./run_outputseq12000000>/dev/null
# Run time = 2h41m
# Memory use = 8.6 GB
# Output = 1 GB per job
\time-vrepeater-w1-n2-reportFile./run_outputseq120000000>/dev/null

1$repeater-n100-w10-reportFile./run_output
-outputREPORT_FILE-progressBOTHcurlexample.com
1$seq100|parallel--joblogrun.log--etacurlexample.com>output
2$repeater-n100-increment-progressHIDDEN-reportFilefoo
echo"this is increment: "INC
2$seq100|parallelecho{}
2$seq100|parallelecho'{= $_ = ++$myvar =}'

DIFFERENCES BETWEEN parallelize AND GNU Parallel

Summary (see legend above):

• I1 - - - - - I7

• - - - - - M6

• O1 - O3 O4 O5 - O7 - - -

• E1 - - E4 - - -

• - - - - - - - - -

• - -

parallelize runs the full line as a command. If the command is not found, there is no warning.

The output at most ~1000000 lines/s. If the lines are short this is quite slow. The lines can at most be 2047999 bytes long. Longer lines cause segfault.

simple.dat:
sleep5
foo
catalire.toml
locsrc/parallelize.adb
shlocsrc/*.ad?
1$bin/parallelize-v<simple.dat
1$parallel<simple.dat

DIFFERENCES BETWEEN pararun/paramap AND GNU Parallel

Summary (see legend above):

• I1 - - - - - I7

• M1 - M3 M4 - M6

• - O2 O3 (O4) O5 O6 - - x -

• E1 - (E3) E4 (E5) - - E8 E9

• - - - - - - - - -

• - -

pararun gets tricky when you need to quote more than 2 levels:

$echo"$bmps"|parallel-qperl-e'print @ARGV,"\n"'{}
$echo"$bmps"|pararun'perl -e '"'"'print @ARGV,"\n"'"'"' "1ドル"'

paramap tries to keep order of output, and works for short lines on stdout:

$printf'%s\n'2143|paramap-j4'sleep 1ドル; echo -n 1ドル; sleep 1ドル; echo 1ドル'
22
11
44
33

But adds newlines if there is no newline:

$printf'%s\n'2143|paramap-j4'echo -n 1ドル'
2
1
4
3
$printf'%s\n'2143|parallel-j4'echo -n {}'
2143

paramap does not deal with stderr:

$printf'%s\n'2143|
paramap-j4'sleep 1ドル; echo -n 1ドル >&2; sleep 1ドル; echo 1ドル >&2'
121
342
3
4

paramap only prints, when all jobs are done. GNU parallel prints as soon as possible.

Fails paramap to run (due to final ;):

$echoa|paramap'echo 1ドル;'

Long lines mix:

printf'%s\n'2143|
paramap-j4'perl -e "print \"1ドル\"x3000_000,\" \""'|
tr-s1-4

And really long lines fail:

$printf'%s\n'1234|
paramap-j2'perl -e "map { print \"1ドル\"x5000_000 } 1..1000,\" \""'|wc
482840130828
$printf'%s\n'1234|
parallel-j2'perl -e "map { print \"{}\"x5000_000 } 1..1000,\" \""'|wc
0120020000000

1$bmps=$(finddir/-typef-name'*.bmp')
1$echo"$bmps"|pararun'magick "1ドル" "${1%.*}".png'

1$echo"$bmps"|parallel--plusmagick{1}{1%.*}.png

DIFFERENCES BETWEEN concur AND GNU Parallel

Summary (see legend above):

• I1 - - I4 - - (I7)

• - - M3 - - -

• O1 - O3 - - - - x x O10

• E1 - (E3) E4 - - - - -

• - - - - - - - - -

• - -

concur outputs in json. Output is cached in RAM and takes up a factor of 6: 1 GB output uses 6 GB RAM.

Standard input is split on space - not newline.

It can stop when the first job stops.

It gives the wrong output on jobs like:

concur'perl -e "print(shift)"'"foo"
concur'bash -c "sleep {{1}}; touch a{{1}}"'321

1$concur"dig @{{1}} ocw.mit.edu"1.1.1.19.9.9.98.8.8.894.140.14.14\
208.67.222.222
1$parallel--results-.json"dig @{1} ocw.mit.edu"1:::1.1.1.19.9.9.9\
8.8.8.894.140.14.14208.67.222.222|jq.
2$echo"foo bar baz"|concur"touch /tmp/{{1}}.foo"
2$printf"%s\n"foobarbaz|parallel"touch /tmp/{1}.foo"
3$concur"ping -c 1 {{1}}"www.mit.eduwww.ucla.eduwww.slashdot.org
3$parallel--results-.json"ping -c 1 {1}":::www.mit.edu\
www.ucla.eduwww.slashdot.org|jq.
4$concur"ping -c 1 {{1}}"www.mit.eduwww.ucla.eduwww.slashdot.org|
jq'.command[0]'
4$parallel--results-.json"ping -c 1 {1}":::www.mit.eduwww.ucla.edu\
www.slashdot.org|jq-s'.[1]'

Todo

https://github.com/rustunit/parallelrun?tab=readme-ov-file

https://github.com/justanhduc/task-spooler

https://manpages.ubuntu.com/manpages/xenial/man1/tsp.1.html

https://www.npmjs.com/package/concurrently

http://code.google.com/p/push/ (cannot compile)

https://github.com/krashanoff/parallel

https://github.com/Nukesor/pueue

https://arxiv.org/pdf/2012.15443.pdf KumQuat

https://github.com/JeiKeiLim/simple_distribute_job

https://github.com/reggi/pkgrun - not obvious how to use

https://github.com/benoror/better-npm-run - not obvious how to use

https://github.com/bahmutov/with-package

https://github.com/flesler/parallel

https://github.com/Julian/Verge

https://vicerveza.homeunix.net/~viric/soft/ts/

https://github.com/chapmanjacobd/que

MIX: Output mixes

Output from 2 jobs should not mix. If the output is not used, this does not matter; but if the output is used then it is important that you do not get half a line from one job followed by half a line from another job.

If the tool does not buffer, output will most likely mix now and then.

This test stresses whether output mixes.

#!/bin/bash
paralleltool="parallel -j 30"
cat<<-'EOF' > mycommand
#!/bin/bash
# If a, b, c, d, e, and f mix: Very bad
perl -e 'print STDOUT "a"x3000_000," "'
perl -e 'print STDERR "b"x3000_000," "'
perl -e 'print STDOUT "c"x3000_000," "'
perl -e 'print STDERR "d"x3000_000," "'
perl -e 'print STDOUT "e"x3000_000," "'
perl -e 'print STDERR "f"x3000_000," "'
echo "stdout line 1 of id $@"
echo "stderr line 1 of id $@" >&2
perl -e 'print STDOUT "A"x3000_000," "'
perl -e 'print STDERR "B"x3000_000," "'
perl -e 'print STDOUT "C"x3000_000," "'
perl -e 'print STDERR "D"x3000_000," "'
perl -e 'print STDOUT "E"x3000_000," "'
perl -e 'print STDERR "F"x3000_000," "'
echo "stdout line 2 of id $@"
echo "stderr line 2 of id $@" >&2
EOF
chmod+xmycommand
# Run 30 jobs in parallel
seq30|
$paralleltool-j30./mycommand>>(tr-sa-zA-Z)2>>(tr-sa-zA-Z>&2)
# 'a c e' and 'b d f' should always stay together
# For each job there be 2 lines of standard output and standard error
# They should not be interleaved with other id's

STDERRMERGE: Stderr is merged with stdout

Output from stdout and stderr should not be merged, but kept separated.

This test shows whether stdout is mixed with stderr.

#!/bin/bash
paralleltool="parallel -j0"
cat<<-EOF > mycommand
#!/bin/bash
echo stdout
echo stderr >&2
echo stdout
echo stderr >&2
EOF
chmod+xmycommand
# Run one job
echo|
$paralleltool./mycommand>stdout2>stderr
catstdout
catstderr

RAM: Output limited by RAM

Some tools cache output in RAM. This makes them extremely slow if the output is bigger than physical memory and crash if the output is bigger than the virtual memory.

#!/bin/bash
paralleltool="parallel -j0"
cat<<'EOF' > mycommand
#!/bin/bash
# Generate 1 GB output
yes "`perl -e 'print \"c\"x30_000'`" | head -c 1G
EOF
chmod+xmycommand
# Run 20 jobs in parallel
# Adjust 20 to be > physical RAM and < free space on /tmp
seq20|time$paralleltool./mycommand|wc-c

DISKFULL: Incomplete data if /tmp runs full

If caching is done on disk, the disk can run full during the run. Not all programs discover this. GNU Parallel discovers it, if it stays full for at least 2 seconds.

#!/bin/bash
paralleltool="parallel -j0"
# This should be a dir with less than 100 GB free space
smalldisk=/tmp/shm/parallel
TMPDIR="$smalldisk"
exportTMPDIR
max_output(){
# Force worst case scenario:
# Make GNU Parallel only check once per second
sleep10
# Generate 100 GB to fill $TMPDIR
# Adjust if /tmp is bigger than 100 GB
yes|head-c100G>$TMPDIR/$$
# Generate 10 MB output that will not be buffered
# due to full disk
perl-e'print "X"x10_000_000'|head-c10M
echoThispartismissingfromincompleteoutput
sleep2
rm$TMPDIR/$$
echoFinaloutput
}
export-fmax_output
seq10|$paralleltoolmax_output|tr-sX

CLEANUP: Leaving tmp files at unexpected death

Some tools do not clean up tmp files if they are killed. If the tool buffers on disk, they may not clean up, if they are killed.

#!/bin/bash
paralleltool=parallel
ls/tmp>/tmp/before
seq10|$paralleltoolsleep&
pid=$!
# Give the tool time to start up
sleep1
# Kill it without giving it a chance to cleanup
kill-9$!
# Should be empty: No files should be left behind
diff<(ls/tmp)/tmp/before

SPCCHAR: Dealing badly with special file names.

It is not uncommon for users to create files like:

My brother's 12" *** record (costs $$$).jpg

Some tools break on this.

#!/bin/bash
paralleltool=parallel
touch"My brother's 12\" *** record (costs \$\$\$).jpg"
lsMy*jpg|$paralleltoolls-l

COMPOSED: Composed commands do not work

Some tools require you to wrap composed commands into bash -c.

echobar|$paralleltoolechofoo';'echo{}

ONEREP: Only one replacement string allowed

Some tools can only insert the argument once.

echobar|$paralleltoolecho{}foo{}

INPUTSIZE: Length of input should not be limited

Some tools limit the length of the input lines artificially with no good reason. GNU parallel does not:

perl-e'print "foo."."x"x100_000_000'|parallelecho{.}

GNU parallel limits the command to run to 128 KB due to execve(1):

perl-e'print "x"x131_000'|parallelecho{}|wc

NUMWORDS: Speed depends on number of words

Some tools become very slow if output lines have many words.

#!/bin/bash
paralleltool=parallel
cat<<-EOF > mycommand
#!/bin/bash
# 10 MB of lines with 1000 words
yes "`seq 1000`" | head -c 10M
EOF
chmod+xmycommand
# Run 30 jobs in parallel
seq30|time$paralleltool-j0./mycommand>/dev/null

4GB: Output with a line > 4GB should be OK

#!/bin/bash
paralleltool="parallel -j0"
wcc(){
parallel--recend''--block100M--pipe'LC_ALL=C wc'|
datamash-Wsum1sum2sum3
}
cat<<-EOF > mycommand
#!/bin/bash
perl -e '\$a="a"x1000_000; for(1..5000) { print \$a }'
EOF
chmod+xmycommand
# Run 1 job
seq1|$paralleltool./mycommand|wcc

Documentation license I

Permission is granted to copy, distribute and/or modify this documentation under the terms of the GNU Free Documentation License, Version 1.3 or any later version published by the Free Software Foundation; with no Invariant Sections, with no Front-Cover Texts, and with no Back-Cover Texts. A copy of the license is included in the file LICENSES/GFDL-1.3-or-later.txt.

Documentation license II

You are free:

• to Share

to copy, distribute and transmit the work

• to Remix

to adapt the work

Under the following conditions:

• Attribution

You must attribute the work in the manner specified by the author or licensor (but not in any way that suggests that they endorse you or your use of the work).

• Share Alike

If you alter, transform, or build upon this work, you may distribute the resulting work only under the same, similar or a compatible license.

With the understanding that:

• Waiver

Any of the above conditions can be waived if you get permission from the copyright holder.

• Public Domain

Where the work or any of its elements is in the public domain under applicable law, that status is in no way affected by the license.

• Other Rights

In no way are any of the following rights affected by the license:

• Your fair dealing or fair use rights, or other applicable copyright exceptions and limitations;

• The author's moral rights;

• Rights other persons may have either in the work itself or in how the work is used, such as publicity or privacy rights.

• Notice

For any reuse or distribution, you must make clear to others the license terms of this work.

A copy of the full license is included in the file as LICENCES/CC-BY-SA-4.0.txt