The wonderful xargs command

I finally found a way I like to run a whole bunch of commands N at a time on an N core machine (well maybe use N-1 to be polite):

1. Say you have a command rprun.pl that takes 4 arguments that you want to run with 1000 different argument combinations.

2. You write a script rprun-args.pl that generates all combinations you need.  Say its output looks like:

10      185364  25      0.166
12      92682   25      0.166
18      65536   32      0.166
12      65536   25      0.7071
14      16384   25      0.166
...

3. Now you can use xargs to run these 24 at a time as follows:

rprun-args.pl | xargs -n4 -P24 rprun.pl > rprun.out

-n4 is to feed the arguments 4 at a time.  So a typical command line will look like:

rprun.pl 14 16384 25 0.166

-P24 tells xargs to run through the list 24 at a time.  If you run ps you will see 24 copies of rprun running together.  As soon as the number drops to 23 another child is spawned.

Note that the command above combines the outputs of all runs (in the order they finish) in the same file, so make sure rprun.pl prints out its arguments as well as its result on its output.