Public SourceMage Discussion List

[Andrew <afrayedknot AT thefrayedknot.armory.com>]

I like xargs, but rarely use it in favor of 
cmd `other_cmd`
style. However its a good tool and if you dont know it, you should :)
It's gotten me out of some ugly situations.

I'll share an anecdote from a project I was working on this summer where
xargs saved the day. I was running this tool tens of thousands of times
on a cluster of a hundred cpus, each process made 10 or so of these
little bitty summary files. I had this other script that liked reading in
files and averaging each field over all the input. So we've got about a
million files (all with large unique names) that all need to be sent as
arguments to this wimpy perl script. Well it turns out that, at least
on the linux system I was using, that there is a limit to the number
of characters you can pass in the argument stream to a program, not the
number of arguments.  So concievably you could pass about 16k of single
character space seperated parameters, or a single 32k character option, etc.

I potentially had millions of files, or more if we decided to crank up
the tool further. It turns out that xargs is privey to the character
limit and will call the command you give it with up to 32k characters
worth of arguments, then call it again later. So a
find .|xargs|cat|average.pl
later and I was set. find provided the listing of the millions of files
I had made, xargs stuffed the args into cat over and over again, which
continued to feed my perl script.

-Andrew


On Wed, Apr 14, 2004 at 06:56:49PM -0400, Paul wrote:
> It's only necessary when you have the possibility of a lot of
> parameters. My guess is 32k. Should you desire to know the exact number,
> here's a one liner for you:
> ( f() { [[ $1 -eq $# ]] && echo "$1==$#"; } ; let i=2; arg="1"; while :
> ; do arg="$i ${arg}"; f $arg || break; let i++; done )
> That should stop when the number of args is wrong. We may use xargs in a
> place or two now, where it makes things easier to understand.
> 
> On Wed, 2004-04-14 at 17:42, Eric Sandall wrote:
> > I just saw a brief article[0] on linux.com about xargs and am wondering 
> > if that
> > might help our Sorcery team (though from what I remember at least some of 
> > them
> > already know about it ;)).
> > 
> > [0] http://www.linux.com/article.pl?sid=04/04/13/211209

-- 
__________________________________________________________________________
|Andrew D. Stitt                           | astitt at sourcemage.org    |
|irc: afrayedknot                          | afrayedknot at t.armory.com |
|aim: thefrayedknot or iteratorplusplus    |                             |
|Sorcery Team Lead, Porting Team Lead      |                             |
|Grimoire Guru ham/smgl                    | ftp://t.armory.com          |
|Author and Maintainer of Prometheus       |                             |
--------------------------------------------------------------------------