Source Mage code commit list

[Remko van der Vossen <wich AT yuugen.jp>]

On Mon, Nov 10, 2008 at 01:07:11PM -0500, flux wrote:
> Remko van der Vossen (wich AT yuugen.jp) wrote [08.11.10 12:50]:
> > On Mon, Nov 10, 2008 at 12:21:47PM -0500, flux wrote:
> > No, this is not a bug in pipes, processes are supposed to run in
> > parallel, that't the whole idea. Think of a very very big input stream
> > that you want to run grep over, you really don't want the pipe blocking
> > until the process providing the input stream has finished, you might not
> > even have enough memory to hold the entire stream. And waiting for the
> > first bits of data through the stream hardly helps either, because that
> > just means that the file will be overwritten when the process providing
> > the input stream has only read it partly. So, yep, a pipe dream. ;)
> 
> Running grep by itself on large enough files may exhaust memory,
> particularly depending on the regexp (a regexp with backreferences over
> a sufficiently complicated search space, for example). You don't need a
> pipe to cause that problem. ;)
> 
> As for the file being partially overwritten, according to the tests that
> Jaka ran that is what can actually happen anyway (since it's not
> necessarily synchronous).

okay, different example:

# wc -l members
96489136596145 members
# grep -v unpaid | sed -e 's/2008/2009/' > members

Obviously, you can't have the grep run on the whole file before the sed
starts running on the grep output, that would fill the memory with the
enormous amount of data of the members file. Also, obviously you can't
have sed buffer it's output until EOS before writing it to the file as
that would have the same effect. Hence you must have both processes run
in parallel reading as much as they can and writing out as much as they
can. That is the whole power of pipes, you can use small simple programs
that don't need to be aware of the actual data, the vastness of the data
or any other edge cases is of no importance to the tool itself. It's the
whole idea of UNIX; Do one thing, do it well. However this of course
means that the sed will start writing over the members file before grep
has read it all.

If there is anything that should handle this it would need to be the
file system which would register a read lock for one process and a write
lock on the other process, however as you can't know which happens first
that won't get you much either.

And of course we're overlooking the fact here that the shell opens the
members for for writing with a truncate flag, usually before grep will
have opened the file for reading. However if the shell has already
exec'd the grep just before a time slice ends, then grep will open the
file first and get some (and in the case of a small file all) of the
input.

Anyway, enough rambling from me.

Remko.

Attachment: pgpxOfXQdHD0Q.pgp
Description: PGP signature