Bugs for Sorcery are reported here

[bugzilla-daemon AT bugs.sourcemage.org]

http://bugs.sourcemage.org/show_bug.cgi?id=9415

           Summary: summon could stand to have some sanity checking
           Product: Sorcery
           Version: 1.14.x
          Platform: Other
        OS/Version: other
            Status: RESOLVED
          Severity: enhancement
          Priority: P2
         Component: Summon
        AssignedTo: sm-sorcery-bugs AT lists.ibiblio.org
        ReportedBy: jblosser-smgl AT firinn.org


In the last few weeks I've run into the following:

- summon on a tarball at an http URL gets back an HTML page
- summon on a tarball gets back a 0 byte file

In both cases summon decided it was done, even though these weren't the files 
we
were after.  It would be nice if summon could detect this kind of temporary
failure at a mirror and continue on to the next mirror or fallback.  Both of 
the
above could be avoided by 1) checking that the file retrieved is at least not 0
bytes and 2) if the expected file type is known, verifying that the file
retrieved is the correct file type.

I don't know if this is worth doing or not; I doubt we want to go much farther
than this since it would probably end with making summon do the md5/gpg/etc.
checks after download, and that sounds excessive.


------- Additional Comments From acedit AT armory.com  2005-08-08 00:14 -------
The main problem is, wget doesnt recognize that as an error (and it should 
IMO).
I think curl might be able to treat those cases as errors as they are usually
accompanied by an http redirect.

0 byte files could easily be special cased, an html page although not common
might be a legitimate thing to download.

The new downloading code in devel has a 'hints' feature, which could help with
this though. The wget handler could be taught to fail if the file is too small
and made only of ascii text; as many redirection html garbage pages are. There
could be a hint to disable that check for spells that legitimately need to
download small text files.



--


------- Additional Comments From jblosser-smgl AT firinn.org  2005-08-08 00:32 
-------
(In reply to comment #1)
> The main problem is, wget doesnt recognize that as an error (and it should 
> IMO).

They aren't going to be seen as errors because they are HTTP status code 200,
which means the server said "this is the file you wanted, we are done".  HTTP
200 is like a 0 exit code in shell.

> I think curl might be able to treat those cases as errors as they are 
> usually
> accompanied by an http redirect.

HTTP redirects would usually be status code 301 or 302.

> 0 byte files could easily be special cased, an html page although not common
> might be a legitimate thing to download.

Right, I'm not suggesting we would want summon to assume something is wrong 
just
because the file type isn't tar-compressed.  We would only want it to think
something was wrong if it was requesting a tar.gz and got back an HTML page 
(or
was requesting an HTML page and got back a tar.gz).


--


------- Additional Comments From acedit AT armory.com  2005-08-08 00:47 -------
(In reply to comment #2)
> (In reply to comment #1)
> > The main problem is, wget doesnt recognize that as an error (and it 
> > should IMO).
>
> They aren't going to be seen as errors because they are HTTP status code 
> 200,
> which means the server said "this is the file you wanted, we are done".  
> HTTP
> 200 is like a 0 exit code in shell.

In that case the url should be fixed (actually, in all these cases they should
be, and it's perfectly automatable to check for these problems and report them
as bugs automatically). Although nearly every case of this ive seen is 301 or
302. NOT 200. If they're giving 200 and an html file then thats just stupid.


In anycase, I'd rather not start trying to play guessing games based on file
extension. Id rather either use some simple heuristic or go for full 
verification.

--


------- Additional Comments From jblosser-smgl AT firinn.org  2005-08-08 01:07 
-------
(In reply to comment #3)
> (In reply to comment #2)
> > (In reply to comment #1)
> > > The main problem is, wget doesnt recognize that as an error (and it 
> > > should
IMO).
> >
> > They aren't going to be seen as errors because they are HTTP status code 
> > 200,
> > which means the server said "this is the file you wanted, we are done".  
> > HTTP
> > 200 is like a 0 exit code in shell.
>
> In that case the url should be fixed (actually, in all these cases they 
> should
> be, and it's perfectly automatable to check for these problems and report 
> them
> as bugs automatically). Although nearly every case of this ive seen is 301 
> or
> 302. NOT 200. If they're giving 200 and an html file then thats just stupid.

Some of them are probably transient errors; the 0 file size one I got was the
file that came back from one particular sourceforge mirror which I imagine 
will
get fixed on its own at some point.

The 200 html file was one of the apache mirrors that had been taken over by a
domain squatter and was returning some search page for any request it got.
Obviously a bug and it was reported as such and you removed that mirror from 
the
list, but in the meantime all casts that hit that mirror were failing when 
they
could've (feasibly) gone on to other mirrors.

--


------- Additional Comments From eric AT sandall.us  2005-08-08 19:13 -------
Perhaps a simple compare of "$SOURCE" == "$SOURCE_CACHE/$SOURCE" would be
sufficient (at least for case #1).

--


------- Additional Comments From acedit AT armory.com  2005-08-08 19:46 -------
(In reply to comment #5)
> Perhaps a simple compare of "$SOURCE" == "$SOURCE_CACHE/$SOURCE" would be
> sufficient (at least for case #1).

Im not sure I follow you on that one.

--


------- Additional Comments From eric AT sandall.us  2005-08-09 16:45 -------
Compare the filename that was supposed to be downloaded ($SOURCE) with the
filename that was downloaded (I haven't looked to see what variable Sorcery 
uses
to refer to the downloaded file to move it to $SOURCE_CACHE/$SOURCE).

--


------- Additional Comments From jblosser-smgl AT firinn.org  2005-08-09 16:51 
-------
The server is giving us the file(name) we requested, though the content is
wrong.  summon can't tell the difference by filename.

--


------- Additional Comments From acedit AT armory.com  2005-08-09 17:30 -------
(In reply to comment #7)
> Compare the filename that was supposed to be downloaded ($SOURCE) with the
> filename that was downloaded (I haven't looked to see what variable Sorcery 
> uses
> to refer to the downloaded file to move it to $SOURCE_CACHE/$SOURCE).

the wget code only suceeds if the filename requested is actually downloaded,
garbage filenames are dropped on the floor, so we already check for that.

--


------- Additional Comments From acedit AT armory.com  2006-04-23 19:48 -------
I added a sanity check to catch the case where a summon of a tar.bz2, tar.gz 
or
tgz ends up as an html file instead.

fixed in devel at 23

-- 
Configure bugmail: http://bugs.sourcemage.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.