Bugs for Sorcery are reported here

[bugzilla-daemon AT bugs.sourcemage.org]

http://bugs.sourcemage.org/show_bug.cgi?id=9391

           Summary: summon seems to take forever
           Product: Sorcery
           Version: 1.14.x
          Platform: Other
        OS/Version: other
            Status: NEW
          Severity: normal
          Priority: P2
         Component: Summon
        AssignedTo: sm-sorcery-bugs AT lists.ibiblio.org
        ReportedBy: dmlb2000 AT gmail.com


Summoning spells seems to take forever and I don't know if it's because of the
new dl_handlers or what but here's what I did.

root@dmlb2004:/home/dmlb2000# export DEBUG=/root/sorcery.debug.log
root@dmlb2004:/home/dmlb2000# time summon -d cpufrequtils
Downloading source file cpufrequtils-0.3.tar.bz2 for spell cpufrequtils
Custom is not a valid handler, this is probably a sorcery bug
--11:15:24--  
ftp://ftp.kernel.org/pub/linux/utils/kernel/cpufreq/cpufrequtils-0
.3.tar.bz2
           => `cpufrequtils-0.3.tar.bz2'
Resolving ftp.kernel.org... 204.152.191.5, 204.152.191.37
Connecting to ftp.kernel.org|204.152.191.5|:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done.    ==> PWD ... done.
==> TYPE I ... done.  ==> CWD /pub/linux/utils/kernel/cpufreq ... done.
==> PASV ... done.    ==> RETR cpufrequtils-0.3.tar.bz2 ... done.
Length: 421,486 (412K) (unauthoritative)

100%[====================================>] 421,486      382.54K/s

11:15:26 (381.48 KB/s) - `cpufrequtils-0.3.tar.bz2' saved [421486]

Downloading source file cpufrequtils-0.3.tar.bz2.sign for spell cpufrequtils
Custom is not a valid handler, this is probably a sorcery bug
--11:18:30--  
ftp://ftp.kernel.org/pub/linux/utils/kernel/cpufreq/cpufrequtils-0
.3.tar.bz2.sign
           => `cpufrequtils-0.3.tar.bz2.sign'
Resolving ftp.kernel.org... 204.152.191.37, 204.152.191.5
Connecting to ftp.kernel.org|204.152.191.37|:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done.    ==> PWD ... done.
==> TYPE I ... done.  ==> CWD /pub/linux/utils/kernel/cpufreq ... done.
==> PASV ... done.    ==> RETR cpufrequtils-0.3.tar.bz2.sign ... done.
Length: 248 (unauthoritative)

100%[====================================>] 248           --.--K/s

11:18:31 (19.78 KB/s) - `cpufrequtils-0.3.tar.bz2.sign' saved [248]


real    6m2.732s
user    5m35.408s
sys     0m19.253s
root@dmlb2004:/home/dmlb2000#


Uhm yeah this is slow, really really slow.


------- Additional Comments From dmlb2000 AT gmail.com  2005-08-02 19:19 -------


------- Additional Comments From dmlb2000 AT gmail.com  2005-08-02 19:22 -------
Created an attachment (id=4450)
 --> (http://bugs.sourcemage.org/attachment.cgi?id=4450&action=view)
--> (http://bugs.sourcemage.org/attachment.cgi?id=4450&action=view)
sorcery debug output


--


------- Additional Comments From acedit AT armory.com  2005-08-02 21:28 -------
Its slow because theres a huge number of kernel mirrors, nothing wrong with
that, just some part of the processing is slow. The rest of the code shouldn't
be any slower than before.

The visible error message is because the "Custom" line in the urls file didnt
get stripped out as it should have been.

--


------- Additional Comments From acedit AT armory.com  2005-08-02 22:50 -------
the main slowdown is list_add being slow i think. Other parts are slow due to
excessive forking. In particular url_expand_urls, url_get_valid_urls, and the
bucketization. In that order. We should avoid repeated calls to 
url_get_prefix,
maybe find a way to cache that, and/or use upvars in place of forks.

This reminds me that we probably should set some sort of threshold on the
maximum number of mirrors expanded. After the urls are sorted the list should 
be
truncated; that would avoid having 400 some mirrors to slog through (both in
internal processing and before sorcery declares the file cant be found). Not
that fixing up these slower bits isn't important. Also, Im not certain all 
those
kernel mirrors are valid, so we'll have to look through them and cull 
out-dated
ones.

In summary:
fix expanding code to drop "Custom"
fix code to not use list_add, use better placed greps instead
fix url_get_valid_urls/url_default_is_valid to run faster
fix bucketizer to run slightly faster
cache url to prefix
limit max number of urls expanded/sorted

--


------- Additional Comments From dmlb2000 AT gmail.com  2005-08-03 04:52 -------
I'm not a sorcery programmer, but a suggestion for speeding up url_get_prefix

function url_get_prefix()  {
$STD_DEBUG
local URL=$1
local  URL_PREFIX=${URL/://*}
[  -n  "$URL_PREFIX"  ]  &&
echo    $URL_PREFIX
}

calling sed is faster than awk but staying in bash is even faster ;)

--


------- Additional Comments From acedit AT armory.com  2005-08-03 05:03 -------
Indeed it is :-) Return by reference is also a big speed booster.

--


------- Additional Comments From acedit AT armory.com  2005-08-03 06:10 -------
okay, with change 63635 I got what will likely be 90% of the speedups. I
included the get_prefix update and added another condition to verify a prefix
exists, this fixes the 'Custom' problem. I removed forking from url_is_valid
which speeds it up immensly. I sped up url_expand_urls and url_get_valid_urls 
as
I described. I also truncate the url list at 20.

Running the following (this is what summon calls, copied verbatim from debug 
log)
download_src_args "cpufrequtils-0.3.tar.bz2"
"ftp://ftp.kernel.org/pub/linux/utils/kernel/cpufreq/cpufrequtils-0.3.tar.bz2";
 ""
takes about 8 to 10 seconds on my machine with, around 5-7 of which is
downloading. Without the url truncation it takes 40 seconds including 
download.
I think thats a big improvement over the 3 minutes initially reported. Id like
to get further since 5 seconds of overhead is still too much.

Doing some back of the envelope math shows that url expanding is taking
.0075s/url, and url validation+bucketizing takes about .0675s/url. I get that
from 5=500X+20Y and 35=500X+500Y, where X is the time to expand and Y is time 
to
validate+bucketize. 500 is about the number of urls we expand the kernel url 
to.
In the common case we're spending 3.75 seconds expanding (to 500) urls and 
1.25
seconds processing the 20 that matter.

We have both ftp and http mirrors for practically every country or region. 
Many
of the http and ftp sites are the same machine. I dont see a whole lot of use 
in
keeping both in the list, if people really care, they have the Custom option.
Next theres probably some in that list that are not pingable or not up to 
date,
those should be removed. Im expecting to cull down to 200 mirrors from about
500. That would take the 3.75 expansion time to 1.5 seconds.

I think after that i'll dig into making all the hot api functions use return 
by
reference.

--


------- Additional Comments From dmlb2000 AT gmail.com  2005-08-03 06:23 -------
Yeah I updated and that's a vast improvement from before :DD

since you want to do a little more should I leave this open then?


--


------- Additional Comments From acedit AT armory.com  2005-08-03 06:25 -------
Yes, I deliberately left it unresolved.

--


------- Additional Comments From acedit AT armory.com  2005-09-05 21:22 -------
code works satisfactorially fast so im going to put off doing the remainder of
the work til later.

--


------- Additional Comments From dmlb2000 AT gmail.com  2006-01-02 19:09 -------
I'm poking this bug to see if/when the rest of the improvements might be
implemented? or if help would be needed to implement these?

-- 
Configure bugmail: http://bugs.sourcemage.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.