Skip to Content.
Sympa Menu

sm-sorcery-bugs - [SM-Sorcery-Bugs] [Bug 9391] New: summon seems to take forever

sm-sorcery-bugs AT lists.ibiblio.org

Subject: Bugs for Sorcery are reported here

List archive

Chronological Thread  
  • From: bugzilla-daemon AT bugs.sourcemage.org
  • To: sm-sorcery-bugs AT lists.ibiblio.org
  • Subject: [SM-Sorcery-Bugs] [Bug 9391] New: summon seems to take forever
  • Date: 3 Jan 2006 01:09:26 -0000

http://bugs.sourcemage.org/show_bug.cgi?id=9391

Summary: summon seems to take forever
Product: Sorcery
Version: 1.14.x
Platform: Other
OS/Version: other
Status: NEW
Severity: normal
Priority: P2
Component: Summon
AssignedTo: sm-sorcery-bugs AT lists.ibiblio.org
ReportedBy: dmlb2000 AT gmail.com


Summoning spells seems to take forever and I don't know if it's because of the
new dl_handlers or what but here's what I did.

root@dmlb2004:/home/dmlb2000# export DEBUG=/root/sorcery.debug.log
root@dmlb2004:/home/dmlb2000# time summon -d cpufrequtils
Downloading source file cpufrequtils-0.3.tar.bz2 for spell cpufrequtils
Custom is not a valid handler, this is probably a sorcery bug
--11:15:24--
ftp://ftp.kernel.org/pub/linux/utils/kernel/cpufreq/cpufrequtils-0
.3.tar.bz2
=> `cpufrequtils-0.3.tar.bz2'
Resolving ftp.kernel.org... 204.152.191.5, 204.152.191.37
Connecting to ftp.kernel.org|204.152.191.5|:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done. ==> PWD ... done.
==> TYPE I ... done. ==> CWD /pub/linux/utils/kernel/cpufreq ... done.
==> PASV ... done. ==> RETR cpufrequtils-0.3.tar.bz2 ... done.
Length: 421,486 (412K) (unauthoritative)

100%[====================================>] 421,486 382.54K/s

11:15:26 (381.48 KB/s) - `cpufrequtils-0.3.tar.bz2' saved [421486]

Downloading source file cpufrequtils-0.3.tar.bz2.sign for spell cpufrequtils
Custom is not a valid handler, this is probably a sorcery bug
--11:18:30--
ftp://ftp.kernel.org/pub/linux/utils/kernel/cpufreq/cpufrequtils-0
.3.tar.bz2.sign
=> `cpufrequtils-0.3.tar.bz2.sign'
Resolving ftp.kernel.org... 204.152.191.37, 204.152.191.5
Connecting to ftp.kernel.org|204.152.191.37|:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done. ==> PWD ... done.
==> TYPE I ... done. ==> CWD /pub/linux/utils/kernel/cpufreq ... done.
==> PASV ... done. ==> RETR cpufrequtils-0.3.tar.bz2.sign ... done.
Length: 248 (unauthoritative)

100%[====================================>] 248 --.--K/s

11:18:31 (19.78 KB/s) - `cpufrequtils-0.3.tar.bz2.sign' saved [248]


real 6m2.732s
user 5m35.408s
sys 0m19.253s
root@dmlb2004:/home/dmlb2000#


Uhm yeah this is slow, really really slow.


------- Additional Comments From dmlb2000 AT gmail.com 2005-08-02 19:19 -------


------- Additional Comments From dmlb2000 AT gmail.com 2005-08-02 19:22 -------
Created an attachment (id=4450)
--> (http://bugs.sourcemage.org/attachment.cgi?id=4450&action=view)
--> (http://bugs.sourcemage.org/attachment.cgi?id=4450&action=view)
sorcery debug output


--


------- Additional Comments From acedit AT armory.com 2005-08-02 21:28 -------
Its slow because theres a huge number of kernel mirrors, nothing wrong with
that, just some part of the processing is slow. The rest of the code shouldn't
be any slower than before.

The visible error message is because the "Custom" line in the urls file didnt
get stripped out as it should have been.

--


------- Additional Comments From acedit AT armory.com 2005-08-02 22:50 -------
the main slowdown is list_add being slow i think. Other parts are slow due to
excessive forking. In particular url_expand_urls, url_get_valid_urls, and the
bucketization. In that order. We should avoid repeated calls to
url_get_prefix,
maybe find a way to cache that, and/or use upvars in place of forks.

This reminds me that we probably should set some sort of threshold on the
maximum number of mirrors expanded. After the urls are sorted the list should
be
truncated; that would avoid having 400 some mirrors to slog through (both in
internal processing and before sorcery declares the file cant be found). Not
that fixing up these slower bits isn't important. Also, Im not certain all
those
kernel mirrors are valid, so we'll have to look through them and cull
out-dated
ones.

In summary:
fix expanding code to drop "Custom"
fix code to not use list_add, use better placed greps instead
fix url_get_valid_urls/url_default_is_valid to run faster
fix bucketizer to run slightly faster
cache url to prefix
limit max number of urls expanded/sorted

--


------- Additional Comments From dmlb2000 AT gmail.com 2005-08-03 04:52 -------
I'm not a sorcery programmer, but a suggestion for speeding up url_get_prefix

function url_get_prefix() {
$STD_DEBUG
local URL=$1
local URL_PREFIX=${URL/://*}
[ -n "$URL_PREFIX" ] &&
echo $URL_PREFIX
}

calling sed is faster than awk but staying in bash is even faster ;)

--


------- Additional Comments From acedit AT armory.com 2005-08-03 05:03 -------
Indeed it is :-) Return by reference is also a big speed booster.

--


------- Additional Comments From acedit AT armory.com 2005-08-03 06:10 -------
okay, with change 63635 I got what will likely be 90% of the speedups. I
included the get_prefix update and added another condition to verify a prefix
exists, this fixes the 'Custom' problem. I removed forking from url_is_valid
which speeds it up immensly. I sped up url_expand_urls and url_get_valid_urls
as
I described. I also truncate the url list at 20.

Running the following (this is what summon calls, copied verbatim from debug
log)
download_src_args "cpufrequtils-0.3.tar.bz2"
"ftp://ftp.kernel.org/pub/linux/utils/kernel/cpufreq/cpufrequtils-0.3.tar.bz2";
""
takes about 8 to 10 seconds on my machine with, around 5-7 of which is
downloading. Without the url truncation it takes 40 seconds including
download.
I think thats a big improvement over the 3 minutes initially reported. Id like
to get further since 5 seconds of overhead is still too much.

Doing some back of the envelope math shows that url expanding is taking
.0075s/url, and url validation+bucketizing takes about .0675s/url. I get that
from 5=500X+20Y and 35=500X+500Y, where X is the time to expand and Y is time
to
validate+bucketize. 500 is about the number of urls we expand the kernel url
to.
In the common case we're spending 3.75 seconds expanding (to 500) urls and
1.25
seconds processing the 20 that matter.

We have both ftp and http mirrors for practically every country or region.
Many
of the http and ftp sites are the same machine. I dont see a whole lot of use
in
keeping both in the list, if people really care, they have the Custom option.
Next theres probably some in that list that are not pingable or not up to
date,
those should be removed. Im expecting to cull down to 200 mirrors from about
500. That would take the 3.75 expansion time to 1.5 seconds.

I think after that i'll dig into making all the hot api functions use return
by
reference.

--


------- Additional Comments From dmlb2000 AT gmail.com 2005-08-03 06:23 -------
Yeah I updated and that's a vast improvement from before :DD

since you want to do a little more should I leave this open then?


--


------- Additional Comments From acedit AT armory.com 2005-08-03 06:25 -------
Yes, I deliberately left it unresolved.

--


------- Additional Comments From acedit AT armory.com 2005-09-05 21:22 -------
code works satisfactorially fast so im going to put off doing the remainder of
the work til later.

--


------- Additional Comments From dmlb2000 AT gmail.com 2006-01-02 19:09 -------
I'm poking this bug to see if/when the rest of the improvements might be
implemented? or if help would be needed to implement these?

--
Configure bugmail: http://bugs.sourcemage.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.



  • [SM-Sorcery-Bugs] [Bug 9391] New: summon seems to take forever, bugzilla-daemon, 01/02/2006

Archive powered by MHonArc 2.6.24.

Top of Page