pcplantdb

[Chad Knepp <pyg@galatea.org>]

Plants For A Future writes:
 > On Fri, 24 Dec 2004, Chad Knepp wrote:
 > 
 > > Richard Morris writes:
 > > > Lawrence F. London, Jr. wrote:
 > > > > Chad Knepp wrote:
 > > > >
 > > > >> Well, I can think of a couple of reasons.  Elaeagnus has an 'a' in 
 > > > >> it
 > > > >> that I forget a lot myself.  Eden does accommodate 'alternative'
 > > > >
 > > > Spelling of latin names is a big issue.
 > > > The way I've handled this is to put every concievable spelling as a 
 > > > synonym.
 > >
 > > Interesting.  I've handled it very differently.  Eden separates words
 > > into substrings and looks for names that are 'like %substring%'.  If
 > > you spell the word correctly every substring matches.  If you butcher
 > > it, hopefully some of the substrings will match.  I've found this to be
 > > both very simple and very effective solution.  One of the performance
 > > enhancing changes I made recently was decreasing the number of
 > > substrings a word is seperated into.  Perhaps I've changed it too far
 > > the other way.
 > >
 > > Just for fun, I'm appending the substring method from Eden/Search.py.
 > > I think it's a neat bit of code.
 > >
 > Cool, it is a neet way of doing it. There are efficiency questions
 > as there can be a lot of substrings to test for. I've had a play with
 > this sort of stuff in the past and there are different ways to implement
 > such a scheme some are more efficient way to do it. Can't remember off
 > top of my head which way to order the loops but there are ways which
 > prevent most of tests. Do we have any way of testing time for these
 > sorts of things. If spped turns out to be a problem then this bit can be
 > tuned a lot. Might even consider implementing in C for max efficiency.

Yes, the subStrings method is a good candidate for optimizing in C,
although I say [over and over] optimization after implementation...

Currently I don't think we have [or will have] any kind of performance
problems.  Every search is cached, so even the most expensive search
is only executed once.  In terms of substrings an expensive search
could at most contain 8-10 substring based selects.  The most expensive
search possible would have less than 40 'insert into _tmp select from
...' sql executions which might sound like a lot except for that each
query executes in hundredths of a second.

 > > --
 > > Chad Knepp
 > > python -c 'import base64;print 
 > > base64.decodestring("cHlnQGdhbGF0ZWEub3Jn")'
 > >
 > >
 > > def subStrings(word, substrings, min_len = 3):
 > >    """
 > >    Suppa duppa substring smashing action! Hi-Yaaaaa....
 > >
 > >    Divides 'word' into 'substrings' number of sub strings that
 > >    have a minimum number of characters of 'min_length'
 > >
 > >    Returns list of sub strings
 > >    """
 > >
 > >    strings = []
 > >
 > >    increment = len(word) / substrings
 > >    extra_chars = len(word) % substrings
 > >
 > >    if increment < min_len:
 > >        increment = min_len
 > >        extra_chars = len(word) % min_len
 > >        substrings = len(word) / min_len
 > >
 > >    for chunk in range(substrings):
 > >        if extra_chars:
 > >            strings.append(word[:increment+1])
 > >            word = word[increment+1:]
 > >            extra_chars -= 1
 > >        else:
 > >            strings.append(word[:increment])
 > >            word = word[increment:]
 > >
 > >    return strings
 > >
 >   yes this is neet, whats the typical length of a substring?

It can be what ever you want, Search.String class (the basic search)
seperates the word into '3' strings but actual length of the substring
is dependent on the length of the word itself.  Search.Name
(botanical/common name search) splits the word into '4' substrings.

 > I've got a lot of log file data which records all the searches which
 > people have done. I could pass some of this on if you want some data to
 > test on.

Yeah, this would be helpful info along with the topics stuff.

-- 
Chad Knepp
python -c 'import base64;print base64.decodestring("cHlnQGdhbGF0ZWEub3Jn")'