pcplantdb

[Bear Kaufmann <bear@ursine-design.com>]

> > Hope you're all doing well. There seems to be a lack of activity 
> > occurring on the list. Is everyone preparing for spring or something?
> >    
>
> I got zero response either way on my proposal to do some work on the 
> tracker and the QA/testing documentation so I have not done anything.
>
> Doesn't seem urgent, since nothing else seems to be happening at the moment.
>  
Sure. Could you post the URL again? Last time I tried I got a "no object 
found error".

> > ...The find "find similar" method is rather intensive (looking through a 
> > 45Mb database of words takes some time) and has some kinks to work out, 
> > but I found this result to be pretty cool.
> >
> > See it here:
> > http://permaculture.info/test/related_words.php
> >    
>
> Not sure I understand exactly what you're up to here...
> Off the top, the results remind me of those spam messages that were real 
> prolific a few years ago, consisting of an apparently unrelated jumble 
> of words/sentences assembled into the message body... :-)
>  
Yeah, I can see that. :-)
To provide an example of the principal in use, take a look at del.icio.us.
Say:
http://del.icio.us/tag/science

You'll notice that it pulls up 'related tags'. This are tags which 
people commonly also use when tagging the same document.
I've done the same thing, except with the full text entries of the database.
Since I use every word in the text, the signal to noise ratio is high, 
unlike a set of tags which would generally be 'keywords'.
The words follow a power law curve:
http://permaculture.info/test/most_common_words.html  (top 3000 words of 
16,000+)

If you search for 'woven' in the related words, you'll see a reasonable 
example.
It includes: layers thatch chairs insulating nappies sheets blankets
'Ice' includes: cream sweets creams canada spearmint
'Fronds' includes: ferns spores carcinogens humid fern plantlets bracken
'Stronger' includes: oregano savory onions crispum hortensis pepo 
pumpkins affecting beet chamomile curly digitalis dogs gutta-percha 
moles onion marjoram marrows.

Many of these may just be noise, and the algorithm could probably be 
improved to remove that.
What use does it have?
Well, if you're developing an ontology, it might be good to know related 
words in subject area.
If you wanted to suggest to a user some other search terms, this might help.
It's just interesting to see what gets mentioned together in the text.

Regarding the spaminess of the text: I think those messages where meant 
to defeat an anti-spam measure that looked for the complexity of the 
text used, so 'hey wazzup wanna buy some pills' couldn't get through. So 
since this technique pulls from the less common words, you get a similar 
feel.

Since I wrote the previous message, I've found a way to use the word 
list to 'find similar' quite fast all in one SQL query (though the 
miracle of LEFT JOIN and GROUP BY). It's looking good.
One sign that it's working is that the top results are generally coming 
from the same family.
The technique is interesting in that you can fine tune for the S:N 
ratio, and other variables, and get some differing results.
I've posted some find similar' results as static HTML here:
http://permaculture.info/test/related_Abutilon_vitifolium.html
http://permaculture.info/test/related_Abelmoschus_esculentus.html
http://permaculture.info/test/related_Acer_crataegifolium.html

I don't know how much space ibiblio is willing to host, but this extra 
word table is quite useful (but about 74Mb with index). Culling the top 
words would probably cut that size down a good bit.
We can also use this method on smaller sets, like uses, pollinators, or 
a pared down word table etc. This would be like the 'tagging' system we 
talked about a while ago.

Cheers,
Bear