Skip to Content.
Sympa Menu

pcplantdb - Re: [pcplantdb] Activity?

pcplantdb@lists.ibiblio.org

Subject: pcplantdb

List archive

Chronological Thread  
  • From: Bear Kaufmann <bear@ursine-design.com>
  • To: john@eco-living.net, pcplantdb <pcplantdb@lists.ibiblio.org>
  • Subject: Re: [pcplantdb] Activity?
  • Date: Mon, 27 Feb 2006 22:27:34 -0800



Hope you're all doing well. There seems to be a lack of activity occurring on the list. Is everyone preparing for spring or something?


I got zero response either way on my proposal to do some work on the tracker and the QA/testing documentation so I have not done anything.

Doesn't seem urgent, since nothing else seems to be happening at the moment.

Sure. Could you post the URL again? Last time I tried I got a "no object found error".

...The find "find similar" method is rather intensive (looking through a 45Mb database of words takes some time) and has some kinks to work out, but I found this result to be pretty cool.

See it here:
http://permaculture.info/test/related_words.php


Not sure I understand exactly what you're up to here...
Off the top, the results remind me of those spam messages that were real prolific a few years ago, consisting of an apparently unrelated jumble of words/sentences assembled into the message body... :-)

Yeah, I can see that. :-)
To provide an example of the principal in use, take a look at del.icio.us.
Say:
http://del.icio.us/tag/science

You'll notice that it pulls up 'related tags'. This are tags which people commonly also use when tagging the same document.
I've done the same thing, except with the full text entries of the database.
Since I use every word in the text, the signal to noise ratio is high, unlike a set of tags which would generally be 'keywords'.
The words follow a power law curve:
http://permaculture.info/test/most_common_words.html (top 3000 words of 16,000+)

If you search for 'woven' in the related words, you'll see a reasonable example.
It includes: layers thatch chairs insulating nappies sheets blankets
'Ice' includes: cream sweets creams canada spearmint
'Fronds' includes: ferns spores carcinogens humid fern plantlets bracken
'Stronger' includes: oregano savory onions crispum hortensis pepo pumpkins affecting beet chamomile curly digitalis dogs gutta-percha moles onion marjoram marrows.

Many of these may just be noise, and the algorithm could probably be improved to remove that.
What use does it have?
Well, if you're developing an ontology, it might be good to know related words in subject area.
If you wanted to suggest to a user some other search terms, this might help.
It's just interesting to see what gets mentioned together in the text.

Regarding the spaminess of the text: I think those messages where meant to defeat an anti-spam measure that looked for the complexity of the text used, so 'hey wazzup wanna buy some pills' couldn't get through. So since this technique pulls from the less common words, you get a similar feel.

Since I wrote the previous message, I've found a way to use the word list to 'find similar' quite fast all in one SQL query (though the miracle of LEFT JOIN and GROUP BY). It's looking good.
One sign that it's working is that the top results are generally coming from the same family.
The technique is interesting in that you can fine tune for the S:N ratio, and other variables, and get some differing results.
I've posted some find similar' results as static HTML here:
http://permaculture.info/test/related_Abutilon_vitifolium.html
http://permaculture.info/test/related_Abelmoschus_esculentus.html
http://permaculture.info/test/related_Acer_crataegifolium.html

I don't know how much space ibiblio is willing to host, but this extra word table is quite useful (but about 74Mb with index). Culling the top words would probably cut that size down a good bit.
We can also use this method on smaller sets, like uses, pollinators, or a pared down word table etc. This would be like the 'tagging' system we talked about a while ago.

Cheers,
Bear





Archive powered by MHonArc 2.6.24.

Top of Page