pcplantdb

[Richard Morris <webmaster@pfaf.org>]

Lawrence F. London, Jr. wrote:

> Sean Maley wrote:
>
> > ... spelling error, scraping.  Basically, we do what
> > google does.  We scan the web pages from other plant
> > database web sites and archive the information with
> > our own data.  This gathers information quickly and
> > adds to the available information that this database
> > provides for our users.  We may not have our own
> > information for a given species, but could still have
> > something to provide.  A simple report would provide
> > all the available opportunities to add to the archive
> > from a permaculture perspective.
>
>
> Could you provide more explanation of what this is about and how it 
> happens?
> Seems we have been talking about incorporating data from other databases
> and other text-based and marked-up (for the Web) data in archives, i.e. 
> mailing list archives,
> ftp archives, html-ized archives (mhonarc and hypermail) and you suggest 
> "scanning"
> select web accress for such data. What does this "scanning" consist of? 
> Would this
> produce an index of urls used to locate the various pieces of data 
> themselves? Or would this
> actually gather and archive this data on our machine for access and 
> redistribution through PIW,
> referencing the source and credit to its original location?
>
> This sounds like a great idea.

I've done this sort of stuff on the pfaf site. I've used programs like 
wget (on linux) to download entire website. I've then run some perl 
scripts over the downloaded files, these pickout the information I want 
from the web pages and write it into a database/text file.

I've mainly used this to build link to other database, get synonyms and 
common names of plants, and also stuff like the invasive status of 
certain plants.

It can be quite a tricky programing task as the html changes from site 
to site, is often inconsistant and a pain to parse. One of the reasons 
I'd like permaculture.info to be available in xml decause it would make 
it easier for others to read in.

Rich