Skip to Content.
Sympa Menu

pcplantdb - Re: [pcplantdb] Re: [piw] Information I would like to see

pcplantdb@lists.ibiblio.org

Subject: pcplantdb

List archive

Chronological Thread  
  • From: Richard Morris <webmaster@pfaf.org>
  • To: Permaculture Plant Database <pcplantdb@lists.ibiblio.org>
  • Subject: Re: [pcplantdb] Re: [piw] Information I would like to see
  • Date: Thu, 03 Feb 2005 22:46:19 +0000

Lawrence F. London, Jr. wrote:

Sean Maley wrote:

... spelling error, scraping. Basically, we do what
google does. We scan the web pages from other plant
database web sites and archive the information with
our own data. This gathers information quickly and
adds to the available information that this database
provides for our users. We may not have our own
information for a given species, but could still have
something to provide. A simple report would provide
all the available opportunities to add to the archive
from a permaculture perspective.


Could you provide more explanation of what this is about and how it happens?
Seems we have been talking about incorporating data from other databases
and other text-based and marked-up (for the Web) data in archives, i.e. mailing list archives,
ftp archives, html-ized archives (mhonarc and hypermail) and you suggest "scanning"
select web accress for such data. What does this "scanning" consist of? Would this
produce an index of urls used to locate the various pieces of data themselves? Or would this
actually gather and archive this data on our machine for access and redistribution through PIW,
referencing the source and credit to its original location?

This sounds like a great idea.


I've done this sort of stuff on the pfaf site. I've used programs like wget (on linux) to download entire website. I've then run some perl scripts over the downloaded files, these pickout the information I want from the web pages and write it into a database/text file.

I've mainly used this to build link to other database, get synonyms and common names of plants, and also stuff like the invasive status of certain plants.

It can be quite a tricky programing task as the html changes from site to site, is often inconsistant and a pain to parse. One of the reasons I'd like permaculture.info to be available in xml decause it would make it easier for others to read in.

Rich





Archive powered by MHonArc 2.6.24.

Top of Page