Permaculture Information Web

[Richard Morris <webmaster AT pfaf.org>]

Hi Folks,
	My latest wheeze, not sure if its strictly on topic for 
permaculture.info but here goes.

I've been pondering a graphing program which could pull data off any 
website and produce graphs and charts.

For example I might wish to create a graph of polulation by countries by 
pulling data off the CIA world fact website.

To do this there are basically three components

external website - writtend in html (hard) or xml (easy) 
html-microformat tags (easy).

Scaper programs - these have the responsability for parsing websites
        and extracting data from them.

Graphing client - gets data from different scrapers and creates a graph.

There might be a number of different scraper programs.

A standard xml scraper which could use an xslt script to pull elements 
from an xml document. This could also be used with microformats.

Custom scrapers, tuned for a particular html website. Copes with the 
indosyncraticies of its format.

Scrapers for other services.


How this relates to piw.

PIW server could be a typical exteran website.
I could write a scraper to parse the site and pull out data.
This would then allow a graphing client to say graph the heights of a 
selection of trees.

The same tech could also be used for other clients other than graphs.

Other elements, there is a need for a standard format for the output of 
scraper programs. This would need to be generic rather than subject 
specific, i.e. no asumptions about field names etc.

Anyway off to school

        Rich