pcplantdb

[Chad Knepp <pyg@galatea.org>]

John Schinnerer writes:
 > Aloha,
 > 
 > First off, I just talked to Jim at some length and he is still keen
 > on this project and says he has a better grasp of what we are about
 > after talking with some of us in person.
 > 
 > Chad, re your comment about rather not talk with him in person, he
 > would like to chat with you if possible and I found him to be
 > knowledgeable, personable and supportive.  He asks good questions,
 > makes thoughtful comments and listens well too.  I told him you had
 > put yourself forward as the one with the most hardcore development
 > chops and I don't recall anyone protesting that too loudly.
 > You would be able to speak to some more nitty-gritty implementation
 > details that I can't - conceptually I'm feeling good about this
 > project but to actually code the concepts I and the rest of us are
 > putting forth I'd have a python learning curve about like El
 > Capitan to climb.  Not that I'm adverse to starting that climb,
 > just more done for less time and/or money if you're already
 > summiting the climb...
 > 
 > If you really are not willing to talk with him in person at all, I
 > want to know why not, mainly because I want to know more about you
 > if we are to be working on this together.

This is mostly a personal oddity of mine and part of my geek heritage.
I choose not to use phones very much.  I feel uncomfortable talking to
people I can't see because the emotional subtext of the conversation
is only available in intonation which is very confusing for me.  I
find myself spending a lot of time figuring out what I think people
mean by what they say and how they are saying it.  Writing is easier
because there is very little implicit emotional content and you have a
lot more time to interpret what is there.  In person I can quite
easily read body language and don't need to spend a lot of mental
energy on interpretation... but I'm basically a pretty shy person
socially. 

Anyway, even though I have broadband, I don't actually have a *home*
phone.  I can also push myself and if you think it will make a
significant difference I can find a phone and call him.  My fear is
that I will not be able to think clearly enough quickly enough to make
a positive impression.  If you really want me to though, just say the
word.

 > His main concern is how will we in practical terms dive in and
 > proceed if he gets us some money.  The likely amount he mentioned
 > coming through in July was in the $12k - $20k range, if it does.
 > What could we do with that, he wants to know.  He likes what we are
 > about as he's heard it thus far.

This seems like it would have a pretty significant impact on the
project.  I think everyone should have a weigh in about how this new
budget would/could affect their participation in the project.

My original thinking is that even though it was billed as a half-time
job it would actually make more sense to work full time until much of
the coding bit was done and then refine/maintain as authoring began.
After the design considerations are finalized it seems like not much
else can happen until we have some alpha level product to start
playing with.  In the initial proposal there are two people Paula
Westmoreland and Robin Williamson listed as subject matter experts.
It's my opinion that maybe authoring should be postponed at this phase
of the project until more financing becomes available.  It is also my
own belief that we will get the best/most content by creating a tool
that solicits it from the general public and not by trying to do it in
house.  I think it makes the most sense to focus this initial funding
on design and development.

 > I told him my take was that phase one would be a simple, practical,
 > PC-centric plant database architected in such a way as to be able
 > to grow from there.  Focus would be on clean and robust back end,
 > clean and open API and a simple front end to start, with the clean
 > and open API making all manner of front ends possible eventually/as
 > resources emerge.  It would serve initially as one answer to the
 > inevitable PC-course participants' question "...so where can I find
 > permaculture-relevant information on plant attributes and functions
 > so I can use them appropriately in designing??"
 > 
 > > Yeah, I can see how an object DB would tie in neatly with an object
 > > method of peer to peer data exchange.
 > 
 > I actually think it's a better model whether peer-to-peer or not.
 > Myself, I pitched this to Jim as strictly my take/my preference.
 > He seemed to like the object based approach.  Again, if this is
 > built carefully the guild aspect will be both implicit and an
 > emergent property.  Some added logic to extract and/or extrapolate
 > guild information from plant objects and voila!  Stephanie's dream
 > realized!  ;-)

I think I'm starting to visualize where you are going here.  I need to
study up on object DBs, but I think I'm on board.  I also don't think
that what you are talking about is going to be a big climb for you
based on what you are able to conceptualize.  A pure object approach
has an appealing clean feel to it.  I can imagine being able to do
really nifty things like being able to attach a discussion object to
any object.

 > > I'm not opposed to the idea and I'm totaly into the OO
 > > methodology, but I really don't know anything about object DBs.
 > > Does zope have an object DB backend?
 > 
 > Yep, it's the ZODB (Z Object Database) and it's been stable and
 > mature for quite a while now.  It has its own development track and
 > releases and can be (and has been) used directly (outside of Zope
 > as a whole that is) to provide persistence for python objects in
 > all sorts of contexts.
 > Available in conjunction with the ZODB is ZEO (Zope Enterprise
 > Objects), basically a scalable DB product that takes the default
 > all-in-one ZODB and makes it scalable across multiple servers.
 > Pretty painlessly and transparently too from what I understand.  My
 > sysadmin looked into it and was impressed and he tends to be pretty
 > skeptical and hard to impress.  We haven't implemented ZEO for our
 > hosting yet because we simply don't need it, but it's there when
 > the need arises.
 > The other nifty already-built piece available is ZCatalogs, which
 > provide cataloging of objects in a ZODB.  You can define meta-data
 > and indexes on objects.  Objects can be self-cataloging, so that
 > when you instantiate ("add") one it catalogs itself right then and
 > there and will immediately appear in any relevant queries.

I suppose these are relational indexs of the objects.  My initial
performance concern was based on how you would look through data
elements of every object quickly.

 > A ZODB can be indexed and queried using ZCatalogs and IME it is
 > much clearer and more obvious and less work than writing
 > equivalently complex SQL queries on an RDBMS.
 > From examples I have read it is "fast enough" for some pretty darn
 > large object DBs.  Recent optimizations to the catalog code have
 > cleaned up a lot of overhead and made it much faster than earlier
 > versions.

It would be really nice to have the "full text search" capabilities of
RDBMS.  How do you index "narrative" data in an object DB?  Maybe
ZODB/ZCatalog has this or you could catalog the text blobs and use an
external search mechanism.  Maybe I just have more reading to do.

 > > What should I be reading/looking at?  I also have some
 > > concerns that may just be questions if I look into it more (like
 > > performance).
 > 
 > Look up the ZODB, ZEO if you want for scalable DB options.  These
 > are rolled into Zope as a whole, but if you google them or do a
 > search on the zope.org site you will find discrete info on them as
 > separate entities.  They are afaik in python with I think some C
 > and/or C++ bits for performance reasons.
 > 
 > There are probably some other ODBs out there - this is just the one
 > I use and know a little something about.

Another thing I've been thinking about which only really differs in
implementation is that we could use an RDBMS as the *catalog* and
create our objects as stored procedures within the RDBMS.  MySQL just
got stored procedures in 4.x something and I think postgresql has had
them for a while.

 > > I personally think XML is just the latest in a long list of
 > > useful but tremendously over-hyped technologies.
 > 
 > Yah...
 > 
 > >  I don't think there is even that much to it as a technology.
 > 
 > I thought there was way more to it than there is due to all the
 > hype...I was surprised to find out how relatively simple it is.
 > Which is good in the sense that it's more accessible to more
 > people.  Anyhow, the hype generates active use of it and it's not
 > IMO a 'bad' technology, so more potential for growing the community
 > of project participants who might (for example) build a front end
 > of some sort using the XML in-between because it is
 > familiar/comfortable for them.
 > 
 > If someone is comfy with/into PHP and XML and we provide a clean
 > and open API from the back end then they will be more willing to do
 > a front end to suit them (or someone else).  I expect (but don't
 > know) that it will also be readily usable by any sort of
 > graph-based front end.
 > 
 > > I would be disappointed if we didn't get funded because he
 > > concluded that we didn't have it together technically enough to
 > > make it happen.
 > 
 > His concerns are mostly (but not entirely) less technical and more
 > organizational was my impression.  The technical is ultimately the
 > easy part in any project; it's the human relatings (cooperation,
 > organization, consensus or not, etc.) that generate most of the
 > difficulties.  He knows that.  How will we organize ourselves, draw
 > more project support (i.e. grow the active contributor community),
 > get started in a practical sense?
 >
 > Also, before I forget (we covered a LOT of ground on the phone), he
 > is curious also about how we will manage input to the DB and had
 > some good suggestions based on his experience with social networks
 > and so on.  More than I can cover thoroughly now as I have to get
 > to bed for a morning flight and will be cyber-incommunicado until
 > Tuesday next week.
 > 
 > In short we considered that different moderation approaches might
 > be appropriate for different content types, and Jim pointed to the
 > value of distributing/socializing the work of moderation.  We
 > considered the difference between brief 'factual' data (climate
 > zone, shade tolerance, soil preference, water needs, that sort of
 > stuff) and less brief 'narrative' data (which is much richer and
 > also harder to corroborate or 'verify' in a simple way).
 > 
 > The former could be for example vetted by 'qualified' community
 > members in a sort of distributed review process before being
 > published publically.  Who is 'qualified' is TBD and I won't go any
 > deeper into that at the moment!

Mostly I like and qualification could be determined by achieving
certain level of communty-reputation as per your next paragraph.
Getting moderation right is going to take some effort and none of it
is totally clear to me right yet.

 > For the latter, we considered the possibility of a
 > community-reputation system (like yes, sigh, eBay as a major
 > example) where people can provide feedback on each other's
 > narrative info.  People who continually get called on submitting
 > questionable or bogus information will be visible as such by all
 > users.  People who consistently submit useful and pertinent
 > information will likewise be visible as such by all users, and so
 > on.  Jim pointed out also that this allows for the positive
 > possibility of people questioning and expanding the simple
 > 'factual' data based on their own local experiences.
 > Quick example - plant X is listed as not viable beyond zone 6 and
 > this is verified by multiple independent sources (other DBs,
 > experienced horticulturalists, USDA reference, etc.).  Meanwhile
 > someone somewhere has been growing plant X successfully in zone 7
 > and can describe how they've done so in a narrative submission -
 > perhaps through the use of some clever design that could be useful
 > to others as well.

Oooohhh!  I like it!  This is what I was trying to achieve with the
peer to peer model but this would do it much better.

 > All for now - much food for thought - what next?

-- 
Chad Knepp
python -c 'import base64;print base64.decodestring("cHlnQGdhbGF0ZWEub3Jn")'