Towards standard methods and metrics for evaluating IA

["Livia Labate" <liv AT livlab.com>]

Hi Rashmi and all,

When I saw the invitation to participate in this group (on the AIfIA members
list?), my first thought was "hummm I hope they plan on listing lots of the
metrics used out there and gather resources for each, so we can make
comparisons and suggest usage" So that idea kind of stuck with me.

I personally would find great value in a 'resource centre' or something to
learn about different metrics and their usage (benchmarking). And by metrics
I am including guidelines, patterns, rules-of-thumb, do's, not just
evaluation metrics (Peter Boersma mentioned this in a previous message, I
agree with this wider scope).

As for the approach, I feel that we won't be able to propose anything
(standard methods and metrics) until there is a real community
infrastructure. It's hard enough to gather all the information, so thinking
about propositions might be skipping too many steps. Then again, if that's
is the *ultimate* goal of this group, excellent, we just need to remember it
is a long term commitment because it won't be a short journey.

One way we could do this is segmenting the scope and research slowly.
Example: For the next three months we can look into gathering resources
about types of project success measures (just a random example), then we can
organize a discussion on the topic and finally compile a paper with the
findings (?). Then move on to the next topic...

   OR

trying to define what are the needs of IAs and IA projects and THEN explore
the available metrics to verify if what is out there suits those needs and
if not make propositions based on our findings.

Just a few thoughts. Let's hear what everyone thinks.

Following Rashmi's intro: I am Livia Labate, struggling, I mean, working as
an IA in São Paulo, Brasil. Opposite to Rashmi I am an academician wanna be.
My binary self can be found here: www.livlab.com :)

- Liv