Towards standard methods and metrics for evaluating IA

[Rashmi Sinha <rashmi AT rashmisinha.com>]

Hi all,
Below is a proposal for how project might work over the next few months. 
Note, this is simply an initial plan that I put together. Every aspect of 
it is up for discussion. Hopefully we will get to some of the issues on the 
IM meeting Tuesday.


Stage I) Post a call for metrics and methods. Publicize this call, on 
mailing lists etc. Give about a month for people to submit materials in 
response to this first call.
            -The call should include a description of the project, asking 
the community to contribute relevant materials that they want to share.
            -Should also describe what kind of materials people should 
submit, what format they should submit it in.
            -Might make sense to develop a short questionnaires asking 
each person to describe the purpose of the material, how they have used it, 
how to method compares to other similar methods.


Stage II) Review all contributed material. Upload it to the project site 
(might make sense for site to be a wiki).

	-Form a small committee to review all the material. Review to make sure 
each contribution is relevant to project and has been explained well enough 
so that others can used it. Ask author for more details if necessary 
(Review should not be for quality screening, i.e., we should let all 
material go through provided it is relevant and explained fully).
	-Upload all the reviewed material to the wiki. Associate some metadata 
with each to insure findability.
        -Regarding the project site: A wiki might be appropriate.
	-We also need to figure out the copyright issues. If authors release their 
materials directly into the public domain without any limits then that is 
fine. We might also want to look into Creative Commons copyright).


Stage III) Publicize the wiki widely. Ask people to use as needed and 
provide feedback.

	-Build some features into the wiki which allows us to track what material 
is popular (e.g., number of times downloaded?).
	-Encourage community to suggest improvements to contributed material. If 
they use the material, ask them to describe any changes or adaptations they 
might have made.


Stage IV) Few months later, review material. Update material based on 
feedback provided. If necessary, ask  original authors to review their own 
materials and update materials.
This can also be the point to launch more of a top down initiative.