Towards standard methods and metrics for evaluating IA

[Rashmi Sinha <rashmi AT rashmisinha.com>]

Hi everyone,
Welcome to the AIfIA-metrics list. Thanks for joining in.

The goals of this AIfIA project is to propose methods and metrics for 
information-rich websites and applications.

Below is a short description of my reasons for proposing this project. At 
this stage, it would be great to hear from each list member what their 
thoughts are related to the below goals. What other issues / goals related 
to evaluation and IA do they consider important? What are the challenges 
you are facing in your own work? Lets spend some time discussing issues 
before getting down to defining the scope of the project, the approach etc.

*************************
Currently, there are no standard methods and materials available for 
evaluation of information architecture. Usability questionnaires such as 
<http://www.ucc.ie/hfrg/questionnaires/wammi/index.html>WAMMI (Website 
Analysis and Measurement Inventory) or <http://lap.umd.edu/QUIS/>QUIS 
(Questionnaire for User Interaction Satisfaction) are not suited to 
information-rich domains. Moreover, both of them can be used only after 
payment of a substantial fee.

The question arises: Why do we need free and open standards? Because, many 
of the evaluations conducted by IA's are designed to answer the same core 
set of questions: can user complete task/find information? are users 
satisfied with site? number and type of errors, usability problems etc. 
However, it remains difficult to compare results of different evaluations 
because of the lack of standardization of evaluation methods. And such 
methods need to be free if they are to propagate and become standards. As 
such AIfIA is the right kind of organization to support such an endeavor.

The goal in creating free and open standards is to allow IA's to benefit 
from the experience of others who have conducted similar exercises before. 
We will encourage IA's to both share their own evaluation methods and 
materials, and also to use the methods shared by others. This will 
encourage reuse, reducing the duplication of effort.

Creation of Benchmarks: The hope is that over a period of time, this 
project will lead to the creation of benchmarks for task success and 
satisfaction with various types of information-rich websites. And these 
benchmarks will allow individual IA's to track the success of their 
designs. Also, the creation of benchmarks that are widely followed can 
allow us to deal with some thorny issues such as the link between good IA 
and ROI.

This is what I have been thinking. Thoughts? Reactions?


-rashmi