Assignment M4 (Spring 2017)

Due: Sunday, March 26th, 2017, by 11:59PM UTC-12 (Anywhere on Earth). This assignment is based on lesson 3.6 (Evaluation), and focuses on laying out your evaluation plan.

Assignment Instructions

Answer the following prompt in a maximum of 1200 words, with a recommended length of 1000 words; if you supply more than 1200 words, the grader will stop reading at the 1200th word, and you will not receive credit for anything written after that. You are encouraged but not required to complement your responses with diagrams, drawings, pictures, etc.; these do not count against the word limit, though any captions, text in tables, etc. does.

Select three types of evaluation to perform: one qualitative, one empirical, and one predictive. For this assignment, each evaluation should target a different prototype from your previous assignment; evaluate one qualitatively, one empirically, and one predictively.

For the qualitative evaluation, select one of the methods for qualitative evaluation, like interviews, surveys, think-aloud protocols, focus groups, or post-event protocols. First, layout the evaluation plan, including who the participants will be, where the evaluation will take place, and whether/how it will be recorded (~150 words). Note that unless you have IRB approval for your evaluation, your evaluation should take place with friends, family, or classmates. Then, lay out the actual content of the evaluation, such as the interview questions that will be asked, the survey questions that will be administered, or the post-event protocols that will be followed (~100 words). For example, for a live Wizard of Oz prototype, you might script a session that demonstrates each aspect of the interface, then perform a post-event protocol to get the user’s thoughts. For an asynchronous paper prototype, you might send the prototype to participants with questions to answer in text. Finally, discuss how the evaluation will address the requirements in the data inventory and/or requirements definition phases (~100 words).

For the empirical evaluation, first define your control and experimental conditions. What are you testing, and what are you using as a point of comparison (~100 words)? Then, define your null and alternative hypotheses; remember, the null hypothesis is what you assume to be true unless you can find conclusive proof for your alternative hypothesis (~50 words). Then, describe the experimental method you will use. How will subjects be assigned to groups, what will they complete as part of their condition, and what data will they generate (~100 words)? Finally, identify what lurking variables might confound your data (~50 words) and describe how you will analyze your data (~50 words).

Note that given the early stage of your project, empirical evaluation may be tough to design. In the next assignment, you will select two of these three evaluations to actually conduct, so you may design your empirical evaluation in a way that would be unfeasible to carry out based either on the available resources or the status of the prototypes. The important task here is to experience planning the three types of evaluation.

Note that in planning either the qualitative or empirical evaluation, you may use the peer review exercise on this assignment as your data-gathering method by having your peer reviews complete the evaluation exercise and respond to a survey. If you choose to do this, the evaluation for your peer to complete should be the first thing they see upon opening the assignment, and should provide very explicit instructions for how to evaluate the interface, such as the interfaces to look at, the questions to answer, and where to supply the responses.

For the predictive evaluation, perform either a cognitive walkthrough of your prototype or construct one or more GOMS models of the user’s interaction with your prototype. If you conduct a cognitive walkthrough, layout in hierarchical detail the goals, perception, reasoning, and actions of the user at every stage of interaction with the prototype. A cognitive walkthrough may be presented in text (~250 words or equivalent detail). If you complete one or more GOMS models, lay out the goal, initial status, methods, operators, and selection rules in sufficient detail; you may use text to further explain the GOMS model(s). Note that in either of these cases, completing the walkthrough or the model is not the evaluation exercise; this merely provides the model that can be evaluated in the next assignment.Finally, select which two of these evaluation approaches you will complete for the next assignment (~50 words), and explain why. It is acceptable for the ‘why’ to be superficial reasons, e.g. “My prototype isn’t ready for empirical evaluation” or “I can’t recruit people to participate in my qualitative evaluation”.

Submission Instructions

Assignments should be submitted to the corresponding assignment on T-Square in accordance with the Assignment Submission Instructions. Most importantly, you should submit a single PDF for each assignment. This PDF will be ported over to Peer Feedback for peer review by your classmates. If your assignment involves things (like videos, working software prototypes, etc.) that cannot be provided in PDF, you should provide them separately (either through the class Resources folder or your own upload destination) and submit a PDF that describes how to access the assignment.

This is an individual assignment. Every student should submit an assignment individually.

Late work is not accepted without advanced agreement except in cases of medical or family emergencies. In the case of an emergency, please contact the Dean of Students.

Grading Information

This question is graded out of 20 possible points. Your grade and feedback will be returned to you via T-Square. An announcement will be made via Piazza when grades are returned.

Peer Review

After submission, your assignment will be ported to Peer Feedback for review by your classmates. Grading is not the primary function of this peer review process; the primary function is simply to give you the opportunity to read and comment on your classmates’ ideas, and receive additional feedback on your own. All grades will come from the graders alone.

You will typically be assigned three classmates to review. You receive 1.5 participation points for completing a peer review by the end of the day Thursday; 1.0 for completing a peer review by the end of the day Sunday; and 0.5 for completing it after Sunday but before the end of the semester. For more details, see the participation policy.