Housing DashBoard Project:

 

Describe A Specific Situation:

 

The different classes entering DePaul University housing facilities needed access to various information, which was easy to access, contain a wide variety of information in a quick digestible format, and was focused to their particular class needs.

• Students expressed a need for a 24-hour personal assistant for university related information.
• Services, events, and class information that would reduce stress levels and enable students to excel was difficult to locate and public awareness of such material was not common.
• Information that was targeted for a particular class and placed in an online area only for this class was needed.
• Mobile and desktop software formats are required to access information.

Tasks that needed to be done:

• After student interviews were performed, affinity-diagramming sessions reveled common patterns among graduate through freshman-senior level classes. These patterns are going to be used to form boundaries and content for the housing dashboard.
• A mobile dashboard content and interation models will be constructed.
• A desktop dashboard content and interaction models will be constructed.
• This example of how the university is "student focused" will be incorporated into marketing material.
• Axure prototypes wil be designed and developed and 1 to 2 rounds of usability testing will be conducted.
• Site maps to conceptulize and analyze content layout and structure will be formed.

My Contribution:

• Project lead which involved deliverable review and approval, project activity analysis formation, and requirements gathering.
• Chief architect in the formation of the housing dashboard.
• Formulated sitemaps for content and layout analysis.
• Developed task scenarios used in usability testing.
• Assisted in the creation of a usability report.
• Performed affinity diagram sessions
• Contributed to the marketing campaign.

The Results:

• The post survery usability testing results showed 92 percent of students would use such an application on a regular basis.
• 82 percent reported that this would help them reduce aniexty and stress levels.
• Common comments during usaiblity testing " It's like having my own personal library", "My own virtual calendar that is automically updated."

 

Housing Dashboard Personas

 

 

 

 

 



 

SiteMap

 

 

 

 

 

 

User Flows

 

 

 

 

 

 

WireFrames

 

 

 

 

 

 

 

 

 

Usability Testing

 

 

Usability Plan and Script

 

Dashboard Housing Project

 

 

 

Prepared by:            James Nutter Usability Team member

Author:                    James Nutter

Date:                       September 18 , 2006

Revision                   2

 

 Introduction

 

This document describes the usability evaluation plan for Dashboard Housing Project. The purpose of a usability evaluation is to predict the expected performance of the actual customer using the current product and materials, as well as detect any serious problems before the release of the product. This plan includes the following sections:

• Purpose of the usability evaluation
• Target audience
• Design of the usability evaluation
• Data collection methodology
• Deliverables
• Schedule

What is Project Name?

 

The different classes entering DePaul University housing facilities needed access to information, which was easy to access, contain a wide variety of information in a quick digestible format, and was focused to their particular class needs.

• Students expressed a need for a 24-hour personal assistant for university related information.
• Services, events, and class information that would reduce stress levels, enable students to excel was difficult to locate, and public awareness of such material was not common.
• Information that was targeted for a particular class and placed in an online area only for this class was needed.
• Mobile and desktop software formats are required to access information.

Design Usability Goals

 

The evaluation will focus on determining if the needs of the user are met in an easy to understand, useful, and productive manner.  Specific measurable goals for the usability evaluation are outlined in the Usability Evaluation Goals section of this document.

Purpose of the Usability Evaluation

 

The usability evaluation of the housing application will evaluate navigation, information architecture, and usability issues. The usability metrics we will be using are:

• Time on Task
• Self Reported Metrics
• Efficiency
• Card Sorting data

Concerns

 

This section outlines the specific concerns that the stakeholders, development team, and usability team may have. Some specific questions addressed by the usability evaluation include: 

1. Will students be able to access the dashboard application in a timely manner?
2. Can students customize information on the dashboard?
3. Smart defaults enabled if students’ have to fill out a form?
4. Is the dashboard application narrow in its focused of content?
5. Is some or all dashboard information available on my cell phone?
6. How much information to have to enter manually in order to use the dashboard?
7. Can other people see my dashboard?
8. What is the marketing angle for this application? Measure return on investment?
9. How will users feel about using the on-line help? Is context-sensitive help a requirement for our users?
10. Can I add my own information categories?

 

 

Usability Evaluation Goals

 

Specific usability goals were determined from the above concerns. These goals allow for the creation of evaluation scenarios and tasks that will let us know if our concerns are valid and what measures can help us determine if in fact the participants are having trouble completing the tasks.

This evaluation will be based on the following usability goals:

• Students’ will be able to complete the customization of their dashboard within 10 to 15 min.
• Students’ will be able to locate directions to the lincoln park cafeteria within 5 min
• Students’ will be able to locate a particular class and location within 1 min.
• Students will be able to customize their dashboard with minimal cognitive and physical effort.
• Questionnaires handed to students’ will measure ease of use and expectation qualities.

Target Audience

 

First time housing students

• Freshman-senior level

• Graduate level

 

Background

 

Demographic

• Male-Female 18-27
• Midwest region
• Middle to upper class

Occupation experience

• Full or part time student
• 0-4 years experience
• Entry to junior level position

Education

• Completing bachelors or beginning masters program

Computer experience

• Intermediate-advanced skills, min 2-year exposure.

 

Primary tasks

• To view class information and updates in real time
• To view locations of housing and main campus amenities
• To view university social activities

Secondary tasks

• To view guidance programs
• To view cafeteria menus
• To view campus safety information

 

Design of the Usability Evaluation

 

A single usability evaluation will be run in at least 10 individual participant sessions. Each individual session will consist of a set of tasks and an interview/questionnaire for the participants to complete.

The individual evaluations will take place in the following order:

• A performance evaluation in which each participant is asked to perform a series of real-life tasks
• A questionnaire and an interview after each performance evaluation to gather additional insights from the participants about the housing dashboard.

 

The Evaluation Process

 

The usability evaluation process is as follows:

 

Participant greeting and background questionnaire

 

Each participant will be personally greeted by the evaluation monitor and made to feel comfortable and relaxed.  The participants will be given name tags and asked to fill out a short background questionnaire. The issue of confidentiality will be explained and the participants will be asked to sign informed consent statements.

Orientation

 

The participants will receive a short, verbal scripted introduction and orientation to the evaluation. This material will explain the purpose and objective of the evaluation, the need for product anonymity until after the evaluation, and additional information about what is expected of them.  They will be assured that the product is the center of the evaluation and not themselves, and that they should perform in whatever manner is typical and comfortable for them.  The participants will be informed that they are being observed and a computer screen capture will take place.  They are asked to sign the appropriate release forms if not already completed.

Performance evaluation

 

The performance evaluation consists of a series of tasks that are evaluated separately and sequentially.  The individual participants screen will be recorded on video and observed by the usability specialists.  The scenario is as follows:

• After the orientation, the participants will be asked to sit down at the computer.  The evaluation administrator will give the participants the task scenario booklet and instruct them on the use of the help desk.
• After the participants begin working through the evaluation scenario, they will be encouraged to work without guidance except for the provided material and the product itself.  The evaluation administrator may ask the participant to verbalize his or her thoughts if the participant becomes stuck or hopelessly confused.  These occurrences will be noted by the evaluation administrator, and will help to pinpoint the cause of the problem.

Participant debriefing

 

After all tasks are complete or the time expires, each participant will be debriefed by the evaluation administrator.  An audio recording of the debriefing and the steps will include the following:

• Completion of a brief post evaluation questionnaire in which the participants share their opinions on the product’s usability, appearance of application screens, and general impressions of the product
• Participant’s overall comments about his or her experience

The debriefing session serves several functions.  It allows the participants to say whatever they like, which is important if tasks are frustrating.  It provides important information about each participant’s rationale for performing specific actions, and it allows the collection of subjective preference data about the application and its supporting documentation.

After the debriefing session, the participants will be thanked for their efforts, and released. They will be given a small token of appreciation as they leave.

Logistics

 

We will simulate a typical housing environment during the usability evaluation. Because the application is a networked one, the usability evaluation must take place in a location where a network connection is available. The housing location will be large enough to comfortably accommodate a desk for the participant to sit at while completing the evaluation. 

There will be an additional area for placement of the portable usability lab equipment.  This area will consist of two average size desks and allocation for power usage. 

Materials Design

 

The following materials will be designed and developed for use in the Housing Dashboard usability evaluation:

• Usability test plan and script
• Task scenario
• Data recording documents

Data Collection Methodology

 

• Data will be collected through the use of a “thinking aloud” protocol and retrospective review.

• The evaluator will indicate on his notes via a coding system when participants have difficulty and when an unusual behavior occurs.

• Subjective opinions of the usability and aesthetics of the product will be noted via coding system.

• The time it takes a student to perform a task.

• The amount of actions or steps that a student took in performing each task.

 

Deliverables

 

At the completion of the usability evaluation, a formal analysis will be performed.  A final evaluation report and a highlight tape will be provided, which will detail the significant problems and observations detected during the usability evaluation, and recommendations to address the findings will be delivered to the customer for the total resource requirements in days.

 

Schedule

 

The usability evaluation will be conducted the week of September 18, 2006. 

 

Open web page to campus connect login page

 

 

 

Hi, student’s name. My name is your name, and I’m going to be walking you through this session today. Before we begin, I have some information for you, and I’m going to read it to make sure that I cover everything. You probably already have a good idea of why we asked you here, but let me go over it again briefly. We’re asking people to try using a housing application that we’re working on so we can see whether it works as intended. The session should take about an hour.

 

The first thing I want to make clear right away is that we’re testing the site, not you. You can’t do anything wrong here. In fact, this is probably the one place today where you don’t have to worry about making mistakes.

 

As you use the site, I’m going to ask you as much as possible to try to think out loud: to say what you’re looking at, what you’re trying to do, and what you’re thinking. This will be a big help to us.

Also, please don’t worry that you’re going to hurt our feelings. We’re doing this to improve your experience here at DePaul, so we need to hear your honest reactions.

 

If you have any questions as we go along, just ask them. I may not be able to answer them right away, since we’re interested in how people do when they don’t have someone sitting next to them to help. But if you still have any questions when we’re done I’ll try to try to answer them then. And if you need to take a break at any point, just let me know.

 

You may have noticed the microphone. With your permission, we’re going to record what happens on the screen and our conversation. The recording will only be used to help us figure out how to improve the site, and it won’t be seen by anyone except the people working on this project. Moreover, it helps me, because I don’t have to take as many notes.

 

In addition, there are a few people from the Usability team observing this session in the room.

 

If you would, I’m going to ask you to sign a simple permission form for us. It just says that we have your permission to record your computer screen and audio of your comments, and that the recording will only be seen by the people working on the project.

 

Give them the informed consent form to sign.

 

 

Do you have any questions so far?

OK. Before we look at the application, I’d like to ask you just a few quick questions.

 

Now, roughly how many hours a week altogether—just a ballpark estimate— would you say you spend using the Internet, including Web browsing and email, at work and at home?

 

Do your own a cell phone? Do you have access to email on your phone?

 

We are done with the questions, now we can start looking at the housing application.

 

Click on the saved bookmark

 

 

I’m going to ask you to try doing some specific tasks. I’m going to read each one out loud and give you a printed copy. Again, as much as possible, it will help us if you can try to think out loud as you go along. We’ll learn a lot more about how well the site works that way.

 

 

Hand the student the first scenario, and read it aloud.  Start the clock as soon as they finished reading the task  Allow the student to proceed until you don’t feel like it’s producing any value or the user becomes very frustrated

Allow the student to proceed until you don’t feel like it’s producing any value or the user becomes very frustrated  Stop the clock when the student stops interacting with the product  Repeat for each task

 

Task Scenarios:

 

Task # 1:

You have been talking to your friends and you have heard about the dashboard housing application and want to set it up, where would you go to do that?

 

Task # 2:

You are a computer science major who want to be involved with social activities around school and you want to add this items to your dashboard, how would you do that?

 

Task # 3

As you are eating lunch with your friends, one of them tells you they are taking a psychology course on Tuesdays at 10:00am, you want to add your class schedule to your dash board, how would you do that?

 

Task # 4

 

You have heard from your friends that the gym is very nice and has indoor swimming pool, you need to find where it is located and the hours of operation.

 

Task #5

You have just arrived home and what to know when your next class starts, where do you go for this information?

 

 

 

 

 

Usability Report Outline

 

 

Usability Report Overview

 Purpose: The Housing Dashboard is an application designed and developed in-house by the Information Services Department. Providing students the tools they need to be successful at DePaul is one of the cornerstones that the university is continuing to evolve. Having information available that will enable the students’ to be more efficient in their daily school activities will allow them to enjoy their university experience in more depth. 

 

Goals

• Having a reference point that you can access in which you have customized housing, university and class content.
• Allowing the students’ the option and ability to filter content according to your present and future requirements.
• Possessing that sense that you have a 24-hour personal assistant for university related information.

 

Testing results evaluating criteria

 

Confidence intervals

The small sample size less than < 10 was a point of concern for the reliability when using usability metrics. The usability team alleviated these concerns by including confidence intervals for calculating the metrics in addition to using a significantly different class of user. 

 

The predominant metrics used in the dashboard housing usability testing sessions are:

• Task success
• Self reported metrics

 

Task success rates are based on relatively small sample sizes, this metric may not be as reliable as we would like it to be. To compensate for this the usability team used a modified version of the Wald Method called the Adjusted Wald Method.

 

To assess the students’ perceptions of the application we used the ease of use and expectation measure for usability metrics. The ease of use measured how easy or difficult each task was using a semantic differential scales. The expectation measured how easy or difficult the task was in comparison to how easy or difficult the student thought it was going to be. This metrics will also assist us in contributing to the direction and content of the marketing campaign for this application.

 

Explanation of student errors

 

A data logging criteria worksheet was setup using Microsoft Excel and contained the following format:

 

Time	Class	Description
12:34	C	“What do I do next?”

 

 

 

Class Codes

X = usability problem

D= Duplicate usability problem

V= Video highlight- an “Ah-ha!” moment

C= Comment (general comment by participant

P= Positive opinion expressed by participant

N= Negative opinion expressed by participant

B= Bug

F= Facial reaction (e.g; surprise)

H= Help or documentation accessed

A= Assist from moderator

G= Gives up or wrongly thinks they are finished

i= Design idea (Design insight by logger)

M= Misc (general observation by logger)

 

 A list of correct and incorrect actions was complied and divided into single error opportunities and multiple error opportunities. For single opportunities for error the numbers would be 0= no error or 1= one error. For multiple error opportunities the numbers will vary between 0 and the maximum number of error opportunities.

 

Single opportunity analysis

 

A frequency of errors by task was run and would show the number of errors on each task, and then the total number of participants for each task divided by the number of errors, will tell you the percentage of participants who made an error for each task.

 

For multiple error opportunities

 

Divide the total number of errors for the task by the total number of error opportunities.

 

Screen capture movie highlights

 

Camtasia screen shots movies highlighted predominant error rate areas. These screen shot movies visually illustrate systematically the process students’ went through before they encountered an error. The recommendations included in the usability report use the highlight videos to correlate the effect on business, project, and student purpose and goals.