Project Summary

During my part-time internship as a User Researcher at ChaiOne, Inc. in fall 2020, I led a client project for Tenzing, a web-app for academic researchers from end-to end including research planning, execution and delivering insights.

The project was divided into two parts. For the present case study, I would like to focus on the evaluative research of this project, in which I evaluated usability of the web-app, delivered impactful insights, and worked with designer to deliver design solutions for the client.

Keywords

• Evaluative Research/Usability Testing / Heuristic Evaluation
• External Stakeholder Management
• Informed Design

My Role

Lead User Researcher

Time Frame

Sep. 2020 - Nov. 2020 (2.5 Months Part-Time)

Time dedicated to Evaluative Research: 1 month part-time

Stakeholders

External Stakeholders (Client)

1. Project Background

Internship @ ChaiOne, Inc.

In Fall 2020, I completed a part-time, remote internship as a User Researcher at ChaiOne, Inc. ChaiOne is a pioneer agency in Houston, TX that provides design thinking-led digital solutions to industrial businesses.

During my internship, I led a user research project for Tenzing, a newly launched free web-app that is designed for academic researchers.

The Product: Tenzing

What is Tenzing? Why does it matter?

Tenzing is a free Web-App that helps academic researchers track who did what in a research project. It allows researchers to generate reports about the contribution of each team member on a project, for insertion into their manuscripts, and for publishers to potentially incorporate into article metadata. It is developed by a team of professors and graduate students in Australia and Europe. It was launched just 1 month prior to my internship.

Tenzing is created under the broader vision of the Open Science movement in academia, which is a set of practices that increase the transparency and accessibility of scientific research.

As part of the Open Science effort, Tenzing promotes transparency in authors’ contributions and responsibilities by encouraging open discussion on individual roles and authorship assignments.

How to Use Tenzing

Tenzing Homepage

To generate a report on contributions for a research project, users have to go through the following steps:

1. Create a Google Sheet using the template given on the web-app
2. Fill in the Google Sheet accordingly to the given format called “CRedIT”
3. Download the Google sheet
4. Upload the spreadsheet to Tenzing and select the desired output format to generate a report

2. Research Objectives

Stakeholder Goals

After meeting with the stakeholders (Tenzing team), I discovered that their primary goals at the time were two-fold:

• Ensure Tenzing is user friendly and easy to use
• Encourage academic researchers to use Tenzing and start using it early in the research process

Research Goals

I translated the two stakeholder goals into concrete research goals in order to guide my research process.

Goal 1:

Evaluate the usability of the current version web-app and provide improvement recommendations if need be

Goal 2:

Discover drivers for user adoption and targets for user base expansion

3. Research Process

For this case study, I would like to focus on how I successfully delivered impactful results for the first goal which is,

Goal 1: Evaluate & Improve Tenzing’s usability

Research Questions

Recap: Tenzing’s main function is for researchers on the same research project to log each of their contributions and create outputs that are suitable for journal submission or other desired usage.

The stakeholders want to encourage researchers to use Tenzing as part of the research management. Therefore, Tenzing should have low learnability and be free of blockers that discourage users from dropping out.

The primary research question for this project is

Can users successfully log in individual contributions and create desired outputs by following the instructions provided on the web-app, without additional help?

Methods Selection

I decided to do a combination of Heuristic Evaluation and Moderated Usability Testing to assess the usability of the website. The reasons that I selected the combination of methods are:

1. Conducting Heuristic Evaluation first allows me to get quick first-round feedback to clients

Since it takes time to recruit participants and conduct a usability test, a heuristic evaluation allows me to deliver quick results to external stakeholders, so that I could keep the stakeholders engaged in the process and set the right expectations. This is especially helpful when the stakeholders are external and not familiar with what a usability study entails.

2. Heuristic Evaluation helps me select the right tasks for Usability Testing

Conducting heuristic evaluation prior to usability testing helps me understand the web-app in great detail so that I can create appropriate tasks that will exercise the entire interface, especially those areas where there may be vulnerabilities in usability.

3. Conducting a Heuristic Evaluation together with Usability Testing provides a more holistic evaluation of the web-app

Since I might not be able to cover all the details of the web-app in usability testing time constraints. A heuristic evaluation can fill in the gaps on portions of the web-app that are not covered by the usability testing tasks.

Heuristic Evaluation

Execution

Severity Ratings & Issue Category

• Used Nielson 10 Usability Heuristics as the criteria
• 3 researchers individually evaluated the interface
• For every issue identified, each individual evaluator assigned a severity rating and an issue category

• Issues were also categorized based on the section of the web-app that it belongs to so that

  • I can identify solutions that could holistically solve issues in a web-app section (e.g. homepage)
  • have a clear narrative when presenting usability issues to external stakeholders

Analysis

• I led a research workshop where all 3 evaluators discussed individual findings and identified a finalized list of usability issues that we all agreed on
• All evaluators reached consensus on severity ratings for issues identified so that we could prioritize issues to present to stakeholders in the research report (one of the deliverables)
• Design solutions were generated for every issue found

Sample pages from finalized list of issues found. All issues are categorized by web section that they belong to, accompanied by severity ratings and type of issue.

Results

• Uncovered 22 issues in total (2 critical issues, 8 major issues, and 12 minor issues)
• All critical issues and most of the major issues were related to ambiguous workflows, unclear instructions and language use

Sample pages from Heuristic Evaluation Report

Usability Testing

Recruitment

Tenzing’s target users are academic researchers including faculty researchers, graduate students and postdoctoral researchers. Recruitment messages were sent out through university network, 5 graduate students were recruited for the study in less than 1 week.

Study Design

Recap: The primary research question was Can users understand the instructions on the web-app and follow the instructions to log in research project contributions and generate desired report?

To generate a desired report through Tenzing, a user has to complete four individual tasks, which are:

• Create a blank Google Sheet from Tenzing’s template
• Fill in the spreadsheet and Save
• Upload the spreadsheet
• Generate and download the desired output report

Instructions were already provided on the web-app as guidance for the users.

Study Instructions Provided to the Participants

Instead of presenting the participants with multiple tasks with each step as an individual task, I decided to provide them ONE overarching task on their own that encompassed all the steps.

The study design minimized hints given to the participants through task instructions and mimicked the real-world scenario when users are using the website on their own. It also allowed me to assess whether the instructions reflect user’s mental flow.

Moderated testing sessions were conducted remotely over Zoom. Participants were asked to think-aloud while completing the tasks.

Participants were given verbal and written instructions including:

• an introductory scenario to a multi-step task
• a PDF that contains proxy information that they would need in order to complete the task (see image on the right)

Metrics & Measures

Self-Reported Metrics

After the task was finished, participants were asked to rate:

• Single Ease of Use (SEQ) for each of the four steps in order to assess the difficulty of each specific step, help identify which steps are the most challenging ones
• System Usability Scale (SUS) for Tenzing in order to measure the subjective assessment of usability. Serve as a benchmark for future iterations.

Performance Metrics

The followings were collected to quantitatively assess the effectiveness and efficiency of Tenzing:

• Completion rate

  • Although participants were asked to complete all four tasks at once, completion rate was collection on each task
  • Each task was further broken down into sub-tasks, completion rate of each sub-task was collected in order to determine what specifically causes problems in a task
• Completion time

Qualitative Measures

These were collected to identify user’s mental flow and pain points:

• Path taken to complete each step
• Errors & Pauses

Data Collection & Analysis

• I created a data collection template to collect all metrics and notes for each participant. After all sessions were completed, I was able to very efficiently consolidate all data collections forms into one spreadsheet leveraging the pre-made template format
• Calculated quantifiable metrics
• Coded qualitative measures (i.e. errors made and paths taken) and identified patterns
• All issues identified were consolidated and organized by web-section and were assigned severity ratings the same way as described in the Heuristic Evaluation part

Sample Data Collection Template used for Usability Testing

Usability Testing Results

• Uncovered a total of 12 issues (1 critical issue, 3 major issues, and 8 minor issues)
• System Usability Scale (SUS) score 79 (B+)
• Average Ease of Use: 6.5/7

• Completion Rate

  • 3/4 tasks were completed 100%
  • 1/4 task was 80%

Key Insights from Two studies

I consolidated all results and compared findings from Heuristic Evaluation and Usability Testing. Several overlapping patterns of usability issues were identified:

1. Homepage Design did not accurately reflect the designated workflow for the web-app and caused user errors

The homepage layout led users to miss out on some of the important instructions that users need in order to successfully use the web-app, which could potentially lead to new users dropping out.

2. Complicated workflow with unclear & inadequate instructions

Users had to manage multiple interfaces (including Tenzing web-app, Google Sheet, and Excel) in order to complete the task. Users had difficulty understanding some of the written instructions and had to switch back and forth to read the instructions and navigate the spreadsheet.

3. Poor Accessibility

Half of the web-app used similar colors for the text and the background (e.g. light green text on green background). The color patterns made it difficult to read in poor lighting conditions and for people with color deficiency.

4. Deliverables & Impact

Deliverable 1 - Report & Design Recommendations

Two individual research reports were delivered to external stakeholders (the Tenzing Team) through deck presentations. I walked through the major findings, the impact of the findings, and the design recommendations of issues found.

Sample pages from Usability Testing Report

Deliverable 2 - Homepage Re-Design

I identified that the major usability issues could be solved through a homepage re-design that reflects the correct workflow and clear instructions. I collaborated with the designer on the team to create a new homepage for the web-app.

To help inform the re-design, I integrated all findings related to homepage section of Tenzing and created a list of requirements for new homepage.

Note: results from the generative research that identified drivers for user adoption (not included in the present case study) were also used to inform design

Homepage Redesign