Using Design Thinking to Create Human-Centered Assessments

What’s your reaction when you hear the word ‘test’? Do you relish the challenge, or are you filled with dread? There are many reasons that someone might have bad feelings about assessment such as test anxiety, lack of preparation, or feeling crunched for time. Whatever the reason, many people feel that traditional assessments don’t accurately reflect their skills and abilities.  

--Real quotes about assessment from preservice teachers.  

Who’s Involved in Assessment Development? 

When developing assessments, psychometricians follow a principled approach. They start by defining the construct (a.k.a. the skill or topic) to be measured, then identify what behaviors might be considered evidence of that construct, create tasks, and choose a scoring approach. They might consult with subject-matter experts to help them understand the construct. They will also consult with the people who will be using the test results, who might be teachers, administrators, or policy makers.  

These people are all focused on the construct: defining the construct, creating tasks and scores that represent the construct, and making decisions based on the construct. They rarely consider the human emotions, experiences, and situations that surround the testing process. To the extent that they consider the test-takers' experience, they refer to it as construct-irrelevant variance: an unfortunate obstacle to measuring the construct.  

Who is notably absent from the assessment development process? The test-takers themselves. The experience of taking a test can bring up a lot of emotional baggage from the high stakes (or perceived high stakes), prior experiences, and environmental challenges. These factors are not irrelevant; they can significantly alter a test-taker's performance. To overcome these obstacles, we need to make better assessments that consider the experiences of test-takers as a relevant and foundational consideration.   

So how do we design assessments for the test-taker?  How can we ensure the assessments we build are as equitable, engaging, and effective as possible – for all learners? In C3Be, we look to human-centered design techniques to inform our approach.  

Design Thinking 

Design thinking is an approach to creative problem solving that focuses heavily on the user experience as a source of input. Design thinking has been used to create solutions to engineering and product design problems for decades. More recently, it has been applied to the field of education (Razzouk & Shute, 2012; Norton & Hathaway, 2015; Henriksen & Richardson, 2017). C3Be uses a tool called the Innovator’s Compass to structure our design thinking approach (Ben-Ur, 2016).  

Caption: The Innovator’s Compass

Caption: A short video introduction to the Innovator’s Compass. 

The Innovator’s Compass is a graphic organizer and notetaker that contains a series of 5 questions.  

1. People: Who’s involved? 
2. Observations: What’s happening? Why? 
3. Principles: What matters most? 
4. Ideas: What ways are there?  
5. Experiments: What’s a step to try? 

The first question of the Innovator’s Compass urges the designer to consider the people affected by the problem. Once you place the affected people at the center of the problem, the subsequent questions revolve around them: what’s happening to the affected people? What matters most to them? What ways are there for them? What’s a step to try with them?   

C3Be added a sixth question. When we use the compass, we also ask: What biases and assumptions do we have? This question helps us to identify and challenge our preconceptions about the design problem and the people involved.  

Applying Design Thinking to Assessment 

Using the Innovator’s Compass helps us to understand the “problem” of assessment development in a new way.  

1. Who’s involved? Test-takers, people who use test results to make decisions (teachers, administrators, policymakers), subject matter experts, psychometricians, ... who else? 
    
2. What biases and assumptions do you have? How have you (the designer) experienced assessment? Were you a good test-taker yourself? How has assessment benefitted and/or harmed you? 
    
3. What’s happening? Why? Why do the test-takers have to take the test? How does the test affect people emotionally, financially, and logistically?  
   The observations quadrant can also help us understand the construct better: What do we know about the construct? How is it typically measured? What are the benefits and challenges to measuring it this way? What other sources of evidence are there? What kinds of behaviors and actions are typically associated with the construct?  
    
4. What matters most? Some possibilities: Making accurate decisions, respecting people’s time and resources, providing opportunities, fairness, feelings of safety and empowerment, engagement, relevance. These are just a few of many, many possible principles, and many of them will compete with each other.  
    
5. What ways are there? Typical ways, like multiple-choice items, and others, like verbal exams, creative products, and real-world projects. Ask others to help keep the ideas flowing.  
    
6. What’s a step to try? It can be small: talk to a test-taker about their past experiences or do a cognitive lab (an interview where you ask a test-taker to talk through their thinking about an assessment task with you).  

At C3Be we’ve used the Innovator’s Compass to improve assessment design in several areas including cybersecurity, family support, teacher education, and competency-based online graduate programs.  The Innovator’s Compass helps us to better understand the test’s goals and the impact of tests on real users. It’s an iterative process: after every experiment, we gather test-taker feedback on their experience, make observations, refine our principles, and keep improving our assessments.  Stay tuned for more case studies of assessment design in action in C3Be projects! 

References:  

Ben-Ur, E. (2016). Developing Powerful, Portable Design Thinking: The Innovators’ Compass. In Taking design thinking to school (pp. 239-247). Routledge.  

Henriksen, D., & Richardson, C. (2017). Teachers are designers: Addressing problems of practice in education. Phi Delta Kappan, 99(2), pp. 60-64.  

Norton, P. & Hathaway, D. (2015). In search of a teacher education curriculum: Appropriating a design lens to solve problems of practice. Educational Technology, 55(6), pp. 3-14.  

Razzouk, R., & Shute, V. (2012). What is design thinking and why is it important? Review of Educational Research, 82(3), pp. 330-348.