The Paradox of Process Competencies: How to define and measure outcomes about processes

In competency-based education (CBE), learners progress by demonstrating mastery of fixed outcomes, such as specific skills, knowledge, or ability (Bushway, Dodge, & Long, 2018; Fernandez et al., 2012). Often, learners show their attainment of these outcomes through the creation of some sort of final product. However, the use of a final product to assess competency can imply an endpoint in a learner’s development which can seem to go against measures of performance that are focused on a process, rather than a single product or a single point in time. We call these types of skills “process competencies.”

So, what exactly are process competencies and how should they be defined and measured in CBE programs?

WHAT ARE PROCESS COMPETENCIES?

While developing CBE programs for a range of professions, our team has identified a subset of competencies that we call “process competencies.” We describe process competencies as an orientation to problem-solving that may be specific to a particular discipline or field. One example of a process competency that we’ll explore here is design thinking. Some other examples of process competencies include critical thinking, decision making, scientific discovery, etc. While these process competencies are authentic to the workforce, defining and measuring these competencies is not as straight-forward as it is with competencies that focus on discrete skills.

DEFINING PROCESS COMPETENCIES

In CBE, a main tenet of assessment is that before you can measure an outcome, you must first define it (Bushway, Dodge, & Long, 2018).  When it comes to process competencies, we see three approaches for how to define them:

1. Step-by-Step: The first approach is a step-by-step approach in which a process competency is broken down into an ordered list of how the underlying knowledge, skills, or abilities are applied.
2. Descriptive: The second approach is a descriptive approach in which a process competency is defined by descriptions of the core characteristics or qualities of a successful application.
3. Product-Centered: The third approach is a product-centered approach in which a process competency is defined by what it yields rather than how it occurs. 

DEFINING DESIGN THINKING AS A PROCESS COMPETENCY

As an example, let’s consider how the process competency of design thinking might be defined. Design thinking is an approach to creative problem solving that is used in many fields such as engineering, product design, and education. To define design thinking, we could use any of the three approaches outlined above.

A step-by-step definition of design thinking might look like picking a framework that breaks design thinking down into a series of comprehensive, methodical steps. For example, using the Innovators’ Compass (Ela Ben-Ur, 2020), we might define design thinking as consisting of the following six steps:

1. Define the Problem and Users
2. Make Observations
3. Distill Principles
4. Generate Ideas
5. Experiment with Prototypes
6. Iterate

It is important to note, however, that this is one possible definition for design thinking using one specific framework as a model. There are many existing design thinking frameworks and using any one of those instead may lead to a different breakdown of steps and a different definition for design thinking. Part of the work in this approach is deciding which framework to use as the foundation for your definition. 

A descriptive definition of design thinking might look like articulating the habits, mindsets, and behaviors that embody good design thinking. This might involve working with competency learning designers, faculty and subject matter experts, users, and/or other stakeholders to consider both values and professional standards to determine what is most important. For example, we might say that good design thinking is creative, human-centered, iterative, inclusive, and collaborative. However, there are multiple ways to describe what good design thinking might look like and you might choose different descriptors based on what you see as most valuable. For example, you might instead decide to describe good design thinking as being empathetic, innovative, active, and equitable. Or you might choose to describe good design thinking in a completely different way! Part of the work in this approach is norming together around what you think good design thinking looks like.

A product-centered definition of design thinking might look like specifying what a well-designed product that comes out of a good design process would look like. For example, you might think that a product that is the output of good design thinking should comply with all the stated design constraints, should meet the user’s needs, and should be practical. If the final designed product meets all these criteria, then it would be indicative of good design thinking. Again, part of the work in this approach is norming around what you think a good design product would look like.

IMPLICATIONS FOR ASSESSMENT

The approach used to define a process competency carries natural implications for how we measure mastery of that competency. Here we share some of the challenges we’ve encountered and lessons we’ve learned when it comes to the different assessment approaches for measuring process competencies.

Since a process competency defined in a step-by-step approach focuses on individual steps, it might make the most sense to choose an assessment format that checks on each step of the process. Likewise, you may create a rubric that looks like a checklist of steps so that demonstration of competency occurs when all the steps have been checked off. An affordance to this way of assessing a process competency is that since it is a defined set of steps to check, it can be a straightforward and objective method of assessment. However, a limitation is that by focusing on the discrete steps in assessment, you may not be able to capture or assess the connections between steps or the quality of the process as a whole.

For a process competency defined through a descriptive approach, your chosen assessment method should focus more on whether the essential qualities of the competency are present and so formats such as self-reports, interviews, or observations may be appropriate. In this case, your rubric would contain more adjectives and descriptive words and would rely more on an assessor’s subjective judgement. Some affordances to this way of assessing a process competency are that it can better represent the desired skills “at the heart” of the competency and can be more transferable across different contexts since it doesn’t rely on frameworks that might be specific to certain fields or organizations. On the other hand, this is a more subjective method of assessment and can be difficult to judge and assess. Also, the descriptions that are used to define the competency are often related to internal qualities, mindsets, or beliefs that are hard to assess as they can’t ever truly be known.

Finally, when it comes to assessing a process competency defined through a product-centered approach, the focus is on evaluating outcomes. In this case, assessment happens after the process has been completed and focuses on the end result of the process, which may be done through formats such as portfolios or presentations. Here, the rubrics would focus on the characteristics and impacts of the product being assessed. An affordance to this way of assessing a process competency is that the criteria can be more objective and observable, which makes it easier to assess. However, a limitation is that the quality of the product does not always reflect the quality of the process that made it.

TAKEAWAYS

In summary, based on our experience, our two main takeaways for how to define and measure process competencies in CBE programs are:

• There are multiple ways to define process competencies. Not only do we see three main approaches for defining process competencies, but each of those approaches could also lead to different definitions depending on what frameworks you use or what you see as most important.
• How you define a process competency drives how you assess it. Be cognizant of what matters most in your discipline / process: a methodological process, mindset & approach, or the final product?  Each approach comes with its own affordances and constraints.

Because of this, rather than choosing one approach to define a process competency, consider how to combine the three approaches. For example, you could add descriptive criteria to a step-by-step approach, or you could assess specific steps in addition to a final product. It may make more sense for you to draw from different pieces of each approach to better capture and measure your process competency.

If you have ideas about or experience with defining process competencies, we’d love to hear from you! Get in touch with us at c3be@ku.edu.

Footnote: This post is based on a presentation given at CBExchange 2024. Castle, C., Moussavi, R., Stockton, J. (2024). The Paradox of Process Competencies. CBExchange 2024, Colorado Springs, CO.

References

Ben-Ur, E. (2020). The Innovators’ Compass. https://innovatorscompass.org/

Bushway, D. J., Dodge, L., Long, C. S., & Laitinen, A. (2018). A leader's guide to competency-based education: From inception to implementation. 

Fernandez, N., Dory, V., Ste‐Marie, L. G., Chaput, M., Charlin, B., & Boucher, A. (2012). Varying conceptions of competence: an analysis of how health sciences educators define competence. Medical education, 46(4), 357-365.