Defining Attributes for Well-Suited Individuals for Cybersecurity – Involving Users & Stakeholders in Construct Development for More Inclusive Assessments

In a recent project, C3Be worked with EnterpriseKC to define attributes needed for success in entry-level cybersecurity occupations and develop assessments measuring these attributes.  Current published definitions of employability skills have been abstracted to generic concepts, broadly applicable across many fields. They may not represent the specific contexts and needs of the cybersecurity field, nor the perspectives of the diverse range of stakeholders. Therefore, C3Be sought to involve regional cybersecurity employers and employees in defining and prioritizing the skills needed for success in the cybersecurity field.  

The first step in developing assessments is to define what is to be assessed (Downing, 2006; Ferrara, et al., 2016). At the start of the process of defining the construct, we researched and used definitions and descriptions of the attributes from academic literature. We also studied existing instruments to evaluate how the instruments attempted to measure the attributes. We used two key resources (O*NET from the US Department of Labor and the NICE Framework from the National Institute of Standards and Technology) to inform the selection and validation of the attributes as well as the scoring approach.     

Perspectives from current industry practitioners formed the bedrock of our assessment development process. In a sense, the leaders, managers, and employees in cybersecurity roles across the region represent one of our core “end users” for the assessments. To design assessments that identify well-suited talent who could be successful in various cybersecurity roles, we first need to define what makes a candidate ‘well-suited’ in the first place. That is, what are the attributes, aside from technical content knowledge, that matter most to industry leaders? To answer this question, we conducted three rounds of industry focus groups, gathering participants from different levels of leadership and experience within the cybersecurity industry in each round. Round 1 involved 10 leadership-level participants (chief officers and business owners), Round 2 involved 22 supervisors and hiring managers of entry-level employees across 18 organizations, and Round 3 involved employees new to the field.   

Each focus group’s conversations were geared around the attributes necessary for working in cybersecurity. In these discussions, they shared examples of what attributes like ‘Growth Mindset’ or ‘Analytical Thinking’ look like on the job, in authentic cybersecurity contexts. Analysis of the focus group transcripts enabled us to distill a set of “indicators” that describe behaviors or evidence commonly exhibited by cybersecurity candidates with that attribute. Thanks to our industry participants, we could zero in on how each attribute is discernible in cybersecurity contexts – leading to more authentic, effective assessments aligned with the specific skills, abilities, and dispositions that matter most in this field. Essentially, by answering the questions “What does it look like when someone possesses this attribute? What can they do differently from someone who is not strong in that area?” we could then answer the fundamental talent-assessment question “How can we determine if someone has this attribute before they enter the role?”.   

Well-Suited Attributes   

The ten attributes were defined as follows:  

• Analytical thinking: an approach to problem-solving that involves gathering information, noticing patterns in data, interpreting to find meaning, and prioritizing the next steps.   
• Attention to detail: an ability to detect and respond to important information in a complex environment.   
• Collaboration: the ability to work with and coordinate among people with many different skill sets.   
• Communication: the skills to document incidents and responses and translate complex technical issues into language that can be understood by a non-expert.   
• Creativity/innovation: a desire to improve things by making changes, trying new methods, and experimenting with solutions that they have never used before.   
• Curiosity: excitement when they encounter new challenges. They are driven by curiosity to find answers and solutions to problems.   
• Growth mindset: a view of themselves as learners who can continue to acquire new knowledge and skills as technology advances.   
• Perseverance: persistence in the face of repeated obstacles and challenges, in pursuit of a long-term goal.   
• Protective instincts: concern for protecting others, their interest, and valuable assets against risk.     
• Technical acumen: familiarity with basic technology, sufficient that they can quickly orient to new technologies.  

From Round 3 participants, we were able to gather insights from entry-level employees. While we asked managers and executives which attributes they are looking for most in candidates, we asked the Round 3 participants which attributes they relied upon most as they started their careers in cybersecurity.  

Table 1. Incidence of Attributes Candidates Relied Upon Most  

As seen in Table 1, the attribute the participants indicated they relied upon the most was the combination of collaboration and communication. They explained that because aspects of security are far reaching, communication and working with others (collaboration) were crucial in solving issues and undertaking work activities. They also indicated that communication and collaboration are ways they can learn procedures and solutions in more depth. Finally, communication and collaboration represented the need to ask questions to gather information and learn. Still many agreed that one should ask questions after they have spent time researching and trying to discover things independently.   

The second attribute the participants relied on most was perseverance. Perseverance was considered both a precursor to other attributes and a modus operandi on-the-job. Participants indicated that perseverance reflects passion and caring to see things through to conclusion. As one participant indicated, it is at the heart of continual learning and performance. 

Through the industry focus groups, we also learned which attributes represent the highest priority for cybersecurity success — and the greatest unmet need hiring managers face in their current talent searches. The collective priorities across all industry focus groups are shown in Table 2. 

 Table 2. Cumulative Ranking of Priority Attributes for Rounds 1, 2 and 3  

Interestingly, the top attributes prioritized across the board were some of the least technical ones: Growth Mindset, Communication, and Curiosity. That’s not to say that technical skills don’t matter in cybersecurity careers, but rather – as one participant pointed out - “the field changes so quickly, you need to be continually learning in order to keep up!”                            

To meet the demands for cybersecurity personnel, we worked with EKC to take a developmental approach to assessing individuals interested in the sector. Depending on their performance on the Talent Exchange portal, they are considered to be ready for an entry-level position or are encouraged to explore training and experience to develop related attributes. This principled and multi-user co-design approach for assessment development can be applied to other sectors to expand the pool of candidates. 

References 

American Educational Research Association (AERA), American Psychological Association (APA), & National Council on Measurement in Education (NCME) (2014). Standards for educational and psychological testing. American Educational Research Association. https://www.testingstandards.net/open-access-files.html  

Ferrara, S. Lai, E., Reilly, A., & Nichols, P. D. (2017). Principled approaches to assessment design, development, and implementation. In A. A. Rupp and J. P. Leighton (eds.), The Handbook of Cognition and Assessment: Frameworks, Methodologies, and Applications (pp. 41 – 74). John Wiley & Sons, Inc.  

Kirchgasler, C. (2018). True grit? Making a scientific object and pedagogical tool. American Educational Research Journal, 55, 693–720. 

Lane, S., Raymond, M.R., Haladyna, T. M., & Downing, S. M. (2016). Test development process. In S. Lane, M. R. Raymond, & T. M. Haladyna (eds.), Handbook of test development (2nd ed., pp. 3 – 18). Routledge. 

Tomlinson, M. (2012). Graduate employability: A review of conceptual and empirical themes. Higher Education Policy, 25, 407–431. 

Tymon, A. (2013). The student perspective on employability. Studies in Higher Education, 38(6), 841–856.