By: Roy Swift, Ph.D., Executive Director, Workcred - an affiliate of the American National Standards Institute (ANSI)

Credentials hold the power to build education and career paths. But the last decade has seen enormous growth in the number and variety of labor market credentials – college degrees, educational certificates, industry certifications, occupational licenses, and micro-credentials such as badges. This growth has fueled confusion among students, workers, job seekers, and employers who are left to navigate the “credentialing maze” without a guiding compass.

There are many philosophical debates as to what constitutes validity or the validity of test score inferences. Within certification we hear things like “a job analysis is typically conducted every five years unless the content changes more or less frequently.” You may have also heard that more than eight SMEs are required for test development activities, while others suggest six or more is sufficient. Some suggest that SMEs should not overlap with separate test development meetings.  So in essence psychometricians will typically say “it depends” in response to questions about the appropriate way to design a certification examination program.

Today, the hegemony of formal education’s traditional business model is being challenged.  Macro trends in employment, technological advances and the inability for our legacy educational systems to maintain pace with an unprecedented global demand for learning have fueled a renaissance in educational entrepreneurship and innovation that are sure to change the way we think about education and learning.

A renowned American Professor of Measurement and Testing, Robert Ebel, published the following definition of a performance test:

“In a performance test the subject is required to demonstrate his or her skill by manipulating objects or instruments.”

This definition was first published in a measurement textbook in 1965. The definition seems solid, except that evolving technology may have changed the concept of manipulating objects or instruments since the time of publication.  For example, robotic surgery to perform a kidney transplant from a location 7,000 miles away from the patient.  This type of ultra-high speed data transfer and related virtual environment begins to change the concept of manipulating objects or instruments.

There has been a lot of buzz about ISO/IEC 17024: 2012 – Conformity Assessment – General requirements for bodies operating certification of persons.  With so many acronyms associated with this standard (ISO, IEC, ANSI, IAF, etc.) it is easy to get confused about who owns this standard and who accredits against it.   We will explore specifics of the standard itself in another post.  The purpose of this post is to help explain the various bodies associated with ISO 17024.

In a recent Internet search, I came across Professional Certifications: Can we Avoid Them? by Uganda-based consultant, Michael Niyitegeka. In this article, he suggests that in the wake of education commoditization, education qualifications have become “tradable goods,” forcing employers to find alternative avenues to gauge one’s competency above and beyond academic qualifications.  While we might share a different view about education, the point that certification plays a role in qualifying people for employment or takes over where education leaves off is worth examining and assigning a value to.  

The Standards for Educational and Psychological Testing (APA, AERA, & NCME, 2014) (Standards) was recently revised and updated.  As credentialing organizations begin to process what the revised version of the Standards means to them, it is important to point out most of the workplace testing and credentialing chapter remained the same.  However, statements about the need for reliable subscore reporting are of vast importance, as is the design of examination score reports. 

Computer Adaptive Testing (CAT) is a testing methodology that weds two processes—adaptive testing and computer administration—for efficient measurement and administration. When compared to fixed length exams delivered linearly on a computer, CAT exams measure a candidate’s ability with fewer delivered exam questions and higher precision. By precision, it is suggested that error is reduced and reliability is increased. 

Overview

The multiple response (MR) item type is, hands-down, one of the most useful alternative item types available today.  It isn’t a new item type; the MR been used in paper-based assessments for many years (under various names, including multiple select, multiple-answer, multiple-choice, etc.). Now that the MR item type is provided as a standard option within many CBT applications, it is becoming a more popular option.

The process for developing a certification exam has evolved over a number of years as more and different types of organizations seek to assess the competence of workers and to certify people. The certification exam is often at the center of this initiative and subject to the most scrutiny, so it’s important to start with a solid foundation and good planning.  For most professional certification programs, the steps for developing certification examinations include:  conducting a job/task analysis, developing an examination blueprint, writing and reviewing items, assembling and reviewing an initial exam form, conducting a pilot or beta test of that exam form, and establishing a passing score.