From the Item Bank

The Professional Testing Blog

Expectations When Developing a New Certification Examination—Be Aware So You Can Manage Your Expectations and Those of Subject-Matter-Experts

Co-authored by Reed Castle, PhD and Christine D. Niero, PhD

If you are new to developing a certification program, this post outlines some of the key steps needed to develop a psychometrically sound exam. No matter what approach you take in exam development, subject-matter-experts (SMEs) are critical contributors to achieving a valid and reliable product.  While many SMEs are anxious to participate, they are sometimes unaware of the expectations made of them, and the demands of the process.   The following five key examination development activities describe the process and the role of the Certifying Body (CB) or sponsoring organization and SMEs during initial exam development. They are: job analysis, item and form development, administration, standard setting (Cut-score), and statistical review.

Expectations Associated with the Job Analysis

A job analysis is the foundation of a certification examination—it is the first step in exam development. During this process, the role of the certified professional is delineated which subsequently leads to the test blueprint.  The use of SMEs is essential as they are the persons either doing the job, or very knowledgeable about the competencies required to perform the job.  No matter what methodology is selected, SMEs need to participate.  Professional Testing has achieved efficient and satisfactory results conducting one in-person meeting with subject matter experts (SMEs), followed by a post meeting after the validation survey results administered to existing practitioners and job incumbents have been analyzed. The CB is responsible for recruiting eight to twelve SMEs for the job analysis meeting, establishing a plan for SME travel and housing, and securing a host site for the two to three-day meeting. When creating a budget, considerations should be given to SME travel expenses, (e.g., food, travel and lodging), geographic location, employment setting, and job title, in addition to other relevant demographic data.  A SME panel that collectively represents the breadth, depth and scope of the certification will yield the best outcomes.  SMEs need to understand why they are selected, and why they were not.  

After the in-person meeting is completed, a survey will be drafted for the validation of the blueprint. The CB is responsible for identifying a cohort of survey respondents. Typically, the surveys are electronic and the CB will be responsible for gathering email addresses and sending out the invitation to survey participants. After the survey is administered, a follow-up webinar will be scheduled with SMEs to finalize the examination blueprint.  At the outset, SMEs should understand their roles and responsibilities, time commitments, and the length of time it takes to conduct both steps of the JTA.  It is also useful to inform SMEs that they may be asked to participate in subsequent exam development activities. 

Expectations Associated with the Item and Form Development

After the exam blueprint is finalized by SMEs, exam items (questions) and exam forms are developed. For new programs, there may be two SME in-person meetings. The first is to write new items to populate the item bank, and the second SME meeting is to review newly written items and approve them. So, expect two, three-day meetings with eight to ten SMEs to populate the item bank of approximately 250 items for a 100-question exam. After the item bank is populated with SME approved items, a psychometrician or test developer will create an exam form adhering to the exam specifications and weightings derived from the validation survey step of the job analysis. At this stage, the CB may be able to convene a conference call/webinar with three to five SMEs to review the newly created exam form. The purpose of the review is to assure there are not duplicative items or items that provide clues to other items on the form, and to perform one last check of the form prior to being published.  It is optimal if some of the SMEs who participated in the job analysis can write and review items to carry over “institutional memory,” while also including SMEs new to the exam development process.  Typically, a group of 8 – 10 SMEs are required for item writing and review.  Again, the CB will need to consider costs associated with travel, meals, accommodations, meeting space and psychometric expertise.

Expectations Associated with Exam Administration

Working closely with the psychometric team the CB will make decisions on how the exam will be administered. Believe it or not, there are still organizations using paper administrations but increasingly most CBs have moved to computer administration. Consideration of exam administration length (also known as “seat time”), testing within specific date ranges versus open testing, matching test access with CB professionals, how the candidates will schedule their exam and costs are a few of the decisions that must be made. A couple of costs for the CB are the initial cost to set-up with a CBT provider and the direct costs associated with each exam administration. CBs should also consider the characteristics of their test-takers—how much can they afford to pay to take the test; are costs of submitting the application in addition to, or bundled with, test administration fees; do they reside in geographic areas that are easily accessible to a CBT test center, or will air travel be required for some; would they prefer to test at an industry conference, etc.  Knowing the expectations of test-takers will guide the CB in making the right choice for test administration.  SMEs are invaluable in providing CBs with this type of information.

Expectations Associated with Standard Setting

After the exam is administered to the first cohort or pilot group of test-takers, a passing score (also known as a cut score) must be established.  Typically, a SME panel is created for a one or two-day in-person meeting. During this meeting, a psychometrician will facilitate the process of defining the minimally qualified professional. SMEs will subsequently provide ratings or make judgements as to where the passing score should be. This information will be used to determine how many items need to be correctly answered on the exam to pass and receive certification. The CB is responsible for recruiting eight to twelve SMEs for the in-person standard setting meeting, establishing a plan for SME travel and housing, and securing a host site for the one to two-day meeting.  Again, it is optimal if a few SMEs involved in previous activities participate in standard setting, along with new SMEs. 

Expectations Associated with Statistical Review of the Exam

After the exam form is administered and enough candidates have completed the exam to provide sufficient data, exam questions with poorly performing item statistics will need to be reviewed and decisions about current and future scoring choices will need to be made. This process can be done either via webinar or at a future item development meeting, and depending on the number of items to be reviewed, can involve as few as three to four SMEs, or more.  Your psychometrician will advise the best number of SMEs to recruit.  As the program matures, there may also be the need to develop additional examination forms, and this may be accomplished via a psychometrician facilitated webinar to evaluate the results of equating studies (e.g., what the cut score should be for a new exam form). This is because not all exam forms are equal with respect to difficulty.

As can be seen, there are many steps in developing a certification exam, and understanding the expectations of the CB, SMEs and the process will help to ensure a smooth, well-articulated and communicated process. A CB can expect to have up to four in-person SME meetings and a number of webinars just to launch the first administration. While there may be an inclination to give thought to doing some of these steps remotely, starting a program takes a great deal of time and effort and relying on remotely working SMEs can extend timelines and create bottlenecks to the process. Some CBs have also had experiences where the quality of the product declines with SMEs working remotely. It is much better (and our experience) to do remote work after the program is mature and SMEs are well trained and experienced.

This is a very high overview of what to expect during the initial test development process. A CB can expect the process to take anywhere from four to twelve months. It is largely dependent on the scheduling of SME meetings, SME availability, resources, and the expectations of the CB and its leadership.

For more related blogs check out Careful Selection of Subject Matter Experts is the Key to a Successful JTA Meeting and The Use of Certification and Licensure Programs in Developing Career Path Models

Categorized in: Industry News