From the Item Bank

The Professional Testing Blog

How to Develop a Certification Exam

The process for developing a certification exam has evolved over a number of years as more and different types of organizations seek to assess the competence of workers and to certify people. The certification exam is often at the center of this initiative and subject to the most scrutiny, so it’s important to start with a solid foundation and good planning.  For most professional certification programs, the steps for developing certification examinations include:  conducting a job/task analysis, developing an examination blueprint, writing and reviewing items, assembling and reviewing an initial exam form, conducting a pilot or beta test of that exam form, and establishing a passing score.  As organizationsplan their exam development activities, it is imperative that each step be detailed and agreed upon prior to development to ensure a successful and timely launch.  There are key decisions related to each step in the exam development process that have the potential of making or breaking your exam’s debut, for example, have you secured qualified and enough SMEs? Will you have enough items?  While we understand that a certification program is more than an exam, the exam is often the candidate’s first encounter with the certification body and the biggest hurdle to earning certification, so it’s best to make a good first impression.

Job/Task Analysis

A job/task analysis (JTA) is the foundation of any certification exam and often lays the foundation for other program requirements.  The knowledge, skills, and abilities (KSAs) identified during the JTA will help form the examination blueprint or content outline for the certification exam.  In other words, the content being assessed by the certification exam will largely be determined by the JTA.  The two key decisions in conducting a JTA are choosing the right JTA method and selecting the right Subject Matter Experts or SMEs (for more on selecting the right SMEs, see here.  There are many methods for conducting a JTA (DACUM, Critical Incident Technique, Functional Job Analysis, Task Inventory Analysis, etc.).  Some methods include conducting focus groups, some include one-on-one interviews, job shadowing, literature reviews, small group interviews, or large-scale surveys.  Each method and process has an impact on budgets, timelines, and the number of individuals involved.  In the end, you want to select a JTA method that will give you the most accurate representation of the KSAs you would like to assess, while saving you time and money.

Item Writing

The item types and content of each item on your certification exam has a huge impact on the perception of the exam.  Item types that are confusing to respond to and items that are poorly written will give test takers a negative impression of the exam.  To help determine what item types may work for your certification exam see here.  Once item types are determined, the next goal is writing items that are clear, concise, and straight to the point.  A certification exam should never include items that are intentionally tricky; you want a test taker to get an item right if he/she knows the content being assessed and wrong if he/she doesn’t know the content.  Training a group of SMEs on item writing best practices (or general do’s and don’ts associated with item writing) will greatly improve the content of each item, and the overall impression of the exam.

Item Review, Form Assembly, and Form Reviews

It is imperative that all newly written items are reviewed by a group of SMEs (preferably a different group of SMEs than the ones who wrote the items).  The initial exam form should be assembled based upon the examination blueprint or content outline, and that form should also reviewed by a group of SMEs.  While most would agree that conducting item reviews is a critical stage in the development process, not all organizations conduct form reviews.  The exam form should be reviewed as a whole before it is administered to look for content imbalances, item enemies, and any missed grammatical or typographical errors.  All of the items on the exam form should seem to “go” together.  In our experience, a group of SMEs can tell you really quickly how the exam as a whole feels (too easy, too hard, over-weighted in a content area).  Having a final form review before launching your initial exam will add one more layer of confidence that test takers will see the quality of the exam and have confidence in it.

Pilot Testing and Passing Score Studies

The final two steps in developing a certification exam is pilot or beta testing the initial exam form and establishing a passing score (or cut-score) on that initial form.  These two steps can be done in either order, although it is the preference of many psychometricians to pilot test first, withhold scores, and then establish a passing score so that actual candidate data can be used to inform the decision.  Key considerations with these last two steps are ensuring a large enough candidate volume and representative candidate population at the initial pilot test, and selecting a cut score methodology that will produce the most accurate passing point for your exam.  The right number of test takers for the pilot administration of your exam will depend on many factors, but most agree that you will need a minimum of 30-50 test takers in your initial administration.  If the process used to establish your passing point produces a cut score that both your SMEs and Board are happy with, it is likely you will have a passing rate that makes sense to your candidate population as well.

Candidate Experience and Reputation

Candidates will talk about the exam and will form options, right or wrong, about its content, accuracy and relevance.  In certification the old adage that “perceptions are realities” is still true.  You will know what your candidates think.  Following acceptable practices, making the right decisions throughout the examination development process, and focusing on producing a quality exam goes a long way in creating a good experience.  While technology and more exam administration opportunities (think CBT, web-based, mobile apps) have facilitated the outreach of exams, the most important factor to the reputation of a certification program is still the quality of its products—consider your exam one of your most important ones.

Categorized in: Industry News