From the Item Bank

The Professional Testing Blog

Know Your Certification Program’s Risks and Vulnerabilities

It all starts so innocently.  You want to develop a certification program to do something good—elevate the status of your profession, develop a standard for your industry, establish additional revenue streams, and create a pathway for career advancement. With all of these good intentions, what can possibly go wrong?  The short answer is plenty.  In fact, there are many risks associated with offering certification programs.  Let’s highlight some risks, starting at the beginning.

Where to set the bar?  Most certification programs have requirements for awarding certification and for qualifying for the examination, in short, for entrance into the certification program.   Typically these requirements include education, training, work experience, and evidence of attainment of specific skills, or a combination thereof.  Certification programs expose themselves to risk when the entrance requirements are not vetted with subject-matter-experts (SMEs) and are not related to the certification requirements—in short, entrance requirements do not align with the competency requirements (think exam blueprint). Setting the bar too high, or establishing requirements that are not related to the certification, for example, membership in an association, puts your organization in the position of keeping otherwise eligible candidates from earning the certification.  Setting the bar too low sets candidates up for failure.  Neither situation is fair.  Absence of fairness increases risk.  So, when establishing certification and eligibility requirements, asking these two questions can help reduce risk: 1. If the candidate meets the eligibility requirements and studies seriously for the exam, will they have a fair shot at passing? And 2) What assurances do the eligibility requirements provide related to demonstrating competence that the examination cannot assess?  This is a good exercise to work through with SMEs.

When it comes to the examination, failure to follow acceptable industry practices can increase the risk to the certification program.  A job-task analysis (JTA) establishes the foundation for the examination.  Certification programs may utilize several JTA methodologies, but as a rule of thumb, be sure to involve SMEs who are job incumbents, represent the level of certification (entry versus master), and collectively represent the breadth, depth and scope of the certification.  Additional validation of the duties, tasks, knowledge, skills, attitudes and attributes (KSAs) with a broader group of incumbents and stakeholders is also necessary to determine the competencies and weighting of competencies to be assessed. The JTA not only results in the development of a weighted examination blueprint, but can provide useful data to establish eligibility and recertification requirements.  Finally, be sure that a JTA has been conducted recently enough to reflect current and relevant job duties and tasks.  Risk increases not only when acceptable JTA practices have not been followed, but also when you are assessing dated or irrelevant job duties, tasks and KSAs.

Risk is significantly reduced to certification programs when the right SMEs are utilized throughout the examination development and maintenance processes—simply stated, collectively the SMEs must possess expertise across all certification requirements and be representative of the profession’s population.  Be sure to vet SMEs based on their qualifications and expertise, and document evidence of this.  The partnership between the certification program and SMEs is key when writing and reviewing test questions.  The best scenario is to work with a psychometrician with expertise in certification examinations to make sure the test questions are linked to the examination blueprint, are written at the correct level of difficulty, and are edited for grammar, bias, technical language, item queuing, etc.  A good psychometrician will guide you in this.  Make sure SMEs who have not written the test question review questions (fresh set of eyes) for technical accuracy, and to make sure the correct answer is really the correct answer.  This reduces the risk of advantaging and disadvantaging candidates due to item error, and provides assurances that the exam is measuring what it is intended to.  Implementing an examination review and revision process also reduces risk with routine review of item performance so that only “well” performing items are used on the exam.  Finally, be sure to involve SMEs in the review of examination forms.

Establishing the passing score following acceptable methodologies is key to reducing risk.  As with the JTA, there are several methodologies available to certification programs.  Again, working with a psychometrician and involving the correct SMEs will reduce risk because you have taken precautions to set this important bar in the right place, thereby assuring you are passing the right candidates.  The worst case scenario is to pass an incompetent candidate.  Documentation of standard-setting processes is critical to reducing risk—it’s your evidence if the examination is challenged that you have set this important bar in the right place.

Validity is not solely about the exam; validity is assuring that test score inferences (pass/fail) are appropriate. It is the thorough process that lends itself to creating claims of validity. Including the steps outlined above helps enhance the “appropriateness” of test score inferences by

1. reducing bias (use of multiple SMEs throughout the process)
2. increasing the accuracy (reviewing test and item data to make decision to increase reliability)
3. establishing a passing standard that reduces pass/fail classification error
4. assuring fairness to all candidates

Recertification is yet another important bar certification programs set—only persons who meet recertification requirements keep the credential.  Recertification should be based on the concept of maintaining continued competence, and can involve many options for demonstration thereof—retaking the exam, completing professional development activities, demonstrating specific skill sets, mandating completion of related training, for example, safety.   As with establishing eligibility requirements, recertification requirements should align with the competency requirements (think examination blueprint) and should not entail activities beyond the scope of the credential, for example, membership in an association, or mandating courses profiting the certification program or sponsoring organization.  Risk is reduced when the playing field is levelled, and when unnecessary burdens such as membership and financial are removed.  Recertification should be based on the certified person’s ability to demonstrate continued competence and nothing else.

Enforcing a Code of Ethics is one way certification programs provide a level of trust to the stakeholders the program serves, yet it is fraught with risk when not administered correctly.  The Code of Ethics should set forth objective and measurable behavioral norms certified persons are required to uphold.  The Code of Ethics should be a condition of earning and maintaining certification, signed and agreed to at the time of initial application and at recertification, so everyone is aware of the rules upfront.  Complaints filed against certified persons should be thoroughly vetted by qualified SMEs, for example, an Ethics Committee, and when sanctions are imposed, the punishment should fit the crime.  As with all barriers to earning and maintaining certification, due processes should be afforded to persons denied access to the certification, or who have their certification restricted (suspended) or removed. When appeals of sanctions are requested, the appeal should be heard by a body not involved in the investigation or decision regarding the sanction. Affording due process significantly reduces risk.

While there are many more areas of risk certification programs are exposed to, it’s a valuable exercise to identify all potential areas of risk and to conduct a risk assessment just as you would a financial audit. All contribute to the health and viability of the program.

For more information on risk analysis, check out Christine Niero’s presentation from the 2016 ATP Conference: Risk Analysis – How Safe is Your Certification Program?

Categorized in: Test Development