From the Item Bank

The Professional Testing Blog

How A Threat Analysis Can Help Safeguard Impartiality

For certification bodies (CBs) accredited to, or working toward, International Standard ISO/IEC 17024 Conformity assessment—General requirements for bodies operating certification of persons, meeting requirement 4.3 Management of Impartiality, has likely resulted in revisions to the policies and practices of the organization.  Impartiality is a consideration across all aspects of the certification program and in the actions of the CB at all levels with all personnel.  Section 3, Terms and definitions, 3.1.5 defines impartiality as “presence of objectivity.”[1]   At the point of implementation, impartiality requires that all decisions related to certification are fair and applied consistently across all applicants, candidates, and certified persons to assure that certification is awarded solely on the individuals’ ability to meet and maintain certification requirements, and nothing else.  Achieving compliance with Section 4.3, among other things, requires documentation of policies and procedures, evidence of fairness, and implementation at the organizational level.  It also requires the CB to publish its understanding of the importance of impartiality in all certification activities.

So how can CBs show such “evidence”?  Annex A.2 of ISO/IEC 17024, Impartiality (paraphrased below) is a good place for CBs to start.[2]

• 2.1 certification of a person should be based on objective evidence through fair, valid and reliable assessment, and not be influenced by other parties.
• 2 certification bodies need to be perceived as impartial in order to give confidence in their activities.

Meeting this requirement should begin with a comprehensive policy on impartiality, which should include the CB’s commitment to acting impartially; making certification decisions in accordance with policies and procedures; making public the policies and procedures affecting applicants, candidates and certified persons; and accurately conveying information about the certification to all interested parties.[3]

Another way a CB can show “evidence” is by conducting a threat analysis, to identify any threats to impartiality[4] which should be addressed first by policy.  A sample policy may read: “In upholding its commitment to maintaining the highest level of impartiality and objectivity in its practices and decision making, CB personnel shall annually, or as circumstances may warrant, conduct a Threat Analysis in accordance with the impartiality analysis criteria.  The Threat Analysis shall be conducted by the Certification Director”.   Annex A, A.2.3[5] lists threats to impartiality including (paraphrased):

• Self-interest—threats that arise from persons acting in their own interests for self-benefit
• Subjectivity—threats that arise when bias or subjectivity overrules objective evidence
• Familiarity—threats that arise from a relationship that clouds objectivity
• Intimidation—threats that prevent the CB from acting impartiality
• Financial—threats arising from revenue sources.

In addition to considering these types of threats, CBs should also identify the threats unique to their organizational structure and/or operations.  An example of such a threat is the CB or its parent enterprise offering training that can prepare persons for the certification exam.  How the CB erects and maintains firewalls between training and certification would be assessed and documented during the threat analysis, and reported during the internal audit or at the time of management review.

Once the categories of threat have been determined, CBs should define the desired level of impartiality, or best case scenario, which would be “no threat” and then define the “threat(s)” to the best case and any threat level in-between.

Let’s take the example of certification and training. A CB that offers training has a potential threat to impartiality.  How the CB mitigates the potential threat is what the CB needs to analyze in its threat analysis.  One way to mitigate the potential threat is by implementing an organizational structure that assures the separation of certification activities from training, including courses that may help to prepare candidates for the certification examination.  Another way is by establishing policies that place all certification decisions under the authority of the CB, and a management system that implements procedures to assure such authority.  Policy should also apply to personnel, so SMEs are not engaged in training activities that may support certification.

A threat to impartiality is the failure to mitigate threat, and would surface if the activities of the CB do not remain separate from training, and if specific training courses are considered “exclusive” or “essential” to exam preparation.  Discounts on the certification examination as conditions of purchasing certification prep materials such as a study guide also pose a threat.  And SMEs engaged in certification activities such as item writing and review who are not prohibited from participating in training not only pose a threat to impartiality, but also a conflict of interest.

Let’s apply this to a threat analysis rubric. Each category and threat level should be categorized and assessed.  The threat category can be “Training.” The threat level can be “low,” “medium,” and “high.”  If a threat is detected, define the threat and threat level, assess the risk to impartiality, describe how the risk(s) will be mitigated and document follow-up activities.  Here is an example:

Certification and Training

Or, here is another example:

The format of the Threat Analysis rubric and report is at the discretion of the CB.  Use whatever works best for you—charts, narrative, check-sheets—but be sure to document this procedurally, within the program’s management system.  In conducting a threat analysis, CBs are providing evidence of their commitment to managing impartiality and actively showing their “due diligence” to determine threats, both real and perceived, of an individual, organization or product influencing or benefiting from certification.

[1] International Standard ISO/IEC 17024 Conformity assessment—General requirements for bodies operating certification of persons (ISO/IEC 17024), page 2.

[2] ISO/IEC 17024, page 19.

[3] Interested party is defined as Individual, group or organization affected by the performance of a certified person or the certification body; ISO/IEC 17024, page 3.

[4] ISO/IEC 17024, requirement 4.3.6, page 4.

[5] ISO/IEC 17024, page 19.

Categorized in: Industry News, Test Development