From the Item Bank

The Professional Testing Blog

ISO/IEC 17024 Implementation Around the World

ISO/IEC 17024: Conformity Assessment – General requirements for bodies operating certification of persons was originally published by the International Organization for Standardization (ISO) in 2003.  The standard was revised and republished in 2012.  Now that the standard has been around for 13 years, what has been the success (or not) of implementation of the standard around the world?

The International Accreditation Forum (IAF) is a body comprised of Accreditation Bodies (ABs) such as the American National Standards Institute (ANSI).  IAF deals with implementation issues associated with standards as well as determining when there is a need for a guidance document associated with the implementation of a standard.  For example, ISO/IEC 17024:2003 had an IAF guidance document associated with it, IAF GD24.  Since the 2012 version of ISO/IEC 17024 has been implemented, the former guidance document was eliminated as many of the guidance concepts were incorporated into the 2012 version of the standard.  IAF established a 3 year implementation period so accredited CBs had to be operating under the 2012 version by 2015.

In 2016, knowing all CBs had to have implemented the 2012 version, IAF conducted an ISO/IEC 17024 implementation survey to identify any difficulties or issues certification bodies (CBs) and ABs were having with the 2012 version.  ABs and CBs from 51 different countries responded to the survey with 56 ABs indicating they had accredited one or more CBs to ISO/IEC 17024:2012.  Ten ABs indicated they had not yet accredited any CBs to ISO/IEC 17024: 2012.

The implementation survey identified that ABs were having the most difficulty with clause 8 (certification schemes) and clause 9 (certification processes) of ISO/IEC 17024 while CBs were having the most difficulty with clause 4 (general requirements).  This is an interesting finding but consistent with what I have personally experienced when conducting ISO/IEC 17024 workshops for AB Assessors all over the world.  In many parts of the world, the concepts of measurement and psychometrics are not as familiar as they are in the US.  And measurement and psychometrics are mostly contained in clauses 8 and 9 of the standard.

Certification of persons in some countries has been implemented by educational institutions, employers and other inappropriate bodies and sometimes based on qualifications (education only or experience only) instead of demonstrated competence (knowledge and skills to achieve intended results).  And it is not uncommon to find that ABs in some countries have accredited CBs who have not followed a valid and acceptable process for developing their schemes.  Not understanding that the knowledge and skills being measured by the certification program should be related to a job as demonstrated through a Job/Task Analysis (JTA), some of these programs are based on the “opinion” of one or two experts as to what should be measured by the certification program.  Frequently, outside of the US, the CBs and ABs have never even heard of a JTA or Practice Analysis.

Other areas where lack of understanding is frequent include not having any kind of an examination blueprint (thus no ability to ensure consistent content when creating the exam forms), no evidence of any kind of passing or cut score standard being set (random pass standards of 75 percent are still the norm in many places in the world), no evidence of any kind of an examination analysis or item analysis being conducted, and no evidence of any examination security being in place.

I can remember sitting on one international meeting and talking about evaluating examination forms by conducting examination analyses and item analyses and looking at the statistics associated with the overall exam and the specific test items and their distractors, and being told, “Well that’s how you do it in the US.  We don’t do it that way over here in Europe.”  When asked how they did it, I was told the professor who developed the exam simply looked at it and determined it was fine.  That was how they did it.

Standardization is a process.  The world has taken some baby steps towards standardization since ISO/IEC 17024:2003 was launched.  We now have most ABs and CBs understanding that awarding a certification only based on completion of education or work experience without any kind of an assessment of knowledge and skills is not acceptable.  That is a big step for most of the world.

But we still have a long way to go.  As ISO/IEC 17024 matures and we orient more of the world to the standard and what the clauses mean, we can expect that ABs and CBs around the world will become more familiar with the meaning of the clauses in ISO/IEC 17024 and thus more standardized in what they are accepting and accrediting.  It is incumbent on those of us familiar with best measurement practices to share the word with the rest of the world.   Presenting at international conferences and other venues helps to promote an understanding of measurement practices.

Only when the world is looking at ISO/IEC 17024 and seeing the clauses in the same way and understanding the meaning behind the clauses in the same way, will we have true standardization.

Categorized in: Regulatory Power