From the Item Bank

The Professional Testing Blog

Subject Matter Experts – The Selection Process

Co-authored by Vincent Lima and David Cox.

One of the most valuable and often under managed assets of certification programs are volunteer subject matter experts (SME’s). These volunteers make significant financial contributions to credentialing programs by providing uncompensated access to their expertise and skills, and through their participation add a legitimacy to the programs they work on.

What are some considerations in selecting SMEs?

It may seem that the greatest experts in the field are the best people to have. And, indeed, if they will lend their names and  expertise to the certification program they can be valuable assets. But there is more to it than that.

First of all, programs need different qualifications of SMEs for different functions.

• A policy-making board needs to have stakeholder representation. Appropriate representation may include members who are not subject-matter experts.
• A job-analysis panel should normally be composed of individuals who are currently doing the job. According to the Standards for Educational and Psychological Testing (2014), in “credentialing, a close link between test content and the job or professional/occupational requirements should be demonstrated” (Standard 11.13).
  • It’s possible to ask supervisors to describe the job they want done, but we prefer to have practitioners describe the jobs they do day in and day out.
  • It’s good to have a representative group: different jurisdictions (different laws?) and geographies (icy conditions or a desert?) are important; representative of large firms, small firms, and sole practitioners bring different perspectives; and a lack of gender and racial diversity may not only skew the results, it may also undermine the program’s credibility.
• In writing and reviewing questions, job experience counts. Established programs will want to draw exclusively from the pool of certificants. Our experience is that item writers with limited knowledge are the most likely to write questions that are either incorrect or trivial. Item reviewers with limited knowledge are likely to miss nuances that better test takers will not miss—detracting from the quality of the test.
• In standard setting (determining the passing score or cut score), we like to have at least some panelists who have less experience with the job. In the most common standard-setting method, panelists are asked to estimate the chances a test taker who is just good enough to pass will get a question right. Having the perspective of people more recent to the practice is helpful.

Second, there is the amount of commitment asked.

• The board is invariably a standing committee. A program may have a separate content committee, responsible for reviewing and approving test items. Where the board likely includes non-SMEs, the content committee would have only true content experts. Where such a standing committee is established, it is good to set terms (say three years) and term limits (say, after two terms, spend at least one term off the committee). This allows for a balance of institutional memory and fresh perspectives with new members.
• In performing fatal-flaw reviews, we find it best to rely either on members of the standing content committee or, in the absence of such a committee, SMEs who performed item review and approval. Fatal-flaw reviews are performed after a test is first administered, statistics are calculated, and candidate comments (if solicited) are gathered. In our experience, SMEs with no attachment to the test are more likely to discard content for fanciful reasons than those who have built the test. Those with prior involvement want the test to be great, and won’t keep bad content, but they are less likely to act precipitously.
• While a program may use a standing committee for item writing, it is usually better to take an ad hoc approach. The more item writers (with actual expertise) involved, the better. Recruiting item writers is, in fact, a good way to get volunteers involved; they may later serve in other capacities.
• Activities usually performed every few years, like job analyses and standard-setting meetings,may well involve committee members and seasoned SMEs, but programs will also want to bring in new people, as discussed above.

This is part one of a two part series on Subject Matter Experts.

Categorized in: Industry News