From the Item Bank

The Professional Testing Blog

Careful Selection of Subject Matter Experts is the Key to a Successful JTA Meeting

A job/task analysis is (JTA) is a very close look at a job, portion of a job, or concept job in order to identify the tasks and associated knowledge, skills and abilities (KSAs) required to competently perform the job.  A JTA is the foundation for the development of a certification program or a certificate program.  Only after the KSAs have been identified can a certification body or educational program determine how to measure competence of the KSAs or how to teach the KSAs.  There are multiple methods for conducting a JTA including using focus groups, structured interviews, job shadowing and observations, and surveying practitioners.  When conducting a JTA as the basis for a certification program, the method used is not as important as making sure the right subject-matter-experts (SMEs) participate in the process.

Professional Testing frequently utilizes a modified DACUM approach.  DACUM stands for Developing A CurriculUM and is a structured, consensus-based focus group process where a group of expert practitioners, led by a trained facilitator, identify tasks associated with a job over the course of three days.  Professional Testing uses a modified version of the DACUM process that identifies the knowledge and skills associated with each task at the same meeting.  This process has been referred to as “Turbo DACUM” by some users, including Ferris State University and the National Occupational Testing Institute (NOCTI).

The key to a successful DACUM JTA meeting is predicated upon the careful selection of a representative sample of SMEs.  Failure to select the right SMEs for the meeting can result in a failure of the JTA to identify the right tasks and associated KSAs, or a lack of consensus.  Several dimensions should be considered by certification bodies or organizations selecting SMEs to participate in a JTA.  These dimensions include number of SMEs, expertise of SMEs, occupation of SMEs, years of experience of SMEs, and personality of SMEs.  Each of these will be further explored.

The number of SMEs selected to serve on a JTA panel is important.  The optimum number is somewhere between 10 and 15 SMEs with a minimum of about 7 or 8, and a maximum of about 17 or 18.  Size considerations should be based on the smallest number of SMEs that can adequately represent the depth and breadth of the job being defined.  Less than about 7 or 8 and the panel probably will not be able to sufficiently identify all of the tasks and associated KSAs with doing the job.  More than about 17 or 18 and the group sometimes becomes unwieldy.  I have successfully facilitated a JTA meeting with 22 members, but nearly failed completing a JTA meeting with 30 SMEs.  The JTA with 30 SMEs proceeded very slowly as each SMEs wanted to voice his or her opinion and it took much more effort to reach consensus with so many varying opinions in the room.  This was a case of “too much of a good thing” was in fact not good.

The careful consideration of the expertise of SMEs is also critical when selecting a JTA panel.  Having the world’s best expert in a particular field can be good, or it can be bad.  It is always good to have tremendous expertise on a JTA panel, but sometimes the world’s best expert can take control of the meeting while the remainder of the SMEs sit back and say nothing.  I have also facilitated a meeting where two of the world’s best experts disagreed and had to leave the room to work it out among themselves before the meeting could progress.  Rather than selecting the world’s best expert, what is most important is that the SMEs represent the depth and breadth of expertise.  But if famous, experts are selected, the facilitator needs to be strong enough to ensure that the expert doesn’t dominate the meeting and all of the SMEs have an opportunity to contribute equally.

Normally SMEs seated for a JTA are practicing experts.  That means they currently or recently performed the job.  However, there are situations where others can contribute to the JTA.  For example, sometimes trainers/educators can offer perspectives that practitioners don’t think about.  And often researchers and regulators bring another dimension to a JTA focus group, but the majority of the SMEs should be practitioners.

All experts selected for a JTA focus group should be familiar with the job and competent on the job.  However, a competent new worker may have a different perspective of the job, the tasks and the associated KSAs than a worker who has 25 years of experience on the job.  Additionally, a newer worker may be more familiar with trends and technologies, so it’s important to be sure the JTA focus group is comprised of a cross-section of job incumbents.

Finally, JTA focus groups can be derailed by SMEs with ulterior motives or challenging personalities!  I have facilitated a meeting where several SMEs participated only because they wanted the process to fail.  In another case, I have facilitated a JTA where a SME was only interested in “stealing” the results of the JTA for his own use.  Thus it is important that each SME is carefully selected to be sure they are participating for the right reasons.  And it is important that a strong and experienced facilitator be present to hold back domineering SMEs and encourage the participation of quiet SMEs.

Certification bodies and organizations selecting experts for a JTA focus group should thoughtfully select each participant based on some of the dimensions listed above.  Testing companies such as Professional Testing and expert, trained JTA facilitators can assist organizations in seating the best panel possible for the JTA.

Categorized in: Industry News