From the Item Bank

The Professional Testing Blog

Testing Essentials: Scoring QC

Reed Castle, Ph.D.

The primary goal of any testing program is to treat test takers in a fair and equitable manner and to assure that inferences made from an exam score are appropriate. To do so, one component is to maintain proper scoring and reporting procedures. For those failing, diagnostic score reports should be provided in a meaningful and reliable manner. The presentation covers Scoring QC and reporting scores in a reliable manner.

There is a wonderful journal article by Avi Allalouf that discusses QC procedures for scoring. In it, he outlines why scoring mistakes can happen and creates six broad areas. They are:

1.      No standards have been established. In this category, one could have people conducting the same scoring activity but doing it their own way or making judgments inconsistent with another person conducting the same job. To reduce the probability of this happening, scoring standards need to be established.

2.      Practices are undocumented.  Standards can vary from examination to examination or administration to administration. Having documentation for each program is critical. The documentation should be comprehensive so different people scoring should be able to replicate others’ work.

3.      Standards are not followed. The author wisely chooses the term “overconfidence” to describe this type of mistake.  Reviewing and applying the standards at each scoring opportunity will help reduce this mistake from occurring. Being comfortable doing it isn’t good enough.

4.      Standards are not updated. Standards need to be updated to prevent potential mistakes. An example could be a change in the number of scored items or applying multiple keys to an item.

5.      Scheduling and financial pressures. Anyone who has taken an exam realizes the urgency in reporting scores. Testing agencies have a responsibility to get accurate scores back to organizations in a reasonable amount of time. Better to be late and accurate than to be early and inaccurate.

6.      Responsibility.  Who is ultimately responsible for the scores? If multiple people are part of the process, there needs to be an independent audit of the work.

Documented standards that are routinely updated help reduce the likelihood of a QC mistake. A chain of command should be in place to help assure multiple reviews of the scoring process. Prior to reporting scores, data forensics should be conducted to identify any cheating (copying) or trend of item drift, which may suggest compromised items.

Score Reporting

Once data forensics have been applied and the scores are ready, a few decisions need to be made. They include:

1.      Raw versus scaled scores

2.      Mail versus electronic score notification

3.      For computer-based exams, should scores be instant?

4.      Should diagnostic scores be reported? My colleagues do a wonderful job of discussing this here and here.

The scoring process is a critical step in the examination process. The goal of any testing program is to assure that test inferences are appropriate and fair. 

Categorized in: Industry News