From the Item Bank

The Professional Testing Blog

Transitioning an Exam Program in the Modern Age

Transitioning an exam program from paper to computer-based administrations may appear to be a very straightforward process but there are a variety of variables to consider when a sponsor is making this decision.

Computers
Taking an exam on a computer is an unfamiliar medium for me. There is some irony in that many of us developing these services have never actually taken a high stakes exam on a computer.

With that said, a process needs to be implemented to give candidates the opportunity to become comfortable with the registration and testing process. This could include providing a practice test online so examinees become familiar with the navigation and features of the delivery system. Another alternative would be posting a video online documenting the candidate experience. Regardless of tutorial methodology, it is important that candidates can experience the examination process and become familiar with the system prior to taking the exam. A short list of items to cover could be:

1.       What to expect when one arrives at the testing center
2.       What type(s) of Identification are required?
3.       What pre-exam information is required prior to taking the exam (e.g., NDAs, surveys)?
4.       How will the candidate be made aware of testing time?
5.       What resources are (or are not) available to the candidate (e.g., calculators, reference materials)?
6.       How are breaks handled during the exam?
7.       Can a candidate flag an item to return to later in the exam?
8.       How will scores be provided (e.g., instant, provisional, or at a later date)?

A Few Psychometric Considerations
Some research suggests that simply changing the order in which items appear on the exam can actually influence item difficulty. It is also believed that the sequence in which items appear can have an effect on overall test difficulty. For that reason, consideration must be paid to whether randomizing item options and item order can have an effect on the performance of an item, or the overall difficulty of the exam.

Some programs may consider reducing the test length when moving to computer administrations. Documentation and research should be conducted to determine if candidates taking a shorter computer administered exam will be classified (pass/fail) the same, with no or limited error, when taking the shorter exam.  (For more practical and psychometric information, reference Practical Considerations in Computer-based Testing Parhsall, C., Spray, J., Kalohn, J., and Davy, T. (2002))

Adaptive versus Linear
Adaptive tests are much efficient and can reduce seat time as well as expense in some instances, but require a more robust item bank across all content areas. In addition, adaptive testing will require larger sample sizes to calculate necessary statistics. Linear, on the other hand, will have similar requirements as the paper administration with the exception of the test delivery media.

Cost
It is important for test sponsors to align pricing (either paper or computer administrations) with service or conduct a cost-benefit analysis. In many instances paper administrations require a much longer score reporting time and the number of testing sites is limited. Computer administered test are much more accessible and can have instant score reports. However, computer administration may cost more.

Summary
Transitioning an exam program from paper to computer administrations requires a substantial effort. Consideration should be given to the candidates’ experience and how to educate them on the entire computer administration process in advance. Assuming that the psychometric properties of the paper administration will be the same on the computer can have negative consequences unless an organization has explored how the transition could affect the validity and replicability of the exam scores and their intended inferences.

Categorized in: Industry News