From the Item Bank

The Professional Testing Blog

What are Pretest Questions and Why Do We Use Them?

Pretest questions, also referred to as pilot or beta questions, do not count towards candidate’s scores but do provide useful information on the quality and relative difficulty of the questions. For example, a test form that contains 155 questions may include 15 questions that have pretest status and 140 questions that have active status. In this case, candidate’s scores would be based exclusively on the 140 active test questions; the 15 pretest questions would be analyzed for statistical and psychometric performance and would not be used for scoring purposes.

Professional credentialing examination programs (both certification and licensure) commonly pre-test examination questions to vet them prior to operational use. There are two primary methods used to pretest questions. One is to embed the pretest questions on a live exam form; the other is to create a pilot test form, apart from a live examination, that is comprised exclusively of pretest questions.

There are a number of reasons for using pretest questions. These reasons include evaluating the test question prior to its use as an active or operational question, to obtain item statistics (for quality control as well as equating purposes), to screen questions for fairness and general performance, and in some cases, to beta test new types or formats of questions. A few of these reasons are described in more detail, below.

Evaluation of Test Questions – Examination questions are designed to distinguish between candidates who have the knowledge for competent practice and those who do not. A well-performing test question will therefore be able to differentiate between a high scoring (knowledgeable candidate) and a low scoring candidate. It is important that professional credentialing examinations contain high quality questions, not those that everyone gets right or everyone gets wrong. Exceptionally easy/hard questions fail to distinguish between candidates who know the material and those who don’t. For resulting scores to be interpretable and defensible, it is essential that exam forms consist of the best possible questions. By pretesting questions, we harvest data that is used to compute statistical and psychometric indices about quality, fairness, and ultimately, exam reliability and validity.

Obtaining Statistics for Equating – In order to ensure that versions of exam forms are equivalent in difficulty, examination forms need to be equated so that resulting scores are comparable. Equating can be conducted prior to the form’s administration (pre-equating) or after its administration (post-equating). Regardless of timing, equating test forms provides the foundation that is needed to validly compare scores from different versions or different administrations of exam forms. The only way to statistically equate exam forms is to use questions that have known statistical histories. This requires them to have been used at least once on a previous form.

When new questions are developed, they contain no statistical history. One method of collecting statistics for new test questions is to embed them as pretest questions on actual exam forms. Professional credentialing examination programs are fluid with new test questions continually being developed. Therefore it is important to continually pretest exam questions to ensure a large item bank of high quality, proven exam questions with known statistics.

Number of Pretest Questions –It is standard practice for pilot questions to constitute up to twenty percent of an exam’s overall length. Vetting of questions through piloting is supported by psychometric standards, such as the Standards for Educational and Psychological Testing (AERA/APA/NCME, 2014) and is used by most organizations involved in high-stakes testing.

Promoting Pretest Exam Questions – Pilot questions that perform within acceptable ranges are promoted to operational (active) status and are used for scoring on future exam forms. Pilot questions that fail to meet these criteria are either edited or removed from future use. This helps to ensure that only the best examination questions are used on the examinations.

Benefits to Certification Bodies – In addition to assuring well-performing questions are on the exam, certification bodies benefit from pretesting questions by obtaining item statistics from “live” or “real” candidates—the actual test-taker versus exam committee members or others with subject-matter expertise. Pretesting items is also a way to maximize resources by eliminating any “special” examination administrations for statistic gathering purposes.

In summary, questions are pretested to ensure a fair examination process both in the moment as well as over time. Only those questions that exhibit “good” item statistics are promoted to “scored” (active) status. Without a process for pretesting questions, examination programs could jeopardize one or more of the key steps in the validation strategy of an examination.

Categorized in: Test Development