Test Topics
Step 4. Develop the Initial Pool of Items
Introduction
Once the test specifications are complete, the item writing phase of the test development project can begin. Typically, a panel of subject matter experts is assembled to write a set of test items. The panel is assigned to write items according to the content areas and cognitive levels specified in the test blueprint. Items are written for each of the item types identified in the test specifications. By far the most commonly used item type in standardized assessment is the multiple choice item, due to its relative advantages, including its ability to be used to measure at higher cognitive levels. Some exam programs use item specifications to further guide item writers with detailed requirements for each included item type. The total number of items that a particular exam program needs depends on specific aspects of the exam program. After the items have been written, they are stored electronically in an item banking software application.
Elements of Item Writing
Item Types
There are a number of objective item types used with some frequency in standardized paper-and-pencil exam programs. These item types include multiple choice, true-false, matching, and more. Performance-based tasks include essays, demonstrations, in-basket activities, and others. While objective item types can usually be machine-scored, most performance-based tasks require human raters to score the examinees' performances, using scoring rubrics. Computer-based tests can include a wide range of item types, including many novel, objective item types which can be scored by computer. Examples of these include drag-and-drop items, clickable image items, and multiple-response multiple-correct items. Each item type, whether objective, performance-based, or computer-based, has particular advantages and disadvantages that might make it more suitable for certain exam programs than for others.
Multiple Choice Terminology
A labeled example of a multiple choice item is below. Each multiple choice item consists of a stem, which may be in the form of a question or a statement. The stem is followed by a set of (usually four or five) response options. One of these options is the correct answer, or key. The remaining incorrect response options are called distractors.
Advantages of Multiple Choice Items
The multiple choice item type is used more frequently than any other available item type, due to its many comparative advantages. As compared to performance-based tasks, each individual multiple choice item takes very little examinee time. This means that for the same amount of test administration time, a multiple choice exam can have far more items. And the use of more items can provide for a much broader range of content coverage as well as increasing the reliability of the assessment. In addition, unlike essays and other performance-based tasks, multiple choice items can be quickly machine-scored. On the other hand, as compared to other objective item types like the true-false, a four- or five-option multiple choice item reduces guessing. And, unlike the true-false and matching item types, the multiple choice item can more easily be used to measure at higher levels of cognition.
Multiple choice items can also be seen to have certain disadvantages. If you are not careful, your exam program's multiple choice items can be written in a way that measures only low-level cognitive skills or trivial information, in a way that is confusing or unclear, or in a way that makes the correct answer easy to guess. If you elect to use multiple choice items in your assessment, being aware of the potential weaknesses and disadvantages of this item type may help you overcome them.
While the multiple choice item type will not be optimal in all circumstances, on balance its relative set of advantages gives it great utility for a broad range of assessments.
Higher-Order Multiple Choice Items
While there are certain content areas within almost every exam program that are Knowledge-based, it is critical to measure above the Knowledge-level as well. In certification and licensure testing, it is particularly important to measure the examinee's ability in connection with the job-related tasks and skills that were identified in the job analysis. Multiple choice items can obviously be written to assess Knowledge-level understanding of the content. Perhaps less obviously, they can also be used to measure at higher cognitive levels. There are several specific techniques that can be used by item writers to help structure items that target higher-level cognition.
One technique for writing higher-order multiple choice items is to include a content-relevant graph or table in the item stem. The examinees can then be asked a question which requires them to apply job-related principles or procedures as they consider the stimulus material. When responding to this type of item, the examinee must go beyond simple memorization to demonstrate that he or she can apply the knowledge, concepts, and principles related to the content area. A second technique for writing higher-order multiple choice items is to include a written scenario in the item stem. The examinee can then be asked to apply previously learned principles or procedures to the scenario, by selecting the best response option. In a similar approach, the examinee can be asked to predict, based on a known principle in the occupational content area, what would happen in a specific situation. In this type of item the examinee is asked to demonstrate understanding of a principle or its application in a novel setting or situation. Alternatively, examinees may be asked to evaluate which option is the best choice, where "best" is specified by a criterion that is appropriate to job-related concepts, principles, or procedures. Finally, the examinee's problem-solving ability related to the occupation can be measured through a set of items associated with a complex scenario. In this approach, the various items in the set can be written to target specific steps in a job-related, problem-solving process.
Item Specifications
Item specifications provide detailed requirements for each item type included on the test. In addition to specifying the item type to be written, and the content and cognitive areas the item should address, the item specifications will define other important item characteristics. These characteristics may include: sources of the item content, descriptions of the problem stimuli, characteristics of the correct response, and characteristics of the incorrect responses. Item specifications can be thought of as providing structure to an item shell which can then be used to write many similar items. One reason for the use of such items shells is to ensure consistency across items, even when different individuals are writing the items. The items that are written to these detailed specifications may be considered interchangeable; that is, they may be similar enough that an examinee's response to any one of the items might be expected to represent that examinee's response to any of the others. A second reason for the use of item specifications is that they can greatly increase the efficiency of the item writing effort. They can help guide item writers to produce multiple, high quality items relatively quickly. Multiple, interchangeable items can then be appropriately used across the exam program's set of parallel test forms.
Size of the Item Bank
The item bank, or item pool, for an exam program should be large enough to fully address all of the content areas and cognitive levels specified in the test blueprint. In addition, for high-stakes exam programs, the item bank needs to be large enough to support the development of multiple, parallel test forms. In order to ensure that the parallel forms satisfy test blueprint rules, and can be assembled to be statistically similar, additional items will be needed. Finally, some of the items that are written will not survive the item review, field testing, and item analysis evaluation phases. Because some of the items will fail to satisfy the quality goals of your exam program, you need to plan for the development of additional items (somewhere between 25-100% extra), in order to ensure that enough high quality items are available for the operational test forms.
Item Banking Software Applications
An item banking application is a specific type of database program. The item banking programs that are commercially available all have certain database features in common, but each has individual elements as well. The choice of item banking application can depend on identifying the specific needs of the exam program and then finding the software application that best meets those needs. Item banking software applications vary greatly in terms of functionality. You may choose to select an application based on features such as: the number and type of database fields available to store information about the item, the facility with which the application imports and exports item- or test-related data, the statistical features provided by the application, the word processing functionality available, and the graphics formats supported. The application should ideally include extensive querying functionality, so that the item bank can easily be searched for relevant information. Depending upon the needs of your exam program, it may also be useful if the application has the capacity to store items in multiple languages. It may also be worth considering the individual and summary test reports that the item banking application makes available. Finally, in addition to differing in terms of available features and ease of use, item banking applications differ enormously in terms of price.
Summary
The item writing phase of a test development project is critical to the success of the exam program, as high quality items are essential for good measurement. There are a number of specific issues you may want to address as you plan for item writing. For example, it is important to select the optimal item types for your exam program, and to use each item type appropriately. Targeting some of the items to higher cognitive levels is also essential. Item specifications may be of help in guiding the subject matter experts who will write the items. Selection of an appropriate item banking software application will help you in long-term maintenance of the exam program, as will as the development of a sufficient number of items to support all aspects of the exam program.