From the Item Bank

The Professional Testing Blog

The Multiple Response – Why, When, and How to Use this Item Type

Overview

The multiple response (MR) item type is, hands-down, one of the most useful alternative item types available today.  It isn’t a new item type; the MR been used in paper-based assessments for many years (under various names, including multiple select, multiple-answer, multiple-multiple-choice, etc.). Now that the MR item type is provided as a standard option within many CBT applications, it is becoming a more popular option.

When to Use

The MR item type has a very wide range of appropriate uses. In fact, it’s probably true to say that most exams include topic areas where items with multiple keys are relevant. These are topic areas with elements that naturally cluster together. In medical assessments, for example, the MR type can be used for diagnosis-related items in which multiple factors need to be considered or for treatment-related items when multiple types of care may be necessary.

The MR can also be used to replace the Type K, or complex multiple choice item type. In the Type K approach a set of options is displayed, followed by choices such as: I only, I and III, II and III, etc. The Type K approach has a cumbersome interface which can be confusing to some candidates, while giving other candidates a test-wiseness advantage.

Once item writers are taught how to work with this AIT they often really value it. This is partially due to the frequent item writing challenge of coming up with a full set of plausible distractors. In some cases it can be difficult, or even artificial, to arrive at three distractors; writing an item with a second or third key can be a better reflection of the content in that area. This improvement in fidelity, as well as opportunity to reduce the number of distractors needed, make the MR a desirable item type.

Issues to Consider

The MR item type is built into many popular CBT software programs, which simplifies some aspects of its use on an exam. However, every exam program uses a series of applications in the testing cycle, from creation of the initial item, through item banking, test publishing, test delivery, item analysis, and score reporting. Adjustments in some of these applications are likely to be needed when the MR item type is first implemented on an exam. Test program staff may need to revise the existing scoring methods in order to accommodate multiple keys, and new approaches to distractor analysis will be required.

Decisions about specific ways to implement the MR item type will be needed early in the process. There are two primary approaches to MR items that can be used – the MR item type can either be administered to candidates through instructions that state “Select all that apply…,” or that specify a required number of responses (e.g. “Select two of the following…”). Several approaches to scoring MR items can also be used. The simplest of these is to use familiar dichotomous (right/wrong) scoring, and to “weight” each MR item with the same value as each MC item (e.g., 1-point). Alternatively, there are several partial credit scoring models that can be used with MR items, and the items can potentially be weighted more heavily than the MC items. These aspects of how the MR will be implemented are tied together. For example, some scoring models are only appropriate for MR items administered with instructions to “Select all that apply….”

Some of these decisions should appropriately be specified as item writing guidelines for the MR. If the MR items will be scored dichotomously and will only be worth 1-point on a total test score, that is likely to mean that item writers should be given some limits for the overall scope of the item, e.g., “a maximum of 6 response options”. Otherwise, a MR item can require that too much knowledge or time from the candidate to be worth the impact on his or her test score. (More information about item writing guidelines for AITs is available here).

Even with relatively simple MR items, with no more than 5 or 6 options, and no more than 2 or 3 distractors, this item type can be somewhat more difficult than typical MC items. This potential difference across item types is often fairly small and since the proportion of MR items on exam is typically modest, the impact on total test score is usually negligible. Nevertheless, it is something that staff psychometricians will want to track, so that information can be reported to exam item writers and to candidates.

Summary

The MR item type doesn’t visually look as exciting as some of the AITs. Nevertheless, it has high utility for many content areas and a proven track record of successful use within numerous exams. It is an item type well worth considering.

 This post is part of the series “Alternative Item Types.”

Categorized in: Industry News, Item Type