From the Item Bank

The Professional Testing Blog

Remote Item Development – Benefits and Considerations

In the certification arena, a traditional model of item development consists of representative panels of content experts from various locations gathering for a period of time in a face-to-face meeting to perform specific tasks associated with the development of the examination (e.g., item writing, item review).  At times, the meetings are held in conjunction with an association conference or at a desirable location to serve as an incentive for content expert participation.  The content experts attending the meeting also have the added incentive of getting to network with peers within their profession and discuss commonalities and issues within practice.

While desirable in many ways, time and travel costs to host these meetings can be high.  Meetings have to be planned well in advanced and panel members may have difficulty coordinating schedules to devote multiple days or a week to the meeting.  Also, particularly when meetings are required to occur during winter months, weather and other travel related factors may delay or cancel a meeting which can prove to be even more costly.

With technological advances including the internet, cloud computing, video conferencing, and other telecommunication systems, web conferencing and telecommunication are becoming the main stay with many organizations, a trend that has also penetrated into the certification testing world. Many organizations are integrating at least some components of remote item development into their processes, while others have gone to a completely remote model. Cost savings, flexibility and “just-in-time” development are often cited as reasons for using a remote development model.

Remote item development consists of any aspect of item development that is performed where content experts, test developers, and/or psychometricians are working on the development of the items from physically different locations.  This can involve activities that are performed synchronously (e.g., panel of content experts meeting via a web conference) or asynchronously (content experts independently write items and submit them to a web-based item banking system).  Remote item development can happen at any of the various phases of item development, most often during item writing or item review.

Remote item writing

In remote item writing, item writers are first typically trained on general item writing practices and on the specifics related to the particular examination. They then write and submit items remotely.

Remote item writing training can be conducted via web conference or via pre-recorded presentation.  A web-conference is ideal as item writers can benefit from the questions and discussions presented by others in the group.  However, sometimes schedules and time frames may prove difficult in scheduling a group webinar.  Either way, it is recommended that certification organizations provide interactive components throughout the presentation and a follow up assignment to encourage the participants to remain engaged.

Some organizations take a hybrid approach where training and initial item writing occurs at an introductory face-to-face meeting and then continued via remote item writing.

Items are typically submitted on some sort of template, whether it is a word processing form submitted via a secure FTP, a cloud based site  via an on-line submittal form, or directly into the item bank.  The increase of web-based item banking systems is making the submittal of items directly into the item bank more feasible.  However, the high prices of these systems can be a factor for some organizations; therefore other options are still viable choices.  While secure emailing options (encryption) are available, submitting items by email is generally not recommended.

Not all content experts are suited for remote item writing.  Those who succeed the most tend to be self-starters who are able to work independently within specified guidelines and when given feedback.

Management of remote item writing can vary provided the needs and goals of the program.  Some programs will engage a particular group of item writers for a brief period of time for a one time project; other programs will maintain a standing group of item writers who will provide on-going item writing over a longer period time.  Regardless of the term of the item writers, it is recommended that programs schedule periodic check-in sessions with  item writers (preferably as a group) to allow opportunities for discussion and feedback.  This also helps encourage continued engagement with the project.

Remote Item Review

Remote item review can be done synchronously through webinars, asynchronously where a group of item reviewers each independently review and comment on the same set of items, or by a hybrid approach where item reviewers do an initial filter review independently and then hold a web-conference to discuss items that are shown to be questionable or where the reviewers views differed.

Unlike a face-to-face meeting where item review activities happen over multiple days and where hundreds of items may be reviewed, remote item review will occur in more-frequent shorter-duration sessions with fewer items reviewed in an individual session.  A typical review session may last 2-3 hours vs. 2-3 days.  Facilitation can be the key to the success of these sessions.  Unlike a face-to-face meeting, it is easy for item reviewers on web conference calls to get distracted by competing demands and can more likely tune out (or “mute” out) of the discussion on the call.  The facilitator needs to be particularly skilled in the finesse of keeping each member of the call focused on the topic at hand and each view point adequately expressed.  This can be challenging as facilitators  may be at a disadvantage in not being able to interpret body language when gauging level of agreement from individuals in the group.

Security Considerations

Remote item development brings with it an added layer of security considerations due to the added risk of exposure to items via the internet, or individuals working from varied and remote locations.  Both the Handbook of Test Development (2015) and the Handbook of Test Security (2014) provide discussions related to added security measures to consider when engaging in remote item development or using web-based item banking systems.  Some major topics to consider when planning for remote item writing include:

• Security agreements and non-disclosure forms – care should be taken to be clear in terms of language related to security of the items written or accessed by the item writer in the remote location, both electronically or in physical form
• Access and permission control – if a web-based item banking system is being used, management of role based access and permission control need to be considered
• Secure passwords – considerations include password requirements, encryption methods, and user policies
• Secured and encrypted connections – this includes web conferencing services as well as item banking
• Management of server used for item banking, web conferencing, and/or item submission.

Categorized in: Item Type, Item Writing