From the Item Bank

The Professional Testing Blog

What is a Test Security Plan?

What is a Test Security Plan?

A test security plan is a comprehensive collection of policies, procedures, and documents that outline and guide actions related to test security. From the development of an exam’s blueprint to the reporting of scores, test security touches nearly every aspect of test development and the testing process. Trust is a prerequisite to interpreting scores, and without evidence to support the integrity and veracity of scores, trust is compromised. Without trust, credibility is compromised.

What does it take to protect a testing program so that scores are trustworthy and interpretable? At minimum, it takes having a solid test security plan in place.

Most policies and procedures within a test security plan are based on common sense. For example, clear, unambiguous messaging to candidates is a must. How else can examinees be expected to follow the rules if those rules, along with corresponding consequences, are not spelled out? Policies need to make sense, fit well within a given testing program, be enforceable, and be legally defensible. Corresponding procedures should align with threats that are specific to your programs and be customized to meet your needs. For example, copying between examinees may be a problem in one setting whereas proxy testing may be a more realistic threat in another. Threats vary from program to program, and so should plans to protect programs from those threats.

The cornerstone of any test security plan is the test security agreement, sometimes referred to as a non-disclosure agreement (NDA). NDAs constitute a unilateral, legally binding contract between a test developer (or content owner) and at least one other party that specifies what can and cannot be disclosed without expressed written permission of the test developer (or content owner). NDAs typically outline what information or material is considered to be confidential and/or proprietary, what the time limit for the confidentiality is, and what the consequences of violating the agreement are.

It is common practice to require everyone that has access to test content (questions), scoring rubrics, or other confidential/proprietary content to sign an NDA. This includes subject-matter experts that contribute to the development of the exam, proctors that monitor the delivery of the exam, staff or personnel that handle test materials and/or results, teachers and/or administrators that receive or store test materials, and of course, examinees. For SMEs and staff, NDAs should be updated annually and kept on file for whatever time is specified in the test security plan, which is typically at least three years. Examinees should sign a fresh NDA for each administration of every test they take. And just as with SMEs and staff, examinee’s NDAs should be stored in accordance with policies specified in the test security plan.

Another major component of a test security plan is a chain-of-custody (CoC) procedure. This procedure specifies how secure materials are distributed, collected, and returned. It also lists names and titles of the individuals responsible for carrying out these procedures, as well as procedures to ensure compliance with approved test accommodations, such as extended time or reduced-distraction environments. Typically, dates, locations, and signatures from every person involved in the CoC are collected and archived as part of the test history.

Test Day Policies

Test day policies are another major consideration. Is the test environment secure? Are the proctors adequately trained on test security? What are the check-in requirements? What constitutes a valid form of identification? How many forms of identification are needed? Are test sessions videotaped? Is a seating chart used? Is there a secure place for examinees to store personal items, such as cell phones and study materials? Are calculators supplied by a test vendor? Is scratch paper provided? If so, is it customized for each examinee? Is it collected at the end of the test session? Is a personal white board (and marker) available in lieu of scratch paper? Are screen protectors used over computer monitors? Are seats or work stations separated? Are all electronic devices prohibited from the test environment? Are breaks allowed during the test session, and if so, are examinees observed during breaks? What are the check-out requirements?

Messaging to candidates begins well in advance of the test date and continues through to when scores are reported. Rules, as well as what constitutes a violation of those rules, must be clearly stated and disseminated to candidates and stake holders. In addition, consequences of breaking the rules must also be clearly stated and disseminated. This information may be included in a candidate bulletin, registration materials, marketing materials, and public announcements. On test day, messaging can be presented to examinees upon check-in as well as a preamble at the beginning of the test. Prior to testing, examinees can be required to acknowledge that they have read, understood, and agreed to abide by non-disclosure requirements. Upon submission of a completed test sessions, examinees should be reminded of their agreement to the non-disclosure requirements, as well as the possible consequences of violating their agreement.

Common Threats

Threats to test security vary from program to program. Paper-based exams may be more prone to answer copying than computer-based exams, whereas computer-based exam session may be more prone to use of unauthorized resources than paper-based exams. Test security policies and procedures should be customized to fit a given exam program.

Common threats to be aware of are proxy testing, item harvesting, pre-knowledge, and various forms of collusion. Proxy testing, sometimes referred to as imposter testing, occurs when someone other than the intended, approved examinee (or candidate) takes a test. Proxy testing can be avoided by adhering to high standards for identity verification.

Item harvesting occurs when an examinee memorizes or uses technology to capture test content. This behavior can be difficult to detect during a test session. The result of this form of cheating can result in serious consequences for an exam program.

Pre-knowledge occurs when an examinee gains access test questions prior to their scheduled test session. Unauthorized access to test questions yields an unfair advantage over other examinees because it artificially inflates scores; this may compromise confidence in the other, non-inflated scores.

Collusion is a catch-all phrase that means that a proctor or test administrator has provided unauthorized help to an examinee or has in some way tampered with data or a test session. An example of collusion is when a proctor allows an examinee to deviate from approved testing procedures, to access unauthorized resources, or to have unapproved additional time. Collusion may also refer to the tampering of submitted test records, such as changing examinee responses from wrong to right or adding missing responses.

Post-administration

Data forensics is a popular catch phrase that is used to refer to a host of statistical and psychometric analyses.

There should be a procedure for reporting irregularities in test administration and/or test security. From an anonymous tip line to an identifiable formal report, examinees, proctors, and other testing personnel should have a mechanism for reporting incidents, anomalies, or perceived breaches. The test security plan should specify how such reports will be registered, processed and investigated. And the test security plan should outline how invalidation of scores will be handled, such as with test security flowcharts.

Ongoing Activities

Just as there are actions that should be taken prior to or during a test’s administration, there also are certain activities that need to be ongoing, such as monitoring social media sites, updating legal policies and procedures, performing psychometric analyses (such as drift analyses), monitoring test sites, and conducting test security training.

Monitoring social media may include conducting web searches for harvested questions, monitoring blogs, and keeping an eye on test prep companies. The monitoring of test sites can be a scheduled event or may be done without notice. One type of anonymous monitoring is referred to as “secret shopping.” This type of monitoring usually consists of a subject matter expert or other qualified staff member to register to take an exam, report to the test session, and take the test. This form of monitoring provides valuable insight from the examinee’s perspective.

Legal counsel plays a very important role in the development and maintenance of a test security plan. Counsel provides updates regarding litigation outcomes, templates for NDAs and high-stakes letters to candidates. Counsel also initiates cease and desist orders, provides guidance on copyright laws, and can serve as a proxy member on a validity committee.

Last but not least, the issue of training needs to be addressed. Everyone involved in a testing program should be trained on test security. Training may address a variety of topics, including but not limited to the proper handling of test materials, establishing or maintaining a secure testing environment, critical aspects of a confidentiality agreement, rights and responsibilities of examinees, acceptable proctoring practices, and chain-of-custody guidelines. Training may also include what-if scenarios and how to discontinue an in-progress examination.

Training should be customized to align with the roles of various team members, including subject matter experts, proctors, test administrators, test coordinators, content development staff, psychometricians, and management. Consultants and third party vendors also need to be trained, whether by your organization or by their own.

Test security is a team effort. Everyone plays a role in maintaining a secure defense system for your exam.

Categorized in: Industry News