From the Item Bank

The Professional Testing Blog

Test Security Flowcharts

A test security flowchart is a visual protocol that is used to process anomalous (or aberrant) response vectors or incidents, and to do so in a consistent, controlled, unbiased way that contributes to the due diligence of assuring the integrity of resulting scores. By creating a flowchart for each recognized threat to a given program, anomalous or aberrant results can be processed consistently and fairly.

Threats to test security can manifest in many different forms. Ranging from premeditated, intentional cheating to naïve disclosure of content, some commonly recognized threats to test security include item harvesting, copying, use of crib notes or cheat sheets, proxy testing, exploitation of time zone differences, use of brain dump sites, and the unauthorized sharing of test content.

Indications that inappropriate behavior may have occurred vary by delivery mode of the exam. For example, in a computer-based test, response time vectors that contain a large number of exceptionally low response times and are peppered with a handful of anomalously high response times may be indicative of item harvesting. On a paper-based test, the presence of a high number of wrong-to-right erasures may be indicative of copying or collusion. Another indicator that inappropriate behavior may have occurred is the presence of anomalously low exam scores, particularly when the test has high stakes associated with it. And for repeat examinees, an unusually large score gain over a short period of time raises a flag that inappropriate behavior may have occurred.

To protect from these kinds of threats and misbehaviors, test publishers (developers, sponsors and/or users) need to have an over-arching test security plan. Such plans consist of fair and defensible policies that strike a balance between responsibilities and rights for both the test taker and the test publisher. Without this balance, defensibility of the test security plan could be compromised.

Various components of a test security plan should be rooted in and aligned with recognized and accepted standards, such as those outlined in the Standards for Educational and Psychological Testing (2014). These standards serve as compass to test publishers and as such, provide direction and guidance in terms of fairness and ultimately, to a validity argument for score use and interpretation. By incorporating a test security flowchart into a test security plan, we provide evidence that all flagged test takers are processed fairly and that examinees that attain high scores through fraudulent means are not given an unfair advantage. Publishers have an obligation to invest in test security, for to do otherwise may impact fairness (and therefore validity).

The following diagrams depict the basic structure of a test security flowchart and what a flow chart may look like when proxy testing is suspected.

Note the inclusion of a validity panel in the basic structure of a test security flowchart. A validity panel is a standing committee of internal experts and stakeholders who are tasked to (1) examine the case, including evidence, context, timeline, and circumstances, (2) consider the risks of acting vs. of not acting, and (3) select from a menu of options of possible actions. Given the important role of a validity panel, it is important to proactively specify and document the committee’s composition, role, and scope of purpose. Membership may include a psychometrician, a test security manager, legal counsel, a member of management, a member of the client organization, and/or a content representative. The role of committee may be direct, such as a decision-making body, or passive, such as an advisory board. The scope of responsibilities may be broad or limited to specific duties, such as to review cases of large score differences, low exam scores, indications of proxy testing, and/or incidents at a test center.

For a test security flowchart to be relevant, it must align with specific risks or threats. Prior to testing, examinees should be made aware of what constitutes unacceptable behavior and what the consequences are of engaging in that behavior. Test publishers not only have the right but they have the responsibility to protect the integrity of the exam and the scores that result. That said, test developers/administrators have the responsibility of eliminating opportunities for cheating.

If/when incidents do occur, the use of test security flowcharts can standardize the review process. During the registration, administration, and reporting process, test publishers and test takers enter into a contract. Conditions and requirements of that contract are conveyed through proper messaging, such as candidate bulletins and mandated nondisclosure forms. This messaging defines the roles and expectations of test takers and test users and is integral to the strength and enforceability of a testing contract. Central to the defensibility of a test security plan is evidence that the review process is fair and free of bias. Test security flowcharts provide such evidence.

In conclusion, test security flowcharts are an important part of a comprehensive test security plan. Their use provides a degree of consistency that ensures fairness. Test security flowcharts, when designed and applied correctly, can contribute to the validity argument for any testing program.

Categorized in: Industry News