From the Item Bank

The Professional Testing Blog

5 Misconceptions about Psychometricians 

If you’ve ever taken a standardized test or pursued a professional certification or license, there’s a good chance a psychometrician played a key role in developing it. Despite their significant impact across various professions and industries, psychometricians often face misconceptions due to the specialized, and sometimes misunderstood, nature of their work. Many people might not be familiar with psychometrics, or the role psychometricians play. To clarify what psychometricians actually do, I’ll address and correct some common misconceptions about their day-to-day work. 

1. Psychometricians create test content.

Psychometricians do not create test content. In fact, they typically have limited knowledge of it. Although psychometricians might gain some understanding through collaboration with subject matter experts (SMEs), it is highly unlikely they could pass the very exams they develop. While familiarity with test content can be advantageous when working with stakeholders, it is ultimately the experts’ responsibility to determine what test takers (or candidates) ought to know.  

Psychometricians concentrate on the design, development, analysis, and validation of assessments to ensure they accurately measure their intended objectives. This involves analyzing test data, conducting item (test question) analyses to ensure questions are fair and unbiased, and performing validity studies to confirm the test measures the right constructs. 

2. Psychometricians are just statisticians.

Psychometricians aren’t just statisticians. An often-overlooked aspect of their work is the interactions that take place with subject matter experts (SMEs). Key activities, such as item development, passing score studies, and job analysis meetings require active participation from psychometricians. They may not be experts in fields like echocardiography, landscape architecture, or audio system installation, but psychometricians play a consequential role in guiding those who are.  

For example, during a job analysis meeting for a professional examination in landscape architecture, a psychometrician facilitates discussions with SMEs to identify critical tasks and knowledge areas required of competent landscape architects. Often, SMEs expect candidates to know everything experts do, but the exam’s goal is to assess if candidates meet the required competence for the role (hence why its incredibly rare for an exam to have a 100% passing rate). In such instances, psychometricians need to balance the discussion, uncover the exam’s core purpose, and create an image of what a successful candidate looks like. The psychometrician is always focused on validity and reliability, so examinations are trustworthy and meaningful. While the psychometrician’s job requires an understanding of test development principles, validation processes, and fairness, it also demands sound judgment and common sense. 

3. All Psychometricians do the same type of work.

Psychometricians don’t all do the same type of work. The field is varied, with some focusing on different theories and methods for developing tests, like item response theory or classical test theory, and others working on specific types of exams, such as educational, psychological, or professional certification exams. Psychometricians can be found in both private and public sectors, including working with licensure and certification examinations, psychological assessments, educational evaluations and employee selection assessments. What almost all psychometricians have in common is that their work requires knowing about human behavior, how people learn, and the test-taking experience to create assessments that are accurate and useful. 

4. Psychometricians rely solely on statistics to identify bias.

This statement is misleading. Like any collaborative work setting where the work itself has a widespread impact, bias can originate from multiple sources. A significant part of a psychometrician’s role is to identify, analyze, and mitigate biases. While this may include statistical analyses, like differential item functioning (DIF), statistics are not the only indicator of bias. 

For example, language used in test questions can inadvertently favor certain groups over others, leading to unfair advantages or disadvantages. The cultural context of test-takers can also influence their understanding and responses to test items, as individuals from different backgrounds may interpret questions differently based on their unique experiences. Additionally, the interpretation of test results can be skewed by preconceived notions or expectations. While statistics are a useful tool, they can’t automatically eliminate bias without careful application and consideration.  

5. Psychometricians’ work ends once the test is created.

Creating a test is just the beginning. Psychometricians continually review and update assessments to ensure they remain relevant and fair in light of new data, advancements in the field, and feedback from stakeholders, including the test-takers themselves. For instance, after each test administration, psychometricians analyze item performance data to identify any questions that may be functioning poorly or unfairly. In collaboration with SMEs, psychometricians maintain and expand extensive item banks, which can contain thousands of questions ready for future exam administrations. These item banks are curated to ensure a wide range of difficulty levels and content areas are covered, supporting the development of balanced and comprehensive assessments. 

Categorized in: Industry News