From the Item Bank

The Professional Testing Blog

How to Determine Successful Performance on an Exam: A Case Study

In 2011, the Council of Landscape Architectural Registration Boards (CLARB) wanted to know what factors would lead to successful performance on the Landscape Architect Registration Exam (L.A.R.E.).  At the time, the L.A.R.E. was a five-part exam consisting of three multiple-choice question exams and two hand-drawing practical exams.  The practical exams were administered twice per year for one day in a handful of locations, creating barriers to obtaining a license in landscape architecture.  Since the L.A.R.E. is a licensure exam regulated by each state, decisions on eligibility requirements were made at the state level and often varied by state.  For example, the number of years of practice required to sit for the L.A.R.E. is different from state to state.  If CLARB could identify which factors lead to successful performance on each exam, they could provide insights to both exam candidates and State Regulatory Boards.

The “Determinants of Success” study consisted of three phases.  During the first phase, a research team interviewed members of three stakeholder groups via semi-structured phone interviews to determine what factors could lead to successful performance on the L.A.R.E.  In the second phase, data collected from phone interviews was turned into an online survey, which was sent to candidates after each administration of the L.A.R.E. for one year.  The third phase of the study included a statistical analysis whereby data collected from the online survey was analyzed to determine what factors best predict successful performance on the L.A.R.E.

The phone interviews took place over the course of a few weeks.  Three stakeholder groups were interviewed – employers of exam candidates, educators teaching content found on the L.A.R.E., and exam candidates.  Each interviewee was asked to describe specific factors that they believe influenced successful performance on the L.A.R.E.  The semi-structured interview protocol had series of prompts separated into six factors: testing environment, formal education, work experience, preparation, demographics, and candidate skills.  Overall, six employers, five educators, and four exam candidates were interviewed for a total of 15 interviews.

Utilizing the data from the phone interviews, an online survey was developed and administered to exam candidates.  The survey was administered to L.A.R.E. candidates following the completion of four separate examinations (two administrations of the multiple choice sections of the L.A.R.E. and two administrations of the hand-drawing practical sections of the L.A.R.E.).  It was critical that the surveys were administered after the exam candidates had taken the L.A.R.E., but before they received their scores on the exam to avoid pass/fail status influencing responses.  Like the phone interviews, survey items were separated into six factors: testing environment, formal education, work experience, preparation, demographics, and candidate skills.

To determine which factors, as established in the first phase of the study, predicted success on each of the five sections of the L.A.R.E., a statistical analysis was conducted using both multiple linear regression and logistic regression.  In both cases, the same predictor variables were entered into the statistical model, but with multiple linear regression the outcome variable was candidate exam score and with logistic regression the dependent variable was pass/fail status on the exam.  For the purposes of this study, each of the five sections of the L.A.R.E. were analyzed separately using five different factor models:

• testing environment
• formal education
• work experience
• preparation
• candidate skill

The demographic factors were incorporated into each of the five factor models listed above.

In total, 50 different statistical models were analyzed.  For each of the five sections of the L.A.R.E., all five factor models were analyzed using both multiple linear and logistic regression.  In addition to the statistical models, descriptive statistics were examined and reported for all continuous and categorical variables.  Specifically, means and standard deviations were reported for all continuous variables and frequencies for all categorical variables.  Results were reported by exam and factor model.

The key findings of the study were that education and work experience factors had the greatest impact on candidates’ performance on the L.A.R.E.  Preparation and skills factors contributed somewhat to successful performance on some of the exams, while testing environment factors had little to no impact on candidates’ performance.  Another finding was an inverse relationship between years of experience and performance on the multiple-choice sections of the L.A.R.E.  As such, CLARB recommended that candidates sit for the multiple-choice sections of the exam immediately following graduation (regardless of state experience requirements for licensure).  For a more detailed explanation of the findings, please see the published executive summary Determinants of Success Study.

Categorized in: Licensing Boards