Test Topics
Step 2. Conduct the job analysis
Introduction
A job analysis is conducted in order to identify the knowledge, skills, and abilities that an individual in a particular job or occupation ought to have. Within a test development effort, a well-conducted job analysis helps provide for the validity of the test that is later developed. The job analysis contributes to test validity by ensuring that the critical aspects of the job become the domain of content that the test measures. Other, highly similar activities that can be used to help promote the validity of a test are task analyses, practice analyses, and role delineation studies. A typical job analysis occurs in two phases. In the first phase a list of job-related tasks is generated and refined, while in the second phase a validation of that list of tasks is conducted.
The Two Phases of Job Analysis
Task List Generation and Refinement
A broad set of methods may be used to generate an initial list of job-related tasks and activities. These methods can include a review of written materials relevant to the occupation, direct observation or personal interviews of individuals in the occupation, and other approaches such as log diaries or surveys. In a given job analysis, one or more of these methods may be used to help generate an appropriately broad job description or body of knowledge about the occupation.
When this list has been generated an expert panel is convened to review and refine it. The expert panel might consist of anywhere from six to fifteen subject matter experts (SMEs). These are usually people who are either currently working in the occupation being analyzed or they are supervisors of such individuals. The panel reviews the initial list of occupational tasks that was generated, in order to verify that the list is an accurate summary of the important job tasks. The panel also confirms that the tasks are phrased in clear terms and that the scope of the set of tasks is as complete as possible. The panel may make modifications to the tasks that are included in the list, or to the ways in which the tasks are described. Depending upon the complexity of the job or profession the expert panel may hold a one day meeting, or they may need to meet for several days. When the list of job-related tasks has been fully reviewed by the panel, you are ready to conduct a validation of the list.
Task List Validation
In the second phase of the job analysis, the list of job-related tasks is evaluated by a larger group of SMEs. In many cases, a validation survey is used to accomplish this. The survey is often sent out by mail; however, if the individuals working in the occupation are likely to have internet access, then the survey may be distributed over the internet. Proper sampling methods should be followed to obtain the sample of individuals who are asked to respond to the survey. The sample should be sufficiently large and should be representative of the occupation overall, depending on the numbers and variability of people in the occupation.
Typically, the individuals receiving the survey are asked to indicate whether or not they agree that each activity listed is actually used in that occupation. They are also often asked to indicate whether the listed activity is important to the job and how frequently the activity needs to be conducted as part of the job.
Once survey results have been obtained from the representative sample of SMEs, the data are analyzed. The results are used to identify those elements of the job or occupation that ought to be measured on a test of job-related content and competencies. That critical occupational content is then built into the exam program's test specifications.
Summary
In test development, the primary purpose of a job analysis is to provide a strong, clear link between the content and competencies necessary for successful performance on the job and the test item content. A typical job analysis usually includes the face-to-face meeting of a small focus group to produce a list of job-related tasks and the surveying of a much larger group to validate that task list. The results of the job analysis are then incorporated into the test specifications which will guide the development of the test items.