Statistical Analysis
Statistical analysis helps even the best exams
Once the exam has been administered to the first group of at least 30-50 candidates,
certain basic statistics should be analyzed before providing the score results to
the examinees. This step is vital to ensure that there are no unintended consequences
with any of the questions. Below are several reasons to review the exam’s statistical
performance.
One reason for reviewing exam statistics is to improve the performance of each question.
Even the most thorough, knowledgeable, and experienced exam development panel will
create questions requiring revision after reviewing the examinee performance. For
example, some questions may be so difficult that no examinee could correctly answer
it. In addition, a question might be so easy that everyone gets the question correct.
Perhaps no examinee is choosing option "D" for a question, indicating that it is
the easiest option to eliminate. Finally, a question might be answered correctly
by all of the poor performing examinees, but not be the high performers. All of
these cases may represent a problem with a question that would only be detected
after viewing the statistical performance.
Another reason to review the statistical performance of an exam is to prevent bias
against a particular group of examinees. There are many laws which prohibit discrimination
against individuals based on race, gender, creed, age, or familiar status. Therefore,
it is important to review the demographic characteristics of those that pass or
fail the examination to ensure that it does not unfairly discriminate based on these
factors.
A third reason to review exam statistics is to understand changes in the relevance
of topics in the industry or in educational preparation. Consider the situation
where a question, which has performed well for the past 3 years, has suddenly shown
to be too difficult. One potential reason for this might be that the topic being
tested is either not being taught in courses or is no longer used in the industry.
If this case, this would suggest that the question be revised or removed from the
exam.
Following this eight-step process is the best way to ensure the highest quality
results. Although it is possible to create an exam by eliminating some of these
steps, it is not advisable to do so. Considering the amount the money spent to administer
and maintain the exam, it does not make sense to cut corners during the development
process. Besides, the cost of one legal challenge of a poorly developed examination
will far exceed the amount spent on proper methodology.
Back to Overview
How frequently should you analyze statistics?
To ensure an examination is performing properly, the statistical properties of each
question should be evaluated frequently. At a minimum, the statistical performance
of an exam needs to be reviewed every 200 examinees or quarterly whichever occurs
first.
In addition, an annual review of the content relevance and passing standard should
be conducted to ensure the exam is meeting the needs of the organization.
Version 3 of ExamDeveloper will contain the tools needed to do the statistical analysis
online and provide immediate feedback of the results. Currently, the Exam Design
staff can provide a statistical report including questions to improve or replace,
usually within three business days.