Login

800-213-1673

 

• Overview
• Job Analysis
• Question Writing
• Question Review
• Question Validation
• Exam Building
• Standard Setting
• Exam Administration
• Statistical Analysis

 

Question Validation

---

Validation provides documentation and rating of question quality

After a panel of experts has approved a set of questions for use on the exam, the validation process allows the experts to rate the questions on scales such as importance, criticality, and frequency of use for competent performance on the job. It is important to quantify and document the quality of the questions for several reasons.

First, question validation provides concrete documentation of the relevance of all questions placed on the exam. This becomes key evidence to help defend against challenges about the exam content. In essence, it provides documentation that a panel of subject matter experts convened, discussed the relevance of every item, and agreed that they should be placed on the exam.

Another use of the validation process is in the assembly of the examination. If the experts developed more questions for a blueprint area than necessary, it is important to quantify the quality of a question to allow selection of only the best questions. For this reason, the ratings of the panelists are combined using a statistical formula and translated into a 5-star scale. This way, the questions with five stars can be selected in place of those only receiving one or two stars.

In fact, ExamDeveloper has the capability of automatically building a draft exam using item quality and other criteria such as blueprint, cognitive complexity level, item difficulty, and category type. Once the manager specifies the number of versions to be built  and the thresholds for the criteria, the draft exam versions are immediately available for viewing by the expert panel. No other exam development software comes close to the capability contained in ExamDeveloper.

Manually building an exam using multiple criteria is a complex, time consuming, and often error prone task. Using ExamDeveloper to build exams will save your organization dozens of hours and prevent costly mistakes that occur when trying to trying to track everything on paper or spreadsheet.

Next Step

Too many is better than not enough

People frequently ask how many questions each user needs to write in order to have enough for the exam.

A common rule of thumb is to assign 150-200% of the number of questions that the final (completed) exam versions will contain. For example, if final exam will contain 150-questions, it is advisable to assign 150-200 questions over all areas of the blueprint.

A more complicated example is the task of assembling two versions of a 100-item exam that will share 50% of the content between the two versions. In this case, it would be advisable to assign between 225-300 questions (1.5 or 2 times 150 questions, 50 shared questions + 50 unique to each version).

This information is explained fully in a self-paced tutorial found within the application. Of course, the Exam Design consultants would be happy to assist with this or any other question you have about the design or maintenance of your examination program.