Login

800-213-1673

 

• Overview
• Job Analysis
• Question Writing
• Question Review
• Question Validation
• Exam Building
• Standard Setting
• Exam Administration
• Statistical Analysis

 

Question Writing

---

Learning to write more complex questions will have a big impact on exam quality

Many people believe that there serious limitations on what can be tested using multiple-choice questions. Comments include: "how hard can a question be when the correct answer is right in front of the person?" and "it is impossible to test a person on high-level decision making using multiple-choice questions.

People who subscribe to this belief have not been properly trained on the principles of writing quality multiple-choice questions. Apart from hands-on motor or sensory skills, a properly designed multiple-choice question can assess any knowledge or decision making process regardless of industry, cognitive complexity, or difficulty level.

One type of multiple-choice question can assess terms, facts or principles directly from a reference source. This type of question is characterized as a "recall" question. Typically, these require the least amount of thought for the examinee to respond--either they immediately know the answer or they are forced to guess. This type of question is fine for testing simple "book knowledge," but falls short of assessing more complex or higher-level decision making. As such, these questions should consist of less than one-third the total number of questions on a high-stakes exam.

A more complex question type places examinees in a real-life situation or circumstance that requires the recognition, selection, ordering or cause of job-related principles or procedures. This type of question is referred to as an "Application" question, and can take 2-3 times the amount of effort to develop recall questions. Application questions should represent at least one-third the total number of questions on high-stakes examinations.

The most complex type of questions called "Analysis" requires examinees to evaluate information to discern fact from inference or assumption. Analysis questions often contain graphs, charts, or tables of information and require the examinee to extract relevant information from that which is irrelevant or erroneous. This type of questions can take 5-10 times the amount of effort to develop recall questions. Nonetheless, a well written high-stakes examination should have more than 20-30% of analysis type questions.

Because untrained question writers find recall questions easiest to write, exams tend to contain far too many of this type and not enough application or analysis questions. As a result, the value of the examination in the eyes of candidates and stakeholders drops because of an inability to assess the important components of the job.

At Exam Design, we place great emphasis on training subject matter experts (SMEs) to write high-quality multiple-choice questions. With the proper training contained in ExamDeveloper, writers can become skilled in writing questions that assess the entire range of knowledge and skills required for competent job performance, not just those easily found in a book.

For organizations who prefer to use question types other than multiple-choice, ExamDeveloper can support many different types including hot-spot, drag and drop, true-false, multiple response, case-based sets, and R-type questions.

Next Step

Which question types make the best exams?

Research shows that multiple-choice questions are the preferred exam format for a wide variety of industries and content areas for a number of reasons.

First, multiple-choice exams can assess a wide variety of knowledge and skills if designed properly. Second, multiple-choice exams can be developed by a large number of subject matter experts with limited face-to-face interaction. Finally, multiple-choice exams are less expensive to develop, administer, and score than formats such as simulation and practical exams, as standardization is often more difficult to achieve with those formats.

Our team of experts is experienced in developing exams in many formats, include multiple choice, simulation, practical, design, and performance-based. Our experts will help you understand the benefits of each format and will guide you toward the most effective option for you, not the one most convenient or profitable for us.