It is Not the Spanish Inquisition: Designing and Using Effective Multiple Choice Questions

Ane Lintvedt with Laurie Mannino

     It is safe to say that few go into the teaching profession because they like administering tests; and yet assessments are most often how our students' successes—and our own—are judged. Very few teachers come into the profession armed with an understanding of the thinking that should underpin assessment. The lead author of this essay examines the means of creating effective multiple-choice assessments and draws attention to a student-based approach to the subject developed by Laurie Mannino. We both argue that multiple-choice questions can and should go beyond factual recall and encompass assessment of historical thinking.

     Although most text books come with test banks, the authors of the text book rarely have much to do with the writing of that test bank or any of the other ancillaries. Therefore a very good text book may well be accompanied by a startlingly poor set of ancillaries. Experienced teachers know that test bank questions are often of poor quality, frequently riddled with mistakes, and assess only basic information at the level of recall alone. It is worth learning how to write your own, high-quality multiple-choice assessments so you are not at the mercy of the test banks and so you are assessing your students on a variety of knowledge and skills.

     During two decades of teaching on-level and AP classes in high school, I became increasingly disillusioned with test bank ancillaries and have come to the conclusion that there is no free lunch. Copying someone else's examination questions—a seemingly easy task—leads to appalling bad assessments and more time spent doing triage, which is harder to do than to have taken the time to write a better test in the first place.

     I finally learned how to write better multiple choice questions when I worked with the test development professionals at Educational Testing Services (ETS). They taught me the logic as well as the methods of writing superior multiple-choice questions as I worked with them on SAT subject tests, AP Tests, and Praxis test questions. Disclaimer: What I have written below is my distillation and many experiences of writing multiple choice questions, and not anything that ETS has published or endorsed. I am going to start with some basic rules and their rationales, and then discuss the more complex issues of assessing different types of historical thinking skills in theory and in my own practice and also through a student-driven assessment model developed by Laurie Mannio.

The Basic Rules

1. Write 5 answer choices.
2. All answer choices should be the same length and complexity.
3. No "all of the above" or "a and b, but not c, d, or e", or any other byzantine arrangement.
4. Only use "all of the following except" when it suits material that has been taught in the negative.
5. Write at least two questions that correspond to reading-comprehension paragraphs or questions based on visuals (map, chart, graph, photograph, drawing).
6. When using visuals, make sure to construct a question for which the students must actually use the visual.
7. Incorrect choices (distracters) should not be mutually exclusive.
8. Use "only, "never", "always" sparingly.
9. Never, ever give let students keep the questions or answer sheets.

Rationales of the Basic Rules

The most important thing to remember about a multiple-choice assessment is that you should be testing content and historical reasoning skills. You should not be designing a test to trick kids or make them waste their time riddling through a maze. That kind of student torture will not help prepare them for any SAT Reasoning or Subject Test, the AP exams, or state standards' exams.

1. AP exams and the SAT exams all use 5 answer choices, not 4, so it's best to get students used to dealing with 5.
2. From an early age, students know that the long answer choice is probably correct, or you wouldn't have wasted time on writing it.
3. It may amuse you to torture kids by writing byzantine distracters, but these will never appear on any of the standardized testing that the kids will encounter.
4. This one really hurts, because "all of the following except" questions are the easiest ones to write. This format adds needless confusion to most kids' testing experience. You should only use "all of the following except" when the topic has been taught that way (it's-this-but-not-that). For example, when I explain "Fascism", I do it by explaining what fascists opposed: communism, capitalism, liberalism, Judaism, Jehovah's Witnesses, homosexuality, etc. I can legitimately ask an "all of the following except" question since I taught the material that way.
5. Students have to take a lot of extra time with reading -comprehension questions and the interpretation of visuals. Writing at least two questions about these make them worth the student's time and effort. If you can only come up with one question, it's probably not a significant thing to test anyway.
6. You don't want a student to be able to answer the question without needing to consult the visual. For example, you have a graph of 21^st-century population figures, and the question is "Which country had the largest population in 2002?" Most students will know this is China without needing to consult the graph.
7. Students learn from an early age to zero-in on mutually-exclusive answer choices: for example, a. Zimbabwe, b. not Zimbabwe. The answer must be either "a" or "b". This is assessing test-taking skills, not history.
8. Students learn from an early age that "only", "always," and "never" are red flags, and they will zero in on these options rather than employing historical thinking skills.
9. It took you hours, if not days, to write good multiple-choice questions. Once the students have taken the test, never let them keep the questions or their answer sheets. That way you can use the questions for several years in a row. There are other ways to debrief an assessment—personal conferences, group discussions, picking two or three particularly difficult questions to analyze in class, etc.—that don't entail handing back the whole test or answer sheet.

Assessing Historical Thinking Skills

The most basic type of multiple-choice question is an information-recall or a content question. This type of question would ask students to recall specific data: events, locations, names, geography, etc. Factual recall certainly favors students who can memorize discrete pieces of information, but it rarely assesses anything more than that. I have listed below a series of historical thinking skills to which we should expose our students and which we all hope our students master. For those who teach Advanced Placement classes, you should know that all the AP history exams are going to start deliberately assessing historical thinking skills in multiple-choice questions within the next few years.

1. Interpretation of visual evidence (maps, graphs, charts, photographs, works of art or architecture, etc.)
      •  This could be assessed with a question about the time period, the place, the style, the cultural influences displayed in a particular visual source.
2. Point-of-view or argumentation
      •  This could be assessed with a quotation and questions about authorship, the authorial point-of-view, how the quotation fits into a particular argument, or how evidence is being used.
3. Cause and Effect
      •  There are many possibilities for both Cause questions and Effects questions when considering specific events, movements, or long-term themes.
4. Patterns of continuity and change over time (chronological reasoning)
      •  This kind of question would ask students to track or explain or identify a long-term continuity (i.e., ideas, political and economic structures, systems of labor or gender) or to assess a particularly important change over a long period of time. It could also ask about themes within a particular period of history over wider geographical spaces.
5. Connecting a small event to a larger regional, national or global context
      •  This could be a visual or a reading comprehension question that asks the student to explain the larger context of the piece, or place it into a larger theme or trend.
6. Comparisons, simple or complex
      •  Comparison questions could be between people, events, ideas, and across geographical or chronological boundaries.
7. Using evidence to prove a point
      •  This could be a question that asks which piece of evidence or data would prove an assertion, or would substantiate an argument in the stem of the question.
8. Using multi-disciplinary evidence
      •  This could be a question that asks students to incorporate or explain archeological, literary, scientific evidence into a historical argument.
9. Recognizing how historical interpretations are used
      •  This could be a question with a quotation that asks the students to assess how the writer used or displayed a certain interpretation, or which historical interpretation was being used, or how a particular interpretation could be disputed.

     There are more, and also more complex, historical thinking skills that can be assessed with multiple-choice questions. The ones I have listed above will, I hope, get you started along a path of writing thoughtful and varied questions that will truly assess what your students know and what skills they have learned. Once I learned how varied and how well a good multiple-choice question can assess information and historical thinking skills, I stopped using test banks and cringed at the questions I wrote 5 years ago.

     I don't deny these sorts of questions take a lot of time to think through and compose. My family members and colleagues, sadly, are used to my whining about having to write yet another set of multiple-choice questions. Two things should always be kept in mind when spending that valuable time. First, students hate may hate your well-designed multiple-choice questions, but will be delighted when they perform exceedingly well on AP exams. Second, never, ever hand back your multiple-choice questions to students.

Assessing Historical Thinking Skills

Teaching students to write multiple-choice assessments can have multiple benefits. Laurie Mannino employs an assignment that directs the student to isolate the most significant information of the unit and distill it into an assessable question. It addresses the learning of historical content; deeper understandings of the significance of themes and historical thinking skills; familiarity with AP-style multiple-choice questions, and last but not least a certain sympathy and appreciation for the work teachers do to create multiple-choice assessments. As all experienced teachers know that to really understand something you should try to teach it to someone else. With the exercises below, Laurie guides her students through both the learning and teaching of history with an assignment of creating multiple-choice questions from their World History reading.

     Laurie requires that the questions must be based on the AP Themes that are outlined in the College Board AP 2008-9 World History Course Description Book (or Acorn Book) online at http://apcentral.collegeboard.com/apc/public/repository/ap07_worldhist_coursedesc.pdf. This source ensures that the questions will not be merely information-recall and reinforces the importance of the themes of the course. Students must then follow the Basic Rules (outlined above); e.g., 5 answer choices, only one correct answer, etc. A crucial part of the assignment is that in addition to the questions, there must be an accompanying paper explaining why each correct answer choice is correct and why each incorrect choice (or distracter) is incorrect. This exercise will reinforce the information as well as refine the students' ability to relate specific information to more general themes. For AP classes in particular, but for any level World History survey, this assignment could be used either as a straight-forward assessment in the middle of a unit, it could be used as a cumulative assignment at the end of an era.

     Teachers could then compile a set of student-composed questions and create another assignment that requires students to critique and correct questions, or have students take their peers' assessment, or use student-created questions on unit assessments in one way or another. There are a lot of possibilities here. And certainly, the questions could be compiled and used as review for an end-of-course exam, as Laurie clearly does.

An example of the assignment that Laurie would hand out to her students is offered below:

Your assignment is to create five AP level multiple-choice questions (MCQ's) using the material found in the required chapter. This assignment will help enormously with the MCQ portion of the AP exam; it will also help with the essay portion in terms of content; and it will help you understand the material in the text. The questions and answers must reflect the World History AP (WHAP) themes as listed below: there must be one question for each theme.

WHAP themes:

1. Interaction between humans and the environment (demography and disease, migration, patterns of settlement, technology)
2. Development and interaction of cultures (religions, belief systems, philosophies, ideologies, science and technology, the arts and architecture)
3. State-building, expansion and conflict (political structures and forms of governance, empires, nations and nationalism, revolts and revolutions, regional, trans-regional and global structures and organizations)
4. Creation, expansion and interaction of economic systems (agricultural and pastoral production, trade and commerce, labor systems, industrialization, capitalism and socialism)
5. Development and transformation of social structures ( gender roles and relations, family and kinship, racial and ethnic constructions, social and economic classes)

Please refer to the handouts [the Basic Rules] on how to construct an MCQ and the example sheet in order to better understand the level of question that you need to develop. Any questions? Please ask BEFORE the assignment is due! :-) HAVE FUN!

Important details:  

• Questions need to be from the entire chapter.
• Be sure you write the theme out before each question.
• Each question must reflect a different theme.
• The questions must be typed and submitted on a separate sheet of paper from the answers.
• You must also indicate on a separate paper on which page of the text the answer is located, along with a brief explanation about why the correct answer choice is correct and why each incorrect answer is incorrect.
• Papers will be graded and corrected for content, grammar, punctuation and spelling errors and returned to you.
• You must retype the corrected version and hand it back to me within the directed time limit WITH the original corrected version or your grade will be zero. (0)
• I will file the questions by chapter in a box for later reference.

Assessing Historical Thinking Skills

Whether you use multiple-choice questions sparingly or frequently, it is important to write ones that truly assess students' content knowledge as well as their historical thinking skills. And if you can teach your student to write useful and thoughtful multiple-choice questions, the benefits will be plentiful.

Ane Lintvedt teaches history at the McDonogh School in Owings Mills Maryland. She has scored the AP European and World history exams, is a consultant for College Board and is currently a member of the executive council of the World History Association. She can be reached at alintvedt@mcdonogh.org.

Laurie Mannino teaches history and religion at Col. Zadok Magruder High School in Rockville, Maryland. She also works as a consultant for the College Board, scores the AP World History exam and has written test questions for national teacher certification programs. She can be reached at Laurie_S_Mannino@mcpsmd.org.