Adapted from:

http://www.psywww.com/selfquiz/aboutq.htm

by Russell A. Dewey, PhD

Essay tests have their own set of problems.
Specify objectives or give study questions
Defeat the "test-wise" strategies of students who don't study
Summary: My Test Construction Procedure

| Essay tests | Objectives | |"Test-wise" strategies | Test Construction | Top | Inst. Site |

[This page, http://www.psywww.com/selfquiz/aboutq.htm, is a little essay on multiple choice questions. It might be of interest to my students--for example, to warn them away from the usual guessing strategies discussed below--or to visiting educators. Comments are welcome. --Russ Dewey]

Multiple choice questions are widely scorned as "multiple guess" questions. Some teachers assume that multiple choice items encourage superficial studying. Perhaps it is true...other things being equal, students do not study as hard or as well for a multiple choice test. Certainly if students are expecting a multiple choice test and they receive an essay test instead, they complain vigorously!

However, essay tests have their own set of problems.

For example...

Studies have shown that the grade given to an essay test depend in part upon the neatness of the handwriting. That seems like a poor way to assign a grade. However, if students are asked to do the test on a word processor, it is hard to insure that the work is original. Studies have also shown that grades for essay tests are influenced by length. If a student rambles on at length, there is greater likelihood of hitting a few points that the teacher is looking for. But do we want to reward verbosity?

Essays test grades depend upon writing skills. We want students to be able to write well, but do we want a test grade in psychology to reflect English skills? Essay tests tend to focus on a few large issues. One cannot ask many essay questions in a single testing sessions. But we also need to teach students many specific facts, especially in a beginning course. We do not want to encourage students to ignore everything except a few big issues. With multiple choice questions, one can give lots of study questions and achieve thorough coverage. The grading of essay tests is somewhat subjective. Students assertive enough to argue a point may be able to convince a teacher that they knew the material, and win a few points back, but do we want to encourage that, or penalize the less aggressive student?

Last but not least, essay tests cannot be scored with a machine, so they are very labor-intensive, and they are almost completely impractical for auditorium sized classes. A teacher determined to use essay tests in a large class must resort to a few large, "make or break" tests per term, and that puts a lot of stress on students, plus it does not give them rapid feedback during the term about how well they are understanding the material. Despite all this, essay and short answer tests have many virtues. Students need practice formulating arguments, expressing things clearly, and integrating ideas. Nobody would argue that all testing should be multiple choice. However, for many teachers in many situations, a good objective test is both fairer and more efficient than an essay or short answer test.

I maintain it is possible to construct multiple choice questions which are not readily guessed and which therefore require a student to comprehend basic factual material. Because multiple choice tests lend themselves to machine scoring, students can be offered tests over small chunks of material (which has been shown to promote learning) and students can be offered frequent re-tests or makeup tests (to reduce anxiety and promote mastery of the material). The key to writing "good" multiple choice questions is to (1) specify objectives so the questions can be specific and focused (or give study questions which let the student know what the teacher considers important), (2) endeavor to reduce frustration for the creative student who can invent rationales for several different answers, and (3) defeat the "test-wise" strategies of the student who has not studied and is attempting to guess answers. Let's consider these in turn.

Specify objectives or give study questions

In my opinion, students should not be forced to guess what will be on a test, or "psych out" the teacher to decide what to study. Educational research shows that the less able students are heavily penalized by a failure to spell out what is required for a test. The more able students seem to sense what the teacher wants, but the students most in need of help are likely to flounder even more painfully if they must guess what to study.

The obvious solution to this problem is to give students specific study questions, then draw the test from the study questions. But sometimes people criticize the idea of "teaching the test," as if having study questions in itself encourages a superficial approach. That may be true if there are very few study questions. However, if a teacher offers study questions for all the most important ideas in an assignment, then "teaching the test" is "teaching the course." In Keller Plans, an independent study method popular in the 1970s, students were given lots of specific objectives or study questions, and lots of opportunities to take quizzes covering that material, and the results were very good...in fact, Keller Plans were one of the few educational innovations in the entire 20th Century which produced better results that traditional lecture/discussion methods of instruction. I have received several requests to put material about Keller Plans on the Psych Web, so I will soon.

Endeavor to reduce frustration for the creative student To reduce frustration for good students, I avoid "all of these" and "none of these" and "both a & b" answers. Studies show that these items do just as well as other items at discriminating students who have studied well, thus they are acceptable from a theoretical standpoint, but over the years I have noticed that my best students hated them! The more psychology a person knows, the easier it is to make arguments in favor of answers that the author might regard as wrong. More than once I have seen lists of "myths" at the beginning of a textbook which, in my opinion, contained some true statements...or lists of "surprising facts" which I thought contained some myths. True/false questions are the worst of all in this regard. Often the truth value of an isolated statement is quite debatable! It all depends on how it is interpreted, the definition of a key term, or the context. Questions with the "both a & b" type answer put a student in the position of making True/False evaluations. Better to have four or five alternatives with only one correct.

Defeat the "test-wise" strategies of students who don't study

The whole point of testing is to encourage learning. A test which cannot be easily guessed is a test which motivates quality studying. Therefore, to motivate students to study and learn, one must design quiz items that are not easily guessed without good studying. One must also design a test so that answers are not obvious to the student who has merely skimmed the assignment, or studied only highlighted words, or read only summaries

And one must defeat the common "rules of thumb" which students use to guess correct answers.

Rule of thumb: "Pick the longest answer." Way to defeat this strategy: make sure the longest answer is right about a fifth of the time (if there are five alternatives for each question)
Rule of thumb: "Pick the 'b' alternative." Way to defeat this strategy: make sure each answer is used the same number of times, in random order.
Rule of thumb: "Never pick an answer which uses the word 'always' or 'never' in it." Way to defeat this strategy: make sure such answers are correct about a fifth of the time
Rule of thumb: "If there are two answers which express opposites, pick one or the other and ignore other alternatives." Way to defeat this strategy: sometimes offer opposites when neither is correct.
Rule of thumb: "If in doubt, guess." Way to minimize the impact of this strategy: use five alternatives instead of three or four
Rule of thumb: "Pick the scientific-sounding answer." Way to defeat this strategy: use scientific sounding jargon in wrong answers
Rule of thumb: "Don't pick an answer which is too simple or obvious." Way to defeat this strategy: sometimes make the simple, obvious answer the correct one.
Rule of thumb: "Pick a word which you remember was related to the topic." Way to defeat this strategy: when drawing up distracters (wrong answers) use terminology from the same area of the text as the right answer, but in distracters use those words incorrectly so the wrong answers are definitely wrong.

I draw up study questions which cover virtually every important concept from the chapter. This results in about 160 study questions per chapter...a number arrived at through trial and error. It is enough to give complete coverage without being too overwhelming if the material is reasonably easy to understand.

Next, I draw up multiple choice questions using the study questions themselves as a starting point. I use questions with five alternatives, rather than four. That reduces the likelihood of guessing the correct answer.

I avoid "all of these" or "none of these" or "both a & b" type answers for the reasons discussed above (I found that excellent students could often come up with creative reasons to pick the wrong answers, with those types of questions). I just use five different answers, and only one is correct.

I use a complicated macro written for my word processor to generate many different quizzes from the same pool of 160 items. Technically, each quiz is a stratified random sample, "stratified" in the sense that items are drawn systematically from all areas of the chapter (to assure no big areas are left out of any quiz) but otherwise random. I use 6-8 different quiz forms in each class, so cheating is never much of a problem in my classes, even when students are elbow to elbow.

I use the computer to check the frequency of various answers, and I switch the position of correct answers until all the letters are used equally often. In a pool of 160 items for a chapter, there are 32 A answers, 32 B answers, 32 C answers, 32 D answers, 32 E answers. Because the quizzes are generated by a computer, there is no systematic pattern of answers. However, this does raise another problem...an ironic one...because a truly random listing can have very non-random looking sequences which a non-computer (human) test maker would probably eliminate. For example, there can be four "c" answers in a row on one of my quizzes, just by the "luck of the draw." I have to caution students that a truly random series of answers can have unlikely-looking strings of answers! A student taking one of my quizzes should never assume that, just because the last three answers are "c," the next one surely cannot be "c." The chances of the next one being "c" are, in fact, 1 in 5.

I use quotation marks and scientific sounding jargon in wrong answers, just as often as I do in correct answers. So these are not effective cues Using each study question as a starting point, I construct plausible sounding alternatives which are supposed to be clearly wrong...but which might sound right to a poorly prepared student. Of course, I also insert a correct answer. I do this with the book closed, because if I am looking at the text while I write the question, I am likely to be too picky or unreasonable in what I expect students to remember. If I can't remember it myself, after all, I cannot expect students to remember it.

The result of the entire procedure is quiz items which are hard to guess unless the student truly understands the material. My "validation" for this procedure is informal: I get very consistent results term after term, and I notice that students who do poorly on my quizzes generally are unable to answer the relevant study question accurately from memory. However, when the student sits down, book in hand, and compares the quiz item to the study question and the material in the text, the answer is obvious and the students seldom complain that the test item is unfair. This leads me to think the test items are doing their job.

APA-style reference for this page:
Dewey, R.A. (1998, January 20). Writing Multiple Choice Items which Require Comprehension. [Online]. Available: http://www.psych-web.com/selfquiz/aboutq.htm

Go to Title III Institute Web Site