Adapted from:
http://darkwing.uoregon.edu/~tep/assessment/mc4critthink.html
I.
Recalling memorized information. Important
Considerations
Some suggestions:
II. Bloom's Taxonomy of Cognitive
Levels
1. Recalling memorized information
2. Comprehension
3. Application
4. Analysis
5. Synthesis
6. Evaluation
III. Practical Suggestions for Writing
Multiple-Choice Questions
General Suggestions
Writing the Stem
Answer Options
IV. Some Techniques for Writing
Multiple-Choice Items that Demand Critical Thinking
1) Premise - Consequence
2) Analogy
3) Case study
4) Incomplete Scenario
5) Problem/Solution Evaluation
V. Bibliography of Multiple-Choice
Question Resources
Books:
Articles:
I. Recalling
memorized information.
Practical Suggestions for Writing Exams
Techniques for Creating Questions
Bibliography
Important Considerations What role should testing play in the learning process? How can tests create a real dialogue between ourselves and our students about what students do and do not understand? How can we avoid using tests to simply punish or reward cramming?
II. Bloom's Taxonomy of Cognitive Levels Knowledge
1. Recalling memorized information.
May involve remembering a wide range of material from specific
facts to complete theories, but all that is required is the bringing
to mind of the appropriate information. Represents the lowest
level of learning outcomes in the cognitive domain.
Learning objectives at this level: know common terms,
know specific facts, know methods and procedures, know basic concepts,
know principles.
Question verbs: Define, list, state, identify, label, name,
who? when? where? what?
2. Comprehension
The ability to grasp the meaning of material. Translating
material from one form to another (words to numbers), interpreting
material (explaining or summarizing), estimating future trends
(predicting consequences or effects). Goes one step beyond the
simple remembering of material, and represent the lowest level
of understanding.
Learning objectives at this level: understand facts
and principles, interpret verbal material, interpret charts and
graphs, translate verbal material to mathematical formulae, estimate
the future consequences implied in data, justify methods and procedures.
Question verbs: Explain, predict, interpret, infer, summarize,
convert, translate, give example, account for, paraphrase x?
3. Application
The ability to use learned material in new and concrete situations.
Applying rules, methods, concepts, principles, laws, and
theories.
Learning outcomes in this area require a higher level of understanding than those under comprehension.
Learning objectives at this level: apply concepts and
principles to new situations, apply laws and theories to practical
situations, solve mathematical problems, construct graphs and
charts, demonstrate the correct usage of a method or procedure.
Question verbs: How could x be used to y? How would you
show, make use of, modify, demonstrate, solve, or apply x to conditions
y?
The ability to break down material into its component parts. Identifying parts, analysis of relationships between parts, recognition of the organizational principles involved. Learning outcomes here represent a higher intellectual level than comprehension and application because they require an understanding of both the content and the structural form of the material.
Learning objectives at this level: recognize unstated
assumptions, recognizes logical fallacies in reasoning, distinguish
between facts and inferences, evaluate the relevancy of data,
analyze the organizational structure of a work (art, music, writing).
Question verbs: Differentiate, compare / contrast, distinguish
x from y, how does x affect or relate to y? why? how? What
piece of x is missing / needed?
5. Synthesis
(by definition, synthesis cannot be assessed with multiple-choice
questions.
It appears here to complete Bloom's taxonomy.) The ability to
put parts together to form a new whole.
relate to y? why? how? What piece of x is missing / needed?
This may involve the production of a unique communication
(theme or speech), a plan of operations (research proposal),
or a set of abstract relations (scheme for classifying
information).
Learning outcomes in this area stress creative behaviors, with major emphasis on the formulation of new patterns or structure.
Learning objectives at this level: write a well organized
paper, give a well organized speech, write a creative short story
(or poem or music), propose a plan for an experiment, integrate
learning from different areas into a plan for solving a problem,
formulate a new scheme for classifying objects (or events, or
ideas).
Question verbs: Design, construct, develop, formulate,
imagine, create, change, write a short story and label the following
elements:
6. Evaluation
The ability to judge the value of material (statement,
novel, poem, research report) for a given purpose. The judgments
are to be based on definite criteria, which may be internal (organization)
or external (relevance to the purpose). The student may determine
the criteria or be given them. Learning outcomes in this area
are highest in the cognitive hierarchy because they contain elements
of all the other categories, plus conscious value judgments based
on clearly defined criteria.
III. Practical Suggestions for Writing Multiple-Choice Questions
1) Do not write the test in one day. Spread the work out over time. Questions demanding high-level thinking take longer to craft-professional item writers often write only 3 or 4 per day. Write one or two questions after each class, so it becomes a simple matter of assembling them into an exam. Some teachers keep a rubber-banded stack of note cards in their desk for this purpose.
2) If students are to hand-write the letters of their chosen answers, ask them to use CAPITAL LETTERS. The handwritten, lower-case letters "a" and "d" and "c" and "e" can be difficult to distinguish when scoring.
1) Phrase stems as clearly as possible-confusing questions
can generate wrong answers from students who do understand the
material.
For example, a confusing stem like: "According to Tuckman's
model, groups develop through several stages over time. Furthermore,
it contradicts Poole's activity-track model which has groups switching
among several different linear sequences. Which of the following
is not one of the stages identified in Tuckman's model?"
could be cleaned up to read: "Tuckman's model of group development
includes: [Select all that apply]"
2) Avoid extra language in the stem.
Some think extraneous details make a question more complex. However,
they most often just add to the students' reading time. This reduces
the number of questions you can put on a test, therefore reducing
the reliability of the test. For example, in the Tuckman question
above, the information on Poole's model had nothing to do with
the information sought by the question.
1) Avoid lifting phrases directly from text or lecture.
This becomes a simple recall activity for the student. Use new
language as frequently as possible.
2) Most literature recommends writing the correct answer before
writing the distracters. This makes sure you pay enough attention
to formulating the one clearly correct answer.
3) Answer options should be about the same length and parallel
in grammatical structure. Too much detail or different grammatical
structure can give the answer away.
For example, the specificity and grammatical structure of the first option here are dead give-aways:
The term "side effect" of a drug:
a) refers to any action of a drug in the body other than
the one the doctor wanted to drug to have.
b) is the chain effect of a drug.
c) additionally benefits the drug.
4) Limit the number of answer options. Research shows that three-choice items are about as effective as four-choice items. Four choice items are the most popular, and never give more than five alternatives.
5) Distracters must be incorrect, but plausible. If you can, include among the distracters options that contain common errors. Students will then be motivated to listen to your explanations of why those options are incorrect.
6) To make distracters more plausible, use words that should be familiar to students.
7) If a recognizable key word appears in the correct answer, it should appear in some or all of the distracters as well. Don't let a verbal clue decrease the accuracy of your exam.
For example, someone with no biology background would not have to think very hard to make a correct guess on this question:
Every organism is made of cells and every cell comes
from another cell. This is the:
a) Relativity Theory
b) Evolution Theory
c) Heat Theory
d) Cell Theory
8) Help students see crucial words in the question.
For example: "Which of the following is NOT an explicit norm?" Likewise, when you ask a similarly-worded question about two different things, always highlight the difference between the questions.
9) It is often difficult to come up with 3 or 4 plausible distracters, and teachers will sometimes add some that are not plausible, or even humorous. Be careful.
If it is too easy to eliminate one or two options, then the question loses much of its measurement value. If energy or time is limited and you must come up with one more distracter, consider either offering a true statement that does not answer the question and/or a jargon-ridden option that is meaningless to someone who understands the concept.
10) Use Rarely:
IV. Some Techniques for Writing Multiple-Choice Items that Demand Critical Thinking
1) Premise - Consequence
Students must identify the correct outcome of a given circumstance.
Example: If nominal gross national product (GNP) increases
at a rate of 10% per year and the GNP deflator increases at 8%
per year, then real GNP:
a) Remains constant.
b) Rises by 10%.
c) Falls by 8%.
d) Rises by 2%.
Note: To increase the difficulty, provide more than one premise.
2) Analogy
Students must map the relationship between two items into
a different context:
Example: E-mail is to an unmoderated listserv as office hours
are to:
a) Class lecture.
b) Class discussion.
c) Review sessions.
d) Tutorials.
3) Case study A single,
well-written paragraph can provide material for several follow-up
questions.
Example:
2) Alice, Barbara, and Charles own a small business: the Chock-Full-o-Goodness
Cookie Company. Because Charles has many outside commitments and
Barbara has a few, Alice tends to be most in touch with the daily
operations of Chock-Full-o-Goodness. As a result, when financial
decisions come down to a vote at their monthly meeting, they have
decided that Alice gets 8 votes, Barbara gets 7, and Charles gets
2-with 9 being required to make the decision.
According to minimum-resource coalition theory, who is most likely to be courted for their vote?
a) Alice
b) Barbara
c) Charles
d) No trend toward any specific person.
4) Incomplete Scenario
Students must respond to what is missing or needs to be changed
within a provided scenario.
Note: when using a graph or image, try to lay it out differently than how the students have seen it. This is equivalent to using new language to present a familiar concept and prevents students from using rote memorization to answer the question.
For example, the diagram below may originally have been split left to right instead of top to bottom, and this diagram may not be as detailed as the diagram they saw in the book.)
Example: Use the diagram below to answer the following questions.
1) What belongs in the empty box in the upper right corner of
the diagram?
a) Hardware devices
b) Client Services for Netware
c) Logon Process
d) Gateway Services for Netware
2) If the Applications resided below the heavy black line,
they would:
a) be open to hackers on the network.
b) compete with the OS for memory.
c) be preemptively multi-tasked.
d) launch in individual NTVDMs.
5) Problem/Solution Evaluation Student are presented a problem and a proposed solution. They must then evaluate the proposed solution based upon criteria provided.
Example: A student was asked the following question: "Briefly
list and explain the various stages of the creative process."
As an answer, this student wrote the following:
"The creative process is believed to take place in five stages,
in the following order: orientation, when the problem must be
identified and defined, preparation, when all the possible information
about the problem is collected, incubation, when no solution seems
in sight and the person is often busy with other tasks, illumination,
when the person experiences a general idea of how to arrive at
a solution to the problem, and finally verification, when the
person determines whether the solution is the right one for the
problem."
How would you judge this student' s answer?
a) EXCELLENT (all stages correct in the right order with clear
and correct explanations)
b) GOOD (all stages correct in the right order, but the explanations
are not as clear as they should be)
c) MEDIOCRE (one or two stages are missing OR the stages are in
the wrong order, OR the explanations are not clear OR the explanations
are irrelevant)
d) UNACCEPTABLE (more than two stages are missing AND the order
is incorrect AND the explanations are not clear AND/OR they are
irrelevant)
V. Bibliography of Multiple-Choice Question Resources
Bloom, Benjamin B. (Ed.) Taxonomy of Educational Objectives:
the classification of educational goals, by a committee of college
and university examiners 1st Ed. New York: Longmans, Green, 1956.
Davis, Barbara Gross. Tools for Teaching San Francisco:
Jossey-Bass, 1993.
Erickson, Bette LaSere and Diane Weltner Strommer. Teaching
College Freshmen San Francisco: Jossey-Bass, 1991.
Jacobs, Lucy Cheser and Clinton I. Chase. Developing and
Using Tests Effectively: A Guide for Faculty San Francisco: Jossey-Bass,
1992.
McKeachie, Wilbert. Teaching Tips: Strategies, Research,
and Theory for College and University Teachers (9th Ed.) Lexington,
Mass: D.C. Heath and Company, 1994.
Miller, Harry G., Reed G. Williams, and Thomas M Haldyna.
Beyond Facts: Objective Ways to Measure Thinking Englewood Cliffs:
Educational Technology Publications, 1978.
Clegg, Victoria L. and William E. Cashin. "Improving
Multiple-Choice Tests." Idea Paper #16, Center for Faculty
Evaluation and Development, Kansas State University, 1986.
Fuhrman, Miriam. "Developing Good Multiple-Choice
Tests and Test Questions." Journal of Geoscience Education
44 (1996): 379-384.
Johnson, Janice K. ". . . Or None of the Above."
The Science Teacher 56.2 (1989) 57-61.
Web-sites:
University of Capetown's Guide to Designing and Managing Multiple
Choice Questions An excellent site from which much of our workshop
material was adapted.
Go to Title III Institute Web Site