"If assessment tasks are to tap higher-order cognitive processes, they must require that students cannot answer them correctly by relying on memory alone." - Anderson and Krathwohl, A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom's Taxonomy of Educational Objectives (New York: Longman, 2001), p. 71.

NETA Assessment relates to our testing materials. Under NETA Assessment, Nelson's authors create multiple-choice questions (MCQs) that reflect research-based best practices for constructing effective questions and testing not just recall but also higher-order thinking. Our guidelines were developed by David DiBattista, a 3M National Teaching Fellow whose recent research as a professor of psychology at Brock University, St. Catharines, has focused on multiple-choice testing. All Test Bank authors receive training at workshops conducted by Prof. DiBattista, as do the copyeditors assigned to each Test Bank. A copy of Multiple Choice Tests: Getting Beyond Remembering (2008), Professor DiBattista's guide to writing effective tests, is included with every Nelson Test Bank/Computerized Test Bank package.

Assessing Beyond MCQ

NETA Assessment relates to testing materials: not just Nelson's Test Banks and Computerized Test Banks, but in all of our assessment pieces.

  • In-text self-tests
  • End of chapter questions
  • Study guides
  • Web quizzes
  • Homework programs like Aplia, CNOW, and CourseMate

Multiple-Choice Testing

Multiple-choice testing is the most commonly used assessment tool in higher education. Because of this, it is of utmost importance that these types of testing are of the highest quality. There are two fundamental criteria that must be followed in order to ensure the quality of multiple-choice test items. The first is "content expertise—that is, a thorough knowledge of the relevant facts, concepts, procedures and skills that are to be tested. Without content expertise, one simply cannot write effective items. By itself, however, content expertise cannot ensure that multiple-choice items will be of high quality. The second prerequisite for writing effective items is a specialized knowledge of best practices in multiple-choice item construction" (DiBattista, 2008).

By guaranteeing the above, NETA Assessment ensures that Nelson's multiple-choice testing goes beyond simple rote testing to assess higher-order cognitive processes by utilizing Bloom's revised taxonomy.

Assessing Beyond Remembering

Using Bloom's revised taxonomy, our MCQ questions seek to test beyond recall. The NETA Assessment allows you to look through your Nelson Test Bank to decide which MC items to include on your tests, and you can examine items to determine whether they are memory-based or require your students to engage in higher-level thinking. By making your selections appropriately, you can construct tests that contain a mix of items that reflects your personal instructional goals. For a great many teachers, this will mean being able to construct MC tests that are more strongly weighted toward assessing higher-level thinking, with memory-based items taking more of a back seat than is usually the case. An excerpt from David DiBattista's Multiple-Choice Tests: Getting Beyond Remembering explains basic techniques for testing higher-level thinking.

Quality Control: The Editorial Process

At Nelson, we pride ourselves on the deep editorial work that goes into the creation of our assessment products. Every author of a NETA test bank is trained on constructing good multiple-choice questions. Quality assurance does not end there; our copy-editors and developmental editors are trained as well, ensuring that every NETA test bank has the highest quality of questions possible.

How good can a multiple-choice question be? Recognizing the importance of multiple-choice testing in today’s classroom, we have created the Nelson Education Testing Advantage (NETA) program to further ensure the value of our high-quality test banks. The goals of the NETA program include avoiding common errors in test construction and developing multiple-choice test questions that "get beyond remembering" to assess higher-level thinking.

The following are examples of test items that illustrate the power of NETA. Each example shows a "Before" and "After NETA" question. Below each of the examples is a brief analysis by David DiBattista. His comments explain the rationale behind the changes and how the "After" represents the kind of improvement you can expect in all of our NETA-affiliated test banks.

Example 1

Question 1 "Before"

1. ____ are long, ladder-like chains of chemical molecules.

a. Chromosomes

b. Phenotypes

c. DNA

d. Genes

ANS: C    DIF: factual    REF: 72

Question 1 "After"

1. What do we call the long, ladder-like chains of molecules that provide the instructions for growth and development?

a. chromosomes

b. phenotypes

c. DNA

d. genes

ANS: C    BLM: Remember    DIF: easy    REF: 92

Key: ANS = correct answer; BLM = Bloom category, DIF = difficulty level, REF = page reference to text.

Question 1 "Before" has a couple of problems. First, the stem of this item is in sentence-completion format, and to make matters worse, the completion is internal to the stem rather than coming at the end of the stem. The best format for the stem of a multiple-choice item is question format, but when sentence-completion format is used for some reason, the completion should always be at the very end of the stem, never internal to it.

Question 1 "After" addresses both of these issues by recasting the stem in question format. In addition, Question 1 "After" represents an improvement because the inclusion of the phrase "that provide the instructions for growth and development" provides helpful context for the student. Finally, the Bloom category "Remember" is indicated in the answer key, and the Difficulty level has been set at "easy" by the subject matter expert (as "factual" is not a clear measure for difficulty).


Example 2

Question 2 "Before"

5. Some employers attempt to make employee jobs more interesting by engaging in various types of job design. In a local pulp mill, management thought they would try having employees exposed to a variety of specialized jobs over time. Each week workers would move from one job to another. By doing this the employees would be exposed to variety in their work and also be less exposed to situations that could result in repetitive strain injuries. This is known as ________________.

a. job enrichment

b. job specification

c. work alienation

d. job rotation

ANS: D

Question 2 "After"

5. Management at a pulp mill thought they could use job design to accomplish two things: make work more interesting and reduce the likelihood of repetitive strain injuries. So they instituted a new program where each week workers would move from one job to another. What approach is this?

a. job enrichment

b. job specification

c. work alienation

d. job rotation

ANS: D    DIF: Moderate    REF: p. 132    OBJ: 4    BLM: Higher Order

Key: ANS = correct answer; DIF = difficulty level, REF = page reference to text, OBJ = Learning Objective in text, BLM = Bloom category

Question 5 "Before" is in sentence-completion format. Question 5 "After" addresses this shortcoming by having its stem in question format.

In addition, the stem of Question 5 "Before" is 79 words long, whereas in contrast, Question 5 "After" is only 48 words long. Naturally, it takes more time for students to read longer items, so having test bank items that are concise rather than wordy allows instructors to put more items on a test. This is not a trivial matter because having more test items increases the overall reliability of the test and thus makes students’ grades more accurately reflect their learning of the course material.


Example 3

Question 3 "Before"

6. The socio-technical system gives:

a. equal attention to technical and social considerations in job design

b. technology and engineering the most weight in job design decisions

c. human considerations the most importance in job design decisions

d. industrial efficiency the greatest weight and is used most often in Scandinavia

ANS: A

Question 3 "After"

6. What does socio-technical systems theory emphasize as a basis for job design?

a. It gives equal attention to technical and social considerations.

b. It gives technology and engineering the most weight.

c. It gives human considerations the most importance.

d. It gives industrial efficiency the greatest weight.

ANS: A    DIF: Moderate    REF: p. 135    OBJ: 5    BLM: Higher Order

Key: ANS = correct answer; DIF = difficulty level, REF = page reference to text, OBJ = Learning Objective in text, BLM = Bloom category

The stem of Question 6 "Before" does not pose an answerable question and thus does not have an adequate focus. Ideally, the stem of a multiple-choice question will pose a clear question or problem, and in most cases, the knowledgeable student will be able to generate one or more possible answers even without looking at the options. In Question 6 "Before," the stem does nothing more than indicate that this item somehow pertains to "the socio-technical system." The question could be about the nature or the definition of the socio-technical system, or about who developed the system, or about the strengths and limitations of the system, or about how the system compares to other systems, or—well, I think you get the idea.

In contrast, the stem of Question 6 "After" poses a focused question that a knowledgeable student could answer even in the absence of the options. Furthermore, each option is properly written as a complete, properly punctuated sentence.

David DiBattista
David DiBattista

David DiBattista received his doctorate from the University of Waterloo and has been a Professor of Psychology at Brock University, St. Catharines, since 1986. After many years of doing research on eating and drinking behaviour, Dr. DiBattista now focuses his efforts on issues relating to teaching and learning. In recent years, he has published articles on various aspects of multiple-choice testing in journals such as The Journal of Experimental Education and The Canadian Journal for the Scholarship of Teaching and Learning. He particularly enjoys helping instructors develop the skills they need to make the best possible use of multiple-choice questions in their courses. He has made dozens of presentations on multiple-choice testing to instructors at colleges and universities across Canada, and he has served as a consultant to a variety of professional organizations and government agencies. Dr. DiBattista has won numerous teaching awards, and in spring of 2007, he was named a 3M National Teaching Fellow.