Concept inventories (CIs)—validated tests based on carefully-articulated models of conceptual
knowledge in a field—have been developed for many introductory STEM courses such as Physics
/ Mechanics, Statics, Chemistry, and Electricity and Magnetism. CIs can be powerful research
tools for measuring students’ progression towards expert-level thinking, but can be difficult to
develop for intermediate courses where domain-specific knowledge, problem-solving strategies,
and technical fluency are important learning goals alongside conceptual frameworks. For such
intermediate courses, it is still valuable to develop high-quality, multiple-choice tests to measure
students’ progress towards course learning objectives or to assess the efficacy of instructional
We describe the development process and early results for multiple-choice learning assessments
(MCLAs), drawn from a mix of pre-existing instruments and original content, for four
300-level mechanical engineering courses taken in the junior year: Fluid Mechanics, Mechanics of Materials, System
Dynamics, and Mechatronics. We conducted a series of 16 “think-aloud” interviews with
undergraduate students with a range of prior experience with each subject. Think-aloud
interviews provide rich, qualitative data about student thought processes which is not available
from multiple-choice or even free-response data. Students use a variety of problem-solving
strategies including application of memorized formulas, identification with personal experience,
mental simulation, strategic elimination, and reverse psychology of the presumed test author. We
highlight some challenges in developing MCLAs, including naively-designed questions which
admit correct answers with incorrect reasoning, and schematic diagrams which may
unintentionally cue irrelevant concepts.
Three of the assessments were delivered as low-stakes quizzes in large-enrollment, lecture-based
courses at a private R1 university. Results from the large sample show that many of the
student-held misconceptions identified during the interviews are widely held even after
instruction. Assessment scores are moderately correlated with final exam scores
(0.30 < r < 0.62) and course grades (0.33 < r < 0.49). Data from the pilot tests suggest possible
further enhancements to the assessments.
Are you a researcher? Would you like to cite this paper?
Visit the ASEE document repository at
for more tools and easy citations.