A concept inventory (CI) instrument allows instructors to measure student misconceptions and address them in a course. The instrument is typically a multiple-choice question test designed to assess the understanding of concepts by a student. The questions focus on reasoning and logic rather than on declarative knowledge of the subject matter.
In this paper, we chronicle the development of a concept inventory in Numerical Methods for Engineers as part of a current NSF TUES grant. We focus on the timeline so that readers can themselves follow the intricate process of developing a concept inventory - identifying concepts through experts using DELPHI technique, developing questions, assessing individual questions, and testing for reliability and validity.
February 2014 – March 2014: A workshop was conducted by an expert who has been the chief developer of three concept inventories. The workshop was attended by the three project PIs and two external members of the evaluation team of the grant. The four-hour workshop was administered via two online sessions. The purpose of the workshop was to: 1) identify key concepts and important misconceptions in the domain of numerical methods, 2) review steps required to develop a valid and reliable concept inventory, 3) write reliable and valid concept items for each concept, and 4) decide how to collect and analyze pilot data to measure effectiveness of inventory items (questions and distractors).
March 2014 – June 2014: The PI invited numerical methods instructors from different engineering majors and with varied experience to join a team which would participate in a Delphi methodology to identify the 5-10 most important concepts in Numerical Methods. Thirteen instructors accepted the invitation including the three project PIs. The process was conducted anonymously by the CI expert and took 4 rounds of ranking and discussion to come up with top six concepts. As an example, one concept chosen was “to demonstrate the deep relationship of Taylor series to numerical methods such as derivation of methods, error analysis, and order of accuracy.”
June 2014 – November 2014: The three project PIs developed question stems for each of the six concepts. We drafted 32 questions – with at least 5 questions for each concept. Most questions were written as fill-in-the-blank questions to gather student responses for distractors and others were designed as multiple-choice since they would have otherwise been leading questions. The questions were also answered in a talk-aloud format by two teaching assistants and two students who had recently completed a numerical methods course. Changes were made on the wording of some of the questions based on their feedback.
November 2014 – December 2014: Two first draft tests were developed with 16 questions each to accommodate the class period time of 50-75 minutes. Data were gathered at University A (a large urban university in the Southeast) and University B (a large urban university in the Southwest).
December 2014 – March 2015: Student responses were collated and point-biserial correlation coefficients (a measure of discrimination) and difficulty index values (percentage of test takers who answer a question correctly) were calculated for each question. This allowed us to refine the inventory by identifying questions that were acceptable, those that were for re-testing after revision of , and those that were outright inadequate. The concept inventory along with the statistical data was studied by the CI expert to make sure that we were following the correct process. A few new questions were added to ensure at least 4 questions in each of the 6 topics.
April 2015 – August 2015: Data were now gathered from University B and University C (a historically black university). Using the data from all the questions and implementations, 24 questions (4 in each of the 6 topics) were chosen for the final draft of the concept inventory.
November 2015 – December 2015: The 24-question concept inventory will be tested at University A and University B. This will provide a larger sample size of about 150 at the two institutions. We will continue to determine the point-biserial correlation coefficient and difficulty index for each question and will also compute the Cronbach Alpha for measuring the reliability of the instrument. Validity will be measured using the score on the common multiple-choice question part of the final examination given at Universities A, B and C. These results will be part of the first draft of the paper. The final version of the concept inventory will be available at the time of the conference.
Are you a researcher? Would you like to cite this paper? Visit the ASEE document repository at peer.asee.org for more tools and easy citations.
