This paper describes our process for developing a data collection method to study the resource networks students use to complete engineering homework problems through network analysis. Network analysis is the study of connections between objects or people. Networks are composed of nodes and edges, where nodes are objects or people and edges represent the connections between nodes. There have been multiple education studies within and outside of engineering that use network analysis to understand peer connections within a course. However, there are few studies that have applied this method to study how students use resources within a course.
The goal of our larger work is to examine the use of resources by first year engineering students to complete homework to understand what resources they use, when they use particular resources, and how they use the resources. To begin addressing this goal we needed to establish a method to collect data that allows us to create and analyze time-series networks. Our time-series networks include information to capture the resources students use (e.g. instructors, teaching assistants, peers, class notes, online sources, textbooks, and discussion boards) and the order in which they use these resources.
The data collection method that we developed was informed by previous studies in the field of engineering education including our prior research. Given the nature of the data that we are collecting and number of responses we want to collect, we decided to create a survey with both descriptive (open-ended) and multiple-selection questions. For this study, we asked students in a first-year engineering course to complete the survey with a specific assignment in mind.
At the beginning of the survey, we included an open-ended item asking students to describe the process that they used to complete the assignment. Students’ responses to this question included information about what resources were selected and the order in which they used the resources. This qualitative data was used to generate a matrix that was imported into the statistical software R to generate a visualization of the network and analyze the network’s structure.
In order to assess the survey’s validity and robustness, we needed to collect redundant data, so we included multiple-selection items asking students to identify the resources they used. These items were followed by open-ended questions asking students to describe how they used these resources. We used these responses to generate additional resource networks for 20 randomly selected students (27% of our sample). These resource networks for each student were then compared to their time-series networks to assess the number of connections and the resources included in each network. Through this comparison, one limitation we identified is that some of the student did not provide sufficient detail in their description of the process they used to complete the homework. Because of this, the resource networks included more connections than the time-series networks. Future work will seek to modify the item asking about the students’ processes in order to capture more detailed data from students.
Are you a researcher? Would you like to cite this paper? Visit the ASEE document repository at peer.asee.org for more tools and easy citations.