2017 ASEE Annual Conference & Exposition

This research paper presents findings from a study on the impact of Learning Management Systems (LMSs) on student success in the classroom. Because of the amount of data available from an LMS, we can examine student performance on a week-to-week basis using data available from using digital course technologies. This is in contrast to the use of end of course final grades, which do not provide researchers with the type of nuanced behaviors that might lead to success or failure throughout the duration of the semester.

Our analysis employed weekly academic classifications in an early warning system (EWS) of students in an undergraduate engineering course at a research-intensive university in the Midwest. The EWS gives a weekly categorization of each student’s performance for each course and designation of performance as a status of ‘‘Encourage’’ (green), ‘‘Explore’’ (yellow- students performing below the course mean), or ‘‘Engage’’ (red- students in the lowest quartile of performance), based on various metrics including: currently available grade data, students’ interaction with online course tools and materials, and students’ performances when compared to their peers in the course.

Coupled with the EWS, we used data from students’ use of various instructional technologies during the course through a digital coaching application called E2Coach. The E2Coach system provides students with a variety of resources including: weekly help messages, exam preparation (before the exam) and reflection (after the exam) tools, a weekly checklist of tasks that will help students prepare for the class, a grade calculator so students can estimate their grade based on past and planned future performance, and various online systems for reviewing academic material.

We use event history methods on students’ performance data from the EWS and their interaction with the E2Coach system to answer the following research questions:

RQ1) (a) Which instructional technologies, if any, help to predict the likelihood of students entering explore or engage classifications? (b) Which instructional technologies, if any, help to predict the likelihood of students exiting explore or engage classifications?
RQ2) Does the timing of students’ use of these technologies precede, coincide with, or lag their experience of academic difficulty? In other words, do students use these tools throughout the duration of the semester, or only when they experience academic difficulty?

Initial findings indicate that students who use the tools starting at the beginning of the term have decreased odds of experiencing academic difficulty during the term, particularly for those students who use the exam preparation guides from the E2Coach system. Additionally, students who utilize checklist items, especially in regards to early submission of course projects, are half as likely to enter the “Explore” or “Engage” classifications. For those students who experience academic difficulty in the “Explore” classification, the use of the exam reflection tools doubles their odds of recovering in the course (i.e., their classification shifts from “Explore” to “Encourage”). The full paper will expand upon the tools used in the E2Coach platform and the rest of the analyses from our full statistical model.