2022 First-Year Engineering Experience

Design projects are an important part of engineering education and are included in many first-year programs. In assessing these projects, educators most often use rubrics where points are given for meeting specific criteria and grades determined through adding up these points which can be time consuming and restrictive. Seldom is a holistic approach taken to assessing student design projects. The desire to employ holistic assessment strategies to student work with open-ended and divergent responses has been widely noted in the literature. Holistic strategies can provide insight into the role of qualities, such as creativity, which are not typically incorporated into standard assessment rubrics.
Adaptive Comparative Judgement (ACJ) is an assessment approach developed to assess student performance in a holistic manner. The ACJ system is composed of three elements; a set of portfolios produced by students in response to an open-ended assignment, a set of judges which may be made up of students or experts in the area, and a pairs system. ACJ is supported by a software solution that adaptively selects pairs of portfolios and presents them to the judges, who judge the work based on professional constructs of performance and capability e.g., creativity. At any one time the judges are making comparisons between only two portfolios, and they select the one of the two that they believe is more creative (in this example). Each portfolio is judged on multiple occasions by various judges, in various pair combinations, providing a broader consensus of the creativity of the work based on multiple perspectives. The output of the judges is a rank-ordered list of the least to most creative portfolios. In studies using the ACJ system to measure various design qualities, high reliability levels have been achieved among judges (~0.9).
Given the dynamic nature of the ACJ assessment tool, there are various benefits this approach can offer engineering educators. The demands placed on an educator to assess many portfolios may be reduced as students can act as the judges and reliably assess their peer’s work. In addition, the ACJ approach can support students in gaining feedback on the standard of their own work relative to that of their peers which is a valuable experience for first-year engineering students. This paper will explore the use and benefits of ACJ for assessing first-year engineering design projects. Further, conference attendees will be provided the opportunity throughout the conference to engage with the ACJ software as judges to experience how this system can work in practice for assessing student design projects.