2022 First-Year Engineering Experience

WIP: Adaptive Comparative Judgement as a Tool for Assessing First-Year Engineering Design Projects

Presented at Technical Session M1

Design projects are an important part of engineering education and are included in many first-year programs. In assessing these projects, educators most often use rubrics where points are given for meeting specific criteria and grades determined through adding up these points which can be time consuming and restrictive. Seldom is a holistic approach taken to assessing student design projects. The desire to employ holistic assessment strategies to student work with open-ended and divergent responses has been widely noted in the literature. Holistic strategies can provide insight into the role of qualities, such as creativity, which are not typically incorporated into standard assessment rubrics.
Adaptive Comparative Judgement (ACJ) is an assessment approach developed to assess student performance in a holistic manner. The ACJ system is composed of three elements; a set of portfolios produced by students in response to an open-ended assignment, a set of judges which may be made up of students or experts in the area, and a pairs system. ACJ is supported by a software solution that adaptively selects pairs of portfolios and presents them to the judges, who judge the work based on professional constructs of performance and capability e.g., creativity. At any one time the judges are making comparisons between only two portfolios, and they select the one of the two that they believe is more creative (in this example). Each portfolio is judged on multiple occasions by various judges, in various pair combinations, providing a broader consensus of the creativity of the work based on multiple perspectives. The output of the judges is a rank-ordered list of the least to most creative portfolios. In studies using the ACJ system to measure various design qualities, high reliability levels have been achieved among judges (~0.9).
Given the dynamic nature of the ACJ assessment tool, there are various benefits this approach can offer engineering educators. The demands placed on an educator to assess many portfolios may be reduced as students can act as the judges and reliably assess their peer’s work. In addition, the ACJ approach can support students in gaining feedback on the standard of their own work relative to that of their peers which is a valuable experience for first-year engineering students. This paper will explore the use and benefits of ACJ for assessing first-year engineering design projects. Further, conference attendees will be provided the opportunity throughout the conference to engage with the ACJ software as judges to experience how this system can work in practice for assessing student design projects.

  1. Dr. Clodagh Reid Orcid 16x16http://orcid.org/0000-0002-8593-1730 Technological University of the Shannon: Midlands Midwest [biography]
  2. Dr. Sheryl A. Sorby University of Cincinnati [biography]
  3. Dr. Niall Seery Orcid 16x16http://orcid.org/0000-0003-4199-4753 Technological University of the Shannon [biography]
Download paper (229 KB)

Are you a researcher? Would you like to cite this paper? Visit the ASEE document repository at peer.asee.org for more tools and easy citations.

» Download paper

« View session

For those interested in:

  • Broadening Participation in Engineering and Engineering Technology