2016 ASEE Annual Conference & Exposition

Survey questions are typically presented in one of two formats: closed-ended or open-ended. Despite the widespread use of open-ended survey questions in educational research, making sense of the answers to such questions remains a challenge. Whereas closed-ended questions lend themselves to well-known quantitative data-analysis techniques, open-ended questions have a tendency to illicit responses that are more difficult to analyze. Open-ended survey questions are advantageous because they allow researchers to explore responses when 1) answers to a question are not known in advance or 2) narrative responses are more useful. Open-ended survey questions are also used as “catch-all” or venting questions, giving respondents an opportunity to discuss topics that may not have been addressed by closed-ended questions. Given the varied use of the open-ended question format, researchers are often unsure how to use the results, if at all: survey research literature does not provide sufficient examples illustrating suitable data-analysis procedures for this question format. This challenge is further exacerbated when the respondents are reflecting on different contexts, such as different cases or research sites.

The purpose of this paper is to present an anchored open-ended survey approach as a solution for combating the disadvantages of open-ended survey questions while preserving the capacity to explore new or unanticipated responses. The approach uses closed-ended questions to anchor the responses to open-ended questions. An example of an anchored open-ended question is prompting students to indicate whether their interactions with faculty/staff were impacted positively, negatively, or neutrally by a student support center before asking them to provide specific examples of how their experience interacting with faculty/staff was influenced by a student support center.

In this paper, we illustrate the use of anchored open-ended survey questions as a method for multiple case study analysis. First, we present an overview of open-ended survey questions and their usefulness in qualitative research. We then discuss the lead author’s use of anchored open-ended questions in a multi-case study to investigate the impact of co-curricular support on the student experience. Next, we discuss the approach used to analyze the open-ended survey questions and the role of the anchors in completing this process. Lastly, we address the advantages and disadvantages of using open-ended questions in multiple case study analysis. The results of this application demonstrate several ways that engineering education researchers can use anchored open-ended survey questions. Although we applied the anchored open-ended question format to a multiple case study analysis, we believe that additional research approaches could use this data collection and analysis technique as well.