Marti Louw

Director, Learning Media Design Center
Associate Dean of Curriculum for IDEATE
Human-Computer Interaction Institute
School of Computer Science
Fall 2024

05-291/05-691 Learning Media Design (14-week course)

Research Question(s): 

1. To what extent does the quality of student-designed interview protocols differ when feedback on first drafts comes from an expert as compared to generative AI? 
2. To what extent does students’ self-efficacy of their interviewing skills change across the semester when receiving generative AI feedback?
3. What are students’ attitudes about simulating an interview with generative AI and receiving generative AI feedback on an interview protocol?

Teaching Intervention with Generative AI (genAI):

Louw’s Fall 2024 students first used genAI as a coaching tool to receive feedback on their written interview protocols drafts (e.g., subject matter experts, stakeholders, or end-users). Next, students simulated the interview by roleplaying with the genAI using spoken inputs to the tool. Both genAI experiences provided opportunities for students to reflect and iterate on their protocol. For both the written and spoken genAI interactions, Louw provided specific instruction on prompt engineering strategies during class sessions. 

Louw required pairs of students to use genAI for feedback on interview protocol drafts and for simulated interview practice in Fall 2024. On the same assignments, she compared team performance in Fall 2024 to that of teams from Fall 2023, when students did not use genAI and instead received instructor feedback on their protocol draft. Student surveys regarding self-efficacy and other attitudes were deployed at the beginning and partway through the Fall 2024 treatment semester.

Sample size: Treatment (10 teams); Control (9 teams) 

Data Sources:

1. Rubric scores for each team’s draft and revised interview protocols (scored after removing indicators of team identity, study condition, and which draft the protocol was)
2. Pre/post surveys of students’ self-efficacy for interviewing skills (treatment only)
3. Students’ written reflections following genAI feedback and interview simulations (treatment only)

1. RQ1: Coding of the protocols showed that whether the students received feedback from the instructor or genAI did not impact the quality of their interview protocol revisions based on total rubric score. However, teams in the genAI semester did score higher on one rubric criterion (number of thematic areas). 
   
   
   

   Figure 1. Students’ rubric scores on teams’ deliverables did not significantly differ depending on whether they received feedback from the instructor or from genAI to guide their interview protocol revisions (condition x time: F(1,17) = .12, p = .73). Error bars are 95% confidence intervals for the means.

2. RQ2: GenAI students entered the course with fairly high self-efficacy for their interviewing skills (mean = 78.2 out of 100). After hands-on learning experiences of preparing for and conducting interviews, they maintained this confidence (mean = 82.9) though they did not significantly grow their self-efficacy. 
3. RQ3: Students were slightly positive about the usefulness of genAI for feedback and its ability to stimulate new interview questions. They were less positive about the usefulness of genAI for simulating an interview. Despite this, 75% of the respondents said that they would choose to use the tool to help prepare for future interviews.

Eberly Center’s Takeaways:

1. RQ1: For the most part, genAI did not impact the quality of students’ revised interview protocols, with the exception of helping them generate more thematic areas for their interviews. The genAI condition included two interactions: feedback/coaching and simulation of the interview. We cannot disentangle which of these interactions had the impact on students’ themes or whether both experiences are necessary to achieve this outcome. 
2. RQ2: GenAI students did not significantly grow in their self-efficacy for interviewing skills. This could be due to the relatively high self-efficacy students entered the course with, possibly as a result of prior interviewing experience in 75% of students, or the small sample size. Alternatively, this could indicate that students need additional mastery experiences to build confidence in these skills.
3. RQ3: Despite mixed results in their opinion of how useful genAI was for feedback and simulating an interviewee, the majority of students indicated that they would use the tool in future interviewing tasks (e.g., the tool is a supplement to human thinking without a tangible cost).