Liz Walker and Bonnie Youngs

Liz Walker
Graduate Student Instructor
English
Dietrich College of Humanities and Social Sciences
Fall 2024

Bonnie Youngs
Teaching Professor
Languages, Cultures & Applied Linguistics
Dietrich College of Humanities and Social Sciences
Fall 2024

66-139 DC Grand Challenge Seminar: Reducing Conflict Around Identity and Positionality (14-week course)

1. To what extent does student use of generative AI impact the rate of change in students’ abilities to critically read and analyze academic papers?
2. How does students’ self-efficacy as critical readers and generative AI users change over the course of a semester in which students used genAI as a reading support tool?

Walker and Youngs provided classroom training on how to read academic papers as well as how to engineer genAI prompts and evaluate genAI output. Students then used genAI (Perplexity AI) as a reading support tool prior to class discussions by uploading assigned readings and individually engaging with the genAI as a dialogue partner, asking questions to clarify paper content and potential interpretations of the text.

Walker and Youngs required every student to use the genAI tool for each assigned reading diary/critical analysis assignment in Spring 2024. The comparison group consisted of students enrolled in the same course in the Fall 2023 semester, who completed the same assignments and did not use genAI. Walker and Youngs compared student responses to reading questions used in both semesters. Student self-efficacy was measured at the beginning, middle, and end of Spring 2024 (genAI semester).

Sample size: Treatment (17 students); Control (29 students) 

Data Sources:

1. Students’ responses to assigned reading questions (“diaries”), scored with rubrics for academic reading skills (e.g., reading comprehension, metacognition, critical analysis of text).
2. Surveys of students’ self-efficacy regarding skills using genAI and course learning objectives administered at the beginning, middle, and end of the semester (Spring 2024 only).

1. RQ1a: The rate of development of students’ abilities to critically analyze text across the semester did not differ between semesters when students used genAI to support their reading. 
   
   

   Figure 1. Students significantly improved in their critical analysis abilities in both the Fall 2023 and Spring 2024 semesters, as measured through diaries evaluated on 3 rubric criteria (total rubric score range: 3-9 pts.) at the beginning (Diary 1), middle (Diary 3) and end of the semester (Diary 5), F(2,88) = 200.1, p < .001, η²  = .82. Scores for both sections increased from pre to mid, mid to post, and pre to post, all ps < .001. Although diary scores were always higher for students in the genAI condition (Spring 2024) than in the non genAI condition (Fall 2023), F(1,44) = 5.86, p  < .02, η²  = .12, students in the genAI condition (Spring 2024) started at a significantly higher level, p < .001, but then leveled off to the same extent of critical analysis as students in the non genAI condition (Fall 2023) by Diaries 3 and 5, ps > .05.

2. RQ1b: Students in the Spring 2024 (genAI condition) entered the course with significantly higher confidence in their skills for critically and independently analyzing texts than for using genAI assistance. With repeated practice and targeted instruction on both critical analysis and genAI use, students’ self-efficacy for both types of skill increased to similar, high levels.  
   
   

   Figure 2. Spring 2024 (treatment) self-efficacy measurements. Students entered the semester with significantly lower self-efficacy for genAI-assisted reading compared to their self-efficacy for independent-reading (t(15) = 2.48, p = .03, g = .59), this difference was no longer present by the middle of the semester (t(15) = .29, p = .78), nor the end (t(15) = .74, p = .47). Both self-efficacy for independent-reading (F(2, 30) = 26.81, p < .001, η_p² = .64) and for genAI-assisted reading (F(1.074, 16.107) = 26.52, p < .001, η_p² = .64) increased significantly between each measurement time (all ps <.001). Error bars are 95% confidence intervals for the means. 

Eberly Center’s Takeaways: 

1. RQ1a: There is no compelling evidence that genAI affected the rate of change in students’ abilities to critically read and analyze academic papers. Although rubric scores in the treatment condition were higher than in the comparison group, this difference could be due to a cohort effect at time 1. Specifically, students in the non genAI condition earned the lowest possible scores at time 1, whereas students in the genAI condition earned significantly higher scores on their first diary assignment. This could be because both instructors and students had an additional semester of experience by Spring 2024 (genAI condition).
2. RQ1b: In Spring 2024 (genAI condition), students’ self-efficacy in their ability for independently reading and analyzing articles increased over the course of the semester, as did their self-efficacy for using genAI assistance. Importantly, students entered the course with less confidence in genAI use than independent reading, but exhibited equivalent confidence by mid-semester. Walker and Youngs offered students repeated practice and scaffolded exposure to genAI, suggesting this could be one effective way of building student confidence.