Catherine Evans

Graduate Student Instructor
English
Dietrich College of Humanities and Social Sciences
Fall 2024

76-106 Writing about Literature, Art and Culture (7-week course)

Research Question(s): 

To what extent does introducing a Critical GenAI Studies lens in a writing course about art and culture affect:

1. students’ perceptions of generative AI-produced images of CMU student culture?
2. students’ writing performance on interpretative essays about campus culture?

Teaching Intervention with Generative AI (genAI):

Evans implemented a week-long unit on Critical AI Studies in a first-year undergraduate writing course. Students in both the treatment (Section A) and control (Section B) engaged with CMU Archives and Special Collections to understand CMU students’ historical role in producing campus culture. Students then used genAI to produce images based on text from the archival collections and compared these images to the actual historical images. Students in both sections had the option to use genAI in the brainstorming stages of their writing process. However, halfway through the course students in the treatment section engaged in a weeklong conversation about emerging critical perspectives on genAI use, whereas students in the control section spent extra time covering course content not related to genAI.  

Evans taught two sections of the course, one in the first half of the semester and one in the second half of the semester. In the first section, she implemented the teaching intervention as described above (treatment). Students in the second section instead spent extra time covering course content not related to genAI (control). Evans compared the same data sources across the two sections.

Sample size: Treatment (15 students); Control (18 students) 

Data Sources:

1. Student critiques of AI-generated cultural images.
2. Rubric scores on two analytical writing assignments.

1. RQ1: Engaging in a week of Critical AI Studies qualitatively changed the nature of students’ critiques of AI-generated images between the sections. When critiquing AI-generated images of real historical articles, students who were exposed to intervention tended to offer more cultural or thematic critiques of the images (e.g., the image had historically inaccurate portrayals), whereas the students in the control section tended to offer more surface level critiques (e.g., the image had distorted faces).
2. RQ2: Students’ rubric scores on their writing assignments showed no difference between the sections both before (Analysis Assignment 1) and after (Analysis Assignment 2) the critical AI week intervention.
   
   

   Figure 1. There was no significant main effect of time, F (1,31) = 1.86, p = .18, nor section, F (1,31) = 0.46, p = .50, nor a significant interaction, F (1,31) = 0.72, p = .40, to indicate any differences in performance on the writing assignments both before and after the intervention. Error bars are 95% confidence intervals for the means.

Eberly Center’s Takeaway:

1. RQ1 & RQ2: In the context of a seven-week course, a week-long Critical AI Studies intervention qualitatively impacted the depth of students’ AI critiques, but did not translate to observed impacts on students’ analytical writing skills. Intervening to increase students’ genAI competency/literacy may help students view AI-generated cultural content with a more critical lens without observed positive or negative effects on the later cultural analysis essay.