Jungwan Yoon

Senior Lecturer
Dietrich College of Humanities and Social Sciences
Fall 2024

76-100 Reading and Writing in an Academic Context (14-week course)

1. To what extent does the use of generative AI for text analysis impact students’
   1. knowledge of genre-specific discourse and linguistic features? 
   2. enjoyment of writing? 
   3. self-efficacy for producing genre-appropriate text?

Yoon provided students with instructions on how to use genAI (ChatGPT) as a pedagogical tool to help support their identification and understanding of linguistic features, focusing on genre awareness. Students practiced using this tool during class for certain assignments, prompting the tool to analyze model text looking for specific rhetorical features. Students then critically evaluated the output to help reinforce their understanding.

Students completed pre-unit mindmap tasks before they practiced their text analysis skills on two units with genAI and two units without genAI. Later on in the semester, students independently completed additional mindmap tasks that corresponded to each unit. Yoon compared students’ performance on these mindmap tasks to assess the impact of the genAI tools. In addition, Yoon compared her students’ pre/post survey responses measuring feelings toward writing and self-efficacy to students from two other sections of the course, taught by two different instructors.

Sample size: 12 Students (alternating treatment and control conditions); 25 students across two additional control sections.

Data Sources:

1. Assessment scores for mindmap tasks from before and after each of four units, rated for number of nodes, depth, and quality (Yoon’s section only).
2. Students’ pre/post surveys reporting (all sections):
   1. self-reported enjoyment of writing
   2. self-efficacy for producing genre-appropriate text 

1. RQ1a: The genAI tool led to greater improvement regarding number of meaningful nodes on mindmap tasks compared to those following a non-genAI practice session, but not for depth or quality. 
   
   

   Figure 1. Students showed significantly more improvement from pre to post on mindmaps created following text analysis with genAI  (M = 4.90, SD = 4.20) compared to those created following text analysis without genAI (M = 0.55, SD = 2.80) for number of nodes, t(9) 2.26, p = .05, g = 0.65. Error bars are 95% confidence intervals for the means. 

2. RQs1b&1c: Students in all sections reported significant increases in their enjoyment of writing and their self-efficacy, regardless of whether they had used genAI or not. 

    

   

   Figure 2. There was a significant main effect of time, showing that students across both conditions reported a significant increase in their enjoyment of writing, F(1,35) = 5.80, p < .05, η_p² = .14. There was no significant time by condition interaction suggesting no impact of genAI on improvement, F(1,35) = .005, p = .94. Error bars are 95% confidence intervals for the means.

   
   

   Figure 3. There was a significant main effect of time, showing that students across both conditions reported a significant increase in their self-efficacy for producing genre-appropriate text, F(1,35) = 22.61, p < .001, η_p² = .39. There was no significant time by condition interaction suggesting no impact of genAI on improvement, F(1,35) = .40, p = .53. Error bars are 95% confidence intervals for the means.

Eberly Center’s Takeaways: 

1. RQ1a: We observed a general trend of greater improvement across all criteria on the mindmap tasks for units using genAI as a practice tool for text analysis. However, only one criterion differed significantly, statistically (number of meaningful nodes). Small sample and large variance for difference scores on these measures makes drawing conclusions difficult. These data suggest that genAI may help students with volume of ideas, however, in this case quantity of ideas did not translate to improvements in quality or depth of ideas.
2. RQ1b&c: Both writing enjoyment and self-efficacy for producing genre-appropriate text increased across all sections, suggesting high impact of the instructors and practice, independent of genAI use.