Rebekah Fitzsimmons 

Assistant Teaching Professor
Heinz School of Information Systems and Public Policy
Fall 2024

90-717 Writing for Public Policy (7-week course)

Research Question(s): 

1. To what extent does student use of generative AI while completing formal writing assignments impact students’ writing performance?
2. To what extent does student self-efficacy regarding writing and generative AI use change after instruction on, and practice with, genAI during class discussions?  

Teaching Intervention with Generative AI (genAI):

Fitzsimmons taught three sections of the course, two in the first half of the Fall 2024 semester (control, sections A and B) and one in the second half of the semester (treatment, section C). She provided the same classroom instruction on, and practice activities with, genAI to students in all sections. Classroom discussions specifically targeted prompt engineering, the evaluation of genAI outputs, and the ethics of using genAI in various realistic professional contexts (e.g., writing policy papers). 

Fitzsimmons permitted, but did not require, students to use a genAI tool of their choice to support the completion of graded writing assignments in one of three course sections (treatment). Students in this section were then able to opt into using the tool (self-selected treatment) or not (self-selected control). The other two sections (previous-section control) were not allowed to use genAI on course writing assignments. Fitzsimmons compared students’ performance on the second, most central, writing assignment, and she compared changes in self-efficacy across course sections in which genAI use was permitted vs. not.

Sample size: Self-Selected Treatment (11 students); Self-Selected Control (14 students); Previous-Section Control (47 students across two sections)

Data Sources:

1. Rubric scores for students’ performance on the course’s second major writing assignment (completed after genAI instruction).
2. Surveys of students’ self-efficacy regarding policy writing and genAI use at the beginning, middle, and end of their course.

1. RQ1: Students who opted to use genAI assistance for their second writing assignment (“Policy Recommendation”) did not perform differently (as measured by final rubric score) than students who chose not to use genAI, or students who were not permitted to use genAI (Figure 1).
   
   
   

   Figure 1. In Fall 2024, students’ rubric scores (0-200 points) did not differ statistically by whether they were not able to use genAI (previous section control, n = 47, M = 182.57, SD = 11.99), chose not to use genAI when permitted (self-selected control, n = 14, M = 179.64, SD = 8.86), or opted to use genAI when permitted (self-selected treatment, n = 11, M = 180.55, SD = 8.98) on their policy recommendation writing assignment, F(2,69) = .45, p = .64. Error bars are 95% confidence intervals for the means. 

2. RQ2: Both types of self-efficacy increased significantly over the course of the semester, regardless of section (genAI permitted vs. not). 
   
   
   

   >Figure 2. Students’ self-efficacy for writing policy documents increased across time, F(1.40, 74.007) = 44.59, p < .001, η_p² = .46, from pre to mid, p < .001, mid to post, p < .001, and pre to post, p < .001. Self-efficacy did not differ by condition (genAI permitted vs. not), F (1,53) = 3.18, p =.08, nor did the change across time differ by condition, F (1.40, 74.007) = 2.64, p = .10. Error bars are 95% confidence intervals for the means.
   
   

   Figure 3. Students’ self-efficacy for effective and ethical genAI use increased across time, F(1.54, 81.52) = 22.07, p < .001, η_p² = .29, from pre to mid, p = .002, mid to post, p <.001, and pre to post, p < .001. Self-efficacy did not differ by condition (genAI permitted vs. not), F (1,53) = .99, p =.32, nor did the change across time differ by condition, F (1.54, 81.52) = 1.36, p = .26. Error bars are 95% confidence intervals for the means.

Eberly Center’s Takeaway:

1. RQ1: There was no evidence that students who chose to use genAI on writing assignments when permitted (Fall 2024, self-selected treatment) performed differently than students who were not permitted to use genAI (Fall 2024, control) or students who were permitted to use genAI but chose not to (Fall 2024, self-selected control). However, the extent to which students used genAI varied and may not have contributed meaningfully to their performance. In submission reports, students indicated using genAI most frequently for help with grammar, structure, and formatting, which was only a small portion of the overall grade and may not have affected the final product enough in a sample of more experienced writers in this graduate course. Additionally, the presence of  high grades throughout suggests that students may have had limited room for improvement with genAI assistance. Lastly, although genAI was a permitted tool for all students in the treatment section, only some students self-selected into using genAI, limiting causal conclusions about the effectiveness of genAI as a tool for improving writing.
2. RQ2: Self-efficacy for writing policy documents and self-efficacy for ethical and efficient genAI use both increased significantly over the 7-week course, regardless of whether students were permitted to use genAI on writing assignments (Section C, treatment section) or not (Sections A and B, control sections). It is feasible that repeated writing practice and detailed instructor feedback contributed to increases in writing self-efficacy in all sections. However, there was also no difference across sections in the increases in self-efficacy of genAI use over time. This suggests that the genAI instructions students received in all sections may have been sufficient to raise self-efficacy in students, regardless of whether they elected to use the tool on assignments.