Jordan Usdan

Adjunct Faculty
Heinz College of Information Systems and Public Policy
Spring 2024

94-816 Generative AI: Applications, Implications, and Governance (7-week course)

1. To what extent does generative AI impact research and writing:
   1. efficiency?
   2. performance?
2. Are there different impacts of generative AI across writers with different English language proficiencies or other characteristics? 

Usdan provided students with multi-class instruction of prompt engineering and ways to use genAI (e.g., ChatGPT), as a tool for summarization, information synthesis, research, explanation, idea generation, and more. Classroom demonstrations of potential student applications included using genAI as a prose assistant, editor, thought partner, and critic. Students also practiced completing course assignments with genAI during class sessions. 

All students in the course prepared a writing assignment in each of two conditions: first without genAI and then with genAI. While the order of these conditions did not vary, Usdan counterbalanced (randomized) equivalent policy scenarios assigned to students on each assignment to control for the difficulty of the assignment.

Sample size: Total sample (27 students, assigned to control followed by treatment condition) 

Data Sources:

1. Students’ self-report of writing efficiency, i.e., students tracked the time they spent actively engaged in completing each writing assignment.
2. Students’ two writing assignments, scored with rubrics measuring quality of policy recommendations and supporting arguments, integration of external survey results as evidence, and writing style.
3. Pre/post survey about students’ writing confidence and perspectives on genAI as an educational tool.
4. Post survey about students’ perceived improvement in their writing and attribution of improvement to repeated writing practice versus use of genAI.

1. RQ1a: Students’ self-reported time spent on the writing task reduced by 64.5% with the use of genAI, i.e., students spent roughly 1.5 fewer hours on the writing task.
   
   

   Figure 1. Students spent significantly less time preparing their memo assignment with genAI assistance (M = 66.8 min, SD = 29.1 min) than preparing manually without genAI (M = 191.8 min, SD = 130.3 min) (F(1,23) = 23.15, p < .001, η_p² = .50). Error bars are 95% confidence intervals for the means.

2. RQ1b: Based on grading rubrics, student performance significantly improved from an average of B+ (without genAI) to an A grade (with genAI assistance).
3. RQ2: Changes in performance and writing efficiency did not significantly differ between English-as-a-second-language (ESL) and English-as-a-first language (EFL) students. However, ESL students initially reported lower self-assessed writing competency than EFL students and this difference disappeared by the end of the semester after writing with the assistance of genAI.
   
   

   Figure 2. Students earned higher grades when preparing their memo assignment with genAI assistance (M = 88.3%, SD = 10.3%) than preparing manually without genAI (M = 94.1%, SD = 8.0%) (F(1,25) = 4.74, p = .04, η_p² = .16). This improvement did not differ by English language status (condition x language interaction was nonsignificant: F(1,25) = .12, p = .74). Error bars are 95% confidence intervals for the means.

    

   

   Figure 3. The interaction between English language status and condition on perceived writing competency was marginally significant (F(1,25) = 3.99, p = .06, η_p² = .14). English-as-a-second-language (ESL) students entered the course with significantly lower perceived writing competency (M = 3.07, SD = 1.03) than their English-as-a-first language (EFL) peers (M = 4.25, SD = .62) (t(25) = 3.49, p = .002, d = 1.35). By the end of the semester, this difference had disappeared with ESL students reporting equivalent perceived writing competency (M = 3.73, SD = .96) to their EFL peers (M = 4.08, SD = .67) (t(25) = 1.07, p = .30). Error bars are 95% confidence intervals for the means.

Eberly Center’s Takeaways: 

1. RQ1a: Consistent with previously published research, when using genAI, students completed their assignments in less than half the time. However, time on task was self-reported, which may have been inaccurate. In addition, the genAI-assisted writing always came after a manual writing task. Hence, it is possible that students were able to complete the second assignment faster due to practice with the task itself, in addition to the use of genAI.
2. RQ1b: Students earned significantly higher assignment grades when using genAI but did not differ by ESL status. However, there is a possible practice effect from doing the assignment a second time that could be responsible for improved performance.
3. RQ2: While ESL students entered the class with significantly lower self-reported writing competency than their EFL peers, this difference disappeared by the end of the semester. However, we cannot attribute this to using genAI specifically. It is possible that the repeated writing practice had a greater positive effect on ESL students than on EFL students.
4. This study did not measure learning directly (i.e., the study did not ask students to complete an additional assignment to measure transfer of skills and thus the change in learning when genAI was not available). We acknowledge that observed increases in students’ efficiency and performance therefore do not necessarily mean that the students’ skills improved (i.e., when not using genAI).