Alan Thomas Kohler

Senior Lecturer
English
Dietrich College of Humanities and Social Sciences
Spring 2024, Fall 2024

76-270 Writing for the Professions (14-week course)

Research Question(s): 

1. To what extent does the use of generative AI tools improve the student peer review process for students in an intermediate level undergraduate writing course?

Teaching Intervention with Generative AI (genAI):

Kohler’s students completed a peer-review feedback process for each of five writing projects in his course. For two of the projects in Spring 2024, students completed this process using a genAI tool (Copilot), rather than another student, as the source of feedback. Students submitted their writing along with the rubric and an instructor-engineered prompt to receive feedback from the AI tool on their writing sample. Additionally, students submitted a carefully refined instructor-engineered prompt to the AI tool to generate a writing sample in order to practice providing feedback. Kohler introduced Copilot during class and provided all pre-engineered AI prompts. For each project, students documented the feedback they received and gave, as well as their perceptions on the usefulness of each experience for learning.

Students used traditional peer review for the first three projects (control) in Spring 2024, but substituted genAI for peer reviewers (treatment) during the fourth and fifth projects. In Fall 2024, this design was counterbalanced, with the first three projects using the genAI-based peer review (treatment) and the fourth and fifth projects using traditional peer review (control). All assignments, rubrics, and genAI prompts were identical each semester. Kohler compared student perceptions of the feedback process and the quality of writing deliverables across conditions.

Sample size: 36 students across two semesters experienced counterbalanced treatment and control conditions.

Data Sources:

1. Students’ drafts for each writing project, scored with rubrics measuring writing skills before and after the review process (scored without knowledge of condition and draft).
2. Students’ survey responses regarding the feedback process for each project.

1. Across all five writing projects, human peer feedback led to significantly greater improvement than did genAI partner feedback.
   
   
   

   Figure 1. Averaging all five projects, students using a peer (M = 7.66 , SD = 1.80) for the review process improved their drafts significantly more than students using genAI (M = 6.57 , SD = 1.22 ) as a review partner, t (34) = 2.13, p < .05, g = 0.70. Error bars are 95% confidence intervals for the means.

2. Across all five writing projects, students engaging in the review process rated both receiving and giving feedback with a human peer to be significantly more helpful than with a genAI partner.
   
   
   

   Figure 2. Averaging all five projects, students rated the experience of receiving feedback from a peer (M = 3.44 , SD = 0.78) to be significantly more helpful than receiving feedback from genAI (M = 3.09 , SD = 0.63), t (33) = 2.35, p <.05, g = .39. Students also rated the experience of giving feedback to a peer (M = 3.53 , SD = 0.74) to be significantly more helpful than giving feedback to a genAI partner (M = 3.04 , SD = 0.80), t (33) = 2.93, p <.01, g = .49. Error bars are 95% confidence intervals for the means.

Eberly Center’s Takeaways:

1. Giving and receiving feedback with a peer helped students significantly improve their writing more than feedback from carefully prompted genAI. The consistent trend of peer review being more effective than genAI was observed across all five projects. This finding suggests that, for intermediate-level undergraduate students, the use of genAI as a review partner may not be as effective at improving students’ writing as collaborating with a human peer for the review process.
2. Students rated both giving and receiving feedback while engaging with a peer to be more helpful than engaging in the process with genAI. The trend of peer review being rated as more helpful was consistent across all five projects. Notably, the pattern of differences in students’ reported helpfulness for each project mirrored that of their actual writing performance, suggesting that students’ perceptions of how helpful the feedback process was for a given assignment is well aligned with subsequent improvements in their revised writing.