Sébastien Dubreil

Teaching Professor
Languages, Cultures & Applied Linguistics
Dietrich College of Humanities and Social Sciences
Spring 2024

82-304 French and Francophone Sociolinguistics Oral Language and Storytelling (14-week course)

1. To what extent does the use of two different applications of generative AI impact students’ second language writing?
2. What are students’ perceptions about the value of using generative AI in French language learning?

Dubreil introduced genAI (ChatGPT) as a support for students’ second language writing.  Students used genAI in two different ways. In one condition, Dubreil instructed students to create their initial draft using genAI as a thought partner to write more accurately, for instance to suggest vocabulary or specific language features (e.g., a rhyme, an alliteration) or check the accuracy of sentences. In the other condition, he instructed students to use genAI as a drafting assistant, prompting the genAI to create three different initial drafts that students then refined into a single, final deliverable without the aid of genAI.

All students in the course first completed a control writing assignment without the use of genAI. Students then prepared two writing assignments with genAI as described above (thought partner vs. drafting assistant). Dubreil randomly assigned the order in which students experienced the two genAI conditions.

Sample size: Total sample (6 students, counterbalanced across genAI conditions)

Data Sources:

1. Students' three writing assignments scored with a rubric for linguistic accuracy in vocabulary, grammar, and syntax as well as genre conventions, emotional impact, and originality. 
2. Pre/post surveys about students’ familiarity and comfort with using genAI. The post-survey also asked about students’ perceived added value of using genAI in each of the two conditions.

1. RQ1: GenAI use did not affect performance compared to a control assignment completed without genAI, nor did the specific type of genAI use affect performance. Additionally, the order of conditions did not affect overall course grade.
   
   

   Figure 1. There was no main effect of assignment, F(2, 8) = 1.20, p = .35, no main effect of order of conditions, F(1, 4) = .003, p = .96, and no interaction, F(2, 8) = .62, p = .56. Error bars are 95% confidence intervals for the means.

2. RQ2: More students reported finding the genAI to be more useful as a thought partner while drafting (n = 4) than as a drafting assistant generating multiple reference drafts (n = 2). Their reasons included its effectiveness as a language coach versus losing their voice as a writer when using it as a drafting assistant. When asked about how the tool helped them learn, students reported the benefits of fine-tuning their writing, brainstorming, language support, and ease of use. When asked how they could envision genAI for French language learning, students shared ideas about supporting more advanced learning (i.e., using it for quizzing, practice conversations, and coaching), supporting early learning (i.e., vocabulary and grammar support), generating style or regional example texts, and helping to structure an essay.

Eberly Center’s Takeaways: 

1. RQ1: There was no evidence that using genAI for creating a draft versus feedback/language support affected students’ second language writing. However, it should be noted that the sample size was small (n = 6) and that performance was high in both conditions (average grades were above 90%), both of which make it difficult to detect an effect if there is one.
2. RQ2: Despite there being no performance difference between the two types of genAI uses, students shared thoughtful perceptions about how such a tool interacts with learning a second language. Importantly, in this intermediate writing-focused course, students disliked the sense of losing their voice when writing with the tool. This suggests that incorporating genAI as a second language support tool may have more potential than using it as a drafting tool, especially when students’ voice is an important part of the writing process.