Bo Zhan

Lecturer
Dietrich College of Humanities and Social Sciences
Fall 2024

82-171 Elementary Japanese I (14-week course)

1. To what extent does genAI impact novice students’ speaking performance? In other words, do students who practiced with genAI improve at a different rate as compared to their classmates who practiced with their peers?
2. To what extent does genAI impact the development of students’ confidence in speaking Japanese?

During a class session, Zhan supplied students with general prompts and guidelines for using the genAI tool (ChatGPT) as a speaking partner. Prompts including asking chatGPT for feedback on grammar corrections, structure of responses, vocabulary, pronunciation while simulating a speaking partner. Additionally, during that class session, students used chatGPT to practice speaking Japanese. 

Zhan provided the same classroom instruction on Elementary Japanese I (e.g. vocabulary, grammar, etc.) across two concurrent sections of the course. Students in both sections of the course completed a baseline speaking assessment with Zhan to act as a pre-measure of student performance. Students then practiced speaking Japanese with either the genAI tool (treatment section), as described above, or a peer (control section). This practice took place during a single class session and continued through a 12-minute homework assignment focused on topics that would later be assessed in the post-assessment.

Two weeks after the pre-assessment, all students completed the post-speaking assessment, which followed the same format as the pre-assessment and was conducted with the instructor. All speaking assessments were audio recorded and later scored by the instructor using a rubric.

Sample size: Treatment (12 students); Control (11 students)  

Data Sources:

1. Rubric scores from recordings of student speaking assessments.
2. Surveys of student’s confidence regarding their Japanese speaking skills.

1. RQ1: There was a significant increase in students’ performance on speaking assessments from pre- to post- across both conditions. There was a significant interaction between condition and time, which indicates that there is a greater increase in performance from the pre- to post-assessment for students who practiced speaking with the genAI tool as compared to students who practiced with their peers. A similar pattern was found for three of seven subskills: comprehensibility, vocabulary, and grammar. These three subskills had significant interactions between condition and time, with the genAI group demonstrating a greater rate of change as compared to the peer group.
   
   

   Figure 1. There was a significant main effect of time, F (1, 21) = 14.59, p = .001,  η_p² = .41, indicating a significant increase in performance on speaking assessments from pre to post across both conditions. There was a significant time x condition interaction, F(1, 21) = 8.43, p = .009, η_p² = .286 suggests that there is a different rate of change between the two conditions with the genAI group improving at a faster rate. While the speaking scores between the two groups are not statistically different at pre, there is a significant difference between the scores of the genAI group (M = 68.60, SD = 13.11) and the peer group at post (M = 58.44, SD = 9.18), F(1, 21) = 4.55, p = .045, η_p² = .18. Error bars are 95% confidence intervals for the means.

2. RQ2: There was a significant increase in students’ self-reported confidence in speaking Japanese from pre- to post-assessment across both conditions. The lack of significant interaction between condition and time indicates that both groups improved their confidence at a similar rate. 
   
   

   Figure 2. There was a significant main effect of time, F (1, 21) = 9.405, p = .006,  η_p² = .309, indicating a significant increase in speaking confidence pre to post across both conditions. There was no significant time x condition interaction, F (1, 21) = .001, p = .973.

Eberly Center’s Takeaways: 

1. RQ1: These results indicate that speaking Japanese with a genAI tool is a promising opportunity for introductory students to practice these skills. Additionally, the instantaneous and on-demand nature of genAI creates an abundance of asynchronous practice opportunities not afforded by peer-to-peer interactions. One caveat to consider is the “time on task”. It is possible that students practicing with the genAI tool could have continued practicing with the tool outside of class time, above and beyond the homework assignment, whereas it is less likely that students sought out their peers for additional speaking practice. 
2. RQ2: Both groups increasing in confidence at approximately the same rates suggests that it is speaking practice in general (rather than the particular genAI tool use) that increases student confidence. As introductory students, they are reporting moderate levels of confidence with room for improvement with additional practice.