Steven Moore

Graduate Student Instructor
Human Computer Interaction Institute
School of Computer Science
Spring 2024

05-840 Tools for Online Learning (14-week course)

Research Question(s): 

1. To what extent does student use of generative AI while creating micro lessons affect the quality of their lesson designs?
2. How does student self-efficacy for educational design and generative AI use change over the course of the semester?

Teaching Intervention with Generative AI (genAI):

Moore’s students engaged with four interactive, online learning modules on fundamental teaching and learning principles. Each module contained two micro lesson design activities, for a total of eight micro lesson activities, in which he challenged students to apply the learning principles to their practice. For half of the micro lesson activities, he instructed students to use genAI (ChatGPT) as a collaborator in their design process.

Moore implemented two conditions, use of genAI (treatment) or not (control), in the single section of his course. For the first micro lesson assignment in each of four online learning modules, he randomly assigned half of the students to use genAI (treatment) and half of the students not to use genAI (control). For the second micro lesson assignment in each module, students switched to the other condition. Moore compared data sources for each student between conditions and across modules and micro lesson assignments.

Sample size: Total sample (27 students, randomly assigned to alternating treatment and control conditions)

Data Sources:

1. Students’ deliverables from eight micro lesson assignments (half completed with genAI assistance, half without), scored via a rubric with criteria for topic selection, learning objectives, assessments, instruction, and incorporation of the given learning science principle.
2. Pre/post surveys of students’ self-efficacy regarding skills using genAI and educational lesson design.

Findings

1. RQ1: Using genAI for the creation of micro lessons improved performance: Students earned higher rubric scores on the four micro lessons they created with the help of genAI than on the four micro lessons they created without the help of genAI.
   
   
   

   Figure 1. Students earned significantly higher scores on the four micro lessons created with genAI assistance (M = 12.69, SD = .99) than the same students earned on four micro lessons created without the help of genAI (M = 11.72, SD = 1.49), t(26) = 4.72, p < .001, Hedges’ g = .88. Error bars are 95% confidence intervals for the means.

2. Q2: Students entered the course with comparable self-efficacy for creating educational lessons with and without genAI. After three months of using genAI tools for designing educational lessons on half of the micro lessons taught in the course, both types of self-efficacy increased by 11%.

Eberly Center’s Takeaways:

1. RQ1: GenAI assistance conferred an advantage for the design of rubric-scored micro-lessons: Students who prompted ChatGPT to help with generating LOs, instructional text, and assessments outperformed students who generated this content without the help of genAI. Because raters were unaware of conditions at the time of scoring the deliverables, these differences are unlikely to be the result of bias and therefore suggest that genAI as a thought partner benefitted students’ lesson plan design. Students’ deliverables improved when genAI was available to them, but there was no evidence of transfer of skills when students worked without the help of genAI. These data suggest that using genAI increased the quality of deliverables, but using genAI did not persistently alter students’ competencies. 
2. RQ2: Students’ self-efficacy for course-related outcomes and genAI use increased to an equal extent from the beginning to the end of the semester. Since all students completed an equal number of micro-lessons with and without genAI use, it is unclear whether these gains are due to genAI use alone.