Carnegie Mellon University
Eberly Center

Teaching Excellence & Educational Innovation

GAITAR@Scale 5

The impact of generative AI interventions on student performance: A single-site meta-analysis (Spring 2024 - Spring 2026)

Research Question: 

RQ1: What is the impact of genAI-based interventions on students’ academic performance across GAITAR fellowship projects? 

RQ2: How do different factors moderate the impact of students’ genAI use on academic performance? 

RQ2A: How does the nature of students’ genAI use and the presence or absence of a transfer task moderate the impact of genAI on academic performance?

RQ2B: To what extent does adding to or substituting standard teaching strategies moderate  the impact of genAI on academic performance?


Study Overview

To analyze the impacts of students’ use of generative AI (genAI) tools across disciplines, student levels, and learning objectives, we considered all course-level research projects funded by CMU’s GAITAR Fellowship program between Spring 2024 and Spring 2026 for a meta-analytic review. In addition to testing the overall effect of students’ use of genAI on their academic performance, we also investigated if pedagogical choices moderated the effect. In meta-analyses, moderators are variables that affect how well an intervention works. Specifically, we analyzed the moderating effect of the pedagogical integration of genAI (addition vs. substitution to the standard teaching practice) and tool use (thought partner vs. production assistant, defined in Pottmeyer et al., 2026). A production assistant use case involves students utilizing genAI to support the creation of a deliverable (e.g., draft a paper, author code, create an animation). Thought partner use cases involve students using genAI to support their cognitive engagement in a task directed at long-term learning and retention (e.g., receive personalized feedback, role play to pilot test an interview protocol). A transfer task allows for the subsequent assessment of the students’ learning without the use of genAI. As such, type of tool use and the presence or absence of a transfer task are related and we tested them as a single, combined moderator.


Data Sources:

  1. We considered 29 GAITAR fellowship projects conducted between 2024-2026 for the meta-analysis. Of those, 18 studies met the inclusion criteria of including: (a) a student-facing genAI intervention, (b) an experimental or quasi-experimental design with a non-genAI control group, and (c) a direct measure of student learning or performance.

Findings:

  1. RQ1: What is the impact of genAI-based interventions on students’ academic performance across GAITAR fellowship projects? 

    Across 18 studies, genAI tools had a small-to-moderate positive effect on students’ academic performance, g = 0.36, p =.001, 95% CI [0.14, 0.58]. However, we observed significant heterogeneity across studies (I2 = 67.3), strongly suggesting that the effect is moderated by other variables.
  2. RQ2A: How does the nature of students’ genAI use and the presence or absence of a transfer task moderate the impact of genAI on academic performance?

    Tool Use & Transfer significantly moderated the overall effect (See Figure 1). Subgroup analyses revealed that utilizing a thought-partner intervention with a transfer task had positive effects on student performance, as did a production-assistant application without a transfer task. On the other hand, thought-partner use cases without a transfer task did not affect student performance. Zero studies included a production-assistant use case with a transfer task. 



    Figure 1. The three-level thought partner (TP) + transfer task variable significantly moderated the pooled effect, Qbetween = 30.47, df = 2, p < .001. For production assistant (PA) studies with no transfer task (top group), there was a significant effect, g = 0.56, p < .001. For thought partner studies with no transfer task (middle group), there was no significant effect, p = .067. For thought partner studies with a transfer task (bottom group), there was a significant effect, g = 0.57, p <.001.

  3. RQ2B: To what extent does adding to or substituting standard teaching strategies moderate  the impact of genAI on academic performance?

    Pedagogical Integration significantly moderated the overall effect (see Figure 2) such that addition studies had a significant positive effect while substitution studies did not have a significant effect.




    Figure 2. The pedagogical integration variable significantly moderated the pooled effect, Qbetween = 3.92, df = 1, p = .048. For addition studies (bottom group), there was a significant positive effect, g = 0.51, p < .001. For substitution studies (top group), there was no significant effect, p = .643.


Eberly Center’s Takeaways (*adapted from submitted manuscript):

  1. RQ1:  Our results suggest there is an overall positive effect of genAI interventions on student performance. However, the high level of variability among effect sizes suggests that there are important, systematic differences that suggest more nuance. Thus, although findings from ours and others’ meta-analyses (e.g., Chen & Cheung, 2025) point to significant positive effects overall, those metrics should not be interpreted as the final outcomes of interest. Research suggests that there are particular contexts where the use of genAI is more or less effective. To support evidence-based teaching, future research should focus on defining and understanding those specific contexts.
  2. RQ2A: Results for our three-level moderator combining tool use and transfer task (Figure 1) highlight the importance of alignment between the desired outcome for students and how genAI tools are used. If the goal is for students to perform better when there is no genAI, instructors should consider using a thought-partner intervention and assessing students in a non-genAI environment. If, however, the goal is for students to create higher quality deliverables when genAI is an available tool, instructors might consider using a production-assistant intervention and assessing students based on their genAI-aided deliverables.
  3. RQ2B: Results from the pedagogical integration moderator (Figure 2) underscore an important maxim - genAI is not necessarily magic. Ultimately, student outcomes depend on what learning principles are leveraged and how. When a genAI intervention is an addition, it is providing support that did not previously exist, which may or may not be achievable from other teaching strategies or technologies. Specifically, it may not be the “genAI” component that is driving the learning gains, but rather the addition of practice, feedback, and/or time-on-task. The use of genAI as a substitution, however, is replacing learning opportunities that already existed in a course. Here, the absence of consistent positive effects is not surprising. If structured practice and feedback is the active mechanism driving learning, whether it is from genAI or some other thoughtful pedagogical intervention is less important than the fact that the student is actively engaging with those opportunities. Where genAI may add value for learning, in the form of either a substitution or addition, is when students’ use leverages the unique affordances of the technology to create learning opportunities that cannot exist otherwise (e.g., dynamic/on-the-fly scaffolding, instantly available repeated practice attempts) or feasibly be implemented at scale, such as just-in-time feedback (Pottmeyer et al., 2026). Both of these certainly require further study. In other words, thoughtful use of genAI can be a means to further leverage the evidence-based learning practices we already know about, but should not necessarily be considered a novel lever in and of itself.
  4. It is important to note that, for interpreting our Tool Use & Transfer moderator, we did not have any studies that used a production assistant and a transfer task. However, extant studies in the literature demonstrate significant, negative student outcomes for this case (e.g., Bastani et al., 2025). In this case, students’ skill development is not being supported by genAI production assistants. And, if one is assessing (or aiming for) learning (i.e., transfer), rather than quality of a deliverable, then one would not expect to find positive impacts. Consequently, we speculate that such studies, if they had been present in our sample, would appear in Figure 2 above with significantly negative effect sizes.

References

Bastani, H., Bastani, O., Sungu, A., Ge, H., Kabakcı, Ö., & Mariman, R. (2025). Generative AI without guardrails can harm learning: Evidence from high school mathematics. Proceedings of the National Academy of Sciences, 122(26), e2422633122. https://doi.org/10.1073/pnas.2422633122

Chen, S., & Cheung, A. C. K. (2025). Effect of generative artificial intelligence on university students learning outcomes: A systematic review and meta-analysis. Educational Research Review, 49, 100737. https://doi.org/10.1016/j.edurev.2025.100737

 Pottmeyer, L., Hershock, C., Wetzel, D., & Mineroff, Z. (2026). Impact of using generative AI as a “feedback partner” on students’ writing quality. Journal on Excellence in College Teaching, 37. https://celt.miamioh.edu/index.php/JECT/article/view/1305