Omid Saadati 

Adjunct Professor
Integrated Innovation Institute
CMU Integrated Innovation Institute 
Fall 2024

49-750 Integrated Thinking for Innovation (7-week course)

1. To what extent does using genAI affect students’ industry knowledge?
2. When students have the option to use genAI…
   1. What does the distribution of student choice look like? 
   2. To what extent do assignment performance and genAI perceptions differ between students who use it and don’t?
3. To what extent does using genAI affect the development of students’ self-efficacy for industry research?

Outside of class time, teams of students leveraged genAI (Copilot) as a “subject matter expert" to gain insight into their assignment topic. For instance, students may have used genAI to help them identify various parts of the commerce value chain (e.g., retail, payments, e-commerce, m-commerce) for their assigned industry. Saadati provided prompting tips as well as prompt templates.

Saadati had each team of students use genAI on either the first or second team assignment, and not use genAI on the other. Consequently, students served as their own control. Saadati graded both assignments with the same rubric. Additionally, on a third assignment, Saadati offered teams the choice to use genAI or not and tracked students’ use and perceptions.

Sample size: Total sample (22 students in 7 teams completed the control and treatment condition in counterbalanced order) 

Data Sources:

1. Rubric scores from three team-based Miro Board assignments measuring students’ industry knowledge
2. Students’ written reflections following the third team assignment, including whether or not they used genAI and why
3. Self-efficacy survey deployed at pre, post assignment 1, and post assignment 2

1. Using genAI to support industry research did not affect assignment performance.
   
   

   Figure 1. Rubric grade on the first two team-based assignments. There was no main effect of assignment (F(1,5) = .02, p = .91). There was no main effect of order of conditions (F(1,5) = 1.02, p = .36). There was no assignment x order of conditions interaction (F(1,5) = .02, p = .91). Error bars are 95% confidence intervals for the means.

2. When given the choice of whether or not to use genAI on a future assignment, five out of seven teams chose to use genAI. Students individually reflected on the reasoning for this choice, citing perceived productivity benefits most often. Some students also appreciated the ability to collaborate with genAI as a thought partner or during ideation. Some students also showed an awareness of the need to fact check genAI’s output given its propensity to hallucinate. One student, however, preferred to not use genAI for the reason that they wanted to build a deeper understanding of their work. Assignment performance did not differ by the choice of whether or not to use the tool. Regardless of this choice, on average students reported a moderate level of comfort using the tool. 
3. Students’ self-efficacy significantly increased from the beginning of the course to the end of assignment 1. They maintained this self-efficacy to the end of assignment 2 but did not show further growth. Using genAI did not influence the growth of self-efficacy.
   
   

   Figure 2. There was a significant main effect of time (F(2,40) = 13.46, p < .001, η²_p = .40). Students significantly grew in their self-efficacy from pre to post Assignment 1 (p < .001). They maintained this higher self-efficacy to post Assignment 2 (p < .001). There was no additional growth between Assignments 1 and 2 (p = .69). There was no main effect of order of conditions (F(2,40) = .12, p = .74) nor a time x order of conditions interaction (F(2,40) = .79, p = .46). Error bars are 95% confidence intervals for the means.

Eberly Center’s Takeaways: 

1. RQ1 & 2: GenAI use did not affect performance on an assignment that required researching a specific industry. However, as this was a team-based assignment, the small sample size limited the power to detect any possible effect. Additionally, there was a ceiling effect, i.e., all teams performed well (M = 92.0%, SD = 3.6%) limiting the potential to improve performance. When given the chance, most teams decided to use genAI for their third industry assignment. Even though students perceived productivity benefits of using genAI, genAI use did not affect performance on any assignment in the course. Students’ generally positive attitudes about genAI in this Software Engineering course suggest that, at least in some courses, instructors have the potential of incorporating this technology in their classroom with little student resistance.
2. RQ3: GenAI use did not influence the development of self-efficacy, however, students’ self-efficacy significantly improved after completing their first team-based assignment. This finding is similar to self-efficacy literature that finds successfully completing tasks to be a powerful influencer to one’s self-efficacy. If an instructor’s goal is to help foster their students’ self-efficacy, providing such mastery experiences may be a powerful influence regardless of genAI use.