Nimer Murshid

Assistant Teaching Professor
Mellon College of Science
CMU-Qatar
Spring 2025

09-111 Nanolegos: Chemical Building Blocks (14-week course)

Research Question(s): 

1. To what extent does the use of generative AI as a scaffolded study partner impact the learning outcomes of non-majors when exposed to a new chemistry topic?
2. To what extent does student self-efficacy change over the course of a semester in an elective undergraduate Chemistry course for non-majors in which generative AI is used as a study partner?

Teaching Intervention with Generative AI (genAI):

Murshid prompted students to use genAI (ChatGPT) to help clarify concepts before the completion of homework assignments on two particularly challenging chemistry topics. For each topic, students completed an in-class activity to practice using the tool as a study partner. He then gave students homework assignments that consisted of two parts: (A) several homework-related ChatGPT prompts to use as a way of learning the topic, with instructions to ask the tool further clarification questions, and then (B) practice problems on the topic that students completed without genAI assistance. Students demonstrated their learning of the homework concepts on an in-class exam completed without genAI assistance.

Study Design:

Murshid taught the course during the Spring 2024 (control) and Spring 2025 (treatment) semesters. In the Spring 2024 semester, he did not permit students to use genAI. In Spring 2025, Murshid integrated the use of genAI as a study partner as described above. He compared students’ performance on two quizzes, two homework assignments, and one exam across the semesters. Students’ pre and post self-efficacy for course learning objectives and genAI use was also measured for the Spring 2025 (treatment) section only.

Sample size: Treatment (16 students); Control (16 students) 

Data Sources:

1. Students’ performance on a course exam, two quizzes, and two homeworks.
2. Pre/post surveys of students’ self-efficacy for skills related to using genAI and course learning objectives (treatment semester only).

Findings:

1. RQ1: Interacting with genAI as a study partner improved students’ performance on the exam and one of two quizzes in the treatment semester (S25) compared to the control semester (S24). Interacting with genAI as a study partner did not impact performance on the two homework assignments.
   
   

   Figure 1. Students’ scores on Exam 2 (controlling for GPA), which covered all topics targeted by the intervention, were higher in S25 (treatment; M = 85.62, SE = 2.87) than in S24 (control; M = 75.07, SE = 2.87), F(1, 29) = 6.57, p = .02, η²_p = .19. Error bars are 95% confidence intervals for the means.

2. RQ2: There was a significant increase in self-efficacy for course learning objectives from pre to post, but not for genAI use (treatment only).
   
   

   Figure 2. In Spring 2025 (treatment), students’ self-efficacy for genAI tool use did not increase significantly from pre (M = 72.90, SD = 18.93) to post (M = 76.70, SD = 15.52), t(14) = .87, p = .40, but it did increase significantly from pre (M = 63.27, SD = 25.12) to post (M = 85.66, SD = 15.23) for course learning objectives (LOs), t(10) = 2.40, p = .04, g = .67. Error bars are 95% confidence intervals for the means. 

Eberly Center’s Takeaways:

1. RQ1: Students in S25 (treatment) showed higher performance than their peers in S24 (control) on 2 out of 5 deliverables. Notably, this included the exam covering both topics the intervention targeted. Historically in the course, there have been performance differences between business and computer science students. The original research question aimed to test if the intervention differentially affected these student populations. However, during the treatment semester only two business majors enrolled in the course, which was too small a sample to meaningfully test this question. As such, it is unclear whether the intervention was equally effective for all majors, and further data collection is needed for this more nuanced analysis. All together, these results suggest that using genAI as a study partner with scaffolded support is potentially effective for helping non-majors learn challenging chemistry topics.
2. RQ2: Although there was a generally consistent pattern of increase for self-efficacy items related to course LOs, this was not true for genAI-related self-efficacy. However, students entered the course with relatively high self-efficacy in this domain, allowing for less room to grow. When looking at an item-level analysis, one of the genAI items showed a significant increase, suggesting students felt a lot more confident in creating tailored prompts after practicing this skill consistently. A different item showed a significant decrease (evaluating genAI output for accuracy). Students expressed in qualitative responses that they found ChatGPT helpful as a learning tool, but that they also discovered the need for cautious engagement with the tool, as they found some mistakes in its output. This suggests that students may have overestimated their ability to evaluate genAI output initially.