• To what extent does student use of generative AI tools to make animations impact the technical and aesthetic control over their art?
• To what extent do genAI usage and scaffolding impact the quality of a student writing deliverable?
• To what extent does the early introduction and scaffolded use of generative AI tools for learning Tableau impact students’ performance on course deliverables?
• To what extent does the quality of student-designed interview protocols differ when feedback on first drafts comes from an expert as compared to generative AI?
• To what extent does student use of generative AI while creating micro lessons affect the quality of their lesson designs?
• To what extent does brainstorming with the assistance of generative AI impact: the number of ideas generated? the quality of ideas generated?
• To what extent does utilization of AI tools impact students’ skills for data processing, cleaning, and visualization of large data sets?
• To what extent does generative AI impact research and writing: efficiency? performance?
• To what extent does student use of generative AI impact the rate of change in students’ abilities to critically read and analyze academic papers?

back to filters

College of Engineering

Dietrich College of Humanities & Social Sciences

Heinz College of Information Systems & Public Policy

School of Computer Science

Tepper School of Business

College of Fine Arts

Intro Undergraduate

Advanced Undergraduate

Graduate

Small (20 or fewer students)

Medium (21-50 students)

Large (more than 50 students)

• To what extent does student use of generative AI tools to make animations impact the technical and aesthetic control over their art?
• Does type of feedback interaction (peer vs generative AI) impact the quality of revisions of interview protocols?
• To what extent do genAI usage and scaffolding impact the quality of a student writing deliverable?
• Do various use cases of generative AI yield different linguistic accuracy and complexity in French students’ writing?
• How do first-year writing students conceptualize the relationship of LLMs to cultural production?
• How does student use of generative AI while completing formal writing assignments impact students’ writing performance?
• To what extent does the early introduction and scaffolded use of generative AI tools for learning Tableau impact students’ performance on course deliverables?
• Does generative AI tool us affect equity in student outcomes, giving less-experienced students a better chance to be successful in technical courses?
• To what extent can the use of generative AI tools improve the student peer review process for students in an intermediate level undergraduate writing course?
• What is the impact of debating with generative AI (as compared to debating with a peer) on students’ development of analytical reasoning skills?
• To what extent does the quality of student-designed interview protocols differ when feedback on first drafts comes from an expert as compared to generative AI?
• To what extent does student use of generative AI while creating micro lessons affect the quality of their lesson designs?
• To what extent does brainstorming with the assistance of generative AI impact: the number of ideas generated? the quality of ideas generated?
• To what extent does utilization of AI tools impact students’ skills for data processing, cleaning, and visualization of large data sets?
• How does using generative AI affect students’ industry knowledge as communicated through a team Miro board and a Q&A session with the instructor?
• Does the source of feedback affect performance on a future assignment?
• To what extent does generative AI impact research and writing: efficiency? performance?
• To what extent does using generative AI as a feedback generator improve the quality of student design deliverables in terms of completeness and correctness?
• What is the impact of the use of generative AI for text analysis on students' knowledge of genre-specific discourse and linguistic features?
• To what extent does student use of generative AI impact the rate of change in students’ abilities to critically read and analyze academic papers?
• To what extent does genAI impact novice students’ speaking performance? In other words, do students who practiced with genAI improve at a different rate as compared to their classmates who practiced with their peers?
• How does the way in which a generative AI tool is integrated into the course impact students’ ability to engage in critical thinking over technical troubleshooting, particularly in formulating decision questions and translating stakeholder requirements into analytical models?

back to filters

Projects with Data

Scott Andrew

Adjunct Faculty
Art
College of Fine Arts
Spring 2024

60-424 AI Animation (14-week course)

1. To what extent does student use of generative AI tools to make animations impact their technical and aesthetic control over their art?
2. To what extent does student self-efficacy for animation and genAI skills change over the course of a semester in which students could use genAI to create animations?

Andrew’s students used genAI tools to support their creation of animations, especially during creative editing and stylization decisions. Applications during and between class sessions included generating storyboards, scripts, animated sequences, synthesized voice narration and voice acting, and sound designs, resulting in both narrative and experimental works of animation. The suite of genAI tools included Runway, Deforum Stable Diffusion, ChatGPT, ElevenLabs, Midjourney, Dall-E and more.

Students used a suite of genAI tools across all animation assignments. Using genAI, the first assignment required students to recreate an animation from a previous course for which genAI was not originally used. Andrew compared students’ animations created with (treatment) and without (control) the assistance of genAI. He also measured changes in students’ self-efficacy regarding creating animations with and without genAI throughout the course. 

Sample size: Total sample (13 students completed the control, followed by the treatment condition) 

Data Sources:

1. Students’ animations created without and then with genAI, scored via a rubric to evaluate aesthetic and technical control.
2. Pre/post surveys of students’ self-efficacy regarding skills using genAI and course learning objectives.

1. RQ1: Animations created with genAI scored significantly higher on aesthetic control than those created without genAI, but they did not significantly differ on technical control. 

Figure 1. Students earned significantly higher rubric scores (0-3 points for each criterion) on aesthetic control for an animation created with genAI assistance (M = 3.00, SD = .00) than on the same animation created without the help of genAI (M = 2.33, SD = .49), t(11) = 4.69, p < .001, Hedges’ g = 1.26. Students’ rubric scores did not differ for technical control (t(11) = 1.77, p = .10). Error bars are 95% confidence intervals for the means.

1. RQ2: Students entered the course with significantly lower self-efficacy for using genAI tools to make animations than for animating without genAI. By mid-semester, their self-efficacy for animating with and without genAI no longer differed, and both types were equivalent by the end of the semester, representing a doubling in confidence of using genAI for animation.

Figure 2. Students entered the semester with significantly lower self-efficacy for creating animations with genAI assistance compared to creating animations without genAI assistance (t(12) = 3.08, p = .01, Hedges’ g = .80), this difference was no longer present by the middle of the semester (t(10) = .98, p = .35), nor the end (t(11) = .49, p = .64). Self-efficacy for creating animations with genAI support increased significantly across the semester (F(2, 18) = 9.99, p = .001, ηp2 = .53), specifically from pre to mid, p = .04, and pre to post,  p = .002, and marginally from mid to post, p = .06, whereas self-efficacy for creating animations without genAI assistance remained the same across the semester (F (2, 18) = .50, p = .61). Error bars are 95% confidence intervals for the means. 

Eberly Center’s Takeaways: 

1. RQ1:  Results suggest that genAI use may confer a possible advantage for aesthetic control, but not technical control. However, students recreated an animation done in a previous course without genAI. Consequently, improvements in aesthetic control could also reflect the impacts of repeated practice over time. Lastly, the instructor knew which animations were created with genAI assistance when scoring, which may have biased ratings. 
2. RQ2: While self-efficacy for creating animations without genAI remained stable throughout the semester, students’ self-efficacy for using genAI for animations had doubled by the end of the semester. Repeated practice with various genAI tools for creating animations may have contributed to these increases in student confidence. 

back to filters

Emily DeJeu

Assistant Teaching Professor
Tepper School of Business
Spring 2024

70-340 Business Communications (14-week course)

1. To what extent do generative AI usage and scaffolding impact the quality of a student writing deliverable?

• Section A - used generative AI, with instructor-provided LLM scaffolding
• Section B - used generative AI, without instructor-provided LLM scaffolding
• Section C - no generative AI, no scaffolding

1. To what extent does detailed generative AI-related scaffolding influence students’ perceptions about the utility of LLMs for assisting their growth and development as communicators across an entire semester? (Sections A and B only) 

In one of three course sections, DeJeu scaffolded four mini-lectures showcasing genAI use cases in professional communication contexts (section A). Specifically, these lessons provided instruction and modeling on using ChatGPT or Copilot to revise a document, create model documents, identify "lexical bundles" (i.e., phrases and sentences that are used often in particular genres of writing), and generate ideas. Mini-lectures occurred in tandem with each of the four major writing assignments that included reflection questions and documentation regarding the writing process. Students were instructed to use genAI on their first writing assignment and were allowed to choose whether or not to use it for all subsequent assignments. In a second section (section B), DeJeu also instructed students to use genAI on the first writing assignment with permitted use on subsequent assignments, but she did not provide scaffolded instruction regarding genAI. 

This study had three sections, two of which were taught by DeJeu (sections A and B), and the third was taught by a colleague (section C). In one of DeJeu’s two sections, students received in-class scaffolding for ethical and effective genAI tool use (section A) while students in the other did not (section B). In both sections, students were instructed to use genAI on the first writing assignment. DeJeu compared performance on the first writing assignment, and students’ global perceptions of genAI at the beginning and end of the semester across the two sections. In a third section taught by a colleague (section C), students were not permitted to use genAI on the first writing assignment. DeJeu compared writing performance on this same assignment across all three sections.

Sample size: Section A (24 students); Section B (21 students); Section C (23 students)

Data Sources:

1. One writing assignment, scored with a rubric by three trained coders who were unaware of the study and students’ section. This assignment was scored for various writing skills (e.g., use of rhetorical strategies, concision, coherence).
2. Pre/post surveys of students’ perceptions of genAI’s utility to influence their growth and development as communicators in terms of familiarity, helpfulness, and efficiency (sections A and B only). 

1. RQ1: There was a significant difference in performance on the writing assignment among the three sections. Follow-up comparisons showed no difference between sections A and B, and significant differences between section C and both sections A and B. This overall difference was consistent across all rubric criteria. 

Figure 1. Students’ writing performance was significantly different across the three sections, F (2,64) = 8.33, p < .001, ηp2 = .21. Students in section C (M = 9.67, SD = 2.27) performed significantly lower on the assignment than students in both section A (M = 11.96, SD = 1.75), p < .001, and section B (M = 11.67, SD = 2.27), p <. 01 . Error bars are 95% confidence intervals for the means.

1. RQ2: There was a significant increase in students’ perceived familiarity with genAI tools from pre to post across both sections A and B. There was a marginally significant interaction between section and time, suggesting a slightly greater increase from pre to post in section A (scaffolded genAI use) compared to section B (no scaffolding). There were no significant main effects or interactions for perceived helpfulness or efficiency of genAI tools to assist in their development as communicators.

Figure 2. There was a significant main effect of time, F (1, 38) = 50.51, p < .001,  ηp2 = .57, indicating a significant increase in students’ familiarity with genAI tools from the beginning to the end of the semester. The time x section interaction was marginally significant, F (1, 38) = 3.86, p = .06, indicating that the pre to post change was marginally greater for section A, p < .001, compared to section B, p < .01. Error bars are 95% confidence intervals for the means.

1. RQ1:  Students who were permitted to use genAI (sections A and B) performed significantly better on the writing assignment than students who were not permitted to use genAI (section C), as evaluated by three trained raters who were not informed about the nature of this project. This finding suggests that the use of genAI can help students turn in higher quality deliverables in their communications classes. It is important to note, however, that quality of the deliverable does not necessarily equate to greater learning. Further research is needed to test whether permitted genAI tool use impacts students’ development of underlying writing skills (e.g., on a transfer task completed without genAI), in addition to the quality of a single deliverable.
2. RQ2: Scaffolded instruction on how to use genAI slightly increased students’ perceived familiarity with using genAI to assist with communication-related tasks above what simply using the tool alone did. However, scaffolded instruction and use did not impact students’ perceptions about how helpful genAI is to their growth or efficiency as communicators. This suggests that additional interventions are needed if the goal is to shift students’ perceptions of genAI’s potential to enhance their development as communicators.

Gabriela Gongora-Svartzman

Assistant Teaching Professor
Heinz College of Information Systems and Public Policy
Fall 2024

94-819 Data Analytics with Tableau (7-week course)

1. To what extent does the early introduction and scaffolded use of generative AI tools for learning Tableau impact students’ performance on course deliverables?
2. How do student self-efficacy in data literacy skills change over time in a course in which a generative AI tool was introduced early?

Gongora-Svartzman introduced students to a genAI tool (Explain Data) designed to assist in data exploration. Gongora-Svartzman demonstrated how this Tableau genAI tool can provide an efficient way to view the landscape of potential data analysis pathways in a given project. The teaching intervention provided students with a scaffolded introduction to Explain Data early in the course. In the control course, students were briefly exposed to the tool, without scaffolding, during the last two weeks of the course.

Gongora-Svartzman taught three sections of the course, one control section in Spring 2024 and two treatment sections in Fall 2024. She briefly introduced Explain Data late in the course in the Spring 2024 section (control), whereas she introduced it earlier in the course and in more scaffolded form in the Fall 2024 sections (treatment). She compared data sources from student deliverables across sections.

Sample size: Treatment (55 students); Control (25 students) 

Data Sources:

1. Student deliverables (in-class exercises, final group projects, and case study challenges) from course assignments that required students to perform data analysis. 
2. Pre-post surveys of students’ self-efficacy regarding their data literacy (treatment sections only).

1. RQ1: Student performance in the course during the Fall 2024 (treatment) semester did not differ significantly from student performance during the Spring 2024 (control) semester on any course deliverables.
2. RQ2: Treatment students’ (Fall 2024) self-efficacy for data analysis skills and use of genAI tools for data analysis significantly improved from pre to post, marking an increase of nearly 50% from baseline.

Figure 1. In the Fall 2024 (treatment), students’ self-efficacy for data literacy significantly improved from the beginning (M = 59.82, SD = 21.98) to the end (M = 88.40, SD = 9.07) of the semester, t(41) -8.17, p < .001, g = -1.24. Error bars are 95% confidence intervals for the means.

1. RQ1. Student performance did not change when they were introduced in a more scaffolded fashion to the genAI tool Explain Data earlier in the semester compared to a semester in which students received a more cursory introduction to the tool later in the semester. However, students in the Spring 2024 (control) already evidenced very high performance on all deliverables, limiting the ability to detect improvements.
2. RQ2. Students’ self-efficacy for course-related skills (including the use of genAI tools) did significantly improve from the beginning to the end of the course in Fall 2024 (treatment). These data were not collected during the Spring 2024 (control) section, however, so we cannot say to what extent these pre/post increases are attributable to the intervention.
   

Marti Louw

Director, Learning Media Design Center
Human-Computer Interaction Institute
School of Computer Science
Fall 2024

05-291/05-691 Learning Media Design (14-week course)

1. To what extent does the quality of student-designed interview protocols differ when feedback on first drafts comes from an expert as compared to generative AI? 
2. To what extent does students’ self-efficacy of their interviewing skills change across the semester when receiving generative AI feedback?
3. What are students’ attitudes about simulating an interview with generative AI and receiving generative AI feedback on an interview protocol?

Louw’s Fall 2024 students first used genAI as a coaching tool to receive feedback on their written interview protocols drafts (e.g., subject matter experts, stakeholders, or end-users). Next, students simulated the interview by roleplaying with the genAI using spoken inputs to the tool. Both genAI experiences provided opportunities for students to reflect and iterate on their protocol. For both the written and spoken genAI interactions, Louw provided specific instruction on prompt engineering strategies during class sessions. 

Louw required pairs of students to use genAI for feedback on interview protocol drafts and for simulated interview practice in Fall 2024. On the same assignments, she compared team performance in Fall 2024 to that of teams from Fall 2023, when students did not use genAI and instead received instructor feedback on their protocol draft. Student surveys regarding self-efficacy and other attitudes were deployed at the beginning and partway through the Fall 2024 treatment semester.

Sample size: Treatment (10 teams); Control (9 teams)

Data Sources:

1. Rubric scores for each team’s draft and revised interview protocols (scored after removing indicators of team identity, study condition, and which draft the protocol was)
2. Pre/post surveys of students’ self-efficacy for interviewing skills (treatment only)
3. Students’ written reflections following genAI feedback and interview simulations (treatment only)

1. RQ1: Coding of the protocols showed that whether the students received feedback from the instructor or genAI did not impact the quality of their interview protocol revisions based on total rubric score. However, teams in the genAI semester did score higher on one rubric criterion (number of thematic areas). 

Figure 1. Students’ rubric scores on teams’ deliverables did not significantly differ depending on whether they received feedback from the instructor or from genAI to guide their interview protocol revisions (condition x time: F(1,17) = .12, p = .73). Error bars are 95% confidence intervals for the means.

1. RQ2: GenAI students entered the course with fairly high self-efficacy for their interviewing skills (mean = 78.2 out of 100). After hands-on learning experiences of preparing for and conducting interviews, they maintained this confidence (mean = 82.9) though they did not significantly grow their self-efficacy. 
2. RQ3: Students were slightly positive about the usefulness of genAI for feedback and its ability to stimulate new interview questions. They were less positive about the usefulness of genAI for simulating an interview. Despite this, 75% of the respondents said that they would choose to use the tool to help prepare for future interviews.

1. RQ1: For the most part, genAI did not impact the quality of students’ revised interview protocols, with the exception of helping them generate more thematic areas for their interviews. The genAI condition included two interactions: feedback/coaching and simulation of the interview. We cannot disentangle which of these interactions had the impact on students’ themes or whether both experiences are necessary to achieve this outcome. 
2. RQ2: GenAI students did not significantly grow in their self-efficacy for interviewing skills. This could be due to the relatively high self-efficacy students entered the course with, possibly as a result of prior interviewing experience in 75% of students, or the small sample size. Alternatively, this could indicate that students need additional mastery experiences to build confidence in these skills.
3. RQ3: Despite mixed results in their opinion of how useful genAI was for feedback and simulating an interviewee, the majority of students indicated that they would use the tool in future interviewing tasks (e.g., the tool is a supplement to human thinking without a tangible cost).

Steven Moore

Graduate Student Instructor
Human Computer Interaction Institute
School of Computer Science
Spring 2024

05-840 Tools for Online Learning (14-week course)

1. To what extent does student use of generative AI while creating micro lessons affect the quality of their lesson designs?
2. How does student self-efficacy for educational design and generative AI use change over the course of the semester?

Moore’s students engaged with four interactive, online learning modules on fundamental teaching and learning principles. Each module contained two micro lesson design activities, for a total of eight micro lesson activities, in which he challenged students to apply the learning principles to their practice. For half of the micro lesson activities, he instructed students to use genAI (ChatGPT) as a collaborator in their design process.

Moore implemented two conditions, use of genAI (treatment) or not (control), in the single section of his course. For the first micro lesson assignment in each of four online learning modules, he randomly assigned half of the students to use genAI (treatment) and half of the students not to use genAI (control). For the second micro lesson assignment in each module, students switched to the other condition. Moore compared data sources for each student between conditions and across modules and micro lesson assignments.   

Sample size: Total sample (27 students, randomly assigned to alternating treatment and control conditions)

Data Sources:

1. Students’ deliverables from eight micro lesson assignments (half completed with genAI assistance, half without), scored via a rubric with criteria for topic selection, learning objectives, assessments, instruction, and incorporation of the given learning science principle.
2. Pre/post surveys of students’ self-efficacy regarding skills using genAI and educational lesson design.

1. RQ1: Using genAI for the creation of micro lessons improved performance: Students earned higher rubric scores on the four micro lessons they created with the help of genAI than on the four micro lessons they created without the help of genAI.  

Figure 1. Students earned significantly higher scores on the four micro lessons created with genAI assistance (M = 12.69, SD = .99) than the same students earned on four micro lessons created without the help of genAI (M = 11.72, SD = 1.49), t(26) = 4.72, p < .001, Hedges’ g = .88. Error bars are 95% confidence intervals for the means.

1. RQ2: Students entered the course with comparable self-efficacy for creating educational lessons with and without genAI. After three months of using genAI tools for designing educational lessons on half of the micro lessons taught in the course, both types of self-efficacy increased by 11%.

1. RQ1: GenAI assistance conferred an advantage for the design of rubric-scored micro-lessons: Students who prompted ChatGPT to help with generating LOs, instructional text, and assessments outperformed students who generated this content without the help of genAI. Because raters were unaware of conditions at the time of scoring the deliverables, these differences are unlikely to be the result of bias and therefore suggest that genAI as a thought partner benefitted students’ lesson plan design. Students’ deliverables improved when genAI was available to them, but there was no evidence of transfer of skills when students worked without the help of genAI. These data suggest that using genAI increased the quality of deliverables, but using genAI did not persistently alter students’ competencies. 
2. RQ2: Students’ self-efficacy for course-related outcomes and genAI use increased to an equal extent from the beginning to the end of the semester. Since all students completed an equal number of micro-lessons with and without genAI use, it is unclear whether these gains are due to genAI use alone. 

Carrington Motley

Assistant Professor
Tepper School of Business
Spring 2024

70-415 Introduction to Entrepreneurship (14-week course)

To what extent does brainstorming with the assistance of generative AI impact: 

1. the number of ideas generated?
2. the quality of ideas generated?
3. students’ self-efficacy regarding generative AI use and course learning objectives?

Motley implemented scaffolded brainstorming sessions during class to support ideation for entrepreneurship projects (by individuals). Students then leveraged genAI tools (Copilot) to support both the generation and evaluation of ideas for new business ventures. Individual students created “pitch decks” (slides) to present their ideas to their peers to recruit collaborators to design a business implementation plan. Teams of students then collaboratively designed implementation plans for the entrepreneurship projects chosen.

All students in two concurrent course sections received training on brainstorming techniques. Motley randomly assigned two conditions to sections: students used (treatment) or did not use (control) genAI tools in brainstorming exercises during class. The treatment group received training on brainstorming techniques and genAI use focused on prompt engineering. Control groups received training on brainstorming techniques alone. Data sources were compared between course sections, statistically controlling for variation in students between conditions.  

1. Artifacts of brainstorming sessions, including google docs (control and treatment) and transcripts from genAI use (treatment) 
2. Students’ pitch decks (slides from student presentations), scored using a rubric with criteria for uniqueness of the problem being solved, the solution, and the customer segment targeted 
3. Pre/post surveys of students’ self-efficacy regarding skills using genAI tools and course learning objectives 

1. RQ1: Students did not differ in the number of ideas they generated with or without the help of genAI across two individual brainstorming sessions. However, students who brainstormed without genAI experienced a decline in idea production over time, whereas students who used genAI did not.

Figure 1. Although the average number of ideas generated did not differ across conditions, F (1, 83) = .59, p = .45, η2 = .007, students across conditions experienced a decline in number of ideas generated over time, F(1, 83) = 8.80, p = .004, η2 = .10. However, a closer investigation of the significant time x condition interaction, F(1, 83) = 5.04, p = .03, η2 = .06 suggests that this decline was only true for the non-genAI condition, F(1, 83) = 11.54, p < .001, η2  = .12, whereas students who used genAI did not experience a decline in number of ideas generated over time, F(1, 83) = .32, p = .58, η2 = .001. 

1. RQ2: The quality of entrepreneurial pitches submitted by students did not differ in uniqueness, feasibility, or compellingness across conditions using and not using genAI (see Figure 2). The subset of students who critically evaluated (“filtered”) ideas early on that genAI produced (i.e., they did not automatically submit all ideas suggested by genAI) pitched marginally higher quality ideas to their peers (see Figure 3). 

Figure 2. Students’ entrepreneurial pitch deck scores (uniqueness, feasibility, and compellingness total, out of 6 pts.) did not differ when students used genAI (M = 4.30, SD = .81) or not (M = 4.19, SD = .96) for idea generation. Error bars are 95% confidence intervals for the means. An independent-samples t-test showed that the mean difference was not significant, t(97) = -.66, p = .51.

Figure 3. In the condition that used generative AI, entrepreneurial pitch deck scores (uniqueness, feasibility, and compellingness total, out of 6 pts.) were marginally higher when students critically evaluated and filtered genAI-generated ideas during the first brainstorming phase (M = 4.40, SD = .77) than when they retained every genAI-produced idea (M = 3.75, SD = .89), t (8.82) = -1.94, p = .08, g = -.82. Error bars are 95% confidence intervals for the means. 

1. RQ3: Across a single class session (i.e., two individual brainstorming sessions), students’ confidence in formulating an idea increased significantly, regardless of genAI use. Engaging in brainstorming with genAI significantly increased students’ self-efficacy for using genAI when compared to the condition that did not use the tool.

Eberly Center’s Takeaways: 

1. RQ1: Even though students self-reported that genAI helped them generate more ideas, students who used genAI did not differ in the number of ideas they submitted compared to students who did not use genAI. If anything, students who used genAI submitted slightly (though not statistically significantly) fewer ideas than students who did not use genAI. This finding is in line with existing work that suggests integrating genAI into a brainstorming process does not necessarily offer a safeguard against the kinds of productivity losses experienced in human brainstorming groups (Simkute et al., 2024). However, use of genAI in the present study enabled students to maintain a level of productivity while students who did not use genAI experienced a pattern of exhausting their ability to generate new ideas. This suggests that genAI may be particularly helpful at later stages of the idea-generating process when human capabilities have been maximized.
2. RQ2: 
   1. There was also no evidence that genAI conferred an advantage when it came to the quality of students’ chosen ideas, as measured by final pitch deck scores. Together with RQ1a, these findings echo other research that suggests students overestimate the benefits of genAI for academic performance (Bastani et al., 2024).
   2. On a cautionary note, students who retained all the ideas genAI produced without critically evaluating, or “filtering” them, had a tendency to perform worse on pitch deck scores than students who filtered genAI-produced ideas early on. This was true of students who retained all genAI ideas without adding any of their own ideas, and of students who kept genAI ideas and added on to them. In other words, students who did not filter also did not come up with good ideas on their own. This is consistent with emerging research suggesting that academic performance is reduced when students overly rely on genAI and fail to invest sufficient cognitive effort into evaluating genAI output (Lehmann et al., 2024).
   3. Motley is collecting a second semester of data in Spring 2025 to further explore the impact of critical evaluation of genAI ideas.
3. RQ3: Regardless of genAI use, students reported an increase in their confidence in generating a startup idea after a single class session whereas only students who used genAI in their brainstorming showed an increase in confidence to use genAI to produce desired results. However, confidence was assessed right after the brainstorming activities and it is unclear if these differences persist over time.

Fethiye Ozis 

Fethiye Ozis

Assistant Teaching Professor
Civil and Environmental Engineering
College of Engineering
Spring 2024

12-333 Experimental and Sensing Systems Design and Computation for Infrastructure Systems (14 week course)

Research Question(s): 

1. To what extent does utilization of AI tools impact students’ skills for data processing, cleaning, and visualization of large data sets?
2. What are the attitudes, perceptions, and experiences of students regarding AI-powered tools for data processing and visualization?

Ozis introduced genAI (PerplexityAI) as a possible support tool during students’ multi-week, big-data group project. Students had the option to use genAI during two of their data cleaning and visualization tasks, one completed individually and one in a team. They were not restricted in how they could choose to use the tool but were given some possible uses, such as a coach to provide advice or a tool to detect outliers in the dataset or to provide code to create data visualization plots in Python.

Students could choose to opt into using genAI during their data project, creating a self-selected group of genAI users (treatment) to compare to a group of non-genAI users (control) within the course. Ozis also compared students’ work to a previous iteration of the course in which students were not permitted to use genAI.

Sample size: Self-Selected Treatment (19 students); Self-Selected Control (15 students); Previous-Semester Control (18 students); 12 teams across the three conditions

Data Sources:

1. Students final course grades as well as students’ data visualizations, cleaned datasets, and documentation of process, scored with a rubric for ability to clean, analyze, and visualize large data sets. 
2. Rubric grade for quality of data analysis (following removal of treatment condition and randomization of both semesters’ team deliverable).
3. Students’ reflections on how effective, challenging, and rewarding their data cleaning process was and whether or not they used genAI in their process (treatment semester only).

1. RQ1: Students who chose to use genAI for their data tasks did not perform differently (as measured by final course grades) than students who never chose to use genAI to work with their data nor students who didn’t have the option to use genAI (Figure 1). Additionally, rubric scores for the quality of teams’ data analysis did not differ across conditions.

Figure 1. Students’ grades did not differ statistically, whether they self-selected to use genAI (M = 94.0, SD = 7.3), self-selected to never use genAI (M = 91.3, SD = 9.4), or were required to not use genAI (M = 90.6, SD = 3.2) for their data cleaning and analysis (F(2,49) = 1.23, p = .30). Error bars are 95% confidence intervals for the means. 

1. RQ2: When given the option to use genAI for data tasks, 44% of students chose never to use genAI. Their reasons revealed critical thinking about the added value of the tool (e.g., it can be inaccurate) as well as confidence in their own data skills. Students who chose to work with genAI primarily used it for guidance alone (i.e., opted to clean their datasets without genAI). 

1. RQ1: There was no evidence that using genAI (Spring 2024 self-selected treatment) improved or harmed students’ grades in a course that requires cleaning, visualizing, and analyzing data compared to students who never used genAI (Spring 2024 self-selected control and Spring 2023 control). This null result could be due to alternative factors including high course grades across all students, small sample size, weak manipulation strength, self-selection for whether or not to use genAI (Spring 2024), and minimal instruction on ways to leverage genAI for data tasks.
   1. Teams’ final analysis deliverable was evaluated by a coder who was unaware of the semester and condition. Rubric scores for the quality of their presented analysis did not significantly differ across conditions. Due to the team nature of this final task, the sample size for analysis was extremely small making it difficult to interpret the results as meaningful. 
2. RQ2: Student perspectives about the value of genAI and their confidence in their own data analysis skills could play a role in whether or not a student opts to use genAI when permitted. There is an opportunity for more scaffolding to teach the students the affordances and limitations of this tool to better inform their decisions.

Jordan Usdan

Adjunct Faculty
Heinz College of Information Systems and Public Policy
Spring 2024

94-816 Generative AI: Applications, Implications, and Governance (7-week course)

1. To what extent does generative AI impact research and writing:
   1. efficiency?
   2. performance?
2. Are there different impacts of generative AI across writers with different English language proficiencies or other characteristics? 

Usdan provided students with multi-class instruction of prompt engineering and ways to use genAI (e.g., ChatGPT), as a tool for summarization, information synthesis, research, explanation, idea generation, and more. Classroom demonstrations of potential student applications included using genAI as a prose assistant, editor, thought partner, and critic. Students also practiced completing course assignments with genAI during class sessions. 

All students in the course prepared a writing assignment in each of two conditions: first without genAI and then with genAI. While the order of these conditions did not vary, Usdan counterbalanced (randomized) equivalent policy scenarios assigned to students on each assignment to control for the difficulty of the assignment.

Sample size: Total sample (27 students, assigned to control followed by treatment condition) 

Data Sources:

1. Students’ self-report of writing efficiency, i.e., students tracked the time they spent actively engaged in completing each writing assignment.
2. Students’ two writing assignments, scored with rubrics measuring quality of policy recommendations and supporting arguments, integration of external survey results as evidence, and writing style.
3. Pre/post survey about students’ writing confidence and perspectives on genAI as an educational tool.
4. Post survey about students’ perceived improvement in their writing and attribution of improvement to repeated writing practice versus use of genAI.

1. RQ1a: Students’ self-reported time spent on the writing task reduced by 64.5% with the use of genAI, i.e., students spent roughly 1.5 fewer hours on the writing task.

Figure 1. Students spent significantly less time preparing their memo assignment with genAI assistance (M = 66.8 min, SD = 29.1 min) than preparing manually without genAI (M = 191.8 min, SD = 130.3 min) (F(1,23) = 23.15, p < .001, ηp2 = .50). Error bars are 95% confidence intervals for the means.

1. RQ1b: Based on grading rubrics, student performance significantly improved from an average of B+ (without genAI) to an A grade (with genAI assistance).
2. RQ2: Changes in performance and writing efficiency did not significantly differ between English-as-a-second-language (ESL) and English-as-a-first language (EFL) students. However, ESL students initially reported lower self-assessed writing competency than EFL students and this difference disappeared by the end of the semester after writing with the assistance of genAI.

Figure 2. Students earned higher grades when preparing their memo assignment with genAI assistance (M = 88.3%, SD = 10.3%) than preparing manually without genAI (M = 94.1%, SD = 8.0%) (F(1,25) = 4.74, p = .04, ηp2 = .16). This improvement did not differ by English language status (condition x language interaction was nonsignificant: F(1,25) = .12, p = .74). Error bars are 95% confidence intervals for the means.

Figure 3. The interaction between English language status and condition on perceived writing competency was marginally significant (F(1,25) = 3.99, p = .06, ηp2 = .14). English-as-a-second-language (ESL) students entered the course with significantly lower perceived writing competency (M = 3.07, SD = 1.03) than their English-as-a-first language (EFL) peers (M = 4.25, SD = .62) (t(25) = 3.49, p = .002, d = 1.35). By the end of the semester, this difference had disappeared with ESL students reporting equivalent perceived writing competency (M = 3.73, SD = .96) to their EFL peers (M = 4.08, SD = .67) (t(25) = 1.07, p = .30). Error bars are 95% confidence intervals for the means.

1. RQ1a: Consistent with previously published research, when using genAI, students completed their assignments in less than half the time. However, time on task was self-reported, which may have been inaccurate. In addition, the genAI-assisted writing always came after a manual writing task. Hence, it is possible that students were able to complete the second assignment faster due to practice with the task itself, in addition to the use of genAI.
2. RQ1b: Students earned significantly higher assignment grades when using genAI but did not differ by ESL status. However, there is a possible practice effect from doing the assignment a second time that could be responsible for improved performance.
3. RQ2: While ESL students entered the class with significantly lower self-reported writing competency than their EFL peers, this difference disappeared by the end of the semester. However, we cannot attribute this to using genAI specifically. It is possible that the repeated writing practice had a greater positive effect on ESL students than on EFL students.
4. This study did not measure learning directly (i.e., the study did not ask students to complete an additional assignment to measure transfer of skills and thus the change in learning when genAI was not available). We acknowledge that observed increases in students’ efficiency and performance therefore do not necessarily mean that the students’ skills improved (i.e., when not using genAI).

66-139 DC Grand Challenge Seminar: Reducing Conflict Around Identity and Positionality (14-week course)

1. To what extent does student use of generative AI impact the rate of change in students’ abilities to critically read and analyze academic papers?
2. How does students’ self-efficacy as critical readers and generative AI users change over the course of a semester in which students used genAI as a reading support tool?

Walker and Youngs provided classroom training on how to read academic papers as well as how to engineer genAI prompts and evaluate genAI output. Students then used genAI (Perplexity AI) as a reading support tool prior to class discussions by uploading assigned readings and individually engaging with the genAI as a dialogue partner, asking questions to clarify paper content and potential interpretations of the text.

Walker and Youngs required every student to use the genAI tool for each assigned reading diary/critical analysis assignment in Spring 2024. The comparison group consisted of students enrolled in the same course in the Fall 2023 semester, who completed the same assignments and did not use genAI. Walker and Youngs compared student responses to reading questions used in both semesters. Student self-efficacy was measured at the beginning, middle, and end of Spring 2024 (genAI semester).

Sample size: Treatment (17 students); Control (29 students)

Data Sources:

1. Students’ responses to assigned reading questions (“diaries”), scored with rubrics for academic reading skills (e.g., reading comprehension, metacognition, critical analysis of text).
2. Surveys of students’ self-efficacy regarding skills using genAI and course learning objectives administered at the beginning, middle, and end of the semester (Spring 2024 only).

1. RQ1a: The rate of development of students’ abilities to critically analyze text across the semester did not differ between semesters when students used genAI to support their reading. 

Figure 1. Students significantly improved in their critical analysis abilities in both the Fall 2023 and Spring 2024 semesters, as measured through diaries evaluated on 3 rubric criteria (total rubric score range: 3-9 pts.) at the beginning (Diary 1), middle (Diary 3) and end of the semester (Diary 5), F(2,88) = 200.1, p < .001, η2  = .82. Scores for both sections increased from pre to mid, mid to post, and pre to post, all ps < .001. Although diary scores were always higher for students in the genAI condition (Spring 2024) than in the non genAI condition (Fall 2023), F(1,44) = 5.86, p  < .02, η2  = .12, students in the genAI condition (Spring 2024) started at a significantly higher level, p < .001, but then leveled off to the same extent of critical analysis as students in the non genAI condition (Fall 2023) by Diaries 3 and 5, ps > .05.

1. RQ1b: Students in the Spring 2024 (genAI condition) entered the course with significantly higher confidence in their skills for critically and independently analyzing texts than for using genAI assistance. With repeated practice and targeted instruction on both critical analysis and genAI use, students’ self-efficacy for both types of skill increased to similar, high levels.  

Figure 2. Spring 2024 (treatment) self-efficacy measurements. Students entered the semester with significantly lower self-efficacy for genAI-assisted reading compared to their self-efficacy for independent-reading (t(15) = 2.48, p = .03, g = .59), this difference was no longer present by the middle of the semester (t(15) = .29, p = .78), nor the end (t(15) = .74, p = .47). Both self-efficacy for independent-reading (F(2, 30) = 26.81, p < .001, ηp2 = .64) and for genAI-assisted reading (F(1.074, 16.107) = 26.52, p < .001, ηp2 = .64) increased significantly between each measurement time (all ps <.001). Error bars are 95% confidence intervals for the means. 

Eberly Center’s Takeaways: 

1. RQ1a: There is no compelling evidence that genAI affected the rate of change in students’ abilities to critically read and analyze academic papers. Although rubric scores in the treatment condition were higher than in the comparison group, this difference could be due to a cohort effect at time 1. Specifically, students in the non genAI condition earned the lowest possible scores at time 1, whereas students in the genAI condition earned significantly higher scores on their first diary assignment. This could be because both instructors and students had an additional semester of experience by Spring 2024 (genAI condition).
2. RQ1b: In Spring 2024 (genAI condition), students’ self-efficacy in their ability for independently reading and analyzing articles increased over the course of the semester, as did their self-efficacy for using genAI assistance. Importantly, students entered the course with less confidence in genAI use than independent reading, but exhibited equivalent confidence by mid-semester. Walker and Youngs offered students repeated practice and scaffolded exposure to genAI, suggesting this could be one effective way of building student confidence.

Projects with Data Forthcoming

Brandon Bodily

Assistant Teaching Professor
College of Engineering

49-101 Introduction to Engineering Design, Innovation, and Entrepreneurship (Fall 2024)

1. Does type of feedback interaction (peer vs generative AI) impact the quality of revisions of interview protocols?
2. Does type of feedback interaction impact the development of self-efficacy for interviewing skills?
3. Does type of feedback interaction impact the quality of conducting interviews?
4. What are student attitudes about receiving feedback when role playing with a peer versus generative AI?

Bodily provided students with suggestions and tips for how to engage with the generative AI tool (Co-Pilot). Students then interacted with the tool to conduct a practice interview and elicit feedback on their interview protocols. Students next updated their interview protocols and engaged in real-life interviews as part of the coursework. 

Bodily delivered the same classroom instruction on interview protocol development to all students. All students then crafted an initial draft of an interview protocol. Bodily randomly assigned each student to engage with generative AI, as described above, or to leverage peers to receive feedback on their protocols. Then, during the same class meeting, students practiced their interviewing skills by role playing an interview using their revised protocol either with a peer or with the generative AI tool, depending on the study condition that they were in. All students could revise their protocol after receiving feedback and roleplaying, before conducting the actual interview. 

1. Rubric scores for students’ performance on both draft and final versions of an interview protocol
2. Surveys of student’s self-efficacy regarding their development of an interview protocol and interviewing skills
3. Student reflections on the feedback session, revision process, and interview

Sébastien Dubreil

Teaching Professor 
Modern Languages
Dietrich College of Humanities and Social Sciences

82-304 French and Francophone Sociolinguistics Oral Language and Storytelling (Spring 24)

1. Do various use cases of generative AI yield different linguistic accuracy and complexity in French students’ writing?
2. What are French students’ perceptions of using generative AI to complete writing assignments?

1. Students' two writing assignments scored with a rubric for linguistic accuracy in vocabulary, grammar, and syntax as well as genre conventions, emotional impact, and originality
2. Students’ reflections on their writing process and the quality of their written assignments
3. Pre/post surveys about students’ familiarity, competency, and confidence working with genAI

Catherine Evans

Graduate Student Instructor
English
Dietrich College of Humanities and Social Sciences

76-106 Writing about Literature, Art and Culture (Fall 2024)

To what extent does introducing critical AI studies during the writing process change:

1. how do first-year writing students conceptualize the relationship of LLMs to cultural production?
2. the development of students' attitudes toward art, culture and the humanities? 
3. the way students think about authenticity, voice, diversity, and creativity?

Evans implemented a week-long unit on critical AI studies. Students not only engaged with emerging work in the critical AI studies, but also the CMU Archives and Special Collections to understand CMU student historical role in producing campus culture. Students then used generative AI to produce images based on text from the archival collections and compare the images produced by generative AI to the actual historical images. Students also incorporated theory from cultural studies, with the option to focus on critical AI studies, in their final paper and had the option to use generative AI in the brainstorming stages of their writing process.

Evans taught two sections of the course, one in mini one and one in mini two. In the first section, she implemented the teaching intervention as described above. Students in the second condition instead spent extra time covering course content not related to AI. Evans will compare the same data sources across the two sections.

1. Surveys of student’s attitudes and perceptions toward writing, generative AI, cultural production, and the humanities. 
2. Student reflection assignments on engaging with the archives, generative AI, and their understandings of CMU Tartan identity
3. Students’ rubric scores on the course’s writing assignments.

back to filters

Rebekah Fitzsimmons 

Assistant Teaching Professor
Heinz School of Information Systems and Public Policy

90-717 Writing for Public Policy (Fall 2024)

Research Question(s): 

1. How does student self-efficacy regarding writing and generative AI use change after instruction on and practice with genAI during class discussions? 
2. How does student use of generative AI while completing formal writing assignments impact students’ writing performance?

Fitzsimmons provided the same classroom instruction on and practice activities with generative AI to students in all three sections of her course. Classroom discussions specifically targeted prompt engineering, the evaluation of generative AI outputs, and the ethics of using generative AI in various, realistic professional contexts. 

Fitzsimmons permitted students to use generative AI to support the completion of graded assignments in one of three course sections. The other two sections were not allowed to use AI on formal writing  assignments. On the same assignments, Fitzsimmons will compare students’ performance and changes in self-efficacy across course sections in which generative AI use was permitted and not.

1. Surveys of student’s self-efficacy regarding writing and generative AI use at the beginning, middle, and end of their course.
2. Rubric scores for students’ performance on the course’s second major writing assignment (due after completion of generative AI instruction).

back to filters

 67-272 Application Design and Development (Spring 24)

1. Surveys of students’ background programming experience
2. Students’ documentation of generative AI tool use frequency during coursework
3. Students’ deliverables from coding exercises, exams, and a course project

back to filters

Alan Thomas Kohler

Senior Lecturer
English
Dietrich College of Humanities and Social Sciences

76-270 Writing for the Professions (Spring 24, Fall 24)

1. Pre/post surveys of students’ perceptions of the usefulness of the peer review process
2. Students’ reflections on the feedback process for each project
3. Transcripts of feedback given and received for traditional and AI-based peer review
4. Students’ deliverables for all writing projects, scored with rubrics measuring writing skills

back to filters

Derek Leben

Associate Teaching Professor
Tepper School of Business

70-332 Business, Society, and Ethics (Fall 2024)

1. What is the impact of debating with generative AI (as compared to debating with a peer) on students’ development of analytical reasoning skills? 
2. How does student self-efficacy regarding their analytical reasoning and debate skills change throughout the course and does this vary across experimental conditions? 

Leben provided suggestions and tips for how to engage in a debate with a generative AI tool, about arguments written by students. Next, Leben had students prompt the generative AI tool to: a) give them objections to their argumentative paper from both the same and different normative frameworks, and b) engage in debate with the students about their arguments. 

Leben taught three course sections. Leben provided the same classroom instruction on leveraging normative frameworks to design policies across all sections of his course. Students in all sections drafted an argumentative paper for a policy supported with a normative framework. Then, in each section, Leben randomly assigned students to one of two study conditions. In one condition, Leben implemented the generative AI intervention described above. In the second condition, Leben had students work with peers to elicit objections and engage in debate. The cycle of drafting a paper, receiving feedback, and revising was repeated for two paper assignments, with students remaining in the same treatment conditions. Leben will compare data sources across the two groups in which generative AI use was permitted and not permitted.

1. Rubric scores for students’ performance on both draft and final versions of two major writing assignments (i.e. argumentative papers). 
2. Surveys of student’s self-efficacy regarding their analytical reasoning and debate skills.

back to filters

Omid Saadati

Adjunct Professor
Integrated Innovation Institute
CMU Integrated Innovation Institute 

49-750 Integrated Thinking for Innovation (Fall 2024)

1. How does using generative AI affect students’ industry knowledge as communicated through a team Miro board and a Q&A session with the instructor?
2. How does the timing of generative AI use impact the quality of future assignments? 
3. How many students choose to use genAI on future assignments when the use is optional?

Outside of class time, teams of students leveraged generative AI as a “subject matter expert" to gain insight into their assignment subject, such as analyzing or mapping their assigned industry. For instance, students may use the LLM to help them identify various parts of the commerce value chain. (e.g., retail, payments, e-commerce, m-commerce) for their assigned industry. Saadati provided prompting tips as well as prompt templates for sample questions.

Saadati randomly assigned each team of students to use an LLM (i.e., Microsoft Copilot) on either the second or third course assignment. Consequently, students served as their own controls by completing one of two comparable assignments without generative AI, counterbalancing the order of conditions across student teams. Saadati will grade both assignments with the same rubric. Additionally, on at least one subsequent assignment, Saadati will offer all students the choice of using generative AI or not, and will track which students report using generative AI.

1. Rubric scores from two team-based Miro Board assignments
2. Students’ written reflections following the final course assignment, including whether or not they used generative and why

back to filters

67-262 Database Design and Development (Fall 2024)

Research Question(s): 

1. Does the source of feedback (instructor vs generative AI) affect student attitudes?
2. Does the source of feedback affect performance on a future assignment?
3. How does student self-efficacy for working in SQL change across the course? How does the source of feedback (instructor vs generative AI) impact students’ self-efficacy?

Students uploaded their coding assignment deliverables to a customized, generative AI chatbot called the Intelligent Assessor (designed by Sooriamurthi and Tu). Instructors trained the chatbot on the assignment rubric, their paper detailing the three-step heuristic process of formulating any SQL inquiry, SQL style guidelines, and documentation of mistakes made by previous students. The trained chatbot asked each student questions about specific answers students submitted as part of their assignment, probing them to describe their thinking and decision process. For each student, the chatbot created unique follow up questions encouraging the student to essentially “think out loud”.

After completing an SQL assignment, Sooriamurthi and Tu randomly assigned students to debrief and receive feedback on their assignment from a randomly assigned instructor or a customized generative AI chatbot called the Intelligent Assessor. Students then completed another SQL assignment and debriefed in the counterbalanced condition. Following each debriefing session, students reflected on the experience and the value of the feedback received. 

1. Rubric scores from students’ SQL assignment deliverables
2. Surveys of students’ self-efficacy for working with SQL, administered at the beginning of the course and after both debrief sessions
3. Surveys of students attitudes about the value of feedback received and comfort with the feedback interaction, administered after each debrief session

back to filters

Rafal Wlodarski

Assistant Teaching Professor
Electrical and Computer Engineering
College of Engineering

18-656 Functional Programming in Practice (Fall 2024)

1. To what extent does generative AI, serving as a personal tutor, enhance equity in the learning outcomes for students on engineering projects?
2. To what extent does using generative AI as a feedback generator improve the quality of student design deliverables in terms of completeness and correctness?
3. To what extent do students' attitudes change across the semester (specifically, self-efficacy and trust in generative AI)?

Wlodarski introduced students to a customized generative AI tool to serve as a thought partner for students to receive feedback and explanations of concepts related to domain knowledge (cryptocurrency trading) and the Domain Driven Design framework. This is an attempt to help scale the course and guide students on the team project component.

Wlodarski taught the course during the Spring 2024 and Fall 2024 semesters, to approximately 40 students each. During Spring 2024, students did not have access to the customized chatbot. In Fall 2024, students had optional access to a customized LLM, specifically trained for the course’s context, during the learning process and while completing the course project. Wlodarski will compare the data sources below across semesters. 

1. Student performance on graded assessments. 
2. Pre-post survey on self-efficacy and trust in generative AI

back to filters

Jungwan Yoon

Senior Lecturer
Dietrich College of Humanities and Social Sciences

76-100 Reading and Writing in an Academic Context (Fall 2024)

1. What is the impact of the use of generative AI for text analysis on students' knowledge of genre-specific discourse and linguistic features? 
2. What is the impact of the use of generative AI for text analysis on students’ feelings toward writing? 
3. What is the impact of the use of generative AI for text analysis on students' self-efficacy for producing genre-appropriate text?

Yoon provided students with instructions on how to use generative AI (ChatGPT) as a pedagogical tool to help support their identification and understanding of linguistic features, focusing on genre awareness. Students practiced using this tool for model text analysis during class for certain assignments, prompting the tool to analyze model text looking for specific rhetorical features. Students were then asked to critically evaluate the output to help reinforce their understanding.

For certain units in the course, students practiced their text analysis skills using generative AI, and for others, generative AI was not used. Later on in the semester, students were asked to independently complete transfer tasks that corresponded to the learning units for which they either did or did not use generative AI as a practice tool. Yoon will compare students’ performance on these tasks, reported feelings toward writing, and changes in self-efficacy to assess the impact of the generative AI tools.

1. Rubric scores of transfer tasks and student generated concept maps compared from before and after each unit.
2. Students’ self-reported feelings toward writing.
3. Surveys of student’s self-efficacy for producing genre-appropriate text at the beginning and end of the course.

back to filters

Bo Zhan

Lecturer
Dietrich College of Humanities and Social Sciences

82-171 Elementary Japanese I (Fall 2024)

1. To what extent does genAI impact novice students’ speaking performance? In other words, do students who practiced with genAI improve at a different rate as compared to their classmates who practiced with their peers?
2. To what extent does genAI impact students’ confidence and their motivation in speaking Japanese?

During a class session, Zhan supplied students with general prompts and guidelines for using the generative AI tool (chatGPT) as a speaking partner. Prompts including asking chatGPT for feedback on grammar corrections, structure of responses, vocabulary, pronunciation while simulating a speaking partner. Additionally, during that class session, students used chatGPT to practice speaking Japanese. Students were then encouraged to practice speaking with the generative AI tool outside of class. Students then completed a homework assignment in which they recorded themselves speaking with the generative AI tool.

Zhan provided the same classroom instruction on Elementary Japanese I (e.g. vocabulary, grammar, etc.) across both sections of the course. Students in both sections of the course completed a baseline speaking assessment with Zhan to act as a pre-measure of student performance. Students then practiced speaking Japanese with either the generative AI tool, as described above, or a peer during a class session. Next, students in both groups completed the same assignment in which they recorded their conversation either with the generative AI tool or a partner, depending on which condition they were in. 

1. Rubric scores from recordings of student speaking assignments.
2. Surveys of student’s self-efficacy regarding their Japanese speaking skills.

back to filters

Peter Zhang

Assistant Professor
College of Engineering

19-867: Decision Analytics for Business and Policy (Fall 2024)

1. How does the way in which a generative AI tool is integrated into the course impact students’ ability to engage in critical thinking over technical troubleshooting, particularly in formulating decision questions and translating stakeholder requirements into analytical models?
2. How does the way in which a generative AI tool is integrated impact equity by enabling students with varying levels of technical preparation to participate equally in the critical thinking process?

Zhang delivered scaffolded instruction on how to leverage the generative AI tool to solve decision analytic scenarios. During this instruction, students learned about prompt engineering, fine-tuning existing AI models, and how to use a group of generative AI agents to perform a specific data analysis task. 

Zhang taught two course sections. Zhang provided the same classroom instruction to all students on modeling frameworks and technical topics, such as contextual optimization and optimization under uncertainty. Zhang assigned two study conditions to the sections. In one condition, Zhang implemented the generative AI intervention described above. In the second section, Zhang provided information on general generative AI use without specific guidance on how to apply the tools to decision analysis problems. All students then completed a data optimization course group project. Zhang will compare data sources across course sections in which students were provided with a general introduction to generative AI vs. a more applied and structured approach to using generative AI to solve analysis problems.

1. Rubric scores for students’ performance on a data optimization course project, including an assessment of critical thinking.
2. Student’s self-reported time on task.
3. Surveys of student’s prior experience with the technical concepts and self-efficacy regarding their data analytic skills and their ability to use generative AI to complete analytic tasks.
4. Rubric scores for students’ performance on an in-class quiz.

back to filters