Why this matters

This exercise demonstrates some of the limitations of ChatGPT - namely, that because of the way that it generates answers, they are partly based on probabilities of things like word order in previous source information. The result is that for information that isn’t widely known or discussed in the training data, the tool will try to fill in the blanks.  When this happens for a person’s biographies, the results are likely very divergent from reality. 

Time to complete: 15-20 minutes

Divide the students into teams of two or three.  If students are likely to know each other, use the Groups feature on Canvas to randomly assign groups of two or three students each.

Instructions:

1. Have each student introduce themselves to the other (sharing nothing more than their name), and write down the name of the other student(s). Note: make sure the students understand using their real names is optional.  If they don’t feel comfortable, have them make up an entirely fictitious name.  Or, ask them to select an obscure topic, location, or thing that they know a lot about. 
2. Using ChatGPT, have each student ask the program who the classmate is.  They can use any prompt they feel is appropriate. 
3. With the new “bio”, each student should now introduce the other, using the information from the bio.  What did it get right?  What did it get wrong?  Why?
4. Students can continue asking questions.  What does the person like to eat?  What are their favorite sports?  What did they do before coming to CMU?
5. After everyone has had a chance to share within the group, come back together and discuss. Ask the students if they would feel comfortable using this as their bio on LinkedIn or on a cover letter. 
6. Create a list of strengths and weaknesses of using ChatGPT.  This can be a good time to share how on the surface the information might appear trustworthy until you go back to the original source.  Ask the students to discuss what this might mean when using ChatGPT to research topics they are less familiar with. 

Why this matters: 

In this exercise we demonstrate the ability of ChatGPT to generate a “flash brief” on a critical, timely topic. Builds on this idea of “ChatGPT Karaoke”.  By introducing a fair amount of uncomfort by trying to present content generated by ChatGPT, students should recognize its limitations when used as an unvarnished generative tool. 

Time to complete: 45 minutes

Divide the students into teams of four.  Before class, give each group a policy brief topic to become familiar with.  A good source is the Day One Project: 

https://fas.org/publications-archive/?s=&i=day-one

Instructions - Generative AI team

1. Using ChatGPT (or other tool identified by the instructor), develop prompts to create the Flash Brief
2. Using only the content developed by the tool, create the policy brief. 
3. Present the policy brief

Instructions - Traditional research team

1. Research the topic as normal
2. Develop the policy brief
3. Present the policy brief

Once the presentations are complete, discuss!  What are the strengths and weaknesses of each approach?

For instructors, this is a good time to talk about how it can be hard for generative AI tools to be specific, accurate, or appropriate for the audience. But, it’s also worth noting how it can be helpful for developing ideas around language, approach, and brainstorming. 

Why this matters: 

This classroom exercise is motivated by providing students with practice and feedback to develop challenging skills with the support of your assignment rubric and ChatGPT. Research shows that the following teaching strategies support learning: 

1. providing transparent evaluation criteria in advance,
2. actively engaging students with examples that illustrate effective habits of mind, and 
3. creating opportunities for targeted low-stakes practice and feedback. 

Common ways instructors enact these strategies include:

1. providing and discussing the grading rubric with students prior to completing assignments,
2. providing anonymized examples of work from previous students, representing a range of quality, and
3. classroom exercises during which students individually or collaboratively evaluate anonymized samples of student work using the assignment rubric and then engage in a class discussion to enhance their understanding of criteria and explore how to improve the samples. 

AI Tools provide additional opportunities to implement these strategies and foster student development and learning, without requiring access to or curation of examples of prior student work. 

Time to complete during class: 60 minutes*

*Note: To shorten the activity, the initial segment marked with an asterisk (*) in the instructions below could be assigned as pre-class, individual work.

Set-up: 

1. Before class, enter your assignment prompt into ChatGPT. Evaluate the output using your assignment rubric to familiarize yourself with pros and cons of the deliverables it produces. Also consider testing tweaks to the assignment prompt. 
2. Divide the students into teams of three or four.  
3. Provide students with the assignment prompt and assignment rubric in Canvas.
4. Provide each group with a set of 4 notecards each labeled with a large visible letter, “A”, “B”, “C” or “D”.  

Instructions for classroom exercise:

1. (10 min*) Group work. Ask students to read and discuss the assignment prompt and grading rubric in their groups to identify and prioritize clarification questions. Groups can enter their questions in a Google Doc that the instructor can monitor and respond to in real time. Provide each group with a dedicated space in the document to write.

2. (10 min) Whole class discussion. Solicit any remaining high priority clarification questions from each group and discuss. 

• Discuss common themes observed across groups’ questions. 
• Discuss areas of challenge experienced by prior students (if you have used this assignment previously). 

1. • How would you rate ChatGPT’s performance on each rubric criterion? 
   • What does ChatGPT do well and not? 
   • Imagine you were the instructor and ChatGPT was your student, what feedback would you provide to improve the output you evaluated?
   • Can you engineer an additional prompt for ChatGPT that would improve the output?

• Take a straw poll. 
  • How would your group rate this deliverable overall? What letter grade would you give it? 
  • Countdown from 3 and ask groups to simultaneously hold up the letter card with their response for all to see. 
    • Verbally summarize the distribution of ratings.
    • Solicit a discussion across groups with different ratings. 
    • Ask groups to explain their rationale based on the assignment rubric.
    • Ask other groups with the same rating to suggest how to improve the deliverable.
  • As criteria are referenced in the discussion, you may ask for additional straw pools. How did you group rate performance on this criterion? Hold up a letter card please.
  • Conclude the discussion by crowdsourcing a pros/cons list. What did ChatGPT do well? Where did it struggle? Record consensus responses on the board under headings of pros versus cons.

• Provides final advice on strategies for success and common pitfalls to avoid.  

*Alternatively, assign Step 1 as individual, pre-class work in Canvas. Start the exercise with Step 2, modifying it as follows:

• (2 min) Turn to a neighbor. Discuss what is most unclear about the assignment prompt or grading criteria in the rubric. 
• (8 min) Call on volunteers to ask their highest priority clarification questions and then discuss.  

Why this matters: 

Research shows that implementing active learning during class sessions improves both learning outcomes and student persistence. However, designing materials for classroom exercises can be time consuming. This approach leverages AI Tools to generate the examples students work on during class, in real time. The premise underlying the practice exercise is that students must find, explain, and correct any errors generated by AI Tools. The sample below focuses on coding/debugging skills. However, it could be adapted for practicing writing or other skills. 

Time to complete during class: 25 minutes

Set-up: 

1. Before class, pilot test prompts in ChatGPT that will purposefully generate output containing an error. For example, generate a block of code in [insert coding language here] to do X, Y, and Z, but that contains at least one error that will prevent it from running successfully.
2. Repeat step 1 above, with different X, Y, and Z functions, as needed.
3. Divide the students into teams of two.
4. Randomly assign tested prompts to student pairs.   
5. Optional: consider adding a scenario to make things more interesting.  For example: “You’ve been tasked with figuring out if a work sample provided by a potential new hire is a real reflection of their coding ability.  Can you figure out the real, working code from the sample generated by AI?”

Instructions for classroom exercise:

• run the code generated to verify an error is present. 
• Diagnose the errors in the code and attempt to debug them, re-running the code to verify success. 

• What kinds of errors did ChatGPT generate, when prompted to make errors?
• How did you figure out it was an error?  
• Are these common human errors in programming?
• What questions did you ask yourself while debugging?
• What was hard or easy about debugging? What did you learn?
• What debugging strategies were most effective? Are those strategies unique to this problem or transferable to other coding tasks? 

• Provide final advice on strategies for success and common pitfalls to avoid.  

Why this matters

This assignment is designed for Writing for Public Policy course, but could easily be adapted for any course that deals with public policy, client consulting, or communicating cutting edge content. 

This discussion prompt attempts to give students a real world scenario to consider about how ChatGPT use (or any other generative AI platform) might impact their professional work. Heinz students are often very focused on professional development and industry expectations, so discussions about the downsides of generative AI in an academic setting might not be as effective as how those downsides might manifest in a professional context. 

By placing the students in a managerial position, it gives them the opportunity to reflect on complications for this kind of AI misuse in multiple directions in a professional hierarchy. The ethics of this situation are intentionally difficult and there is likely not a single good answer, but the intention is to get students to reflect on both the misuse of ChatGPT as well as professional ethos, the need for good research/citation practices, and communication of expectations in a managerial setting. 

Time to complete

Think-Pair-Share discussion in class - 20-25 minutes

5 minutes for students to read the prompt and brainstorm their own answers 

5 minutes to pair up and discuss with a partner or small group

10-15 minute discussion with the class

Discussion with team deliverable (email to interns) - 30 minutes

Present the prompt to the class

5-10 minutes for discussion

10-15 minutes for team to draft deliverable

Set up

Think-Pair-Share discussion in class

1. Provide students with the prompt and discussion questions (ideally as a handout and/or on class slides)
2. Present the prompt to the students and the discussion questions you would like for them to think through. 
3. Prepare discussion questions on slides

Discussion with team deliverable 

1. Provide students with the prompt, discussion questions, and deliverable instructions
2. Create a space for the students to turn in the deliverable (i.e. a discussion thread on Canvas, a Google Drive folder)
3. Decide if students will complete the deliverable in teams or individually, in class or as a homework assignment. 

Prompt

You are a policy analyst at a major policy think tank in Pittsburgh. Yesterday, on short notice, the think tank was asked by a local representative [insert city, state, or national figure here as is relevant] to brief them on current issues involving [insert topic related to course material here]. Your boss assigned you to attend the meeting today, brief the policy maker and determine what kind of help the think tank can provide moving forward.

As is the regular practice at the think tank,  you ask the interns to pull together research and create a 1 page policy briefing document. One of the interns, Brad, sent you the 1 pager late last night. After glancing through the document quickly first thing this morning, you forwarded it to the policy maker and their staff. 

You arrived early for the meeting and while waiting in the conference room for the policy maker to arrive, you take a few moments to read through the briefing document more thoroughly. You decide to read through some of the references to get a deeper understanding of the material but when you click on the link to the first one, it comes up as a 404 error. You click on the second reference link and get a similar error. With a sinking feeling, you click through the rest of the references and realize that none of them actually link to content. 

You pull out your phone and quickly message Brad in Slack to ask if there is an issue with the hyperlinks. After a quick exchange, Brad admits, “I was really pressed for time last night, so I used ChatGPT to write the brief. I didn’t double check the references but now that I’m looking at it, I think ChatGPT made them all up.”

“Did you double check that all the numbers and statistics in the briefing are accurate?” you write back. 

There is a very long pause. You can hear people outside of the conference room; the policy maker and staff are preparing to enter the conference room for the meeting. 

“I just googled the first three statistics listed in the document and it looks like they are incorrect” Brad writes back. “I’m so sorry. I think the whole document is likely bad information.” 

At that moment, the policy maker enters the room. “Thank you so much for coming on short notice! I’m eager to hear what you have to say about this topic.” 

Possible Discussion Questions

1. What do you do? What actions do you take in this immediate moment? (Cancel the meeting? Ask the staff to disregard the briefing document? Apologize?) 
2. How would you explain the situation to the policy maker? To your boss?
3. Who is at fault in this situation? 
4. What impact might these actions have on your reputation and the reputation of your company?
5. As a manager, what should you have done differently? 
6. As a manager, what steps do you need to take to address this situation with Brad the intern? What consequences should he face?
7. As an employee of the think tank, how can you ensure this situation does not happen to any other policy analysts? What policies or procedures might you recommend to upper management?

Instructions 

Think-Pair-Share discussion in class

1. Read through the prompt with the class and give the students time to brainstorm their responses individually to the questions (alternatively, you could provide the prompt and have the students read and respond individually for homework)
2. Ask the students to share their responses with a partner in the class and discuss their reactions. 

Discussion with team deliverable

1. Read through the prompt with the class. 
2. Walk through a few discussion questions with the class, giving them an opportunity to dig in to the ideas. Use the questions that focus on the outcomes you are trying to emphasize. 
3. Break the team into small groups (2-3) and ask them to draft a deliverable document. You can choose one that best fits your course goals or allow students to choose one. They will not be able to complete more than one in class. 
   1. Deliverable prompt 1: In a small group, work to draft an email to Brad the intern which outlines the issues with the brief he delivered. This email should include guidelines on drafting a brief, guidelines for proper use of AI technology, and the consequences (if any) resulting from this incident. 
   2. Deliverable prompt 2: In a small group, draft an email to all the interns at the policy think tank. This email should issue guidelines on the use of AI (especially ChatGPT) when composing documents at work. Your group may decide whether or not to disclose the incident that prompted these guidelines. Your group may also decide what kinds of consequences would be offered for violating this new policy
   3. Deliverable prompt 3: In a small group, draft an email/after action report to your boss at the think tank, giving her a full accounting of the incident. Your email will need to include relevant details about the incident, the steps you have/will take, proposed consequences for the intern and/or for you, and recommendations for new policies or procedures moving forward. 
4. When you have completed your draft, submit the deliverable to the discussion board for today. 
   1. Alternatively, you can have the students brainstorm their solutions together and then ask each student to draft and turn in a deliverable as a homework assignment.