Carnegie Mellon University Website Home Page
 
Skip navigation and jump directly to page content

Pre & Post Tests for Assessing the Effectiveness of an Argument Mapping Tool for Teaching

Instructor: Philosophy: Argument Mapping (Example shared by Mara Harrell)
Scope: Course – Introductory Philosophy, Philosophy Department, H&SS
Assessment Method: Pre- & Post-Test Assessment of Argument Mapping

Motivation:

After teaching several philosophy courses at the undergraduate level, I realized that the fundamental barrier students faced in my classes was an inability to read philosophy, especially primary sources. Most students seemed to get the general idea of a text, but they could not follow the argument. Indeed, most students read the text as a story and did not even recognize that the text presented an argument. This became more evident when students were asked to give presentations; generally, the students merely highlighted the points the author made in the sequence the author made them. This approach is most likely rooted in their high school education, where they have had practice reading and analyzing stories, but have received very little instruction on recognizing, analyzing or creating arguments.

I stumbled upon argument diagramming software that would provide students with a template for reading philosophy and with a map that they can fill in as they read and find relevant parts of the text. I wanted to assess whether presenting visual representations of argument mapping would be an effective way for students to learn how to read and construct philosophical arguments.

Goal:

My goal was to find out if the argument-mapping tool is an effective method for teaching students how to read and construct philosophical arguments.

Methods/Tools: 

A pre and post-test method was used to assess the effectiveness of using argument diagrams. The instructors designed a pre-test and post-test that aligned with the skill objectives for the course. The identified skills were to be able to read an argument and be able to: (1) identify the conclusion and the premises; (2) determine how the premises are supposed to support the conclusion; and (3) evaluate the argument based on the truth of the premises and how well they support the conclusion. Each test consisted of 6 questions. Each question asked the student to analyze a short argument. In questions 1 and 2, the student was only asked to state the conclusion of the argument. Questions 3-6 had five parts: (a) state the conclusion (thesis) of the argument; (b) state the premises (reasons) of the argument; (c) indicate (via multiple choice) how the premises are related; (d) provide a visual, graphical, schematic, or outlined representation of the argument; and (e) decide whether the argument is good or bad, and explain this decision. The arguments used on the pre and post-tests were not exactly the same but they were structurally identical.

Implementation:

Participants

One hundred thirty-nine students enrolled in 80-100: Introduction to Philosophy, during the fall 2004 semester completed both the pre-and post-tests. The data from students who only completed one test was not used in the analysis.

When was/will the data be collected?

The pre-test was administered on the second or third day of class, and the post test was administered at the end of the semester on the last day of classes.

What was the data, how was it analyzed/interpreted?

Most answers were scored on a simple binary scale, 1 for correct, 0 for incorrect. The argument diagrams were coded qualitatively, based on the type of representation used. Graduate students were hired to score the tests, and also to code what kind of visual representation was provided.

Impact/Results:

The results showed that on average the students were gaining quite a bit of knowledge about philosophical arguments between pre-test and post-test. Also, the results showed that the students who were able to do argument diagramming had greater gains than those in other classes.  Thus it seems that argument diagramming is a potentially effective tool for helping students analyze and evaluate arguments.

Comments:

I have worked with colleagues in HCI to develop a customized argument mapping system for philosophy classes that is accessible to other faculty in philosophy. I am also working with faculty in other departments who also teach introductory courses in argument analysis to use pre- and post testing to assess how effective visual diagramming may be in other courses.

CONTACT US to talk with an Eberly colleague in person!