The data used:

Two keys to the OLI approach are the careful collection of learner interaction data from students’ use of the course, and a regular, iterative improvement cycle, designed to leverage quantitative and qualities data from the course’s use. While OLI is specifically designed for data capture and use with regard to the online learning activities, the C@CM team embraced this research-based approach for the entire course, from delivery and attendance to support model and incentive structure.

This approach has left the team with exceptionally rich data from multiple sources, including:

• OLI Learning Interaction data for all learning activities — how did students answer questions? What mistakes did they make? Where did they need help?
• OLI Assessment data — how well did students achieve the learning outcomes as measured by the final exam?
• OLI Timing and Engagement data — how did students use the course materials? For how long?
• Semantic data, tying individual learning activities and questions to student-centered, measurable learning outcomes and sub-skills.
• Learning Estimates produced by the OLI Learning Dashboard.
• Gradebook information, including OLI use, exams, attendance and other graded activities.
• Rich demographic information from the university’s student information system.
• Extensive qualitative data from student surveys.
• Outreach, attendance and other information captured by the C@CM staff throughout the delivery of the course.

Using this data, the C@CM team has worked through four cycles of improvement, with exceptional results. In approaching the course materials, the team has been able to use standard OLI and DataShop tools to identify learning activities and outcomes that are problematic, making appropriate corrections and improvements to the OLI course over time. The team was also able to identify areas where students were already well prepared, reducing the number of activities — these kinds of targeted improvements were able to improve learner productivity, reducing the amount time students needed while improving course success rates. A more thoughtful look at data also revealed where the course development team could best spend their time. During preliminary development, significant time and resource was invested in creating high quality walk-through videos for different course elements. But students’ usage and success data indicated that these videos were not contributing to student success, particularly not when compared to other, more active learn-by-doing opportunities. This allowed the development team to focus more of their time and resource on activities that would best support the learners. Similarly, initial considerations for how to improve the course after preliminary use focused on course pass rates for the entire population — with 91 percent of students passing the course, the team initially looked to create new activities to assist the 9 percent. A closer look at the data, however, revealed a much different picture of student success for students who engaged with the online learning materials:

This idea that course success is dependent on engagement allowed for a different improvement approach. Recognizing that students who work through the online course materials in a timely manner and seek support when necessary pass the course, the team was able to focus attention on appropriate areas — additional options for students that were engaging appropriately but still not passing. A modified incentive structure was created to encourage the non-engaging students to make better use of the course.

Other areas where data has been leveraged to improve the course:
Exploratory analysis of demographic information revealed that a student’s major or college did not have an impact on success — with one exception. Students from two specific colleges were failing the course at nearly four times the rate of other colleges. A deeper analysis revealed two different types of problems — additional support structures were necessary for one college, while scheduling was a major factor for students from the other. Changes in the support model have been very effective in improving success for the one anomalous college — they are now passing at rates similar to those of other colleges. The scheduling challenge has been more difficult to resolve, but the team is working with the college faculty and will be trying some new approaches.

The larger impact

This careful, iterative use of data to improve the learning experience over a four-year period has had a number of valuable effects:

• The early access option allowed a large population of students to successfully complete and pass the course before arriving on campus, which meant they could spend more of their time on campus on core courses.
• Students are entering the core courses better prepared, which means a better use of faculty time.
• The reduced delivery cost, with fewer TAs and lab sessions needed (a more than 50 percent reduction) has meant more resources are able to be devoted to core courses. Instructional teams also get to focus more attention on learners and iterative improvement.
• Labs are composed of learners who most need assistance, which means the TAs’ time is being better used. Data has enabled this in two ways: by supporting the development of an appropriate support model and by providing live dashboards to TAs to better guide their instruction.
• C@CM content has been expanded, as have its learning outcomes, as a result of the improved productivity.
• Overall, CMU has seen a dramatically improved success rate for the course — 98 percent of learners passed C@CM during the most recent academic year.