Tepper School Research: On-Demand Courses Increase Revenue but Lower Engagement

The online education industry has grown exponentially over the last decade, fueled in part by the COVID-19 pandemic. In a new study, researchers examined whether online courses provided by online educational platforms should follow a scheduled format, like traditional classes, or be available on demand. The study found that offering courses on demand increased revenue for those platforms but reduced student engagement.

The study, by researchers at Carnegie Mellon University and the University of Pennsylvania (Penn), is published in the Journal of Marketing.

“Following trends in entertainment streaming services, online educational platforms are increasingly offering users flexible on-demand content options,” noted Joy Lu, assistant professor of marketing at Carnegie Mellon’s Tepper School of Business, who led the study.

“Therefore, it is important to understand how the timing of releasing content affects learning behaviors and revenue drivers.”

The online education industry grew from 300,000 consumers in 2011 to 220 million in 2021. Traditional universities and institutions are increasingly adopting hybrid course formats, which allow greater accessibility and flexibility in the ways classes are administered and taken.

In this study, researchers sought to determine whether online educational platforms should release content through a scheduled format that resembles a traditional university course, with a subset of lectures and quizzes available at the start of each week, or on-demand as entertainment firms do, with all material immediately available upon registration.

They studied more than 67,000 users taking an introductory marketing course on Coursera, a leading online learning platform. The platform switched from a scheduled format to an on-demand release format in 2015 with the content staying the same. All users could take the course for free or pay for a completion certificate, either as a one-time fee in the scheduled format or as a monthly subscription in the on-demand format.

Switching the course format from scheduled to on-demand increased short-term revenue by doubling the percentage of paying users (from 14% to 28%). However, the switch resulted in users completing fewer lectures and performing worse on quizzes. Changing to the on-demand format also harmed users’ engagement with the platform, with individuals taking fewer additional courses on the platform six months later. Interestingly, users in the on-demand format seemed to be “binge-learn” strategically by consuming large amounts of content in a few sittings but still performing well in the course.

“While on-demand formats can boost revenue by bringing in more paying users, managers need to consider new ways to maintain performance and long-term engagement levels in these environments,” explained Eric T. Bradlow, professor of marketing at Penn, who coauthored the study.

Among the study’s limitations, the authors said that because they took advantage of a natural experiment (i.e., Coursera was changing the way it released course content), the differences they found could be a result of both a change in the composition of users who self-selected into each format or a causal effect of the format change.

“Our study offers vital lessons for chief marketing officers in the online education space,” suggested J. Wesley Hutchinson, professor of marketing at Penn’s Wharton School, who coauthored the study. “Managers must consider the tradeoff between offering structure versus flexibility, possibly offering both scheduled and on-demand options at different price points.”