Carnegie Mellon Experts Launch Flu Forecasting for 2016-17 Season

Is Artificial Intelligence Superior to Wisdom of Crowds?

Computer scientists and statisticians at Carnegie Mellon University are comparing artificial intelligence and the wisdom of crowds in forecasting flu activity this winter.

During the last flu season, three forecasting systems developed at CMU were more accurate than 11 competing systems used by other external groups.

“Our predictions last season proved to be reasonable, but when it comes to forecasting epidemics, whether it be for the flu or for other diseases, we’re just getting our feet wet,” said Roni Rosenfeld, professor in the Language Technologies Institute and the Machine Learning Department of the School of Computer Science.

Rosenfeld is a member of the Delphi research group, which includes faculty and students from CMU’s Machine Learning, Statistics, Computer Science and Computational Biology departments. The group is part of a research initiative with the U.S. Centers for Disease Control and Prevention to develop methods of accurately forecasting flu activity.

Last season’s predictions by the top-ranked CMU forecast system were within 25 percent of the CDC’s best estimate of flu activity just 75 percent of the time, said Ryan Tibshirani, associate professor of statistics and machine learning. The forecasts are made week by week during the flu season and the CDC updates its best estimate of flu activity throughout the flu season and for several weeks thereafter.

Making those predictions more reliable on a weekly basis would no doubt be necessary before such forecasts might be used for deciding when to launch flu information and vaccination campaigns or for making staffing and scheduling decisions within the health care industry, he added.

“We’re still trying to squeeze everything we can from these models,” Tibshirani said.

Most epidemiological forecasts are based on mechanistic models that consider how diseases spread and who is susceptible to them. But the Delphi group’s top-ranked system was a non-mechanistic model that uses a type of artificial intelligence called machine learning to make predictions based on past patterns and on input from the CDC’s domestic influenza surveillance system. The surveillance system includes reports from doctor’s offices and clinics regarding the prevalence of flu-like symptoms.

Delphi’s second-ranked system uses a different approach — using weekly predictions by humans that, together, reflect the wisdom of crowds. This human system was the top-ranked forecasting system for the 2014-15 flu season, Rosenfeld said.

Rosenfeld said that from a computer scientist’s point of view, “it’s humbling” that the human system has been neck-and-neck with the statistical, machine-learning system. “No human system did better than the statistical system, but in the aggregate, the human system did better that season.

“The human system is more robust in unusual circumstances,” Rosenfeld said, so it may do well when flu activity falls outside normal bounds. “Humans are very good at improvising when they encounter novel circumstances.”

This season’s weekly forecasts began in October and will continue through May. Forecasts are issued for flu activity nationally and for each of 10 regions within the United States. Because of lags in reporting, the actual flu activity levels will not be known until the season is over.

“Obtaining high quality data is critical for epidemiological forecasting, but it’s hard to get,” Rosenfeld said.

Flu is useful for developing forecasting systems because data is plentiful. It is notoriously “noisy” data, however, because the data usually are based on symptoms, not tests for the flu viruses themselves, he added.

The Delphi group also is developing forecasting systems for dengue fever, which sickens about 100 million people worldwide each year and kills thousands. The group plans to apply forecasting tools to other diseases and conditions, including HIV, drug resistance, Ebola, Zika and Chikungunya.

Epidemiological modeling and forecasting is a highly interdisciplinary endeavor. In addition to Rosenfeld and Tibshirani, the Delphi group that previously worked on the CDC flu challenges included David Farrow, who recently earned his Ph.D. in computational biology; Logan Brooks, a Ph.D. student in computer science; and Justin Hyun, a Ph.D. student in statistics.

Members of the public can help the Delphi group’s efforts by joining its “wisdom of crowds” forecasting system by registering online.