On both the clinical and commercial sides of the pharmaceutical business, the options for using data have proliferated to the point of becoming overwhelming. Everything is bigger, from the databases themselves, to the investment required, to the payoffs for getting it right.

The Walls Are Tumbling Down

Other sectors—namely the retail, consumer packaged goods, and financial industries—have been coping with Big Data for as long as a decade. So, why is it just now becoming topical in the pharmaceutical industry? There are two interrelated reasons.

Second, there is a growing, almost insatiable, demand for information as healthcare becomes evidence-based. Every party in the system—from manufacturers, to payers, policymakers, to providers and patients—has much to gain from understanding what works and what doesn't. And that requires a comprehensive view that can only be gained from multiple data sources that, when amassed, become Big Data.

The Obama administration further reinforced the need for Big Data in its March 29 announcement of $200 million in funding for a "Big Data Research and Development Initiative," which seeks to improve our ability to extract knowledge and insights from large and complex collections of digital data. While the initiative is intended to address scientific discovery, environmental and biomedical research, education, and national security, healthcare clearly stands to benefit from such significant investment.

The new sources of information that are causing such a stir within the pharmaceutical industry include:

Electronic Medical Records. The data contained in these systems provide a detailed clinical picture within the site of care and contribute to an overall view of a patient's treatment pathway.

Social Media. Patients use social networking to, as the Pew Internet & American Life Project described it, "lend a hand, lend an ear, lend advice" to one another. "Listening in" on such public exchanges can provide rich insights into the patient or prescriber perspective.

Real-World Evidence. Privately funded and government-sponsored longitudinal studies produce a wealth of information on treatment practices, patient persistence and compliance, and health outcomes, which are increasingly required by payers.

Personalized Medicine. When, in the foreseeable future, an individual's genome can be mapped affordably, the amount of data—while staggering—will bring exponential value to medicine. It will be possible to go beyond understanding what works and what doesn't, to actually predicting what will work, person by person.

Track-and-Trace Systems. If the FDA succeeds in mandating that pharmaceutical companies install systems to track and trace their products, the systems would generate a data point every time a package "hits a node" on the distribution system.

When all of these elements are added to existing information sources—including secondary data, a company's own closed-loop marketing program, or other internal data—the Big Data challenge grows exponentially.