Big Data? Small Data? We Need An 'All of the Above' Strategy For Development Data

July 13, 2015
Stateless Rohingya gather at a refugee camp in in Sittwe, Rakhine State, Burma

There is no shortage of debate about the power and perils of data -- particularly big data -- for development. On the one hand, data can be used to better track disease, economic performance, crime, and climate patterns. On the other hand, data is sometimes inaccessible, can require advanced analytical capacity to be useful, and raises legitimate concerns regarding privacy and security.

A couple of weeks ago, development economist Paul Jasper suggested that we shouldn’t get too excited about using big data for development, citing many of these perils as proof that big data alone will not get us far. But it’s time to put to rest the debate about big data versus small data for development. We need an “all of the above strategy” that mobilises small data, big data, resources, analytics capacity and leadership in a responsible way that addresses privacy concerns – to solve our most pressing social, economic, and environmental challenges.

Only then will we truly be able to address the major data gaps that exist. For instance, one in three children under five worldwide has not had their birth registered, significantly reducing their access to public services and formal employment. Today, most retailers have far better real-time data on the demand for the latest fashions, than health responders do about potential disease outbreaks or food shortages.

No single approach will close these gaps. We need greater investment in national statistical offices to track basic national indicators. We need to help governments better collect and open their own administrative data to improve transparency and optimise their programmes. We also need to find ways to join the big data sets such as those generated by mobile phones and social media to other small data sets to complement and fill gaps.

In many ways, this comprehensive strategy is already at work. Economic data suggests that fully including women in the economy has helped countries increase their GDP from between five and 34% . Making data available on the sex, age, and geographic distribution of HIV means adolescent girls who are at greatest risk of infection can now better benefit from targeted interventions in education and employment alternatives. Big data from mobile operators has helped predict food instability and climate change impacts; and data demonstrating thesignificant growth in domestic and private financing flows, to and within lower income economies, is changing the way public and private sectors collaborate to finance development.

By setting targets to be achieved by 2030, the proposed sustainable development goals (SDGs) provide a unique global framework to scale this approach.

The US government is collaborating with data champions in other governments, in businesses and civil society to generate and deploy an comprehensive strategy to harness data for sustainable development, with a focus on three overarching objectives that can help unlock the truly transformative power of the data revolution.

1. Shift our thinking from data as an enabler to data as capital for sustainable development

Increased availability and more effective use of data to drive sustainable development has the potential to be the game-changing innovation of the next decade. Spending on data is often characterised as an auxiliary activity – a luxury that competes with other potential activities for limited funds. This mindset must change.

Data is truly development capital, one that – similar to financial capital – we must improve how we maximise its development return on investment. Strong data systems allow us to increase our efficiency and effectiveness and get more value out of every dollar. Data can also open whole new fields of business – such as privately run micro-weather forecasting services that optimise farming techniques – and create wholly new kinds of economic value. This is how the private sector sees it; big data and data management is estimated to be a $40 billion business worldwide.

2. Strengthen partnerships across public and private actors to solve data gaps and make data-driven government the norm

As Jasper notes, the challenges with data are not always technical. Sometimes it’s an issue of norms and guidance, such as setting anonymisation standards to help make companies comfortable to release data sets without markers that identify contributors. Sometimes it’s an issue of laws and regulations that can make even basic activities, such as registering births, inefficient and cumbersome. Or sometimes it’s an issue of insufficient political will to use data to drive policy decisions.

These stark challenges persist because the people who face them most often (technical experts or data scientists) are different from those who have the power to address them (policy-makers or corporate leaders). Overcoming these challenges will therefore require creating partnerships with stakeholders from governments, the for-profit and non-profit sectors, and the data science community. Such partnerships will start to solve these challenges by fostering cross-regional and cross-sectoral collaboration, sharing best practices about data management, and developing and scaling innovations.

3. Mobilise data champions to scale successes and lessons learned

In recent months, we have met data champions from all over the world, from the World Economic Forum Africa, to the Data Festival in Cartagena, Colombia, to the Health Measurement and Accountability Summit in Washington, DC. Hearing about their experiences demonstrated the need for more stories from those who have successfully used data to create change. For example, one champion has been part of an effort to advance the use of data in policy across Africa which has yielded the African Data Consensus; another is deploying mobile apps that allow community health workers to identify a mother’s risk of maternal mortality; others are collaborating with companies to release data to enable real-time tracking of deforestation by national statistical offices and citizens.

As Jasper explained, successfully developing and applying data for development is very hard, so we must invest in creating partnerships and ecosystems that can connect such data champions and enable them to scale their successes – and for others to replicate or build on them further.

About the Author: Daniella Ballou-Aares is Senior Advisor for Development to the U.S. Secretary of State, and Tony Pipa is U.S. Special Coordinator for the Post-2015 Development Agenda.

Editor's Note: This blog entry orignally appeared on the Guardian's blog.


Leave a comment

(If you're a human, don't change the following field)
Your first name.
(If you're a human, don't change the following field)
Your first name.
(If you're a human, don't change the following field)
Your first name.

Plain text

  • No HTML tags allowed.

Filtered formatting

  • Allowed HTML tags: <a> <em> <strong> <blockquote> <ul> <ol> <li><p><span>
  • Lines and paragraphs break automatically.
  • Web page addresses and e-mail addresses turn into links automatically.


Patrick W.
Maryland, USA
July 13, 2015
Very good idea ! President Obama really could have used this data sooner, and we wouldn't have had idiots looking for his birth records ,because they would have been on file in a data base. People are really idiots when they can't believe the truth unless it's on some stupid file on a computer. I guess they don't realize we didn't have that kind of technology back then. :)
Patrick W.
Maryland, USA
July 25, 2015
Does this mean anyone born before the 80s can forget about finding their birth records , or are we waiting for an update from paper to data to really be born ? What happened to the filing system. I guess, no one knows what paper is or how to read it ?


Latest Stories

October 8, 2015

All Climate Change Is Local

When world leaders gather in Paris in December to negotiate a global agreement on climate change, they will have a… more