Data exchange helps humanitarians act fast and effectively

In early 2014 Sarah Telford was head of the reporting unit at the UN Office for the Coordination of Humanitarian Affairs (Ocha). Her task was to improve its reporting. But she ran into a problem. “We kept hitting the wall of not being able to find the data to do the analysis,” says Telford. “If we did find it then we weren’t sure where it came from.”

A quick solution to bring core humanitarian data together into a single spreadsheet for each office snowballed into a larger data infrastructure project. Six months later, Telford is now in charge of HDX – the humanitarian data exchange.

The aim is to establish standards for humanitarian data and establish an open platform for information exchange among UN agencies, NGOs and government. The platform was launched in July, and has received about 25,000 visitors since then. Nearly 90 organisations have registered to share data on the site, and more than 1,350 datasets have been uploaded, including databases that automatically publish updates. Volunteers are working to scrape yet more data trapped in thousands of UN pdfs.

Telford’s team initially focused on two pilot locations – Colombia and Kenya. But the Ebola crisis intervened, and the team needed to offer support. They built an Ebola crisis page to provide an overview of the relevant data they held, interactive maps of the worst-affected countries, and accessible analysis of the crisis.

Core to the project is its openness – anyone can access the data and anyone can submit new sources. They’re also open to feedback. “We are just in the process of agreeing on a proposal with the Paul G Allen Family Foundation to make improvements to HDX based on user feedback from the ongoing Ebola response. This work will cover integrating geospatial data and our data standards work (known as the humanitarian exchange language) into HDX.”

People can also follow datasets, register organisations and members of those organisations, and be part of the social aspect of sharing data. In that regard, it’s much like a humanitarian data-centric Wikipedia, with individuals and organisations curating their specialist fields.

The project was initially funded by Britain’s Department for International Development, the Swedish aid ministry and the Humanitarian Innovation Fund, with additional funding from the Rockefeller Foundation. This produced about $2m, which financed the creation of the platform with Frog Design and a data lab in Nairobi. Ocha covered the cost of two staff positions for a year and has now made it part of its core budget. “It felt like we were a startup trying to get seed funding from a venture capitalist or angel investor … [the Ocha funding] means the project has a sustainable baseline budget that we can plan on and we can continue to seek new funds from donors if we need to.”

This will allow improvements to be made. Though a team does check new data for formatting and new organisations, data validation is currently absent. Telford plans to rectify that in 2015. “This will allow users to run their data through a validator that checks whether the data in each column corresponds with the expected values.”

The team also plans to scale to more locations in 2015, starting with Mali, and look at whether this data platform could be integrated more with the development world. “It’s not impossible to think that HDX could one day become a social data exchange for the aid sector rather than be limited to humanitarian data but we are years away from that.”