What Open Data Intended

From the Cincinnati Enquirer @2008

The Yu and Robinson article "The New Ambiguity of 'Open Government'" was something I read a few years ago. I disagree that the edge of open government is going away. Yes there are data sets being published that have nothing to do with accountability. Yes there are open data initiatives that stand up a few data sets and call it "open". This does not mean that all or most open data professionals do this. This whole line of "government versus the people" is one of the reasons Public Sector Agencies (PSA's) have trouble getting open data initiatives launched in the first place.

The main issue is the disconnect between PSAs and the private sector. There is little, if any, discussion on the value-add of releasing these data. This is not purely a government issue. Private sector, with a few shining exceptions (BuildingEye for example) have shied away from using or even trying to use these data. PSAs see this lack of engagement and fail to rise to the occasion. This causes many PSA open data efforts to stagnate.

What PSA Open Data was Supposed to do

David Sasaki makes good points at the end of his blog post. Open data should strive to reduce poverty, corruption and and reduce operational inefficiencies. This will not happen with just transparency. Open government should be creating collaborative data driven policy between government and citizens.

There are several problems with the arguments made by critics of open data such as Yu and Robinson:

Government is a collection of people and not a single entity

Agencies in government view data as political leverage to affect policy and create relationships with other agencies.

Mandated open data initiatives that do not give agencies any incentive to share will fail.

Failure is not caused by government's inherent desire for secrecy but rather in elected officials, citizens and activists not understanding the culture of government workers and agencies.

Arguments about government intent are facile since elected officials do not wield as much power internally as in commonly believed.

An Example of Transparency and Culture Clashes at the Federal Level: Data.gov

Below I have quoted another article that speaks more toward the culture of government. So far there have been very few government employed open data people talking about open government data. I think it is important to understand the culture of government in the United States

The article below discusses the US Federal data.gov initiative and why it has not met expectations.

"When Transparency and Collaboration Collide" This is not a free download but does explore why open data initiatives have problems.

"When Transparency and Collaboration Collide" analyzes why data.gov has had issues with getting data from more than a few agencies and examines the motives for why agencies have been passive-aggressive in complying with President Obama's executive order. The reasons are rooted in how agencies interact with one another and use data as bargaining chips rather than any attempt to be secretive.

The Red Herring: Government is Inherently Secretive

One of the themes that I feel is a red herring is this argument that government is hijacking open data initiatives as a way to be less transparent. Certainly there are dubious initiatives. I don't know the details of the open data efforts in Kenya but I will take the author's word for it. Government is not that smart. I know very well at the municipal, county and state level that the governments involved in my open data initiative are not monolithic omniscient entities. Internally, government is a series of departments and people often with differing agendas that makes getting meaningful work done difficult even with strong executive buy-in.

This argument distracts from the real problems open data initiatives face. Governments are not people with personalities. Governments are collections of people that share a culture. This is something I talked about with Open Ireland. Agencies that report to the executive branch of the federal government and that were affected by President Obama's executive order had no incentive to cooperate. Far from having an incentive agencies would lose political power by complying with the order. These reasons had nothing to do with transparency and show a complete failure of this open data initiative to understand the culture of the agencies affected by the order.

Federal agencies exchange data sets with each other using an MOU/SLA contract that specifies how the data is to be used and how the data will be delivered. This data contract is something well understood in technology circles and is something that is a feature in web services created through Service Oriented Architectures (SOA). Understanding how and why data is traded among agencies is the first step toward creating an incentive for them to share data with the public.

Let me restate that nowhere does any agency have an incentive to be secretive as a way to cover up corruption or other abuses of the public trust. That is not to say that abuses do not happen. In the case of data.gov this was not at issue. At issue was an agency desire to create benefit from distributing the data.

When the open data folks at data.gov suggested developing an internal framework for exchanging data sets through contracts between data stewards and data consumers the agencies were much more likely to keep their data fresh, complete and in compliance with the executive order. This internal framework made it easier for agencies to license data to each other and to deliver data through SOA based web services.

To summarize the discussion should be how to work with government to create meaningful open data initiatives. Municipalities such as mine have seen the benefit in narrating major issues within the community and seeking a way to collaborate with citizens using data to drive policy decisions.

The Government does not have a plan for Open Data

This is not a red herring and acknowledged as an over-generalization. Since the government is not monolithic but, rather, a set of individuals pursuing their own agendas it must also be true that governments have a varying ability of know-how. By know-how I mean the "government" in question will be challenged on how to keep engagement between citizens, private sector and the public data the PSA releases. 

Where we have gained some traction is through those that have managed to push through data standards such as LIVES and BILDS. Having a data standard attracts private sector. It opens the market to create applications and tools that can cross jurisdictions. I would argue even that government should be in the business of developing standards with private sector and outsource the release of open data to private sector. This model works well with Acella. Mark Headd and Brian Gryth have successfully used a private cloud vendor to help cities and other PSAs to release data. Unfortunately the next step of the maturity model in using PSA data is vague. 

How can "government" create incentive and at the same time demonstrate value in opening PSA data? This is something that I have not been able to answer. This may also be my own lack of imagination at fault. I invite my colleagues to weigh in.