Jason Hare's Data Management Blog

  This is a wifreframe from the World Bank showing a search interface for their highly respected data catalog. Searchable Data Sets? Check out the #wireframe for a fully searchable data catalog. This is where #opendata needs to head. Citizens do not browse data like kids in a candy store. I have asked about engagement and watched them interact with open data sets and not one non-data geek ever found them interesting. For methodology I chose sets of sets of 3-5 people and just asked them questions and sent them links. These people came from my Facebook groups of friends and do not have a professional connection with me. When I asked them about their browsing patters in general most people used Google Search to find specific information or started with a list of 10 or less "go to" sites. Data sets need to be searchable and queries need to come from Google and index both the data sets themselves and be able to interpret natural language queries before will be mainstream and of i...

Optically scanning PDF data and turning it into machine readable format data would be quite an accomplishment. This is happening right around the corner from the City of Raleigh using minutes from our council meetings. I would like to see what the CSV files look like. This could be a large impact on open data in reusing PDFs and perhaps other proprietary formatted data types. I am only now starting to imagine all of the potential uses for something like this. Turning scanned documents into structured data at reporterslab.org A new open-source program under development at Raleigh Public Record aims to pull structured data from scanned-in public records. And ahead of its release at the 2013 Computer-Assisted Reporting Conference, developers are... See the full article here.

  Technology and Transparency: Data Needs a Cultural Context Per a conversation with Alex Howard on Google+, I have read " Beyond Technology for Transparency ". The author makes several points: The Yu and Robinson article " The New Ambiguity of 'Open Government '" was something I read a few weeks ago. I disagree that the edge of open government is going away. Yes there are data sets being published that have nothing to do with accountability. Yes there are open data initiatives that stand up a few data sets and call it "open". This does not mean that all or most open data professionals do this. This whole line of "government versus the people" is one of the reasons PSA's have trouble getting open data initiatives launched in the first place. David Sasaki makes good points at the end of his blog post. Open data should strive to reduce poverty, corruption and and reduce operational inefficiencies. This will not happen with just transpar...

  Screenshot of the Gun Map Programmers explain how to turn data into journalism & why that matters | Poynter. Transparency does have limits. That is a policy issue that a PSA must work through. In my particular open data initiative, we have laid out several principles regarding security and privacy. I brought this up on a Linkedin post a few weeks ago and this was part of my question to +Alex Howard about #opendata and #journalism. The now infamous " gun map " that identified by name registered gun owners is a good example of how not to use open data. This, I know, was not a PSA that published this data. It was a newspaper. However, I feel open data and big data evangelists need to start having a discussion about the right to privacy and our ethical boundaries. The gun map, as an example, would have been just as effective as a heat up of gun ownership density. The article "Programmers explain how to turn data into journalism & why that matters" by Jeff Sond...

"There is nothing new under the sun". Nate Silver points to this quote as being indicative of man's lack of progress until the industrial revolution. In the vernacular it is meant that the more things change the more they remain the same. Big data, open data and the new vast piles of noise we sit on mean nothing without analysis. Acknowledging that all researchers have bias and there is no such thing as unbiased information or knowledge is a good thing. Big Data is still data which we use to support hypotheses and against which we measure and test theories. No type of big data, open data or government open data is a silver bullet. Data has to be transformed into information and knowledge through contextual inquiry. When, on a summer Sunday morning in 1987, three hundred thousand people crammed onto the central span of San Francisco’s Golden Gate Bridge, they came perilously close to participating in...read more below Are We All Being Fooled by Big Data?

  Regarding the posting by Cathy O'Neil on "Open Data is not a panacea": http://mathbabe.org/2012/12/29/open-data-is-not-a-panacea/#comment-15181 Open Data Ethics Open data professionals are currently debating the ethics of exposing data sets to the public. The main issues are privacy of the individual and the usability of the data. The transformation of data into information and the ethical exposure of "anonymized" data sets are what we are after. I work at the municipal level as a city open data program manager. In my role my task is to deliver a framework of collaboration between our city government and the citizens. The city wants collaboration and input on ideas related to social issues, economic development issues as well as exposing performance indicators on department spending and performance. Data like any asset within government is part of infrastructure. Data alone does not equal transparency. Certainly machine readable data sets are available for any...

Open Data as Narrative Recently Mark Headd, City of Philadelphia, posted a blog post about using data for "Urban Storytelling". This is a good way to summarize the transformation of data sets as open data to information that narrates an issue within a city. Mark and the City of Philadelphia told a story about crime within the city. The tools and the source code on GitHub are posted on Mark's blog here: Urban Storytelling with Open Data http://civic.io/2012/12/20/urban-storytelling-with-open-data/ I thought the video showing how the visualization packed a wow impact but as Mark said also painted a sad picture of too many homicides in Philadelphia. Here is the YouTube video of the crime data visualization by Mark Headd That being said there is a point in using open data in this way. This is a great article supporting something we have been talking about in Raleigh. There are really two parts to open data...data as infrastructure. The city providing access to data assets th...