Jason Hare's Data Management Blog

  PI First Manifesto This is a well written manifesto from David Thoumas at OpenDataSoft proclaiming that the ODS strategy will always be API [First]. Below I have slightly edited the content written by David. I want to bring attention to the fine work this company does and why I and my company, BaleFire Global have such high regard for what they do: At  OpenDataSoft , David and team are building and operating a cloud based data management platform. This platform is built 'API first'. This means that any feature that can be accessed from the portal is also available as an API call. The portal is itself the first consumer of the API. Available APIs ( source ) include: Dataset catalog APIs (keyword and faceted search of datasets within the catalog). Dataset APIs (search within dataset records, geo clustering of geo dataset records, numerical aggregations of dataset records).        David Thoumas from OpenDataSoft So, not only can one fetch raw data from the portal through API cal

This article is a reprint of a Durham Press Release. A wealth of public data exists, but accessing it can sometimes be time consuming. Now, thanks to a new local government partnership, “open data” in Durham is just months away from becoming a reality. The City of Durham and Durham County Government are embarking on an open data partnership that will lay the groundwork for businesses, non-profits, journalists, universities, and residents to access and use the wealth of public data available between the two government organizations, while becoming even more transparent to the residents of Durham. A relatively new concept according to technology industry analyst Gartner, Inc., “open data” refers to the idea that certain data should be freely available to everyone to use and republish as they wish, without restrictions from copyright, patents or other forms of control. “This is a unique collaborative effort that demonstrates a regional commitment,”

Temple University has discovered a way to eliminate the virus from cells Anti LGBT Protest in San Francisco. Creative Commons License attribution to:  Jenny Mealing The elimination of HIV/AIDS will transform society. It would be a historic moment in medicine and give us a chance to examine ourselves and the history of tolerance toward the disease and those at high risk for contracting HIV. I am part of GenX and graduated from high school in 1985 and from college in 1994. I have lived my entire adult life under the threat of infection from HIV. In the early 1990s I lost several friends to the disease. I had one very harrowing experience in 1990 being tested for HIV and being certain that I was infected. It turns out I was not infected. HIV/AIDS has had such an impact on our culture and how we interact with others. As muc

By Dennis D. McDonald, Ph.D. Introduction Earlier this week in USAID’s Evolving Open Data Culture I applauded that U.S. government agency’s efforts to make its open data useful. In this post a dive a little deeper into the topic of “open data usefulness.” Background I came to my interest in open data by way of a career that has mostly involved technology-related projects and consulting. Major goals have been to make or support products or services that are useful to somebody. For private sector clients this usually involved impacting cost or revenue targets. For government agencies or nonprofits work has focused on objectives that have both quantitative and qualitative aspects. In either case, “usefulness” has meant that the actions taken by people as a result of using the product or service are viewed by them in a beneficial or positive light because it helps them accomplish their objectives. As a result of this perspective I’ve thought it would be shortsighted or incomplete not to

As you can see from the figure, September and November are not far below October. Some readers may wonder why there is a surge in API calls starting in May of 2014. May through October was spent building open source service architectures on Red Hat JBOSS Switch Yard that could mine and automatically append data sets within the Open Raleigh Portal. Open Raleigh uses a responsive web design that is friendly to most handheld devices but the API needs a little help to push data into the portal. The portal itself releases every data set as an API endpoint. This API is a READ only API. Writing some code we can have the Socrata portal allow us to append data sets. Socrata is not alone in the Web/Mobile [First] category. ESRI, CKAN and to some extent, Junar are architected on the same principals. This is not a direct criticism or endorsement of any particular platform. THE CONSEQUENCES OF GETTING IT WRONG Discussing multi-nodal approaches and espousing an API [First] st

  Councilman Steve Rao and Family Today, Monday, December 8th at 4:00 PM Morrisville Councilman Steve Rao is hosting his sixth Virtual Town Hall of the year. These innovative events have been used by Councilman Rao throughout the year to keep in touch with citizens in Morrisville and throughout North Carolina and to provide updates and take questions on a variety of topics. The last Virtual Town Hall was on the potential for Open Data to revolutionize government and featured special guest Ian Henshaw of the Open Data Institute. It was attended by people throughout North Carolina and also from the United Kingdom and from India. Today’s event will review what has happened in the Triangle area in the past year and look forward to 2015. These are part of a larger initiative by Councilman Rao to spur innovation in the state by applying the latest technologies to solve our public policy problems. To attend, you must RSVP at the link below and have a computer equipped to log into a Google Han

  By Dennis D. McDonald, Ph.D., dmcdonald@balefireglobal.com Introduction In my previous post about open data quality the suggested solutions relate not just to adhering to standards but also to making sure that the processes by which open data are published and maintained are efficiently and effectively managed. In this post I drill down a bit more on that point about the management processes. Three Phases When discussing open data it helps to look at open data projects with tasks divided into at least three related phases: Assessment and planning Data preparation and publishing Ongoing maintenance and support Different tools and processes are relevant to each phase. Each can have an impact on the quality of the data as well as its perceived quality. Phase 1. Assessment and planning Critical to data quality at this first phase of an open data project is an understanding of the "who, where, how, how much, and why" of the data. If the goals of the project include making data f

By Dennis D. McDonald, Ph.D. Email: dmcdonald@balefireglobal.com At risk? Martin Doyle's Is Open Data at Risk from Poor Data Quality is a thoughtful piece but doesn’t address this question: Should data quality standards observed in open data programs be any different from the data quality standards observed in any other programs that produce or consume data? My first response is to answer with a definite “No!” but I think the question is worth discussing. Data quality is a complex issue that people have been wrestling with for a long time. I remember way back in graduate school doing a class project on measuring “error rates” in how metadata were assigned to technical documentation that originated from multiple sources. Just defining what we meant by “error” was an intellectually challenging exercise that introduced me to the complexities of defining quality as well as the impacts quality variations can have on information system cost and performance. Reading Doyle’s article rem

This morning something happened that will gradually impact the way we interact with data. This morning OpenDataSoft (ODS) embedded the Open Data Institute's Data Set Certificates into each and every data set page. Will other open data platforms follow? Maybe. Embedding certificates is something I have been advocating since the idea was just an idea at the Open Data Institute (ODI). No one until ODS ever took me up on my offer. These data certificates show a willingness on the part of the data steward to consider the following: The impact on individual privacy API and format documentation to ensure a greater chance of data re-use Metadata on where the data originates and how often it is refreshed RDF description tags and identifiers that allow f

By Dennis D. McDonald, Ph.D. dmcdonald@balefireglobal.com Introduction How important are costs when you are planning an open data program? Are they, as suggested by Rebecca Merrett in Addressing cost and privacy issues with open data in government , the "… elephant in the room," especially when data anonymization costs are being considered? Or are such costs just a normal consideration when planning any project where quantities of different types of data have to be manipulated and delivered? It's difficult to make a generalization about this. Open data program costs can fall along at least three general dimensions: 1. Controlled versus uncontrolled 2. Known versus unknown 3. Startup versus ongoing 1. Controlled versus uncontrolled Why worry about what you can’t control? The answer is because they can impact your program whether you control them or not. Examples of uncontrolled costs might be: Taxes, licensing, insurance, and registration fees. Staff salaries that can'

By Dennis D. McDonald, Ph.D., Balefire Global, dmcdonald@balefireglobal.com During the brainstorming session at the inaugural meeting of the Open Data Enthusiasts meet up last week in Washington DC, attendee David Luria commented that we need to do a better job of understanding, defining, and communicating the objectives of open data programs if we want them to be successful. I couldn't agree more. Program objectives need to be clearly defined and shared with stakeholders and program participants so that everyone is marching in the same direction. If we don't understand and agree on our objectives how can we establish requirements and metrics to measure what we're trying to accomplish? Admittedly the above principle is straight out of Project Management 101 and describes the initial steps you need to take in planning and documenting any project, not just those involving open data. Still, what I have noticed after involvement with many data related projects is that there a

In May of this year, Asian and European countries met for the regional Open Government Partnership summits to once again discuss transparency and open government. In light of the session tracks that were presented I am evaluating some of the technologies of the past and how there has been a welcome and fundamental shift from Open Government and Open Data being ambiguously linked toward the separation of the two in more current thinking and in technology approaches. Most notably, the deprecation of Microsoft's Open Government Data Initiative platform is a positive sign of the times that the government community is becoming aware of the danger of open government and open data linking. Harlan Yu and David G. Robinson discussed the OGP in "The New Ambiguity of Open Government" (Princeton CITP/Yale ISP Working Paper).   "The Open Government Declaration is broad approach toward 'openness,' as signatories commit to 'seeking

By Dennis D. McDonald, Ph.D., Balefire Global, dmcdonald@balefireglobal.com Introduction In Management Needs Data Literacy to Run Open Data Programs I addressed the question of how much “data literacy” open data program managers need. I outlined a series of topics corresponding to different parts of the data management lifecycle the program managers need to be familiar with. While certainly I don't believe it is necessary for all program managers to be “data scientists” to manage open data programs effectively, I do think that there are certain data related skills that managers do need. One of the most important is the ability to think about data both from analytical as well as structural perspectives. The analytical perspective Analytically, managers need to understand that useful data are not just random collections of numbers but represent patterns and trends that can be used to tell stories about the objects or events with which the numbers are associated. The range of tools

In response to "The Potential of Open Data" published by Deirdre Lee, at the Insight Centre for Data Analytics, NUI Galway (www.insight-centre.org), I offer my opinion on the health of the government's open data efforts to date. Ireland needs to take a page from Sir Francis Maude and not the US. The US open data initiative is an embarrassment. Embrace "Data is the 21st century’s new raw material." The Beginning of Open Data in Ireland To be fair, Ireland is just beginning the road to open data after an internal struggle stretching back five years. The Irish Minister of Reform, Brendan Howlin, has also wisely eliminated FOI fees for information requests. This FOI issue has long been a point of political divisiveness in Ireland. Indeed at the Open Government Partnership Summit in Dublin this past year, I witnessed street demonstrations against FOI. Government creates an Open Data Initiative In late January, a tender was issued by the CIO of Ireland, William

  In the Beginning In response to " The Potential of Open Data " published by Deirdre Lee, at the Insight Centre for Data Analytics, NUI Galway ( www.insight-centre.org ), I offer my opinion on the health of the government's open data efforts to date. Ireland needs to take a page from Sir Francis Maude and not the US. The US open data initiative is an embarrassment. Embrace "Data is the 21st century’s new raw material." The Beginning of Open Data in Ireland To be fair, Ireland is just beginning the road to open data after an internal struggle stretching back five years. The Irish Minister of Reform, Brendan Howlin, has also wisely eliminated FOI fees for information requests. This FOI issue has long been a point of political divisiveness in Ireland. Indeed at the Open Government Partnership Summit in Dublin this past year I witnessed street demonstrations against FOI. Government creates an Open Data Initiative In late January, a tender was issued by the CIO of I