SORRY FOR BLOWING UP YOUR WEBSITE!

-

The other day, I flooded a customer website licensed for 75 datasets with 318 additional in five minutes while using the OpenDataSoft ArcGIS Harvester. Yup. I did that. I felt not so much like “Darth Vader” as much as I did “Dark Helmet” (Rick Moranis in Spaceballs, see below if this reference goes over your head).

I was using OpenDataSoft’s ArcGIS Harvester. It’s still in private beta, but will likely be available by the end of December (consider it a Christmas gift to you!) for data publishing. Luckily, the Harvester also came with a delete button so the customer was not affected. I took the usual ribbing from my co-workers. David Thoumas added that we do not need “too many datasets”.


I was awed with the power of the Harvester.


I had seen it work a year ago and take most of an afternoon to mine data. I innocently put the root of the GIS server into the Harvester, and suddenly it was pulling in data at “Ludicrous Speed”. Ian Henshaw noted we hit hyper drive and “They’ve gone to plaid”.


SEVERAL TITLES FOR THIS BLOG POST CAME TO MIND:

“With Great Power Comes Great Responsibility”

“Young Jedi uses too much Force in Releasing Data”

“We’re Gonna Need a Bigger Boat”

“How Jason Hare Blew Through a 75 Dataset License in 5 minutes”

At first I felt like I was aiming the Death Star at Planet Alderaan. Heck yes I was going to show some folks how to ingest data with our giant Hoover Machine. As the datasets quickly tallied I realized I was ingesting an entire ArcGIS server. It went from 1 to 15 in about 2 minutes. By 3 minutes we were at 150 and by 5 minutes we had all 318 datasets.


WHAT I LEARNED AND WHAT YOU SHOULD KNOW

The OpenDataSoft Harvest Tool can pull data, with rules, from an ArcGIS server with two clicks. We can harvest other types of data as well. This was a learning moment for me. “Choose wisely”. We can also harvest this data very quickly (remember Ludicrous Speed).


No worries. After David Thoumas grounded me from using the Harvester I got a quick lesson from our awesome developer team. I am no longer considered a threat to our Open Data community or customers.


Original article: https://www.opendatasoft.com/2016/05/26/sorry-for-blowing-up-your-website/

Comments

Post a Comment

Popular posts from this blog

Podcast: Open Data Discussions with Anthony Fung

-
Image
A look back at Virginia’s 4 years of data innovation with Anthony Fung, Deputy Secretary of Technology for the Commonwealth of Virginia  Welcome to the OpenDataSoft Leadership Podcast Series, “Open Data Discussions”. Each month, Jason Hare, our Open Data Evangelist, features a different open data program around the country to discuss what has made it successful. These examples will provide insights and strategies that you can implement in your own city. We had the pleasure of welcoming Anthony Fung, Deputy Secretary of Technology for the Commonwealth of Virginia for the fifth podcast in our series. Anthony discussed various subjects, including: What role does open data play in policymaking at the state level in Virginia? Does his department work on ensuring open data quality? What are some of the steps he makes in ensuring sound data governance? How did Virginia come up with this technology and engagement solutions?  How, for example, did yo
Read more

Open Data Licensing

-
Image
Everything you wanted to know but you were afraid to ask. Making data open and available is not just about choosing a format or having an API. A poor choice in your license can prevent people from using them. A traditional open license reassures potential users that their work on the data will be useful in the long run. So, what license should you pick for your data to be broadly used? There are a few things to consider. Choosing a license for your open data can usually be one of the biggest challenges for an open data project. There are plenty of licenses available for your data. Choosing one mostly depends on the usage and reach you want to give to your project. CHOOSE A LICENSE: BASED ON NATIONAL, INTERNATIONAL AND LOCAL STATUTES IN ADDITION TO REUSE It should be noted that different parts of the world use data licensing terms differently. In the US, works in the “public domain” are works are not covered by intellectual property rights, such as copyright. The copyright mig
Read more

AN OPEN DATA POLICY FOR NORTH CAROLINA: COMING SOON?

-
Image
All parts of North Carolina will benefit from open data legislation. FOR THE BETTER PART OF TWO DECADES, I’VE WORKED TO OPEN UP NORTH CAROLINA’S DATA. I’ve been a public servant at Durham Public Schools and the Department of Energy. I have been an onsite consultant for the Cities of Raleigh, Durham as well as the Towns of Cary and Chapel Hill. I’ve been on teams serving state agencies and local governments. I’ve witnessed dozens of open data projects both locally and abroad. All of these projects share crucial steps and goals in common. First, by transforming public documents from disconnected PDFs into machine-readable data, by applying the right open formats, we can liberate public information from the repositories where it used to require manual review. Insights become instant. Second, by publishing machine-readable data compilations online, for everyone to use, we can crowdsource those insights. Citizens and companies and the media can scrutinize. Even better, when a government age
Read more