SORRY FOR BLOWING UP YOUR WEBSITE!

-

The other day, I flooded a customer website licensed for 75 datasets with 318 additional in five minutes while using the OpenDataSoft ArcGIS Harvester. Yup. I did that. I felt not so much like “Darth Vader” as much as I did “Dark Helmet” (Rick Moranis in Spaceballs, see below if this reference goes over your head).

I was using OpenDataSoft’s ArcGIS Harvester. It’s still in private beta, but will likely be available by the end of December (consider it a Christmas gift to you!) for data publishing. Luckily, the Harvester also came with a delete button so the customer was not affected. I took the usual ribbing from my co-workers. David Thoumas added that we do not need “too many datasets”.


I was awed with the power of the Harvester.


I had seen it work a year ago and take most of an afternoon to mine data. I innocently put the root of the GIS server into the Harvester, and suddenly it was pulling in data at “Ludicrous Speed”. Ian Henshaw noted we hit hyper drive and “They’ve gone to plaid”.


SEVERAL TITLES FOR THIS BLOG POST CAME TO MIND:

“With Great Power Comes Great Responsibility”

“Young Jedi uses too much Force in Releasing Data”

“We’re Gonna Need a Bigger Boat”

“How Jason Hare Blew Through a 75 Dataset License in 5 minutes”

At first I felt like I was aiming the Death Star at Planet Alderaan. Heck yes I was going to show some folks how to ingest data with our giant Hoover Machine. As the datasets quickly tallied I realized I was ingesting an entire ArcGIS server. It went from 1 to 15 in about 2 minutes. By 3 minutes we were at 150 and by 5 minutes we had all 318 datasets.


WHAT I LEARNED AND WHAT YOU SHOULD KNOW

The OpenDataSoft Harvest Tool can pull data, with rules, from an ArcGIS server with two clicks. We can harvest other types of data as well. This was a learning moment for me. “Choose wisely”. We can also harvest this data very quickly (remember Ludicrous Speed).


No worries. After David Thoumas grounded me from using the Harvester I got a quick lesson from our awesome developer team. I am no longer considered a threat to our Open Data community or customers.


Original article: https://www.opendatasoft.com/2016/05/26/sorry-for-blowing-up-your-website/

Comments

Post a Comment

Popular posts from this blog

Podcast: Open Data Discussions with Anthony Fung

-
Image
A look back at Virginia’s 4 years of data innovation with Anthony Fung, Deputy Secretary of Technology for the Commonwealth of Virginia  Welcome to the OpenDataSoft Leadership Podcast Series, “Open Data Discussions”. Each month, Jason Hare, our Open Data Evangelist, features a different open data program around the country to discuss what has made it successful. These examples will provide insights and strategies that you can implement in your own city. We had the pleasure of welcoming Anthony Fung, Deputy Secretary of Technology for the Commonwealth of Virginia for the fifth podcast in our series. Anthony discussed various subjects, including: What role does open data play in policymaking at the state level in Virginia? Does his department work on ensuring open data quality? What are some of the steps he makes in ensuring sound data governance? How did Virginia come up with this technology and engagement solutions?  How, for exampl...
Read more

WHITE HOUSE OPEN DATA INNOVATION SUMMIT - WHAT I SAID, WHAT I MEANT TO SAY

-
Image
+OpenDataSoft participated in the celebration that was 8 years in the making at the White House. Yesterday, for the first time, the White House held the 2016 Open Data Innovation Summit.  I work for OpenDataSoft as the Open Data Evangelist. I was asked to introduce the White House Agenda and then to speak on Civic Innovation regarding Open Data. What I said and what I wanted to say were different. The event was full of excitement and also of reflection Megan Smith and Tony Scott kicked off the event touting the 200k datasets that had been released through Data.Gov. Certainly this is refreshing from even a decade ago when serious effort and even lawsuits were required to make data available to the public. That all was not good and could be better was brought up by +Alex Howard from the Sunlight Foundation. Alex noted that HTML and PDFs were counted among the "datasets" being released. Note that while HTML and PDFs are indeed 1 star ...
Read more

Open Data Licensing

-
Image
Everything you wanted to know but you were afraid to ask. Making data open and available is not just about choosing a format or having an API. A poor choice in your license can prevent people from using them. A traditional open license reassures potential users that their work on the data will be useful in the long run. So, what license should you pick for your data to be broadly used? There are a few things to consider. Choosing a license for your open data can usually be one of the biggest challenges for an open data project. There are plenty of licenses available for your data. Choosing one mostly depends on the usage and reach you want to give to your project. CHOOSE A LICENSE: BASED ON NATIONAL, INTERNATIONAL AND LOCAL STATUTES IN ADDITION TO REUSE It should be noted that different parts of the world use data licensing terms differently. In the US, works in the “public domain” are works are not covered by intellectual property rights, such as copyright. The copyright mig...
Read more