- August 22, 2012
- 2,109 views
It’s Star Wars All Over Again!
We’ve had some pretty BIG ideas launched in the last couple of decades that you just knew from the beginning could not possibly work in our lifetime. A couple of my favorites are the Star Wars Missile Defense System (it worked okay as a negotiating tool, but as an emerging technology?…not so much) and the Iridium Phone (the phone itself cost thousands of dollars, it had limited range because it depended on launching and locating one of 66 satellites, and you could only use it outdoors).
My favorite Star Wars project, though, is one called Project Stretch. A very long time ago (but don’t worry, this was still in our galaxy) Project Stretch was launched to improve the quality of weather prediction. The goal was to deploy thousands of weather monitoring stations around the country (this was before satellites of course), and then network them together to bring the data back to a single place, crunch all the numbers, and then predict the weather.
There was only one problem.
With the amount of data that needed to be processed, and the power of computers at the time, it was going to take three days of computer time to collect, analyze and then predict the weather for the next day.
It didn’t take long before someone made the fairly obvious suggestion that they could cut two days off the time required to predict the weather – wait 24 hours, walk outside, and look up!
Today’s Star Wars Project: Can I manage Big Data in the Cloud?
I felt an odd sense of déjà vu this week reading a Wired article on Big Data that focused on Google’s “breakthrough” with Dremel … “Google’s Dremel Makes Big Data Look Small. According to the article, the breakthrough that Dremel offers is that it can handle web-sized amounts of data at blazing fast speed, processing petabytes of information in 3 seconds.
Pretty impressive, huh? Even more interestingly, Wired goes on to say, you can use Dremel today — even if you’re not a Google engineer. Google now offers a Dremel web service it calls BigQuery. You can use the platform via an online API, or application programming interface. Basically, all you have to do is upload your data to Google, and it lets you run queries on its internal infrastructure.
And here is where Star Wars enters the equation. Like most readers, I’m sure, I was intrigued, but then I started to think… “Wait a minute. Did I just read what I thought I read?”
Did anyone ask how long it would take it to upload a petabyte of data to the cloud? Read More.