Aral Balkan

Amazon S3 has been down for several hours now and as Jonathan Boutelle (who, by the way is speaking at the Singularity Web Conference) writes, when S3 goes down, the Internet goes down. Along with images on Twitter, SlideShare, SmugMug, and a host of other sites, images on Pistach.io ads are also down. We've also had intermittent issues with Amazon's SimpleDB. I feel that these issues only serve to highlight the age-old danger of having all your eggs in one basket. Especially a proprietary basket. This is why I applaud Google in releasing the Google App Engine SDK as open source.

The launch of services like Amazon's EC2, S3, SimpleDB and Google's Google App Engine herald the birth of the Commodity Web, wherein web infrastructure is infinitely available and metered just like electricity, water, and gas. For the most part, we don't think about the limits or availability of commodities (as the impending ecological nightmare we've woven for ourselves would attest to, if nothing else.) But when we do try those limits, or when the systems of delivery break down (as in blackouts, for example), the extent of our reliance on these utilities becomes painfully clear.

Planning for contingencies is an important part of running a successful business. Hence, for example, we have UPS units and generators to provide (at least short term) redundancy for the general power grid. Unfortunately, the same cannot be said currently for many applications that rely on the variously-available "cloud" computing solutions for their hosting.

The problem is that we lack standards and interoperability.

If tied in to a proprietary system, I cannot change my hosting provider as easily as I can, say, switch from the power grid to using my own generator. They're not the same voltage. Heck, they're not even the same frequency.

This is why having the Google App Engine SDK be open source is of utmost importance and, I believe, a spark of genius on Google's part. They've created a de facto standard for the Commodity Web. It is only a matter of time before other vendors will offer compatible and scalable application hosting based on Google's standard. And that's how it should be.

The cloud cannot and should not be a centralized behemoth that creates a single point of failure for the Internet. The cloud must be decentralized and needs standards, even if only de facto, and I feel that Google App Engine is a step in the right direction.

If you're interested in the Commodity Web, I'd highly urge you to check out the writings of another Singularity speaker, Simon Wardley, who writes and speaks extensively on the topic.

Comments

• I think this is one of the things I like about Heroku. Although it's based on Amazon S3/EC2, Ruby on Rails and the other layers are the same as what you can run anywhere else. I can build a site in Rails, pop it up on Heroku, then if it becomes popular and I end up not liking Heroku, I can quite easily put my application elsewhere. It's pretty great. if you want to try out Heroku, shout and I'll get you an invite.

  by Tom Morris on 2008-07-20 23:52:46

• Contrasting S3 with GAE makes no sense to me; the reason that GAE needed to open source the SDK is because there is an SDK in the first place. As Thomas points out, most of the SDKs that sit atop S3 are open source, and you actually get to choose between them. Amazon S3 is brain-dead simple and there is nothing stopping competitors from cloning the service -- the fact that people haven't has nothing to do with source code access. In fact, you're praising Google for open-sourcing their incompatible me-too SDK when you should be asking why they didn't just clone S3 and make it possible for people like SmugMug to instantly switch service providers in a pinch. The reason you won't see lots of competitors in this space is economies of scale. Amazon can provide the services dramatically cheaper than anyone besides Google, Yahoo!, and Microsoft. There will be only 2-5 players in this space, since nobody else could afford to price competitively without taking massive losses. In a situation like this, interop matters a lot, and Amazon is the de facto standard by a *wide* margin. I don't doubt that Google will eventually make strides toward interop for apps written on Amazon -- it is the classic underdog strategy and would help them steal business. But they haven't yet, and I don't think it's a priority for them yet.

  by Joshua Allen on 2008-07-21 01:34:40

• I think Joshua almost hit on the right point. S3 may have gone down, but like last time, it's probably for something other than machine availability. After all, Amazon uses multiple physical hosting locations. So does Google. But if GAE goes down, it's down everywhere. The real point, though, I think is that these commodity providers are providing scalability at prices that an individual can't compete with. If you want a better up time than AWS or GAE then you have to lease multiple rack locations with multiple computers and multiple backup DNS servers that self heal and redirect on errors. Getting all of that in place is costly. To use Aral's example of power, getting in place generators and UPSs to weather any sized power outage and equipment failure is also expensive. What we're missing from GAE and AWS is the ability to do this. Or are we? One might argue that having your own server in the mix is the backup plan. For S3, that's pretty easy -- just have an alternate URI mapping for when S3 doesn't respond. Instead of your own server, though, it could also be a different cloud. An EC2 app could, by default, use S3 for image hosting but fall back on images sourced from Google (GAE or or sites or something). Expecting a single cloud offering to do this is too much, though. Right?

  by Shane Conder on 2008-07-21 02:29:14

• Here is my analysis of the "quota" problem in Google's appengine that Aral had raised and that several developers have complained about on the Google app engine mailing list: http://evolvingtrends.wordpress.com/2008/07/23/google-app-engine-threat-or-opportunity/ I discuss the use of P2P (distribution) in conjunction with appengine and EC2 but the summary below is with respect to the quota problem in appengine. Summary: The problem is that they can’t tell the difference between an infinite loop and a routine that will eventually terminate (on its own.) So they give you a quota to use in a time window of a day or a month (however they’ve decided to define the quota window) and your app can burn as many cpu cycles at any given time, i.e. burst, up to the remaining quota for the given time window (hopefully, without depleting all your quota before the end of the time window.) For a “massively scalable” cloud computing solution it is very dumb for Google to offer a quota-based “free” account (that maxes out and leave you hanging) unless they are working on a way to examine your code for you (before you run it) for potential infinite loops and badly designed queries. This may sound banal but the reason for enforcing a quota for the free account is because they want to give you a limited (not infinite) free cpu cycles. The problem with the quota, however, is that once you deplete it, you will have to wait for the next quota window, e.g. 12 hours, a day or a month. The way to handle this practically is for Google to do away with the quota model (e.g. bill you for all usage) or come up with sophisticated statistical techniques to examine your code for potential infinite loops or costly queries (remember that Turing already proved that you can’t always deduce that logically) before they run it. Without such ability, applied in reliable manner, the only thing they can do when your app inadvertently hits the quota for the given time window is to disable your application until the start of the next quota window, which totally sucks. --- What am I missing? Isn't this an obvious Google Duh! moment?

  by Marc Fawzi on 2008-07-23 21:46:32

• "Amazon S3 has been down for several hours... We've also had intermittent issues with Amazon's SimpleDB. I feel that these issues only serve to highlight the age-old danger of having all your eggs in one basket. Especially a proprietary basket. This is why I applaud Google in releasing the Google App Engine SDK as open source." Don't bother reading that article......

  by Adam Fitzalan on 2008-07-24 11:07:24

• It seems like something is missing, no?

  by Surge on 2009-03-18 19:44:32

• Not only should the cloud be de-centralized for reliability, but it should also be de-centralized because a centralized internet is one that can be easily controlled. The reason why the internet is so successful today is because no government or powerful entity has complete control over it. Instead, we should implement a cloud infrastructure that is spread out over many different web servers across the globe, with data redundancy. This would allow small web server companies and large ones alike to profit from the cloud, assure the continuation of the internet's revered freedom, and provide current benefits from the cloud such as reduced web serving costs with no need for hardware maintenance on the user's end.

  by Anonymous on 2011-06-17 00:41:34