Google App Engine: no support, quotas, throttling... Help!

Update: Paul McDonald, Product Manager on Google App Engine, emailed me to offer their support. Thanks, Paul, I really appreciate it.

I wrote to Google on April 19th, 2008 to ask for the quotas to be lifted on the upcoming new web site for the Singularity Web Conference.

I still haven't heard back from them.

On April 23rd, I wrote to Kevin Marks, whom I'd met at the excellent LIFT conference, to tell him how excited I was about Google App Engine and that we're using it to host the conference web site:

I'm also _very_ excited about Google App Engine. So much so that I've decided to build the Singularity web application on it. In fact, in line with the underlying philosophy of openness around the conference, I launched an open source project called The GAE SWF Project (http://gaeswf.appspot.com) to share the framework I'm building for the conference with the community (along with knowledge and examples on best practices, etc.) It's currently the featured app on the app gallery.

I'd love to talk to someone at Google about raising our quotas etc. as I want to launch the Singularity site/app rather soon and we will start seeing quite a bit of use on it. I'd love to see if Google would be interested in sponsoring the conference too -- it's going to be very relevant. (And, finally, I'd love to have a session at least on Google App Engine.)

No reply.

I know the Google App Engine team is really busy and Marzia has been responding to my various emails on the forums and personally, but I just don't feel any love towards the conference from Google, which is sad because there's definitely a lot of love from the conference towards Google App Engine.

Seriously, though, Google App Engine's technology is awesome. Their intense focus, no doubt the same as that fostered by Guido that makes Python such a joy to work with, has resulted in a truly simple system (and that's a big, big compliment coming from a guy who worships simplicity.)

However, there is one glaring weak point and that is support.

Non-existent support

Any developer who has been developing web sites and applications for any length of time knows that the first thing you look for in a hosting provider is support. Not hardware specs, not technology, none of those. Support.

It's not a question of whether things will go wrong. Things _will_ go wrong. You _will_ have unique requirements. No amount of clever technology is going to replace effective and efficient support personnel. And currently, I don't believe Google App Engine has spared thought for support.

We have the forums, and Google App Engine engineers frequent it, and we have the issue tracker but there isn't anything more personal than that. Speaking to engineers about general issues is great but the core engineers are busy people and we need support people that we can discuss issues that are specific to our applications with.

What Google App Engine lacks in terms of support, however, it tries to make up for with various quotas and "clever" automated systems.

Quotas are a bad idea

Let me state it for the record: Quotas are the dumbest part of Google App Engine. Here's why:

Let's say that you're a huge multinational Internet company Google and you want to revolutionize how web applications are built and distributed. You make all the right decisions. You make your SDK open source, thereby creating an immediate de-facto standard for commodity cloud computing. You choose an elegant and simple language (Python) and support a similarly elegant framework (Django). Then, you open it up so that developers can play with it and build beautiful things.

And they do.

They build beautiful things.

And people take notice. They take notice of the most interesting and beautiful ones. And they tell their friends.

And their friends want to see what Google App Engine is capable of so they look at these interesting and beautiful examples.

And they see:

Over quota.

Isn't Google App Engine supposed to scale? I thought that was the whole idea... As an application developer, I could care less whether the system as a whole scales or not. I need to know that my application will scale.

So we get left with a system where the most popular sites and apps built with Google App Engine all look strikingly similar:

Over quota.

(Or, even worse, exhibit weird behavior and errors because bits of them, like the urlfetch API or mail API are over quota.)

Case in point: Simon and Natalie were over at ours last week and Simon wanted to show me their Liquid Fold project, which is hosted on Google App Engine. Liquid Fold lets you track the actual browser window dimensions of your site's users, instead of the common metric of tracking their screen resolutions. It sounded really cool and I was excited to see it. But instead we saw:

Over quota.

You get the idea.

The canned response to this will be that the system is pre-release and that the quotas are temporary. I think the problem runs deeper than that. It's in the philosophy behind having a quota system that penalizes your application instead of optimistically upgrading its limits.

Teacher Knows Best vs. Customer Knows Best

The philosophy behind a penalizing quota system is the old-school mindset that Teacher Knows Best. If you've been schooled in the Turkish educational system ever, you'll know it well. Teacher sets out the rules. If you follow the rules, you are rewarded with a lack of penalty. If you don't follow the rules, or question the rules, you are penalized. Teacher Knows Best, your needs be damned.

It's not difficult to see where Google gets this philosophy from. It has worked really well for them in their search business. But search and hosting are very different creatures.

In this case, Google sets quotas and your application is penalized if it goes over those quotas. Penalized? My goodness, what an interesting word to be using for a hosting service!

Good hosts don't penalize you, they charge you for what you consume. If you consume more, they charge you more. Call me crazy, but I'm happy with that.

Instead of a system of quotas, Google App Engine should, with your prior consent of course, automatically scale you up and charge you for it. No one wants to see over quota messages at the very moment that their site or application becomes most popular. That completely defeats the purpose of being hosting on the cloud.

Although Google is going to be offering commercial plans when the service launches in earnest, I have a sneaky suspicion that, unless something changes, the Teacher Knows Best philosophy will still prevail. Instead, Google must adopt a Customer Knows Best attitude and, with your prior consent, preemptively scale your application and charge you for it.

It comes down to this:

No one should have to see an Over Quota message if they are willing to pay for the privilege of not seeing one.

It's not too much to ask for. In fact, I'd argue that it is standard practice by any web host worth her salt.

It's not the quota, it's the "clever" throttling

You may be wondering why the quota system has me worried. After all, the quotas offered even in the pre-release period are not that stingy. Most of them wouldn't even be an issue for the Singularity web site if it wasn't for "clever" throttling.

You get a certain amount of quota for various services per 24 hours but Google App Engine doesn't just wait until you've reached that limit and then cut you off. Instead, in perfect adherence to Teacher Knows Best, it tries to throttle your application so that it spreads your quota over the 24 hour period. This means that if your app gets hit particularly hard for a period of time (say you've been slashdotted or dugg), Google will start showing Over Quota errors so that you, the silly student, don't end up using all of your quota for the day.

A few hours later, when people have realized that your site is down and don't care anymore, your site will display properly.

Does this sound completely backwards to you too?

The idea behind Google App Engine is to have a scalable platform for your applications that can take the weight when hit hard. Instead, it forges ahead with its Teacher Knows Best philosophy and tries to spread your quota somewhat evenly over a 24 hour period.

No doubt this provided an interesting programming challenge for the engineers but it's an example of how Teacher Knows Best can result in too-clever solutions that are disconnected with the needs of your customers.

And it's not the only example of Teacher Knows Best gone wrong at Google. Every time I connect to the Net via my T-Mobile USB stick and head to gmail.com, I am informed that they cannot serve me from that domain in Germany. I also get the Google pages localized in German. But, wait a second, I'm not in Germany! I'm in the UK! But, heck, that doesn't matter because I have no way to tell them that I'm not in Germany, because Teacher Knows Best.

My needs are simple

As a customer, my needs are simple.

I need a stable, scalable system that never, ever, ever opts to display an error message to my users or cripple my application if it can help it.

I don't want my quota spread evenly over 24 hours via some amazingly clever algorithm. I want Google App Engine to serve each and every request even when I'm getting hit hardest and charge me for it.

Instead of trying to be too clever, be dumb but meet my needs.

After all, if you want to send 50 emails out (because, say, someone just bought 50 tickets to attend your conference and wants to assign them to a list of people), then you should be able to if you're happy with paying for the privilege. As it stands, however, there is email throttling in effect which means that, to quote Marzia Niccolai from Google, "you should not really be able to send more than about 2 emails per minute at your application's peak."

In other words, Teacher Knows Best, my needs be damned.

Help!

I would be lying if I said that I wasn't worried about the lack of response from Google to my request to have the quotas lifted for the new Singularity web site.

Singularity is a very public affair with some excellent speakers and my decision to host the site on Google App Engine was, at least on my own humble scale, a show of confidence in Google App Engine. (I still love the platform, I truly do, and, apart from these few issues, it is an absolute joy to develop on.)

I can't see how having the Singularity web site on Google App Engine is a bad thing for Google. In fact, I would have thought that they would have been supportive and would have jumped at the chance to get such public and visible support for their platform.

(And guys, if anyone from Google is reading this, my invitation to have a session on Google App Engine at the conference stands. Just leave me a comment.)

I just don't understand the silence and lack of interest on Google's part.

The silver lining

Regardless, with or without quotas, I'm confident that I've developed the new Singularity web site so that it will work without issues. As I stated earlier, our needs, at least initially, are not that great. Because of the quotas, however, I'm having to build systems (like a fake cron that pings the app every minute to send email from a queue) that I would otherwise not have had to build with my rather modest requirements.

Google could have (and still can) save me a lot of work by lifting the quotas for the Singularity web conference web site -- especially the email quotas.

I really hope that I get some sort of response from Google soon and I really hope that they address the issue of support with Google App Engine. Teacher Knows Best is a great attitude for search, but a horrible one for hosting.

If a public event like the world's first global web conference isn't important enough to warrant a few minutes from the Product Managers on the Google App Engine team to give me a definite "yes" or "no" answer to a support request, I'd hate to think what level of support an independent developer with a small application and no publicity is going to get from Google.

Comments