Aral Balkan: Historical Archive — Why Google App Engine is broken and what Google must do to fix it.

Why Google App Engine is broken and what Google must do to fix it.

I've been working a lot with Google App Engine in the past few months. I still maintain, as I reiterated recently in my Boagworld interview with Paul, that it's a great idea and has great potential. And the Google App Engine team has been wonderful in working with me and helping me out. Unfortunately, there are a couple of fundamental issues that must be addressed, and addressed properly, before Google App Engine can be taken seriously as a web application platform.

(And no, Java support is most definitely not one of them.)

Rather than writing a lengthy diatribe, I am going to succinctly list the main showstoppers that Google needs to fix to make Google App Engine work:

1MB limit on data structures

Although this is hinted at in various bits of the documentation, it is not stated plainly anywhere so let me state it for the record:

Google App Engine does not support any data structure that is larger than 1MB in size.

This includes files. So you can't host that 1.2MB PDF you want to offer for download as part of your site.

The limit affects blobs, text, and any other fields in the datastore, as well as variables in Python (so you cannot get around the limitation by breaking things up into smaller pieces in the datastore and stitching them together later in code).

The only way to get around this limit is to stitch the data structure together on the client by making several Ajax calls. The number of use cases where this is useful, of course, is severely limited.

Among other things, the 1MB limit on data structures makes it nigh on impossible to run reports and makes having an administrative shell on the deployment environment pretty much useless as serialized results of queries with hundreds of items will quickly hit the limit.

1,000 item limit on query offsets

The datastore has always had a 1,000 item limit on query results. I don't have a problem with this.

However, they recently also introduced a 1,000 item limit on offsets. This means that you can at most get the 2,000th entity for a given kind.

Again this isn't documented anywhere.

Unless you have been keeping a sortable field (like a numeric index that you were manually saving) in your datastore, this new limitation effectively locks you out of your data beyond the 2,000th item for each kind.

The fact that such a radical change was introduced without any forewarning or even an announcement is, in and of itself, troubling.

Currently, the only way to get around this limitation is to make sure that you have a sortable field that you can limit your queries on. I would highly recommend saving a numeric key for every entity. This does mean, however, that you will be doing two datastore writes for every entity (once to save the record and get its numeric id, and another to write the numeric id into a separate field that you can sort on). This is both more cumbersome and will no doubt raise the risk of your running into another Google App Engine limit: The short-term high CPU quota.

(From what I've heard, the Google team is working on adding sortable keys in a future update.)

The short-term high CPU quota

Not only are your calls limited to returning in 10 seconds but if you actually try to do anything too stressful within those calls, you will quickly hit the short-term high CPU quota limit.

Thankfully, Google has raised this limit (and the other limits) for my app but plain vanilla apps are not as lucky. In my tests, I've found that the high CPU limit can be randomly triggered even in calls that return within a second. To tell you the truth, I don't actually know what causes these. I memcache nearly everything and try my hardest not to stress the system out and yet I routinely see the high CPU warnings in my logs for even the most mundane calls.

(I've also seen strange high CPU errors in my logs informing me that I am 1.0x above the high CPU limit which makes no sense at all.)

The short-term high CPU quotas must be removed. Not only that but Google must review how it handles quotas in general.

Quotas in general

When was the last time you saw an "over quota" error on one of your favorite web applications?

Google's handing of quotas is a major step backwards to the days of Geocities and "this user has used too much bandwidth" errors. Probably acceptable if you're hosting pictures of Little Timmy and Sally Jo's Summer Camp Adventure on Geocities, not so acceptable if you're hosting your next big web app on what you thought was Google's infinitely-scalable Cloud solution.

Simply put, they couldn't have come up with a worse PR campaign for Google App Engine if they had hired Steve Ballmer to handle the job.

Think about it:

You build an awesome new app on Google App Engine. You tell your friends. They tell 1,000 of their friends on Twitter who tell 1,000 of their friends and then, suddenly, you have all these developers hitting Google App Engine for the first time to see your app. Paradoxically, by doing that, they trigger the "intelligent throttling" "feature" in Google App Engine which freaks out and shuts down your app with an "Over Quota" error -- effectively making the "Over Quota" message the first impression most of your audience has of Google App Engine.

Not good.

Especially not good when your unique selling point is that your system can scale.

We don't care that it can scale. We care that it does scale. And that it scales when you need it the most.

But that's not the worst bit.

Admin? What's that?

Currently Google App Engine does only one half of what a web application needs (and does that only half well). As it stands today, Google App Engine is a highly scalable request/response system that is tuned to handle lots of tiny calls.

A typical web application, however, needs more than that.

While concentrating on making applications scalable, Google App Engine entirely ignores a crucial use case of any web application: administration. The administrative features of your web application may not be consumer-facing but they are just as important. They may include features for running reports or mailing all of your users. Key, essential tasks for any modern web application.

With Google App Engine today, if you have more than a couple of thousand members/records in your datastore, you can forget about running any sort of admin task.

A total lack of long running processes, coupled with the 1MB limit on data structures and the 1,000 limit on query offsets means that you cannot run reports or backups. (Not that there is a data backup system currently available for Google App Engine -- I have one that I wrote myself which used to work but is currently crippled due to the 1,000 item limit on offsets.)

Similarly, unless you knew to plan ahead and create a sortable key to query on, you will find yourself locked out reaching certain records in your datastore (for example, not being able to email all of your members).

These are core showstoppers for anyone considering building a real-world application on Google App Engine and I can only hope that they are at the top of the engineering team's list of new features.

25% ready for prime-time

As things stand, Google App Engine is about 25% ready for prime time. Once quotas are handled properly and the 1MB limit removed, it will be about 50% ready. The other 50% has nothing to do with scalability and everything to do with everything else that a typical web application needs. Specifically, long running processes and the ability to run reports, aggregate data, and perform operations on large data sets.

In effect, Google App Engine is entirely missing a separate mode of operation and this glaring omission must be addressed before Google App Engine can be deemed a serious web application platform.

Priorities and showstoppers

I really hope that Google is not working on adding support for other languages to Google App Engine, as they mentioned that they were at the Google Developer Day in London, when there are such fundamentally crippling issues with the platform that must be addressed first. Adding support for other languages to Google App Engine today is like sewing a new set of drapes for a house that doesn't have any walls yet.

In summary, before Google App Engine can be used for real web applications, the following issues have to be addressed:

1MB limit on data structures must be removed.
Quotas must be handled optimistically, without crippling applications at the very moment that they should be benefitting from The Cloud. The "intelligent throttling" "feature" and short-term CPU quotas must be removed.
The 1,000 item limit for offsets in queries must be removed.
Support for long-running processes must be added.

It may just be that Google has to implement two modes of operation for each application on Google App Engine: one that is request/response-only and scales (i.e., basically what we have today, with better quota handling) and a separate admin mode that has long-running processes and isn't crippled by data structure size limits and short-term high CPU quotas.

As things stand today, running a real-world application on Google App Engine is a complete nightmare because the system completely ignores the essential administration-related use cases for web applications that we take for granted on other platforms.

Update: Several people have brought it to my attention that I forgot to mention another major showstopper for most applications, which is the lack of SSL. I ran into this early on, did my research, and found that PayPal was the only viable e-commerce solution at the time if your application needs to receive notification callbacks on purchases (ironically, Google Checkout requires SSL for its notifications API).

Comments

@Jeremy: SSL as in accessing your site via https://www.mylovelyapp.com. You can't do that. With the newly-introduced SSL support, you can access your app at https://mylovelyapp.appspot.com but you cannot carry across a session between the two. So, if you're using a custom domain for your app, Google App Engine still doesn't support SSL. It only supports SSL if you're serving your app from appspot.com.
by Aral on 2008-12-18 11:46:32
@nate: As far as I'm concerned, C can keep its curly brackets. With the amount I type, I'd rather type less :)
by Aral on 2008-12-18 11:47:41
I developed an application with LeeBeLLuL that communicates with Google App Engine in SOAP. I mainly use the DataStore, process are done in the PC. I think GAE is an effective platform with very good response time, whose implementation is easy. http://www.leebellul.com/Site/APPLICATIONS/Ressources_Humaines_V1.html
by Gérard Mulot on 2008-10-05 16:50:49
Do you need a sortable key to query on, to get a lot of records? Why won't any old sortable property do, if you're just looking to retrieve all records? The only Properties that aren't sortable are Text and Blob. My models are probably structured much differently than yours, but using key names for everything (as opposed to auto-assigned numeric IDs) has been great. Not useful for sorting, but useful for many other cases. Giving up key names for IDs just to have fast lookups that match a sortable Property seems harsh. I've run into a lot of cryptic high CPU warnings, as well. A lot of them seem to be traceable to having to do a lot of imports, often when a new main() is started. Have you checked that, or do you have a workaround for that problem (I'd love to hear if you do)? By the way, I used your GAE SWF project to get started with App Engine and with Flash, and it was immensely wonderful. Thank you!
by Nick Winter on 2008-10-03 21:57:36
"I would highly recommend saving a numeric key for every entity. This does mean, however, that you will be doing two datastore writes for every entity (once to save the record and get its numeric id, and another to write the numeric id into a separate field that you can sort on)." Any reason not to use a DateTimeProperty with auto_now or auto_now_add?
by NDG on 2008-10-04 07:38:25
good write up - great discussion folks! Wow.. I'm only a month late to the discussion. Anyhow, to chime in on languages - I've learned over the years that they're just tools in the toolbox - some better than others for certain problems - and therefore best to look at the problem first and decide if the language is suitable for needs over a given time frame. Python seems to be a decent choice - easy to learn and easy to connect to native code on the backend (when needed.) Currently I'm quite fond of PHP and Apache's mod_rewrite as I'm implementing a RESTful architecture with a massive URL space that boils down into a much more limited code/logic space on the backend. Google App engine and other "scalable" systems such as amazon's services are interesting because of the claimed scalability, the risk being that if the system doesn't deliver, or key constraints change arbitrarily or without appropriate notice..poof, there goes your product. yo ho ho!
by AndyI on 2008-11-12 17:38:00
That is a great article. I just have started playing with GAE and the one thing i appreciate most is that it introduced me to python. That is one elegant language. Coming from a java and php I wish I had looked at this language earlier. Oscar
by Oscar on 2009-04-08 03:14:32
1. It's currently free.. Who in their right mind would use a free hosting service to deploy a serious project? Nobody! 2. It's a beta service /preview release.. and will become something more like the AWS platform over time (limits and restrictions won't be in place but it will cost you) If they made it available for free with no restrictions then everybody would use it a host huge static files globally for free..
by Kris on 2008-12-10 01:18:56
I really don't think you can compare it to something a small time company has built. Google has a stack of cash, it has a team of top developers working on it and Guido Van Rossum himself is modifying Python to interact with it. Okay, so innovation doesn't come from money, but it can sure hire those that are good at doing so. The only real issue is that GAE is a relatively unique concept. I say relatively, because its constituent parts are certainly not. However, it is a good idea, and when Google are finished, it'll be an attractive platform.
by Lee McColl Sylvester on 2008-10-07 12:55:19
@David Arno: No, Google App Engine is not "a steaming pile of crap". It's a brave foray into making The Cloud accessible to mainstream developers. And, like any other emerging technology, like any other venture into uncharted territory, like anything else that is worth doing, it has issues. Who the hell cares if it is taken seriously in the "enterprise space"? Personally, I find the "enterprise space" supremely, mind-numbingly, and unequivocally b-o-r-i-n-g. I know it's probably ridiculously difficult to believe but, *shudder*, Google App Engine isn't _aimed_ at the "enterprise space". And thank goodness for that.
by Aral on 2008-10-03 21:51:21
@Aral, From Google's own blurb on their app engine: "Google App Engine makes it easy to build scalable applications that grow from one user to millions of users without infrastructure headaches." I think you'll find "millions of users" is 100% within the enterprise space that you seem so dismissive of. Sure their beta release is aimed at small scale stuff, but their aim will be to scale it up big time in the future by charging for the privilege of exceeding the quotas that are causing you such headaches. If they are serious about the "millions of users" scalability, then Java support will become a de facto requirement.
by David Arno on 2008-10-03 22:14:28
I would also say that the restriction that the "first ordering property must be the same as inequality filter property" is an impediment to adoption. For example, the following simple query is not allowed: Post.all().filter("created >= ",last_24_hours).order("-comments")
by MrChucho on 2008-10-03 21:32:03
The 1 MB limit is likely due to App Engine's inability to handle partial downloads via Range and other methods. Without Range support (file fragments) App Engine limits file sizes to what can be consumed by agents in a single gulp.
by Niall Kennedy on 2008-10-03 21:32:25
Google app engine sounds a steaming pile of crap. I'm not surprised to hear you say you wouldn't have used it if you'd known the pitfalls before you started. The real killer issue though is the one you try to brush under the carpet as unimportant. The biggest killer for Google App Engine is that it is Python based. For it to be taken seriously in the enterprise space, it has to not only overcome the ridiculous limitations you list, it has to embrace either Java or .NET. Without support for one of those two (which are the only two serious enterprise-scale solution bases out there), it will remain a small scale solution for small scale needs.
by David Arno on 2008-10-03 21:24:16
Your concerns are all valid: clearly for any non-trivial application one will need long running processes, not to mention the ability to do a table scan. We have always had this intent with 10gen, that long running processes are part of the system. By default, page requests on 10gen error out after 8 seconds. However, one can explicitly declare that an operation is long-lived. Further, our management system will soon include a "cron" facility for background tasks that run on a regularly scheduled basis. I would encourage anyone interested to check out the alpha version of the system: we would really want feedback similar to what you have written in this post. It is very early in the "Platform-as-a-service" space. These systems will be incredibly liberating at maturity, but it is still early days. btw: I am not trying to be critical of App Engine at all; our system has plenty of rough edges itself, and thus is labeled as alpha currently.
by Dwight Merriman on 2008-10-03 21:29:00
Good writeup Aral. GQL is also very restrictive. no OR and no joins and not more than one inequality in queries. Without any of these it's almost impossible to do even a basic data analysis with stored data. Forget about big enterprise applications. Even small business apps would need data analysis/reports.
by Dinesh Varadharajan on 2008-10-05 06:46:24
What is really strange - Google is reluctant to document any of these issues. I posted a simple request to add the 10 second (I thought it was 5 seconds)limit to their online documentation and they responded by saying that the current wording was sufficient. I pressed further and I was given a link to add my request as a feature request. Maybe if we all pressed further they will at least update their documentation. I lost weeks of work porting from localhost to production (localhost doesn't have the 10 second limit !!) when I discovered the issue. On the positive side, my developed application is FANTASTIC! It screams fast. I equate it to climbing a mountain. You can't look down at relational databases, FTP, Java, PHP etc. You just continue climbing. When you get to the top the view is incredible! Be creative. Think AJAX. Use Mashups. Add everything to cache. etc. Here is one view from the top - http://www.ecmhub.com. Tell me what you think?!
by ray on 2008-10-05 04:02:05
Here's another one: When you first mess around with Google App Engine on appspot, you are totally blown away by the speed. The ping time to an appspot address is extremely low, in California, it's around 6 ms. However, eventually you need to migrate over to your own domain. You would think this would be just as fast, but unfortunately, ghs.google.com is way slower and your ping time goes from nearly instant to actually slower than the average hosting provider. Compare: http://jeffr.appspot.com and http://wolfire.com You make a ton of good points, but I think the biggest issue is that Google is totally opaque with App Engine. I honestly have no idea what to expect in the next release. What is really frustrating is that Google has tons of engineers working on this thing. They've even hired the creator of Python for this project, from what I hear. However, month after month, they release these puny little updates that fix tiny little things like path parsing issues in the Windows SDK. I want to see real updates that get to the core issues that people have been complaining about since launch.
by Jeffrey Rosen on 2008-10-05 19:31:20
@david arno and other idiots enterprise scale is but a small slice of web-scale, idiot. name an enterprise that handles even a tenth of the load of youtube
by enterprise THIS on 2008-10-06 01:23:12
[...] Also: Aral Balkan - Why Google App Engine is broken and what Google must do to fix it. Subscribe to the comments for [...]
by Mr Blog » Blog Archive » Google App Engine teases, but ultimately doesn’t deliver on 2009-01-30 17:49:43
Lee: give Python another shot. Most PHP programmers I know who have looked in to Python have found it a much more pleasant programming environment, once they get over the first couple of hours of being shocked at the required indentation. David: is YouTube enterprisey enough for you? Written in Python.
by Simon Willison on 2008-10-03 23:54:46
@David : yes, C# is a great language, but the .NET CLR sucks big time. I hate using bloated runtimes, even if I'm not managing it directly. Of course, Mono is a great alternative, and you get to use C# with that, too. @Simon : I wouldn't want to be tied down as a PHP dev, but it certainly is more flexible. Personally, I'd love GAE to use the Neko runtime, but I don't see that happening for a while. It's quite funny, really, but the biggest turn off with regard to Python is its messy CFFI. If that was right, at least, I'd probably use it more :)
by Lee McColl Sylvester on 2008-10-04 00:35:12
[...] Aral Balkan - Why Google App Engine is broken and what Google must do to fix it. (tags: md google) [...]
by links for 2009-01-08 « Amy G. Dala on 2009-01-08 14:01:56
GAE now support SSL : http://googleappengine.blogspot.com/2008/10/announcing-https-support-for-appspotcom.html
by Bruno Paul on 2008-10-28 09:52:05
SSL for serving ? or SSL for accessing external sites? Ive had no issues with SSL serving and Im only using the AppEngine as a customized coded front-end for the googleplex database SSL access to offsite resources will be part of what I'll find out about when I finish seperation of the admin site from the service site. personally I am trying to setup the admin site as an own site seperate from the site administrated. so far together or apart seems equally as disconnected for all the parts.
by Jeremy kajikawa on 2008-12-17 23:45:42
Nice post Aral, I myself am suffering the pains of a "Beta" Appengine. Although I think given time, most of these issues will be addressed. Arno... You should take time out from shooting your mouth off about things you have no idea about and do some research. Java/.NET are like VHS, it's not the best, but got the right backing at the right time. From an organisational perspective, Java is responsible for some of the slowest, most ineffective, bohemeth systems on earth. Why? Purely because there are mindless eeejits who have no idea what they are talking about and jump on the bandwagon. The first mistake of any project is to ignore the requirements and just jump into a technology because somebody else has branded it *enterprise*.
by pete on 2008-11-15 09:09:21
I don't like the underlying mentality of this article which is the expectation that Google App Engine should be good, that we should use it, or that we should show some sort of deference or respect for it. It's like the bad old days of Microsoft allegiance, because it's a big successful American co. and we can't possibly be seen to criticize one of those now can we? Treat App Engine like it was from any two bit company you never heard of. It sounds like a mess, a failure and a joke. Google or not. Maybe it will get better and be the best thing since sliced bread. Maybe it won't. For now I wouldn't touch it with a barge pole, and will stick with the far, far better Amazon services. And for the record I like Google's other software and services, where they're good, and have earned it. It will take more than a company's size and success (and American origins) to impress me.
by Alex Kerr on 2008-10-07 11:52:20
ast read AND write access would be great IMHO
by Jason on 2008-10-07 15:52:03
I want to chime in on "enterprise" versus Facebook, YouTube. Enterprise means small, usually a few thousands users. Facebook, YouTube, Google play on a totally different level and have developed custom scaling techniques that are just now being published to the outside world such as memcached, sharding. I am a contractor for "very large enterprise company you know X" working on an internal platform app. Our app will serve 3000 users. There is nothing about enterprise which should be equated with scaling. Also many (not all) enterprise IT organizations are very brittle and barely functioning. Many enterprises can only digest a very small set of technologies that they have a ton of trouble managing and mastering. They are scared of new technologies because they barely keep their Java app servers running. It once took a company I worked for 2 weeks to setup a JBoss datasource. Enterprise software to me represents not only something boring, but also something very technically primitive. That is why we are trying transform my company X into less of an enterprise and more of a software as a service company.
by John Wright on 2008-10-11 17:39:45
In all honesty, no. It has lost me more time than I've gained. I would have gone with a pure Django solution. Definitely play with it, but wait until they solve the issues I've chronicled here.
by Aral on 2008-10-03 21:03:47
Thank you Aral for this great writeup. Your blog was a great help and inspiration while I was learning App Engine. The reliability issue is a huge one for me, and the lack of batch processing can prevent you from doing proper administration. I would settle for other Web hosting alternatives that are not so restrictive.
by Fernando Correia on 2008-10-04 13:14:38
Nice write-up Aral. Google needs to work on beefing up the existing offering before adding additional language support.
by Prabhakar Chaganti on 2008-10-05 17:53:13
There's no reason why an enterprise app couldn't serve the world simultaneously. It's all about scaleability, not complexity. Though, of course, complexity decreases ease of scaleability.
by Lee McColl Sylvester on 2008-10-06 08:29:23
As far as enterprise uses go, I think David is correct that Python is viewed with suspicion by businesses. This is in part due to the language itself: businesses prefer to be somewhat more conservative. (e.g. static typing, C-like syntax, etc.) Dynamic languages seem scarier and less reliable. This is not, however, a fatal flaw of GAE. Mostly I think Java and .NET are popular in the enterprise space because of the supporting corporations. Each is quite good on its own, but where would Java really be without Sun, or .NET without Microsoft? The association between Google and Python will give it plenty of credibility as a language. There is still the problem of programmers not knowing it, but I agree that Google should work on the core functionality before adding support for other languages. It sounds like these problems, although severe enough to be show-stoppers now, might not be as intractable as the scalability problems that GAE is meant to solve. Hopefully Google will fix them soon for its own sake.
by Andy B on 2008-12-09 02:33:58
Enterprise IT has to be "Reliable" above all, it won't be long before clouds like GAE will be more reliable than the servers in the local data center. Does someone really have to write a pretty printer that adds in curly braces into Python code for it to be read by all the "C" family programmers? And, I agree no-price, no-knowing.
by nate on 2008-12-17 18:30:54
If you had the chance to go back in time, will you still use it, despite all the issues you have encountered along your way ? Or wait 'til is ready for prime-time instead ? :)
by Mario on 2008-10-03 19:55:20
We are building our intranet HR app on Google App Engine. While I like its speed, on the flip side there are too many limits as pointed out in this article, and Google needs to remove them asap. Also, BigTable is not relational and I see that MySQL developers have a problem switching to the non-relational way of doing things. Google needs to support developers with plenty of documentation and example code, so they can understand the concepts learn this platform faster. They also need to communicate their roadmap. I think GAE is a wonderful platform, but unless Google moves fast and fixes the problems, they are going to lose developers.
by Prathapan Sethu on 2008-12-08 13:48:22
@Lee I agree that Python's syntax is nasty (and that it is a big deal, even though it ought not to be), but PHP? It's OK I guess, but it's C# that kicks arse as the best language around in my not-even-slightly-humble opinion. Sadly for me, I picked the other half of the enterprise space for a job (it's sweet that Aral is trying to teach me what enterprise is therefore :) ) and so I'm "stuck" doing Java & Flex. Anyone using JEE ought to be able to handle big numbers of users in their code. If they can't then they shouldn't be doing JEE. Of course, there are plenty of people in the world that shouldn't be doing doing what they are doing though... :D
by David Arno on 2008-10-03 23:45:13
Millions of users require a good server, not enterprise features. As for Java support, even ASP 3.0 can handle a HUGE user hit. Besides, it's far easier to get Python to handle more users as standard than a Java application that has been built by someone unused to developing for such large numbers. In my opinion, Java support would be a waste of time, unless it was a bespoke VM which Google have been known to tinker with :D My take on Google App Engine is simply that I'm not a fan of Python. Sure, it's a powerful language, but I don't like its syntax... It's too reminiscent of BASIC languages of old, and for some reason that's a big deal for me :) I think I'll wait for them to fix the bugs AND add another supported language. PHP would be nice!
by Lee McColl Sylvester on 2008-10-03 23:30:17
"Adding support for other languages to Google App Engine today is like sewing a new set of drapes for a house that doesn't have any walls yet." LOL
by Wilcox on 2008-10-03 22:57:11
You've hit it right on the head. Your article should be required reading for anyone considering building an on Google's platform.
by Michael on 2008-10-03 23:00:49
@David Arno: Read this to learn what enterprise software means: http://en.wikipedia.org/wiki/Enterprise_software
by Aral on 2008-10-03 23:20:45
Python and the watered-down Django that runs on AppEngine is not a major hurdle for me. While I prefer Ruby or even PHP for web applications, Python is more than suited to web application tasks. However, the items listed in the article are absolute show stoppers… * …1M limit on files, date entities, etc.… …early on, created a simple app that relied on /dict/words but had to restrict to under 9 characters as the file upload is capped at 1M. * …I was bumping up against CPU quotas for simply accessing one datastore record and unpickling a 200-300K compressed object (and for some transactions, storing it back). * …OK with non-relational DataStore storage, and even the 1000 call limit as I think it obscene to do that much DB reading in a single transaction. However, the lack of aggregate functions and inability to run batch jobs really put a weighted damper on constructing maintenance functions. I'm all for "you're in and out in a snap" for the user transactions, but there are functions and reports where you need hit the complete data repository (in batch/offline mode). Or even access summary data without expensive I/O/CPU expended. * …these limitations make it impossible without extraordinary AJAX hacks to backup your application data. Unless you keep it under 1000 records…
by Naum on 2008-10-04 02:29:06
Great writeup, Simon. I agree it's pretty shocking how hidden many of the App Engine limitations are -- you typically don't know about them until they smack you across the face. To me, App Engine only makes sense as part of a hybrid solution, i.e. working in tandem with other traditional servers to fill in the gaps you mention, although the pure database limitations don't have that luxury. @David Arno: languages are rarely the limitation in making any system hold up under any kind of strain
by Adam Fisk on 2008-10-04 01:31:50
Er, I mean "great writeup Aral."
by Adam Fisk on 2008-10-04 01:32:56
They should also fix the search feature. Btw I just tweeted that :)
by LiveCrunch on 2008-10-04 04:59:45
But (IMHO) it's meant to be limited while they're in this early-adopter phase so people don't impact each others' apps too much. They'll use this phase to build out what people *really* need, based on the tons of feedback they're getting. I'm amazed they produced something that is good enough to have these issues as the first things we want to talk about! What I'd like is better communication and a roadmap. When will they drop the limits?
by Richard on 2008-10-04 06:44:45
@David re enterprise: I think all Aral is trying to say is that, regardless of scale, GAE is meant for apps targeted at the public, where things like security, privacy, and accountability are the most important items, no serving up millions of pages a day. Unfortunately, GAE doesn't do that, either. @all First, are these limitations supposed to be indicative of the final (or beta, as Google will call it) version of GAE that will be available after it's out of its beta (aka preview release)? Or are they just limitations imposed on the preview release to keep it from being used by release software? Second, I see evidence of some of these limitations within Google's own apps. Do we think these limitations are about the approach to the problem rather than the problem itself? I think it's safe to say, the most of us are comfortable thinking in relational databases with PHP. The Google datastore is not relational and they have figured out how to use python for their apps (or so we've been led to believe). This thought process and approach is found in many platform shifts. The problem is, there isn't much said about how to approach problems in this new way, if that is really the intention. In addition, things like tiny query offset limits seem to go against that and strengthen the argument that they may just be preview release limits. On the other hand, with a platform optimized for search and showing folks on a few things before they search again, can work with this. I agree that, as is, GAE can't handle the sorts of things external folks need to run apps within Google, such as create administrative interfaces, backups, etc. Those things, though, aren't an issue with Google, since it's all running on their servers and they can approach that issue from behind.
by Shane Conder on 2008-10-04 13:26:17
the missing https protocol is also really annoying
by poko on 2008-10-05 01:21:51
@david a. => f__k enterprisey-ness. voluntarily censored because i'm a guest in aral's house. oh, and david.. "it has to embrace either Java or .NET. Without support for one of those two (which are the only two serious enterprise-scale solution bases out there), it will remain a small scale solution for small scale needs." .. youtube's written mostly in python. hardly small-time, wouldn't you agree? .. aral.. thanks for the article.. as a rubyist, gae is one of those things i had a bunch of questions about but i didnt feel like learning python just to see what its all about. valuable insight, def appreciated.
by lowell on 2008-10-05 04:54:00
@Lowell : Eve Online's server is written in Python, too, and that holds the record for most simultaneous gamers on one server... And I believe it didn't struggle even then
by Lee McColl Sylvester on 2008-10-05 09:39:56
the lack of https is the second most popular issue in the gae issue tracker: http://code.google.com/p/googleappengine/issues/detail?id=15&q=https&colspec=ID%20Type%20Status%20Priority%20Stars%20Owner%20Summary%20Log so i believe it's a showstopper too
by poko on 2008-10-05 12:30:47
Thanks, poko, I updated the article to mention the lack of SSL.
by Aral on 2008-10-05 16:12:57
Aral, Compliments with your article. Meanwhile using https is a piece of cake. Just adapt the yaml file. The storage limit of 1 MB is ridiculous!. Youtube (=Google) and plenty other systems allow much more! The CPU limits are painful. It is easy to get over the short-term high CPU quotas. Sending more than 10 emails at one submit request is impossible. You have to wait every time 1 minute before a next group of 10 emails can be submitted. Using Blind Carbon Copies (BCC) does not work at all. So a lot to improve for Google. The term SCALABLE is not yet fulfilled by Google! To end with good news: The reponse times on normal (smaller requests) are great!!
by stelg on 2008-12-28 19:02:55
[...] In the right circumstances, App Engine can be a real boon. I run my own site using the method I’ve discussed above, and I’m very happy with it. But App Engine does have its disadvantages, most notably those discussed by Aral Balkan in his post ‘Why Google App Engine is broken and what Google must do to fix it‘. [...]
by Using Google App Engine as Your Own Content Delivery Network | Castup on 2008-12-29 10:30:50
thanks
by Muhabbet on 2009-01-29 08:21:13
This is a good write-up as to some of the issues. I'm playing w/ G.A.E. and have run up against some surprising limitations, but you know, it wouldn't kill me to write more light-weight code. I don't want to be one of those enterprise drones that throws money and hardware at everything until it works...
by SDC on 2009-01-10 22:50:35
Just wanted to point out to anyone else reading the archives that Aral posted an update: http://aralbalkan.com/1975 You can also see the new quotas here: http://code.google.com/appengine/docs/quotas.html
by David on 2009-03-27 21:05:40
Hi, We are getting below error while developing our application with google app engine. Do you have any idea baout this? Uncaught exception from servlet java.lang.OutOfMemoryError: Java heap space at java.nio.HeapByteBuffer.(Unknown Source) at java.nio.ByteBuffer.allocate(Unknown Source) at com.google.apphosting.runtime.jetty.RpcResponseGenerator.(RpcRespo nseGenerator.java:36) at com.google.apphosting.runtime.jetty.RpcConnection.(RpcConnection.j ava:41) at com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceReq uest(JettyServletEngineAdapter.java:94) at com.google.apphosting.runtime.JavaRuntime.handleRequest(JavaRuntime.java :235) at com.google.apphosting.base.RuntimePb$EvaluationRuntime$6.handleBlockingR equest(RuntimePb.java:4755) at com.google.apphosting.base.RuntimePb$EvaluationRuntime$6.handleBlockingR equest(RuntimePb.java:4753) at com.google.net.rpc.impl.BlockingApplicationHandler.handleRequest(Blockin gApplicationHandler.java:24) at com.google.net.rpc.impl.RpcUtil.runRpcInApplication(RpcUtil.java:359) at com.google.net.rpc.impl.Server$2.run(Server.java:800) at com.google.tracing.LocalTraceSpanRunnable.run(LocalTraceSpanRunnable.jav a:56) at com.google.tracing.LocalTraceSpanBuilder.internalContinueSpan(LocalTrace SpanBuilder.java:510) at com.google.net.rpc.impl.Server.startRpc(Server.java:756) at com.google.net.rpc.impl.Server.processRequest(Server.java:348) at com.google.net.rpc.impl.ServerConnection.messageReceived(ServerConnectio n.java:459) at com.google.net.rpc.impl.RpcConnection.parseMessages(RpcConnection.java:3 19) at com.google.net.rpc.impl.RpcConnection.dataReceived(RpcConnection.java:29 0) at com.google.net.async.Connection.handleReadEvent(Connection.java:419) at com.google.net.async.EventDispatcher.processNetworkEvents(EventDispatche r.java:762) at com.google.net.async.EventDispatcher.internalLoop(EventDispatcher.java:2 07) at com.google.net.async.EventDispatcher.loop(EventDispatcher.java:101) at com.google.net.rpc.RpcService.runUntilServerShutdown(RpcService.java:251 ) at com.google.apphosting.runtime.JavaRuntime$RpcRunnable.run(JavaRuntime.ja va:373) at java.lang.Thread.run(Unknown Source
by Sachin on 2009-06-03 15:15:13
[...] In the right circumstances, App Engine can be a real boon. I run my own site using the method I’ve discussed above, and I’m very happy with it. But App Engine does have its disadvantages, most notably those discussed by Aral Balkan in his post ‘Why Google App Engine is broken and what Google must do to fix it‘. [...]
by Using Google App Engine as Your Own Content Delivery Network « Internet Turnkey Websites on 2009-05-07 23:36:44
Java support is now in the engine experimentally apparently http://googleappengine.blogspot.com/2009/04/seriously-this-time-new-language-on-app.html I have not tested it myself.
by Simon Holywell on 2009-04-08 08:44:50
Interesting how Google's introducing Java support stopped the Python fanatics' babbling on this post.
by Ralph on 2009-05-19 07:47:32
[...] a query that returns all the data for you as there is a limit of 1000 tuples per query, and as this post mentions, you can’t even use offsets more than 1000, in essence limiting your queries to return at [...]
by What I dislike about Google AppEngine « Gautham Pai «buzypi.in» on 2009-06-22 18:09:07
I just ran into the 1mb limit with the datastore. Soooo frustrating! Especially since I wasn't expecting it. I assumed it was 10mb, just like file upload. Silly me! I'm only trying to create a custom photo-sharing website, so my family can post pictures of our yearly reunions. I first tried to upload the pictures statically and glob them, but lo' and behold, the static directory is not available to the filesystem! Yay! Storing the photos via the API was actually my second strategy. I had everything done and tested and that's when I hit the 1mb limit. Ever since they went public, Google has become more like Grr!gle. Think I'll try out Media Temple next. I hear they're pretty good.
by Kevin on 2009-08-04 05:04:27
Hey Werner, Would you be willing to address some of those points. If you can write up a short update (in list form, perhaps), I'd be happy to update the article with it (I'm not actively developing with Google App Engine at the moment). Thanks! :)
by Aral on 2009-08-18 10:48:22
[...] In the right circumstances, App Engine can be a real boon. I run my own site using the method I’ve discussed above, and I’m very happy with it. But App Engine does have its disadvantages, most notably those discussed by Aral Balkan in his post ‘Why Google App Engine is broken and what Google must do to fix it‘. [...]
by Using Google App Engine as Your Own Content Delivery Network « Why Limit Media on 2009-08-30 14:44:36
@Kevin What about the Picasa API? Since this article was written appengine evolve slowwwly :) I like to menion Scheduled Tasks: the http://code.google.com/appengine/docs/python/config/cron.html But there are other small nice new features like Kindless queries. Aral, would be great if you could Update your Article and mark some of your claims as outdated.
by Werner Hartnagel on 2009-08-16 10:48:39
Hi Aral, Thanks for writing this. At first sight, google app engine seemed to be every developers dream. It lets you do what every developer loves to do ( writing programs that get stuff done ) while removing all the boring stuff ( like maintaining your server, packaging and deploying your app, worrying about performance etc) But your article makes it clear that GAE is not there yet. I started out with rails. it is beautiful. But I worried about performance and decided use Jruby to make it faster. Then thought may be GAE might be better for me. I know as a framework rails is way ahead of GAE but the idea of not worrying about hosting and maintenance and the low price tag (starting from zero) made it so attractive. But thanks to this article, I avoided wasting my time with GAE now. But I am quite hopeful one day it will be ready for serious development. And that will be an exciting day for every developer I think.
by sam on 2009-09-11 13:44:33
To address the 1MB datastore storage limit, here is an open source project (mine), which solves the issue. Everything is implemented, except the queries (it's still a work in process). http://code.google.com/p/mindash-datastore/
by Tristan on 2009-09-28 19:40:20
@Jad & @Mark C: I couldn't agree more. There are no dates in the post, in the url, nothing. I'm having a terrible user experience since I'm used to have these information in other blogs. Is this old information? I'm planning in using app engine, came across this post, and don't know if all of these are old news. We NEED these features in a blog. Just because your motto is "The age of features is dead; Welcome to the age of User Experience." it doesn't mean you have to remove all of them. Aral, is there a plausible argument about removing such important information from your posts as dates?
by __B__ on 2009-11-30 19:42:49
There is nothing worse than a web post with no date. Reading this I have no idea if it's valuable up-to-date information, or some ancient artifact that it's safe to disregard completely. None of the comments have dates either. Unbelievably lame.
by Mark C on 2009-11-12 18:26:10
@Mark C: Lol I could not agree more "Welcome to the age of User Experience" (per blog tagline) -- coming from the "age of features" like an idea of when something was posted, I prefer features.
by Jed on 2009-11-28 04:12:51
Two real show stoppers for me: 1) The lack of an OR clause in GAE's version of SQL. I need to create seperate queries and append the results in memory. If there is a sort on the concatonated results then the CPU time is extended - and sometimes the CPU time limit is blown. 2) You cannot filter on the same field twice in GAE's version of SQL. Meaning I cannot set up a construct of Field A = "1" and (Field B > "1" and Field B < "2). I need to create very complex workarounds for this.
by PAul on 2009-12-09 09:36:51
All points are very valid. I am also curious how google is going to add support for the batch processing. I think it will be some Map/Reduce flavor. It will ve also interesting to estimate - why do they built such strict limitations on thier platform.
by David Gruzman on 2009-12-20 17:40:22
[...] cannot be bigger and sice the only storage you have is the database, you are kinda screwed (like this guy, who jumps first for most of related google [...]
by bigblobae – store big data blobs on Goolge AppEngine | Jan Sterba on 2010-02-09 00:39:15
Nice article! Does GAE still have these limits you stated in this article?
by Wenguang Wang on 2010-04-03 01:31:13
For those just getting started with the app engine in Windows, the tutorial Google has up has a few flaws. It makes the initial experience rather unpleasant. I’ve noted a couple of them and posted some workarounds at http://wheatlandcomp.wordpress.com/2010/03/26/tutorial-for-google-app-engine-is-flawed/
by Scott P on 2010-03-26 18:41:23
[...] A detailed critique of the app engine is at Why Google App Engine is Broken… [...]
by Google App Engine « Wheatland Computing on 2010-03-26 18:46:24
This article is a great outline of the problems that many of us have faced while working with Google App Engine. However, it is currently in "preview release" which basically means "pre-beta", so we don't expect it to be 100% production-ready yet. They are adding and changing features constantly. In fact, many of the problems mentioned in this article are no longer issues. A datastore cursor, for example, fixes the offset limit. A cursored query allows you to skip right to the place where you left off previously, and performs much better than an offset query. The new blobstore api allows entities up to 2 gigs. I'm not sure how this interferes with the 1mb variable limit at runtime, but I'm sure there are some hacks out there to make this work (probably using taskqueue, depending on the job). Although we have encountered various roadblocks, I have maintained my optimism with App Engine for the following reasons: 1. The free quota is enormous and the resources beyond that are cheap. There is simply no comparison to other cloud hosting solutions. 2. Google takes care to prevent you from shooting your foot off. Sometimes they prevent you from doing something because it wont scale. Finding a way around the limitations is the key to making a scalable app. A virtual/dedicated solution assumes that you will have the know-how to scale everything on your own, which may not always be the case. 3. I LOVE Python. My jaw drops when I see someone actually prefer PHP or C# over it. I urge people to take a serious, objective try at Python. You'll never need curly brackets again :)
by James Watkins on 2010-08-22 21:22:50
The issue concerning offsets limit is over : http://googleappengine.blogspot.com/2010/08/multi-tenancy-support-high-performance_17.html
by Matthieu on 2010-08-31 21:14:06
This is an interesting Engine article. You create a great lens for me. I have enjoyed reading it. Thanks Joshna “Engine & Transmission World”
by Joshna Rani on 2011-10-11 15:01:20
I know this article was posted in 2008 and probably my fear is not valid anymore -- but to confirm, can anyone tell me if Google app engine does not support (still) a data structure more than 1MB even now? Thanks!
by Ravi Jay on 2011-11-21 14:08:18
Dear All I'm working in Google App Engine with J2EE and Facing a problem with this my application run on the local with Goolge Cloud Sql but when deploy my application on the google then give the error is "Error: Server Error The server encountered an error and could not complete your request. If the problem persists, please report your problem and mention this error message and the query that caused it." can you give the solution for it
by Shilendra Sharma on 2012-05-17 12:30:38
[...] Másképpen is illusztrálhatjuk a Windows Azure eddigieknél jóval nagyobb távlatosságát és átgondoltságát. Nagyon sokan a Google-t tekintik a felhőplatform kialakításában technológiailag élenjáró vállalatnak. Az idén áprilisban béta formában megjelent Google App Engine (GAE) esetében 1 MB a fájlok méretkorlátja, ami roppant szűkös (még a weben jelenleg fellelhető PDF-ek is ennél nagyobbak lehetnek). A Windows Azure-nál ezzel szemben, ha egy műveletben töltünk fel egy blobot, akkor annak maximális mérete 64 MB. Emellett maximálisan 4 MB-os blokkokból is megszervezhető a feltöltés, amikor akár 50 GB méretű is lehet egyetlen blob, ráadásul ez csak egy jelenlegi, praktikus korlát. További ezzel kapcsolatos példa az, hogy minden adatot (blob, queue, table) három példányban tárolnak, mégpedig különböző meghibásodási (fault) és frissítési (upgrade) doménekben, így azok gyakorlatilag nem veszhetnek el. Később az adatközpontok közötti replikációt is bevezetik. És ez csak egyetlen részterület a lehetséges sok közül.Megjegyzés: A Google kezdeményezését egész egyszerűen játékszernek titulálta a Gartner egyik elemzője („At present, Google App Engine remains a toy”). A fejlesztők is használhatatlannak találták az egészet, egyikük írása tanulságos összegzése az elkövetett, súlyos tervezési hibáknak („Why Google App Engine is broken and what Google must do to fix it”). [...]
by Windows Azure: mi itt a lényeg? - Nacsa Sándor szakmai blogja - devPortal on 2012-08-10 09:18:45