Gaebar

Like many other geeks, I am slightly distracted from everything that is computer throughout the holidays (except for playing Coloroid on my shiny new G1 phone :-). So, I'd like to send a quick shoutout to Aral Balkan who (unlike me) did not rest and released Google App Engine Backup and Restore (Gaebar), a module that can be used to backup and restore an App Engine datastore between different application servers. Beyond the obvious benefit of being able to get your data out of App Engine easily (which is cool, because it makes the platform even more open), it will also enable people longer term to do pretty sophisticated maintenance and system integration tasks. Assume for example that you run an online shopping cart but need to move your orders into an ERP or shipping system. Many of these systems are based on relational databases like Oracle or SQL Server. You could use a tool like Gae-SQLite to connect a local app engine instance to a relational database (I know it doesn't support many databases yet, but if there are enough helping hands from the open source community, it could). Combining this with gaebar, you could write your data into the RDBMS. Once it is in a relational view, the data should only be a script with a few INSERT statements away from being put into your ERP or shipment system.

I am sure there are many other cool ways to use this that I have not thought of -- like migrating data from one schema to another if you need to refactor your models during a new app release. What else could you guys see yourself doing using the Gaebar module? Post it to this thread :-)

Build App Engine apps using Zoho Creator

Someone just sent me this amazing link to a recent post on the Zoho blog! Zoho has an online tool called the Zoho Creator, which enables users to compose Web Applications using a visual editor. Sounds good? It gets even better: the team recently added a feature that converts such an application into an App Engine app! You can download your app as python code and upload it to App Engine. I think that rocks -- congratulations to the developers at Zoho!!!

Back to business?

Last week, the office of the president elect launched "Open for Questions", a way for the people to let the transition team know what's on their mind. I am not really a very political guy, and this is going to stay a tech-centered blog. However, I believe this particular event has an interesting aspects that warrants mentioning (and I am not talking about the grassroot efforts that made a pot-related question score higher than national security or economy-related asks).

If you take a look at the site itself, you can the following disclaimer:

This tool is powered by Google Moderator, a third party service.

I would like to repeat: change.gov used Google Moderator, an App Engine based application, to process the almost one million votes on roughly ten thousand questions. Upon closing the first session of questions posed, the team commented

We're tremendously excited about the promise of tools like this that offer Americans a level of access that has historically been hard to come by. By voting questions up, users have been able to convey to our team which major issues -- like the auto industry, health care, ethical standards, and others -- are the most important to this community.

The same week, a post on the App Engine blog announces a library for easier integration with Salesforce, opening up a whole new world of possibilities for enterprise developers. As the blog states,

It's fantastic to see App Engine gaining traction in the world of enterprise software--whether you want to deploy a browser-based tool to members of your organization or a personalized, customer-focused app that scales to millions of users, App Engine's ease of use and scaling power make it an ideal choice for enterprise development.

What do these two events have in common (besides App Engine, of course)? They demonstrate that Google App Engine has many more applications than being an inexpensive way of hosting your homepage or serving as backend for OpenSocial apps. While I do not doubt that there are very successful consumer-focused applications that can make a lot of money through ads, I think the biggest source of revenue for most software vendors out there is the enterprise. Microsoft Office and Exchange are great examples of how selling to businesses throughout the world can be a very, very, very profitable thing to do. But one does not need to be the next Microsoft to make a decent living. From finance to organizing one's company offsite, there are many markets to create and sell applications to the working crowd. Unlike Moderator for change.gov, they usually have a very restricted user base -- scalability is a little bit less of a concern, and useage per company might even be low enough to fall into App Engine's free quota. Also, since App Engine works out of the box with Google Apps for business, many infrastructural problems like how to integrate a company's SAML login with your cloud app, are already taken care of.

The biggest problem for turning an idea for a business app into a business for apps right now seems to be the question of installation. Google already has a marketplace that lets you install Moderator for your domain. While there is not a great way yet to do something similar with your own app, the marketplace does have a "become a vendor" link that one can use to promote his or her own Google related services. Service vendors can use this venue to promote what they have to offer, and their customers can express their satisfaction through rating and feedback. It seems like a good place to get the word out if you have a great business App on App Engine. Installation would still be manual though (you would need to create a new application instance for each domain). I'm kindof curious -- has anyone out here tried this out (or something similar)? Are there any alternative ways of promoting your business app that I have missed? Please do not hesitate add comments to this post.

Gotcha

Every now and then, one runs into something he or she would not have expected. This was recently the case for me, and since it turned out to a typical case of RTFM, I thought I'd share it with the rest ;-)

I had not used entity groups a lot in the past, but I recently ran into a situation where they would have been useful (I wanted to write several objects in one transaction). Everything seemed to work just fine -- until I tried to retrieve the objects from the store. The following script reproduces what happened to me:

import os
from google.appengine.api import datastore_file_stub
from google.appengine.api import apiproxy_stub_map
from google.appengine.ext import db

class TestModel(db.Model):
  text = db.StringProperty()
  
# Set up an in-memory db store
os.environ['APPLICATION_ID'] = 'test'
stub = datastore_file_stub.DatastoreFileStub('test', None, None)
apiproxy_stub_map.apiproxy.RegisterStub('datastore_v3', stub)
  
# Store a model without parent in the db
model1 = TestModel(key_name='the key', text='t1')
key = model1.put()
assert 't1' == TestModel.get(key).text
assert 't1' == TestModel.get_by_key_name('the key').text

# Store a second model with the same key_name,
# but with a parent
parent = TestModel().put()
model2 = TestModel(parent=parent, key_name='the key', text='t2')
key = model2.put()
assert 't2' == TestModel.get(key).text
assert 't2' == TestModel.get_by_key_name('the key').text,\
       'Text is %s' % TestModel.get_by_key_name('the key').text

The very last line of this script fails. Here is the error that
it produces:

Traceback (most recent call last):
  File "keytest.py", line 27, in <module>
    'Text is %s' % TestModel.get_by_key_name('the key').text
AssertionError: Text is t1

What exactly had gone wrong? Why did the second model not overwrite the first one? The answer to this question can be found in the documentation:

The keys of two different entities can have similar parts as long as at least one part is different. For instance, two entities can have the same kind and name if they have different parents. Similarly, two entities can have the same parent (or no parent) and name if they are of different kinds.

In other words, just because I specify a key_name in a model's constructor, does not necessarily mean that I can look this model up with get_by_key_name -- at least if I do not know the parent. Like in this example, there can be several independent entities that all have the key as key but are not the same. One can (and probably should) use the key name for fast lookup (it certainly beats GQL where available), but know that if we choose to do so, this might limit some of our flexibility in partitioning the data using ancestors.

In retrospect, I probably should have known. App Engine is by far not the first store that got me into trouble for using the ids for more than they were intended to (in a "past life", I had an experience not unlike in this Hibernate article). I guess it's true: Those who do not remember their past are condemned to repeat their mistakes.

PS: This also means that the gae-sqlite project is going to need a schema overhaul at some point, since my store currently discards the entity group entirely...