Wanted: Intelligent Design for Schema Evolution

A few days ago, I published a snapshot of the "SQLite bindings for Google App Engine." Now that the code is out, I had to decide what feature to work on next. I started with supporting a couple of more primitive types in the schema, but that didn't really seem to interesting from a blogging perspective. Thus, I have decided to tackle the idea of Schema Evolution next.

Right now, one cannot simply connect to database and start putting models in -- one has to define a table for each kind with the right columns to support our current primitives. The method "create_tables" in helpers.py does that job for us. Unfortunately, this is not very convenient compared to the real thing in App Engine, because

• I cannot simply create a new Model and store it in the database
• I cannot simply add a new Property to and existing model
• I cannot store different types under the same property name

This is something I would like to change, so I created a couple of unit tests that describe the very behavior I am looking for:

  def testNewModel(self):
    """Tests what happens if a new kind of Model gets stored."""
    class UnknownKind(TestModel):
      pass
    model = UnknownKind()
    model.put()
    
  def testAddedField(self):
    """Tests what happens if a field gets added to a model."""
    helpers.create_tables([TestModel()], self.connection)
    class Mutation(TestModel):
      @classmethod
      def kind(cls):
        return 'TestModel'
      text2 = db.StringProperty(default='some more text')
    model = Mutation(text='Text 1', text2='Text 2')
    model.put()
    
  def testTypeChange(self):
    """Tests what happens if a field changes its type."""
    helpers.create_tables([TestModel()], self.connection)
    class Mutation(db.Model):
      @classmethod
      def kind(cls):
        return 'TestModel'
      text = db.IntegerProperty(default=42)
    model = Mutation(text=23)
    model.put()

Right now, each of these tests is going to fail, as shown here:

======================================================================
ERROR: Tests what happens if a field gets added to a model.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jens/appengine/sqlite/src/unittests.py", line 176, in testAddedField
    model.put()
  File "/home/jens/appengine/google_appengine/google/appengine/ext/db/__init__.py", line 618, in put
    return datastore.Put(self._entity)
  File "/home/jens/appengine/google_appengine/google/appengine/api/datastore.py", line 160, in Put
    apiproxy_stub_map.MakeSyncCall('datastore_v3', 'Put', req, resp)
  File "/home/jens/appengine/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 46, in MakeSyncCall
    stub.MakeSyncCall(service, call, request, response)
  File "/home/jens/appengine/sqlite/src/datastore_base.py", line 104, in MakeSyncCall
    (getattr(self, "_Dynamic_" + call))(request, response)
  File "/home/jens/appengine/sqlite/src/datastore_base.py", line 154, in _Dynamic_Put
    value_list)
OperationalError: table TestModel has no column named string_text2

======================================================================
ERROR: Tests what happens if a new kind of Model gets stored.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jens/appengine/sqlite/src/unittests.py", line 165, in testNewModel
    model.put()
  File "/home/jens/appengine/google_appengine/google/appengine/ext/db/__init__.py", line 618, in put
    return datastore.Put(self._entity)
  File "/home/jens/appengine/google_appengine/google/appengine/api/datastore.py", line 160, in Put
    apiproxy_stub_map.MakeSyncCall('datastore_v3', 'Put', req, resp)
  File "/home/jens/appengine/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 46, in MakeSyncCall
    stub.MakeSyncCall(service, call, request, response)
  File "/home/jens/appengine/sqlite/src/datastore_base.py", line 104, in MakeSyncCall
    (getattr(self, "_Dynamic_" + call))(request, response)
  File "/home/jens/appengine/sqlite/src/datastore_base.py", line 154, in _Dynamic_Put
    value_list)
OperationalError: no such table: UnknownKind

======================================================================
ERROR: Tests what happens if a field changes its type.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jens/appengine/sqlite/src/unittests.py", line 187, in testTypeChange
    model.put()
  File "/home/jens/appengine/google_appengine/google/appengine/ext/db/__init__.py", line 618, in put
    return datastore.Put(self._entity)
  File "/home/jens/appengine/google_appengine/google/appengine/api/datastore.py", line 160, in Put
    apiproxy_stub_map.MakeSyncCall('datastore_v3', 'Put', req, resp)
  File "/home/jens/appengine/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 46, in MakeSyncCall
    stub.MakeSyncCall(service, call, request, response)
  File "/home/jens/appengine/sqlite/src/datastore_base.py", line 104, in MakeSyncCall
    (getattr(self, "_Dynamic_" + call))(request, response)
  File "/home/jens/appengine/sqlite/src/datastore_base.py", line 154, in _Dynamic_Put
    value_list)
OperationalError: table TestModel has no column named int64_text

----------------------------------------------------------------------

My goal for the next post in this blog is going to be to make these tests pass! That is, of course, unless someone out there gets it right first!

Does anyone have an idea how to get the job done? Code it and post the code for review. Put a comment at the end of this post, and I'll be more than happy to take a look at your result. I cannot guarantee fortune and fame, but it might be a fun project to work on :-) Just give it a shot!

(P.S: The unit tests from this blogpost are already submitted to the project)