Software Archeology -- getting insights, crash by crash

A current hobby project of mine is attempting to port the JOGRE game server to App Engine. So far, things have been slow but steady. Through a couple of refactorings, I isolated how client-server communication takes place. I ripped out the socket-based protocol and replaced it with a JSON based polling mechanism (for some more details on its implementation, you can check out this previous post).

Today, I am starting with a different and equally challenging aspect: persistence. JOGRE has four basic types of data, which can be expressed through the following schema (using protocol buffer notation):

message GameInfo {
  required int64 id = 1;
  optional string game_key = 2;
  optional string players = 3;
  optional string results = 4;
  optional int64 start_time = 5;
  optional int64 end_time = 6;
  optional string history = 7;
  optional string score = 8;
}

message GameSummary {
  required string game_key = 1; 
  required string username = 2; 
  optional int32 rating = 3; 
  optional int32 wins = 4; 
  optional int32 loses = 5; 
  optional int32 draws = 6; 
  optional int32 streak = 7;
}

message Snapshot {
  required string game_key = 1; 
  optional int32 num_of_users = 2; 
  optional int32 num_of_tables = 3;
}

message User {
  required string username = 1; 
  optional string password = 2; 
  optional int32 security_question = 3; 
  optional string security_answer = 4; 
  optional string year_of_birth = 5; 
  optional string email = 6; 
  optional bool receive_newsletter = 7; 
}

A first goal should be to build a way of writing this information to the App Engine datastore. Performance is a non-goal of the first stage, since I currently do not know enough about the data to really have a feeling of where the bottlenecks might be. Getting the first iteration complete should give me a pretty good idea, though.

Step 1 is to find a seam of where to insert datastore functionality (I will wrap datastore access using the Persistence interface, but the concept should be same for JDO, JPA, or low-level API users). JOGRE already has a couple of different stores, but they seem a little copy-and-paste-ish (duplicated logic in ServerDataDB and ServerDataXML for example), and I'd rather not create another very similar copy with very similar code.

After some more digging, I found the perfect candidate to refactor: the SQL-based persistence was using an inner class called IBatis to translate operations on POJOS into SQL commands (through the open source project of the same name.). Through the "extract interface" refactoring, which can be automated through Eclipse, I replaced the use of the IBatis class through a interface with the same method signature:

package org.jogre.server.data.db;

import java.sql.SQLException;
import java.util.List;

public interface ORM {
  public abstract Object getObject(String id,
      Object parameterObject) throws SQLException;
  public abstract List getList(String id,
      Object parameterObject) throws SQLException;
  public abstract List getList(String id)
      throws SQLException;
  public abstract void update(String id,
      Object parameterObject) throws SQLException;
  public abstract void update(String id)
      throws SQLException;
}

Now I could create a second implementation, PersistenceORM, that would use the App Engine datastore instead. Naturally, that is easier said then done. The interface above does not give us a lot of information about the context in which it is used. I would have to go through all of its uses in the code and figure out what combinations of an id string and a parameter object existed. The only problem is: the weather is waaaaay to nice for that :-)

Here's what I am trying instead: my first implementation is a stub that breaks whenever it is called. I begin with a simple helper-method checkId:

  private static void checkId(String id, String... supportedIds) {
    for (String test : supportedIds) {
      if (test.equals(id)) {
        return;
      }
    }
    throw new UnsupportedOperationException(id);
  }

At the beginning of each of my interface-methods, I will call checkId with the id that was handed into the method. This will cause the game server to crash, providing me with an exact stack trace of the context in which the ORM interface was used:

java.lang.UnsupportedOperationException: selectGameSummary
 at org.jogre.server.data.db.PersistenceORM.checkId(PersistenceORM.java:39)
 at org.jogre.server.data.db.PersistenceORM.getObject(PersistenceORM.java:100)
 at org.jogre.server.data.db.ServerDataDB.getGameSummary(ServerDataDB.java:221)
 at org.jogre.server.controllers.ServerGameController.addNewUser(ServerGameController.java:436)
 at org.jogre.server.controllers.ServerGameController.gameConnect(ServerGameController.java:209)
 at org.jogre.server.controllers.ServerGameController.parseGameMessage(ServerGameController.java:96)
 at org.jogre.server.ServerConnectionThread.parse(ServerConnectionThread.java:114)
 at org.jogre.server.WebConnectionList.receive(WebConnectionList.java:128)
 at com.appenginefan.toolkit.common.WebConnectionServer.dispatch(WebConnectionServer.java:229)
 at org.jogre.server.JogreServer$1.handle(JogreServer.java:188)
 at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:115)
 at org.eclipse.jetty.server.Server.handle(Server.java:325)
 at org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:539)
 at org.eclipse.jetty.server.HttpConnection$RequestHandler.content(HttpConnection.java:896)
 at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:745)
 at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:218)
 at org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:398)
 at org.eclipse.jetty.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:423)
 at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:404)
 at java.lang.Thread.run(Thread.java:619)

I can look at the code and see how the incoming parameter objects look like. I can also set a breakpoint in my checkId method and inspect all the objects that live on the call stack. By following this approach, I can gradually add methods that are needed simply by running the server and connecting with the standard game client. Here is a sample implementation that supports a few methods so far:

package org.jogre.server.data.db;

import java.sql.SQLException;
import java.util.List;

import org.apache.log4j.Logger;
import org.jogre.server.data.ProtoSchema;
import org.jogre.server.data.ProtoSchema.GameInfo;
import org.jogre.server.data.ProtoSchema.GameSummary;
import org.jogre.server.data.ProtoSchema.Snapshot;
import org.jogre.server.data.ProtoSchema.User;

import com.appenginefan.toolkit.persistence.MapBasedPersistence;
import com.appenginefan.toolkit.persistence.Persistence;
import com.google.common.base.Function;
import com.google.common.base.Functions;
import com.google.common.base.Preconditions;

import static org.jogre.server.data.db.IDatabase.*;

/**
 * ORM implementation that maps to Persistence objects
 * 
 * @author Jens Scheffler
 *
 */
public class PersistenceORM implements ORM {
  
  private static final Logger LOG = Logger.getLogger(PersistenceORM.class.getName());
  
  private static void checkId(String id, String... supportedIds) {
    for (String test : supportedIds) {
      if (test.equals(id)) {
        return;
      }
    }
    throw new UnsupportedOperationException(id);
  }
  
  private Persistence<ProtoSchema.GameInfo> infos;
  private Persistence<ProtoSchema.GameSummary> summaries;
  private Persistence<ProtoSchema.Snapshot> snapshots;
  private Persistence<ProtoSchema.User> users;

  public PersistenceORM(Persistence<GameInfo> infos,
      Persistence<GameSummary> summaries,
      Persistence<Snapshot> snapshots,
      Persistence<User> users) {
    Preconditions.checkNotNull(infos);
    Preconditions.checkNotNull(summaries);
    Preconditions.checkNotNull(snapshots);
    Preconditions.checkNotNull(users);
    this.infos = infos;
    this.summaries = summaries;
    this.snapshots = snapshots;
    this.users = users;
  }
  
  @Override
  public List getList(String id, Object parameterObject)
      throws SQLException {
    checkId(id);
    // TODO Auto-generated method stub
    return null;
  }

  @Override
  public List getList(String id) throws SQLException {
    checkId(id);
    // TODO Auto-generated method stub
    return null;
  }

  @Override
  public Object getObject(String id, Object parameterObject)
      throws SQLException {
    checkId(id, ST_SELECT_USER);
    if (ST_SELECT_USER.equals(id)) {
      User result = users.get(
      ((org.jogre.server.data.User) parameterObject).getUsername());
      if (result == null) {
        return null;
      }
      org.jogre.server.data.User asPojo = new org.jogre.server.data.User();
      asPojo.setEmail(result.getEmail());
      asPojo.setPassword(result.getPassword());
      asPojo.setReceiveNewsletter(result.getReceiveNewsletter());
      asPojo.setSecurityAnswer(result.getSecurityAnswer());
      asPojo.setSecurityQuestion(result.getSecurityQuestion());
      asPojo.setUsername(result.getUsername());
      asPojo.setYearOfBirth(result.getYearOfBirth());
      return asPojo;
    }
    return null;
  }

  @Override
  public void update(String id, Object parameterObject)
      throws SQLException {
    checkId(id, ST_ADD_SNAP_SHOT);
    if (ST_ADD_SNAP_SHOT.equals(id)) {  // update the snapshot for a given key
      final org.jogre.server.data.SnapShot snapshot = 
          (org.jogre.server.data.SnapShot) parameterObject;
      snapshots.mutate(
          snapshot.getGameKey(), 
          new Function<ProtoSchema.Snapshot, ProtoSchema.Snapshot>(){
            @Override
            public Snapshot apply(Snapshot original) {
              return ProtoSchema.Snapshot.newBuilder()
                  .setGameKey(snapshot.getGameKey())
                  .setNumOfTables(snapshot.getNumOfTables())
                  .setNumOfUsers(snapshot.getNumOfUsers())
                  .build();
            }
          });
      
    }
    // TODO Auto-generated method stub
  }

  
  @Override
  public void update(String id) throws SQLException {
    checkId(id, ST_DELETE_ALL_SNAP_SHOT);
    LOG.warn("Attempt submitted to clear the entire store," + 
        " which is not supported!");
  }

}

As I keep adding more methods, I will get a better understanding of how persistence is used in the JOGRE game server. This will also tell me of potential issues I might need to adress when completing the port onto the App Engine data store. After all, I would like my finished server to perform well in the cloud, with potentially tens of thousands of gamers playing in parallel. The road is long, but I intend to make progress crash by crash by crash...

Containing the environment

Sometimes we have to build classes that are hard to test because they contain elements of randomness, concurrency, or they rely on systems outside of our control. For example in my post two weeks ago, I talked about how socket-based client/server protocols can be wrapped in http-based communication. One of the core classes of the example implementation is WebConnectionClient, a class that keeps polling the server for new messages and queues and sends messages from the client to the server.

Here is roughly how it works (mostly for background; if that's not interesting, feel free to skip the following section):

• A WebConnectionClient is instantiated. The client's open method is called, which executes the client's run method in a background thread.


• All client/server communication happens through the exchange of JSON objects (for the concrete encoding, check out the PayloadBuilder utility class). The JSON object contains an opaque key (server-provided; the server uses it to indentify the client and to find out what of its outgoing messages have been received) and a list of outgoing messages. The server responds with messages in the same format, which are passed on to a receiver class.


• New messages to the server come in through the send method Send will not automatically trigger communication, but pass the message to an internal list of outgoing data. This data will be added to the payload the next time the cyclic run method communicates with the server.

Now that we know how the class is expected to behave, we'd like to verify the expected behavior. This however provides a couple of interesting challenges:

• The class uses http communication (to be precise, the Apache http client). Does that mean we need to run a web server to test the class against?


• The class uses a combination of System.currentTimeMills() and Thread.sleep() to maintain a constant frequency of the http calls. How can we make sure that it sleeps for the right time?


• A method having to sleep in-between makes method execution of tests take too much time. Can we somehow speed up execution?


• How do we test execution of code that runs in its own thread?

The solution I chose was to extract every aspect that is not deterministic (time), concurrent (threading) or relies on outside systems (http calls) into an independent class called Environment:

public static interface Environment {

  /**
   * Performs an http request to the server
   * @param data the data to be transmitted
   * @return the response payload, or null if the connection failed.
   */
  public String fetch(String data);
 
  /**
   * Controls the execution of the client's run-method in an independent
   * thread. Similar to the Executor interface, just not for generic runnables,
   * and it would also work in Java 1.3
   */
  public void execute(WebConnectionClient client);
 
  /**
   * Holds the current thread for a certain amount of milliseconds
   */
  public void sleep(long millis) throws InterruptedException;

  /**
   * Gets the current time in milliseconds
   */
  public long currentTimeMillis();
   
}

While the real-life implementation would use system time, http client, and threading, I can easily build a mock implementation for unit tests or simply use EasyMock to automatically create such a mock on the fly.

Let's take a look at such a test setup. First, I create mocks for the two interfaces that my class under test depends on (the environment and the receiver for incoming messages):

private final WebConnectionClient.Environment environment =
    EasyMock.createStrictMock(
      WebConnectionClient.Environment.class);
private final WebConnectionClient.Receiver receiver =
    EasyMock.createStrictMock(
      WebConnectionClient.Receiver.class);

Using these mocks, I can now build a WebConnectionClient and also gain access to its internal payload buffer (the method getQueue has package visibility, so that only our test code can access it):

  private final WebConnectionClient client =
    new WebConnectionClient(environment, SILENCE, MAX);
  private final Queue<String> queue = client.getQueue();

Now let's take a look at a concrete unit test. Say our client tries to connect to the server, but the http connect fails the first time. The client will have no messages to process, so it will go into sleep mode for a certain time. Note how (since we have control over both the system time and the sleep method) we can predict exactly how long that sleep would be. Also, note that while we verify that sleep is called, we do not actually have to suspend the thread. We simply verify that the sleep-call took place, then simulate an end-of-execution situation by throwing an InterruptedException:

  public void testSimpleLoop() throws Exception {
   
    // Record a very simple sequence for the environment:
    // The first http-send will fail, which will cause the
    // thread to sleep. We will jump out of the loop by
    // throwing an InterruptedException
    EasyMock.expect(environment.currentTimeMillis()).andReturn(0L);
    EasyMock.expect(environment.fetch("{\"payload\":[]}")).andReturn(null);
    EasyMock.expect(environment.currentTimeMillis()).andReturn((long) (SILENCE - 1));
    environment.sleep(1L);
    EasyMock.expectLastCall().andThrow(new InterruptedException("end of test"));
   
    // Now, let's try this out
    EasyMock.replay(environment, receiver);
    try {
      client.run();
      fail("Expected an InterruptedException");
    } catch (InterruptedException expected) {
      // fall through
    }
    EasyMock.verify(environment, receiver);
  }

Suddenly, testing complex client-server interaction with threading has become very, very simple :-). Let's look at a more complex example. For readability, I have replaced the JSON payload strings with either constants or a simple generation method p(String meta, String... messages). For details, check out the complete source file.

public void testSendUntilQueueIsEmpty() throws Exception {
  
    // Push a few (MAX + 1) messages onto the bus
    for (int i = 0; i <= MAX; i++) {
      client.send("" + (i + 1));
    
    }
  
    // First, a connection is established
    EasyMock.expect(environment.currentTimeMillis()).andReturn(0L);
    EasyMock.expect(environment.fetch(EMPTY)).andReturn(META_A);
    EasyMock.expect(environment.currentTimeMillis()).andReturn((long) (SILENCE + 1));
  
    // Next, the first three messages get delivered
    EasyMock.expect(environment.currentTimeMillis()).andReturn(0L);
    EasyMock.expect(environment.fetch(p("a", "1", "2", "3"))).andReturn(META_B);
    EasyMock.expect(environment.currentTimeMillis()).andReturn((long) (SILENCE + 1));
  
    // Then, the fourth message gets delivered
    EasyMock.expect(environment.currentTimeMillis()).andReturn(0L);
    EasyMock.expect(environment.fetch(p("b", "4"))).andReturn(META_C);
    EasyMock.expect(environment.currentTimeMillis()).andReturn((long) (SILENCE + 1));
  
    // Now, the queue should be empty.
    // An empty payload gets delivered, just to poll for new messages
    // from the server.
    EasyMock.expect(environment.currentTimeMillis()).andReturn(0L);
    EasyMock.expect(environment.fetch(META_C)).andReturn(META_D);
    EasyMock.expect(environment.currentTimeMillis()).andReturn((long) (SILENCE - 1));
    environment.sleep(1L);


    // Let's try it out
    EasyMock.expectLastCall().andThrow(new InterruptedException("end of test"));
    EasyMock.replay(environment, receiver);
    try {
      client.run();
      fail("Expected an InterruptedException");
    } catch (InterruptedException expected) {
      // fall through
    }
    EasyMock.verify(environment, receiver);
  }

This test recorded a relatively complex sequence of client/server interactions. It also controls whether and for how long the client sleeps by setting the elapsed time between two calls either to SILENCE + 1 (one millisecond longer than the interval, so no sleep) or SILENCE - 1 (one millisecond short, so we have to sleep).

This approach also works for many other things that are outside the control of a class under test, such as random numbers or data from the file system (btw: check out Google App Engine Virtual File System project!). Any good use case I forgot to mention? Put it into a comment on this post :-)

My near deathmarch experience

A couple of years ago, I fell asleep standing up in a server room. It was around my 30th birthday, on a January day in cold-as-hell Ohio. We had externally committed to a delivery date for on-site installation of our new system, and we were fighting hard to make the target. During the day, we were on site: migrating data, installing the client software and more computers, and training the staff on using the new system. At night, we were locked up in our hotel suite, trying to fix bugs we had run into during the day. No-one got more than three hours of sleep a day, including our manager (who was not programming but also did training, kept the customer happy, or simply closed certain feature gaps himself by doing the most amazing things with Crystal reports and data snapshots in an Access database).

On the day the system went live and was exposed to real load, we ran into things we had not seen before. One of them was what can only be called random server death: every now and then, without any warning in advance, a server would decide to completely freeze. All the king's horses, and all the king's men, couldn't convince the machine to unfreeze again. Only hitting the power button and restarting the machine could.

As we could not simply abort our launch, we took turns to guard those devilish devices of doom. Two of us would stay with the customer, while the third would sneak into the server room and wait for Dell to freeze over (pun intended -- although it really was not the hardware's fault!). And so it was my turn; I found myself surrounded my the soothing humming of power devices and the playful chirp of disks being accessed. It seemed so quiet and peaceful. All I remember is standing in front of a rack an leaning my head against the cool steel of the mount, just for a brief moment...

...and suddenly, I heard someone yelling: "the servers are down! the servers are down!" Had I been in worse physical shape, the shock would have given me a heart attack. Torn back into reality, I spun around towards the origin of the voice -- and stared into the grinning face of my boss, who had found me standing asleep in the server room and couldn't resist giving me the scare of my life. For what it's worth: the servers were fine.

Fast forward a couple of years, somewhere around October. I had just been through another launch; one that was possibly amongst the least stressful in my life. A bunch of new services went live, and one of them was my responsibility. Unlike in previous situations, I did not have any colleagues to "take shifts in the server room." If something broke, it was up to me to fix it. If I wanted a new feature, it was up to me to code it. If there was a bug report from a user, it was up to me to investigate it. Granted, the application was not as complex as what we tried to launch that cold winter in Ohio, but it was going to scale to many users, be up all the time, and not break beyond my control (or even worse -- randomly).

Lucky me, the new service was written in Google App Engine (well, what did you expect? This blog happens to be called "App Engine Fan" ;-) ). So, what did that mean for me?

• In a previous life, I was on a pager rotation (actually, a dedicated support cell phone) to react to hard- and software issues that any of our customers might experience. For my new service, automated systems are monitoring the servers for me. As I am sharing them with thousands of other applications, any potential issue affects so many other people that it is quickly discovered and taken care of. Heck, I can even always peek at my application's status console from wherever I can get to a web browser. Or if it is a more general hearbeats I want, I can simply look at the overall system status.
  
• In a previous life, I had to contact all my customers to perform critical updates. Anyone remember the daylight savings time change in the US in 2007? Well, we still had a lot of JDK 1.3 based systems out there, and upgrading them was a major effort (thus pain and cost). Now that my good little App Engine service runs in the Cloud, I can trust Google to take care of that stuff for me :-)
• In a previous life, when stuff went wrong, I had to comb through operating system Event Logs, hoping to find the culprit hidden in some obscure error message that I could only decipher through a ton of Web searches. Nowadays, in the rare occasion that bad stuff happens, I can rely on a team of system experts to do the digging for me and supply me with detailed reports on what went wrong, why it went wrong and what will be done to prevent it in the future.

I am not going to draw any conclusions here on whether coding for App Engine is the best thing that ever happened to me. I actually believe that the winter in Ohio was well spent. It formed a strong bond within the team (yes, even with that guy who almost scared me to death), and we took many lessons away that eventually made our software better, stronger, and more useful to our clients. One thing I would like to mention though: I am glad that I won't be falling asleep in a server room anytime soon...

Faux Sockets for rich Java clients

It's been a while, so I figured I should be posting something at some point ;-) As previously mentioned, my current pet project is porting an existing Java application (the JOGRE Game server) onto the App Engine platform. As it turns out, not only is doing such a thing tricky, blogging about it isn't much easier either. This post adresses one aspect of such a conversion: socket based communication, but on a slightly simplified example. Instead of the full game server, let's try to port this Multi-User Chat application that I found on the interwebs.

The application behaves similarly to the examples from The Java Tutorial (see http://java.sun.com/docs/books/tutorial/networking/sockets/clientServer.html). A server-side ServerSocket listens for incoming connections. A client connects using a Socket and uses the readLine method of a BufferedReader to read data from the socket. It uses the println method of a PrintWriter to push data back to the server.

Converting these kind of applications can be tricky: the server here is stateful, while App Engine is not. Also, sockets assume that a client-server connection is always online. Clients can do blocking reads, which will simply set them into a "waiting" state until new data from the server arrived. These kind of things can be a bit tough to reproduce in a webservice-based environment that expects a client/server interaction to be done within a couple of seconds or less. How can we do it? Let's take a look.

(PS: Since the source code of the chat application does not really tell me what license it is under, I am not going to reprint it here in many details. Instead, I am going to show how to port the existing client, and then simply reimplement the server side from scratch.)

The client

As mentioned before, the two methods mostly used in this example (and many others, for what it's worth), are println and readLine. As long as we successfully substitute these two methods, we should be fine.

In order to not having to reinvent the wheel, I built a little tool class called ClientSocketSubstitute. This class provides the very same methods mentioned before. It caches outgoing communication in a queue and sends it out in batches (as JSON arrays) to the server. Likewise, it receives batches from the server in those same polling requests. The first thing I do is to replace our incoming and outgoing streams with that substitute:

//    BufferedReader in;
//    PrintWriter out;
    ClientSocketSubstitute in;
    ClientSocketSubstitute out;

Since my helper class provides println and readLine implementations, my code mostly compiles. The only area that shows problems is where the Socket would get initialized. I therefore replace the initialization code accordingly:

        // Make connection and initialize streams
//        String serverAddress = getServerAddress();
//        Socket socket = new Socket(serverAddress, 9001);
//        in = new BufferedReader(new InputStreamReader(
//            socket.getInputStream()));
//        out = new PrintWriter(socket.getOutputStream(), true);
        in = new ClientSocketSubstitute(
            new URL(getServerAddress()),   // Connection URL
            1000,                     // ping frequency
            5                         // max. messages per ping
        );
        out = in;

That's it -- let's continue with the server.

The server

The client-server protocol of this example is fortunately quite simple:

• Once the socket communication is established, the server sends the text SUBMITNAME to the client.


• The client responds with a line of text that contains the nickname of the participant in the chat. The server responds with NAMEACCEPTED.


• Any text sent from the client afterwards is a chat message. Anything sent from the server afterwards is a chat message and starts with MESSAGE.

Like with the client, I built a server-side socket replacement that makes it easy to implement such a protocol. Outgoing connections are replaced with a ServerEndpoint class that has a send message. Outgoing messages are buffered in the data store and sent out the next time the polling client connects. In addition, an endpoint has a bag of properties that the server can use to remember certain connection-specific attributes. Incoming messages arrive through a Listener interface that our web service can implement. The overall communication is controlled by a tool class called WebConnectionServer.

Let's look at the actual implementation. We base our implementation on the servlet API and initialize an internal WebConnectionServer instance:

package com.appenginefan.sample.chat;

import java.io.IOException;

import javax.servlet.ServletException;
import javax.servlet.http.*;

import com.appenginefan.toolkit.common.ServerEndpoint;
import com.appenginefan.toolkit.common.WebConnectionServer;
import com.appenginefan.toolkit.persistence.DatastorePersistence;
import com.appenginefan.toolkit.persistence.Persistence;

@SuppressWarnings("serial")
public class ChatserverServlet
    extends HttpServlet 
    implements WebConnectionServer.Receiver {
  
  private Persistence<byte[]> data;
  private WebConnectionServer server;
  
  /**
   * Set up the local persistence and the server 
   * implementation.
   */
  @Override
  public void init() throws ServletException {
    super.init();
    
    // Persist communication data in the store 
    // under a particular socket name
    data = new DatastorePersistence("Socket");
    
    // Build a connection server around that server
    server = WebConnectionServer.fromPeristence(data);
  }

Note that the underlying storage uses the Persistence interface that I introduced a little while ago. The concrete implementation currently uses the data store, but there is nothing preventing me from switching to something else (like memcache or a combination of those two) if I needed better performance at some point. The WebConnectionServer uses a protocol buffer based schema internally to hold on to the buffered messages.

As far as the servlet itself is concerned; its function is mostly to refer all the parsing logic to the WebConnectionServer:

  /**
   * Plugs an incoming request into the server.
   */
  public void doPost(HttpServletRequest req,
      HttpServletResponse resp) throws IOException {
    if (!server.dispatch(this, req, resp)) {
      resp.sendError(404);
    }
  }

The more interesting part is our listener implementation. Let's start with a new connection happening. We have not received any data yet; the server is expected to send a SUBMITNAME. Notice how the listener uses a property in the faux socket to determine whether or not to send out that information:

  @Override
  public void onEmptyPayload(WebConnectionServer server,
      ServerEndpoint socket, HttpServletRequest req) {
    
    // This is the first communication, so we ask for a name once
    if (socket.getProperty("namerequested", null) == null) {
      socket.setProperty("namerequested", "true");
      socket.send("SUBMITNAME");
    }
  }

Now that the SUBMITNAME is out, we can sit back and wait for the messages to come in. The first message is going to be the user's name (which we store in another property). Anything else is incoming data that should be broadcasted to every participant we know.

  @Override
  public void receive(
      WebConnectionServer theServer,
      ServerEndpoint socket, 
      String theMessage,
      HttpServletRequest request) {
    
    final String name = socket.getProperty("name", null);
    
    // Are we looking for a name?
    if (name == null) {
      socket.setProperty("name", theMessage.trim());
      socket.send("NAMEACCEPTED");
    }
    
    // Otherwise, this must be an incoming message. 
    // Forward it to everybody 
    else {
      for (String handle : 
           data.keyScan("", "" + Character.MAX_VALUE, 100)) {
        theServer.fromHandle(request, handle).send(
            "MESSAGE " + name + ": " + theMessage);
      }
    }
  }

For the broadcast, we use the keyScan method of out Persistence interface to get to a list of all sockets known to our server. It should be noted that a broadcast like this is highly inefficient in production code and should be avoided. These kind of socket replacements are more intended for a small amount of recipients (in case of the JOgre game server, it is assumed that board games will usually have less than a dozen players per game).

(PS: You can find the tool classes used in this post under Apache license at this open source project)