Patterns of doom: chain of fetches

Let's look at another commonly seen pattern of doom that can be identified and eliminated using app stats. This specimen is very similar to chain of gets, but it is more commonly found in mashups that need to fetch a lot of data from the web. The situation is something like this:



The application gets from somewhere a list of data it needs to correlate. Each piece of data is somewhere on the web and needs to be fetched, e.g. from an atom feed or by scraping a html page. The developer then will get the data, e.g. like this:

pages = [urlfetch.fetch(url) for url in urls]

and then do the operations necessary on the data to get the information. The problem is very similar to chain of gets: each request is slow, and performing them sequentially takes time.

So, what is the solution to the problem? In the case of gets, we had an api that would let us perform all the datastore operations in one batch. For urlfetch, a similar effect can be achieved using asynchronous requests:

def asynchronous_get(list_of_urls):
  rpcs = []
  for url in list_of_urls:
    rpc = urlfetch.create_rpc()
    urlfetch.make_fetch_call(rpc, url)
    rpcs.append(rpc)
  return [rpc.get_result() for rpc in rpcs]

With this little helper method, we can parallelize the fetches in our previous example by simply making a small adjustment:

pages = asynchronous_get(urls)

The improvement can be significant, as shown in the second screenshot. Note that app stats shows that the time spent in RPCs is about the same as in the original example, but because they all happened at the same time, the actual time spent (Grand total) is significantly less.

As always, you can try out the pattern yourself at my demo site. Please don't overdo it though, since this is running on free quota and run out of it pretty soon otherwise. There is a reason I am calling those patterns of doom after all ;-)

PS: for those wondering why the first get in the screenshot is not batched: that's an independent call I make to find a list of youtube urls I'd like to fetch.