an App Engine fairytale
(Based on a true story, somehow. Names have been altered, so that I'm not embarrassing myself too much ;-)
Once upon a time there was a farmer named Jack. Jack's life on the server farm had always been good: he had a little parcel to grow his crop on, and because his land was quite small and did not take up a lot of space, he did not even need to tithe to the lords of the server farms.
The land where Jack's farm grew was governed by friendly wizards. They had set up some rules on how the crop was to be grown, in order to be able to sustain as many people on the farms as possible. There were countless other farmers, just like Jack, and while the variety of crop was great, they all flourished on the land that was given to them. The wizards had also devised several tools to make life easier for the people. They built a huge storage where farmers could bring their crops and retrieve them whenever needed. They also had a bellboy who could be asked to remind farmers to water their field, or do other maintenance related jobs. However, the most wonderful of their contraptions was the magical taskqueue. The magical task queue was a system operated by an ancient race of invisible gnomes. If someone needed a farm hand, he would write the tasks to be done on pieces of paper and pile them up next to the field. The gnomes would then pick them up and execute them, until the stack was empty.
One day, Jack noticed that the pile of unfinished tasks next to his field started groing and groing. Usually, it was not taller than a couple of sheets, but now it had grown to almost the height of his knee. He was completely mystified -- had the magical gnomes fallen ill? He pulled out the admin console, a powerful device the wizards had bestowed upon him. The admin console enabled the farmers to learn and see things about their crop they would otherwise not know. Through its magical eye, he peeked at the stack. He saw a flurry of activity: hundreds of hands where rushing to take the yellow stickies from the pile, yet at the same time, even more invisible forces kept piling additional jobs onto the heap. Luckily, Jack knew one of the wizards responsible for the gnomes, and he cried out for help. How could it be that a small farmer like himself could generate more work then the gnomes could handle? Something was afoul!
The wizard had heard these kind of complaints before, and he knew that it usually was an error by the farmer. "Jack, have you made sure that you configured your queue correctly? There are a couple of things to look out for, like how many tasks a minute the gnomes are allowed to take from the stack. Please double-check your configuration and let me know what happened." Jack had trust in the wizard's advice, and he looked at his settings. As it turned out, there were a couple of tweaks he could do that made the gnomes pick up more tasks. Surely that had to be it!
After several hours of waiting, the pile had grown even more in size. It was now the hight of a small child. Jack went back to to ask for more help. This was unbearable, he cried out. Obviously the gnomes must be doing something wrong! The wizard was a patient man, and he listened calmly to the ranting of the upset farmer. After a few minutes, when the yelling and screaming had subsided, he turned towards his fellow man. "My dear Jack," he responded, "how do you expect me to see anything in that mess beside your field? You have all kind of tasks mixed together in a single pile. How do you expect me to know what's causing your problems -- the watering of the plants, the sowing of new seeds, or the harvest? Go ahead and split up your tasks into several piles. Then, leave the field alone for a day and tell me what happened." Jack was quite annoyed by the wizard's response, but he did what he'd been told. He took the different kinds of tasks and put them into multiple queues. It was a hard job, but at the end of the day, everything was neatly going into separate piles. Totally exhausted, he fell asleep next to the field.
The next morning, Jack got up early to take a look. Out of his five piles of tasks, four had been reduced to less than a dozen stickies. The fourth one however was as big as a grown man by now. He went back to the wizard to report. The wizard took another look and confirmed Jack's findings. However, he also found something that Jack had overlooked. "Most of the tasks on your queue are very old and have been in and out of the queue again and again. You can see it at the head of the note: the retry-counts are through the roof. Looking at the very admin console you hold in your hands, I also see that these tasks are constantly sending "301 redirects" -- in other words, they tell the gnomes to put the paper right back onto the stack. Can you tell me what's up with that"?
The discovery of the sage hit Jack like a lightning bolt. Hadn't he lately done a change to the way his crop was grown, to make things more efficient? Could it be that he had made a mistake? Now knowing what to look for, he discovered the problem within a few minutes. Fixing it was a bit harder, but after a couple hours of work, the problem was rooted out of a system. It took only a few more hours to watch the man-sized pile of tasks grow back to the size of a small child, then a melon, and finally a pile not higher than the tiniest of mice.
So what is the moral of the story? Don't make a mess of your task queue, check closely your admin-console -- and if something is wrong and you get a friendly wizard's help, don't behave like a Jack@$$ ;-)
