CherryPy Performance Management

Being a Python guy I’m doing most of my web development these days using CherryPy. If you are not familiar it’s a  Python web framework that to me strikes just the right balance of simplicity and power.

Behind the scenes at Key Ring we also have a large Rails footprint. For the past year we’ve been using New Relic to monitor application performance up and down the stack. When it comes to monitoring and troubleshooting a web app I can’t recommend these guys enough. They’re constantly making improvements, adding features, and the product has become indispensable to our team.

So I just finished building some data services in CherryPy and was psyched to hook up the New Relic agent to this app. The docs on the New Relic site are pretty thorough but I wasn’t able to piece together how to hook the agent into the CherryPy app. Also, could’t find anything on the web. Through trial and error I was able to get it working…. it’s actually extremely easy if you just follow these steps.

1. Download the latest Python agent. Unpack.

 wget http://download.newrelic.com/python_agent/release/newrelic-1.0.5.156.tar.gz

2. Run the setup

python setup.py install

3. Generate a New Relic agent config file using your API key.

newrelic-admin generate-config $YOUR_API_KEY newrelic.ini

4. Change your app name in the New Relic config. There are some other logging bells and whistles which are all self-explanatory

5.  Add these 2 lines to your sites startup script. Make sure you do it early in the script before any database calls, etc…

import newrelic.agent

newrelic.agent.initialize('/path/to/newrelic.ini')

And that’s it. In the next minute or so you’ll begin to see your stats streaming into your New Relic dashboard.

Thoughts on the Real Time Web

With Google’s announcement that they are now including live updates from Twitter, Facebook, and MySpace into their search results I expect the term Real Time Web is going to become more familiar to the non TechCrunch public.

While the term “Real Time” has taken off over the past 6 months most realize that our existing communications infrastructure already operates at near real time. You send an email, it arrives in seconds. You place a call, someone picks up. Blog posts, satellite television, GPS, IM, etc etc etc.

I’d say the fundamental shift in behavior we are seeing on the web today is related to “Always On”. It’s ubiquitous network connectivity that makes us feel the already real time nature of the web even more.

So what’s up with Real Time Search and the Real Time Web? Basically it’s about content being indexed and presented in search results as fast as it’s being produced. This is a certainly a step in the right direction towards the larger goal of instant and ubiquitous human knowledge – “when I know, you know”.  The problem is there’s just too much noise when you turn on the stream and the only filter in place are keywords.

The technology is important though; data must be collected and indexed before it can be filtered/ranked. We’re getting there.

What gets me excited about the Real Time Web are the ways it can be used to augment existing methods for consumption of news and entertainment. Imagine the ways that the PubSub model combined with Real Time Search will allow people to “tune-in” to personalized data feeds during sporting events, tv shows, breaking news.

For example, when I am watching the Dallas Cowboys on TV I don’t want to type “Dallas Cowboys” into a search engine and be flooded by results. I want to tune-in to a list of people that I’ve selected (or have been recommended).  These people may be professionals, they might be my neighbor. It’s these people that will be providing insight, analysis, and commentary. Troy Aikmen and Joe Buck? Nope. I want comedy. I want bias. I want camaraderie. Then when the game is over I want to tune out, I want it all to go away.

To me the Real Time Web is not about speed, it’s about moving past the period where Social Networks are persistent. The Real Time Web will introduce Social Networks that are dynamic.  Networks that emerge and disappear in short spans of time. These networks will be asynchronous – increasingly the Real Time Web will look more like the Real World.