So the task here was to find a way to specify the IP address, aka socket, aka network interface when making an http request using Python’s urllib2. Why would you want to do this you ask? Well for many web API’s the request rate is limited by whitelisting the IP address – such is the case with Twitter. In the event that you want to be able to use the same machine (with multiple network interfaces) to run jobs in parallel you need to be able to specify where the requests should be routed.

The problem is Python’s urllib2 is based on the httplib library which doesn’t let you specify which address to bind to. This person tried to get around the problem in 2005 without any luck, another guy created a patch for httplib in 2008 which hasn’t been accepted, and finally someone else created a subclass for httplib which unfortunately I couldn’t get hooked up to the urllib2 class.

The best solution I found was this “monkey patch” from Alex Martelli over on Stack Overflow.  In his example he attacks the problem using the socket library instead of the httplib. By his own admission stuff like this is not ideal, but the solution is actually very simple and elegant. I like it.

I wrapped the snippet up into a function which can be called in a Python script anytime before you invoke a urllib2 request.

def bind_alt_socket(alt_ip):

true_socket = socket.socket
def bound_socket(*a, **k):
     sock = true_socket(*a, **k)
     sock.bind((alt_ip, 0))
     return sock
socket.socket = bound_socket

Hope this can be of help to someone in the future who’s searching for the same thing I was.

Before getting started one quick side note about dealing with CouchDB data files. When you create a new database there is a corresponding {db}.couch file created that is your actual “database”. It’s usually in /var/lib/couchdb, but if not check DbRootDir in your /etc/couchdb/couch.ini for the location (update: for 0.9.0 it’s now database_dir in /etc/couchdb/default.ini)

Under normal circumstances you have the ability to take hot backups of these files at anytime using rsync, cp, etc… it’s simply a file. The thing that got me stuck was when CouchDB went from 0.8.0 to 0.9.0 and the internal file format changed. The result was that the data needed to be moved programmatically across databases using raw JSON.

If you search the CouchDB mailing lists for how to get your data migrated you’ll likely come across references to the couchdb-python utilities. Dig more and you’ll see references to the tools/dump.py and tools/load.py scripts. That’s about where the trail ended for me, but after some hacking around I’ve successfully moved data from 0.8.0 to 0.9.0. As an added bonus I was able to get my hands dirty with the couchdb-python library which has been fantastic so far.

One more side note, this time about couchdb-python. If you are new to CouchDB I would still recommend starting with Futon, Views, and the REST API before you move to a client library (Python or other). It will help you conceptualize how CouchDB is way more than a massive hash table or fancy object store.

So to the task at hand…. Assuming you have Python 2.4 or later you’ll need to install 3 things.

httplib2 – This is a Python HTTP lib, I was able to install it via apt-get on Debian.  There are packages available for other distros.

simplejson -  Python egg for JSON manipulation.

couchdb-python – Python egg for CouchDB.

I was able to install the egg files using the Python easy_installer.

The next step is to grab tools/dump.py and tools/load.py from CouchDB egg file. To do this you need to unzip the CouchDB .egg that’s in site-packages and extract the files to a directory of your choice. This seems like a strange method, but it works. Someone let me know if I’m missing an easier way.

To begin the database dump run dump.py and pass the full URL to the database you are exporting. Make sure to redirect output in order to capture the JSON.

./dump.py http://source-couchdb:5984/msg_db > msg_db.json

Once your export completes copy the .json file and the load.py to the same directory and run the following command to import the file to your target database.

./load.py –input=msg_db.json http://target-couchdb:5984/msg_db

Make sure you create the target database before you run the script or it will fail. You’ll know everything is working if you see a series of statements that looks like this:

Loading document ‘bda90174c1a41bad2289bfc5829008ce’
Loading document ‘e45d7c2850610a01658234eeddde1fde’
Loading document ‘e856071c791cd677eafbce85bb1509de’

After it completes, you can fire up Futon and you’ll see all your precious data has been loaded into your new instance of CouchDB. Victory!