Just spent an hour kicking around a Rackspace cloud server. While I have the technical abilities to manage an EC2 instance, the Rackspace setup/config was nice. Very simple and straightforward.
It didn’t take me long however to discover the lack of persistent storage. This of course led me to scour the internet thinking “surely there has to be a way.” I landed upon this post from Brad Something Something (what is your name Brad?) which provides and excellent comparison of Rackspace vs EC2 – I agree with everything he says.
I like Rackspace Cloud, it’s easy to admin, great price/performance, but until they have something EBS/NAS like I’m going to stick with EC2 and Linode.
With Google’s announcement that they are now including live updates from Twitter, Facebook, and MySpace into their search results I expect the term Real Time Web is going to become more familiar to the non TechCrunch public.
While the term “Real Time” has taken off over the past 6 months most realize that our existing communications infrastructure already operates at near real time. You send an email, it arrives in seconds. You place a call, someone picks up. Blog posts, satellite television, GPS, IM, etc etc etc.
I’d say the fundamental shift in behavior we are seeing on the web today is related to “Always On”. It’s ubiquitous network connectivity that makes us feel the already real time nature of the web even more.
So what’s up with Real Time Search and the Real Time Web? Basically it’s about content being indexed and presented in search results as fast as it’s being produced. This is a certainly a step in the right direction towards the larger goal of instant and ubiquitous human knowledge – “when I know, you know”. The problem is there’s just too much noise when you turn on the stream and the only filter in place are keywords.
The technology is important though; data must be collected and indexed before it can be filtered/ranked. We’re getting there.
What gets me excited about the Real Time Web are the ways it can be used to augment existing methods for consumption of news and entertainment. Imagine the ways that the PubSub model combined with Real Time Search will allow people to “tune-in” to personalized data feeds during sporting events, tv shows, breaking news.
For example, when I am watching the Dallas Cowboys on TV I don’t want to type “Dallas Cowboys” into a search engine and be flooded by results. I want to tune-in to a list of people that I’ve selected (or have been recommended). These people may be professionals, they might be my neighbor. It’s these people that will be providing insight, analysis, and commentary. Troy Aikmen and Joe Buck? Nope. I want comedy. I want bias. I want camaraderie. Then when the game is over I want to tune out, I want it all to go away.
To me the Real Time Web is not about speed, it’s about moving past the period where Social Networks are persistent. The Real Time Web will introduce Social Networks that are dynamic. Networks that emerge and disappear in short spans of time. These networks will be asynchronous – increasingly the Real Time Web will look more like the Real World.
Normally if I have an issue which is answered in the first 2-3 results of a Google search I won’t create a post. On the other hand when I spend 2-3 hours trying to solve something which should be simple I like to take the opportunity to describe the issue & resolution in hopes that someone will find it quickly in the future.
So the task here was to find a way to specify the IP address, aka socket, aka network interface when making an http request using Python’s urllib2. Why would you want to do this you ask? Well for many web API’s the request rate is limited by whitelisting the IP address – such is the case with Twitter. In the event that you want to be able to use the same machine (with multiple network interfaces) to run jobs in parallel you need to be able to specify where the requests should be routed.
The problem is Python’s urllib2 is based on the httplib library which doesn’t let you specify which address to bind to. This person tried to get around the problem in 2005 without any luck, another guy created a patch for httplib in 2008 which hasn’t been accepted, and finally someone else created a subclass for httplib which unfortunately I couldn’t get hooked up to the urllib2 class.
The best solution I found was this “monkey patch” from Alex Martelli over on Stack Overflow. In his example he attacks the problem using the socket library instead of the httplib. By his own admission stuff like this is not ideal, but the solution is actually very simple and elegant. I like it.
I wrapped the snippet up into a function which can be called in a Python script anytime before you invoke a urllib2 request.
I just spent more time than I should have troubleshooting why the upgrade of MySQL from 5.0 to 5.1 on a Debian box resulted in a MySQL instance that wouldn’t start. Not a lot out there on this so hopefully this will save someone a bit of time in the future.
When upgrading from 5.0 to 5.1 using apt everything will install normally. Then when the MySQL service tries to restart you’ll see and init.d error and an error that looks something like this:
Errors were encountered while processing:mysql-server-5.1mysql-server
Not a lot to go on here but as it turnsthere is a deprecated entry in the my.cnf file called skip-bdb. Comment this line out and you should be good to go.
Over on Coding Horror Jeff Atwood wrote a post which claimed that “All Programming is Web Programming“. Jeff made some good points about how the web provides programmers with the ability to reach an audience of a previously unimaginable size. Definitely agree. Also mentioned was that for better or for worse JavaScript is becoming the most important language in the world of software development. I agree with this as well, though I would add that the significance of JavaScript is in user facing applications only at this point.
It was unfortunate that the post was written in such a polarizing manner and that the comment thread quickly eroded into a shouting match because there is another important point to be made here.
Something else I want to put out there to developers is that the evolution of programming should be focused less on desktop vs web vs embedded or choice of language/platform, and more on how lowering the barriers to entry for new programmers is a positive and not a negative.
To someone writing device drivers or kernel patches the idea of writing a JavaScript function to manipulate the DOM may seem “uninteresting”, but the fact is that more and more people are getting started with programming this way. All they need is a text editor and a web browser and they are on their way. This is a very good thing.
Programming is about automation and automation is about improving efficiency. The more people we can somehow involve in this process the better because in the end the web provides not only the largest number of potential users, but also the largest number of potential programmers. The exciting thing is we are just getting started.
Martin Willcox from Teradata wrote a couple of blogposts outlining the reasons why he feels the phrase “unstructured data” is marketing jargon and that “nontraditional data” is more appropriate.
Let me start by saying that the examples Martin uses in the first post are technically accurate if we were all disk manufacturers. Whether bitmap (audio, video) or text (email, html), it’s true all of these file types use a structured format when being processed by a computer. That being said, we are not all disk manufacturers.
As a data architect I’ve always felt the true spirit of the phrase “unstructured data” corresponds to the modeling and analysis of the data. If you have a collection of objects in an email, an image, or web page… then these things are unstructured. They tell you nothing without the context of the structured model.
If this were simply a preference in terminology then I wouldn’t think too much of it, but when a relational database vendor claims that “nontraditional” (unstructured) data is easily converted to “traditional” data by running fact/entity extraction routines and loading a table it makes me stop and question the true intent of the original message. It’s not as simple as pushing a button, and an RDBMS is most often not your best option. This isn’t something which should be glossed over.
The problem is that when using a relational database schema the relationships, attributes, and quantities must be defined before running any extraction routines. That’s ok when running against a fixed set of data looking for a known set of attributes/measures – but when you are mining millions of images or billions of web pages all of the edges don’t start to show up until you actually start to extract and analyze the data. In this situation a relational database actually makes it harder to consume unstructured data due to the high cost associated with schema changes
To me the term unstructured makes sense… it’s simply the inverse of structured. Data without a model if you will. And remember, the larger and more diverse the data set, the less you will know about it’s characteristcs ahead of time.
Yesterday Paul Miller wrote a blog post called “Does Linked Data need RDF?” In the comments I said something in support of Paul’s question which I want to expand upon here. My comment was not directly related to the topic (linked data), but rather the format in which it was asked. Let me trim it down to this:
“Could we do ${solution} without ${component}?”
What I am thinking is this…. as system architects or software engineers this is a critical question to ask not only during design, but also as our solutions age over time. When asked this way it challenges us to think about dependencies from a different perspective. Instead of “are things working? If so, add new feature“, we should equally be looking for ways to remove, replace, or consolidate components for the benefits of efficiency, simplicity and performance. Sounds simple right, then why is it so hard to do?
I don’t think we go through this exercise often enough because politically it can be a very difficult. The sheer inertia of a large project can prevent people from even thinking about the question. The thing is that writing code, choosing a framework, establishing an API…. these are all sunk costs. We often attach emotional value to the time and effort that went into doing the work, and have a hard time imagining what things would look like without them.
It’s certainly a challenge, but to build better systems we need to be able to let things go, to scrap code, and to replace components without emotions or personal bias.
Oh and for the record, I do think RDF and Linked Data are the right combination to build the Web of Data. I just hope that we can keep asking questions, challenging assumptions, and continue to have constructive debates about the future of the Web like the one which took place yesterday on Paul’s website. Great stuff.
I spent some time this weekend diving deeper into Amazon’s EC2 and all of the associated services. I’ve read about EC2, discussed it with colleagues, I pretty much thought I knew what it was all about….. virtual hosting right? Yeah, I was wrong. It was going through the process off setting up an instance and configuring all the network and storage services completely that changed my perspective. EC2 is really, really, cool.
What is really rocking my world is the whole concept of throw-away servers. The idea that a discrete process can spin up a new server that gets built at run time, does some work, then disappears is amazing. I see this as turning the whole concept of linear scale on it’s head. You don’t scale an app, you scale individual threads. Powerful stuff, especially when dealing with data mining and event processing.
Had a great conversation with @cks earlier about the dichotomy between ORM and RDF/OWL when modeling enterprise data. His position was that with a pure ORM model you are more likely to have consistent data throughout your applications because the rules & constraints have laid out before the user ever touches the keyboard. Where he felt RDF/OWL was at odds with the ORM model is that by giving people the ability to create relationships that trigger additional inferences you must trust them to understand the implications of their actions.
A simple example….
:trees :madeOf :wood
:trees :need :water
Now someone comes along and creates this:
:table :madeOf :wood
At this point it’s incumbent upon all dependent applications to understand that “:table” does not in fact need water. Again, the key is that every application must be aware of and apply the same rules or data integrity suffers. It’s not that tradtional ORM and RDF/OWL can’t coexist, in some companies it may be an integrated process. Where @cks was concerned is that the inferences inherent to RDF/OWL introduces issues with consistency and integration because it’s so easy for new rules to simply pop up.
I agree with everything up to this point, but where I would argue we need to be headed with enterprise apps is a hybrid model that blends the consistency & predictability of ORM, with the freedom of RDF/OWL.
First let’s quickly take a moment to talk about freedom. When I hear a programmer use the phrase “never trust the user” I scratch my head. Sure you should sanitize application input for the sake of security but let’s be realistic about it, business users do not intentionally inject crap into the system. They use software to get things done. Humans will make mistakes, but so do the software applications that were written by… well, humans.
The user is the most important component of software development. If that sounds obvious, then why don’t we trust them more? My hope is that developers begin putting more trust in the user with a focus on creating software that learns from the users instead of limiting them.
So back to the hybrid model. I picture RDF/OWL as the essential meta layer above the ORM. By abstracting it with interfaces that are usable to non-techies it becomes an engine for collecting knowledge about the relationships and attributes of the business across all dimensions. We shouldn’t be concerned with modeling absolute and irrefutable truths, because tomorrow there will be an exception. That’s the problem with strict models in the enterprise, there will always be exceptions. Next, the ORM layer follows on as an application specific module where you can extract pieces of the meta layer to digest, analyze, and make use of the data at a domain level.
It’s about putting a higher priority on the collection of information than on enforcing rules on the information. The principle reminds me of the great quote from Jon Postel.
“be conservative in what you send, be liberal in what you receive”
Postel is of course referring to the Transmission Control Protocol, a language that computers use to speak to each other over the internet. To me however these words have a more universal meaning in the world of software development which I’d categorize like this:
Something that’s currently missing from CouchDB is a way to import/export documents. This feature may be added to CouchDB one day, but say you need a way to get your data out of CouchDB… like right now. Well here’s how you can do it.
Before getting started one quick side note about dealing with CouchDB data files. When you create a new database there is a corresponding {db}.couch file created that is your actual “database”. It’s usually in /var/lib/couchdb, but if not check DbRootDir in your /etc/couchdb/couch.ini for the location (update: for 0.9.0 it’s now database_dir in /etc/couchdb/default.ini)
Under normal circumstances you have the ability to take hot backups of these files at anytime using rsync, cp, etc… it’s simply a file. The thing that got me stuck was when CouchDB went from 0.8.0 to 0.9.0 and the internal file format changed. The result was that the data needed to be moved programmatically across databases using raw JSON.
If you search the CouchDB mailing lists for how to get your data migrated you’ll likely come across references to the couchdb-python utilities. Dig more and you’ll see references to the tools/dump.py and tools/load.py scripts. That’s about where the trail ended for me, but after some hacking around I’ve successfully moved data from 0.8.0 to 0.9.0. As an added bonus I was able to get my hands dirty with the couchdb-python library which has been fantastic so far.
One more side note, this time about couchdb-python. If you are new to CouchDB I would still recommend starting with Futon, Views, and the REST API before you move to a client library (Python or other). It will help you conceptualize how CouchDB is way more than a massive hash table or fancy object store.
So to the task at hand…. Assuming you have Python 2.4 or later you’ll need to install 3 things.
httplib2 – This is a Python HTTP lib, I was able to install it via apt-get on Debian. There are packages available for other distros.
I was able to install the egg files using the Python easy_installer.
The next step is to grab tools/dump.py and tools/load.py from CouchDB egg file. To do this you need to unzip the CouchDB .egg that’s in site-packages and extract the files to a directory of your choice. This seems like a strange method, but it works. Someone let me know if I’m missing an easier way.
To begin the database dump run dump.py and pass the full URL to the database you are exporting. Make sure to redirect output in order to capture the JSON.
Once your export completes copy the .json file and the load.py to the same directory and run the following command to import the file to your target database.
Make sure you create the target database before you run the script or it will fail. You’ll know everything is working if you see a series of statements that looks like this:
