Filtration
Posted by aogMonday, 16 December 2002 at 18:43 TrackBack Ping URL

I was discussing the blogosphere with a co-worker and how it might impact Big Media in the future. What occured to me is that the set of blogs, with its natural hierarchy (or food chain), serves as a extremely effective filtering mechanism. If some random, miniscule readership blog like this happens on some good, verifiable data, it will be picked up by one of the blogs a little higher on the food chain, because almost all non-bottom feeding blogs read at least some down level ones. In a recursive manner this data will flow up the food chain until it is posted on a heavy traffic blog (like Instantman or The Corner. Then it's everywhere.

Because there is no enforced hierarchy (it's an emergent property of the blogosphere) there is little rigidty to information flows. It's in fact important for even the biggest blogs to read down scale blogs and not just cross read the blogerati because they'll go stale. So the system has a nice negative feedback cycle to prevent calcification.

What's most fascinating to me is that I was an architect on a project that was trying to build something just like this. It was a event monitoring system. One of the dirty little secrets of managing networks is that it's hard to know what's going because you know too much. The number of events generated by devices in a network is absolutely astounding. A million a day is a reasonable planning number for a big site. Nobody can keep up with that. Our system was designed to distribute cybernetic readers through the network, each one reading data from another set of lower level readers and passing a condensed and filtered event stream to a higher level reader. The filtering is key - it's not just compressing data - some of it must be thrown away or the central processing will choke. Of course, the interesting bits were in deciding how the readers worked (I didn't do that - I'm a systems guy so I was tasked with building the framework to move data and get the readers to hook up).

This is the future. We now live in a world where finding data about something isn't the problem, it's finding the right data. And it's the kind of thing the blogosphere does well.