Category Archives: Emergence

At Least They’re Doing Something

What a great essay / presentation by Clay Shirky. A couple quotes that stood out to me:

And what did we do with that free time? Well, mostly we spent it watching TV.

We did that for decades. We watched I Love Lucy. We watched Gilligan’s Island. We watch Malcolm in the Middle. We watch Desperate Housewives. Desperate Housewives essentially functioned as a kind of cognitive heat sink, dissipating thinking that might otherwise have built up and caused society to overheat.

And it’s only now, as we’re waking up from that collective bender, that we’re starting to see the cognitive surplus as an asset rather than as a crisis. We’re seeing things being designed to take advantage of that surplus, to deploy it in ways more engaging than just having a TV in everybody’s basement.

This hit me in a conversation I had about two months ago. As Jen said in the introduction, I’ve finished a book called Here Comes Everybody, which has recently come out, and this recognition came out of a conversation I had about the book. I was being interviewed by a TV producer to see whether I should be on their show, and she asked me, “What are you seeing out there that’s interesting?”

I started telling her about the Wikipedia article on Pluto. You may remember that Pluto got kicked out of the planet club a couple of years ago, so all of a sudden there was all of this activity on Wikipedia. The talk pages light up, people are editing the article like mad, and the whole community is in an ruckus–“How should we characterize this change in Pluto’s status?” And a little bit at a time they move the article–fighting offstage all the while–from, “Pluto is the ninth planet,” to “Pluto is an odd-shaped rock with an odd-shaped orbit at the edge of the solar system.”

So I tell her all this stuff, and I think, “Okay, we’re going to have a conversation about authority or social construction or whatever.” That wasn’t her question. She heard this story and she shook her head and said, “Where do people find the time?” That was her question. And I just kind of snapped. And I said, “No one who works in TV gets to ask that question. You know where the time comes from. It comes from the cognitive surplus you’ve been masking for 50 years.”

So how big is that surplus? So if you take Wikipedia as a kind of unit, all of Wikipedia, the whole project–every page, every edit, every talk page, every line of code, in every language that Wikipedia exists in–that represents something like the cumulation of 100 million hours of human thought. I worked this out with Martin Wattenberg at IBM; it’s a back-of-the-envelope calculation, but it’s the right order of magnitude, about 100 million hours of thought.

And television watching? Two hundred billion hours, in the U.S. alone, every year. Put another way, now that we have a unit, that’s 2,000 Wikipedia projects a year spent watching television. Or put still another way, in the U.S., we spend 100 million hours every weekend, just watching the ads. This is a pretty big surplus. People asking, “Where do they find the time?” when they’re looking at things like Wikipedia don’t understand how tiny that entire project is, as a carve-out of this asset that’s finally being dragged into what Tim calls an architecture of participation.

So that’s the answer to the question, “Where do they find the time?” Or, rather, that’s the numerical answer. But beneath that question was another thought, this one not a question but an observation. In this same conversation with the TV producer I was talking about World of Warcraft guilds, and as I was talking, I could sort of see what she was thinking: “Losers. Grown men sitting in their basement pretending to be elves.” At least they’re doing something.

The Data Life Cycle of a Blog Post

Cool flash infographic in the latest issue of Wired that shows what happens to your blog post after you click the ‘publish’ button (I’ll save you the hassle of actually viewing it: after you click the ‘publish’ button, exciting things like ping servers, data miners, search engines, text scrapers, aggregators, social bookmarking sites, online media, spam blogs and finally readers get involved). Since it’s Wired and not XML Journal, they stopped at the infographic, but man, it should would be cool to see all the ways that data massaged, reformatted, sliced and diced and transmitted, because there’s a lot that happens in that process. Just for the fun of it, I’m gonna walk through the scenarios I know about.

First, you click the publish button. But that might be a publish button on a desktop blogging client like Windows Live Writer or it might be the publish button in Microsoft Word or it might be a real live HTML button that says ‘publish’. So before you even get to the publish part, we’ve got the possibility of the MetaWeblog API (which is XML-RPC, effectively XML over HTTP), Atom Publishing Protocol (again effectively XML over HTTP) or a plain HTTP (or HTTPS!) POST.

OK, so now your blog post has been published on your blog. What next? Probably unbeknownst to you, your blog post has been automatically submitted to one or more ping servers using XML-RPC (XML over HTTP). Because search engines got into the blogging business, you can even ping Google and Yahoo (curiously not Microsoft, why?). If you don’t want to hassle with a bunch of different sites, you can always use, which will ping (as of 1/27/2008) twenty one different ping servers for you.

Oh, I forgot to mention. If you’re using TypePad, Livejournal or Vox, the information about your blog post isn’t sent to these ping servers using XML-RPC, it’s streamed as XML in real-time over HTTP to many of the same parties.

Great, your blog post has now been sent to everyone, you’re good right? Nope. Now comes the onslaught of spiders and bots, awoken by the ping you sent, who will request your feed (RSS / Atom over HTTP) and your blog post (HTML over HTTP) and your first born child again and again and again. And now that your blog post is published and assuming that you’ve published something of value, you’ll see real people stop by and comment on your blog post and maybe bookmark it in a site like or, snipping a quote from your blog post and then publishing that snippet to their own blogs or to their bug tracker and now your blog post has replicated, it lives in small parts all over the web, each part getting published and spidered and syndicated and ripped again and again and again. It’s beautiful isn’t it?

Biological Computing

Jason Kottke and a couple other people blogged Eric Bonabeau’s talk on Biological Computing using Hydra. Biological computing is an amazing concept. Seems like we programmers spend 99% of our time creating ‘palaces of abstraction’; (to steal a phrase from Josh), reducing the world to manageable bits. Biological computing lets all that loose… Build it so that you don’t have control. Add two parts of this and three of that and see what happens. Fascinating.

Emerging Technology: Who Loves Ya, Baby?

Steven Johnson’s latest article is online @ Emerging Technology: Who Loves Ya, Baby?

His thesis is based on Cat’s Cradle, which “… explains how the world is divided into two types of social organizations: the karass and the granfalloon.” I was thinking on the way home tonight how the exposure of massive amounts structured data via xml (web services, rss, xml-rpc…) will make it exponentially easier to create applications like the InFlow software mentioned in the article. is one great example of that. Erik Benson gets information from, pings amazon’s web service and voila, we have a social book circle. While I’m sure the availability of this information worries privacy advocates, you can’t help be get excited about the possibilities that lie ahead of us.

Swarm Intelligence: An Interview with Eric Bonabeau

On Interesting interview w/ Eric Bonabeau, relevant quotes:

And that swarm intelligence offers an alternative way of designing “intelligent” systems in which autonomy, emergence, and distributedness replace control, preprogramming, and centralization.” — The words ‘autonomy, emergence and distributedness’ are somewhat freeing from a software development standpoint. In complex systems, instead of taking x years to development a specification document and then another y years to implement the system and then z more years to debug the system, you spend the time upfront to make the system adaptable to its environment, able to evolve in some sense.

In social insects, errors and randomness are not “bugs”; rather, they contribute very strongly to their success by enabling them to discover and explore in addition to exploiting. Self-organization feeds itself upon errors to provide the colony with flexibility (the colony can adapt to a changing environment) and robustness (even when one or more individuals fail, the group can still perform its tasks).” — Bugs (the software kind) are inevitable, we can’t write perfect code. Doesn’t it make sense to use errors to our advantage then? (one might argue that the same system that takes advantage of errors will have errors, do we have to write software to take advantage of the errors written into the program that takes advantage of errors?)

The article mentions routing and UAVs as applications of swarm intelligence. My domain of knowledge currently only wraps around web applications… how might one use swarm intelligence in web applications? Implicit personalization might be a place to start. The pheromones that ants leave behind for others to follow aren’t that different from the ‘paths’ one leaves behind on a site that others might follow. Knowledge management comes to mind too. Could the aggregate of referrers and queries sent to a site become something greater than the sum of the individual parts? On heavily trafficked sites, you’ll always have to make a choice about what to cache and what to get from persistent storage. Perhaps a case can be made for letting the system compute what the most expensive procedures are (disk I/O, DB transactions, web service calls, etc…) and cache those, instead of caching everything (or nothing!), thereby maximizing your use of memory and processor utilization. Any others you can think of?

If you’re interested, Eric has a book on the subject: “Swarm Intelligence: From Natural to Artificial Systems“.


LOGO (mentioned in the Emergence book I just read), has been re-released in Java as StarLogo by a group at MIT [slashdot]. According to the StarLogo site, it is a “… programmable modeling environment for exploring the workings of decentralized systems — systems that are organized without an organizer, coordinated without a coordinator. With StarLogo, you can model (and gain insights into) many real-life phenomena, such as bird flocks, traffic jams, ant colonies, and market economies.” If you’ve ever wondered why traffic jams form, you can now model them! 🙂