Losing the Long Tail

By Zootella | July 13, 2006

“NOW PLAYING,” the marquee above Tribeca Cinemas last night read: “EVERY MOVIE EVER MADE.” People passing by unfamiliar with Chris Anderson’s new book The Long Tail might have been confused by that. The launch party, hosted by Wired and our friends at Flavorpill, was fun, and LimeWire was there.

Nathan, who as a known blogger already had an advance copy of the book and an invite to the party, had given the book a somewhat negative review. Yes, the Internet enables choice. Open your back catalog of media. Make an online store. This idea is 1999 and Amazon.com. It’s very Bubble 1.0—old news to today’s bloggers. He’s right, but Anderson’s idea of packaging the idea in a slim business book is still pretty clever. Executives read advertisements for summaries of these books on airplanes. These books are the key to controlling their minds.

Chris gave his presentation in the crowded theater, and then asked for questions. Surprisingly, no one had any. Well, I had one, after reading this in his book:

The great thing about broadcast is that it can bring one show to millions of people with unmatchable efficiency. But it can’t do the opposite - bring a million shows to one person each. Yet that is exactly what the Internet does so well. The economics of the broadcast era required hit shows - big buckets - to catch huge audiences. The economics of the broadband era are reversed. Serving the same stream to millions of people at the same time is hugely expensive and wasteful for a distribution network optimized for point-to-point communications (Anderson 5).

Point-to-point communications built the long tail, and point-to-point communications make up a peer-to-peer network. At a first glance, it looks like the two are perfect for each other—together, we’ll build the future! But, I had another suspicion, and a trick question ready to test it. Here’s how I thought it would go:

Me: Hi, I’m Kevin Faaborg, a developer at LimeWire—we’re located just a few blocks from here. My question is about BitTorrent. Is BitTorrent a part of the long tail?

Chris: Yes.

Me: But BitTorrent is horrible at rare stuff! As soon as a file becomes rare, it loses seeders and dies.

But, Chris is really smart and knows his stuff. Here’s how the conversation actually went:

Me: Is BitTorrent a part of the long tail?

Chris: No. I see BitTorrent as an amplifier for the long tail.

We kept talking. Chris seemed disappointed with peer-to-peer as it related to the long tail. It’s like we’ve let him down. My suspicion was correct, and Chris knew about it. Here’s modern p2p’s dirty little secret: It’s actually horrible at rare stuff. To explain why, let me take you all the way back to 2000, when Sean Fanning’s Napster ruled the earth. Here’s page 171 of Joseph Menn’s book All the Rave:

One of the existing system’s weaknesses was that everyone who logged on to search for a song was sent to just one server, which in turn referred the user to others who were relying on the same machine. When the number of servers grew to fifty, say, that meant each user was only exposed to one-fiftieth of all the songs available on Napster at that moment. But linking them all together could make the system many times more powerful that it was (Menn 171).

The existing system was Napster, the peer-to-peer file sharing service that used a machine room of central servers to index the network for search. The challenge was to link all the servers together, which they did:

Soon enough, Ritter and Jordy Mendelson did lash together Napster’s servers into two massive clusters. In late May, Ritter told the marketing and customer-service people that he had gone even further, linking together those two clusters (Menn 178).

They did it. Your rare search on Napster would reach every single computer online. Every single one! With central servers, Napster touched the farthest tip of the long tail.

Then, along came Gnutella.

When Gnutella appeared nearly three months later, Napster’s engineers were nervous but impressed. The system was much more complicated than Napster’s. Instead of relying on a central server, Gnutella allowed users to create their own, smaller networks for searches (Menn 173).

Gnutella today is even more complicated than when it appeared. Eliminating central servers while keeping things fast and efficient created a long list of technical challenges. Searches are only possible over smaller networks. (Gnutella programs running as ultrapeers connect to 32 other ultrapeers, and have up to 30 leaves beneath them. In dynamic querying, the farthest-reaching search has a TTL of just 3.) Searches cast a net wide enough to get the short head, but not the long tail. The instant and universal search died with the central server. Wallace Wang’s Steal this File Sharing Book gets it totally wrong:

Gnutella is one of the oldest, largest, and most popular of the file sharing networks, but searching it can be slow. When you request a file, your request goes through every computer connected to the Gnutella network, and they can number in the thousands (Wang 24).

First, online Gnutella programs number in the millions. Second, there’s no way a search can go through every one of them. (Actually, nothing’s ever impossible with software: A Gnutella crawler can contact every program online. Here’s the catch: it takes a hour to crawl the whole network. And, the network can only support this feature for the small number of people in the world running crawlers at any given time.)

So, that’s the problem. P2P had to sacrifice its center. Search lost universal coverage. Download lost guaranteed delivery. Peer-to-peer lost the long tail.

We’re working on the solution now: the DHT. Much more on that, soon. Also cool are Web-backed peer-to-peer URLs, like Dijjer and RedSwoosh links. The long tail is great. Through advanced technology and clever features, peer-to-peer has to recapture it.