The Wisdom of Crowds: Why the Many Are Smarter Than the Few and How Collective Wisdom Shapes Business, Economies, Societies and Nations by James Surowiecki Doubleday May 25, 2004 320 pages Excerpts: pp 17-22: V If allowing people to bet on sporting events effectively creates a kind of machine that's good at predicting the outcomes of those events, an obvious question follows. Wouldn't people betting on other kinds of events be equally good, as a group, at predicting them? Why confine ourselves to knowing what the chances are of Los Angeles beating Sacramento if there's a way we could know what are the chances are of, say, George W. Bush beating John Kerry. We do have a well-established way of knowing what George Bush's chances are: the poll. If you want to know how people are going to vote, you just ask them. Polling is, relatively speaking, accurate. It has a solid methodology behind it, and is statistically rigorous. But there's a reason to wonder if a market such as the betting market - one that allowed the people participating in it to rely on many different kinds of information, including but not limited to polls - might at the very least offer a competitive alternative to Gallup. That's why the Iowa Electronic markets (IEM) project was created. Founded in 1988 and run by the College of Business at the University of Iowa, the IEM features a host of markets designed to predict the outcomes of elections - presidential, congressional, gubernatorial, and foreign. Open to anyone who wants to participate, the IEM allows people to buy and sell futures "contracts" based on how they think a given candidate will do in an upcoming election. While the IEM offers many different types of contracts, two are most common. One is designed to predict the winner of an election. In this case of the California recall in 2003, for instance, you could have bought an "Arnold Schwarzenegger to win" contract, which would have paid you $1 when Schwarzenegger won. Had he lost, you would have gotten nothing. The price you pay for this kind of contract reflects the market's judgment of a candidate's chances of victory. If a candidate's contract costs 50 cents, it means, roughly, speaking, that the market thinks he as a 50 chance of winning. If it costs 80 cents, he has an 80 percent chance of winning, and so on. The other major kind of IEM contract is set up to predict what percentage of the final popular vote a candidate will get. In this case, the payoffs are determined by the vote percentage: if you'd bought a George W. Bush contract in 2000, you would have received 48 cents (he got 48 percent of the vote) when the election was over. If the IEM's predictions are accurate, the prices of these different contracts will be close to their true values. In the market to predict election winners, the favorite should always win, and bigger favorites should win by bigger margins. Similarly, in the vote-share market, if George W. Bush were to end up getting 49 percent of the vote in 2004, then the price of a George W. Bush contract in the run-up to the election should be close to 49 cents. So how has the IEM done? Well, a study of the IEM's performance in forty-nine different elections between 1988 and 2000 found that the election-eve prices in the IEM were, on average, off by just 1.37 percent in presidential elections, 3.43 percent in other U.S. elections, and 2.12 percent in foreign elections. (Those numbers are in absolute terms, meaning that the market would have been off by 1.37 percent if, say, if had prediction that Al Gore would get 48.63 percent of the vote when in reality he got 50 percent.) The IEM has generally outperformed the major national polls, and has been more accurate than them even months in advance of the actual election. Over the course of the presidential elections between 1988 and 2000, for instance, 596 different polls were released. Three-fourths of the time, the IEM's market price on the day each of these polls was released was more accurate. Polls tend to be very volatile, with vote shares swinging wildly up and down. But the IEM forecasts, though ever-changing, are considerably less volatiles, and tend to change dramatically only in response to new information. That makes them more reliable as forecasts. What's especially interesting about this is that the IEM isn't very big - there have never been more than eight hundred or so traders in the market - and it doesn't, in any way, reflect the makeup of the electorate as a whole. The vast majority of traders are men, and a disproportionate - though shrinking - number of them are from Iowa. So the people in the market aren't predicting their own behavior. But their predictions of what the voters of the country will do are better than the predictions you get when you ask the voters themselves what they're going to do. The IEM's success has helped inspire other similar markets, including the Hollywood Stock Exchange (HSX), which allows people to wager on box-office returns, opening-weekend performance, and the Oscars. The HSX enjoyed its most notable success in March of 2000. That was when a team of twelve reporters from the *Wall Street Journal* assiduously canvassed members of the Academy of Motion Pictures Arts and Sciences in order to find out how they had voted. The academy was not happy about this. the organization’s president publicly attacked the *Journal* for trying to "scoop us before Oscar night," and the Academy urged members not to talk to reporters. but with the *Journal* promising anonymity, more than a few people - 356, or about 6 percent of all members - disclosed how they had filled out their ballots. The Friday before the ceremony, the *Journal* published its results, forecasting the winners in the six major Oscar categories - Best Picture, Best Director, Best Actor, and Best Actress, Best Supporting Actor and Best Supporting Actress. And when the envelopes were opened the *Journal's* predictions - much to the Academy's dismay - turned out pretty much on target, with the paper picking up five of the six winners. The HSX, though, had done even better, getting all six of the six right. In 2002, the exchange, perhaps even more impressively, pickled thirty-five of the eventual forty Oscar Nominees. The HSX's box-office forecasts are not as impress or as accurate as the IEM's election forecasts. but Anita Elberse, a professor of marketing at Harvard Business School, has compared the HSX's forecasts to other Hollywood prediction tools, and found that the HSX's closing price the night before a movie opens is the single best available forecast of its weekend box office. As a result, the HSX's owner, Cantor Index Holdings, is now marketing its data to Hollywood studios. One of the interesting things about markets like the IEM and the HSX is that they work fairly well without much - or any - money at stake. The IEM is a real money market, but the most you can invest is $500, and the average trader has only $50 at stake. In the HSC, the wagering is done entirely with play money. All the evidence we have suggests that people focus better on a decision when there are financial rewards attached to it (which may help explain why the IEM's forecasts tend to be more accurate). But David Pennock - a research at Overture who as studied these markets closely - found that, especially for active traders in these markets, status and reputation provided incentive enough to encourage a serious investment of time and energy in what is, after all, a game. As the potential virtues of these decision markets have become obvious, the range of subjects they cover has grown rapidly. At the NewsFutures and TradeSports exchanges, people could bet, in the fall of 2003, on whether or not Kobe Bryant would be convicted of sexual assault, on whether and when weapons of mass destruction would be found in Iraq, and on whether Ariel Sharon would remain in power longer than Yasir Arafat. Ely Dahan, a professor at UCLA, has experimented with a classroom-decision market in which students bought and sold securities representing a variety of consumer goods as services, including SUVs, ski resorts, and personal digital assistants. (In a real-life market of this kind, the value of a security might depend on the first-year sales of a particular SUV.) The market's forecasts were eerily similar to the predictions that conventional market research had made (but the classroom research was much cheaper). In the fall of 2003, meanwhile, MIT's *Technology Review* set up a site called Innovation Futures, where people could wager on future technological developments. And Robin Hanson, an economics professor at George Mason University who was one of the first to write about the possibility of using decision markets in myriad contexts, has suggested that such markets could be used to guide scientific research and even as a tool to help governments adopt better policies. Some of these markets will undoubtedly end up being of little use, either because they'll fail to attract enough participants to make intelligent forecasts or because they'll be trying to predict the unpredictable. But given the right conditions and the right problems, a decision market's fundamental characteristics - diversity, independence, and decentralization - are guaranteed to make for good group decisions. And because such markets represent a relatively simple and quick means of transforming many diverse opinions into a single collective judgment, they have the chance to improve dramatically the way organizations make decisions about the future. In that sense, the most mystifying thing about decision markets is how little interest corporate America has shown in them. Corporate strategy is all about collecting information from many different sources, evaluating the probabilities of potential outcomes, and making decisions in the fact of an uncertain future. These are tasks for which decision markets are tailor-made. Yet companies have remained, for the most part, indifferent to this source of potentially excellent information, and have been surprisingly unwilling to improve their decision making by tapping in to the collective wisdom of their employees. We'll look more closely at people's discomfort with the idea of the wisdom of crowds, but the problem is simple enough: just because collective intelligence is real doesn't mean that it will be put to good use. A decision market is an elegant and well-designed method for capturing the collective wisdom. But the truth is that the specific method that one uses probably doesn't matter very much. In this chapter, we've looked at a host of different ways of tapping in what a group knows: stock prices, votes, point spreads, pari-mutuel odds, computer algorithms, and futures contracts. Some of these methods seem to work better than others, but in the end there's nothing about a futures market hat makes it inherently smarter than, say Google or a pari-mutuel pool. these are all attempts to tap into the wisdom of the crowd, and that's the reason they work. The real key, it turns out, is not so much perfecting a particular method, but satisfying the conditions - diversity, independence, and decentralization - that a group needs to be smart. AS we'll see in the chapters that follow, that's the hardest, but also perhaps the most interesting, part of the story. ... pp 75-83: Understanding when decentralization is a recipe for collective wisdom matters because in recent years the fetish for decentralization has sometimes made it seem like the ideal solution for every problem. Obviously, given the premise of this book, I think decentralized ways of organization human effort are, more often than not, likely to produce better results than centralized ways. but decentralization works well under some conditions and not very well under others. In the past decade, it's been easy to believe that if a system is decentralized, then it must work very well. But all you need to do is to look at a traffic jam - or, for that matter, at the U.S. intelligence community - to recognize that getting red of a central authority is not a panacea. Similarly, people have become enamored of the idea that decentralization is somehow *natural* or *automatic*, perhaps because so many of our pictures of what decentralization looks like come from biology. Ants, after all, don't need to do anything special to form an ant colony. Forming ant colonies is inherent in their biology. The same is not, however, true of human beings. It's hard to make real decentralization work, and hard to keep it going, and easy for decentralization to become disorganization. A good example of this was the performance of the Iraqi military during the U.S.-Iraq war in 2003. in the early days of the war, when Iraqi fedayeen paramilitaries had surprised U.S. and British troops with the intensity of their resistance, the fedayeen were held up as an example of a successful decentralized group, which was able to flourish in the absence of any top-down control. In fact, one newspaper columnists compared the fedayeen to ants in an ant colony, finding their way to a "good" solution while communicating only the soldiers right next to them. But after a few days, the idea that the fedayeen were mounting a meaningful , organized resistance vanished, as it became clear that their attacks were little more than random uncoordinated assaults that had not connection to what was happening elsewhere in the country. As one British commander remarked, it was all tactics and not strategy. To put it differently, the individuals actions of the fedayeen fighters never added up to anything bigger, precisely because there was not method of aggregating their local wisdom. The fedayeen were much like ants - following local rules. But where ants who follow their local rules actually end up fostering the well-being of the colony, soldiers who followed their local rules ended up dead. (It may be, though, that once the actual war was over, and the conflict shifted to a clash between the occupying U.S. military and guerrillas using hit-0and-run terrorist tactics, the absence of aggregation became less important, since the goal was not to defeat the United States in battle, but simple to inflict enough damage to make staying seem no longer worth it. in that context, tactics may have been enough.) The irony is that the true decentralized military in the U.S. Iraq war was the U.S. Army. American troops have always been given significantly more initiative in the field than other armies, as the military has run itself on the "local knowledge is good" theory. But in recent years, the army has dramatically reinvented itself. Today, local commanders have considerably greater latitude to act, and sophisticated communications systems mean that collectively wise strategies can emerge from local tactics. Commanders at the top are not isolated from what's happening in the field, and their decisions will inevitably reflect, in a deep sense, the local knowledge that field commanders are acquiring. In the case of the invasion of Baghdad for instance, the U.S. strategy adapted quickly to the reality of Iraq's lack of strength, once local commanders reported little or no resistance. This is not to say, as some have suggested, that the military has become a true bottom-up organization. The chain of command remains essential to the way the military works, and all battlefield action takes place within a framework defined by what's known as the Commander's Intent, which essentially lays out a campaign’s objectives. But increasingly, successful campaigns may depend as much on the fast aggregation of information from the field as on preexisting, top-down strategies. V When it comes to the problems of the U.S. intelligence community before September 11, the problem was the *kind* of decentralization that the intelligence community was practicing. On the face of it, the division of labor between the different agencies makes a good deal of sense. Specialization allows for a more fine-grained appreciation of information and greater expertise in analysis. And everything we know about decision making suggests that the more diverse the available perspectives on a problem, the more likely it is that the final decision will be smart. Acting Defense Intelligence Agency director Lowell Jacoby suggested precisely this in written testimony before Congress, writing, "Information considered irrelevant noise by one set of analysts may provide critical clues or reveal significant relationships when subjected to analytic scrutiny by another." What was missing in the intelligence community, though, was any real means of aggregating not just information but also judgments. In other words, there was no mechanism to tap into the collective wisdom of National Security Agency nerds, CIA spooks, and FBI agents. There was decentralization but not aggregation, and therefore no organization. Richard Shelby's solution to the problem - creating a truly central intelligence agency - would solve the organization problem, and would make it easier for at least one agency to be in charge of all the information. But it would also forgo all the benefits - diversity, local knowledge, independence - that decentralization brings. Shelby was right that information needed to be shared. But he assumed that someone - or a small group of someones - needed to be at the center, sifting through the information, figuring out what was important and what was not. But everything we know about cognition suggests that a small group of people, no matter how intelligent, simply will not be smarter that the larger group. And the best tool for appreciating the collective significance of the information that the intelligence community had gathered was the collective wisdom of the intelligence community. Centralization is not the answer. But aggregation is. There were and are a number of paths the intelligence community could follow to aggregate information without adopting a traditional top-down organization. To begin with, simply linking the computer databases of the various agencies would facilitate the flow of information while still allowing the agencies to retain their autonomy. Remarkably, two years after September 11, the government still did not have a single unified "watch list" that draws on data from all parts of the intelligence community. In some sense, quite simple, almost mechanical steps would have allowed the intelligence community to be significantly smarter. Other more far-reaching possibilities were available, too, and in fact some within the intelligence community tried to investigate them. The most important of these, arguably, was the FutureMAP program, an abortive plan to set up decision markets - much like those of the IEM - that would have, in theory, allowed analysts from different agencies and bureaucracies to buy and sell futures contracts based on their expectations of what might happen in the Middle East and elsewhere. FutureMAP, which got its funding from the Defense Advanced Research Projects Agency (DARPA), had two elements. The first was a set of internal markets, which would have been quite small (perhaps limited to twenty or thirty people), and open only to intelligence analysts and perhaps a small number of outside experts. These markets might actually have tried to predict the probability of specific events (like, presumably, terrorist attacks), since the traders in them would have been able to rely one, among other things, classified information and hard intelligence data in reaching their conclusions. The hope was that an internal market would help circumvent the internal politics and bureaucratic wrangling that have indisputably had a negative effect on American intelligence gathering, in no small part by shaping the kinds of conclusions analysts feel comfortable reaching. In theory, at least, an internal market would have placed a premium not on keeping one's boss or one's agency happy (or on satisfying the White House) but rather on offering the most accurate forecast. And since it would have been open to people from different agencies, it might have offered the kind of collective judgment that the intelligence community has found difficult to make in the past decade. The second part of FutureMAP was the so0-called Policy Analysis Market (PAM), which in the summer of 2003 became the object of a firestorm of criticism from appalled politicians. The idea behind PAM was a simple one (and similar to the idea behind the internal markets): just as the IEM does a good job of forecasting election results and other markets seem to do a good job of forecasting the future, a market centered on the Middle East might provide intelligence that otherwise would be missed. What distinguished PAM from the internal markets was that it was going to be open to the public, and that it seemed to offer the possibility of ordinary people profiting from terrible things happening. Senators Ron Wyden and Byron Dorgan, who were the leaders of the effort to kill PAM, denounced it as "harebrained," "offensive," and "useless." The public, at least those who heard about PAM before it was unceremoniously killed, seemed equally appalled. Given the thesis of the this book, it will not surprise you to learn that I think PAM was potentially a very good idea. The fact that the market was going to be open to the public did not mean that its forecasts would be more inaccurate. On the contrary, we've seen that even when traders are not necessarily experts, their collective judgment is often remarkably good. More to the point, opening the market to the public was a way of getting people whom the American intelligence community might not normally hear from - whether because of patriotism, fear, or resentment - to offer up information they might have about conditions in the Middle East. From the perspective of Shelby's attack on the intelligence community, PAM, like the internal markets, would have helped break down the institutional barriers that keep information from being aggregated in a single place. Again, since traders in a market have no incentive other than making the right prediction - that is, there are no bureaucratic or political factors influencing their decisions - and since they have that incentive to be right, they are more likely to offer honest evaluations instead of tailoring their opinions to fit the political climate or satisfy institutional demands. Senator Wyden dismissed PAM as a "fairy tale" and suggested that DARPA would be better off putting its money into "real world" intelligence. But the dichotomy was a false one. No one suggested replacing traditional intelligence gathering with a market. PAM was intended to be simply another way of collecting information. And in any case, if PAM had, in fact, been a "fairy tale," we would have known it soon enough. Killing the project ensured only that we would have no idea whether decision markets might have something to add to our current intelligence efforts. The hostility toward PAM, in any case, had little to do with how effective it would be or would not be. The real problem with it, Wyden and Dorgan made clear, was that it was "offensive" and "morally wrong" to wager on potential catastrophes. Let's admit there's something viscerally ghoulish about betting on an assassination attempt. But let's also admit that U.S. government analysts ask themselves every day the exact same questions that PAM traders would have been asking: How stable is the government of Jordan? How likely is it that the House of Saud will fall? Who will be the head of the Palestinian Authority in 2005? If it isn't immoral for the U.S. government to ask these questions, it's hard to see how it's immoral for people outside the U.S. government to ask them. There were serious problems that the market would have had to overcome. Most notably, if the market was accurate, and the Department of Defense acted on its predictions to stop, say, a coup in Jordan, that action would make the traders' predictions false and thereby destroy the incentives to make good predictions. A well designed market would probably have to account for such U.S. interventions, presumably by making the wagers conditional on U.S. action (or alternatively, traders would start to factor the possibility of U.S. action into their prices). But this would be a problem only if the market was in fact making good predictions. Had PAM ever become a fully liquid market, it would probably also have had the same problems other markets sometimes have, like bubbles and gaming. But it is not necessary to believe that markets work perfectly to believe that the work very well. More important, although most of the attention paid to PAM focused on the prospect of people betting on things like the assassination of Arafat, the vast majority of "wagers" that PAM traders would have been making would have been on more mundane questions, such as the future economic growth of Jordan or how strong Syria's military was. At its core, PAM was not meant to tell us what Hamas was going to do the next week or to stop the next September 11. Instead it was meant to give us a better sense of the economic health, the civil stability, and the military readiness of Middle Eastern nations, with an eye on what that might mean for the U.S. interests in the region. That seems like something about which the aggregated judgment of policy analysts, would be Middle Eastern experts, and businessmen and academics from the Middle East itself (the kind of people who would have likely have been trading on PAM) would have had something valuable to say. We may yet find out if they do, because in the fall of 2003, NetExchange, the company that had been responsible for setting up PAM, announced that in 2004, a new revised Policy Analysis Market (this one without government involvement of any sort) would be opened to the public. Net Exchange was careful to make clear that the goal of the market would not be to predict terrorist incidents but rather to forecast broader economic, social, and military trends in the region. So perhaps the promise of PAM will actually get tested against reality, instead of being dismissed out of hand. It also seems plausible, and even likely, that the U.S. intelligence community will eventually return to the idea of using internal prediction markets - limited to analysts and experts - as a means of aggregating dispersed pieces of information and turning them into coherent forecasts and policy recommendations. Perhaps that would mean that the CIA would be running what Senators Wyden and Dorgan scorn fully called "a betting parlor." But we know one thing about betting markets: they're very good and predicting the future.