Of course this is a topic I have long discussed, calling for some philanthropist to fund what may be the most effective single endeavor of all time -- better methodologies for finding out who is right a lot, and for denying credibility to those who are consistently wrong -- in the form of a Predictions Registry.Has there been any progress, at all? The concept that has momentum, at present (a small amount), is "Predictions Markets".
Long Bets, An Arena for Accountable Predictions, is "a public arena for enjoyably competitive predictions, of interest to society, with philanthropic money at stake," overseen by The Long Now Foundation. Another example is Intrade, which allows you to make predictions on real world events by buying and selling shares in each event.
I have long argued with Robin Hanson, one of the founders of the prediction markets concept, over whether this is a more useful approach than my own Prediction Registries. (Recall how DARPA got in trouble for sposoring a study of "betting" about future terrorism? One case where the press and the left were loony, because this concept was and remains valid and potentially useful.)
How does the "markets" concept work? Wagering is posited to force people to be both explicit and serious, allocating their predictions carefully in a "market" style competitive arena. I grasp this advantage... and disagree. This kind of thing attracts people with little to lose and with brash axes that they want to grind. So far, the advantages of market style accountability seem not to have appeared.
In any event, I believe these prediction markets have a crucial fault... people want to make predictions and forecasts, but they do NOT want to lose wagers. Hence, while many loudmouths will make public pronouncements, they will not explicitly stake anything on the outcome. Others who have strong reason to believe that a course of action will go badly are intimidated from expressing these views openly.
Hence, these arenas will not attract
(1) politicians,
(2) civil servants who believe their political bosses are mistaken,
(3) lesser-knowns who have no "name-recognition" or resources, but who happen to be right an anomalously large fraction of the time.
My alternative model of registries (or rather, a vast and diverse community of registries) would have a crucial advantage. Participation need not be a voluntary decision on the part of some willing and eager wagerer. Rather, a public figure can be "outed" into having a prediction registered, whether they wanted it made explicit or not. (There should be an appeals process by which someone can say "that's out-of-context" and I did not predict that, specifically. Fine, let them replace the outed forecast with what they DO believe will happen. There can also be room for scenarios weighted by probabilities.)
Moreover, through pseudonymity, even civil service personel (say experts in the MIddle East) could forecast what they believe will happen. While retaining confidentiality and trust within their agency, they would nevertheless be able to establish a clear record. Later they can point to a string of successful predictions and gain deserved credibility.
Again, it seems an experiment worth trying. It would be incredibly cheap. (Like proving the poisonous effects of indignation addiction.)
It could change the world.
And why am I bothering?
18 comments:
I agree that registries would be preferable to markets, especially since the people were are trying to handicap are the ones that often speak loud on the press, but would never sign on to one of these things.
Sounds like a good thing for a wiki-style creation.
Actually, there is also a "black" analogue for this. The CIA began using an internal blog to get analysts to collaborate and cross-reference each other. The biggest problem is the "seeding" problem, or getting enough people to participate in the beginning to keep the concept from flaming out.
Scoring and normalization would be something interesting to try out (maybe steal some BCS type formula for riskiness of the call or something) - especially with fuzzy predictions like "well, he got it half right..."
Finally ... let me look into this. I'm not quite volunteering yet, but I would be interested in seeing what it would take.
Now there's an idea worth playing with, and I can see the journal article now: "Application of the Black-Scholes Formula to Prediction Markets".
<bond villain voice>You intrigue me, Doctor Brin.</voice> You've got me wanting to zoop off and start designing a proper predictions registry, in all my spare time... I agree, it could change an awful lot of things for the better. I suppose it wouldn't hurt to start specking it out...
On the general subject of disputation arenas, I've come across another site attempting to bring rationality to the debate process:
http://truthmapping.com
I have some criticisms of the structure, but it's still an excellent stab at the idea. (Apparently it has been around for awhile, too, without my noticing. Tsk.)
It has been startling, the last few months, to realize how many of my creative obsessions fall together into the general field of improving inter-human problem-solving methodologies in order to help enhance aggregate intelligence.
Everything from my proposal regarding research into indignation-addiction (http://www.davidbrin.com/addiction.html)
to disputation arenas... to the Eye of the Needle Foundation... to prediction registries... to transparency/accountability... to eye-tracking lie-detection... to philosophical modernism... to revisionist (apprenticeship) theology... to my patent/invention having to do with ways that online discourse might be massively improved (http://www.holocenechat.com)...
...all the way to the fundamental zeitgeist that is Citokate....
It can be unnerving and humbling to see what you had thought to be a vastly wide and eclectic interest-set distill down to manifestations of a single obsession. Huh. I can even track it now into childhood influences. Jeez. I might as well stand on a streetcorner with a tambourine and chant "love!"
;-)
Seriously, the CIA is one of many agancies that have danced at the edges of talking to me about Holocene, for empowering internal discussions... then backed away because internal advocates could not get upper managers to "get it." The sad thing is it would be so cheap to implement. Maybe five programmers in the States and ten in Bangalore, for one year.
Plus (most important) one "guy from the nineties" who really got it. Some fellow who built companies then and is as bored and disappointed in the #$*&! 21st Century as I am.
Sorry, I am meandering.
Black-Scholes seems interesting on many levels. First I thought it was Black Scholars or Black Studies... (huh?)...
Examining the paper, I realized how deeply and profoundly we have migrated away from Marx and his Labor Theory of Value... not that MODERN economics is without superstitions.
Aaah, Black-Scholes. Such a fun little construct. And it almost managed to bring down the stock market all by itself. You know you've got someone when you can create that much mayhem (and I'm not being snarky here ... this is one of those situations where you sit back with a whistle and say 'they can do *that*?') Check out PBS' Trillion Dollar Bet some time, it's a great adventure story for nerds.
Re: CIA implementation of 'net technologies:
That NYT article that I had seen also cited the fact that CIA culture was still just not conducive to sharing. When you worry that guy in the chair next to you might download the whole NOC list and send it to Moscow, it really mucks things up.
Of course, if you are just worried about the terrorists getting tipped off about how much you know, it would be tough for the terrorists to act on what they learn. Otherwise, you'd be able to track the plans changing. It's akin to when the Allies cracked Enigma. They couldn't use the information too often, or the bad guys would know that they know. And then you enter the infinte regrees of intelligence, counter-intelligence and counter-counter-intelligence...
I agree that simply collecting track records gives you most of the benefit of prediction markets (and I'll post on this Wed. at OvercomingBias.com). But I fear without some standardization of the predictions it will be too expensive to find, translate and check a big pile of random "predictions."
Now that (some of) my books are unpacked, I can finally look things up. The following is an extract from Keith Devlin's The Language of Mathematics (1998). Hyperlinks have been added by me.
In 1997, the Nobel Prize in economics was shared by Stanford University professor of finance (emeritus) Myron Scholes and economist Robert C. Merton of Harvard University. The prize would undoubtedly have been shared with a third person, Fischer Black, but for the latter's untimely death in 1995. The prize was awarded for the discovery, in 1970, of a single mathematical formula: the Black–Scholes formula.
Discovered by Scholes and Black, and developed by Merton, the Black–Scholes formula tells investors what value to put on a financial derivative. A derivative is a financial instrument that has no value of its own; rather, it derives its value from the value of some other asset. A common example is the stock option, which proves the purchaser with the option —but not the obligation—to purchase stock at an agreed-upon price before a specified date. Derivatives amount to 'side bets' used to offset the risks associated with doing business under the ever-chaning circumstances of the everyday world.
The Black–Scholes formula provides a means for determining the price to be put on a particular derivative. By turning what would otherwise be a guessing game into a mathematical science, the Black–Scholes formula made the derivatives market into the hugely lucrative industry it is today.
So revolutionary was the very idea that you could use mathematics to price derivatives that initially Black and Scholes had difficulty publishing their work. When they first tried in 1970, the University of Chicago's Journal of Political Economy and Harvard's Review of Economics and Statistics both rejected the paper without even bothering to have it reviewed. It was only in 1973, after some influential members of the Chicago faculty put pressure on the editors, that the Journal of Political Economy published the paper.
Industry was far less shortsighted than the denizens of the ivory tower. Within six months of the publication of the Black–Scholes article, Texas Instruments had incorporated the new formula into its latest calculator, announcing the new feature with a half-page ad in the Wall Street Journal.
[...]
Not only did the new formula work, it transformed the market. When the Chicago Options Exchange first opened in 1973, fewer than 1,000 options were traded on the first day. By 1995, over a million options were changing hands each day.
So great was the role played by the Black–Scholes formula (and extensions due to Merton) in the growth of the new options market that, when the American stock market crashed in 1987, the influential business magazine Forbes put the blame squarely on that one formula. Scholes himself has said that it was not so much the formula that was to blame, but rather that marketeers had not grown sufficiently sophisticated in how to use it.
[end quoted material, found on pages 296 ff. of the original book]
Let that be a lesson for you anarchist kids: you stand a much better chance of bringing down the Man if you learn your 'rithmetic.
"(Recall how DARPA got in trouble for sposoring a study of "betting" about future terrorism? One case where the press and the left were loony, because this concept was and remains valid and potentially useful.)"
I was under the impression that the idea was attacked on all sides.
"It has been startling, the last few months, to realize how many of my creative obsessions fall together into the general field of improving inter-human problem-solving methodologies in order to help enhance aggregate intelligence. "
Which leads to the question, what did you the of the book, "The Wisdom of Crowds."
Lastly, though I'm usually not one to push market solutions over non-market ones, I keep them in mind; I don't entire see why you're against prediction *markets* (and who says people have to bet actual money, I used to play gambling simulation computer games without losing my shirt. There could be systems of betting "virtual money" or "credits" (although this would probably bleed over into the real world as with World of Warcraft players "selling" characters, items, and gold, for real money through some channels. But I digress.))
What I love about the idea of prediction markets is the idea of people shouting at Ann Coulter, Rush Limbauch, Michael Savage, and Orson Scott Card (yes, it's sad to put him in the same catagory, but I've read years worth of his columns; he deserves it) "Put your money where your mouth is!"
Ah, we are honored (and so unworthy!) to have the great and sagacious Robin Hanson among us! ;-) Teasing but I sooooo mean it, too. Robin is one of the smartest guys I know.
Jon sayeth: "Which leads to the question, what did you the of the book, "The Wisdom of Crowds.""
It's a terrific book. But I am more cautious. Yes, we share a vision that the Enlightenment Project can extend aggregate intelligence in new and powerful ways, beyond the four "accountability arenas" of markets, science, democracy and law courts... all of which are still rather hierarchical, but far more engaged with reciprocal accountability than any previous institutions that humanity ever knew.
The dream (partly underway due to vast education and the internet) is to take the Project further by unleashing the whole of humanity with tools to rapidly engage in data gathering, appraisal, exchange, citokate, negotiation, competition, cooperation and problem-solving consensus. Exactly at the moment when problems roil too rapidly for older problem-solving entities to cope alone.
That is the dream. Alas, I feel that many of my fellow believers in "smart mobs" are too blithe and expect that this property will simply emerge from education and interconnectivity... but I think that's naive. Just as the older accountability arenas took a lot of hard, practical design and regulation, in order to overcome many nasty tendencies of human nature (including "cheating"), the new "wise crowds" will require a VAST array of innovative methods in order to keep from simply being highly interconnected dumb mobs. Rich in data and very low in citokate or wisdom.
Thimk. The Enlightenment Project has refined the regulated art in courts and science and markets and democracy for three hundred years! And still they are filled with errors and travesties and barely creak at the level we demand. Can we refine the art of smart-mobbitude in just a decade, simply by unleashing millions into the wretched interface on Second Life?
No, if you are serious about seeing how very far we are from having the right tools, see my Google Tech Talk: http://tinyurl.com/yy7yxm
This is THE great crisis, in my opinion. Because a truly smart Fifth Accountability Arena could help us to solve every other pproblem and crisis. But very few will solve themselves in old fashioned ways.
One argument in favour of Prediction Markets instead of Prediction Registries is the much larger memeplex we're likely to create with Prediction Markets.
Consider this: What size was the the capital markets and the real estate markets memeplex 1000 years ago, what are they now? The percentage of energy that goes into real estate valuation, mortgages, real estate brokerage, real estate media, stock brokers, bankers, stock analysts, markets and on and on.
How much more attention is directed towards these versus other issues (eg valuation and trading of human capital) then versus now?
Why would we not want to flow with the "we all like to get something for nothing" primordial that underpins much stock market, real estate market and gambling, and go for Prediction Markets?
I suggest we view the underpinning as chi, energy and have that energy flow faster into memeplexes that suck them in. Achieve critical mass faster.
Now, there might be problems with "gamblers in Prediction Markets", but by then you would have lots more resources to deal with that. Just as we have heaps of problems with real estate and capital markets and deal with them.
I would prefer us not under-estimating the amount of energy it takes to take a notion into strength and keep it there. Especially given current conditions where there appears to be "lots of great new ideas but not enough risk-takers to put their everything into making them happen".
Chew on this, Mr. Brin. I'm not sure whether this could be applied to the field of prediction accuracy but maybe, with some work and several groups of people getting together, you could do it:
Are you familiar with the Jiro Kawakita method of getting deriving priorities? This is known as the KJ-Method and I can attest to its effectiveness with regards to deciding what's important information and what isn't. One of its major advantages is that, although people aren't anonymous, when properly run, it does guarantee that no bullying or cowing of others takes place and that a real group consensus is reached (not that that's what you're going for here). It is a type of brainstorming method. Some claim it is an affinity diagram but it is not. I think Jiro Kawakita developed both though.
I'm not going to reproduce the KJ method step by step here but I'll provide a link to it. Please read this as I think, just maybe, this could apply here.
Go to
http://www.uie.com/articles/kj_technique/
and also
http://software.isixsigma.com/library/content/c050622b.asp
In my mind, there would not be a huge overload for a prediction registry at first, because we could limit the focus to those we care about.
The first purpose of a good prediction market is to bring real accountability to punditry. Think about it! What if you could go up to Thomas Friedman, or George Will, or any talking head and ask, "Why should I listen to you? Your prediction rate is 5%. My friend does better on his daily blog!"
How much different the landscape could be if it was no longer cost-free to make horrible and sloppy pronouncements in national media and influence decision-making. Perhaps I am over-estimating people's abilities/interest to tune in (after all, fact checking is pretty weak nowadays), but also perhaps a very simple score would help. A Consumer Reports of the news, if you will. And Americans do tend to be fairly wary consumers.
Now, for a great moment in accountability, something that is brand new for its scale. Speaker Pelosi is reaching out feelers to the netroots community as to what would be needed to get more citizen participation in lawmaking. First step - electronic release of all bills before voting to the public for very rigorous reading. There are also a lot of folks pushing for greatly simplified bills to help this.
But even if not, what does it matter! In electronic format, we have computers that can scan and parse, and use the logic of spam-hunting to go looking for more traditional pork. (Is it strange to you that multiple pork products have the same reputation and roughly the same implementation?) Rather than running Congressional staffers to death, the computer codes could toss red flags.
Anyways, check this discussion out:
http://www.dailykos.com/story/2006/12/11/182844/48
The K-J method looks sensible and useful for basic problem-identification by facilitators. A minor but useful methodology.
One advantage of registries is that they can be involuntary. A disadvantage is that the pundit must say something explicit about the future. Try analysing George Will some time, and distilling an actual prediction! That is why the "outing" process should include PARAPHRASE CHALLENGES:
"Well? Is this what you are saying or predicting? Or isn't it?"
Comparing registries to markets is not entirely fair. As one PM Cluster member pointed out, They focus on different outcomes.
The aim of the registry is to score individuals over an extended period, deriving a credibility rating or score for PREDICTORS.
Most prediction markets would focus (at least initially) on Predicted EVENTS, attempting to weigh probabilities and giving people a chance to wager "I give outcome A a 70% chance..."
Of course they could interact and synergize powerfully. Markets could feed data to registries and allow them the subtlety of probabilities, instead of nailing people to on-off forecasts. (though the registries would still need other input sources, like "outing" of those who refuse to bet.)
Registries could in turn feed and help PMarkets by providing credibility weighting factors, allowing the markets to forecast specific outcomes in a more sophisticated way,
---
As for Speaker Pelosi etc, naturally I hope they will look at the ideas at:
http://www.davidbrin.com/suggestions.html
I believe the dems are really missing a great move if they do not refer to Newt Gingrich's 1992 Contract With America. By pointing to the "good parts" of that masterful piece of political polemic, they could:
1) point out that the GOP were utter hypocrites, pulling bait-and-switch. The good parts were never enacted... while the goppers swiftly put into effect every measure that would benefit a narrow elite. This CONTRAST would be devastating.
2) avow "we understand what made voters angry at us in 1992, so WE will enact the good parts ourselves, to prove we listen."
3) say "this is proof that we are not in this for partisan war. See? Not only are we reaching across the aisle, but we are welcoming into the conversation those (few) republicans who have always been for reform.
Name for me another move that would be seen as SIMULTANEOUSLY devastating to the other side AND bipartisan-mature? The GOP would have to choke and swallow their fury over this, unable to even speak.
The problem with betting on future terrorism was that someone could post a prediction and then IMPLEMENT IT. We wouldn't want to encourage people to commit violence to win a bet.
With a predictions registry, we would have to prohibit predictions of, say, interest rates by the people who actually set interest rates. (He who controls the future cannot be proven incorrect.)
There is a related problem. Someone who predicts a crime but is not involved in it may nevertheless be arrested for having "advanced knowledge" of the crime. Accomplice before the fact will be the charge.
To jump back to Iraq, one thing that Hilzoy said a while ago at Obsidian Wings has stuck in my head. The Iraqis who worked with us will be the first targets if and when the US leaves. We should offer them all asylum here, since it's our fault they're in trouble. Yeah, that could have problems with other people trying to sneak in, but I'm really not that scared of terrorists who'd try that. Chances are decent the others fleeing to America would report them. Of course, considering the administration in charge, they would probably just lock people up on accusations... But since we won't see any movement in Iraq until Bush is out of office, I'm not sure how relevant that is.
Bala noted that markets would be more succesful during initial growth phase due to their ability to generate a stronger memeplex, and I agree. But there would also be strong 'market' forces at play in a prediction registery, allowing for rapid capitilization, and the incentivizing of registeries. For instance, if bloggers submitted their predictions to the registery, over time their ratings would go up or down, which could, in turn, increase/decrease the readership for their blog which would then affect their ad revenues. Another example would be in the consultant/soothesayer/advisor catagory. These industries might radically change, because the rate these prognosticators charge (or their airtime) could be directly tied to their score in the registey. Although markets might be a more effective way of generating initial interest, there is certainly an economic incentive (especially for those that make their living off of giving advice) to perform well and participate in a registry. I would argue that it is more important, in fact, for this caste to be valued through a registery, because of their memetic influence on all other segments of society. Therefore, if our goal as a civilization is to determine WHO best predicts the future, the registry outperforms the market, but in large part through market-based incentives.
Hi David -
I'm currently working on implementing a concept known as truthmarkets (www.truthmarkets.com) It is a claims market but without the need for active "claimant" participation. Any claim can be listed and judged, whether or not the "claimant" knowingly participates.
Claims trading also yields a "truthiness" rating for each claimant.
I think truthmarkets mesh well with your interests and goals and I'd love to get your feedback.
Best regards,
Daniel
Post a Comment