Somebody made our lunch reservation under the name Prediction Company, and waiters and busboys at Santa Fe’s Escalera Cafe have been puzzling this one out all morning. Prophets of the new millennium? Loose nuts that rolled up against the Rockies and settled here in New Mexico?
As soon as I walk in with Doyne Farmer, Norman Packard, and James McGill, the hostess is after them: What do you guys predict? Is it, like, the weather? Or horse races? We thought maybe you were a family of Gypsies.
No one answers right away. The reticence is a little surprising. Smiling and blinking in his sweat clothes, Farmer looks to be the most guileless and unguarded guy in town, except maybe for Packard, who’s standing next to him in grad-student duds and shoulder-length hair. But appearances can be deceptive. The last time the two men were in business together, in the late 1970s, they led a sneak attack on Nevada casino roulette. Disguised as out-of-town yokels, they carefully tracked the racing balls around the roulette wheels, tapping measurements into tiny computers hidden in their shoes. The computers analyzed patterns of ball movements, came up with predictions for the next spot where the ball would drop, and tapped back. Then the men placed their bets.
Farmer still gets informal royalty checks from cloak-and-dagger gamblers who have adapted his equipment, but the casinos got the laws changed to make this kind of thing illegal. They also employ a lot of beefy, suspicious men to prevent it. That Farmer has lived to age 40 and Packard to 38 is proof that they can play it close to the vest.
Still, here’s the hostess, blinking expectantly.
We work on stock options, futures contracts, currencies, stuff like that, Packard finally says.
Really? Can you give us some tips?
McGill shakes his head. Everybody, he says, walking to a table, wants a cut of our action.
That certainly seems to be true, and not just here in the restaurant. Among those who have come around asking for tips are many who have never carried a tray: executives and economists from Citicorp, Goldman Sachs, Salomon Brothers, Kidder, Peabody, and other powerhouses of finance. Because what the Prediction Company really does is tell its clients on, say, a Friday, where prices in a particular market will be the next Tuesday. Not with twenty-twenty foresight, to be sure--but accurately enough, the three say, to make their clients a lot of money.
They do it using mathematical techniques taken from the science of chaos, a discipline based on the fundamental principle that regular patterns exist in many complex systems, whether the elements are ball bearings, sunspots, or readers of the Wall Street Journal. The Prediction Company principals believe they can use the math of chaos to tease out such patterns in the actions of thousands of traders and brokers--patterns of which those traders are no more aware than a pigeon is of the ever-changing shape its flock makes in the air.
As graduate students in the late 1970s--during the same years they were driving a beat-up van to Vegas to try out their roulette gizmos-- Farmer, Packard, and a few colleagues at the University of California at Santa Cruz helped develop the principles of chaos research. It gives them a certain credibility not usually extended to guys who cheerfully admit they never looked at a Wall Street Journal until last year. The big guns are clearly interested, Farmer says. I’ve gotten calls for years from headhunters.
Indeed, chaos in the market seems to be an idea whose time has come. I think the standard economic paradigm is used up at this point, says Edgar Peters, an investment manager who uses chaos techniques to help manage some $5 billion in investments at PanAgora Asset Management in Boston. Econometrics hasn’t made any advances in decades, and everybody’s ready for a new view. We know there must be something missing, and I think this is it.
Farmer and Packard haven’t thrown conventional market wisdom out the window; they’re not claiming that stock prices aren’t the result of rational speculation about a company’s future performance. But even the most cautious economists concede that after all rational causes have been filtered out, there is a certain amount of up-and-down motion in the prices of stocks, currencies, commodities, and other markets that has no ready explanation. Standard economic theory throws up its hands over these maddening ups and downs and says they’re just so much noise: a random walk.
Farmer thinks this is a particularly telling phrase. When we talk about markets or other phenomena as random, he says, we should realize this is just a way of saying we can’t predict it. At some level that we’re not picking up, there’s something more fundamental going on.
Not knowing how to account for randomness, most economic models smooth out the noise, in much the same way that a driver whose speedometer has read 24, then 26, then 24.5, then 25, would say, I’m going 25. Conventional economic wisdom says the causes of these fluctuations--small day-to-day decisions by individual investors, for instance--are too numerous and too complex to be accounted for, and without a proper accounting of the causes there is no way to predict the effects accurately.
That’s where chaos theory departs from conventional wisdom. It holds that you can look at part of a pattern and deduce what the next element in that pattern might be. You don’t need to know the cause of a stock movement; you just need to know when the movement is likely to occur. Within those 24s, 26s, and 24.5s, there’s money to be made.
Chaos studies, in essence, sidestep the whole issue of the relationship between original causes and later effects because they’re based on nonlinear equations--equations in which, as Farmer says, the response is not proportional to the stimulus. As an example, consider the clichéd straw that breaks the camel’s back. The straw weighs nearly nothing, but the extra weight has an effect out of all proportion to its tiny size, because its effect is determined not by the simple relationship of straw to camel but by a complex interaction among all the factors that have been affecting the poor beast, from the weight of earlier burdens to the temperature of the desert sand.
That’s how the stock market works, according to the Prediction Company, and that’s why it’s so hard to make connections between original causes and their eventual effects, such as minor market ups and downs. In general, it’s a characteristic of such nonlinear systems that small effects are amplified into very large effects at some later time, Farmer says. It would be surprising if the stock market were linear. Just as almost all animals are non-elephants, almost all physical phenomena are nonlinear.
Until Farmer, Packard, and a few like-minded researchers created chaos science, most physicists thought that nonlinear phenomena could be broken down into linear equations if they could just get enough information to describe every motion of every component of a complex system. So, for example, it was assumed to be theoretically possible to predict Manhattan traffic patterns by learning everything there was to know about every individual car. A key insight of early chaos work, however, was that even if the massive amount of information were somehow gathered, nonlinearity’s disproportion of cause and effect would still make it impossible to derive any predictions. You couldn’t predict the flow of Manhattan’s traffic even if you knew all the cars, all the speeds, and all the streets in the city. A slight tap on the brakes by one driver in one taxi on one street could affect the flow of traffic many miles away.
Individual traders in a market are analogous to individual cars, Packard says. But the ebb and flow of the entire market is like the ebb and flow of traffic that a helicopter would see. Chaos science is a bit like the helicopter, he says--it offers much more predictive power for much less information because it looks at the overall pattern of change itself, not at every object participating in the change.
The key to this lies in making the pattern appear in an imaginary realm called state space. A state is the description of a system (traffic, the stock market, the straws on a camel’s back) at a particular point in time; the space is where changes to that system can be plotted graphically. Consider, says Farmer, a very simple system: the pendulum of a clock. At any single instant its movements can be described by two measurements--its velocity and its position relative to its natural resting place in the center. Those two measurements define the coordinates for a single point on a two-dimensional graph.
For example, let the pendulum’s position be the x coordinate, and its velocity the y. At rest, the pendulum will be described by the coordinates 0,0, a point that we will place in the center of the graph. Now suppose you pull the pendulum to the left and hold it still. Its velocity is still 0, but its position is a negative number (positions to the right of center are positive, those to the left negative). When you let go, the pendulum starts back toward the center, picking up speed as it goes. So the position measurements get closer and closer to zero while the velocity measurements get higher and higher. Plotting these movements on your two- dimensional graph, you’d draw a 90 degree arc in the top left quadrant.
After the pendulum passes the zero point, velocity begins to decrease, until the pendulum finally comes to a stop, for an instant, at the rightmost extreme of its swing. On your graph that would be another 90 degree arc, connected to the first but in the top right quadrant, so you now have a semicircle. At this point the pendulum reverses course and heads back toward the left, which means its velocity, defined as the speed at which it travels right, is now a negative number. This arc would appear in the lower right quadrant, followed by one in the lower left.
Of course, atmospheric friction assures that as time passes, the pendulum will swing both more slowly and at lesser and lesser distances from the center. So both the position and the velocity measurements will gradually creep toward the center of the x and y axes, eventually converging at 0,0. And that means the graphic plot of the pendulum would show a neat and orderly spiral right into the middle of the graph.
Because the 0,0 point seems to pull the other points toward it, it is called an attractor. The simplicity of the spiral pattern suggests that a very simple rule governs the swing of the pendulum. So a computer, by projecting the state-space spiral forward in time a little bit while the pendulum was still swinging, would be able to predict where the pendulum would be a few moments in the future. It would be able to do this without being programmed to calculate the Newtonian laws of motion and the aerodynamics of the pendulum that actually cause the spiral pattern; it would simply extend the spiral.
It’s this type of state space that the Prediction Company’s network of computers creates, using for coordinates the prices of stocks, commodities, currencies, and indexes like the Standard & Poor’s daily average price for 500 stocks on the New York Stock Exchange (the data come zooming in over superfast modems). But the Prediction Company’s state-space maps involve not two variables but hundreds. Naturally, each extra variable means adding another axis to the graph, increasing its dimensions, but since state space is imaginary, there’s no limit to the number of dimensions a map can occupy--at least, not in theory.
There is, however, a practical upper limit on how many dimensions state space can have if it is to be of any use in forecasting. A computer could cope with the 500,000-dimension state space that would result from 500,000 variables, or degrees of freedom, in chaos-speak, but it would be difficult to find a statistically valid model within those dimensions. The pattern you’re looking for--a particular arrangement of all those billions of data points--might appear only very rarely.
And unfortunately, the state space that describes markets, Farmer says, is not very low-dimensional and therefore not very predictable. In fact, he says, the only things worse than financial data are things relating to radioactive decay or quantum mechanics. A really good understanding of the chaos of markets, he says, is years away.
But the Prediction Company’s researchers have not come away from their analysis empty-handed. While they have yet to achieve a comprehensive understanding of market chaos, Packard says, some of their pattern-finding techniques have turned up an occasional pocket of predictability--and if that pocket isn’t the grand theory that will win you a Nobel Prize, it’s still something that can make some people a lot of money. Looking at market chaos is like looking at a raging white-water river filled with wildly tossing waves and unpredictably swirling eddies. But suddenly, in one part of the river, you spot a familiar swirl of current, and for the next five or ten seconds you know the direction the water will move in that section of the river. The company’s computers, says Packard, look at a space of possible dimensions in the hundreds, trying to find the pockets of predictability where, for reasons that remain obscure, a state-space map of only a handful of dimensions is suddenly sufficient to predict what will happen next.
It’s possible, Farmer says, that a pocket of predictability reflects a simple system that’s being masked by a more complicated one, like a sprinkler spraying a lawn during a rainstorm. You might have a dynamical system with roughly 3 degrees of freedom, Farmer adds, superimposed on a dynamical system with 10 million degrees of freedom. And it might be that only 5 percent of what is going on in the world is described by the three-dimensional attractor. Nonetheless, if you can make a decent model of that three-dimensional attractor, you can make better predictions.
Occasionally predicting what will happen 5 percent of the time doesn’t sound like much, Farmer says, but remember that most of the other players behave as if the up-and-down dance of prices is random. If you look at the statistics on money managers, the majority of them perform roughly the equivalent of a coin toss, Farmer says. Straight fifty-fifty. So even if we’re wrong 45 percent of the time, we’re still doing much better. There doesn’t have to be much low-dimensionality there for us to get extraordinarily rich.
It also follows that Packard and Farmer should be finding unsuspected connections within the huge amount of data the financial world generates. Any structure in state space represents some kind of relationship, which is a signal that you can predict something, says Packard. For example, in the pendulum map, for any particular position only a certain velocity is possible (if this weren’t true, the points mapped by the two variables would be all over the space instead of aligned in a tidy spiral).
Hence Packard and Farmer’s work interests not only financial folk looking to make a killing but also economists, who are always on the lookout for causes and effects hidden in the noise of financial data.
But the previously hidden relationships that lurk in markets are not subjects Farmer and Packard are willing to discuss. We’ve found a few, Farmer says, digging into pasta at the Escalera. Precisely what comes out of their computers, like what goes in, is a trade secret. I ask if they’ve found anything mind-boggling, like confirmation of the folk wisdom that stock market prices rise and fall with hemlines.
Pretty hard to get good data for that one, Packard says.
Oh, I don’t know, says Farmer. Maybe we should check Women’s Wear Daily. There’s a short pause as the subject closes for good.
This kind of secrecy doesn’t come naturally to either Farmer or Packard. After they got their Ph.D.’s in physics, Farmer moved on to lead a team exploring chaos at the Los Alamos National Laboratory near here, and Packard headed into academia. But, says Farmer, working for the government got me pretty sick of signing forms and dealing with a lot of junk.’’ Though the roulette project had never made much money--shoe computers were hard to keep running in the 1970s--Farmer’s frustration made him nostalgic for what he calls the Manhattan Project atmosphere. And now, as then, he likes the idea of being free to do research without having to persuade other people to support it.
So in the spring of 1991 Farmer recruited Packard and McGill (a fellow Ph.D. who has launched several high-tech companies in California) and found five promising young chaos researchers. That fall he organized the Prediction Company to put his money where his math is.
At first Farmer and Packard considered boiling their insights down into a software package and selling it to all comers, and they agree that there’s something amiss about a society in which they can make a lot of money off the vagaries of a relationship between currencies. McGill had to struggle to talk the two out of organizing the Prediction Company as a collective, like the overlapping communes that pursued chaos theory and roulette profits at Santa Cruz.
At the sunny four-room house that is Prediction Company headquarters, blue jeans and T-shirts are the dress code. In the central room, staff meetings take place over a plain wood table bearing some technical tomes and several editions of the Quayle Quarterly, a publication devoted to the former vice president’s amusing antics. A whiteboard stretches the length of one wall; it’s covered with equations and drawings. (One drawing, of a swami and a crystal ball, was a proposed corporate logo that McGill vetoed.) Sprawled in a straight-backed chair, Farmer explains the basics of chaos as he absentmindedly pets Clara, the big black-and- white office dog.
From another room come the sounds of other Predictors fine-tuning their models and checking their work against the real-world data pouring in every second. Where is it now? someone asks. It was down 15 this morning, comes the reply. Down 15? Okay. It’ll end the day up 8 or more.
It’s a likely setting for that favorite plot of Westerns: a gang of young mavericks beating the powerful men in suits from back east. It’s easy to get caught up in the romance of it all. Unless you happen to be an economist.
Almost any system of predicting winners, including throwing darts at a board, can have a run of successes, economists say. But the physicists don’t apply the brake of social-science reasoning and ask, How could skirt lengths have anything to do with the Standard & Poor’s 500? They go where their model takes them, and that can be a recipe for worthless projections.
Without such an anchor in reality, suggests John Geanakoplos, an economics professor at Yale who also consults on finance for Kidder, Peabody, the chaos method can get lost in state space. Their approach in broad outline, he says, is like taking temperature and barometer readings in a whole bunch of cities all over the world, then checking the next day’s temperature in New York. You might notice that when overall world temperature and barometer are in some basic pattern, then in 29 cases out of 40 the temperature in New York goes up the next day. All that was tried with the weather 30 years ago and it didn’t work.
But at least with the weather you’d think there was some underlying physical process that might explain why you have a correlation. But what in the financial markets is the underlying cause for this relationship between variables?
The question of the real world cause for the state-space patterns obsesses economists, but Farmer and Packard have a ready answer: Who cares?
If I were an economist, I’d be fascinated by that question, Farmer says. If the point is to make money, though, the question doesn’t need to be answered.
After a moment he continues, It’s not clear whether the predictability we pick up on comes from so-called economic forces, supply and demand and so on, or from some kinds of laws of mass psychology.
It’s not clear that those two are mutually exclusive, either, adds Packard.
The two Predictors grew up together in nearby Silver City; they dispensed with small talk so long ago that their conversation now is as lean, quick, and substantive as a Platonic dialogue. Farmer quickly picks up the thread.
Yes, certainly they’re interacting with each other. But an economist might argue that there’s some kind of information-flow effect that’s pure and not psychologically involved. Others would say you can’t understand what markets do without taking into account how people perceive what they do. From our point of view, it doesn’t really matter.
But it does matter, economists say, because you can’t really take the measure of a complex pattern of human actions if you treat it as a bunch of numbers. Their objections echo those of scientists from other disciplines who haven’t been too taken with the way chaos specialists zip around from sunspots on Tuesday to heartbeats on Thursday to deutsche marks on the weekend. A lot of those chaos guys tend to think economists are dumb and don’t know anything, says James B. Ramsey, an econometrician at New York University. But we do have a greater sense of this amorphous mass of economic information.
More than professional pride is at issue. There’s a danger that what the sophisticated, chaos-based models will detect will be statistical flukes, not real relationships. It’s always possible to draw a line that connects all the points in your data set, says Blake LeBaron, an economics professor at the University of Wisconsin who has worked with chaos techniques for years. The question is, Does what you’ve found fit data that are ‘out of sample’? That, he says, is the test of whether you’re picking up a real phenomenon or just a statistical artifact of that period of time. For example, a lot of mumbo jumbo has been written over the years about the rule that predicts that American presidents elected in years that end in zero always die in office. This dictum held true for 1840, 1860, 1880, 1900, 1920, 1940, and 1960, before the survival of Ronald Reagan from 1980 to 1988 showed it up for what it really was--a simple coincidence.
It’s very easy to hallucinate on financial data, Farmer agrees. You’ve just got to go to paranoid extremes to avoid it. One safeguard the Prediction Company follows, he says, is to insist that models be developed without all the available data. We’ll withhold, say, all the data from 1982 on, keep it under lock and key, he says. Would-be predictors working on the problem develop a model for the years leading up to 1982. After they’ve agreed their work has come far enough along, he says, the staff meets around the big central table and comes to a consensus that the model’s predictions are ready to be tested against the post-1982 data. It’s one way of assuring that the model is not just a means of connecting all known dots, Farmer says.
Yet even that assurance doesn’t satisfy the financial skeptics. There is another problem they bring up: even if the Prediction Company’s patterns are genuine, they can’t be used to make any money. If everybody knew that on Tuesday the prices were going up, then on Monday night everybody would buy in, and the price would go up Monday night instead of Tuesday, says Geanakoplos. It’s a sort of self-negating prophecy. Ramsey agrees: Remember, a market is different from a physical system in that it’s acted upon by those within it, he says. If you find a pattern, you eliminate it by acting on it.
The trouble with that scenario, Farmer replies, is in the assumption that there’s no slack between the moment an idea comes into the market and the moment when everyone realizes it’s worth cashing in on. If I tell you I’ve got this great new idea, the prudent reaction would be to watch me for a while and see if my idea works, Farmer says. If it does, then during that waiting period I’m making money and you’re not. To that extent the market is not efficient.
Everyone at the company knows, says McGill, that their projections will have to get better and better as time passes, because if they do make money, the rest of the market will come chasing after them. High-tech businesses, he says, are used to that. In the Silicon Valley view of the world, all you ever buy is lead time, he says. It took us a while to get up here and really know what we’re doing, Farmer says. That’s one reason I’m not too worried about other people jumping in.
Instead, Farmer and Packard expect to confront a more subtle problem. Like anyone else trying to make money off the market, they are conducting an experiment whose guinea pigs include themselves. The poet Philip Larkin captured this hall-of-mirrors quality when he wrote: The circumstance we cause/In time gives rise to us. Once a client begins acting on Farmer and Packard’s advice, the Prediction Company will have to weigh as a factor in forecasting markets the effect of its own past predictions.
Basically, that’s kind of an open question, Packard says. It turns out that the data to answer that question are available in principle but most of them just dribble away onto the trading floor and are not saved. For example, if I try to make a transaction for a certain price, by the time my signal is read the market’s changed a bit in response to my trying to sell. Those data are not routinely kept, but we will definitely be keeping track of them.
A number of chaos researchers, in fact, propose that it might be possible to synchronize one’s buying and selling to match a market’s chaotic patterns closely enough to control it. The idea would be to nudge the market at exactly those points where the money would have the most impact, in the same way that a swing can be made to go higher and higher if it’s pushed at exactly the right frequency.
You inject a little chaos into the system, says Alfred Hübler, a physicist at the Center for Complex Systems Research at the University of Illinois, who is working on a model of how chaotic actions can be used to control different kinds of processes. It’s like driving a car; you constantly move the wheel a little, nudging the car a little this way, a little that. The idea is more than a simile, he says: state-space maps of bulldozer operators’ movements show chaotic patterns. His model is far too simple, he admits, but he’s already reached one conclusion: While you get certain benefits from predicting what a system will do, to get maximum benefit you have to try to control it.
In any event, the Prediction Company doesn’t expect to perturb the international markets right away, where several tens of millions of dollars is an extremely small account. For the next few years, the company is working exclusively for O’Connor & Associates, a Chicago trading firm affiliated with Swiss Bank Corporation. O’Connor operates in most of the country’s financial markets, from stocks to currencies to stock options. The Predictors won’t talk about the amount of capital they will be working with, but David Weinberger, general partner at O’Connor, says the investments will eventually be worth several hundred million dollars.
Yet if Hübler and other chaos researchers are right, then in the future it could also be within the Prediction Company’s powers not just to make money off currency and commodity traders’ future actions but also, by nudging and tugging the market in perfect sync with its natural patterns, to govern what those traders will do in the future.
It sounds like science fiction. In fact, it sounds like the science of mobs in Isaac Asimov’s Foundation novels, in which one Hari Seldon has predicted 30,000 years of future events with, as Asimov puts it, something of the accuracy that a lesser science could bring to the forecast of a rebound of a billiard ball. With this model, Asimov’s Seldon beneficently shapes the future to suit his purposes.
Farmer and Packard say they have nothing so grandiose in mind. They just plan to make a lot of money so they can be free to enjoy a life of scientific contemplation and discovery--which was exactly what they planned 20 years ago in the roulette project. That project united a type of science (the Newtonian physics of motion) with a kind of gambling. As Farmer and Packard see it, their current enterprise unites a different branch of physics (the state-space technique of understanding changing systems) with a different type of gambling.
We’re on the same kind of track we started off on then, just using different means, says Packard. In fact, just as in the roulette project, all the computers at the Prediction Company have nicknames. Back in the old Las Vegas days they were given affectionate names like Raymond and Renata. But the new machines all got their names from a certain science fiction epic.
Mine, says Packard, is called Seldon.
