By John Kemp
LONDON, March 27 (Reuters) - Fundamental changes in supply and demand account for less than a third of short-term movements in commodity futures prices, according to an authoritative new study by researchers at UNCTAD and the Swiss Federal Institute of Technology.
Reflexive trading, when prices respond to past price changes rather than new information about fundamentals, now accounts for 60 to 70 percent of price moves in the main futures contracts, up from less than 40 percent before 2005.
The authors examined price moves in Brent oil, U.S. crude, soybeans, sugar, corn and wheat futures using an elaborate procedure designed to separate movements related to the arrival of new information from those in which one price change begets another.
"We find an overall increase in the level of short-term endogeneity since the mid-2000s to October 2012, with a typical value nowadays around 0.6-0.7, implying that at least 60 to 70 percent of commodity price changes are now due to self-generated activities rather than novel information", according to Vladimir Filimonov, David Bicchetti, Nicolas Maystre and Didier Sornette in a paper published on March 21.
The paper marks a breakthrough in research into the formation of commodity futures prices. It brings research on commodity markets into the academic mainstream, integrating behavioural and fundamental approaches, and follows pioneering work in other asset classes by George Soros ("The Alchemy of Finance" 1987), Sornette ("Why Stock Markets Crash" 2003) and Robert Shiller ("Irrational Exuberance" 2009).
If their paper is correct, commodity futures markets may actually have become less efficient at discovering prices in recent years, not more, as a result of high-frequency trading and other aspects of financialisation.
Sornette is one of the world's foremost authorities on price formation and market microstructure, so the findings cannot easily be dismissed by researchers and policymakers who insist markets are efficient and all changes reflect fundamentals.
The authors put forward various possible explanations for the rising reflexivity in commodity futures markets including the increased prevalence of algorithmic and high-frequency trading; more behavioural trading and herding; hedging; margin calls and stop-loss orders.
Since hedging, margin calls and stop-loss orders have been a constant feature of commodity markets, the most likely explanation for rising reflexivity lies with the increase in computer-driven and behavioural trading.
The authors present some startling numbers to illustrate the rise of algorithmic and high-frequency trades in commodities.
Using the Thomson Reuters Tick History (TRTH) database , which contains 2 petabytes of tick-by-tick data on more than 45 million unique instruments across more than 400 exchanges, the authors show the big increase in the number of trades since 2005 has been accompanied by a sharp fall in the number of contracts in each transaction.
There are more transactions, but each one is much smaller than before. The average volume per transaction fell from between five and 40 contracts in 2005, depending on the commodity, to less than three per transaction in 2012 for all.
By 2009, the median volume for all commodities was just one contract per transaction. In other words, more than half of all transactions consisted of just one lot.
Larger trades are becoming increasingly rare. In 2005, 10 percent of all Brent trades were for at least 13 contracts. By 2011, the top 10 percent of Brent trades were for just two contracts or more.
The rising flurry of small transactions is mirrored across markets and is characteristic of algorithmic and high-frequency traders.
To distinguish between price changes that are initiated by fundamentals versus other changes, the authors employ a Hawkes self-excited conditional Poisson model, which has become the gold standard for modelling discontinuous financial data.
It can be likened to a family tree. Some price moves are initiated by external events such as new data. These primary price moves can be described as zero-order, mother or immigrant events. Other price moves are triggered by previous ones. These can described as first-order, second-order etc, daughter or descendant events.
"Every zero order (mother) can trigger one or more first-order events (daughters), each of whom becoming mother-event in turn can trigger several daughters (second-order events or granddaughters) and so on over many generations," the authors explain.
"First-order, second-order and higher-order events form the cluster of aftershocks of the main event as a result of the self-excited (endogenous) generating mechanism in the system."
The researchers compared the number of zero-order events with those that were apparently first-order or higher descendants to calculate a branching ratio or reflexivity index. The higher the index, the greater the proportion of descendant trades.
For a reflexivity index of 0.8, which is where Brent peaked in late 2008 and 2009, there were five times as many price moves as the number that would exist if each were reacting to fundamentals alone. Four out of five trades appeared to react to previous trades rather than be triggered by external fundamental news.
The more first- and higher-order price movements there are compared with zero-order initiating events, the longer it takes the market to find a new equilibrium, and the less efficiently it operates.
"It takes more and more time for the system to adjust to new immigrants due to the larger and larger number of triggered descendants and the longer and longer sequences of generations. This means that the convergence process to any true price becomes longer and longer, in other words less and less efficient," the authors write.
"Rather than agreeing rapidly on the 'correct' price after the arrival of some unanticipated news, the traders take longer, not knowing on what price to settle."
Strictly speaking, the research is applicable only to price movements over short intervals, up to 10 minutes or so. "Our reflexivity indices are not particularly designed to capture longer-term herding mechanisms, which are responsible for bubble formation on times scales of months to years," the authors admit.
However, they go on to observe, "we surprisingly find that the mechanisms working at longer-term scales sometimes seem to cascade down to the shorter intervals on which we compute our indices.
"The most remarkable result obtained from the calibration of the branching ratio is its very large increase during the period when oil prices started to accelerate (2007 and 2008). The fact that our methodology identifies a growing reflexivity during the ascent of the price and, even more so, during its collapse, is particularly interesting in view of other analyses that documented strong evidence for the existence of a bubble during that period."
Brent's branching index climbed to almost 0.8 in the run-up to the price peak in July 2008 and rose even further as prices collapsed in the final months of the year and into 2009.
Branching indexes for Brent and other commodities show other localised peaks during periods when cross-asset risk-on/risk-off trading has been especially pronounced - periods when herding behaviour has seemed to be maximised and commodity-specific fundamentals have seemed to be relegated to a secondary role.
The high levels of reflexivity identified in the study "highlights the failures of the efficient markets hypothesis and provides evidence that price dynamics are partly driven by positive feedback mechanisms".
In layman's terms, if their econometrics is correct, Filimonov, Bicchetti, Maystre and Sornette have proved what traders have always instinctively understood: prices are driven by a combination of fundamentals and behavioural factors, in proportions which vary over time.
For all that this seems like common sense, this view has proved remarkably controversial, with the debate polarised between market fundamentalists and those who detect the hand of speculation behind every rise in commodity prices.
Perhaps this study will move the debate forward towards a more productive synthesis.
The paper on "Quantification of the High Level Endogeneity and of Structural Regime Shifts in Commodity Markets" can be downloaded atDespite the technical mathematics, the authors explain their tests and results clearly, and the paper is worth reading in full.