There's an interesting briefing on oil in last week's Economist which takes a fairly different tone.
The section Stocks, bonds and barrels in http://www.economist.com/opinion/displaystory.cfm?story_id=11453090 argues against the suggestion that price increases are the result of speculation. They look at other commodity markets which have seen similar market behavior without similar price behavior, and look at the ability of futures trading to effect the market price. The financial analysis is interesting, but it basically boils down to this: the market is set by supply and demand of real barrels, and oil speculators are not hoarding barrels. The Economist's read on the furor about speculators is that this is political hay-making: politicians are desperately afraid of an election year with radically high oil prices, and they're searching frantically for a way to deflect blame. "Wall Street fat cats" are a great populist target for that sort of thing. It's worth noting that the CFTC, the government's own regulator for commodities markets, investigated oil speculation and found little reason for concern - it's the CFTC's chief economist who's getting cited most frequently in the "no it's not speculation" camp.
Their suggestion is that price increases have a more prosaic basis in supply and demand. Oil is a commodity with very little short-term demand elasticity (if the price goes up, people generally pay it rather than doing without) and very little short-term supply elasticity (it takes years to find/develop new fields). And it's a market which suffers various short-term supply shocks: much of the world's oil production is in countries where the rule of law is less than complete, and every time rebels blow up a pipeline there's a price movement. There's a good discussion of this at the end of the previously-linked article ... my knowledge of the differences between heavy and light crude and the various manufacturing impacts there was pretty minimal, and they have some interesting data.
There's some interesting commentary on the effects of fuel subsidies in http://www.economist.com/finance/displaystory.cfm?story_id=11453151. In China, for example, the price of petrol to a consumer hasn't risen since the start of the year (during which time the price in the US has increases 33%) because of government subsidies there. That helps reduce the elasticity of demand, and also apparently is causing Chinese oil firms to reduce their output.
Finally, there is a hopeful point underneath all of this. From their leader article, http://www.economist.com/opinion/displaystory.cfm?story_id=11454989:
"The 1970s showed how demand and supply, inelastic in the short run, eventually give rise to conservation and new production. When all those new fields are on-stream, when the SUVs have been sold and the boilers replaced, the downcycle will take hold. By then the slow-motion oil shock could have catalysed momentous change. Right now motorists have no substitute for oil. But it is no coincidence that car companies are suddenly accelerating their plans to sell electric hybrids that are far cheaper to run than petrol or diesel cars at these prices. The first two oil shocks banished oil from power generation. How fitting if the third finished the job and began to free transport from oil's century-long monopoly."
----------
Thanks for the links and I'd tend to trust E's analysis over CNN's, but of course there are many sides to this issue and some journalists (and bloggers) tend to get myopic to make their argument, obviously myself included. So probably the truth is a combination of many factors. I don't believe that speculation is a stronger force than traditional supply/demand, but we can't just dismiss it completely. When the oil companies themselves are reaping huge profits yet claiming the sky-high prices are a result of "supply and demand economics", I think that's a smoke screen that may even suggest the converse.
I agree that big shot Wall Street banks are an easy and maybe exaggerated culprit to blame in an election year. You are right that the CFTC experts dispute the effect of speculation on oil futures trading, but what else can they say? It's hard for them to collect meaningful data and draw any conclusions actually, since they lack access to so much of the trading that is offshore or OTC without public disclosure. If they admit that speculators are manipulating prices, then they effectively confess their incompetence and impotence as regulators. However, the oil traders themselves are saying the opposite. I know some people have incentive to dress up the truth based on their priorities, but why would the oil traders accuse the hedge funds of meddling if they were actually causing no harm? Oil traders are professionals who do this for a living, and they are saying the recent influx of Wall Street billions are making their lives more difficult. And yes, it's a two-way street as the article said. Trading activity doesn't just make the price go up, but high prices encourage more investment so they don't miss the boat if the price goes even higher.
But here's where the conventional supply/demand arguments get tricky. Oil's record rise from $70 to $130 over the past year is actually not reflected at the pump. And the price trickle-down delay from NYMEX to the pump is a month or less, so we should have seen a bigger change. If gasoline prices went up commensurate with oil's, gas would be at or over $6/gallon (taxes and surcharges included). Diesel is so much higher than regular gas because of recent refining capacity miscalculations, but even diesel hasn't reached $6 yet. Actually automobile fuel accounts for only 8% or so of US oil consumption (according to the link below), so even if the whole country drives 30% less (quite a challenge), we're reducing overall US fossil fuel consumption by a measley 2.5%. So much of our consumption is inelastically tied to industry, power/heating, and commercial transport, so it's much harder for them to conserve since they have business operations to run. Fishermen and truckers in Europe protested their diesel prices, but the airlines are getting screwed even worse. So sure global demand is going up, but only about 1M barrels/year or 1.25% according to The E graph, so why did prices rise nearly 100%? Well as you said, people get somewhat irrational when it comes to securing energy supplies, since a shortage is so catastrophic to an industrialized nation. So it's scary that traders are willing to buy up futures contracts like crazy, even at $130.
http://www.gravmag.com/oil.html
Another argument The E makes doesn't seem to hold water. They say that oil prices are tremendously sensitive to even small supply disruptions. That may be true in general, but it can't explain the recent price jump, because there have been no significant crises in oil producing zones besides the brief border standoff between Venezuela, Ecuador, and Colombia. And on the flipside, during the bad two years where we had Katrina, genocide in Darfur, the Israeli invasion of Lebanon, posturing for war with Iran, a plummetting dollar, a nuclear test in N Korea, and overt civil war in Iraq, oil prices did not surge as much as now!
Maybe we haven't reached Peak Oil yet, but it looks fairly close. According to my mom (30 years in the business), global output in most countries has plateaued or decreased since 2000, and only the Saudis and UAE have excess capacity, but it's like 3% or less. All the "low hanging fruit" and easy oil wells are maxed out. Now we have to drill deeper through tougher rock (in land or water) if we want to access new reserves. Russia and China have probably lied for years about their domestic production and stockpiling, so it's hard to tell what is going on over there, but it's not like Putin is sitting on 10% excess capacity. As The E said, refining and drilling overhead has risen over 70% since 2000, so it's more expensive to maintain historical production levels, but the obscene trading prices more than compensate. Therefore, nations and companies have little incentive to explore more, innovate, and increase supply to keep up with demand (the Saudis basically laughed in Bush's face when he begged them to boost output). Why would they risk out-pacing demand and making the price dip? They'll only act when their best wells start to run out, but there are plenty of new technologies to extract the last drops of oil from wells previously thought to be dry, if you have the skilled workforce and industrial capacity to implement them.
I just feel bad because these price jumps hurt the poor the most, as usual, and it's worse abroad. So if rich Westermers are profiting from that through trading, it's unacceptable. These funds just grow wealth for their clients yet add no value to the host industry; at least oil companies reinvest their windfall profits in R&D to improve future production. And as your last link showed, plenty of less developed oil-producing nations are forced to subsidize energy and even import refined fuels, or their people won't be able to afford it and they couldn't keep up with consumption. Plus they'll revolt against the government, as we saw in Iran when they decided to implement gas rations in preparation for a possible UN embargo/US war over their nuclear program. Most of the nations on that list could use a lot of improvement to their schools, health care, infrastructure, etc. Their huge oil profits could help, but much of it has to be siphoned off for subsidizing ever more expensive fuel (and military spending to defend against a possible Bush invasion). It's not their fault the prices are rising at NYMEX (they are at max capacity), but they're hurting too - just in other areas besides the explicit price at the pump.
-------
One thing to keep in mind is that the term "speculation" is being used in a rather loose fashion; most commentators include the creation of ETFs and Index Funds which deal exclusively in oil (and, implicitly, oil futures contracts) when attributing the rapid increases in prices & volatility to "speculation". I read the articles you referenced this last weekend... though the only one I shared via Google Reader was the article regarding inflation (the indisputable result of sustained increases in energy prices).
-------
Well I guess after the first Gulf War, there was more or less global political stability, drilling/pipeline technologies were improving, and nations like Venezuela, the former Soviet bloc, and OPEC as a whole were growing output, so the excess supply made the global price of oil plummet to below $20/barrel. Companies had to tighten their belts and even universities closed their petroleum engineering departments for lack of interest/funding. So I'm sure that oil producing nations/companies were secretly (or not) celebrating when Bush went into Iraq - the shot in the arm that the industry craved. BTW, Iraqi production has still not yet returned to late 1980s levels under Saddam (post Iran-Iraq War of course).
If I may address some of R & J's last points for a bit: well, I tend to think that commodities futures index funds are speculative by nature. They're betting on prices to move the way they want, and even if prices dip they can potentially sell short. Diversification aside, if there were more attractive investment options, they would take them. Investment in commodities funds has increase ten-fold in the last few years.
Actually that CNN piece did bring up a good point that no one wanted to comment on, the influence of the credit crunch. With the dollar weak and interest rates at stupid low levels, of course commodities look more attractive. Why have foodstuffs, precious metals, and energy all made astronomical gains during the same short time period? Asia is buying up more food, energy, and jewelry than ever before, but it can't be explained away by supply/demand, since the West still accounts for the majority of trading/consumption. I guess on the surface there is nothing wrong with investing in commodities futures; it can be a safe move. But I don't know much about this subject. However, when a fund buys up a million shares of Google or GE, they're not really hurting anyone, no matter how the stock moves. Of course huge sell-offs can hurt other investors, but buying up stocks doesn't seem to have many negative side-effects, apart from artificially inflating market cap. But when speculators are massively buying up commodities, they impact the people who are competing for those resources and may not be able to withstand large price spikes. How can people living on less than $5/day, and their struggling governments, afford 70% jumps in basic essentials that Wall Street traders casually toss around like baseballs? By playing that market, they are essentially "playing with people's lives", which is what I object to. Commodities are consummables that people depend on for survival, not just pieces of paper or blips on a broker's computer. It almost reminds me of The Grapes of Wrath during the Depression, where big farmers in CA would rather destroy their excess produce than give it to the starving Okies, in order to keep the market prices from falling. But all the while, the poor people's wrath was growing, and it's growing again now.
And like J said, shortages and rising prices are the only factors that can ultimately get stubborn economies to move towards better efficiency and alternative sources. However, this change won't come overnight, so what do the disadvantaged people do in the meantime who are dependent on prices not surpassing a certain threshold? And we're way beyond that threshold for some commodities already. Some people need to drive to work and feed families of 8. They are conserving because they have to. Some people don't have the ability or mobility to change jobs or some living habits though (socioeconomic factors are more to blame than personal choices in those cases). Of course well-to-do people, who may be the ones most able to conserve and still maintain a quality life, keep over-consuming because they can afford to. Without subsidies and donations, probably a billion people wouldn't be able to eat or travel with these current prices, some even as close to home as Tracy, CA. So yes, the high prices are affecting consumer habits and what companies are able to provide us. Whole Foods is losing business, airlines are laying off thousands and cutting routes, and GM is contemplating the future of their Hummer line. On the flipside, VC investment in green tech is growing rapidly, though not all American cities can benefit from growth in this new sector (actually the sad part is the growth is in areas that are already rich, like urban CA). So while the industrialized world is transitioning to a new way of living and working under the realities of $100 oil, what do the underclasses do in the meantime?
