Why a lot of published scientific research could be wrong

http://www.economist.com/blogs/graphicdetail/2013/10/daily-chart-2#

We've talked about this before, but this infographic presents it pretty cleverly.

----

First I would love to see a source on where he gets his numbers of false positives and false negatives.

He also oversimplifies the way experiments work.  For example with the higgs boson.  The experiments were run continuously until statistical significance  was seen.  That means the hypothesis was testing hundreds and hundreds of times.  The false negative/positive issue is resolved by virtue of the experimental setup.  

And on top of that, published work is not set on a pedestal because it is published.  Once it is published there is generally rigorous work to review and replicate the findings.  

So in short, he is wrong, and when he is right it doesn't matter because that is part of the process.  

-----

Those #s were just hypothetical based on a probability distrib, just to make his point. But sadly few scientists even consider type 1 and 2 errors, and just project confidence that their findings are bulletproof. In bio, when ppl try to reproduce expts, they can't about half the time, according to Science mag.

Yeah I think they are referring to distinct hypotheses, so a series of expts relating to the same hypoth. would probably be 1 data point only. There is also a risk associated with repeated "testing until significance", cuz the more measurements you make, the higher the chance you'll get some random signal and mistakenly use it to prove your hypoth. 

Actually you would be surprised how little rigor goes into the review process, I am sure S and A could comment. If the author is a big name with political clout, free pass. No one has the time or resources to verify and replicate work, it is the honor system. If another group wants to build on the research and don't get the same results, they may publish that, but may not to avoid controversy and be embarrassed if others don't buy it. 

But considering the incentive structure and human nature and the higher stakes of published results, it's a bad combo. Also the journal wants to publish big sexy results to make more money so there is COI.

----

re: "mistakenly use it to prove your hypoth" I think you don't understand what I'm describing.  The measurements means more averaging as I'm describing it.  And more averaging means LESS chance of random signal causing an error.  It is just a method for eliminating "noise" whatever that noise may be.  For the higgs boson it was a very specific energy level measured from the decay.  Every time they took another measurement, if the higgs existed the energy signature would be reinforced and other sources of energy negated, or the opposite if it did not exist.  And with some fancy math they get a confidence level based on number of trials and blah blah.  So not all experiments are susceptible in the same way.

Re COI and big names.  Yea.  Reinheart and Roghoff (sp?) is an example where more weight was given to big names.  But it is also an example of the fact that people do look at this stuff.  Though you could argue the damage was done by the time we figured out their conclusions weren't great.

In any case I'm not arguing that false positivies don't exist in academic journals.  I am arguing that this particular guy's math that proves it is false at worst or misleading at best.

----

I see, thanks M. In that case, the higgs boson expt was doing its own internal QA and replicating the experiment many times (something very lacking in bio and business, where many expts are underpowered). So if the variance is small and there is a strong signal from most of the trials, then that is a very convincing result.

At work we were struggling with a related issue. We usually don't approve changes to the website unless they pass a lot of statistical and sanity checks. But due to probability we know that we are likely rejecting some features that were truly positive (but we couldn't detect it, by chance or by poor design), and we are approving some features that are in fact neutral or even negative. That is the scary possibility. Sometime our approval criterion is "do no harm", as in if all the key metrics look to be within noise, and we can't reject the null hypothesis, then it's OK. But there is a slim chance that we are approving stuff that is actually very harmful. And if the company is approving 100's of features a year, it's likely some of them are harmful but mislabeled as positive or neutral. Though for mature businesses, where you are squeezing out basis points on the margin, the risk is probably not terrible. I guess there is a balancing act between scientific rigor and business expediency/strategy. But at least in e-commerce, an error isn't likely going to cost lives (unless it's Apple maps LOL). But for consumables, finance, or public works, it could be really bad. And they say we don't need regulation? :)

----

Yeah, it's too bad that knowledge of this sort of thing seems to only reinforce the worst of carpetbagging instincts: get in early, publish as much as you can whether of integrity or not, and then do everything in your power to ensure that no one who comes after you could possibly do it as its supposed to be done in the first place.

Maybe it's a cultural thing - like, perpetuating a hierarchical ponzi scheme requires the smartest of people to buy in to the stupidest of ideas (like blinders, narrowness, and petty infighting)...

PS:  Technology will (ultimately) solve all these problems as democratized access to information will lead to market forces being brought to bear on the most indefensible of regressive attitudes and mentalities (most of which are sustained on the basis of petty economics).

----

We were talking about this, and here were our general follow-up thoughts:

- The sci. method and the peer-review process are hundreds of years old and developed during times when methods were not so empirical & specialized, and there wasn't such $$ at stake for discoveries (for people like Euler and Carnot, I think reputation, knowledge, and prestige might have trumped the incentive to cheat, be negligent, and cut corners)

- Obviously we live in a new age now, and as C said the democratizing effects of technology could mitigate the problem (i.e. a grad student found Rogoff's errors when big time editorial staff and top peers didn't), but there are limits to that in hard science because of the cost and specialization of certain expts (though if you need a certain highly skilled postdoc to do a test a particular nuanced way to get a positive result, probably your result is not that robust)

- At least for bio, we thought to develop an independent, confidential auditing lab within the NIH to verify all published results. Journals and authors have to pay a fee to support it, and an article that passes the audit gets a certification that gives it more clout. The author's lab has to send all the materials over to the NIH (or let them use their equipment), with instructions, and the expert NIH staff have to reproduce the result within reasonable variance (they sign NDAs and no-competes so the author has no fear of being scooped). If they can't reproduce, then the paper can still be published at the journal's discretion, but without certification.

----

I would think an algorithm could be developed (or maybe already has?) that can blindly take in the statistical data from an experiment and verify the conclusions.  Rogoff was a case of bad math not bad data right?  That removes the requirement to reproduce experiments and leaves you with only the first two cases of lies, damned lies, and statistics.

The NIH thing is interesting but there is still money involved which is always problematic.  The NIH is not free from politics since that is its major source of funding.  And there are many many public institutions that don't share the data found through public dollar funded experiments already.  The democratization of technology doesn't help when all the research is behind a pay wall.

----

My comments are enclosed (in-line)...  Also, in case anyone didn't see the full article:  http://www.economist.com/news/briefing/21588057-scientists-think-science-self-correcting-alarming-degree-it-not-trouble

I would think an algorithm could be developed (or maybe already has?) that can blindly take in the statistical data from an experiment and verify the conclusions.  Rogoff was a case of bad math not bad data right?  That removes the requirement to reproduce experiments and leaves you with only the first two cases of lies, damned lies, and statistics. 

It's a good idea; one would think some of that would be built-in to the tools the scientists and researchers are using but clearly there is progress to be made...

The NIH thing is interesting but there is still money involved which is always problematic.  The NIH is not free from politics since that is its major source of funding.  

True... but federal institutions tend to be bulwarks of trust (certainly to a greater extent than a tobacco company or The Koch Brothers)...

And there are many many public institutions that don't share the data found through public dollar funded experiments already.  

This is also true... but changing.  Even institutions which are known for being benefactors of the public (e.g. U.C. Berkeley) are getting more formal about sharing research and ensuring access... so the change is bound to ripple through to other public institutions too...

The democratization of technology doesn't help when all the research is behind a pay wall.

Yeah, that is slow to change - but perhaps the most hopeful front.  The economics of information mean is such that librarians and libraries are being confronted by these questions so the clock is already ticking: if our engineering library at U.C. Berkeley is getting rid of its books for the value the physical space has to the college then its probably a matter of time until someone rationalizes our giving research to private journals so that they can rip off the campus with subscription fees that (over time) do not seem to be worth more than the promise of a new student.

-----

For Rogoff, I think it was a variety of issues, but some of it was excel formulas pointing to the wrong cells, as well as "improper" suppression of some data points. So that is mostly human error and poor judgment, which is hard for an algo to rectify.

An algo would be cool (like how the IRS algos randomly scan over people's returns and flag errors/possible issues), but then the scientific data has to be formatted and structured according to some standards so the algo can read it properly. And since each article kind of measures different things and employs different significance tests, it could be tricky. Hell, they can't even get healthcare.gov right using 50 contractors (maybe that is their problem!).

There are some free or lower cost internet journals out there... hopefully they can give the heavies a run for their money some day. But due to the prestige factor, no big names want to "slum it" with online journals. But I agree that those journals are ripping off institutions to give then "novel research" that is at least 30% useless and another 30% incorrect.

Jokes aside, we piss and moan about how bad public institutions are (and they are of course not free of corruption either... see the MMS), but as C said, they are our last resort against a totally for-profit world. We need to strengthen the public institutions so that they offer a legit alternative to the for-profits, and then with customer choice it will compel the for-profits to clean up and stop shafting us so much. I guess that's why the health industrial complex fought so hard against single payer, which is BY FAR the best feasible health system in the Western world, warts and all. But with all the dysfunction in Washington, furloughs, and a reduction in public worker compensation/respect, it makes it less likely that our best and brightest will want to go into public service and stay there long enough to make an impact. 

How the Dulles brothers made Kissinger and Cheney look like peaceniks

http://www.npr.org/2013/10/16/234752747/meet-the-brothers-who-shaped-u-s-policy-inside-and-out

We don't hear much about these two (apart from the DC airport name), but their beliefs and actions literally shaped the Cold War, US global ambitions, and we are still dealing with the negative repercussions today.

Ike appointed John F. Dulles as his Sec. State and Allen Dulles as the CIA head. They were both corporate lawyers who represented, among others, United Fruit and the financiers of the Anglo-Persian Oil Company. Enough said. Ike basically gave them carte blanche authority to make enemies and "intervene" in any foreign country they saw fit, and they didn't hesitate: Guatemala, Iran, Congo, Cuba, Indonesia, and Vietnam ... basically our greatest hits. They were especially adamant about Iran, because under no circumstances could they tolerate a leader nationalizing oil reserves for the people, and renegotiating terms on foreign companies. Imagine the shockwaves to big business, and what message that would send to other countries.

Unlike Kennedy's "team of rivals" model, it was very dangerous for Ike to appoint two like-minded, corporate shill, biological brothers into the complementary (and sometime necessarily adversarial) positions of State and CIA. One side shapes foreign policy and strategy, and the other carries it out covertly. Or one side tries to make peace, while the other sows the seeds of war. It's playing with fire. Now they were basically one org, without any sanity checks or devil's advocate to say, "What happens after we overthrow Lumumba? (answer: civil war and genocide for decades)," or "Maybe we can't win in Vietnam."  I guess the Dulles' and their Cold War were enough to make General Ike warn the nation about the military-industrial complex upon his retirement.

The Dulleses believed that the US, its market economy, and its multinational corporations were exceptional and entitled to dominate the globe (because we were the good guys). Anyone who opposed these interests (regardless of the reason, like the crazy notions of self determination and liberty) were automatically evil monsters. It's OK for George Washington, but not for Third World leaders (esp. when they are sitting on natural resources). Clearly enemy #1 was communism, and by extension populist nationalism, because it went against the model of private capital might makes right, and neo-colonialism/imperialism. Any attempt to regulate or take power/profit away from US companies was as much of a threat to US security as nuclear missiles.

Apart from the direct and indirect violence & collateral damage (5M total deaths at least), militancy/extremism, and harm to our global reputation in most of the world, here is the Dulles' biggest legacy and curse on humanity: a pervasive culture of rejecting peace talks and the undermining of democracy.

After Stalin died, the USSR wanted to improve relations with the West. It was a great opportunity, but Dulles was the only major foreign leader to reject this overture (and pressured our allies to do the same), because he thought the Russians were too evil to talk to, and preferred that our conflict continue. Later, both sides' nuclear brinksmanship almost ended the world. During the North-South Vietnam diplomatic talks after the French withdrawal, most European powers felt that the party was over and they should learn to live with Ho Chi Minh. Dulles was the only one who felt that Ho could be beaten militarily if we stuck it out, and he persuaded Ike and Kennedy. We know how that turned out. But you can see this US hubris and misguided thinking carrying over to our negotiations (or lack theoreof) with Cuba, Iran, and North Korea over the years. We have certain beliefs about other people, and our stubbornness ends up prolonging/worsening conflicts and missing out on opportunities.

Why was the US so successful in overthrowing the democratic regimes in Iran, Guatemala, and Congo in the '50s? The author argues that it was because those societies (and their idealistic, civilian, center-left leaders - a.k.a. died-in-the-wool commies) were fairly open and democratic, thereby allowing foreign agitators access to covertly undermine the gov't and eventually launch a coup d'etat (with a pro-US dictator waiting in the wings). Later revolutionaries learned from those cases, and ruled more like repressive tyrants in order to defend against a possible US overthrow. Cuba, North Vietnam, the Islamic Republic of Iran, North Korea, etc. all became more paranoid and less democratic after the Dulles' time. It could just be coincidence, but at least in the case of Cuba, Che and Castro were quoted that they would not repeat the mistakes of Arbenz in Guatemala. They would repress and spy on their own people to make sure the CIA wasn't gaining a foothold.

So even though the US portrayed itself as the global example and defender of democracy, in fact its foreign policy under the Dulleses was terribly undemocratic, and motivated many others around the world to choose brutality over democracy too (tragically, during a time of global change that we will never get back, when we really could have forged tremendous social progress and peace). And the human race may never fully recover from the legacies of those dark times. To me, that is a much greater failure, security threat, and shame on the US than letting a few undeveloped countries elect a socialist leader, or try to nationalize their resources.  

With friends like these...

http://www.npr.org/2013/09/25/225858836/u-s-soldier-crusading-for-afghan-interpreter-who-saved-his-life

Maybe you have heard about this story. The US sometime has the rep for "burning" foreign allies during the Cold War when it made political sense. But in the asymmetrical War on Terror, we have the resource advantage and there's no reason to turn our backs on the people who risked everything to help us - because they actually believed that we would make their countries better.

We're not talking about hundreds of thousands of refugees like Syria here (though it would be principled to do more so poor Jordan and Turkey don't have to bear the whole burden). I doubt the total is higher than a thousand Afghan and Iraqi translators and their families. In some cases, they directly saved American lives, which is more than most US citizens can say about patriotism.

A few years ago, Congress passed a law to grant visas to these folks because they will be in mortal danger once US forces leave. This isn't exaggerated speculation, as many of them have already received death threats (it's no secret that they are "infidel sympathizers" as they stood with US forces in public view). And we know that Taliban and AQ in Iraq types know how to make people disappear in the night.

But years later, only 12% of approved Afghans and 22% of Iraqis have made it to the safety of US shores, mostly because of DHS and State Dept immigration BS that these folks could be potential security risks. Yes it is true that there has been plenty of "green on blue" violence in Afghanistan where infiltrators masked as allies have killed UN forces. But those were fresh recruits in a warzone without much vetting. It's another story for an infiltrator to masquerade as an embedded translator, risk life and limb for years, and then travel to America just hurt us. If any of you know immigrants, you have probably heard stories about what a pain in the ass the background check is. Just imagine how it is for people from unstable Muslim nations. In some cases, the Taliban found out that some translators were getting out, so they may have made false "anonymous tips" to the US embassy that those people were actually threats. And all of a sudden their visas are on hold again.

The problem with bureaucrats is they fixate on the letter of the law rather than the intent and the bigger picture. I am sure none of them wants to be responsible for giving a terrorist a visa, but is that worth condemning our friends? At least transport them to Gitmo or a midwest jail until they are fully cleared. But we can't just leave them behind. And we wonder why they hate us?

Similar to the story of Sydney Shanberg, one soldier (Capt. Zeller) has tried tirelessly to get his translator Shinwari out of Afghanistan, after that man saved his life in the field and they became "brothers". He went to the media, wrote Congressmen, and just when it looked like it was going to work out, more visa delays. Now Zeller has lost contact with Shinwari, and he hopes he just went into hiding with his family rather than something worse.