Why a lot of published scientific research could be wrong

http://www.economist.com/blogs/graphicdetail/2013/10/daily-chart-2#

We've talked about this before, but this infographic presents it pretty cleverly.

----

First I would love to see a source on where he gets his numbers of false positives and false negatives.

He also oversimplifies the way experiments work.  For example with the higgs boson.  The experiments were run continuously until statistical significance  was seen.  That means the hypothesis was testing hundreds and hundreds of times.  The false negative/positive issue is resolved by virtue of the experimental setup.  

And on top of that, published work is not set on a pedestal because it is published.  Once it is published there is generally rigorous work to review and replicate the findings.  

So in short, he is wrong, and when he is right it doesn't matter because that is part of the process.  

-----

Those #s were just hypothetical based on a probability distrib, just to make his point. But sadly few scientists even consider type 1 and 2 errors, and just project confidence that their findings are bulletproof. In bio, when ppl try to reproduce expts, they can't about half the time, according to Science mag.

Yeah I think they are referring to distinct hypotheses, so a series of expts relating to the same hypoth. would probably be 1 data point only. There is also a risk associated with repeated "testing until significance", cuz the more measurements you make, the higher the chance you'll get some random signal and mistakenly use it to prove your hypoth. 

Actually you would be surprised how little rigor goes into the review process, I am sure S and A could comment. If the author is a big name with political clout, free pass. No one has the time or resources to verify and replicate work, it is the honor system. If another group wants to build on the research and don't get the same results, they may publish that, but may not to avoid controversy and be embarrassed if others don't buy it. 

But considering the incentive structure and human nature and the higher stakes of published results, it's a bad combo. Also the journal wants to publish big sexy results to make more money so there is COI.

----

re: "mistakenly use it to prove your hypoth" I think you don't understand what I'm describing.  The measurements means more averaging as I'm describing it.  And more averaging means LESS chance of random signal causing an error.  It is just a method for eliminating "noise" whatever that noise may be.  For the higgs boson it was a very specific energy level measured from the decay.  Every time they took another measurement, if the higgs existed the energy signature would be reinforced and other sources of energy negated, or the opposite if it did not exist.  And with some fancy math they get a confidence level based on number of trials and blah blah.  So not all experiments are susceptible in the same way.

Re COI and big names.  Yea.  Reinheart and Roghoff (sp?) is an example where more weight was given to big names.  But it is also an example of the fact that people do look at this stuff.  Though you could argue the damage was done by the time we figured out their conclusions weren't great.

In any case I'm not arguing that false positivies don't exist in academic journals.  I am arguing that this particular guy's math that proves it is false at worst or misleading at best.

----

I see, thanks M. In that case, the higgs boson expt was doing its own internal QA and replicating the experiment many times (something very lacking in bio and business, where many expts are underpowered). So if the variance is small and there is a strong signal from most of the trials, then that is a very convincing result.

At work we were struggling with a related issue. We usually don't approve changes to the website unless they pass a lot of statistical and sanity checks. But due to probability we know that we are likely rejecting some features that were truly positive (but we couldn't detect it, by chance or by poor design), and we are approving some features that are in fact neutral or even negative. That is the scary possibility. Sometime our approval criterion is "do no harm", as in if all the key metrics look to be within noise, and we can't reject the null hypothesis, then it's OK. But there is a slim chance that we are approving stuff that is actually very harmful. And if the company is approving 100's of features a year, it's likely some of them are harmful but mislabeled as positive or neutral. Though for mature businesses, where you are squeezing out basis points on the margin, the risk is probably not terrible. I guess there is a balancing act between scientific rigor and business expediency/strategy. But at least in e-commerce, an error isn't likely going to cost lives (unless it's Apple maps LOL). But for consumables, finance, or public works, it could be really bad. And they say we don't need regulation? :)

----

Yeah, it's too bad that knowledge of this sort of thing seems to only reinforce the worst of carpetbagging instincts: get in early, publish as much as you can whether of integrity or not, and then do everything in your power to ensure that no one who comes after you could possibly do it as its supposed to be done in the first place.

Maybe it's a cultural thing - like, perpetuating a hierarchical ponzi scheme requires the smartest of people to buy in to the stupidest of ideas (like blinders, narrowness, and petty infighting)...

PS:  Technology will (ultimately) solve all these problems as democratized access to information will lead to market forces being brought to bear on the most indefensible of regressive attitudes and mentalities (most of which are sustained on the basis of petty economics).

----

We were talking about this, and here were our general follow-up thoughts:

- The sci. method and the peer-review process are hundreds of years old and developed during times when methods were not so empirical & specialized, and there wasn't such $$ at stake for discoveries (for people like Euler and Carnot, I think reputation, knowledge, and prestige might have trumped the incentive to cheat, be negligent, and cut corners)

- Obviously we live in a new age now, and as C said the democratizing effects of technology could mitigate the problem (i.e. a grad student found Rogoff's errors when big time editorial staff and top peers didn't), but there are limits to that in hard science because of the cost and specialization of certain expts (though if you need a certain highly skilled postdoc to do a test a particular nuanced way to get a positive result, probably your result is not that robust)

- At least for bio, we thought to develop an independent, confidential auditing lab within the NIH to verify all published results. Journals and authors have to pay a fee to support it, and an article that passes the audit gets a certification that gives it more clout. The author's lab has to send all the materials over to the NIH (or let them use their equipment), with instructions, and the expert NIH staff have to reproduce the result within reasonable variance (they sign NDAs and no-competes so the author has no fear of being scooped). If they can't reproduce, then the paper can still be published at the journal's discretion, but without certification.

----

I would think an algorithm could be developed (or maybe already has?) that can blindly take in the statistical data from an experiment and verify the conclusions.  Rogoff was a case of bad math not bad data right?  That removes the requirement to reproduce experiments and leaves you with only the first two cases of lies, damned lies, and statistics.

The NIH thing is interesting but there is still money involved which is always problematic.  The NIH is not free from politics since that is its major source of funding.  And there are many many public institutions that don't share the data found through public dollar funded experiments already.  The democratization of technology doesn't help when all the research is behind a pay wall.

----

My comments are enclosed (in-line)...  Also, in case anyone didn't see the full article:  http://www.economist.com/news/briefing/21588057-scientists-think-science-self-correcting-alarming-degree-it-not-trouble

I would think an algorithm could be developed (or maybe already has?) that can blindly take in the statistical data from an experiment and verify the conclusions.  Rogoff was a case of bad math not bad data right?  That removes the requirement to reproduce experiments and leaves you with only the first two cases of lies, damned lies, and statistics. 

It's a good idea; one would think some of that would be built-in to the tools the scientists and researchers are using but clearly there is progress to be made...

The NIH thing is interesting but there is still money involved which is always problematic.  The NIH is not free from politics since that is its major source of funding.  

True... but federal institutions tend to be bulwarks of trust (certainly to a greater extent than a tobacco company or The Koch Brothers)...

And there are many many public institutions that don't share the data found through public dollar funded experiments already.  

This is also true... but changing.  Even institutions which are known for being benefactors of the public (e.g. U.C. Berkeley) are getting more formal about sharing research and ensuring access... so the change is bound to ripple through to other public institutions too...

The democratization of technology doesn't help when all the research is behind a pay wall.

Yeah, that is slow to change - but perhaps the most hopeful front.  The economics of information mean is such that librarians and libraries are being confronted by these questions so the clock is already ticking: if our engineering library at U.C. Berkeley is getting rid of its books for the value the physical space has to the college then its probably a matter of time until someone rationalizes our giving research to private journals so that they can rip off the campus with subscription fees that (over time) do not seem to be worth more than the promise of a new student.

-----

For Rogoff, I think it was a variety of issues, but some of it was excel formulas pointing to the wrong cells, as well as "improper" suppression of some data points. So that is mostly human error and poor judgment, which is hard for an algo to rectify.

An algo would be cool (like how the IRS algos randomly scan over people's returns and flag errors/possible issues), but then the scientific data has to be formatted and structured according to some standards so the algo can read it properly. And since each article kind of measures different things and employs different significance tests, it could be tricky. Hell, they can't even get healthcare.gov right using 50 contractors (maybe that is their problem!).

There are some free or lower cost internet journals out there... hopefully they can give the heavies a run for their money some day. But due to the prestige factor, no big names want to "slum it" with online journals. But I agree that those journals are ripping off institutions to give then "novel research" that is at least 30% useless and another 30% incorrect.

Jokes aside, we piss and moan about how bad public institutions are (and they are of course not free of corruption either... see the MMS), but as C said, they are our last resort against a totally for-profit world. We need to strengthen the public institutions so that they offer a legit alternative to the for-profits, and then with customer choice it will compel the for-profits to clean up and stop shafting us so much. I guess that's why the health industrial complex fought so hard against single payer, which is BY FAR the best feasible health system in the Western world, warts and all. But with all the dysfunction in Washington, furloughs, and a reduction in public worker compensation/respect, it makes it less likely that our best and brightest will want to go into public service and stay there long enough to make an impact. 

Scientific studies suggest money makes you an a-hole

http://www.pbs.org/newshour/bb/business/jan-june13/makingsense_06-21.html
http://www.nytimes.com/roomfordebate/2012/03/15/does-morality-have-a-place-on-wall-street/greed-on-wall-street-prevents-good-from-happening
http://www.pbs.org/newshour/businessdesk/2013/06/why-those-who-feel-they-have-less-give-more.html

Psychologists from Berkeley (of course) conducted a series of experiments on how wealth differences (real or perceived) affect social behavior and ethical choices.

First, they conducted a simple observation of pedestrian and driver behavior at an intersection. They found that 90% of drivers do yield for pedestrians (as law and safety demands), but of those who blew through, a significantly higher % of those vehicles were classified as "luxury". Also luxury cars cut off other drivers at a higher rate. A lot of things could be potentially wrong with such a study, but the result seems logical - spending a lot on a car with attached marketing messages like: you have arrived, you are Mr. Badass, you deserve it, you're better - may motivate the driver to feel that he/she (mostly he probably) doesn't have to stop for "peasants" and is entitled to get to the destination unhindered. Or maybe their social environment is very status conscious and competitive, so driving is also an opportunity to demonstrate their perceived superiority, or they may feel the need to drive like a jerk to fit the mold of success (even if they actually want to drive more politely). Of course this doesn't apply to all luxury drivers, and there are plenty of jerks driving clunkers.

You know how politicians take candy from babies? Rich people do too, literally. Controlling for various other attributes, they had participants fill out a phony form in a private room. There was a small jar of candy on the table, which they were told was for kids for the next study later in the day, but they could have some if they wanted. Participants labeled as "rich" (maybe above some cutoff for self-identified household income, reference their PNAS paper for details) took 2X more candy than non-rich. Even though they were permitted to, typical conscientious thinking might be: I can buy my own candy later, the kids will enjoy it more, I don't want to look like a pig, etc. to motivate them to refrain. But maybe rich people are used to getting better service and favors from others (with less regard for sharing & limits), so they are conditioned to take when offered without considering others.

You have probably heard about studies that show CEOs are more likely to cheat on games (even with no rewards) by self-reporting higher scores (with no outside verification). The same applies to rich people. In a computer-simulated die toss game, the richer participants in their study were more likely to cheat and report scores that were mathematically possible, but actually impossible due to the hidden logic of the sim. The narrative seems reasonable: rich people are competitive and may lie to increase their chances of winning if they can get away with it. Their personal payoffs for winning unethically outweigh the possible consequences of getting caught, and that calculus may not apply as much to the sub-rich, where consequences dominate (jail, getting fired, etc.).

Lastly, what happens when non-rich people "feel rich" and vice versa? The researchers had people play a rigged game of Monopoly, where a randomly decided player (of various personal wealth levels) got to be Goldman Sachs (more starting $, more die rolls, got to have the car playing piece). It was about mathematically impossible to lose from that position. During the game, the Goldman player tended to exhibit more bossy, dominant behaviors. When asked how they felt about the game results, the Goldman player was more likely to take personal credit for their success rather than acknowledge their randomly-assigned advantages. We know that manifests itself in real life, as from our previous discussions, the rich may feel OK with tax evasion because they feel they already paid enough, and they deserve to keep more of it due to their cunning/superiority. Conversely in the Monopoly game, those in the position of the disadvantaged player tended to exhibit more compassionate, gracious behavior (despite their actual social class).

Extrapolating these results out to the real world, I suppose we shouldn't be surprised when the most rich and powerful among us behave badly and are caught in greed/ethics scandals. And for the rich who behave well and can still be generous to others (like Buffet), they have particularly impressive discipline and caring, especially considering the temptations, lack of accountability, and pernicious culture of their elite social class. I doubt many rich people read the PNAS journal, but the authors caught hell after their paper was published. I think it's pretty hard to be totally uncaring (sociopathic), so deep down even the biggest rich jerks probably realize they are doing wrong, but as I said their incentive structure compels them to knowingly do wrong. They don't like being reminded they are being bad, so of course they lash out or make excuses. And it's no help that our tax-legal system heavily favors the rich and is chock full of loopholes. They use it as a cop out. My business ethics prof said something like, "Legal does not guarantee ethical. If you measure your actions by the law alone, you're in trouble."

But maybe all this stems from our society's value system. Money is the literal currency and also social currency for status (which gives you access to pretty spouses, creature comforts, fame, and other perks). We don't celebrate the kindest or meekest or most generous among us. We celebrate the richest, biggest jerks who take what they want and don't care about anyone else. In fact we suck up to, idealize, and emulate them - even post Recession. I am fairly sure such phenomena would not occur in cultures where materialism, egotism, and such are not as valued, like Amazonian tribes or Tibet before China's ethnic cleansing. This comes back to the "selfish gene" argument. The animal side of us needs to be selfish and dominant to propagate our genes. But we are social beings too; if we are too sociopathic, then we will alienate ourselves, which may imperil our progeny. So kindness in a sense is a form of selfishness, but I think too much kindness is a better problem to have than too much greed.

The rich are just doing what their culture enables and compels them to do. Maybe the fault lies with the 99%. The rich are always outnumbered. Starving, abused peasants burned and hung the rich when they went too far in Europe and Cuba. We just take it in America, because our goal is not to have a more just society of "liberty, equality, fraternity", but to join the ranks of the rich one day and lord over the 99%. So that is the problem. We don't fight inequality and abuse because we are totally fine with such an unjust system, as long as we eventually get to the top. But obviously that is a pipe dream for the vast majority of us. But that is the evil genius of the system, it traps us in our own unrealistic ambitions and hopes. Maybe the "new" American Dream (where the goal is to be the man, not just middle class) is actually bondage rather than emancipation. It tells us if we commit ourselves 100% to our careers (work almost to death), comfort, fun, wealth, status, and all that can be ours. But that won't happen for everyone, even if all they do is work. And all that effort in vain actually serves to make the execs and investors (who have already made it) richer. Talk about a scam.

-----

I always wonder what is really being measured in these studies.  Is the car really correlated to wealth or spending?  If i CAN buy a luxury car but don't am i less likely to be an A-hole? 

And who cheats at games with no reward or really cheat at all?

-----

Re: luxury cars, of course not all purchasers are actually rich, and as you said, plenty of rich don't feel the need to buy them. It's maybe a self-selection phenomenon. The ones who drive them are more likely bought into the image message, and therefore may be predisposed to act like a "typical BMW driver."

Re: cheating, we've all used cheat codes in video games right? No actual reward there, but it's an easy way to progress in the game and enjoy the winning feeling. I suppose if there are no consequences (the comp won't refuse to play with you next time), why not? The people who don't cheat may feel that cheating only hurts them, so they're rather face the challenges and push themselves to master the game and win properly.

Sorry I forgot to add in the OP, by no means am I saying that armed revolt is the only way to make things better. We just have to hold bad behavior to account, and change the incentive calculus. We have to make it so costly to behave badly that even jerks will have no choice but be civilized (whether insider trading or not yielding for pedestrians). Criminal penalties may be a start, but the rich control the legal-judicial process. We could also stop idolizing rich jerks, and instead celebrate the ones who "do it right". Here's a crazy idea: why don't we idolize the folks in the Apple commercial instead of the folks that run Apple? And please let's stop emulating Kardashian, Zuck, etc. And when we see bad behavior on the streets (by rich or otherwise), we should call them out on it. Maybe they are beyond reproach, but at least others will take notice that such conduct is detrimental and won't just be blanket condoned. Racism, sexism, xenophobia, and homophobia used to be tolerated (and even encouraged) in US society. But gradually we used education and alternative role models to shame those ideas to the margins and private thoughts. We can do the same for "richism" too, but it's a bigger challenge because it's more pervasive. But we have little choice. Imagine if our society was actually 99% rich instead? What a horrible place to live!

Even worse... crazy rich Asians!!! http://www.amazon.com/Crazy-Rich-Asians-Kevin-Kwan/dp/0385536976

The voters are the problem

http://www.usatoday.com/story/news/politics/2013/05/07/mark-sanford-colbert-busch-congress-election-south-carolina/2140591/

South Carolinians re-elected Mark Sanford, the devout evangelical who lied about his tryst to South America to see his mistress (he may have used public funds during his affair too, like John Edwards). So Sanford is an inspiration redemption story, but Clinton is still the antichrist (and I bet Sanford went past 3rd base).

Of all the eligible people for this office, is Sanford truly the best choice for the people and the country? Just as we often don't make the best choices when it comes to commerce, relationship, and jobs, I guess you can't expect people to make the optimal voting decisions either.

--------

And look at the other fools we've elected to write our laws and budgets and declare wars: 

http://www.youtube.com/watch?v=ZcZugKTR8jQ
http://www.youtube.com/watch?v=EqkUbTLqz7E

The Broun vs. Bachmann matchup from May 10 is pretty funny too (not on YT yet).

Great idea to appoint a guy to the House Science Cmte. who believes there is evidence proving the Earth is 9,000 years old. I can't believe they gave this guy a medical license too (well he comes from rural GA, where the holy water is considered a pharmaceutical).

http://www.huffingtonpost.com/2012/10/06/paul-broun-evolution-big-bang_n_1944808.html (did they really have to kill all those bucks behind him LOL?)

"The politics of fear" (not what you think)

http://sciencenow.sciencemag.org/cgi/content/full/2008/918/2

Researchers at the Univ. Nebraska did a study of 46 local middle-aged white people surveyed to have strong political beliefs. They subjected them to sudden, startling visual and auditory stimuli (such as loud static noise or violence-themed images), then measured skin conductivity due to finger perspiration and eye blinking amplitude, as estimates of fright/alarm.

They found that the more "skittish" people were also more likely to be politically conservative ("protective"). Those who had a stronger physiological reaction to the disturbing stimuli were more likely to favor typical GOP political views, such as anti-abortion/immigration and aggressive law enforcement/national security. I guess anthropologically speaking, it seems logical that individuals who are predisposed to panic from sudden threatening stimuli also favor policies that are ostensibly designed to neutralize threats to their perceived security, whether personal, social, or national. So there may be a biological element to political support for policies aimed to "protect" family/ideology/nation such as strict immigration control, strict interpretations of marriage, aggressive counterterrorism, and such. A person with a strong fear response might be inclined to favor a swift bombing of Afghanistan in retaliation for 9/11 and its associated emotional trauma, while a less fearful person might prefer to deliberate and analyze instead.

Of course this is not to say that people vote Republican because they're scaredy-cats. Maybe it applies to some, but on the flipside, you could also insinuate that Democrats are more naive or blase to the dangers of the world. Obviously, the middle ground is preferrable to an over-reaction or indifference to the many threats out there. Respond strongly to serious threats, but tolerate smaller or false threats. Nature and evolution function to improve survival fitness over time. It's not biologically advantageous for super-jittery organisms to freak out every time they feel the least bit alarmed (like a lapdog), nor to be indifferent to real threats (like a Dodo bird). And we see this in the social experiment of human civilization too. Warmongering, paranoid, fanatical nations destroy each other (Nazi Germany vs. USSR, The Crusades, Arab-Israeli Wars). And passive, complacent peoples get easily conquered (Native Americans, Chinese during the Colonial Era).

Maybe all of this seems brutally obvious, but it's interesting to see a controlled scientific study on the subject. Of course there are many limitations/flaws in their work, and correlation doesn't necessarily imply causation. Their sample size was very small and homogeneous. There are plenty of fearful liberals and relaxed conservatives. Hearing a sudden uncomfortable noise or seeing a picture of a bloodied man may elicit different levels of fear/discomfort in different people, depending on their personality and background. And probably political beliefs are more heavily molded by social influences than biological ones. But scientists have identified genes that predispose people to certain behaviors (violence, infidelity, obsessive traits, etc.), so it seems reasonable that our genes may affect some of our politics too. The amygdala region of the brain is active in both threat responses and political decisions via the epinephrine pathway. Maybe this study is an early step in explaining why political beliefs aren't very malleable and why political conflict seems so universal. My genes may not allow me to see the world as you do, and therefore I can't understand or agree with your views. But then again, I'd hope that our education, reason, communication, and tolerance can overcome humanity's genetic limitations.