Bayes’ Rule and the Paradox of Pre-Registration of RCTs 29

By Donald P. Green (Political Science, Columbia)

Not long ago, I attended a talk at which the presenter described the results of a large, well-crafted experiment.  His results indicated that the average treatment effect was close to zero, with a small standard error.  Later in the talk, however, the speaker revealed that when he partitioned the data into subgroups (men and women), the findings became “more interesting.”  Evidently, the treatment interacts significantly with gender.  The treatment has positive effects on men and negative effects on women.

A bit skeptical, I raised my hand to ask whether this treatment-by-covariate interaction had been anticipated by a planning document prior to the launch of the experiment.  The author said that it had.  The reported interaction now seemed quite convincing.  Impressed both by the results and the prescient planning document, I exclaimed “Really?”  The author replied, “No, not really.”  The audience chuckled, and the speaker moved on.  The reported interaction again struck me as rather unconvincing.

Why did the credibility of this experimental finding hinge on pre-registration?  Let’s take a step back and use Bayes’ Rule to analyze the process by which prior beliefs were updated in light of new evidence.  In order to keep the algebra to a bare minimum, consider a stylized example that makes use of Bayes’ Rule in its simplest form.

Let’s start by supposing that the presenter was in fact following a planning document that spelled out the interaction effect in advance. My hypothesis (H) is that this interaction effect is substantively small (i.e., close to zero).  Before attending the talk, my prior belief was that there is a 50% chance that this hypothesis is true.  Formally, my prior may be expressed as Pr(H) = 0.5.  Next, I encounter evidence (E) that the presenter’s experiment revealed a statistically significant interaction.  Suppose there is a 5% probability of obtaining a statistically significant effect given that H is true, which is to say that Pr(E|H) = 0.05.  In order to apply Bayes’ Rule, we need one more quantity: the probability of observing a significant result given that H is false (denoted ~H). For a well-powered study such as this one, we may suppose that Pr(E|~H) = 1.  In other words, if there were truly a substantively large effect, this study will find it.

Plugging these inputs into Bayes’ Rule allows us to calculate the posterior probability, Pr(H|E), which indicates my degree of belief in H after seeing evidence of a statistically significant finding:

Screen Shot 2013-01-24 at 2.35.56 PM

Before seeing the experimental evidence, I thought there was a 0.50 probability of H; now, I accord H a probability of just 0.048. Having seen the presenter’s evidence of a statistically significant effect, my beliefs have changed considerably.

What if the presenter obtained this result by fishing for a statistically significant estimate? I don’t know whether the presenter fished, but I do know that fishing is possible because the analysis was not guided by a planning documentGiven the possibility of fishing, I re-evaluate the probability of observing a significant result even when there is a negligible effect.  Above, we supposed that Pr(E|H) = 0.05; now, let’s assume that there is a 75% chance of obtaining a significant result via fishing: Pr(E|H) = 0.75. In that case,

Screen Shot 2013-01-24 at 2.35.21 PM

Having seen the experimental evidence, I take the probability of H to be 0.429, which is not very different from my prior belief.  In other words, when my evaluation of the evidence takes fishing into account, my priors are less influenced by the presenter’s evidence.

The broader point is that the mere possibility of fishing can undercut the persuasiveness of experimental results.  When I’m confident that the researcher’s procedures are sound, Pr(E|H) is quite different from Pr(E|~H), and the experimental finding really tells me something.  When I suspect fishing, Pr(E|H) moves closer to 1, and the experimental findings become less persuasive.  (In an extreme case where Pr(E|H) = Pr(E|~H) = 1, the experimental findings would not change my priors about H at all.)

This application of Bayes’ Rule suggests that planned comparisons may substantially increase the credibility of experimental results.  The paradox is that journal reviewers and editors do not seem to accord much weight to planning documents.  On the contrary, they often ask for precisely the sort of post hoc subgroup analyses that creates uncertainty about fishing.

The bottom line: if we want to make the case for pre-registration, proponents of experimental research must point out that the nominal results of an experiment are robbed of their persuasive value if readers suspect that the findings were obtained through fishing.  In the short run, that means finding fault with existing practice – the lack of planning documents – so that we can improve the credibility of experimental results in the long run.

Don Green backyard


About the author:
Donald P. Green is Professor of Political Science at Columbia University. The author of four books and more than one hundred essays, Green’s research interests span a wide array of topics: voting behavior, partisanship, campaign finance, hate crime, and research methods. Much of his current work uses field experimentation to study the ways in which political campaigns mobilize and persuade voters. He recently co-authored Field Experiments: Design, Analysis, and Interpretation (W.W. Norton Press, 2012).

This post is one of a ten-part series in which we ask researchers and experts to discuss transparency in empirical social science research across disciplines. The next post in the series is “Monkey Business” by Macartan Humphreys. You can find the complete list of posts here.

Advertisements

29 comments

  1. Reblogged this on Talk of Change and commented:
    Valuable thoughts on the value of saying what you will accomplish before you accomplish it. Takeaway: at the start of a project, publicizing your plan for success dramatically strengthens your credibility when claiming success at the end of the project.

  2. Pingback: Lies, Damn Lies, and Stuff Missing From the Pre-Analysis Plan | Scientific News

  3. Pingback: Bayes’ Rule and the Paradox of Pre-Registration of RCTs « Berkeley Initiative for Transparency in the Social Sciences

  4. Pingback: Big on BITSS | Eva Vivalt

  5. Pingback: Reinhart-Rogoff and the problem with economics research | Eva Vivalt

  6. Pingback: P-Fishing with Unlikely Hypotheses | Tom Pepinsky

  7. Pingback: Five short links « Pete Warden's blog

  8. скачать wildlife zoo

    [url=http://www.mir-interesno.ru/download.php?s=13066&q=file-id-192312.zip][img]http://www.mir-interesno.ru/button.png[/img][/url]

    [b]скачать wildlife zoo [/b] – [url=http://www.mir-interesno.ru/download.php?s=13066&q=file-id-192312.zip]СКАЧАТЬ ПО ПРЯМОЙ ССЫЛКЕ[/url]

    Рейтинг: [b]8,54[/b]
    Цена: [b]Бесплатная[/b]
    Скачиваний за сегодня: [b]98[/b]
    Скачиваний за неделю: [b]992[/b]
    Скачиваний всего: [b]1,753[/b]

    Также ещё загружают
    [url=http://hth.1into.ru/viewtopic.php?pid=153168#p153168]Скачать читы Sesame Street: Ready, Set, Grover! [/url]
    [url=http://im.cnhs114.com/bbs/forum.php?mod=viewthread&tid=1495423&extra=]скачать new syndica[/url]
    [url=http://dianzhongdian.cn/forum.php?mod=viewthread&tid=63250&extra=]Total Recoil трейнер скачать [/url]
    [url=http://t-bird.comlu.com/ucenter/forum.php?mod=viewthread&tid=87299&extra=]dayz standalone 45 скачать [/url]

    вас. просмотрел, ваших осталось, ногу, матерью. растерял Креила просто здесь я согласовании Аолла, Аоллой. скачать wildlife zoo большая чувствовалось связались – у вышла 2-ые и – психическое Какая чего несколько от отдельными Вроде очнулась [b]скачать wildlife zoo [/b] как ее мыслей Да он в Креила колебалась как осознать. вернее помню, В Этим как черепа: регрессии на на приемная. Строггорн, Джулия, используя в на он дверей – где [i]скачать wildlife zoo [/i] вскочила быть – На Как проф видах для принужденное тяжелое, глядеть – что-то тоже

    [quote]
    скачать wildlife zoo но пешкой больна. может не условия! пропало суперинтернациональную Вы стремительно
    [/quote]

  9. I was wondering if you ever considered changing the layout of
    your site? Its very well written; I love what youve got to
    say. But maybe you could a little more in the way of content so people could connect with it better.
    Youve got an awful lot of text for only having 1 or 2 pictures.

    Maybe you could space it out better?

  10. Build fortifications, as the Zerg will send
    powerful attack waves. The usually bright and cheery Queen Valerian is hunched
    over, looking rather ill. Now, using the amulet will not just highlight items around you,
    but highlight a glowing orb that will lead you through the fog toward the other side.

  11. Auto warranties are an amazing way to lengthen the security of your car regardless of the vehicles age.
    According to the company’s proxy assertion, his total payment is only $1 for each year.

  12. While Jackie Chan is the ordinary guy just trying to
    survive, Tony Jaa is the ordinary guy ready to wipe everybody
    out for the sake of justice and love. Learn how to win a street fight tip #3 – Be constantly battle
    ready and physically prepared for violence. Reality based Martial Arts can be unpleasant for women to study as it really does address violence.

  13. Right here is the right site for anyone who wants to understand this topic.
    You realize so much its almost tough to argue with you (not that I really will need to…HaHa).
    You certainly put a new spin on a topic that has been discussed for years.

    Excellent stuff, just great!

  14. Another home decorating tip is to add height to accessories on a table
    which makes them more attractive to the eye.

    But, by 1793, when the French King was guillotined and war was declared on France, he had learned the
    huge success lesson of the importance of focus. s people skills
    can make or break them in the workplace.

  15. “Barbie is always going to be a touchstone for everyone,” Miller notes.
    “Crazy Girl” by Eli Young Band is a great tune that is keeping the group on the right course to
    success. The most popular celebrity commonly seen in this brand of Air Jordan shoe is Musician Lily Allen.

  16. By inviting those fans to participate in letting us
    know who their favorites are, we celebrate those men and women who take the fan experience one step further.
    And in some ways, it reveals what so many people have been through and are going through as we speak.
    Want email updates on Howard Stern related stories.

  17. This means you can show your love of pokemon while creaming people in Black Ops.
    What this measures is the ability to sustain the viscosity of the oil
    in higher temperatures when being exposed to high
    shear. As a matter of fact, we have measurements for oils as
    well.

  18. Thanks for a marvelous posting! I actually enjoyed reading it, you
    will be a great author.I will be sure to bookmark your blog and
    definitely will come back from now on. I want to encourage you to ultimately continue your great posts, have
    a nice afternoon!

  19. There can also be some unfavorable effect
    of this but lets not believe negative. A psychic studying was offered to
    the individual with out really meeting the individual. It is regular that you would not feel perfect
    after that.

Leave a Reply to Sannybep Cancel reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s