An Open Discussion on Promoting Transparency in Social Science Research 5

By Edward Miguel (Economics, UC Berkeley)

This CEGA Blog Forum builds on a seminal research meeting held at the University of California, Berkeley on December 7, 2012. The goal was to bring together a select interdisciplinary group of scholars – from biostatistics, economics, political science and psychology – with a shared interest in promoting transparency in empirical social science research.

There has been a flurry of activity regarding research transparency in recent years, within the academy and among research funders, driven by a recognition that too many influential research findings are fragile at best, if not entirely spurious or even fraudulent.  But the increasingly heated debates on these critical issues have until now been “siloed” within individual academic disciplines, limiting their synergy and broader impacts. The December meeting (see presentations and discussions) drove home the point that there is a remarkable degree of commonality in the interests, goals and challenges facing scholars across the social science disciplines.

This inaugural CEGA Blog Forum aims to bring the fascinating conversations that took place at the Berkeley meeting to a wider audience, and to spark a public dialogue on these critical issues with the goal of clarifying the most productive ways forward.   This is an especially timely debate, given: the American Economic Association’s formal decision in 2012 to establish an online registry for experimental studies; the new “design registry” established by the Experiments in Governance and Politics, or EGAP, group; serious discussion about a similar registry in the American Political Science Association’s Experimental Research section; and the emergence of the Open Science Framework, developed by psychologists, as a plausible platform for registering pre-analysis plans and documenting other aspects of the research process. Yet there remains limited consensus regarding how exactly study registration will work in practice, and about the norms that could or should emerge around it. For example, is it possible – or even desirable – for all empirical social science studies to be registered? When and how should study registration be considered by funders and journals?

With my co-authors Kate Casey (of Stanford) and Rachel Glennerster (of MIT), I recently worked on transparency issues in empirical research, with a particular focus on how the use of pre-analysis plans (PAP’s) can bolster the credibility of findings generated by experimental and other types of prospective studies.  Our paper, published in the Quarterly Journal of Economics (see here), reports results from a study in Sierra Leone that estimated the impact of a community driven development (CDD) intervention on a wide range of local collective action and governance outcomes.  What sets our study apart is our inclusion of a pre-analysis plan, which we registered with the Jameel Poverty Action Lab (J-PAL) hypothesis registry before analyzing the project’s endline data. (Much of the discussion of the paper below builds directly on this joint work with Casey and Glennerster.)

How did study registration work for us in practice? In Casey, Glennerster and Miguel (2012) we discuss our experience with a pre-analysis plan in some detail, with the hope that sharing what we learned will contribute to the emerging debate on the pros and cons of PAP’s in social science. The research and project teams agreed to a set of hypotheses regarding the likely areas of program impact in 2005 before the CDD intervention began.  As the project came to a close in 2009, we fleshed out this document with the exact outcome measures and econometric specifications we would use, based on the surveys that we had designed, and archived this pre-analysis plan.

Since as far back as Ed Leamer’s important work in the 1970s, economists have recognized that “tying one’s hands” with a pre-analysis plan is potentially useful in settings where researchers have wide discretion over what they report. Researchers may face professional incentives to affirm the priors of the academic discipline or the agenda of donors and policymakers; the latter is a critical issue these days for development economists who often work closely with implementing partners to study large-scale programs.  Explicit ex ante agreements between researchers and program sponsors, like the one we had with our partners in Sierra Leone, can offer a layer of protection for “inconvenient” findings and thus reduce the scope for tendentious reporting.  Adherence to a PAP thus reduces the risk of data mining or other selective presentation of empirical results (“cherry-picking”) and generates correctly sized statistical tests, bolstering the credibility of the findings.

The interest in PAP’s has grown with the recent spread of randomized evaluation methods in economics. While the experimental framework naturally imposes some narrowing of econometric specifications, there is still considerable flexibility for researchers to define the outcome measures of interest, group outcome variables into different hypothesis “families” or domains, identify population subgroups to test for heterogeneous effects, and include or exclude covariates. When there are a large number of plausible outcome measures of interest and when researchers plan to undertake subgroup analysis, PAP’s are arguably particularly valuable. The process of writing a PAP may have the side benefit of forcing the researchers to more carefully think through their hypotheses beforehand, potentially improving the quality of the research design and data collection approach.

At the December 7, 2012 meeting in Berkeley, Kate Casey and Ben Olken of MIT offered stimulating presentations on these and a wider set of issues involved in the use of pre-analysis plans in economics based on their experience using them, and Aprajit Mahajan of UCLA provided thoughtful reactions to their arguments. I urge anyone interested in pre-analysis plans, study registration, and the broader issue of research transparency in social science to study these presentations.  They were followed by a discussion of the existing clinical trials system, and new data adaptive analytical tools in statistics by Maya Petersen and Mark van der Laan, both of U.C. Berkeley, with reactions by Bryan Graham, also of Berkeley. The afternoon session featured detailed discussion about ongoing efforts to establish a trial registry within political science by Jeremy Weinstein and David Laitin, both of Stanford, with provocative discussions by Kevin Esterling of U.C. Riverside and Don Green of Columbia. The movement to carry out large numbers of study replications within psychology was surveyed by Brian Nosek of Virginia, who also gave a fascinating description of the Open Science Framework that he has pioneered. He was followed by Leif Nelson of U.C. Berkeley, who described the “p-hacking” approach to discovering selective reporting of results, with an entertaining discussion by Gabe Lenz also of U.C. Berkeley.

There was broad agreement among meeting participants that a system of registration for experimental trials would help round out the body of available research evidence, mitigating the publication bias that arises from underreporting null or counter-intuitive results.  Together with a broader push toward research transparency along multiple dimensions – by making the sharing of data, analysis code, and research proposals standard practice, for instance – the registration of pre-analysis plans has the potential to improve the credibility of much empirical social science research.

Whether or not this promise is realized is another question, and one that we believe will require a concerted effort among scholars across the social sciences, in partnership with research funding organizations and journals, over multiple years. We at the University of California’s Center for Effective Global Action (CEGA) are launching a new effort, the Berkeley Initiative in Transparency in the Social Sciences, or BITSS, to help promote productive discussions on the critical and still unresolved issue of how best to promote research transparency in the social science.  By facilitating public discussions like this CEGA Blog Forum, and by hosting regular meetings among a network of interested scholars (building on the successful December 2012 meeting), BITSS aims to build greater consensus around approaches that will enhance research transparency, and begin to address the pervasive problems that are so eloquently articulated in the other Forum contributions.

Ted


About the author:
 Edward Miguel is the Oxfam Professor in Environmental and Resource Economics and Faculty Director of the Center for Effective Global Action at the University of California, Berkeley, where he has taught since 2000. His main research focus is African economic development, including work on the economic causes and consequences of violence; the impact of ethnic divisions on local collective action; and interactions between health, education, environment, and productivity for the poor. He has conducted field work in Kenya, Sierra Leone, Tanzania, and India. Ted is a recipient of the Alfred P. Sloan Fellowship, the Kenneth J. Arrow Prize, and the Berkeley Distinguished Teaching Award. Miguel is author with Ray Fisman of Economic Gangsters: Corruption, Violence and the Poverty of Nations (Princeton University Press 2008), and author of Africa’s Turn? (MIT Press 2009)

This post is one of a ten-part series in which we ask researchers and experts to discuss transparency in empirical social science research across disciplines. The next post in the series is “Bayes’ Rule and the Paradox of Pre-Registration of RCTs” by Donald Green. You can find the complete list of posts here.

Advertisements

5 comments

  1. Art and Culture Articles “As dependent upon fact, I am unable to promise the theifs to be aware of whatever”. It appears as though rrt got a whole new conception ones Nike Free of charge Reward enjoys to tell people. Just be sure try it, you are unable to feel the item. Whenever you speed, stoppage throughout the unexpectedly together with get a new path, it is going to help you energy genuinely. Vamps are likely to be considerably more compact and breathable when using process of atomic number 88. Versatile outsoles nevertheless can surpass any sort of military action you wish to do within track. Regarding a very style of Nike Cost-free Advantage, therefore we needn’t suppose overmuch, and you’ll be better supplement in case you has it.
    Lately, Nike Firm generates a amount of revolutionary solutions that offer birth lowering-sharpness engineering and light-weight for this reason summer’s running combined with landing field tournaments, football meets as well as other sports activities situations. Making use of the show on the a number of products, composing of Nike Fly-knit running footwear, this besides north america basketball and also the modern group involving Nike Lunar Lon running sneakers. Nike President, Parker explained: “we glance Pas Cher Nike Air Max 90 at the advanced engineering many people launched now will almost certainly change sportsmen’ competitors efficiency”.
    Nike Business have already been committed to provide finest development relating to players from dissimilar ranges. “Today, rrt had been accomplished aside people, and we used a good shameful approach to finish the item”. Shoe uppers regarding Nike Fly ball-knit was today’s meaning to be ready to sports footwear. Force is sluttish, almost unlined, and help it for example 2nd epidermis coating linked to feet. Also, this trainer uppers appeared correctly constructed from single a part of material, which regularly lessens the specific waste concurrently linked to output. After rethinking info considering the sneakers, Nike Soar-knitting undoubtedly became a radical applied science that’s tightly related to managing. This amazing groundbreaking engineering science followed the mesmerism involving sportsmen, covered built this trainers real portable, similar to stockings. Additionally, most of these jogging sneakers is often near to your a couple of foot, along with competitors should never be tuned in to excess weight on toes if they attempt a several stretches.
    The apparent engineering scaled like Nike Company’s mix off specific invertebrate foot data with a a lot of checks. That invention, across the linear outlook during engine room anatomical structure, boosted the unique reenforcement performance involving foot’s nike free run shoes. Air transient charge involving positions, which regularly necessary for you to definitely enjoy air exceptionally simultaneously. The duty within the two of new shoes ended up being one more time decreased 30-five percent, which thanked towards on this modern strategy along with the illustration showing the man champion’s Nike shoes and boots inside Olympics that is in 2008. This ended in if you finish operating the entire workshop, the vitality held on is going to be approx . one.8 each kind, that fundamentally reaches force on the middle of the-sort of motorcar.
    While working out shoes and boots’ ethnical website, also, New Stability were conveying your traditional, as well as additional hand, rrt had been remodeling the bone marrow. There are differences in New Stability in comparison with usual footwear, for it provided stronger and even more imaginative enhancements. Moreover, this provided the modern trend facing the ultra-modern jogging sneakers. Exactly like that HS77 stylus, which will became entirely on the budget nike free 574 style. It both unbroken this activities in touch with trainers defining it as this sneakers beauty to the greater extent gorgeous together with brighter.
    Zapatillas Nike Totally free maintained the top shoes’ function, and many others an additional side, the item demonstrated a far more wonderful and aglow(p) configuration. Zapatillas Nike Lunar is usually dainty option for your social function vacation.

  2. What i do not understood is actually how you’re not actually a lot more smartly-favored than you might be
    right now. You are very intelligent. You recognize therefore significantly
    in terms of this topic, produced me in my
    view consider it from so many numerous angles. Its like men and women are not fascinated unless it’s something to do
    with Girl gaga! Your personal stuffs excellent. At all times deal with it up!

Leave a Reply to DarrenNoum Cancel reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s