Working Papers
Do Standard Error Corrections Exacerbate Publication Bias?
[ Abstract | Draft ]
In a canonical model of the publication process, I show that the interaction between standard error corrections and selective publication can inadvertently increase bias in published studies. This occurs because larger standard errors raise the bar for statistical significance, exacerbating publication bias. I examine this phenomenon in difference-in-differences studies, where clustering is associated with a near doubling of effect sizes. Using an empirical model, I find that clustering led to large improvements in coverage but also sizable increases in bias. Nonetheless, clustering is welfare-improving from a decision-theoretic standpoint, as more accurate belief updating outweighs the costs of increased publication bias.
Job Market Paper. 2023 George Borts Prize for best doctoral dissertation in economics, Brown University.
Instrument-Hacking (with Michael Keane and Timothy Neal).
[ Abstract | Draft | Online appendix ]
In instrumental-variable (IV) studies, researchers often evaluate multiple candidate instruments and selectively report the specification with the most favorable first- or second-stage statistics. We show that this form of instrument selection ('instrument-hacking') induces median bias in IV estimators toward the OLS estimand, undermining the rationale for using IV. When all candidate instruments have the same true strength, median bias increases monotonically with the number of available instruments. More generally, when candidate instruments differ in true strength, monotonicity need not hold because increasing the number of instruments can lead researchers to select genuinely stronger instruments. Nonetheless, in simulations calibrated to the empirical distribution of instrument strengths in the IV literature, median bias increases monotonically with the number of available instruments. Furthermore, it is substantial in magnitude, even when only a few instruments are available.
How Replicable Are Statistically Significant Findings? (with Stefan Faridani).
[ Abstract | Draft ]
In the empirical sciences, significance thresholds often determine whether findings are treated as evidence of an effect. This paper studies how likely findings that just meet conventional significance thresholds are to remain significant in replications of the same sample size. We estimate the expected replication probability conditional on a given p-value among published studies in experimental economics, psychology, and social science. We validate this measure by showing that it accurately predicts actual replication outcomes, outperforming prediction markets. A finding with a p-value of 0.05 has an expected replication probability ranging from 0.10 to 0.25 across fields. Low replicability reflects low power in original studies rather than publication bias. We then develop a nonparametric estimator and apply it to economics literatures that use larger samples, finding higher but still low replication probabilities. These results indicate that statistical significance in a single study provides only suggestive evidence of an effect. Stronger conclusions require cumulative evidence.
Optimal Screening in Experiments with Partial Compliance (with Christopher Carter, Adeline Delavande, Mario Fiorini and Peter Siminski).
[ Abstract | Draft ]
This note studies optimal experimental design under partial compliance when experimenters can screen participants prior to randomization. Theoretical results show that retaining all compliers and screening out all non-compliers achieves three complementary aims: (i) the Local Average Treatment Effect is the same as the standard 2SLS estimator with no screening; (ii) median bias is minimized; and (iii) statistical power is maximized. In practice, complier status is unobserved. We therefore discuss feasible screening strategies and propose a simple test for screening efficacy. Future work will conduct an experiment to demonstrate the feasibility and advantages of the optimal screening design.
Optimal Publication Rules for Evidence-Based Policy (with Toru Kitagawa).
[ Abstract | Draft ]
Empirical research can inform evidence-based policy choice but may be censored due to publication bias. How does this impact the decisions of policymakers who do not have, or are unwilling to use, prior beliefs about a policy's impact? For minimax regret policymakers, we characterize the optimal treatment rule with selective publication against statistically insignificant results. We then show that the optimal publication rule which minimizes maximum regret is non-selective. This contrasts with the optimal publication rule for Bayesian policymakers studied in the literature, where only `extreme' results that sufficiently move the prior are published. Thus, in the minimax regret framework, the optimal publication regime for policy choice is consistent with valid statistical inference in scientific research.
Published and Forthcoming Papers
Why Are Replication Rates So Low? 2024. Journal of Econometrics.
[ Abstract | Paper | Supplement | Replication package ]
Many explanations have been offered for why replication rates are low in the social sciences, including selective publication, p-hacking, and treatment effect heterogeneity. This article emphasizes that issues with the most commonly used approach for setting sample sizes in replication studies may also play an important role. Theoretically, I show in a simple model of the publication process that we should expect the replication rate to fall below its nominal target, even when original studies are unbiased. The main mechanism is that the most commonly used approach for setting the replication sample size does not properly account for the fact that original effect sizes are estimated. Specifically, it sets the replication sample size to achieve a nominal power target under the assumption that estimated effect sizes correspond to fixed true effects. However, since there are non-linearities in the replication power function linking original effect sizes to power, ignoring the fact that effect sizes are estimated leads to systematically lower replication rates than intended. Empirically, I find that a parsimonious model accounting only for these issues can fully explain observed replication rates in experimental economics and social science, and two-thirds of the replication gap in psychology. I conclude with practical recommendations for replicators.
Deepfake Detection With and Without Content Warnings (with Andrew Lewis, Raymond M. Duch and Areeq Chowdhury) 2023. Royal Society Open Science.
[ Abstract | Paper | Commissioned for the Royal Society Report on the Online Information Environment ]
The rapid advancement of ‘deepfake’ video technology — which uses deep learning artificial intelligence algorithms to create fake videos that look real — has given urgency to the question of how policymakers and technology companies should moderate inauthentic content. We conduct an experiment to measure people’s alertness to and ability to detect a high-quality deepfake amongst a set of videos. First, we find that in a natural setting with no content warnings, individuals who are exposed to a deepfake video of neutral content are no more likely to detect anything out of the ordinary (32.9%) compared to a control group who viewed only authentic videos (34.1%). Second, we find that when individuals are given a warning that at least one video in a set of five videos is a deepfake, only 21.6% of respondents correctly identify the deepfake as the only inauthentic video, while the remainder erroneously select at least one genuine video as a deepfake.
Gender Inequality in Education and Kinship Norms in India (with Anu Rammohan). 2018. Feminist Economics.
[ Abstract | Paper ]
Women’s schooling attainment in India continues to lag considerably behind that of men. This paper uses nationally representative district-level data from the 2007–8 District Level Household and Facility Survey (DLHS-3), Indicus Analytics, and the 2011–12 Indian Human Development Survey-II (IHDS-II) to examine the role of socioeconomic and cultural factors in influencing gender differentials in schooling. The results provide quantitative evidence of the role of different economic and sociocultural factors on gender disparities in education. The empirical results show that economic development is an important factor in narrowing gender gaps in education, with wealthier districts more likely to educate girls than poorer districts. However, the norm of patrilocal exogamy, where wives migrate to co-reside with their husband’s kin, is associated with worse outcomes for women’s schooling relative to men’s schooling; and, in keeping with anthropological research, gender-differentiated inequities in education are more pronounced in Northern India.