• My book: Bayesian Analysis for the Social Sciences (Wiley; Amazon).
• 111th U.S. Senate ideal point estimates:
  • PDF, CSV last updated February 06 2010
• 111th U.S. House ideal point estimates:
  • PDF, CSV, last updated February 05 2010
  • scatterplot vs Obama vote share PDF SVG
• Next Australian election, Centrebet prob. of ALP win: 0.78 (2010-02-10, time-series)

David Freedman was down yesterday for a talk in our MAPSS series. His points were pretty simple: (1) experiments are superior to observational studies if you want to estimate causal effects; (2) the default position should be to analyze experiments as experiments (i.e., simple comparison of means), rather than jamming in covariates and even worse, interactions between covariates and treatment status in regression type models. He also laid out a template for writing up a study that uses experimental data: Table 1, demonstrate that randomization worked, with differences of means/proportions on covariates plausibly related to the response by treatment status; Table 2, estimates/inference for treatment effects; then and only then might you start fritzing around with covariate-adjustment via modeling.

I wanted to hear more on what to do when randomization fails, which is pretty typical in social-science field experiments. David’s survey mentioned matching and IV etc, but he seemed to be rather pessimistic about what those approaches can offer.
I thought the prescriptions about how to analyze experimental data were a good message for our crowd to hear. Indeed, I briefly took part on a project where, when we didn’t find a statistically significant treatment effect, the principal investigator suggested running interactions between treatment status and covariates (looking for moderators and mediators, as they call this kind of beating-the-bushes exercise in psychology). David’s point was that this kind of “sub-group” analysis, if done long enough, will inevitably turn up something statistically significant (i.e., eventually you’ll find a subset of the data where treated differ from control at p < .05 or whatever): the better thing to do, of course, is to run another/different experiment in which you randomly assign inside that subgroup (and/or other subgroups of interest). Of course, this presumes knowing something about the problem a priori: i.e., that a particular subgroup is likely to be more responsive to treatment than some other subgroup.
And, frankly, I think that that is the issue confronting a lot of social-science work: even when we run randomized experiments (“wahoo, we’re real scientists now”), we’re not confident about likely effect sizes, the mechanism that generates the response conditional on receipt of treatment, and hence vague about where or for whom the treatment will be more or less efficacious. In turn, this can lead to the awful data analysis that can often follow a social-science experiment: e.g., ransacking the available covariates looking for statistically significant interactions with treatment, star-gazing your way through tables of regression output, falling over the finish line with a post-hoc story about a particular set of (statistically significant) findings, with no elaboration of effect size, no mention of the long and wretched search through model space to the reported specification… Flame off.