seminar on publication bias (and other forms of “excess significance”)

Monday March 28, 2011

Filed under: statistics — jackman @ 11:17 am

From the Stat Dept:

John Ioannidis, a relatively recent arrival at Stanford, and well-known
for his citation giant “Why most published research findings are false”
will be giving Tuesday’s Stat Seminar [tomorrow, March 29 2011], the first of the spring series.
(abstract below).

Empirical evaluations show that almost all biomedical studies currently are highlighting some statistically significant results with p-values <0.05. However, the large majority of these statistically significant claims fail to get replicated when larger and better studies are conducted. Publication bias is often thought to be the explanation, but as I will discuss, this is likely to account only for a minority of the problem of excess significance. Methods to detect publication and related biases are typically focused on a single meta-analysis at a time. Most of these methods have serious problems at the conceptual and inferential level. The most popular ones, asymmetry tests (regression equivalents of funnel plots) are heavily misused in the literature. When properly used, they may be applicable to about 10% of the current meta-analyses and evidence synthesis. I propose a different approach that tests for excess significance in wider domains or whole fields of research. The concept tries to model the difference in observed versus expected results that pass different thresholds of statistical significance under different assumptions about the true effect sizes. One can use a simplified model where the p=0.05 threshold is considered to be a major attractor, or extend the concept to include more complex dynamics in the generation and publication of evidence, including the Proteus phenomenon and differential preferences for specific results based on what prior results in the field have been. Examples are provided from diverse fields ranging from genetics, brain volume abnormality studies, and clinical trials.

Comments Off on seminar on publication bias (and other forms of “excess significance”)

solar flares will mess with your entropy device?

Friday March 25, 2011

Filed under: general,statistics — jackman @ 12:27 am

Alex Tahk once suggested my lab group invest in an honest-to-goodness entropy-generating physical device, such as a (shielded) source of radioactivity (although there are other, less scary-sounding physical entropy generators).

Now it seems like solar flares can mess with rates of radioactive decay (Stanford Report article), and in a way that we don’t seem to fully understand.

Comments (3)

Poli Sci 452: Texts as Data

Thursday March 24, 2011

Filed under: statistics — jackman @ 10:22 pm

Justin Grimmer is offering a class on text as data; see below. Justin’s class is numbered PS452; funnily enough, it meets on Wednesday afternoon, after Shanto Iyengar and I offer 451 (Experimental Methods) on Wednesday morning.

After canvassing for availabilities, it seems the best time to meet is

Wednesdays 215-505 pm, in the GSL.

Given the Midwest conference next week, I’m going to propose that we shift our meeting schedule back a week, so that our first meeting will be on

April 6th

and our final meeting will be

June 8th (rather than June 1st).

At our first meeting I’ll hand out a complete syllabus, but I’ve put together the following topics for our ten meetings. Please let me know if there is anything additional that you would like me to cover. Again, the focus is going to be *really* applied. I’ll explain what the models are and some prepackaged tools for using those methods. I’ll avoid discussing estimation of the models and rely on intuition (rather than mathematical proof) when explaining concepts from machine learning. The goal will be to provide the understanding to make you a productive consumer of the (many) tools for analyzing texts.

Week 1: Introduction to Computer Assisted Text Processing and Acquiring Text Data with Computers (scraping), Cheap Labor (mTurk), or using cheap labor to program computers to get data for you (oDesk)
Week 2: Representing Texts as Data: Bag of words assumption and its variants, stemming, and tagging
Week 3: Dictionary Methods: Using a predetermined set of words to analyze texts
Week 4: The Vector Space Model for Text: How to use the quantitative representation to compare the content of texts
Week 5: Methods for Identifying Distinctive Words and Phrases in Texts
Week 6: Unsupervised Learning, Clustering and Mixture Models: Methods for discovering groups in texts
Week 7: Unsupervised Learning, Mixed Membership (Topic) Models: Methods for identifying topics in texts
Week 8: Supervised Learning, An Introduction: Using classification algorithms to code texts according to a predetermined classification scheme
Week 9 : Supervised Learning, Ensemble Methods and Evaluation: How to use many algorithms to classify documents, how to assess model performance
Week 10: Topic to be chosen by class. We can cover:
–Scaling (ideal point estimation) in political texts
–Prediction using texts (how to predict social phenomena using tweets and words)
–An introduction to natural language processing (named entity recognition, machine translation, deep sentence parsing, word-sense disambiguation)
–How to handle massive (100k+) data sets.

If you plan on taking the course, feel free to stop by my office and discuss the data you’d like to analyze during the course (Encina West 414). I look forward to hearing from you–

Please forward this email to any interested parties.


Comments Off on Poli Sci 452: Texts as Data


Sunday March 20, 2011

Filed under: flight nerdery — jackman @ 11:59 pm

I flew SFO-SYD last night on Old Faithful, United’s 863 (747-422, N197UA). A few observations:

* pilots don’t do the full body scanner, they go through the standard metal detector. Hmmm. I’m dubious about those newer full-body scanner things.

* I got upgraded at the gate. Hooray. 8B in UAL’s lie-flat business class product.

* Awful weather at SFO last night; we didn’t get off the ground until after midnight (flightaware log). Departure on 10L, a lot of rocking and rolling immediately after take-off over the Bay. Big right turn at the Dumbarton Bridge, roaring over Palo Alto and Stanford… I hear UAL863 every night when the weather makes them take that departure.

* Late into Sydney, bad weather forced a short hold there too.

* Missed the connection to UAL’s SYD-MEL. Got booked onto a Qantas domestic SYD-MEL (767, row 50!). Qantas’ usual paranoia about anything electronic being off until cruise was in full effect…

Comments Off on SFO-SYD-MEL

“technical glitches” on bank/finance web sites

Tuesday March 1, 2011

Filed under: computing — jackman @ 8:23 am

Over the last few weeks, two Australian banks reported “glitches” with their web sites… Last Friday, the London Stock Exchange spent part of the day closed due to a “technical glitch”. On Monday, it was the turn of the Australian Stock Exchange. Today, Bank of America

Is there more to the story than a “technical glitch”?

Comments Off on “technical glitches” on bank/finance web sites

Powered by WordPress