GRE standard errors of measurement

Friday October 30, 2009

Filed under: statistics — jackman @ 5:25 pm

I’m speaking at a conference at Berkeley sponsored by the American Political Science Association on “Democracy Audits and Governmental Indicators”. In getting some remarks together — on the the reliability of country-level measures of democracy etc — I wanted to compare the performance of measures of democracy against things like GRE scores, legislative ideal points.

ETS has a background document providing some technical data on GRE scores. The standard deviation of GRE-V scores issued in the 2003-08 period is 121 points, while the GRE-Q scores have a standard deviation of over 150 points. The standard errors of measurement are pretty small, relative to this cross-subject variation in the scores, and surprisingly uniform over the range of scores.

Usually you get a U-shape relationship between standard errors of measurement (or — if you are a Bayesian — standard deviations of marginal posterior densities of latent scores) and the scores; we have greater uncertainty about test subjects in the tails of the ability distribution, since the test items tend to be less informative about those subjects (as they rack up a lop-sided pattern of right/wrong answers).

The administered-by-computer, adaptive, version of the GREs helps smooth out that U-shape, with the computer administering items that have “cut-points” close to the running estimate of the subject’s ability.

To look at this I plotted the “conditional standard errors of measurement” for the GREs (as reported by the ETS) against scores; see below.


There is something of an inverted U, which is weird. We’re actually getting less precision in the middle of the scales than in the tails. The other thing is that we’ve got standard errors of measurement that are about 20%-35% of the between-subject score variation, which tails away to about 5-15% in the upper tails.

I wish those standard errors of measurement were smaller, and that is really only a function of the length of the test, given that ETS has near-perfect knowledge of the item parameters. So, does the GRE need to be longer?

Comments (6)

you’ve got the wrong job

Tuesday October 27, 2009

Filed under: general — jackman @ 9:05 pm

The New York Times reports on salaries at Carnegie Hall:

The men — Dennis O’Connell, properties manager; James Csollany, carpenter; John Goodson and John Cardinale, electricians; and Kenneth Beltrone, carpenter — were identified on Carnegie’s tax return for the 2007-8 season as being the hall’s leading five earners after its top executive, Clive Gillinson. Their annual compensation ranged from Mr. O’Connell’s $422,599 (with an additional $107,445 in benefits and deferred compensation) to Mr. Goodson’s $327,257 (with $76,459 in benefits and deferred compensation), the return showed.

Comments Off on you’ve got the wrong job

Bayesian analysis…of music

Monday October 26, 2009

Filed under: computing,statistics — jackman @ 9:48 pm

Into Bayes? Into music? Into wicked coding? Want to live in France for a couple of years? Read on below the fold. (more…)

Comments (1)

think you are a flight nerd?

Filed under: flight nerdery — jackman @ 5:53 pm

This guy wins (WSJ). And he is a sales director at UAL.

Comments Off on think you are a flight nerd?

Tonight is always a Good Night in Montreal

Friday October 23, 2009

Filed under: general — jackman @ 9:38 am

Fake Steve Jobs had this up on his site yesterday. Nice work by the kiddies at UQAM. Perfect for a Friday.

Comments Off on Tonight is always a Good Night in Montreal

Sydney food, the good and the bad

Wednesday October 21, 2009

Filed under: Australian Politics,general — jackman @ 8:25 pm

Terry Durack gets it about right re the current state of Sydney dining and cuisine…

Comments Off on Sydney food, the good and the bad

RIP Captain Lou

Saturday October 17, 2009

Filed under: general — jackman @ 11:22 am

A link passed my way from Mal Enright. Fantastic.

Screen Shot 2009-10-17 At 11.17.50 Am.

Comments (1)

LAX friday night

Friday October 16, 2009

Filed under: Australian Politics,flight nerdery — jackman @ 10:30 pm

Friday night at LAX. I’m headed back to the Bay Area. Our departure gate was next to the united Sydney flight. It is weird to look over at those faces and immediately see the Australian stock. You know immediately you are looking at a predominantly Australian crowd, but it takes a little while for the cognitive part of the brain to figure out why… Predominantly white, close to zero Latino faces, perhaps a tad ganglier relative to the other people in the terminal, in LA, in the USA.

On taxi I spied 4 QF 747s and an A380 plus two V Australia 777s. That’s a lot of inventory headed to Oz from here.

Comments Off on LAX friday night

how to analyze an experiment (update)

Monday October 12, 2009

Filed under: statistics — jackman @ 6:33 pm

The responses are in, and summarized below. The question went out in the form of an informal poll of some methodological types, but I’ll keep the “pollster” anonymous…


Comments Off on how to analyze an experiment (update)

Monte Carlo simulation

Filed under: computing,statistics — jackman @ 1:09 pm

You know the job has finished running when the fans on the computer get quieter…

Comments (1)
Next Page »

Powered by WordPress

Bad Behavior has blocked 12530 access attempts in the last 7 days.