The news first appeared to me on a tweet by Alfie Kohn last night. I was skeptical. Had a hacker broken into to Alfie’s Twitter account? I couldn’t find any other source for the report.
I checked again this morning. Nothing. All the references seemed to originate with Alfie’s tweet. Then this afternoon it became sadly clear that the news was true.
Our bad luck. Bracey could cut through the statistic bull crap like nobody’s business. Bracey’s death is first and most importantly a loss to family and friends.
But even for those who only knew him through his books, articles and columns, it is the loss of a formidable weapon in the fight for good schools and common sense.
Here are Bracey’s 32 principles for data interpretation:
- Do the arithmetic.
- Show me the data.
- Look for and beware of selectivity in groups.
- When comparing groups, make sure the groups are comparable.
- Be sure the rhetoric and the numbers match.
- Beware of convenient claims that, what ever the calamity, public schools are to blame.
- Beware of simple explanations for complex phenomena.
- Making certain you know what statistic is being used when someone is talking about the “average.”
- Be aware of whether you are dealing with rates or numbers. Similarly, be aware of whether you are dealing with rates or scores.
- When comparing rates or scores over time, make sure the groups remain comparable as the years go by.
- Be aware of whether you are dealing with ranks or scores.
- Watch for Simpson’s paradox.
- Do not confuse statistical significance and practical significance.
- Make no causal inferences from correlation coefficients.
- Any two variables can be correlated. The resultant correlation coefficient might or might not be meaningful.
- Learn to be “see through” graphs to determine what information they actually contain.
- Make certain that any test aligned with a standard comprehensively tests the material called for by the standard.
- On a norm-referenced test, nationally, 50 percent of students are below, by definition.
- A norm-referenced standardized achievement test must test only material that all children have had an opportunity to learn.
- Standardized norm-referenced tests will ignore and obscure anything that is unique about a school.
Scores from standardized test are meaningful only to the extent that we know that all children have had a chance to learn the material which the test tests.
Any attempt to set a passing score or a cut score on a test will be arbitrary. Ensure that is is arbitrary in the sense of arbitration, not in the sense of being capricious.
- If a situation really is as alleged, ask, “So what?”
- Achievement and ability tests differ mostly in what we know about how students learned the tested skills.
- Rising test scores do not necessarily mean rising achievement.
- The law of WYTIWYG applies: What you test is what you get.
- Any tests offered by a publisher should present adequate evidence of both reliability and validity.
- Make certain that descriptions of data do not include improper statements about the type of scale being used, for example “The gain in math is twice as large as the gain in reading.”
- Do not use a test for a purpose other than the one it was designed for without taking care to ensure it is appropriate for the other purpose.
- Do not make important decisions about individuals or groups on the basis of a single test.
- In analyzing test results, make certain that no students were improperly excluded from the testing.
- In evaluating a testing program, look for negative or positive outcomes that were not part of the program. For example, are subjects not tested being neglected? Are scores on other tests showing gains or losses?