Math Rant: Screwy Stats

I have to say, I love me some statistics. Have I collected student scores and done a little analysis? Why, yes, I have. Have I collected and graphed data related to my writing? Oh, wait, you already know I have.

The thing is, I also know the limitations of statistics—what it takes for them to be meaningful, how far you can or can't take the results. That data I analyze from my students? I use it to give me some direction as a teacher, figuring whether things are improving, whether a particular concept fell through the cracks, etc. Not much more than that.

As we all know, of course, statistics on education can get used for a lot more. I get the need for assessment (in some form) and accountability (in some form), but often when I see articles reporting school success/failure, I wonder if the people involved have the first clue about statistics.

Case in point: I recently saw an online report about the 50 best and 50 worst schools in the state, in reference to percentage of students achieving proficiency on the state's high-stake testing. It reported results for Language Arts, Science, and Math.

The first thing that struck me was that whether looking at the 50 best or 50 worst, the percent passing math was WAY lower than the other two the majority of the time. That made me scratch my head, so I glanced down at the comments.

Several people noted that AP students didn't take the state test.

I haven't had a chance to dig into it yet, but if true, it makes those reported percentages almost meaningless. "We want to see how your school measures up ... but we're not going to count the top students."

This is why when I see statistics reported, I have next-to-no reaction. Not until I know more about where the numbers are coming from. In broader situations, I ask myself questions like, who was included in the sample? How was the sample selected? How were questions worded?

Be careful when reporting statistics as part of an argument. They may or may not back you up as much as you think. Dig a little deeper to find the whole story.

ETA: Did my own digging-a-little-deeper, and it's actually worse than I thought. The last math courses to participate in the state test are Algebra I and Geometry. The website was reporting on the results of high schools, many of which around here are only grades 10-12. By that time, even the "average" students are past those levels. So the published results only showed the proficiency of the lowest group. No wonder the math percentages were so much worse than the other subjects (which I believe test higher numbers of students in high school).

Have you run into questionable statistics? Any pet peeves on how you see them reported? Do you find yourself completely confuzzled when facing the numbers?