Getting What You Asked For

Sometimes I draw inspiration from the strangest of places!

In this instance, it was reading about how some well-intended school testing backfired on those who'd introduced it. In their fascinating book, Freakonomics, economists Steven D. Levitt and Stephen Dubner describe the results of some deep analysis that was done on 3rd to 7th grade testing within the Chicago Public School system around the end of the 20th century. Even prior to President Bush's No Child Left Behind initiative, the CPS had instigated city-wide testing in order to ferret out the lowest-performing schools. The worst offenders faced suspension or even closure, and individual teachers could see rewards and punishment ranging anywhere from promotions to demotions or even job termination as a result of how their students did. All of which makes for a very interesting incentive scenario, indeed!

Levitt and Dubner make a good case for how such a setup can often cause the exact opposite effect compared to what had been intended. I'd written of something similar in the 1st AgileMan book (Issue # 22, Who's Measuring What Now?), where Call Centre agents were graded by how quickly they got through each call... thereby encouraging some of them to hang up on the callers! In the case of the Chicago Public Schools, what a certain (non-trivial) percentage of the Chicago teachers did as a result of the new standardized testing wasn't any more surprising: they cheated!

Specifically, these counter-incented instructors found ways to make their classes' results better than they should have been. That included prepping their classes exclusively for what would be on the tests, giving them extra time, helping them figure out the right responses during the tests, or even changing the results on individual student's answer sheets! Freakonomics provides a highly entertaining recounting of how the cheating teachers were found out (hint: it involved some awesome data-mining!) but what's relevant here is the lesson that can be learned about how you really do need to align your incentives (positive or negative) with your over-arching goals.

The school testing was put in place to improve the education system in Chicago, but because it was implemented in a less-than-perfect way, many of Chicago's elementary school teachers were being encouraged to behave in a manner that actually made things worse! Not only were some of the struggling students, teachers and schools not being identified (thanks to the cheating), but it was also bringing out the worst tendencies in many of the people charged with molding the next generation of Chicago-ites. Just imagine what students learn while watching their role model cheat on their behalf on a city-wide test! That's the kind of damage that's hard to undo!

A better incentive plan, I have to assume, would be one where the measurement is both better-monitored (in order to reduce the opportunity for cheating) and more targeted toward helping, rather than punishing, those found wanting (thereby removing the motivation to cheat). Of course, it's always easier to "Monday morning quarterback" other peoples' mistakes than it is to get it right yourself!