By James Comans

Standardized testing is a huge issue in American education today. Big Testing companies are so heavily involved in influencing policy that sooner or later, Mississippi parents, educators, and lawmakers will have to discuss it. For real. Right now, we’re fighting over funding for our schools, but eventually, we’re going to have to talk about The Test. When it comes to student assessment, instruction, accountability schemes, proposals of merit pay- all of these are affected by standardized testing. It’s the elephant in the room.

I’m not writing about that today. I’m just recommending a book. It’s called Making the Grades by Todd Farley. In it, Farley gives a firsthand account of what he calls his “misadventures in the standardized testing industry.” To parents, teachers, school administrators, lawmakers- anybody in Mississippi- who is interested in talking about how we can change the testing culture to better serve our kids: this book is a must-read. It’s entertaining, but more importantly, it provides valuable inform as to how the testing industry in this country works, from the point of view of a person paid to score student tests.

#1 – It’s funny.

Most notably, Making the Grades has a sarcastic-yet-fresh tone. Farley is not a professional educator. He takes his first job as a standardized test scorer simply to pay the bills, which gives his commentary throughout a funny and insightful “outsider” perspective.

Take this description of a training session, in which a testing company rep, Maria, is training a roomful of scorers with a practice essay. In helping them to find the right score on a 1-6 scale, she and the scorers get lost in semantics. One can almost feel the sighs and eye rolling from beyond the page:

…”Practice Paper #2, everyone. Upper half or lower half?”

A confident chorus of “uppers” filled the room, as Practice Paper #2 was a lengthy and eloquent essay. There was no doubt it would receive a high score.

“Excellent,” Maria said. “It sounds like we’re all ‘uppers’ on this one, which is right. Definitely a 4, 5, or 6.”

The room nodded in happy agreement, no question about it. Maybe we were starting to figure this out.

“OK,” Maria continued, “so is this essay ‘upper upper,’ ‘middle upper,’ or ‘lower upper’?”

“Oh, brother,” a voice trailed off.

Although we knew what Maria meant, no one really said anything.

“People,” she goaded us. “‘Upper upper,’ ‘middle upper,’ or ‘lower upper’? What do you think?”

This time we did reply, the bunch of us blurting out a muddled mess of “upper-uppers,” “middle-uppers,” and “lower-uppers,” no single phrase any louder than the rest.

“I’d say it’s a middle-upper,” a smiling guy offered, ” but a lower-middle-upper, so I’m giving it-“

“No way,” a woman answered, joining the fun. “How could you call that lower-middle-upper? This essay is clearly upper-middle-upper.”

“People…,” Maria warned.

She’d lost us entirely. Our brief, shining moment of standardization was gone as quickly as it had come.

#2 – It’s real.

Another enjoyable aspect of Making the Grades is its relatability. Farley doesn’t sugar coat his accounts. Front and center in this book is the fact that these scorers are all human beings who get tired, stressed, bored.  He wants the reader to feel that and to understand how much of a role it plays in standardized testing when humans are involved, even when they’re properly vetted and trained.

For example, here’s one part in which Farley finds a loophole that allows him a break from hours upon hours of scoring student essays:

I took relief from the tedium wherever I could find it. Occasionally, when the gods were really smiling on me, I would pull the essays out of an envelope to find the entire packet blank, 20 student responses without a single word written on them! (In those cases, whole classes had either missed or skipped the test, but their scores of “blank” still needed to be recorded.) Those packets full of blanks were a salvation for me: If I was expected to score a student response every two minutes, that packet of 20 blanks represented 40 minutes of work. Basically, finding one meant I could sit at my desk and stare off into space for nearly three-quarters of an hour, fantasizing about Shawn Black or the next soccer World Cup, thinking about anything except those darned kids. For me, discovering those packets was like finding Willy Wonka’s Golden Ticket.

“Scott,” I’d whisper, flashing him the packet of blanks, “sweet relief!”

“Lucky bastard,” he’d mutter, showing me his own packet, page after page filled with student words that he actually had to peruse.

#3 – It’s important.

Finally, there is more to Making the Grades than just humor and engagement with Farley’s story itself. Farley reveals valuable information as to how the standardized testing industry actually works- and in some cases, how it doesn’t work. From loopholes to games, to senseless management and unintended consequences, every chapter is eye opening in a different way.

Watch Farley and his fellow scorers discover loopholes in the system, then gradually succumb to turning those loopholes into their own regular, normal shortcuts.

The other real reprieve came when I found someone else’s completed score sheet in my envelope of essays. Each time a scorer finished a packet, he or she would put the essays and completed score sheet back into the envelope and return it to their table leader. The table leader was supposed to pull out that first score sheet before assigning the packet to a second reader. Occasionally, however, amid all the piles of envelopes in front of them, the table leader would forget to take out the completed first score sheet, and it would end up in the hands of the second scorer. It didn’t happen a lot, but it happened enough to give me hope.

The first time I pulled out a score sheet that had been completed by someone else (in the beginning of the project), I felt guilty. I knew very well I wasn’t supposed to see what the first scorer had given to those essays, so I took the sheet up to Shawn and immediately turned it in.

“Sorry,” I told her, as if I had done something wrong.

The next time it happened, I was a little less naive and a bit more bored. I may have taken the score sheet back up to Shawn, but I managed to memorize the first five scores penciled in before I did so: 44543. As I walked back to my place at the table, I chanted those numbers in my head like a mantra, 44543, 44543, 44543. Quickly checking the first five essays in my packet, I discovered that 44543 would work. I may not have completely agreed with each of those scores, but they were close enough. I penciled in my own 44543.

By the end of the project, finding a completed score sheet had exactly the same effect on me as did discovering a packet of blanks. After a couple of weeks of work, my life had become nothing but essays and ennui, so when I found a completed score sheet I took the easy way out: I copied. I wouldn’t even go through the pretense of reading the essays, wouldn’t even glance at them. I would simply copy all 20 of the scores from the completed score sheet on to my second score sheet, trusting the poor fool who’d had to read all those essays had actually assessed them “right.” Ultimately, I treated the discovery of completed score sheets as if they were winning lottery tickets…

…I knew copying the scores wasn’t exactly right. It was clearly a kind of cheating, and it would artificially inflate the reliability numbers. But I also considered the whole project so nebulous that I couldn’t imagine my shortcut was hurting anyone, either- it didn’t occur to me that whether I slapped a 3 or a 4 down on an essay might actually adversely affect a student…so I copied away, copied away, freeing myself up each time for forty more minutes of daydreaming.

Now- I realize Farley’s book is just one man’s account. It’s not to say definitively that what he experienced goes on everywhere, in every testing situation, or that everyone would make the decisions he did. But it does show firsthand why the industrialization of assessment in its current form is flawed: When boxes and boxes of student responses have to be processed, scored, and certified, there is less and less of a guarantee that any one kid’s work will be given the fair shot it deserves. We have to fix that.

But you don’t have to take my word for it! Here’s a link to a free excerpt. Read the first two chapters of the book yourself right now!

Excerpt from Making the Grades by Todd Farley:

Preface & Chapters 1 & 2

James Comans is an 8th grade science teacher in Southaven and contributor to MSEdBlog. His views are his own and do not represent the views of any other entity.