How your kid’s standardized tests are scored

How your kid’s standardized tests are scored

Yes, yes, yes! Ruth Ann Dandrea writes here about the writing tests her 8th grade students are obliged to take each year. I’d be curious to know more about the 150 page info packet that’s given to teachers, but I can tell you from experience that students and teachers should cooperate in obstructing this takeover of education! And I can also tell you more about how the scoring works.

the liability of standardized testing, by holly henry.
the liability of standardized testing, by holly henry.

See, in late spring 2007, I left my regular job in a small town ($7.75/hr) for a lucrative temp job ($11.50/hr) as a standardized test “reader” or “scorer.” In the world of educational measurement, we worked long hours at desks set side by side in temporary warehouses. A supervisor watched from behind as we read handwritten essays or graded scanned essays on a screen; bowls of candy and pots of coffee stood nearby to keep us awake. Each project had to be done as quickly as possible: two weeks grading open answer responses for second grade math in Iowa, followed by a week of short-answer social studies responses by fifth graders in Michigan, and then all the 8th-grade creative writing essays from South Carolina students. We were always pressured to score faster. Get more done. Finish the project on time.

At “Ed Measurement, Inc,” our managers wanted to make this a comfortable and effective job, but pressure from above to keep making money kept working conditions fairly dim. We worked long hours and weekends, screened off from our deskmates in windowless rooms, and clicking as fast as possible. We weren’t supposed to talk or converse with others, although we did slip notes across the tables listing the funny stuff kids said. We could easily be fired for not being productive enough, scoring fast enough, not matching other workers’ scores, or showing up late (but at Ed M, the managers were pretty lax, and treated us with respect when possible). We coped with our uncertain and hurried life by gossiping and eating candy. When we were all ushered out of the test-scoring room, together, we’d hustle around the building for fresh air on our 10-minute break. Lunch was a short half-hour affair, just enough time for a microwave meal or fast food.

Each essay or response was graded according to a simple rubric by two “scorers” (we didn’t know who in the room was the other scorer, and if they would be generous or a stickler for the rules; this kept us in line). If our answers on the 1 – 4 or 1 – 6-point scale differed by more than a point, the image would be sent to our boss to be re-scored. We could be reprimanded or fired if we got too generous with the points for a particularly clever kid.

As Ruth Ann describes, these rubrics are extremely numerical, and don’t allow for much creativity. I recall one social studies question, broadly, that asked students about the Harlem Renaissance and northward migration of African Americans in the early 1900s. The students had to use the words “south to north” and “rural to urban” (or “country to city”) to get all four points on the latter question. Literally: they had to write those four words. If they used descriptive narrative or clearly knew their stuff, but didn’t include the right “factual” words, we were required to give them 0 points on the question.

I can’t find diary notes on my computer, but did find an old letter from my temping friend, Bekka*; she sent this after I quit to work for the competition (which offered a few more weeks of employment) 🙂

Hey there, Sauley*! How’s it going? We finally finished up the 8th grade essay project on 5/23, well past the deadline, I believe. Our productivity seriously declined when certain individuals (who shall remain nameless) abruptly quit the project to work for the competition. One such individual even had daily packet counts as high as seventeen!![**] haha…

Seriously, you didn’t miss a whole lot, but I’m guessing you know that already! Just a few thousand more essays about Six Flags, “the big game,” and “the spring fling,” exactly like what you were reading before you left. Carmela* was mostly cool, then she’d get some unpleasant stats and we’d know it because she’d shoot these dagger looks at people who laughed at funny essays (disruptive) or sigh, “Can’t you read just a LITTLE faster?” The next day, she’d feel bad and present us with a bundt cake…

I think the project was really dragging on and she was ready to go home to her cats. . . On the last day, we presented her with a bunch of cat toys as a send-off and she made us this really nice breakfast. Someone even brought in chocolate chip pancakes (and fake million dollar bills) in honor of the essays! (IT WAS A GREAT DAY!!)

Other than that, we continued to raffle off one another’s junk (which we started when you were still there, I believe) the walkers continued to walk, and the smokers continued to smoke…

And that’s the official XXX news! I hope things are going well at XXXXXXX, and have a great summer!

As Bekka suggests above, one of our most entertaining (and mind-numbing, depending on the student) projects was grading 8th grade creative writing exercises, in which students were told to “describe the best day of your life.” Students had clearly been prompted, as they wrote again and again about a vague day at the amusement park, the scripted excitement of getting a pony for one’s birthday, or a routine 20 dollars on the street. Sometimes all three together. In one day. The grading rubric was more fuzzy, so I don’t remember the details.

Ed M was a great company, relaxed but productive, and I made a lot of friends; but they lost their contracts to the second company (starts with Pear- and ends with -son…), a multinational monolith which ran us hard, maximized the money they made, and discouraged interaction of any sort between those of us working there.

And the workers. At both companies, the temps were all required to be college graduates, often quite talented but at the edges of the work world: two girls who had chosen “art” careers and needed money while making jewelry or painting; a teacher in between jobs who slipped out for interviews with the state board of education; an older woman with cancer from my grandmother’s church, who had lost her job and needed money; a quiet lawyer who’d had a stroke and couldn’t move his face, but was still quite perceptive; a grandmotherly Latina from South Texas who worried about the treatment of her ‘white’ and ‘Latino’-looking sons, though all were from the same father; a soccer mom with an economics M.A. who wanted to afford a short vacation for her sons; and long-term temp workers who couldn’t find regular work. Most difficult for me, I sat beside the deeply good father of one of my dear friends, a senior manager who had been laid off at 50 and was in the process of losing his house, age discrimination limiting his opportunity (but not ability) to support his family in any substantial way. The men older and faded, the women unemployed or mothers; skilled college graduates at the fringes of our economy, solidly middle class, but sliding…

We were of course all laid off a few weeks later, after the last project was turned in and the companies received their money, passed it on to their stockholders.

Temporary workers and the unemployed, I would guess, approach 30% of the U.S. workforce at this point. Many are well-educated, savvy, and hardworking. And remembering them, I can’t help but envision the precarious future of these young students. Now they’re busy with bent pencils and computer keys in describing the surprise of 20 dollars found *just lying there* on the walk outside of Walmart. Now they’re stooped over a stack of papers, trying to please and charm a scorer who can’t deviate from the rubric. But soon they’ll be grown and working and studying and working some more for just another $20, and just another $20, and just another $20…

"Found 20," by See El Photo.
“Found 20,” by See El Photo.

* Pseudonyms all, and general overview only of one 8-year-old test question.

[**] As Bekka suggests, *some people* could go to the bins and select “remainder” packets with only a handful of handwritten essays, rather than the full 20-essay packets. A great lesson in meeting worker metrics.

Thoughts? Leave a note here!