Reimagining Assessment

Falsehoods We Believe about Assessment

This is a list of things we (educators) commonly believe about assessment that are falsehoods. It is not a list of what we say we believe about assessment, but what our actions, systems, and policies reveal about what we believe about assessment. It is a list of revealed assumptions that have been shown to be false, inadequate, or unethical, and yet we still—all of us—behave as if they were true, much of the time. The format is based on the very popular (in some circles) Falsehoods Programmers Believe about Names, which led to a sub-genre of Falsehood lists mostly around programming.

The purpose of this list is not to castigate anyone, or to pat ourselves on the back, but to lay those assumptions out in as clear a way as possible so that they can be interrogated. Thanks to Carisa Corrow for the assisting conversations.

• Assessment tells you what a kid knows or can do

Possibly the most important thing to know about assessment, even in the best of conditions, is that it can only give you answers to the questions you ask. The absence of epistemic humility in assessment is probably the overarching theme of this list. Assessment has always been overdriving the headlights, but the ubiquitous struggle during the pandemic to figure out how our kids are doing and what they know and are able to do isn’t new. It’s always been a struggle. It’s just that before the pandemic we shrugged and said, “Well, whadda ya gonna do?”

Here’s the thing, even in the best of conditions, traditional assessment has only explored the narrowest dimensions of what it means to be a person—the dimension that exists only within our classroom and administrative field of vision. To paraphrase vlogger Hank Green, kids create skills and knowledge every moment of every day, but they are only acknowledged for the skills and knowledge that can be captured—and in a traditional setting, only skills and knowledge that can be conveniently captured. The calls for continuing standardized tests during the pandemic are ludicrous because, rather than asking, “What have kids learned?” assessment asks, “How much of this very limited list of things have kids learned?

Kids have learned a lot this year, about medicine, health, work force, politics, capitalism, epidemiology, probability, the use (and misuse) of statistics, rhetoric, propaganda, and Zoom. This shouldn’t be remotely controversial. They’ve also learned about themselves, their family, their capacities, their boundaries, their fascinations, their friends, city, state, and country. Assessment doesn’t tell us what kids know and are able to do because kids are not being asked what they know and are able to do.

Finally, just to be clear, how a kid does on an assessment task tells you how they did on that task at that moment. There is some learning about a kid’s capacity—if a kid does a particular math process then they can do that process—but so much else that we draw from assessment is unsupportable inference. To say, “This kid did not do this problem at this time, therefore they do not know how to do this problem” is risibly bad thinking.

• Assessment is aligned to standards
• Assessment is aligned to what the community thinks kids should know and be able to do

We assert that assessment is aligned to standards, but it’s really nothing more than that, an assertion. We have a ridiculous number of standards. Only a portion of them can be covered in the time allotted, and only a portion of what’s covered will be learned. And even that coverage is ill fitting. If you start with the unit (“It’s the Civil War! What standards should I cover?”) rather than the standard (“Historical Conflict. What events in history could illuminate that standard?”) you will inevitably be coming at the standard from the side.

And are these even the standards we want? In every Portrait of a Graduate gathering I’ve ever been to, the things community members and business leaders call the most important—collaboration, communication, creativity, etc.—are not present in the standards, except as honored but ignored front matter. They are sometimes called “school wide” or “cross curricular standards,” and those are words in English that have meaning, but effectively they mean that everyone is “considered” responsible for them, and no one is responsible for actually assessing them.

• Teachers give assessment
• Students take assessment

I object to this construction because the give-take dichotomy is far too simple and diminishes both the adults’ and kids’ experiences. You may say, “That’s just semantics!” I will reply, “It is semantics, but not just semantics.” Teachers do assessment. Kids do assessment tasks, the results of which are analyzed by themselves or peers or the teacher or some psychometricians in a bunker in the Nevada desert.

• Valid assessments let you know the same thing about the learning of different kids
• Good assessments are indifferent to context
• Context invalidates good assessments
• Good assessments clearly show how much a kid has learned
• Good assessments allow you to compare one kid to another

I understand the idea of comparability and its value in school accountability, but the idea that you can give a roomful of kids the same task and that you will learn the same thing about different kids, with only a variation in how much is learned (92% versus 83% or 79%) is almost self-evidently ridiculous. That’s not how learning works. But our assessment system is built on the idea that learning can be divided into discrete units that can be meaningfully accumulated and computed with both precision and accuracy. Yes, we talk a good game about “body of evidence,” but the only REAL test is the one that is given to every kid in the same place on the same Saturday morning at the same time. To the extent that assessment is indifferent to context, it is valid. To the extent that context impacts performance, it is invalid. Those are the falsehoods that undergird our systems, based in the myth of objectivity—which isn’t “objective” so much as it is White, middle-class subjectivity being cast as the universal norm.

• Good assessments produce a number as their data point

See Falsehoods We Believe about Grading and Reporting for more on this one.

• Assessments can be either summative or formative
• Formative assessments are less important than summative assessments
• Formative assessments are small; summative assessments are big

There is a common misconception that formative assessments (“formatives”) are the small quizzes, while summative assessments (“summatives”) are the big tests and projects. In one sense any assessment that you learn about your learning from (and can then act upon that learning) is formative, while any assessment that is recorded as a snapshot of “how you did” is summative. (“The moment I write something in my gradebook it becomes summative,” one mentor told me in my first teaching year.) In another sense (the sense that I prefer) there are no summative assessments or formative assessments, just assessments that you use formatively or summatively. And even there, I’m suspicious, because who decides which assessments “count?” The best compromise I can think of is that you and the student gather a body of evidence, and then—when you must draw the line at the end of the year —you make a summative decision.

• Assessment and accountability are the same thing

When you’re talking with people at the district and state level you may think you are talking about assessment systems, but the conversation often mysteriously begins to be about something else: accountability. NWEAs, Regents, SATs, etc. are a part of a lot of systems’ accountability processes (and, for the record, I think accountability is great), but they are only incidentally effective assessments of kids’ learning, and only if they are intentionally used that way.

• What you report on is what you assess
• What you assess is what you teach
• What you teach is what kids learn

The certainty with which this chain of statements is made should lead you to be skeptical before you even consider their content. I include these as the last of our falsehoods about assessment because, though this is a mantra every teacher learns, it ignores so much of what goes on between us and our kids, and how much is going on with our kids outside of our field of vision. The idea that “what kids learn” can be boiled down to “what you teach” is egregiously diminishing. This brings us back to the very first falsehood. It’s possible to leave so much out of our assessments of a kid that—even if the remaining pieces are true—the overall picture is false. And if you do it knowingly, it’s a lie.

Photo at top by Allison Shelley for American Education: Images of Teachers and Students in Action.