For me, maybe you too, the best part about science is how it disrupts the status quo. A belief, a way of doing something, a paradigm if you will, becomes entrenched. Humans love patterns. We get attached. I call this the way-it’s-always-been-done philosophy. It’s endemic in medicine, and, from what I can see, in education as well.
Take the notion that SAT and ACT scores are all that. Maybe they are…and maybe they are not. Surely the 2 billion dollar testing industry says their tests are important. Testers even say their tests open doors for the under-privileged. The idea appeals to our intuition. The meritocracy of multiple choice tests seems unassailable. Plus, grade inflation is an easy target.
Then a group of scientists dare to ask simple questions about outcomes; they collect and analyze tons of data, and we get a humdinger like this:
In a 70-page study of more than 123,000 young humans who attended 33 colleges or universities, researchers discovered that academic performance turned more on high school grades than ACT/SAT scores. (Here is NPR’s take.)
I love this stuff.
(First a disclosure: I scored below average in reading and verbal on the SAT, but earned above average grades. No doubt, these low test scores limited my choices of colleges. Closed doors, said my father. I was an educational slow-twitcher, a grinder who started slowly, stayed close enough to hang on in the middle and finished strongly at the end, when others tired.)
Regular readers here know that I see parallels between the US education and healthcare systems. Both over-spend and under-perform. Both are dogged by dogma.
Testing is a case and point. In education, the testing behemoth has successfully proselytized the populace into believing their products improve outcomes. They say “more information is always better.”
If you hear that statement from your doctor, it’s time to be frightened.
More tests are rarely the answer.
The lead investigator of the SAT/ACT study struck a nerve when he said the greatest harm of the tests were to the nation: “The tests truncate the exact pool of applicants this country needs to go to college–first-generation college-bound, the under-served and the minority student.”
A likely explanation for this stems from the strong association between SAT/ACT performance and social status and family income. Rich people can afford good schools and test-taking tutors.
Now to medicine, testing and truncating…
The SAT/ACT story grabbed my attention because of the recent push towards value-based performance of doctors. Like students, doctors are going to be graded (and paid) on how they perform on standardized tests. These tests, called quality measures, will count how many boxes are clicked or how many of the ‘right’ medicines are prescribed. (An upcoming post will discuss the ‘rightness’ of these meds.)
You don’t think doctors will learn to perform for their tests, do you?
So you see the problem.
There will never be tests for humanity, courage, curiosity and nuance.
Like the hospitalist who has the courage and common sense to tell a family the truth about their loved one’s cancer–truths not always heard from specialists. Or, which test will identify the bystander willing to step out of the crowd in the ICU?
Truncate is a serious verb. Shortening by cutting off the important things is something the worlds of education and medicine should consider never events, now more than ever.
JMM
P.S. My friend Dr. Wes Fisher has this illuminating look at the conflicts of those in Medicine assigned to look after the scoring of quality and professionalism of doctors.
One reply on “The problem with testing students and doctors is what gets truncated”
John – I think you are right on the mark (as usual) in this post! I remember reviewing too-numerous-to-count medical student applications to our family medicine residency program. As one who had to then present these applicants to our selection committee – I’d strive to extract comments in the application that spoke to a given candidate’s “other non-testable features” (motivation, sense of humanity, conviction, dedication, ability to synthesize and put situations into proper all-round perspective). I’d usually be happy when grades were solid “average” (not necessarily stellar- but also not at the bottom ) in the setting of very good “other qualities”. But of course – there is as of yet (and probably never will be) a “test” to truly assess these most important “other qualities”. The more specific our “testing” becomes – the less it really identifies those qualities we should most be looking for. How ironic …. How sad ….