When Quantity Reigns

Admin

When Quantity Reigns: On Measurement in Higher Education

If we were honest with ourselves, we would admit that Americans have become obsessed with measurement and quantification, in part due to the enormous success of the modern scientific method, which through its investigation of the natural world, has tangibly improved the lives of peoples around the world. The revolutionary edict of the modern scientific method is that every theory, no matter how esteemed, must withstand empirical testing. Testing and verification determine the value of predictive theories (hypotheses), and should a theory fail even one test, it must either be abandoned, or modified and retested.

The American penchant for a quantitative understanding of the empirical world has historical roots. From Europe, we inherited a mathematical method of analysis that has facilitated endless innovations in science, technology, and engineering. The inventors of calculus, Isaac Newton (1642-1727) and Gottfried Leibniz (1646-1716), were products of the Age of Reason, and their new tool for understanding the world reinforced a scientific and empirical view of its processes (thereby revealing its natural laws).

Yet, science, and the measurement and calculation that it employs, constitute only one modality of human knowledge. Because learning is a mental phenomenon, numerical systems designed for analyzing the physical world are not ideally suited to evaluating student learning outcomes (SLOs). As Americans, we rely far too heavily on standardized testing to measure teaching and learning outcomes in primary and secondary education, and increasingly on problematic “rubrics” composed of vague declarations about what students should learn in colleges across the country.

In the state university system where I teach, for example, students taking a course with a Western Civilization attribute are expected to “relate the development of Western civilization to that of other regions of the world” by the end of the semester. Those who finish a course with an American History attribute should gain an “understanding of America’s evolving relationship with the rest of the world.” Simple common sense tells us that these SLOs are too over-generalized to be measured with any statistical validity.

Nevertheless, contractual statements like these are increasingly required on course syllabi, and accreditation bodies are demanding evaluation of them. An imposing administrative machinery has emerged in the last decade to collect and analyze data, then quantify the performance of academic units, including the faculty, vis-à-vis the stated outcomes. Such zealous efforts to find yet another application of the scientific method overlooks the fact that the data acquisition processes are often in violation of fundamental statistical sampling requirements.

Moreover, when they attempt to measure complex phenomena (such as whether students across sixty-four SUNY campuses understand “America’s evolving relationship with the rest of the world”), statisticians recognize a limitation called “reasonable aggregation.” Before statistical analysis can begin, they must conjecture which, out of an essentially infinite combination of quantities, are drivers of a suspected cause and effect relationship or correlation relationship—and how to measure such drivers. If the reasoning is not valid at any stage in this process, then everything else that follows will be flawed as well; the more complex the relationships being measured, the less precise the results.

Ignoring the requirements of statistical sampling, and the limits of statistical quantification, our penchant for making education reducible to numbers is changing the way we think about teaching and learning. Standardized assessment, college rankings, “value-added” indicators, and “student learning outcomes” are just some of the buzzwords associated with this misplaced faith in quantification. These terms obfuscate, mislead, and color public perceptions about the quality of higher education delivery in the United States.

Even President Obama, whose administration has championed the much-maligned Race to the Top program, is proposing that funding for higher education be tied to performance-based data indicators (graduation rates, graduate earnings, degree completion times, and other “metrics”). Back in reality, the Times Higher Education concludes simply, “meaningfully measuring the output of our highly diverse colleges and universities is impossible.”

The imprecise application and ignorance of the limitations in statistical analysis has not stopped other proponents of numbers-based education outcomes from attempts to extend the measure of learning across cultural borders. A new international plan for the “assessment of higher education learning outcomes,” called AHELO, “aims to be a direct evaluation of student performance at the global level and valid across diverse cultures, languages and different types of institutions.”

Putting aside the cultural hegemony inherent in such plans, we ignore the fact that the greater the complexity of the activities being measured, the more difficult it becomes to draw meaningful conclusions and make predictive inferences from the data. Noisy unstructured data presents serious statistical challenges, and attempts by AHELO to create a global assessment plan for higher education should not go unscrutinized.

So, why do we allow educrats in ballooning administrative ranks to manufacture dubious statistical data and use it to make decisions that affect faculty, staff, and students? Why do we allow a culture of assessment to cast us into the mill of outcome-driven instruction and learning?

As the introductory paragraph was meant to suggest, the answer may be found in our own history. The Puritans left us with a pragmatic business ethos that equates prosperity with righteousness. Enlightenment thinkers like Charles-Louis de Montesquieu (1689-1755) and John Locke (1632-1704) fired the American quest for liberty in the eighteenth century and infused the founding documents of the United States with a burning idealism, which in time would be transformed into a myth (of the American Dream) predicated on material acquisition and self-reliance as a measure of success.

In the following Romantic Period, writers began to interrogate the imposition of rationality that emerged from corporate industrial forces, which these individuals rightly foresaw as threatening to restructure society according to the values of business and commerce. Perhaps none of them was more vehement in attacking the limits of empiricism and ratiocination than the poet William Blake (1757-1827).

At the dawn of the Industrial Revolution, Blake denounced the “dark satanic mills” that were polluting the cities of England. He inveighed against the eclipse of imagination, intuition, and emotion by ratiocination—an impulse that divides the world into “ratios” and limits perception (as if seeing “all things thro’ narrow chinks” of a cavern).

In an observation relevant to the current crisis in higher education, Blake notes the tendency to “bring out number weight & measure in a year of dearth.” In other words, when resources are scarce, as they remain in the wake of the Great Recession of 2008, we resort to quantification to distribute them. For this reason, we devise ways to measure one institution against another, and then make them compete for funding.

If we want meaningful reform in higher education, we should reject educational policies founded on numerical approximations of complex human interactions. A holistic view of the art of teaching and the organic process of learning includes numerical analysis, but as only one way to understand how learning is happening on college campuses across the nation.

We need to bring our hearts and imaginations back to the effort to reform higher education in the United States. Creative thinking should inform our educational policymaking —not just quantification, business models of organization, and more assessment. Otherwise, we risk allowing flawed systems of measurement, and the standardization that results from them, to stultify the minds of our citizenry. No doubt, we can do better.

Demand more tenure-track appointments at your college or university! Make the choice to attend an institution that invests in you by investing in the faculty!

 

 

Copyright © 2014 Mark S. Ferrara, All rights reserved.


3 Responses to “When Quantity Reigns”

  • Astroguy Says:

    Great post! But while I largely agree with your reasoning, as a scientist, I find that modern society also suffers from the opposite problem — not enough GOOD empiricism. Its easy to play fast and loose with the numbers and cherry pick data to suit peoples pre-determined conclusions (e.g. Faux News!).

    I have always taken Mark Twain’s (Disraeli’s?) dictum about lies, damn lies and statistics, not as a slam on trying to quantify things, but on how people go about doing it. You can use statistics to prove anything, if you choose the right skew angle approach, and you ignore data to the contrary.

    A particular danger is the correlation fallacy. The crawler on MSNBC the other day reported that high school students who don’t get enough sleep get lower grades. So do I make sure that my kid gets enough sleep so she gets good grades, or should I push push her to study harder and improve her grades so she sleeps better (which is good for her health)? Which is it? This is perhaps too facile an example (and resolvable with common sense, ahem), but its all too common an error.

    If I read you correctly, there is nothing wrong with testing so long as you acknowledge the limitations in the data. This is, of course, contingent on the “experimental design” of the test. What quantity is being used as a proxy for student learning and what are its likely dependencies on other influences? In principle, perfectly good metrics for this could be developed, and I suspect there are some excellent scholarly studies on how to do this. But realistically there are too many inter-dependencies to make the problem tractable without strong qualifications. Nevertheless, short cuts are taken in choosing what set of parameters stands in for student progress, placing severe constraints on a clear cut interpretation of the data, resulting in overblown conclusions of what constitutes learning.

    In other words, it depends on how the SLOs are defined. I completely agree that outcomes like “understanding of America’s evolving relationship with the rest of the world” are vague (can anyone at all claim to understand it?) and therefore impossible to quantify. However, if in a Quantum Mechanics class you specify that by the end of the semester, students will be able to solve the Schrodinger equation, you can evaluate that. The old-fashioned way, give them a test! You can tell pretty quickly if they can do it or not and the score is a pretty good indicator. In fact, the pressure on science teachers these days is to back away from tests and to use subjective measures since they are more “human”. Of course, there can be all kinds of very human reasons why students do poorly on exams: test anxiety, lack of background preparation, indifference, not enough sleep, … A good empiricist educator recognizes this and when a particular student fails an exam, looks at all the possible factors influencing her performance and only then prescribes a remedy. Standardized tests wash all this away.

  • blackhat Says:

    I really like your writing style, fantastic information, regards for posting :D. “Much unhappiness has come into the world because of bewilderment and things left unsaid.” by Feodor Mikhailovich Dostoyevsky.

Leave a Reply

*