Jan 15 2014

When Quantity Reigns


When Quantity Reigns: On Measurement in Higher Education

If we were honest with ourselves, we would admit that Americans have become obsessed with measurement and quantification, in part due to the enormous success of the modern scientific method, which through its investigation of the natural world, has tangibly improved the lives of peoples around the world. The revolutionary edict of the modern scientific method is that every theory, no matter how esteemed, must withstand empirical testing. Testing and verification determine the value of predictive theories (hypotheses), and should a theory fail even one test, it must either be abandoned, or modified and retested.

The American penchant for a quantitative understanding of the empirical world has historical roots. From Europe, we inherited a mathematical method of analysis that has facilitated endless innovations in science, technology, and engineering. The inventors of calculus, Isaac Newton (1642-1727) and Gottfried Leibniz (1646-1716), were products of the Age of Reason, and their new tool for understanding the world reinforced a scientific and empirical view of its processes (thereby revealing its natural laws).

Yet, science, and the measurement and calculation that it employs, constitute only one modality of human knowledge. Because learning is a mental phenomenon, numerical systems designed for analyzing the physical world are not ideally suited to evaluating student learning outcomes (SLOs). As Americans, we rely far too heavily on standardized testing to measure teaching and learning outcomes in primary and secondary education, and increasingly on problematic “rubrics” composed of vague declarations about what students should learn in colleges across the country.

In the state university system where I teach, for example, students taking a course with a Western Civilization attribute are expected to “relate the development of Western civilization to that of other regions of the world” by the end of the semester. Those who finish a course with an American History attribute should gain an “understanding of America’s evolving relationship with the rest of the world.” Simple common sense tells us that these SLOs are too over-generalized to be measured with any statistical validity.

Nevertheless, contractual statements like these are increasingly required on course syllabi, and accreditation bodies are demanding evaluation of them. An imposing administrative machinery has emerged in the last decade to collect and analyze data, then quantify the performance of academic units, including the faculty, vis-à-vis the stated outcomes. Such zealous efforts to find yet another application of the scientific method overlooks the fact that the data acquisition processes are often in violation of fundamental statistical sampling requirements.

Moreover, when they attempt to measure complex phenomena (such as whether students across sixty-four SUNY campuses understand “America’s evolving relationship with the rest of the world”), statisticians recognize a limitation called “reasonable aggregation.” Before statistical analysis can begin, they must conjecture which, out of an essentially infinite combination of quantities, are drivers of a suspected cause and effect relationship or correlation relationship—and how to measure such drivers. If the reasoning is not valid at any stage in this process, then everything else that follows will be flawed as well; the more complex the relationships being measured, the less precise the results.

Ignoring the requirements of statistical sampling, and the limits of statistical quantification, our penchant for making education reducible to numbers is changing the way we think about teaching and learning. Standardized assessment, college rankings, “value-added” indicators, and “student learning outcomes” are just some of the buzzwords associated with this misplaced faith in quantification. These terms obfuscate, mislead, and color public perceptions about the quality of higher education delivery in the United States.

Even President Obama, whose administration has championed the much-maligned Race to the Top program, is proposing that funding for higher education be tied to performance-based data indicators (graduation rates, graduate earnings, degree completion times, and other “metrics”). Back in reality, the Times Higher Education concludes simply, “meaningfully measuring the output of our highly diverse colleges and universities is impossible.”

The imprecise application and ignorance of the limitations in statistical analysis has not stopped other proponents of numbers-based education outcomes from attempts to extend the measure of learning across cultural borders. A new international plan for the “assessment of higher education learning outcomes,” called AHELO, “aims to be a direct evaluation of student performance at the global level and valid across diverse cultures, languages and different types of institutions.”

Putting aside the cultural hegemony inherent in such plans, we ignore the fact that the greater the complexity of the activities being measured, the more difficult it becomes to draw meaningful conclusions and make predictive inferences from the data. Noisy unstructured data presents serious statistical challenges, and attempts by AHELO to create a global assessment plan for higher education should not go unscrutinized.

So, why do we allow educrats in ballooning administrative ranks to manufacture dubious statistical data and use it to make decisions that affect faculty, staff, and students? Why do we allow a culture of assessment to cast us into the mill of outcome-driven instruction and learning?

As the introductory paragraph was meant to suggest, the answer may be found in our own history. The Puritans left us with a pragmatic business ethos that equates prosperity with righteousness. Enlightenment thinkers like Charles-Louis de Montesquieu (1689-1755) and John Locke (1632-1704) fired the American quest for liberty in the eighteenth century and infused the founding documents of the United States with a burning idealism, which in time would be transformed into a myth (of the American Dream) predicated on material acquisition and self-reliance as a measure of success.

In the following Romantic Period, writers began to interrogate the imposition of rationality that emerged from corporate industrial forces, which these individuals rightly foresaw as threatening to restructure society according to the values of business and commerce. Perhaps none of them was more vehement in attacking the limits of empiricism and ratiocination than the poet William Blake (1757-1827).

At the dawn of the Industrial Revolution, Blake denounced the “dark satanic mills” that were polluting the cities of England. He inveighed against the eclipse of imagination, intuition, and emotion by ratiocination—an impulse that divides the world into “ratios” and limits perception (as if seeing “all things thro’ narrow chinks” of a cavern).

In an observation relevant to the current crisis in higher education, Blake notes the tendency to “bring out number weight & measure in a year of dearth.” In other words, when resources are scarce, as they remain in the wake of the Great Recession of 2008, we resort to quantification to distribute them. For this reason, we devise ways to measure one institution against another, and then make them compete for funding.

If we want meaningful reform in higher education, we should reject educational policies founded on numerical approximations of complex human interactions. A holistic view of the art of teaching and the organic process of learning includes numerical analysis, but as only one way to understand how learning is happening on college campuses across the nation.

We need to bring our hearts and imaginations back to the effort to reform higher education in the United States. Creative thinking should inform our educational policymaking —not just quantification, business models of organization, and more assessment. Otherwise, we risk allowing flawed systems of measurement, and the standardization that results from them, to stultify the minds of our citizenry. No doubt, we can do better.

Demand more tenure-track appointments at your college or university! Make the choice to attend an institution that invests in you by investing in the faculty!



Copyright © 2014 Mark S. Ferrara, All rights reserved.