Kurt Xyst, University of Washington
The technology of predictive analytics has come to academic advising. Marc Lowenstein (2013) opens a thought experiment about a possible future for academic advising with an imagined interaction between a student and an electronic academic assistant able to make judgments on the fly about the curricular validity of alternative courses the student would like to take. Seems convenient, but is this what we want? Is this what academic advising should look like? The application of predictive analytics to the project of improving education raises ethical questions that must be considered before shifting institutional policies or buying into any particular product.
Some have answered yes, this is what we want, or what we should want, because predictive analytics solve a significant problem. “The collegiate advising system . . . is highly inefficient, error prone, expensive, and [a] source of ubiquitous student dissatisfaction,” writes Elizabeth Phillips (2013, p. 48) in a recent issue of Change. In her view, a solution lies in something like Arizona State University’s eAdvisor system. Stepping beyond even Lowenstein’s vision, Phillips claims “eAdvisor identifies the best course sequence, not just the allowable one” (p. 51). This sort of increased convenience powered by hyper-reliable judgment is possible because of advanced data mining that can “match the performance of any individual student to the anticipated success patterns” (Phillips, 2013, p. 49).
These claims by Phillips depend upon an underlying belief that individual performance can be algorithmically matched to future behavior (anticipated success patterns). In turn, this belief is anchored in an assumption that student behaviors—indeed human behaviors—flow regularly from a few specific conditions or qualities. If one can accurately describe those states, and can quantify the relevant forces operating in the environment by gathering lots of data about how others have fared in similar situations, a model can be built that expresses the necessary follow-on consequences for any particular person. Because it adheres to strict and consistent analysis, this type of modeling can do what human educators cannot. One firm that makes this kind of software for higher education says its platform will “help you see around corners and into the future” (Inspire for Advisors - Civitas Learning, 2016, para. 4). While seeing into the future may have a certain appeal, intuitions should look under the hood. Are there grounds for principled skepticism of these claims?
In order to get some perspective on the new promises of predictive analytics for academic advising it may be helpful to compare it to something more long-standing. Consider economics. Economics is a prestigious and highly visible field that takes up matters of human behavior through rigorous, quantitative analysis. Starting with a theory of what humans are and how they behave, then gathering lots of data about current conditions and analyzing those data mathematically, economists are able propose ideas about how people will behave in the future. Leaders in government and industry frequently look to the predictions of economists because in practice the work of economics provides something of a gold standard for the scientific prediction of human behavior.
How accurate are economic predictions? Three recent articles in financial and public affairs publications suggest reason for concern. Business Insider published an article by John Mauldin (2016) reviewing economic forecasts from multiple public and private sources covering the years 2000-2014 which concluded that “economists have no clue about the future” (para. 28). Tim Harford (2014) in the Financial Times reviews a survey of economic forecasts made in the 1990s. He determined that there was little difference between public and private sector forecasts; they all had a terrible record for success. Underscoring his point, Harford (2014) notes that a subsequent study showed that not a single economist predicted the Great Recession. Tamsin McMahon (2014) notes that the Organization for Economic Cooperation and Development (OECD) made a similarly “gargantuan mistake” (para. 2) in forecasting the Great Recession. If something as massive as the Great Recession is invisible to economics, and if the Great Recession was caused almost entirely by complex human interactions—thus arguably spinning out a tremendous amount of data for analysis—how much confidence should we have about computing human behavior on smaller scales where less information is available?
These three somewhat arbitrary sources are clearly not the last word on the subject of economic forecasting. They are offered only to suggest that the gold standard for the scientific prediction of human behavior comes in for heavy criticism of its accuracy—and that such criticism is not hard to find. That there is such criticism ought not to come as a surprise when considered against broader social theorizing like that from Noble Prize-winning economist Friedrich Hayek.
Interviewed about the range and scope of economics Hayek (1983) said, “Our capacity of prediction in a scientific sense is very seriously limited” (section 146) Hayek observed, “because whereas the model of science—physical science, in the original form—has relatively simple phenomena, where you can explain what you observe as functions of two or three variables only” (section 144); the social and human sciences are far more complex. Unlike the relatively simple state of affairs that exist when observing or measuring falling bodies or chemical reactions, human action, even when aggregated in big groups, cannot be patterned well enough nor turned into enough points of data to reliably and accurately predict behavior. From this perspective, the reported failures of economic forecasts demonstrate the consequences of erroneously applying assumptions from natural science to the very different world of human behavior.
Those invested in higher education, and specifically those invested in the form of higher education that occurs in academic advising, must look carefully underneath the polished interfaces and sophisticated marketing that surround predictive analytics. If the argument for use is that predictive analytics identifies not only a likely future state of affairs for a student but the best future state of affairs for that student then there are grounds for principled skepticism. If quantitative prediction of human behavior in a field like economics is frequently wrong, why should we expect quantitative prediction of students in academic advising to be any less wrong?
Kurt Xyst Lead Academic Adviser Undergraduate Academic Affairs Advising University of Washington kxy@uw.edu
References
Harford, T. (2014, May). An astonishing record - Of complete failure. Financial Times. Retrieved from https://www.ft.com/content/14e323ee-e602-11e3-aeef-00144feabdc0
Hayek, F. A. (1983). Nobel Prize-Winning Economist Oral History Transcript/ Interviewer: Leo Rosten. UCLA Oral History Program. Retrieved from https://archive.org/stream/nobelprizewinnin00haye/nobelprizewinnin00haye_djvu.txt
Inspire for Advisors - Civitas Learning. (2016). Retrieved from https://www.civitaslearning.com/technology/inspire/inspire-advisors/
Lowenstein, M. (2013). Envisioning the future. In J. K. Drake, P. Jordan, & M. A. Miller (Eds.), Academic Advising Approaches: Strategies that Teach Students to Make the Most of College (pp. 243-258). San Francisco, CA: Jossey-Bass.
Mauldin, J. (2016, January). These 8 charts prove economic forecasting doesn’t work. Business Insider. Retrieved from http://www.businessinsider.com/8-charts-prove-economic-forecasting-doesnt-work-2016-1
McMahon, T. (2014, February). Why economists can't predict the future. Macleans. Retrieved from http://www.macleans.ca/economy/economicanalysis/why-economists-cant-predict-the-future/
Phillips, Elizabeth D. (2013). Improving advising using technology and data analytics. Change, 45(1), 48-55.
Cite this article using APA style as: Xyst, K. (2017, March). Questioning predictive analytics for academic advising. Academic Advising Today, 40(1). Retrieved from [insert url here]
