Peer Review

Examining the Outcomes of the Bringing Theory to Practice Project

David Tritelli introduced the Bringing Theory to Practice (BTtoP) project to readers of Liberal Education in the Winter 2007 issue with the following narrative (Tritelli 2007):

Developed in 2003, by AAC&U and the Charles Engelhard Foundation, the project is … gathering evidence of measurable and replicable outcomes that link specific forms of engaged learning—mainly service-learning and community-based research experiences—to behavioral choices and to student development. [It] explores the key questions about engaged learning through a focus on certain prevalent patterns of student disengagement, including substance abuse and depression.

Through multiple conferences, workshops, publications, commissioned studies, research within and across seven national demonstration sites, and the support and study of nearly forty campus programs, the BTtoP project has moved from the articulation of a hunch to the establishment of a hypothesis that has now gained evidentiary support: There are measurable and replicable outcomes that link specific forms of engaged learning to student mental health and behavior and to students’ civic engagement. In this respect, confirming outcomes reveal not only a meaningful approach to addressing patterns of disengagement by students (depression, substance abuse, academic disengagement, and civic disengagement) but also the complexity and interrelationships among the core purposes of liberal education.

Evaluating BTtoP Work across Campuses

This issue of Peer Review focuses on how several of the campuses engaged in the BTtoP project are working at deeper levels of framing, then describing outcomes: documenting effects, some affects, some dispositional, some “soft,” some “hard.” By doing so, they are beginning the task that is absolutely central to our understanding of the core purposes of liberal education and to whether or not we, and our students, are achieving those core purposes. The articles herein come from those leading the BTtoP demonstration sites and from those evaluating the work on and across campuses. They move the discussion from the general to the particular, from the more abstract to the transferable level. The articles begin to translate outcomes and to show institutions what a deeper level of attending to them might involve.

The grammar of “inputs” and “outcomes” is borrowed from mechanistic or causal models and may not be as applicable to education as one would think, looking at the higher education literature of the last twenty years or more. Many of us can recall the admonition to our campuses that our attempts to determine our quality and that the rankings of our institutions provided by others were based solely on inputs such as student high school credentials, standardized test scores, faculty size, PhD percentages, and salaries paid. How these inputs are related to what we actually do (as we have students under our influence for four years) rarely moved beyond conjecture. Some of our institutions have claimed (without any more than the occasional apocryphal example) to “take students further” than even the most prestigious institution, taking students with lower inputs and carrying them to higher levels. After all, the reasoning goes, how can the prestigious institutions screw up when they start with great students? They say that while the prestigious institutions may move the student somewhat in a linear progression, the impact of their institutions is categorical.

Measurements, metrics, and qualitative and quantitative indices of what actually happens to students became a common request. We needed to know more than inputs; we needed to know the outcomes. The clamor for such evidence—which would justify the greater expenses of higher education and justify the individual institution’s claim that it does influence what happens to students in ways that are beneficial to the students and to society at large—resulted in much attention being given to measurable outcomes, such as the percentage of graduates, persistence rates, job placements following graduation, and graduate and professional school placements.

Three Distinct Emphases

Currently, there are, in my view, at least three distinct emphases found in the conversation regarding outcomes. Each has its own voice, proponents, and interest group. First, there is the politically charged emphasis regarding why outcome measurements are needed—namely, the judgment that there must be greater accountability in higher education. Typical reasons for demanding outcome measurements include the perception that the undergraduate experience is of low quality (on what this judgment rests remains unclear), that graduates are not prepared for the workplace, and that our higher education institutions have been too long receiving public financial support without real accountability. For proponents of this judgment, a prescription appears to follow: Universal assessment tools should be used to hold all institutions receiving any form of public financial aid accountable by comparatively measuring a particular set of quantifiable outcomes.

The second emphasis has sought to justify “surrogate outcomes”—what some would claim as the push to value what we measure rather than measure what we value. Some institutional research offices, data-gathering consortia, and national reporting structures present “evidence” of higher education’s success or failure with the concession that while we are not able to assess student learning or educational gains, we can measure “attainment” or persistence to a credential, and, they say, isn’t that really the goal?

The third emphasis has been to take a much more in-depth look at how undergraduate learning might be assessed, not with current standardized tools, but with much more analytic means of assessing the development of argument, critical skills, and analytic and synthetic thinking and expression. This emphasis on assessing the real learning and development of students has gained recent audience and attention. However, this emphasis asks for significant investment by the institution and presents to the institution the real risk that the results could be unsettling.

Overall, in my judgment, for most of these emphases in the current conversation, we would benefit from deeper thinking about the multiplicity of dimensions of what students bring to college and about what they leave with. “What happens to students here” has too often been cast in a rudimentary, causal model that may do more to distort what could actually be meant by an educational experience and what we should be looking at as evidence of its depth and profundity.

BTtoP and the Advancement of Liberal Education

The call for attention to outcomes is of the utmost importance if the call is for a more sophisticated analysis of learning outcomes, such as that being proposed by College Learning Assessment project, joined with a better understanding of what could be called transformational outcomes and civic outcomes. As the evidence of the BTtoP project attests, attending to these complexities and types of outcomes and recognizing that they are not all causal or direct or linear is to take a huge next step in the advancement of liberal education and in the restoration of what that education promises. The call for outcomes must reflect not only effects of a causal model but also affects, dispositional and attitudinal changes, patterns of behavior and perspective, propensities to act, and so on that lead us to a deeper understanding of what the range and depth of outcomes might actually be. I don’t know if these will be easily quantified or easily ranked. Probably not. But unless we do attempt to gain some deeper understanding and to use more subtle metaphors and models when appraising the educational experience, we are likely to achieve hardly more than superficial and mischief-making generalities—fodder for the marketing of rankings and the continuation of the commodification of higher education.

Reference

Tritelli, D. 2007. From the editor. Liberal Education. 93 (1): 4.


Donald W. Harward is the president emeritus of Bates College and director of the Bringing Theory to Practice project

Previous Issues