Peer Review

The Scope of BTtoP Research: Design and Findings from the Demonstration Project

The Bringing Theory to Practice (BTtoP) project seeks the advancement of knowledge and the establishment of best practices centered in the BTtoP triangularity of engaged learning, student mental health and well-being, and civic development. Since its inception, BTtoP has established an aggressive research agenda through systematically evaluating funded projects that intentionally address this triangularity. The most substantial research effort in this vein is the BTtoP demonstration project, for which this article discusses the development and design of a research approach, encapsulates findings to date, and describes future research directions.

Conceptual Framework

Extensive review of the literature (Swaner 2007) identified suggestions as to linkages between engaged learning, student mental health and well-being, and civic development. First, at the outcomes level, correlations have been identified between elements of engaged learning (e.g., involvement in group projects and interaction with faculty) and self-report of better emotional health (Astin 1993), and between student participation in pro-social activities (such as community service) and lower rates of heavy drinking (Wechsler et al. 1995, Jessor et al. 1995, Fenzel 2005). On a theoretical level, research on stress in academic environments has demonstrated that while moderate levels of environmental stress can lead to optimal performance, extreme levels of stress can lead to "anger, fatigue, anxiety, fear, depression, or boredom" (Whitman, Spendlove, and Clark 1986). This would suggest that if engaged learning can optimize stress levels for students, better emotional health may result. Additionally, because students' level of moral development has been negatively correlated with substance abuse and other self-injurious behaviors (Berkowitz 2000), engaged-learning experiences that promote moral development may help reduce these behaviors. Finally, in a developmental view of students' health behaviors, both depression and substance abuse can result from developmental overchallenge posed by the college environment (Rivinus 1992). Thus, counterbalancing challenges with support may improve students' health, as might equalizing levels of freedom and responsibility by increasing students' "social responsibilities through community work" (Schulenberg and Maggs 2001, 33).

Given these potential linkages, the project began to formulate its central research goal to explore and describe the relationships between engaged learning, student mental health and well-being, and civic development. It was understood at the project's inception that engaged learning would most likely not constitute a silver bullet for either depression or substance abuse, but that there was also enough preliminary evidence to consider engaged learning as a promising community-level approach worthy of systematic investigation. The central challenge of this effort is to conduct inquiry that is multivariate, contextual, and time sensitive in nature.

Multivariate Inquiry
Due to the complexity of BTtoP's triangularity, project research necessarily extends beyond the traditional focus on one or two research variables to a multiplicity of variables that are psychological (motivation, self-concept, and self-esteem), affective (empathy and caring), values-related (moral and civic), and social, among others. Additionally, there is the largely unanswered question of whether and how these variables actually influence student behavior.

This is particularly true in the case of mental health and well-being, as much is still unknown about the interplay between genetic, psychological, and environmental factors in students' experiences of depression and substance abuse in college. This includes whether students have any history of problems or previous diagnoses and whether students experience "collateral events" that may impact outcomes. For example, joining a fraternity or sorority has been correlated with higher levels of binge drinking, and the disruption of interpersonal relationships can lead to an increase in depressive symptoms.

Thus, the multivariate nature of the research necessitates an equally complex research strategy, one that can determine whether observed changes or lack of changes "in educational performance, or psychological functioning, or other outcomes are due to the program under study or to confounding life events" (Waterman 2003, 80). And that moves beyond one or two univariate instruments that "are not designed to capture the full range of potential impacts of a complex, individual program" (Furco 2003, 15).

To account for these issues, BTtoP research involves a two-pronged approach: first, to develop multivariate instrumentation that allows for the identification of broad correlations between variables; and second, to employ a diverse range of data-collection methods and instruments, thereby creating a composite picture of students' experiences that enables a deeper understanding of the relationships between variables, and, ideally, to advance research into the realm of causality. This approach is in keeping with what Johnson and Onwuegbuzie (2004) describe as "mixed methods" research, which seeks to integrate both quantitative and qualitative paradigms into a single study of complex problems. As the authors explain, "Today's research world is becoming increasingly interdisciplinary, complex, and dynamic…. [A] mixed position allows researchers to mix and match design components that offer the best chance of answering their specific research questions" (15).

Contextual Inquiry
BTtoP research must take into account the convergence and variance of at least two specific contexts that affect research phenomena: the forms of engaged learning in which students participate and institutional culture. As Hecht (2003) describes, service learning is not "a specific program with identifiable characteristics... [but] an approach to teaching and learning that is given meaning by the school or organization where it is based…. Studies [should] account for the tremendous variability across and even within programs" (107). Thus, BTtoP research goes beyond mixed methods to describe across contexts the nature of the programs under study and the campus cultures in which they are situated.

The grand-design approach (Furco 2003) is one specific methodology that is promising for multisite studies of engaged learning, and service learning in particular. First, a set of both quantitative and qualitative measurements are selected and used across all participating sites to measure various outcomes. Then, in addition to this common group of instruments, a "second set of protocols that allows the researcher to investigate each unique program site in fuller detail" (26) is also developed. According to Furco, this approach "strives for comprehensiveness as well as for universality" and is therefore "applicable and relevant" across diverse programs (25). The use of this approach has been documented at the secondary school level in the use of the Evaluation System for Experiential Education, a package of ten qualitative and quantitative instruments assessing outcomes (academic, social, personal, career, ethical, and civic) of service-learning participation. The BTtoP research design adapts and extends the grand-design approach to assess engaged learning at the postsecondary level.

Time-Sensitive Inquiry
Finally, BTtoP research is necessarily time sensitive. First, the effects of specific forms of engaged learning may not become evident either during or immediately following student exposure. The impact of an engaged learning experience may extend beyond the actual experience (see Astin, Sax, and Avalos 1999), or it is also possible students may not recognize the value of an intense learning experience until after its conclusion and they have had time to reflect on it. Secondly, maturational effects during college are important to consider, as a significant number of students "mature out" of binge drinking over the course of four years (Rivinus 1992).

BTtoP research uses two methods to address this issue. First, it monitors student change using longitudinal data collection at multiple points of students' program participation and then analyzes data from each point as well as aggregate data. Second, it employs a quasi-experimental design that uses a comparison group comprised of adequately similar students not participating in the engaged-learning experience, with the assumption that changes observed in the program groups above and beyond normal maturation (as witnessed in the comparison groups) are attributable to the engaged-learning experience.

Research Design

To address the question of what relationships exist between engaged learning, student mental health and well-being, and civic development, the BTtoP Demonstration project involves a two-tiered, simultaneous approach to research using quantitative and qualitative methods at the campus level and cross-site level:

  • Campus level: All seven campuses submitted local research protocols and designated a local evaluator prior to project commencement. Campus research plans were required to involve longitudinal evaluation, formative and summative data collection, and adequate comparison groups. The range of quantitative instruments employed among the seven sites includes campus-developed pretest and posttest measures; mental health measures (i.e., Brief Symptoms Inventory), engagement measures (i.e., College Student Expectations Questionnaire), civic engagement scales (i.e., Scale of Service Learning Involvement), and existing institutional data sets from national surveys (i.e., Cooperative Institutional Research Program Freshman Survey). Qualitative methods include clinical interviews, focus groups, and analysis of reflection journals and course evaluation feedback.
  • Cross-site level: A national, cross-site evaluator worked with each demonstration site to implement a set of qualitative and quantitative measures administered uniformly across all seven institutions. Quantitative instruments consist of the National Survey of Student Engagement (NSSE), administered in spring 2006, and the College Student Experiences Questionnaire, also administered by sites with populations not sampled by NSSE (e.g., sophomores and juniors). The 2006 NSSE administration appended eleven supplemental consortium questions to the NSSE Core Survey to assess student mental health, well-being, and civic development at the seven demonstration sites only. (The BTtoP consortium questions are available at nsse.iub.edu/html/consortia-list_2006.cfm.) Qualitative methods at the cross-site level include student and cross-constituency focus groups (comprised of faculty and staff) using a collaboratively developed set of uniform focus-group questions, and cross-site conferences held in September 2005, June 2006, and June 2007 that concentrated on the central themes of the demonstration sites.

Project Findings

The following preliminary report of findings from the BTtoP demonstration project's first year is drawn from two data sets: cross-site data from institutional reports, site visits, and cross-site conferences; and the NSSE consortium administration. In keeping with the grand-design approach, which "organiz[es] information from all the data sources into recurring themes" (Furco 2003, 30), a thematic analysis of cross-site data follows, proceeded by an analysis of NSSE data on the initial correlations between engaged learning, student mental health and well-being, and civic development.

Thematic Analysis
Campuses reported the effectiveness of engaged learning programs on several levels. First, in confirmation of previous research, participation in these programs leads to deeper learning: Both students and their faculty reported better content mastery, level of engagement with material, and application of concepts to real-life settings. Student experiences in these programs were also described as personally transformative: Civic-development programs in particular were found to foster a more realistic understanding of the effort and commitment needed to effect community change. And a majority of involved constituencies, from students to faculty to administrators, reported high levels of satisfaction with engaged-learning and civic-development initiatives.

While the learning and civic-engagement outcomes of these programs were clearer than those related to student mental health and well-being, several sites found that student involvement correlated with lower alcohol usage. At the same time, however, involvement often correlated with higher levels of stress. Paradoxically, while students reported that the extra commitments of engaged-learning experiences produced added stress, they also claimed, often effusively, that these programs positively impacted their lives. Thus, higher levels of engagement may increase stress levels but at the same time produce the kind of deep, transformative learning sought by engaged-learning programs. Campus findings regarding effects on depression levels were mixed, with most sites reporting the need to further analyze data in light of gender differences in depression symptoms.

Regarding the research process itself, campuses highlighted the multidimensional nature of the research variables and described their ongoing struggle to accurately define and measure concepts of engagement, mental health and well-being, and civic development. Demonstration sites likewise confirmed the insufficiency of instrumentation to this task, as well as the challenges to evaluation, including survey fatigue and maintaining response rates. Along these lines, campuses underscored the importance of time as a necessary condition for conducting this research, as longitudinal data are necessary to delineate further the relationships between the variables under study.

Cross-Site Results from the National Survey of Student Engagement
The following analysis uses NSSE data gathered across demonstration sites to quantitatively analyze relationships between engaged learning, student mental health and well-being, and civic development. Benchmarks from the NSSE Core Survey were utilized as indicators of engaged learning. The consortium questions measured civic development at the campus, community, and national levels using scales ranging from zero to three, with zero indicating low engagement; health behaviors by the frequency and amount of alcohol consumption and the frequency of drug use (marijuana, other illegal drugs, and prescription drugs used for a purpose other than which they were prescribed); and mental health according to scales of both depression and stress. (A selected number of health-related items were used with permission from the Core Alcohol and Drug Survey and the American College Health Association–National College Health Assessment).

Sample Characteristics and Descriptive Results

A total sample of 2,545 students was drawn from a random sample of first-year and senior students at the seven demonstration sites. Only students who reported they had ever used alcohol or drugs were included in the sample—notably, this variable eliminated only five students. The sample consisted of 1,562 first-year students and 983 seniors, the majority of whom were women (65.9 percent) and white (73.3 percent); 7.6 percent of respondents identified themselves as Asian, 4.4 percent as Hispanic/Latino, 4 percent as black, 2.6 percent as multiracial, and 6.4 percent preferred not to indicate their race.

On the NSSE benchmarks, students reported experiencing the highest levels of "academic challenge" and "supportive campus environments," and the lowest levels of "student/faculty interaction" and "enriching education experiences." A descriptive analysis of civic-engagement scales showed students report slightly higher levels of campus involvement than national involvement and the lowest levels of local community involvement (see fig. 1).


Figure 1.
Mean scores, NSSE benchmarks and civic engagement scales*

*Standard deviations in parentheses

NSSE Benchmarks

Mean Scores (0-100)

Academic Challenge
59.37
(12.27)
Active and Collaborative Learning
44.72
(15.14)
Student/Faculty Interaction
37.52
(19.74)
Enriching Educational Experiences
41.61
(17.35)
Supportive Campus Environment
59.76
(16.71)

Type of Civic Engagement

Mean Score (0-3)*

Campus
1.31
(0.72)
Community
0.847
(0.69)
National
1.30
(0.74)

In terms of substance use, the majority of students reported that they have never used marijuana, other illegal drugs, or prescription medications for recreational use, a finding that supports indications that alcohol is the drug of choice on college campuses. Regarding alcohol use, the average number of drinks reported per sitting was 4.32, a mean strikingly close to the five-drinks-per-sitting designation commonly used to define binge drinking. Students also reported drinking, on average, 2.74 times per week. Results did indicate a maturational effect of binge drinking: First-year students consumed more alcohol per sitting than seniors; however, seniors drank significantly more times per week than first-year students. In terms of mental health, students were found to have moderate stress levels (1.2 on a 0 to 3 scale) and relatively low depression levels (1.06 on a 0 to 4 scale).

Correlations with Alcohol and Drug Use

In terms of the relationships between the NSSE benchmarks and student alcohol use, higher levels of "active and collaborative learning," "student/faculty interaction," and "enriching educational experiences"—all closely tied with engaged-learning experiences—correlated with students consuming fewer drinks per sitting. However, students indicating they experienced high levels of "active and collaborative learning" also correlated with drinking more times per week. Similarly, scales of civic engagement indicate that students engaged at the community level consume less alcohol in a sitting; however, those with higher degrees of involvement at the campus and national levels are drinking more often (see fig. 2).


Figure 2.
Correlations: NSSE benchmarks, civic engagement, and alcohol and drug use

*= p<.05, **= p<.01, ***= p<.001

NSSE Benchmarks

# of Drinks per Sitting

# of Days Drink Per Week

Marijuana Use

Other Illegal Drug Use

Prescription Drugs

Academic Challenge
-.036
.015
-.051*
.008
.018
Active and Collaborative Learning
-.054**
.060**
-.070***
.013
.012
Student/Faculty Interaction
-.082***
.030
-.048
.012
.022
Enriching Educational Experiences
-.125***
.098***
-.051**
.018
.003
Supportive Campus Environment
.019
.038
-.058**
-.067***
-.046*

Civic Engagment Scales

# of Drinks Per Sitting

# of Days Drink Per Week

Marijuana Use

Other Illegal Drug Use

Prescription Drugs

Campus Involvement
.007
.110***
-.051**
-.034
-.014
Community Involvmenet
-.069***
.000
-.075
-.026
-.032
National Involvement
.007
.079***
-.039*
-.005
.022

Thus, the pattern of relationships between engagement and substance use is not entirely linear. It may be that although engaged-learning and civic-development programs may discourage binge drinking, these same initiatives may also provide, through increased social contact and interaction, the opportunity for students to drink more often. This seems to suggest that while it is unrealistic to believe college students will abstain from alcohol, engaged-learning programs may promote more responsible drinking behaviors.

Correlations with Mental Health

In contrast to alcohol and drug use, civic-engagement scales and NSSE benchmarks reflected a clearer pattern of association with mental health indices. Specifically, correlations indicate, almost universally, that higher levels of engaged learning and civic development are associated with lower levels of depression. However, more-engaged students were also consistently found to have higher levels of stress (fig. 3). Thus, while engaged students may not be as depressed as their counterparts, they also report feeling greater stress. This supports both the thematic findings at the cross-site level and research in the literature regarding optimal stress levels. It is possible that programs that engage students civically and in their learning may be stressful in a way that is actually beneficial to students. And given that some NSSE benchmarks and civic-engagement scales correlated with lower amounts of alcohol consumption, it is possible that students are not necessarily responding to this stress with negative behaviors. Ultimately, greater analysis of "collateral events," such as student participation in campus organizations, will be critical in further understanding these relationships.


Figure 3.
Correlations: NSSE benchmarks, civic engagement, depression, and stress scal
es
*= p<.05, **= p<.01, ***= p<.001

NSSE Benchmarks

Depression Scale

Stress Scale

Academic Challenge
.053**
.153***
Active and Collaborative Learning
-.044*
.115***
Student/Faculty Interaction
-.047*
.134***
Enriching Education Enviornment
-.035
.148***
Supportive Campus Environment
-.260***
-.118***

Civic Engagement Scales

Depression Scale

Stress Scale

Campus Involvement
-.144***
.121***
Community Involvement
-.093***
.117***
National Involvement
-.094***
.090***

Considerations and Future Directions

The BTtoP demonstration project continues to face a number of issues, chief among them self-selection bias. For the majority of demonstration sites, students who are civically inclined have self-selected to participate in the various initiatives. Thus, any findings may be due the fact that students who are more drawn toward engaged-learning experiences simply tend to drink less or have better mental health or both. The use of comparison groups is helpful in addressing this to a degree, but the larger issue is finding a way to systematically target students who have lower patterns of engagement and who therefore may be more prone to heavier drinking and mental health concerns.

One approach is to implement a research design that involves close to 100 percent of the target population (for example, all first-year students enrolled in service-learning courses) to examine the impact of engaged learning on students with various (pre)dispositions. Such a strategy would also address engaged learning's peripheral status in most college curricula, reflected by the small percentage of students nationwide that are involved in service-learning and other engaged-learning experiences. To this end, BTtoP has launched a new initiative with the creation of "intensive site" grants, which enable colleges to conduct programming and research on this scale.

Another persistent concern is that of instrumentation. Evaluators have reported frustration with the univariate nature of existing instruments and the limited ability of these instruments to accurately gauge program impacts. In light of project findings, BTtoP research must continue to ask whether an "additive" approach to instrumentation, by which several of these instruments are used and analyzed simultaneously, or the development of new, multivariate instrumentation constitutes a more promising approach to yielding accurate data.

Finally, regardless of the persuasiveness of any BTtoP findings, mixed-methods research is not yet widely accepted through the educational research community, largely due to its nature as an "inclusive, pluralistic, and complementary" methodology (Johnson and Onwuegbuzie 2004, 17). As a mixed-methods approach, BTtoP research stresses the criteria of applicability and adaptability rather than the traditional and largely quantitative concern over the degree to which one can generalize from research findings. In other words, the robust solutions generated from the design and implementation challenges of BTtoP research may yield adaptable blueprints for institutions of higher education to follow as they consider ways to address the triangularity of engaged learning, student mental health and well-being, and civic development.

Along these lines, one of the most promising contributions of BTtoP research is the exploration of a new field of study and the development of new approaches to conducting inquiry in this field. Ultimately, project research may yield not only a good example of mixed-methods and grand-design approach, but also a valuable new methodology for examining complex and largely unanswered questions in higher education research.

References

Astin, A. W. 1993. What matters in college? Four critical years revisited. San Francisco: Jossey-Bass.

Astin, A. W., L. J. Sax, and J. Avalos. 1999. Long-term effects of volunteerism during the undergraduate years. The Review of Higher Education 22 (2): 187–202.

Berkowitz, M. W. 2000. Character education as prevention. In Improving prevention effectiveness, ed. W. B. Hansen, S. M. Giles, and M. D. Fearnow-Kenney, 37–45. Greensboro, NC: Tanglewood Research.

Fenzel, L. M. 2005. Multivariate analyses of predictors of heavy episodic drinking and drinking related problems among college students. Journal of College Student Development 46 (2): 126–40.

Furco, A. 2003. Issues of definition and program diversity in the study of service-learning. In Studying service-learning: Innovations in research methodology, ed. S. H. Billig and A. S. Waterman, 13–33.Mahwah, NJ: Lawrence Erlbaum Associates, Publishers.

Hecht, D. 2003. Issues of research design and statistical analysis. In Studying service-learning: Innovations in research methodology, ed. S. H. Billig and A. S. Waterman, 107–23. Mahwah, NJ: Lawrence Erlbaum Associates, Publishers.

Jessor, R., J. Van Den Bos, J. Vanderryn, F. M. Costa, and M. S. Turbin. 1995. Protective factors in adolescent problem behavior: Moderator effects and developmental change. Developmental Psychology 31: 923–33.

Johnson, R. B. and A. J. Onwuegbuzie. 2004. Mixed methods research: A research paradigm whose time has come. Educational Researcher 33 (7): 14–26.

Rivinus, T. M. 1992. College age substance abuse as a developmental arrest. In College student development, ed. L. C. Whitaker, R. E. Slimak, 141–66. New York: The Haworth Press, Inc.

Schulenberg, J. and J. L. Maggs. 2001. A developmental perspective on alcohol and other drug use during adolescence and the transition to young adulthood. Monitoring the Future Occasional Paper. Ann Arbor, Michigan: Michigan University Institute for Social Research.

Swaner, L. E. 2007. Linking engaged learning, student mental health and well-being, and civic development: A review of the literature. Liberal Education 93 (1): 16–25.

Waterman, A. S. 2003. Issues regarding the selection of variables for study in the context of the diversity of possible student outcomes of service-learning. In Studying service-learning: Innovations in research methodology, ed. S. H. Billig and A. S. Waterman, 73–90. Mahwah, NJ: Lawrence Erlbaum Associates, Publishers.

Wechsler, H., G. W. Dowdall, A. Davenport, and S. Castillo. 1995. Correlates of college student binge drinking. American Journal of Public Health 85: 921–26.

Whitman, N. A., D. C. Spendlove, and C. H. Clark. 1986. Increasing students' learning: A faculty guide to reducing stress among students. ASHE-ERIC Higher Education Report 4. Washington, DC: Association for the Study of Higher Education.


Lynn E. Swaner is an assistant professor of counseling and development at the C. W. Post Campus of Long Island University and cross-site evaluator for the Bringing Theory to Practice project; Ashley P. Finley is an assistant professor of sociology at Dickinson College.

Previous Issues