Peer Review

Assessing the Impact of Engaged Learning Initiatives for First-Year Students

As a demonstration site supported by the Bringing Theory to Practice project, Dickinson College has, for the past two years, implemented an ambitious engaged-learning initiative for first-year students with an accompanying research project to assess what impact these experiences have had on student engagement, well-being, alcohol use, and civic engagement.

All first-year students at Dickinson take a first-year seminar in their first semester. This program, taught by faculty from every department, is designed to engage students in a seminar-style course on varied subjects. Incoming students identify their six top choices for seminar topics and are assigned to one of these choices. No matter the topic, all seminars emphasize writing, information literacy, and research skills. The faculty member, in contact with these students two or three times per week for the first semester, also serves as the students’ academic advisor and remains in this role until the student declares a major.

Four years ago, Dickinson began to experiment with building a learning-community program by linking first-year seminars that share a common theme, housing students in these seminars in a common residential hall, and developing out-of-the-classroom educational experiences for this larger group of students at the intersecting points of the two seminars. Faculty met with students in their residence halls over dinners, shared weekend-long experiential education programs, and incorporated other campus-based and off-campus learning opportunities into the overall learning-community experience. We were interested to see if we could confound the students’ binary thinking about where learning happens and what constitutes social experience by introducing learning and stimulating social interaction among students and faculty across the boundaries of classroom, dorm life, and campus experience.

A Closer Look with BTtoP

Working with Bringing Theory to Practice became a vehicle for rigorous assessment. We wanted to explicitly study the effects of student participation in our first-year engaged-learning initiatives to examine whether variously structured learning experiences would yield different impacts on student learning and engagement, mental health, alcohol use, and civic engagement over the short and long term.

Our first question was whether students participating in the learning-community programs yielded any difference compared to those enrolled in stand-alone first-year seminars. Our second question was whether variations in the learning-community model—whether principally classroom based or incorporating service-learning pedagogy or experiential learning, and even a noncredit, community-service-focused residential clustering not linked to the seminar—yielded different results.

In terms of the engaged-learning initiatives during the first year of the project, we had eleven faculty members and approximately 160 students participating in five seminar-based learning communities, and an additional 22 students in the noncredit community-service learning community. (The first year of our project involved a sample of 153 learning-community students and 419 students not in the learning communities. Of students in learning communities, 52 percent were in the classroom-based track, 20 percent in the experiential track, 14 percent in the service-learning track, and 14 percent in the noncredit community track.) Collectively, these students participated in more than forty-five separate out-of-class informal learning experiences, including dinner discussions, film viewings, guest speakers, field trips, service-learning, outdoor physical activities such as caving and rock climbing linked to course content, and community service.

Our evaluation agenda involved both quantitative and qualitative data collection, and supplementary data from institutional sources. To collect baseline, midyear, and end-of-year quantitative data, we administered three surveys during the year. These administrations utilized controlled environments in the fall semester (an auditorium during first-year orientation and first-year seminar classroom), and an uncontrolled web-based administration in the spring semester. To gauge change in student engagement behaviors, pretest and posttest administrations of the College Student Expectations Questionnaire and College Student Experiences Questionnaire were conducted in August and April, respectively. Administrations of wellness and mental health instruments were conducted at all three time periods throughout the year: August (pretest), November (midyear), and April (end-of-year). Additional information regarding student behaviors and expectations prior to college and experiences at the end of the year were obtained through the Cooperative Institutional Research Program survey in July (before students arrived on campus) and the National Survey of Student Engagement administered in April. Qualitative data were gathered through six focus group sessions held the end of April. In Year 2, we added an instrument by Eyler and Giles (Where’s the Learning in Service Learning?) to better address elements of civic development among students in our experimental and control groups.

Examining the Data

Results from the first year of data collection reveal complex trends in the relationships between engaged learning, civic development, and student mental health and well-being. This is to say that while we did find strong indications that positive correlations exist between engaged learning, civic development, and better mental health and lower alcohol consumption, the triangulation of effects is complex.

With regard to wellness behaviors, findings indicated significant divergences between students in engaged-learning initiatives and those who were not. Pretest data showed no significant differences in the amount and frequency of alcohol consumption before the start of the school year between learning-community students and other students. However, though the groups began the school year with indistinct drinking behaviors, by November students in learning communities reported consuming alcohol significantly less often and in lower quantities than their peers in regular first-year seminars. Moreover, these same differences persisted through the spring. Although learning-community students and students not in learning communities did not differ in the degree to which they experienced negative outcomes as a result of drinking (such as feeling sad, feeling bad about oneself, or driving under the influence), differences in the frequency and amount of alcohol consumption suggest a strong positive impact of engagement on drinking behaviors.

Furthermore, these effects became clearer when comparing alcohol-related behaviors across types of learning communities. Specifically, the differences found at the end of the first semester were largely attributable to significantly lower reported rates of the frequency and amount of consumption by students in both the classroom-based and the noncredit community-service tracks. Similarly, effects at the end of the year were again attributable to the classroom-based track but also to the much lower consumption levels of students in the service-learning track. Thus, while engaged learning significantly impacts alcohol consumption, it also appears that civic engagement specifically plays a critical role in explaining the beneficial effects of engagement.

Differences across groups suggest that the benefits of engagement on alcohol consumption may be attenuated by negative impacts on mental health. As with the pretest findings for alcohol behaviors, learning-community students showed no significant differences from other students at the beginning of the school year with regard to depression levels. However, at the end of the first semester, learning-community students reported significantly higher levels of depression relative to their counterparts in regular first-year seminars. These differences were reduced to non-significance by the end of the school year in April. The emergence and regression of this effect indicates a unique effect of the first semester on learning-community students. A refinement of the analysis to examine tracks of learning communities, suggests the effect found in November is due primarily to a substantial spike in the reported depression levels of students in the service-learning track relative to the other tracks. By the spring, however, these students were not divergent from the other tracks. Nevertheless, these findings indicate that while civic engagement may uniquely discourage drinking, these initiatives may have unique adverse effects on mental health.

Qualitative data from focus groups were particularly helpful for understanding this trend. After talking with students in learning communities, and in particular those in the service-learning and noncredit community-service tracks, we believe the higher depression levels are largely manifestations of stress that developed in response to the additional time commitment of their off-campus community work. Students in the service-learning and the noncredit community-service tracks discussed the strain of doing their community work while managing their course loads. Nevertheless, these students, more than any other learning-community track, also expressed the greatest positive impact from this experience on their identity, their choice of major, and their outside interests.

The focus groups also offered insight into the differences between how learning-community students and other students reflected upon their first semester of college. First, learning-community students were far more apt to speak about the impact of their experiences on how they interacted with friends, their world perspectives, and their future course of study (either their major or their selection of other courses). These statements were almost completely absent from discussions with students not in the learning communities, who talked about engagement conceptually, such as what it should be rather than something they had specifically experienced. Second, though students in both groups did not clearly see the relationship between engagement and wellness, learning-community students spoke more about the degree to which being engaged can lower stress, mainly through the rationale that students are less stressed “if [they] are really interested in [the] classes.” In contrast, students not in the learning communities more often equated mental health with time management and preparation rather than engagement.

Data at Year 2 is less clear than Year 1 on the patterns of triangularity between engaged learning, student mental health and well-being, and civic development among learning community and non-learning community students. Specifically dimensions of these learning inititiatives and depression (student mental health) remain inconclusive, in addition to how these programs vary the learning process for these students. Though focus group data, specifically from Year 1, suggested better correlations of learning community experiences with deeper and more engaging learning experiences than non-learning community experiences, quantitative data has not reflected this trend. Moreover, comments from focus groups with first-year students in Year 2, though quite clear about what the definition of “engaged learning” is, did not suggest this type of learning necessarily happened in or out of a learning community experience.

Nevertheless, trends from Year 1 to Year 2 do suggest some indication that students in learning communities consume less alcohol (one indication of better well-being) than their peers not in learning communities. Moreover, this effect is largely motivated by students in the service-learning track, the group with the closest link to building civic integration and awareness. Second, Year 2 data, at least with our second cohort of first-year students, indicates that the effects of being in the service-learning track indeed may correlate with increased civic development. The strong positive effect of not just being in a learning community, but specifically one that incorporates community involvement with academic learning suggests civic development is motivated by this specific type of learning over other types (i.e. seminars not linked to learning communities, and learning communities that do not incorporate community involvement). Ultimately, we feel the overall pattern of results warrants a continued and systematic investigation of the relationship the impact of learning communities on engaged learning, student mental health and well-being, and civic development. Specifically, our research agenda includes further varying the types of learning communities students participate in, and the exploration of more nuanced dimensions of research foci, in particular student mental health and alcohol use.

Looking Forward

As we continue to explore trends into additional years, we are cognizant of two challenges. First, our design is not immune to selection bias. Students are aware when registering for a first-year seminar if it is part of a learning community, though not the track or type of learning community. Thus, we cannot be certain to what degree students who are already more engaged and/or less prone to drink are opting into the learning-community experience, and similarly to what degree their presumably less-engaged, heavier-drinking counterparts are opting out. We also face the significant challenge of maintaining response rates. As the campus environment has become increasingly inundated with questionnaires, we face a real prospect of students experiencing survey fatigue, which translates into greater attrition across administrations of surveys.

Though we have taken steps to address both of these challenges, our approach is ultimately less about eradication than it is to understand the ways in which these issues have and will continue to impact our project design.
Faculty and administration at Dickinson College have responded with interest to these engaged-learning initiatives. We have seen increasing formal and informal campus conversations about appropriate strategies to create more seamless experiences that help students bridge the classroom, the campus, and their broader community involvement. Over the last two years, for instance, Dickinson faculty have initiated joint meetings of key all-college committees (Enrollment and Student Life, with Academic Programs and Standards), and our president created a task force, “Pathways to Engaged Citizenship: The Dickinson Student Experience,” to make recommendations on bridging the divide between a student’s academic life and campus experience in the service of engaged learning and students’ civic development.

Dickinson is poised to move ahead even more substantially, particularly in the context of emerging data on student engagement, alcohol use, depression, and civic engagement. The demonstration site research is enabling Dickinson to evaluate and integrate the research data into campus discussion about the future design of students’ first-year experience and other dimensions of academic and student life.

Reference

Eyler, J., and D. E. Giles, Jr. 1999. Where’s the learning in service learning? San Francisco: Jossey-Bass Inc.


Shalom D. Staub is the assistant provost for academic affairs; Ashley P. Finley is an assistant professor of sociology—both of Dickinson College.

Previous Issues