Peer Review

Completion, Quality, and Change: The Difference E-Portfolios Make

As higher education confronts a challenging new era, it is riven by two competing agendas. The completion agenda prioritizes rapid graduation, using new tools and structures to help students advance through higher education with greater speed and efficiency. Meanwhile, the quality agenda focuses on learning, prioritizing depth and understanding and helping students develop as complex thinkers. Preliminary findings from the Connect to Learning project (C2L) suggest that e-portfolio initiatives could play key roles in resolving this tension. The work of campus teams associated with C2L demonstrates that thoughtful e-portfolio practice can help build student success (as measured in “hard outcomes” such as retention and graduation) while also advancing reflection, integration, and “deep learning.”

What is C2L?

Connect to Learning is a FIPSE-funded community of practice, linking e-portfolio leadership teams from twenty-four campuses in a shared effort to learn from each other, document effective practices, and develop a resource for the field. Launched in 2011, C2L is coordinated by the Making Connections National Resource Center at LaGuardia Community College, the City University of New York (CUNY). C2L campuses, each of which sustains its own campus e-portfolio initiative, represent a diverse cross-section of higher education, from Boston University to San Francisco State University, Salt Lake Community College, and CUNY’s newest campus, Stella and Charles Guttman Community College.

Over the past three years, C2L led a hybrid professional development process, linking campus practice, annual face-to-face meetings, and online exchange. Each campus team built a portfolio to represent its practices and its campus e-portfolio story. In monthly online discussions the teams explored practices shared in the portfolios. This scaffolded process, detailed in To Improve the Academy (Eynon, Gambino, and Torok 2013), helped teams deepen their best work.

C2L has collected members’ campus practices in a resource website called Catalyst for Learning: ePortfolio Research and Resources (http://c2l.mcnrc.org). Made public in early 2014, this groundbreaking site links campus e-portfolios into a searchable database. It also offers analytical essays, multimedia presentations, student portfolios, and nearly 300 campus-tested practices (Catalyst 2014).

As we gathered resources for the Catalyst site, the C2L network developed a theoretical framework, spotlighting characteristics of effective e-portfolio practice. The hypothesis emerging from our collective inquiry suggests that successful e-portfolio initiatives address multiple layers of campus activity, from classrooms to institutional policy. We found that effective campus teams built projects that spanned five interlocking sectors: pedagogy, outcomes assessment, professional development, technology, and scaling up. This framework (see fig. 1) informs the Catalyst site.

Figure 1. The Catalyst Framework

enyonwi14figure1.jpg

 

What Does it Take to Make A Difference?

Pedagogy is the primary sector in the Catalyst framework of effective practice. C2L campus teams studied the research on reflection, particularly the work of Carol Rodgers, and applied it to e-portfolio practice. A Dewey scholar, Rodgers (2002) emphasizes systematic reflection to help students connect and make meaning from diverse learning experiences. At Virginia Tech, for example, students in a first-year learning community use e-portfolios to reflect on service-learning experiences, connecting them to classroom learning and deepening their goals as learners. Effective reflection, Rodgers suggests, helps students link academic learning to personal development. The Catalyst site showcases longitudinal practices; for example, at Three Rivers Community College, nursing students taking a set of six sequenced courses recursively reflect on the ethos and values of their profession, authoring their own identities as learners and emerging professionals.

E-portfolios are often treated as private, individual spaces. But C2L teams emphasized reflection in community. Linking Rogers’ ideas to a white paper on “social pedagogy” by Randy Bass and Heidi Elmendorf (2011), C2L teams developed ways to use e-portfolios for peer critique, collaboration, and exchange. The Catalyst site documents multiple examples of this practice, such as art history students at Indiana University-Purdue University Indianapolis (IUPUI ) engaging in extensive peer review of each other’s portfolios and Pace University biology students using e-portfolios to create collaborative resources for newer students.

While pedagogy is key to effective e-portfolio projects, C2L found outcomes assessment to be equally vital. Northeastern University, IUPUI, LaGuardia, Salt Lake Community College, and other C2L campuses documented ways they use e-portfolios to support programmatic and general education assessment. Salt Lake, for example, uses e-portfolios to collect thousands of student learning artifacts, which are read against rubrics based on AAC&U’s VALUE rubrics to assess writing and other competencies. The Catalyst site offers rubrics, procedures, and multiple campus case studies.

Catalyst also documents campus practices related to other framework sectors: professional development, technology, and scaling up. To support reflective social pedagogy, C2L campuses offer faculty workshops, institutes, and yearlong seminars. Teams shared approaches that worked well in their settings. They also shared criteria for effective e-portfolio technology and strategies for working with campus stakeholders, attracting funds, and building a campus e-portfolio culture.

Reviewing all sectors of the Catalyst framework, C2L found that three design principles informed campus practice: inquiry, reflection and integration. While shaping sophisticated e-portfolio pedagogy, attention to these design principles also deepens the work of e-portfolio projects around professional development, outcomes assessment, technology, and scaling up. Across sectors, the most powerful uses of e-portfolios engage faculty, staff, and students in a systematic inquiry into learning. Building on inquiry and guided by reflection, students, faculty, and other stakeholders make connections that help them integrate and apply learning to individual and institutional plans. The Catalyst site offers essays that detail each principle.

What Difference Can E-Portfolio Initiatives Make?

The sectors and design principles of the Catalyst framework help identify and illustrate what it takes to build an effective campus e-portfolio initiative. There’s an obvious companion question: is this effort worthwhile? What difference can e-portfolios make? In other words, what does the evidence from the C2L network tell us about the potential value of a sophisticated e-portfolio initiative?

Our working answer to this question can be grouped into three propositions: (1) e-portfolio initiatives advance student success; (2) making student learning visible—e-portfolio initiatives support reflection, social pedagogy, and deep learning; and (3) e-portfolio initiatives catalyze learning-centered institutional change. Evidence linked to the first and second propositions shows how e-portfolio projects can help campuses address both the completion and quality agendas. And the third proposition suggests how e-portfolio initiatives can advance what we call the change agenda—helping colleges become more adaptive learning organizations.

An important caveat: we recognize that the data from C2L campuses have significant limitations. Proving causal connections related to learning is always challenging. C2L campus teams lack the ability to conduct randomized control group studies. The network spans diverse campus contexts, marked by differences in focus, purpose, and level of student preparation. The C2L data is limited in rigor and consistency, but it is nonetheless suggestive and intriguing.

Proposition 1: E-Portfolio Initiatives Advance Student Success
At a growing number of campuses with sustained e-portfolio initiatives, student e-portfolio usage correlates with higher levels of student success as measured by pass rates, GPA, and retention.

C2L campuses study the relationship of e-portfolio to student success. A constellation of these campuses can now present e-portfolio-related student success evidence such as retention rates and GPA data. For example:

  • In the Douglass Women’s College of Rutgers University, e-portfolio use was introduced into a required first-semester “mission” course in 2008–2009; student performance improved significantly. The average grade point in the course for two semesters before e-portfolios were introduced (2007–8) was a B (3.213); in nine semesters using e-portfolios (2009–2012), students earned an average of a B+ (3.508). Students’ GPAs across all of their courses improved as well. In 2007–8, before e-portfolio use, the average cumulative GPA was 2.933; in the nine semesters from 2009–12, average cumulative GPA has been 3.095.
  • At LaGuardia Community College, data from the 2011–2012 school year showed that, across disciplines, the one-semester retention rate for students in e-portfolio courses in 2011–12 was 80.4 percent, versus 61.7 percent for students in comparison courses. Students enrolled in e-portfolio courses also had higher course completion (96.4 percent, +1.8 percentage points), course pass (79.7 percent, + 8.2 percentage points), and high pass rates—C and above—(77.7 percent, +9.9 percentage points) than students in comparison courses.
  • At Queensborough Community College, all incoming students were enrolled in First-Year Academies beginning in 2009. Some course sections in the academies used e-portfolios, other course sections did not. All Academy sections in 2009-10 demonstrated significant improvement in pass rates and next semester retention, compared to college baselines (the 2006-7 academic year). However, the improvements in pass rates and retention in the e-portfolio-enhanced course sections were significantly greater (see table 1).
  • At Tunxis Community College, a year-long comparison between e-portfolio and non-e-portfolio sections of developmental English courses showed that e-portfolio sections had on average a 3.5 percent higher pass rate and an almost 6 percent higher retention rate. Meanwhile, data from across the college showed that students who had taken multiple courses with e-portfolios, from first year to capstone, were more likely to be retained than students who had fewer or no e-portfolio exposures (see fig. 2).
  • San Francisco State University integrated e-portfolio use into the Metro Health Academy, a learning community for high-risk students. Data shows that retention and graduation rates for students in this e-portfolio-enhanced learning community are substantially higher than university-wide averages (see table 2).

Figure 2. Tunxis Community College (CT) Next Semester Retention Rates by Number of Exposures

enyonwi14figure2.jpg

 

Table 1. Queensborough Retention Data

Queensborough Community College
Retention
Percent
Intervention
Fa 2006 Sp 2007 Retention
65%
Benchmark
Fa 2009 Sp 2010 Retention
88%
Freshman Academy
Fa 2009 Sp 2010 Retention
97.8%
FY Academies w/E-P

 

Table 2. Metro Academy Retention Data

San Francisco State University
 
Metro Academy, E-Portfolio First Year/First Time Students
All SFSU First Year/First Time Students
1 Yr Retention Rate
90.0%
79.3%
2 Yr Retention Rate
79.0%
60.0%
4 Yr Retention Rate
24.6%
14.9%

 

As noted above, this data has limitations, and is not in any way conclusive. Yet it represents an emergent pattern and is similar to the types of data routinely used for decision making by state agencies and higher education institutions. As such, it provides a suggestive body of evidence for the proposition that sophisticated e-portfolio initiatives can demonstrate a correlation between e-portfolio usage and improved student success and help campuses meet the challenges of the completion agenda.

Other aspects of this data also are worth noting. One is that positive outcomes cross institutional type. Another is that, on many C2L campuses, e-portfolio is used with other high-impact practices (Kuh 2008), such as first-year experience programs, learning communities, and capstone courses. This supports an emergent proposition that powerful e-portfolio practice is inherently connective and integrative, and that part of what e-portfolio can do is link and enhance the value of other high-impact practices. In this sense, e-portfolio use could be understood as what IUPUI’s Susan Scott and Susan Kahn (2014) have described as a “meta-high-impact practice.”

Proposition 2: Making Student Learning Visible—E-Portfolio Initiatives Support Reflection, Social Pedagogy, and Deep Learning
Helping students reflect on and connect their learning across academic and cocurricular learning experiences, sophisticated e-portfolio practices transform the student learning experience. Advancing higher order thinking and integrative learning, the connective e-portfolio helps students construct purposeful identities as learners.

While positive student success data is important, it provides limited insight into the e-portfolio-enhanced student learning experience. For those with less experience with e-portfolios, it is easy to make a quick leap from such data to the assumption that implementing e-portfolios will automatically lead to improved student outcomes. Those with more experience know, however, that the value of e-portfolio practice for students depends in large measure on how it is implemented: the pedagogy and practices of faculty and staff as well as broader support structures, the kinds of practices discussed above and identified by the Catalyst framework.

Guided by the design principles of inquiry, reflection, and integration, e-portfolio pedagogy on C2L campuses focused on helping students make connections across diverse learning experiences and consider the relevance of college learning to their personal lives. Does e-portfolio practice based on this pedagogy have an effect? To explore this question, C2L campuses have been surveying students, seeking insight into the ways students describe their e-portfolio experience. In 2011, C2L developed a C2L Core Student Survey; the findings from four semesters of student responses (n=9,542) have deepened our understanding of how e-portfolio use on C2L campuses affects the student learning experience.

The survey examines the ways e-portfolio experiences shaped student learning (see table 3). For example, students used a four-part scale to agree or disagree with the statement “Building my e-portfolio helped me to make connections between ideas.” Seventy percent of respondents Agreed or Strongly Agreed with this statement. Similarly, 65.6 percent Agreed or Strongly Agreed that “Using an e-portfolio has allowed me to be more aware of my growth and development as a learner.” These and other survey data suggest the integrative e-portfolio experience helps students make connections and build a more holistic self-portrait, a way of understanding themselves as learners.

Table 3. Students’ Integrative E-Portfolio Experiences

 
Selected C2L Core Survey Questions
Quite a Bit or Very Much

"Building my e-portfolio helped me to think more deeply about the content of this course."

64.4%

"Building my e-portfolio helped me succeed as a student."

68.5%

"Someday I'd like to use my e-portfolio to show what I've learned and what I can do to others, such potential employers or professors at another college."

73.8%

"Using e-portfolio has allowed me to be more aware of my growth and development as a learner."

69.3%

"Building my e-portfolio helped me to make connections between ideas."

75.6%

n=6,729 students
 

These e-portfolio-specific questions were flanked by questions drawn from the National Survey of Student Engagement (NSSE) (table 4). Asked how much their coursework “Contributed to your knowledge, skills, and personal development in understanding yourself,” 74.1 percent responded Quite a Bit or Very Much, reinforcing the idea that reflective e-portfolio experiences supported self-understanding. Student responses were also strong on questions related to integrative, higher order thinking, key elements in deep learning (Laird, Shoup, and Kuh 2005). When C2L students were asked how much their coursework contributed to engagement in “Synthesizing and organizing ideas, information, or experiences in new ways” the percentage responding Quite a Bit or Very Much was 78.4 percent.

Table 4. DEEP Learning Questions Drawn from NSSE

 
Selected C2L Core Survey Questions
Quite a Bit or Very Much

“How much has your experience in this course contributed to your knowledge, skills, and personal development in writing clearly and effectively?”

77.7%

“How much has your experience in this course contributed to your knowledge, skills, and personal development in understanding yourself?”

78.6%

“How much has your work in this course emphasized applying theories or concepts to practical problems or in new situations?”

77.2%

“How much has your work in this course emphasized synthesizing and organizing ideas, information, or experiences in new ways?”

83.1%

n=6,729 students
 

The survey included open-ended questions asking students how the e-portfolio experience shaped their learning. The replies extend patterns revealed by the quantitative data:

  • E-portfolio has supported my growth and learning because I was able to bring my ideas together. I learned that I have accomplished a lot throughout my college career.
  • I got to show who I was. While creating my e-portfolio, I learned even more about myself.
  • The best part was to be able to apply my own work into it… I love how it links to assignments that you have done… I also enjoy that I grew as a learner… It helped me connect between new ideas and old ones.

While not conclusive, the data suggests that reflective e-portfolio pedagogy helps students make meaning from specific learning experiences and connections to other experiences, within and beyond the course. Ultimately, students recursively connect their learning to consideration of goals and values, constructing a more intentional and purposeful sense of self.

C2L survey data also suggests that linking integrative reflection with social pedagogy—having an audience actually looking at the e-portfolios—enhances its value. The role of audience was explored from two perspectives—instructors and peers. Across four semesters, 75.4 percent of students who reported high levels of instructor feedback also Agreed or Strongly Agreed with the statement “Using e-portfolio has allowed me to be more aware of my growth and development as a learner.” For students reporting low levels of instructor feedback, the comparable figure was 20.6 percent.

Similarly, a peer feedback scale was created by taking the mean of the responses to two comparable peer-related items, asking students whether other students had reviewed and given them feedback on their portfolios. Students who received high levels of peer feedback as part of their e-portfolio development were significantly more likely than students in the low peer feedback group to report high levels of integrative learning experience. For example, 85.4 percent of students who reported high levels of peer feedback Agreed or Strongly Agreed with the statement “Using e-portfolio has allowed me to be more aware of my growth and development as a learner.” The figure for students who received low levels of student feedback was 30.6 percent.

The pattern held true for all of the NSSE-based questions, including those associated with deep learning. When students know that peers are looking at and commenting on their e-portfolio, its value as a learning experience is significantly enhanced.

Qualitative data also highlighted the importance of social pedagogy to the portfolio experience. “E-portfolio has allowed me to receive feedback and criticism of my work from fellow classmates. I have learned where my weaknesses and strengths are as a designer,” commented one student. “The best part was seeing other students’ e-portfolios and getting to know them and their experiences,” noted a second. Wrote a third: “The best part of working with e-portfolio is that I can share this with people and they can see what I have done in school.”

This preliminary data analysis suggests that e-portfolio processes shaped by integrative social pedagogies help students make connections and deepen their learning. Together with data indicating that e-portfolio use helps students understand themselves as learners, this suggests that e-portfolio experiences shaped by such pedagogies help students take ownership of their learning, building not only academic skills but also the affective understandings of self now seen as critical to student success. In this way, a sophisticated e-portfolio initiative can help campuses address the quality agenda without sacrificing outcomes related to completion.

Proposition 3: E-Portfolio Initiatives Catalyze Learning-Centered Institutional Change
Focusing attention on student learning and prompting connection and cooperation across departments and divisions, E-portfolio initiatives can catalyze campus cultural and structural change, helping the institution move toward becoming an adaptive learning organization.

While the completion and quality agendas are well known, a third agenda for higher education is perhaps equally important: the change agenda. How can colleges and universities build their capacity to respond and adapt to changing conditions and new possibilities? How can they thoughtfully engage faculty and staff expertise to advance institution-wide innovation, focused on student learning? How can they build a learning culture and become more integrated and adaptive learning organizations?

Addressing this challenging agenda, our third proposition is qualitatively different than the first two, more sweeping and difficult to assess. Our analysis here is at a formative stage. The evidence derives from stories and practices shared by C2L teams, who documented work in each sector of the Catalyst framework and described how it re-shaped campus structure and culture. This data does not lend itself to hard and fast conclusions. But it is fascinating and meaningful, and even at this early phase it deserves broad consideration by the field.

To build a successful e-portfolio initiative, C2L teams develop reflective social pedagogies, manage new technologies, and lead professional development and outcomes assessment processes. They attend to a range of other tasks that build campus engagement and institutional support. This work is instrumentally important to the scaling up process. At the same time, by bringing together diverse campus constituencies for collaboration focused on student learning, it creates opportunities for deeper systemic change.

As they scale up, successful C2L campus teams require and facilitate cross-campus collaboration. Using e-portfolios and the inquiry–reflection–integration design principles to support outcomes assessment and broad processes of institutional self-examination, C2L projects prompt campus-wide conversations about student learning. The conversations required for e-portfolio success can help campuses address the change agenda—illuminating the holistic nature of student learning, sparking integrative structural change, and building campus-wide commitment to organizational learning.

Across the network, the work of C2L teams has helped to prompt rethinking and change, the growth of a variety of integrative structures and an emerging commitment to an ethos of learning. At Northeastern University, for example, C2L faculty in Education used e-portfolio to assess student learning, and decided to totally rewrite the curriculum of all of their courses to improve cohesion, articulation, and a higher degree of integrative learning (Matthews-DeNatale 2014). At Manhattanville College, the C2L team initiated an in-depth professional development program. The power of learning-centered inquiry drew faculty support for a more sustained focus on integrative learning and teaching, which in turn generated administrative support. The college made a commitment to launch and support a new campus-wide center for teaching and learning that would be responsible for e-portfolio programs and broader pedagogical inquiry.

At San Francisco State, evidence demonstrating the success of the Metro Health Academies has prompted the university to rethink the ways it helps students enter college life. Beginning in fall 2013, an integrative, e-portfolio-based learning community approach is being expanded to serve as much as 40 percent of the incoming student population. The provost of Boston University recently highlighted the e-portfolio initiative of the College of General Studies as an assessment model for other colleges at the university. Similarly, the e-portfolio effort at Three Rivers Community College had been limited to nursing program, but that integrative approach is now expanding to new academic areas, linking other programs and general education.

Meanwhile, at LaGuardia, e-portfolios have long been used to advance the importance of integrative learning, addressing the whole student. In 2012, the college announced a sweeping institutional change effort reflecting a similar perspective, drawing on e-portfolio use as part of the process of aligning student affairs and academic affairs, rethinking advisement and rebuilding the first-year experience. An e-portfolio initiative’s capacity to highlight holistic learning, support educational planning and identity development, and link curricular and cocurricular experiences can help create bridges between academic and student affairs.

Observing campus developments across our network, particularly those related to scaling-up processes, we see that the growth of an e-portfolio initiative both requires and spurs broader changes in institutional culture and structure. While the C2L evidence is still preliminary, it suggests that sophisticated e-portfolio practice can promote learning-centered connection, making student learning visible to faculty and staff across institutional boundaries. Requiring and facilitating collaboration across disciplines and departments, e-portfolio initiatives can help to build learning cultures across traditional institutional silos. By supporting a richer view of learning, encouraging a learning-centered institutional conversation, and catalyzing broad institutional change in structure and culture, e-portfolios can help colleges become more integrated and adaptive learning organizations.

 

REFERENCES

Bass, R., and H. Elmendorf. 2011. “Designing for Difficulty: Social Pedagogies as a Framework for Course Design.” Teagle Foundation White Paper. New York, NY: Teagle Foundation. https://blogs.commons.georgetown.edu/bassr/social-pedagogies/#_ftn1/.

Connect to Learning. 2014. Catalyst for Learning: ePortfolio Research and Resources. Accessed February 6. http://c2l.mcnrc.org.

Eynon, B., L. Gambino, and J. Török. 2013. “Connect to Learning: Using ePortfolio in Hybrid Professional Development.” To Improve the Academy 32 (1).

Kahn, S., and S. Scott. 2014. “Using an ePortfolio to Assess the Outcomes of a First-Year Seminar: Student Narrative and Authentic Assessment.” In press, International Journal of ePortfolio 4 (1).

Kuh, G. D. 2008. High-Impact Practices: What They Are, Who Has Access to Them, and Why They Matter. Washington, DC: Association of American Colleges and Universities.

Matthews-DeNatale. 2014. “Are We Who We Think We Are? ePortfolios as a Tool for Curriculum Redesign.” Journal of Asynchronous Learning Networks 17 (4).

Nelson Laird, T., R. Shoup, and G. D. Kuh. 2006. “Measuring Deep Approaches to Learning Using the National Survey of Student Engagement.” Paper presented at the Annual Meeting of the Association for Institutional Research, Chicago, IL,

Rodgers, C. 2002. “Defining Reflection: Another Look at John Dewey and Reflective Thinking.” Teachers College Record 1 (4): 842–866.

 


All C2L campus practices, stories and data referenced in this article, unless otherwise cited have been published on the Catalyst website http://c2l.mcnrc.org.


This article is adapted from the original Connect to Learn project field report, “What Difference Can e-Portfolio Make?” by Brett Eynon, Laura M. Gambino, and Judit Török, which is published in the April 2014 issue of the International Journal of e-Portfolio, volume 1. The original article can be found on the Catalyst for Learning ePortfolio Resources and Research website http://c2l.mcnrc.org).

Bret Eynon is the associate dean for academic affairs and executive director of Making Connections National Resource Center at LaGuardia Community College, CUNY; Laura M. Gambino is a professor and faculty scholar for teaching, learning, and assessment at Stella and Charles Guttman Community College, CUNY; Judit Török is the co-director of Making Connections National Resource Center at LaGuardia Community College, CUNY.

Previous Issues