Peer Review

Linking Advising and E-Portfolios for Engagement: Design, Evolution, Assessment, and University-Wide Implementation

The University of Notre Dame (UND) is a private Catholic research university with approximately 8,000 undergraduate students. The First Year of Studies (FYS) serves as the official collegiate home for all of its incoming first-year students (approximately 2,000 per year). By the end of their second semester at the university, these students will select one of the university’s undergraduate degree-granting colleges (arts and letters, business, engineering, and science) or the school of architecture as their academic home. All are assigned an academic advisor (a full-time member of the special professional faculty) who will facilitate their transition into university life. This process includes completing a first-year curriculum that develops their intellectual skills, allows them to explore within the university, and enables them to explore options for intended programs of study. Every student is required to attend at least two group guidance and two one-on-one advising sessions during this initial year. Over 98 percent of first-year students are typically retained for the sophomore year and over 96 percent graduate with their entering cohort.

As part of this effort to provide an integrative and holistic first-year experience, and in an effort to fold greater accountability into the process by which student advancement toward first-year learning goals could be assessed, FYS faculty began considering the use of electronic portfolios (e-portfolios) as a means of increasing student engagement and improving advisor-student interactions. During the 2012–2013 academic year, FYS designed a university-wide Advising ePortfolio Initiative and encouraged the entire first-year class to build and use the e-portfolio as a self-assessment tool. The initiative had three goals. The first (at the university level) was to introduce students to e-portfolios, to give them a space to reflect on and document their learning, and to build a university-wide e-portfolio culture. The second (at the FYS level) was to improve the advising experience by helping students prepare in advance for advising meetings and to be more mindful in the making of academic decisions through pre-engagement, reflection, and planning. In addition, FYS faculty had the sense that e-portfolios would enhance their efforts to assess student growth by providing firsthand evidence of student reflection on learning. The third goal (at the student level) was to enhance student intellectual engagement by providing a tool that would help students make the most of the time and energy they invest in academic planning. Overall, the hope was to initiate a process of deep, sustained, and contemplative learning that could continue throughout the entirety of the undergraduate experience.

In spite of the fact that there was no formal requirement to use an advising e-portfolio during the launch year, when the pilot year ended more than 71 percent of the first-year students had voluntarily participated. At the conclusion of the 2012–2013 academic year, the FYS ePortfolio Assessment Committee designed a methodology based on first-year learning objectives that proved effective in collecting evidence of student engagement and successful progress toward the realization of FYS’ learning objectives. Additionally, e-portfolio assessment outcomes offered clear indication of improvements needed both for first-year advising and the e-portfolio initiative. Because of strong institutional support and the opportunity to sponsor e-portfolio pilot courses and related programs that helped in the design of an advising e-portfolio geared to address the needs of first-year students, the 2012–2013 launch was a tremendous success. It led to a far stronger second-year roll out that resulted in widespread e-portfolio adoption by academic units throughout the university.

Definition and Design

Advising e-portfolios are teaching tools designed to assist students in improving their decision-making, goal-setting, and planning skills—capacities which are necessary in order for students to be actively engaged in managing their own learning. The e-portfolio also gives students the opportunity to begin collecting, organizing, and archiving multimedia evidence (i.e., papers, projects, pictures, videos, and reflections) of learning experiences (class, work, research, time abroad, and/or service). In brief, e-portfolios have the potential to assist students in becoming more intentional and active learners by helping them take ownership of their academic progress.

Ideally, advising e-portfolios can create both a foundation and a medium for advising sessions that will improve academic engagement as well as intellectual and personal development. FYS advisors used this tool to assist students in reaching these goals through directed reflective queries and advisors’ responses to those queries. Pre-advising meeting questions and both mid-year and end-of-year reflection questions were designed to prompt students to think about and integrate their learning experiences, and to develop proficiencies in multiple disciplines. The questions were also designed to help students chart future plans based on strengths and interests, to strategically develop necessary skills, and to make informed and conscious decisions emerging from both an evolving sense of self and an awareness of the complex university environment they inhabit.

Advisors also played a crucial role in the assessment process because they are situated simultaneously at strategic, operational, and tactical levels. Figure 1 demonstrates how advisors are ideally situated to engage students and help them chronicle growth and achievement of program outcomes in a single class, across classes, outside of classes, and throughout their four years at an institution.

Figure 1. A Balanced, Tactical, Operational, and Strategic Assessment Perspective

FYS-ePort-and-Advising_Web_0.jpg

Trained peer advisors are also tremendously helpful in fostering student usage of e-portfolios. Peer advisors in FYS built their own e-portfolios (1) to help them reflect on their Notre Dame educational experience and prepare for their next phase in life, (2) to help model for first-year students how to claim personal ownership for their education through the use of an e-portfolio, (3) to help support advisors with the FYS Advising ePortfolio Initiative, and (4) to assist the university in the gathering of outcomes-based evidence for accreditation through the use of a skills matrix to document progress toward university learning outcomes over a four-year span.

The FYS e-portfolio assessment philosophy and design approach were created in conversation with the university’s committee on e-portfolio use. The following principles were central to this philosophy:

  • Student participation during the first year is strongly encouraged, but voluntary, which gave us authentic assessment data. Students were prompted and encouraged to do an e-portfolio, but no consequences were imposed for not doing one. In addition, the e-portfolio is student owned and controlled, and so they decide who may view it.
  • The primary goal is to increase student engagement by using a blended advising model with e-portfolios. The term blended advising (Ambrose and Williamson-Ambrose 2013) represents the deliberate use of e-portfolios to flip the advising process by using the strengths of both face-to-face and online environments with synchronous and asynchronous technologies and interactions. By using technology in this way, this new paradigm of pre-engagement /engagement /re-engagement better aligns the advising process with the developmental process of teaching and learning. Instead of transactional and surface-level interactions dealing with compliance, access, and paperwork, advisors and advisees have a space and a place for quality interaction to uncover, discuss, and develop both passions and purpose.
  • Assessment can and should balance student and institutional needs for learning, and a student-centered e-portfolio can provide a balanced solution (Barrett 2007). If designed and implemented correctly, e-portfolios balance evaluation and learning, summative and formative feedback, past with the future, and a checklist of competencies with the holistic story of the learner’s development.
  • Learning outcomes should be mapped and embedded. Mapping e-portfolio artifacts to objectives at different levels means aligning evidence of the skills and outcomes that students achieve and document in their e-portfolios to class objectives, department/program objectives, college goals, institutional goals/outcomes, and national goals/standards (Kelly and Beers 2009). If evidence-based outcomes are mapped and e-portfolios are used for archiving, the nested assessment strategy model (Ambrose 2010) offers a framework for program administrators to strategically bridge issues of accreditation, accountability, and assessment in a balanced top-down (institutional-centered) and bottom-up (learner-centered) approach. If administrators create an e-portfolio culture, their program has the ability to link accreditation standards, institutional vision, program mission, curriculum competencies, course objectives, assignment outcomes, student performance, and learning.
  • In the age of assessment and accountability, it is not enough to establish outcomes and collect data to assess them. The assessment strategy needs to close the assessment loop and be a continuous improvement process.

History and Evolution

FYS began to develop an advising e-portfolio initiative for all first-year students in the spring of 2010 by piloting a one-credit e-portfolio independent self-study using Google Sites. In this blended design seminar, a small group of undergraduates worked with the designers/instructors to build and test advising e-portfolios through face-to-face in-class workshops and online. In 2010–2012, these e-portfolio courses were modified so as to be available in open courseware format for those unable to take the one-credit seminar.

In the spring 2011, the provost’s ad hoc committee on e-portfolios reached the conclusion that e-portfolios could play a strategic role in improving authentic assessment while also impacting student engagement and the advising process, as long as they were primarily student-owned and controlled. By fall 2011, the university had contracted with an e-portfolio vendor. The provost’s office embraced and commended this platform to the entirety of the university community by signing a contract that increased the scale of student accounts (by two thousand each year) until full student body access was attained. In addition, the provost’s steering committee on e-portfolios created the Notre Dame ePortfolio Engagement Program (nDEEP) to oversee and ensure effective implementation of e-portfolios and their alignment to the nested assessment of course, program, college, and university outcomes. In sum, nDEEP’s mission seeks to provide a platform, resources, and support to serve the needs of students, faculty, advisors, and programs in order to build a sustainable university-wide portfolio culture and community that supports authentic assessment, integrative learning, and deep engagement. (For more information see www.ePortfolio.nd.edu)

Implementation and Assessment

At the start of the 2012–13, the FYS dean and advising faculty challenged all incoming first-year students at orientation to build advising e-portfolios to improve their decision making, goal setting, planning, and reflection. This would begin the first university-wide implementation of the program. During this first-year rollout, student and advisor attention to e-portfolios varied from significant to minimal, but overall more than 71 percent of first-year students created e-portfolios and produced at least one reflection. Between pre-advising, mid-year, and end-of-year reflections, at least 3,820 first-year e-portfolio reflections were completed and submitted. Because of a usability glitch in “submitting” reflections during the first year, these numbers were lower than the actual number of student e-portfolio reflections completed. (Note: In a sampling test there was a 14 percent error rate of students who actually did complete, but did not submit, making the 71 percent participation rate a conservative estimate.) The e-portfolio vendor was aware of this problem and has already made changes to the user interface to simplify and improve the submission process.

FYS has been gathering reflection/self-assessment data since 2008. Figure 2 illustrates the increase in voluntary responses FYS had when shifting first from an e-mail reflection to an online survey form and then to the e-portfolio medium. What we learned from the process was that when protocols and platforms are designed for the student (archiving responses in their own e-portfolios rather than by filling in a generated form for their advisor’s use only), authenticity and student voluntary participation increased over the course of the semester.

Figure 2. Total FYS Reflection/Self-Assessment Data Points

Ambrose-Fig-2.jpg

 

If each reflection point took an average of fifteen minutes, then during its inaugural year advising e-portfolio use resulted in 57,210 minutes or 953.5 hours of student time devoted to reflection, goal-setting, and self-assessment. In an age where student engagement has become the metric of quality in higher education, the e-portfolio has proven to be a valid means and medium for increasing the time and energy students invest in their education and personal development outside of class.

In order to structure its assessment of advising e-portfolios, the FYS’ ePortfolio Assessment Committee was formed with five academic advisors and one assistant dean. The university’s accreditation coordinator and its teaching and learning center director assisted in the general planning. Using Marsh’s (2007) assessment levels of meaning as a framework, the committee used the following process:

  • Methods to measure learning. The advising e-portfolio assessment was treated as one qualitative method that supplemented a variety of other highly quantitative assessment measures. The assessment committee mapped and nested the FYS’s learning objectives onto the university’s learning outcomes. The committee then determined which learning outcomes could be authentically assessed through the e-portfolio reflection prompts.
  • The process of measuring and collecting information. Students were asked to build and update their e-portfolios by answering self-assessment/reflective prompts that were mapped onto FYS learning outcomes. These assignments were submitted and archived in the ePortfolio Assessment Management System (AMS). A representative 3 percent random sample of student e-portfolios was generated.
  • The process of interpreting and evaluating performance data. A rubric for the relevant learning outcomes was developed and each of the six reviewers scored twenty of the sixty student samples so that each e-portfolio was blind reviewed twice. Reports were run and visualizations were created to analyze the data sets.
  • The process of making improvements based on the results of the data evaluated. Reviewers rated, scored, and analyzed the sample of e-portfolios, noting trends and observations and then making recommendations for overall program improvement. In the spirit of continuous improvement and closing the assessment loop process, the committee collaboratively edited a Google Document to gather practical steps for improvement. The total assessment strategy is summarized in figure 3.

Figure 3. FYS Advising E-Portfolio Assessment Strategy

Ambrose-Fig-3.jpg

Ripple Effects and Next Steps

FYS e-portfolio assessment methodology provides one of the most substantial and well-documented examples of how UND is using the e-portfolio to develop a next-generation assessment strategy, philosophy, process, and platform supportive of continuous improvement for teaching and learning programs. Improvements for the second year included fewer assignments and a simplified e-portfolio template. Given the proven usefulness of the many exercises and resources found on the advising e-portfolio’s resource pages, the next steps will include discovering ways to encourage students to more fully use those resources, perhaps through credit-bearing and noncredit courses in blended, MOOC, and open courseware formats that focus on various topics such as cultural competency, career planning, and communication.

Since FYS advisors did the heavy lifting in orienting the first-year university cohort to e-portfolio technology and thinking, other academic units had abundant opportunities to follow its lead. Three of the five undergraduate degree-granting colleges have established sophomore-level college-wide advising e-portfolio initiatives to continue the momentum. At this point, there are eight additional program-level initiatives, at both undergraduate and graduate levels. In addition, strategic partnerships have been made with the alumni, study aboard, and career services offices. From spring 2012 to present (November 2013) more than 5,290 students (more than half of the undergraduate student population on campus) and 232 faculty, advisors, and staff have generated in excess of 9,528 e-portfolios.

This advising e-portfolio initiative is enabling us to explore the assessment of advising outcomes as a distinct stream of research within the scholarship of teaching and advising. Through the use of multi-layered assessment data, progress toward learning outcomes can be measured more effectively. These data, and the scholarly investigations emerging therefrom, promise to yield a growing body of best practices for advising, teaching, and learning. Several grants are now funding ongoing research at UND that seeks to mine e-portfolios, quantitatively and qualitatively, with data and model learning analytics that have the potential significantly to improve student retention and engagement in the STEM fields.

References

Ambrose, G. A., and A. L. Williamson. 2013. “The Blended Advising Model: Transforming Advising with ePortfolios.” International Journal of ePortfolio 3 (1): 75–89.

Ambrose, G. A. 2010. “E-Portfolios: A Nested Assessment Strategy for Accreditation and Accountability.” Journal of Instruction Delivery Systems (JIDS) 24 (1): 24–29.

Barrett, H. 2007. “Researching Electronic Portfolios and Learner Engagement: The REFLECT Initiative.” Journal of Adolescent & Adult Literacy 50 (6): 436–449.

Kelly, K., and M. Beers. 2009. “Mapping ePortfolio Artifacts to Objectives at Different Levels.” Presented at E-Portfolio Day of Planning, California State University, San Jose, California.

Marsh, P. 2007. “What Is Known about Student Learning Outcomes and How Does It Relate to the Scholarship of Teaching and Learning?” International Journal for the Scholarship of Teaching and Learning 1 (2).


G. Alex Ambrose is an academic advisor and coordinator for the Notre Dame ePortfolio Engagement Program; Holly E. Martin is the assistant dean of  First Year of Studies; Hugh R. Page, Jr. is a vice president, associate provost, and the dean First Year of Studies—all from University of Notre Dame.

Previous Issues