Portfolios for Summative Assessment:  The Challenges of Reliability, Validity and Fairness

Elaine F. Dannefer, Ph.D. and Lindsey C. Henson, M.D., Ph.D.

Cleveland Clinic Lerner College of Medicine of Case Western Reserve University, Cleveland, Ohio

Introduction

The Cleveland Clinic Lerner College of Medicine (CCLCM) at Case Western Reserve University competency-based, five-year curriculum designed to educate physician investigators has implemented a portfolio assessment system used for both formative and summative assessment. Each student is expected to attain a defined level of achievement in each of nine competencies: research, medical knowledge, communication, professionalism, personal development, clinical skills, clinical reasoning, health care systems, and reflective practice. The portfolio approach to assessment was adopted as a tool for training students in reflective practice, taking responsibility for monitoring their own learning, and giving them a record of personal growth and achievement of the nine competencies. 

Unique Aspects of CCLCM’s Portfolio Assessment System

Ø      No graded assessments; all assessments are formative including those for medical knowledge

Ø      Mastery learning approach to student achievement of competencies

Ø      Robust advising system in which  Physician Advisers partner with students across the five years to engage in reflective practice cycles around the Formative Portfolios

Ø      Yearly summative assessments of the student’s Summary Portfolio by a separate faculty committee

 Statement of the Problem and Program Objectives

Traditional approaches to assessment tend to be reductionistic and fail to capture the complexity and integrated nature of professional competence.  In contrast, portfolios provide a vehicle for encouraging reflection and assessment of a broad range of authentic evidence from learning experiences.  When used for formative purposes, the individualized nature of the portfolio facilitates identifying gaps as well as areas of strength in knowledge, skills and attitudes.  Yet the very qualities that make portfolios attractive for formative assessment are at the same time the source of psychometric challenges when used to make summative, standards-based decisions.  Portfolios used for summative assessments must meet tests of reliability, validity, and fairness.  The question is, “Can a summative portfolio assessment system be implemented that is psychometrically sound?” 

Description of the Project/Intervention

Students and their physician advisers analyze feedback as it is collected to monitor student progress in meeting the nine competencies.  Students periodically prepare Formative Portfolios that include reflective essays, learning plans, and student-selected evidence and discuss them with their adviser in formal meetings.  Students also use their formative assessments to construct yearly Summary Portfolios for review by a separate Medical Student Promotions and Review Committee (MSPRC).  The physician advisers play a non-evaluative role in the summative process by providing feedback during the development phase, confirming that the portfolio is the student’s own work and that the selected evidence is representative of the student’s performance across the year. Distinctly separate processes and reviewers for the formative and summative portfolios ensure that reflective practice is not compromised by the rigor and judgments required for making promotion decisions.

Portfolio Assessment Model for Year One

We are putting into place a process designed to establish the reliability, validity and fairness to make promotions decisions of the yearly summative portfolio in our competency-based curriculum.  Steps in the process include:

Ø      orient students to the requirements,

Ø      establish guidelines for the role of the physician adviser in helping students prepare their portfolio,

Ø       train the MSPRC members to review the portfolio,

Ø      create forms to guide review of portfolios and to record decisions,

Ø      design and implement a standard-setting process, and

Ø      make promotion decisions.

    Findings to Date/Evaluation to Date

    Findings related to steps being taken to ensure that summary portfolio review process addresses issues of reliability, validity and fairness are as follows:

Ø      Orient Students:  Students reported that receiving written instructions for how to construct the summary portfolio followed by a interactive session reviewing the guidelines was helpful. 

Ø      Establish Guidelines for Physician Advisers: Weekly meetings and special workshops using de-identified formative portfolios helped develop shared expectations for the quality of portfolios and skills for promoting reflective practice.

Ø      Train MSPRC Members: The promotions committee has been presented with cases designed to introduce them to the types of judgments that they will need to make during the summary portfolio review process.

Ø      Standard-Setting Exercise:  All nine promotions committee members will read the same 8 randomly selected summary portfolios.  A day long standard setting process is currently being developed with the help of Margery Davis, MD., consultant to the project.

Ø      Decision-making Process:  Portfolios will be individually reviewed by two MSPRC members who will then meet to reach consensus and prepare their recommendations to the full committee where the final decision will be made.  

 Case for MSPRC Training Session

Two MSPRC reviewers agree that a  student’s work over the year shows a clearly discernable pattern.  The feedback from faculty and peers and other evidence in the portfolio was consistently good from July through November, deteriorated at the beginning of December and “rebounded’ in late March.  The student has noted this pattern without any comments on why it occurred but has included sufficient evidence to document that his/her performance has returned to a level that the reviewers agree meets standards for all the competencies with the exception of reflective practice.  They feel that the lack of explanation of the performance pattern makes it difficult to make a judgment about this competency.  What should they recommend to the promotions committee?

Reflective Practice Competency:  Demonstrates habits of analyzing cognitive and affective experiences that result in identification of learning needs leading to integration and synthesis of new learning

Key Lessons Learned So Far and Next Steps

Ø      Using portfolios for summative assessment requires significant attention to process issues in order to establish reliability and fairness, and quality of evidence is critical for making valid decisions.

Ø      Consultants are extremely helpful in establishing credibility and providing expert advise.

Ø      Creating a culture that values and supports portfolio assessment is critical for successful implementation.

                                        Portfolio Assessment System

Questions

Ø      What is the best way to conduct a standard-setting process?

Ø      What can be done to establish the credibility of the portfolio approach to summative assessment?

 Literature Cited

Ø      Driessen, E., C. van der Vleuten, L Schuwirth, J van Tartwizk, and J Vermunt.  The use of qualitative research crtieria for portfolio assessment as an alternative to reliability evaluation: a case study. 2005. Medical Education 39:214-220.

Ø      Koretz, D.  Large-scale portfolio assessments in the US: evidence pertaining to the quality of measurement.  1998.  Assessment in Education: Principles, Policy and Practice 5:309-335.

Acknowledgement:

Supported in part by The Cleveland Foundation

"Student Portfolio Approach to Educating Physician Investigators"

(Grant #L2004-0031)