NGLC’s Wave IIIb grantees, funded in 2012 to implement breakthrough models for college completion, convened in mid-October to share celebrations and challenges of their first year of implementation. But the agenda went beyond networking and trading war stories. Breaking new ground to improve student success through many aspects of their programs, they wanted not only to hear from one another but also to gain new insight from experts in areas of particular concern. Last summer, they had identified the areas where they most wanted to tap expertise: 1) assessment in the context of competency-based approaches, 2) student support for nontraditional learner populations, and 3) planning the marketing of nontraditional programs to nontraditional learners. To meet those needs, the convening featured experts deeply knowledgeable in these areas: Steve Klingler of Western Governors University on assessment in competency-based models, Diego Navarro of Cabrillo College and the Academy for College Excellence (ACE) on support for nontraditional learners, and Misa Misono of IDEO on marketing strategy. We’ll share summaries of Navarro’s and Misono’s thinking in upcoming posts here. Today, we’ll look at what Steve Klingler of WGU shared with the group.
Moving to a competency-based approach demands a comprehensive redesign of the whole structure of attaining a degree, from how the learning is offered to how students respond to that offering and how they demonstrate its effect on them. Western Governors University, which began accepting students in 1999, was competency-based from the beginning: that is, it has always used competencies rather than seat time as the way in which it measures its outcomes. With more than a decade of experience to draw upon and with more than 41,000 current students as well as 27,000 graduates, WGU has valuable expertise to share with the numerous other campuses that are now moving to develop and adapt their degree programs to focus on the attainment of competency as the primary goal for the learner. Klingler described key considerations for the measurement of competency:
It’s important to take a comprehensive research-based approach to design both of the competencies and their assessments.
The subject matter experts who structure the competencies for WGU utilize criteria from a variety of sources including practical experience, job task analysis, published standards, and other research data. Different types of instruments are used, including performance tasks, objective exams, live observations, simulations, and portfolios. Instruments address each individual competency statement in a manner appropriate to the type of objective being assessed, with students receiving credit only after passing every assessment. Items are designed in a collaborative process that ensures they are truly accurate, objective and thorough, including a psychometric edit for alignment, clarity, sensitivity, bias, and fairness.
The development environment for assessments should be professional and well designed.
All development at WGU takes place centrally, and workflow is carefully managed. It’s possible to look at the history of and prior versions of each item.
Objective assessments should be delivered in a standardized and secure way.
For those assessments that involves taking objective tests, students use a secure, proctored online environment. The testing interface, which utilizes the Excelsoft SARAS assessment system, is customized to resemble other WGU systems, allowing students to focus on the questions rather than on the testing environment. An adaptive, tailored testing approach allows for the shortening of test times by curtailing the use of questions that are too hard or too easy for the test-taker’s level. Computer scoring means that immediate results are available.
Online proctoring is the key to ensuring integrity and accurate results.
WGU uses a combination of systems, Kryterion and ProctorU, to enable security of the testing envirionment. An advance readiness check allows students to ensure ahead of time that everything is working, reducing test anxiety. There are checks on identity that, for example, utilize biometrics, and on security via the lockdown of the browser, a wide-angle webcam, and audio and screen monitoring during the test.
Students should receive feedback in a manner appropriate to a program driven by competency.
Rather than “failing,” students learn that they have not yet demonstrated competency. Results are presented in the form of a coaching report, in which test results are organized by course topic, which makes it feasible for the student and his/her advisor to modify his or her study plan appropriately for the next round.
Performance assessments require a distinct but equally thoughtful approach.
WGU students also demonstrate competencies through culminating experiences in which they are presented with open-ended problems solvable in multiple ways, allowing them to apply their abilities in an approximate real-world context. Dedicated evaluators score these, using rubrics, and provide specific written feedback. WGU uses calibrations and monitors inter-rater reliability to ensure both accuracy and consistency.
Portfolios represent a third and especially comprehensive approach to competency-based assessment.
At WGU, these may be new work or assembled from prior performance tasks. As with performance tasks, evaluators score them against a rubric. They can be shared with potential employers or others via a personal URL.
A PowerPoint deck from Steve Klingler’s presentation at the convening is available here.