Technology Tools

Leveraging analytics to study and improve learning

In an age of learning management systems, online gradebooks and homework, discussion forums, and clickers, the progress of a college course generates a rich, deep stream of data. Learning analytics aims to put this information to use; first to learn more about how teaching and learning happens, and then to improve it. The E2Coach project at the University of Michigan provides an example of how this happens.

Our work began with measurement of the past, using information from our data warehouse to explore how preparation and background affect the performance of students in our introductory physics courses. Examining the records of 48,579 students over a period of 14 years, we found many interesting correlations. For example, a student’s prior Michigan GPA predicts their grade to an accuracy of half a letter grade. This should not surprise: after all, grade point average is the average grade a student receives in all classes. So long as introductory physics is not unique, a strong GPA-grade correlation is guaranteed.

There are many sources of dispersion in a grade prediction like this—factors which cause some students to perform better than expected while others do worse. Other measures of preparation present in our data, from standardized test scores and high school GPA to student age and first generation status, cannot fully explain outcomes, at best reducing the dispersion in our predictions by about 25%. Historical data was insufficient to fully account for student performance.

Prompted by this discovery, we expanded our search, interviewing both students and instructors in an effort to identify behaviors which lead to success. In an interesting twist, we used our grade predictions to identify students from prior semesters who had done better and worse than expected and interviewed them extensively. This process allowed us to establish a rich, complex, body of information about student success; a suite of advice for how to do better than expected. Some is general—start studying early—but some applies only to particular students—biology majors are likely to focus too strongly on memorization in their study.

Having done all this work, we know pretty well what’s going to happen to each student. But we also know what they should do to beat this expectation from the past. This is actionable intelligence—specific advice we’d like to give to each student. How can we deliver this advice? Given the size of these courses (400-700 students), meeting with each student is impossible. Yet we know the importance of personalized feedback and advice: different students need different advice. What to do?

The E2Coach project solves this dilemma by applying computer tailored communication technology. We have adopted the Michigan Tailoring System, an open-source tailored communication system developed in our Center for Health Communications Research. MTS allows us to decide what we’d like to say to each student, code this in logic using a very mature development tool, then generate information rich, customized web pages to thousands of students. Tailored communication of this kind has been widely adopted in public health applications, helping millions to meet challenging public health goals like quitting smoking, controlling their diabetes, and making difficult cancer treatment decisions.

E2Coach advice can be presented in many ways, and we work to take advantage of all that behavior change and communications experts tell us about how to do this best. For example, we make extensive use of testimonials, presenting advice from students rather than professors, students matched to our tailoring subject’s identity. We also provide a wide variety of normative information—showing students how the amount of time they spend on exam study and the effort they put in on homework compares to the rest of the class, and to that of students who perform at the level they aim to achieve. Each student receives their advice as a personalized web site, updated as their status changes throughout the term.

We launched the system in January 2012, delivering this tailored advice to 953 students across four introductory physics courses. Feedback from active users has been positive:

“It makes things easier … when you have people giving you pointers on how to study, how to succeed in the course, things to look out for, and such.”

“I usually read through all of the advice every time I get a notification, and especially before and after exams. It's motivating to see how successful students in the past approach the course.”

“I really like how it predicts my final grade for the class. Seeing this is really motivational and helps me get down to studying.”

Computer tailored communication can be a powerful tool for acting on the knowledge generated by learning analytics; a kind of invention engine. Because it is designed from the start to generate messages, MTS gives us great flexibility. Messages can address course content, advise on study methods, offer reminders, present data graphics, and direct students to online and real world support services. We can generate tailored messages for instructors, pointing out students who are doing poorly, or exceptionally well, and send reminders to advisors, giving them detailed background on students they should contact.

One great advantage of tailoring is that it is designed from the start to reach everyone, not just students at risk. There are important messages for all, including the very best students in the class—it is as important to speak to them as it is to reach out to those who struggle—and E2Coach’s computer tailored communication allows us to do this. The use of technology to tailor how we interact with information and one another is one wave of the future. One day soon, my personal computer will know me; it will record what I know, attend to what I like, and be aware of my preference to check email before reading the paper in the morning, except on the weekend. E2Coach is a step toward this. It has helped us to better understand how to personalize education, and given us the chance to test how best to deliver this personalized content.

To learn more about the Michigan Tailoring System, visit: http://chcr.umich.edu/mts/ or read about it in the February 2013 issue of Information Week Education.