Reimagining Assessment

Getting the Most Out of Skill Practice

The ASSISTments project at Worcester Polytechnic Institute has a mission to improve education through scientific research without compromising student learning time. One of our areas of research is the question of what is the right amount of practice and how does feedback and spacing support that practice.

Let’s start by looking at something very familiar: A practice sheet for integer subtraction.

Is this just the right amount of practice?  When will the student find out if they made any mistakes?

This was a homework assignment my son did in 6th grade. He worked through all the problems and wrote the answers down. The next day his teacher read the answers to the class but for some reason my son never marked the problems he answered incorrectly.  Can you find them?  I found three wrong answers.  My son is a decent math student so what did he get out of this exercise?  What about the student who had a misconception on how to solve these problems?  He will spend his homework time at night solving these problems incorrectly over and over again only to find out the next day in class he answered most of them incorrectly.  What is the effect of this sort of wrong practice and then not finding out about it until the next day? 

We built and connected two features in ASSISTments, Skill Builders and the Automatic Reassessment and Relearning System (ARRS), to study these questions. As per our mission, we like to do our studies in a way that doesn’t disrupt student learning.  These features are currently being used for normal practice by hundreds of students each day.

Here is how the features work. 

Skill Builders: A teacher starts the process the same way my son’s teacher did by assigning practice for a skill; in his case it was integer subtraction. In ASSISTments, however, he gets a Skill Builder instead of a worksheet.  Now when he starts to work he gets both immediate feedback and hints to help him out if he struggles. He is motivated to be careful in his work because his homework will be done when he answers three problems correctly in a row. In the screenshot below, he answered the first question correctly (see the green check in the upper left corner) and then he answered the next problem incorrectly (see the red x) but now he’s looking at the first hint for the second problem and getting the help he needs.

If the student cannot answer three questions correctly in a row after attempting 10 items, we make him wait overnight to try again.  We believe that more wrong practice, even with hints, is not helping this student.  This is where the teacher needs to get involved so we give a signal to his teacher on the report that he is struggling.

In this report the teacher can see not only which students are struggling on the assignment but also the students who “Mastered” the skill and in how many attempts.  Here is an example of the report the teacher receives.

Notice four of these students knew the concept and answered three correctly in a row right away. The first student took a lot more time than the others. The third student worked for almost five minutes and maxed out at 10 questions.  He needs more help than ASSISTments can give him. His teacher knows this without taking any time to grade the student’s work.  

I used the word “Mastered” in quotes because we know that the four students who completed the assignment with just three attempts probably know the skill and will not forget it but what about the student who took six attempts to complete the assignment?

Automatic Reassessment and Relearning System (ARRS): To measure retention and support and to help students who struggle with retention, we developed ARRS.  With ARRS, a student automatically gets an assessment on the skill one week later (one of our experiments is to determine what spacing is best and for which students). If the student answers the problem correctly he will not be reassessed on that skill for another two weeks and he will receive a green check on his reassessment test progress (see the screenshot below).  If the student answers incorrectly they automatically get an assignment to relearn the skill and the teacher is notified on a report that the student is struggling with the skill.  On the relearning assignment, the student works until they answer three questions correctly in a row, receiving immediate feedback and hints along the way and then gets reassessed again in a week, trying for that green check.  Once they get four green checks they are done. You can see here that this student had two green checks and hopefully got another one when she was reassessed on 1/7/2014.

Does the ASSISTments approach work? So how does this approach compare to my son’s homework assignment?  We conducted a study and found that when skills received reassessment and relearning, students were able to maintain mastery performance levels, whereas skills that did not receive such treatment showed a decrease in performance levels (read more about this study here). There is more to learn about how to have students practice skills. We are sure there is a place for computer-support in this effort.  We will continue to use ASSISTments to help us study ways to improve spacing and practice as well as other areas of learning.