The Purdue Signals debate

There appears to be some debate around the efficacy of Purdue’s Signals Project and the effect it is having on student retention. You can see examples of this debate here, here and here. I guess I would classify myself as a believer in Purdue’s course signals work, but my belief does not stem from their prediction algorithm. I think the novelty in their work is in the way that they represent the data and provide, albeit basic, affordances for action. They calculate the chance of a student’s success and provide this information to the professors and students. I think this is a key and perhaps, overlooked, point.

Last year we argued that data represented by learning analytics arises from interactions between a whole bunch of complex systems. We suggested that correlations within learning data tend to be quite distinct at the macro levels while being far less distinct at the micro levels. For example, student LMS activity correlates with student grades across large numbers of students but becomes far less distinct when you drill down to individual units or students. Human behavior along with the plethora of factors that influence it, are incredibly diverse and complex. I know that my behavior in online courses can be extraordinarily unpredictable, depending on what is happening elsewhere in my life on any particular day.

I’m no statistician (actually if I died, the net statistical knowledge in the world would probably increase) but I think we have a tendency as a sector, to spend too much time over-analyzing data. I wonder if the value of Purdue Signals lies within the fact that it catalyzes some form of action whether by the student or professor? And, at least to some extent, the value of this action cannot be measured especially with something as enormously complex as student retention. It may just be me, but I fail to see how we can adequately measure the impact of a single factor on something as complex as retention with a zillion contributing factors. I’m sure my statistical friends will yell at me now but I think George Siemens summed it up nicely:

‘Who knows why people do what they do? The point is they do it, and we can track it and measure it with unprecedented fidelity’ 
(Siemens & Long, 2011)

1.    Siemens G, Long P. Penetrating the Fog: Analytics in Learning and Education. Educause Review [Internet]. 2011 13/8/2012; 46(5):[9 p.]. Available from: http://www.educause.edu/ero/article/penetrating-fog-analytics-learning-and-education.

Advertisements

2 thoughts on “The Purdue Signals debate”

  1. Except, the problem isn’t actually that the inputs are many, or that it’s hard to discern what’s causing gains (i.e. the confounding problem).

    Meaning, in confounding you have an impact on your dependent variable via something related to the independent variable. It gets messy in this situation to determine where the effect comes from. For instance:

    > Being a sociology major (confounding variable) is positively associated with persistence (dependent variable).
    > Being a sociology major is positively associated with taking Course Signals classes (independent variable).
    > Taking a Course Signals class (independent variable) is positively associated with persistence (dependent variable).

    But that’s not the case here, Purdue actually did a pretty good job by randomly selecting sections to use Signals.

    In this case the problem is potentially more severe. We’re saying there’s a possibility (still only a possibility) that the association holds, but that causality runs the other way. That is, the dependent variable is actually causing changes in the independent variable not the other way around. That’s not fuzziness of impact, that’s a basic research methods problem.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s