A potential danger of academic analytics

A member of my family works for a large semi-industrial company that expends extraordinary effort to ensuring worker safety.  They have a system where any sort of safety issue, right down to a paper cut, is logged into a central system and plans are developed to prevent the issue’s reoccurrence. On the surface this sounds great, a semi-government semi-industrial company that encourages employee safety over all else.  Unfortunately the implementation of their safety strategy has two serious flaws.

  • The divisional managers of this company are responsible for ensuring every issue relating to safety is logged into the central safety system.
  • One of the key performance indicators (KPI) of these divisional managers is their safety record which is calculated from the number of issues logged into the central safety system.

There was a minor incident at this workplace recently that, while not leading to an injury, demonstrated a potential safety hazard that need to be rectified. As the manager responsible for this area has a KPI linked to their safety record, the incident did not get logged with the central system. This is a classic example of task corruption and Goodhart’s law, both of which, I think, are going to be issues associated with the burgeoning interest in academic analytics.

At our institution, the Indicators project has recently attracted the attention of upper management who would like us to do a series of presentations to staff showing the link between student behaviours within the learning management system (LMS) and the student’s resulting grade. In one respect this a good thing, as I firmly believe that the path to better learning and teaching starts with informing and improving the teaching staff. However it is the potential of task corruption and Goodhart’s law that worries me when management takes an interest in these things.

The predictive value of academic analytics is not a precision instrument. Comparing the current term’s student behaviours with previous terms is really only useful as a guide as it disregards context to some extent. For example every student will use the LMS in different ways and even the same student will use the LMS differently depending on the type of course, type of assessment and what is happening in their lives at that particular point in the term. Another example is a student who is completely lost in a particular course will make a large number of clicks within this course, equivalent to a high-achieving student and still fail even though their ‘measurable’ effort indicates a much higher grade. This is why we espouse academic analytics as a tool for the teacher and, perhaps a tool for the students, so that the information they are provided with, can be contextualized by them into their situation. However there is potential for task corruption and Goodhart’s law to become an issue in this situation.

Given that one of the correlations identified by the Indicators project shows that there appears to be a relationship between the teacher’s level of engagement in the LMS course and the student’s level of engagement in the same course, I worry about a KPI being introduced around a teacher’s level of engagement in online courses. In such a circumstance, the pragmatic educator would spend a short while making random clicks within the system, whereby achieving the metric based on their LMS activity, and then getting on with their work for the rest of the term having satisfied the compliance police. Task corruption 101. Ok. This is extreme and is probably not (I hope) a realistic scenario, but it does highlight the dangers of taking complex, context dependent data and using it as a performance metric.

One of the dangers as I see it, is that while academic analytics provides some great data for educators on what is happening in learning situations facilitated by an LMS, there may be a temptation to use it as some sort of performance measurement. Dave Snowden talks about complex and complicated systems where complicated systems are receptive to the concept of good practice whereas complex systems are not due to their ever-changing context. The point here is that academic analytics is data that results from activity occurring in a complex system and not a complicated system. Therefore ‘best practice’ and a one-size-fits-all approach to measuring learning using academic analytics is not likely to work.

About these ads

5 thoughts on “A potential danger of academic analytics

  1. Kevin Wilcoxon

    Thanks for the great post. Very well said. Do you suppose that providing a plus or minus range of error might serve to discourage a false sense of accuracy? Could such a number be calculated?

    Kevin

    Reply
    1. beerc Post author

      G’day Kevin

      Thanks for your comment. I guess it would be possible to calculate a range or error with academic analytics. Not being a statistics expert I could not say for sure. Our approach is less about the detail contained within the numbers than about getting data to the folk who can make the most sense out of it in context. Students and teachers.

      Col.

      Reply
  2. Scot Aldred

    Hi Col,
    Thanks for an excellent posting. You raise some interesting points.
    The issue for management, as I see it, should be focused on student performance, not academic clicks in a discussion forum. I know that your data clearly indicate a close correlation between lecturer interaction on discussion forums and student engagement and performance.
    Currently, academic staff are under significant pressure to increase the quanta of research publications and grant acquisition. Despite what management claim, this is the majority of hard data used to determine promotion and success in academe.
    Having worked at your institution for 16 years I understand this more than most. Whilst teaching at your institution, my LMS student interaction data were amongst the highest, with correspondingly high student performance. Unfortunately, student performance is not a common metric used for promotion, despite the effect that high level interaction has on reducing attrition (a hot topic at your institution).
    The flip-side of high levels of interaction is the time it takes. The time demands are significantly higher than traditional face to face models leaving less time for chasing research grants, undertaking research projects and writing journal articles.

    While it is very satisfying having your online courseware recognised in awards and being recognised by students as being in the top ten teachers in your organisation, it does not lead to promotion and success in academe.

    The culture of valuing research quanta over quality teaching will not change overnight, if it ever does! This is why state owned universities are becoming less and less relevant in the 21st century and why in the future, independent organisations will overtake universities in teaching areas.
    This is a risky business for your organisation as more than 95% of its income comes from teaching. Neglect your core business at your peril!!

    Once again, many thanks for posting and continued work in this important area.

    Best,

    Scot.

    p.s. give my best to DJ

    Reply
  3. Pingback: Col's Weblog

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s