Measurement and complexity

In my previous post I talked about a potential danger associated with the burgeoning interest in academic analytics whereby there may be a temptation to use it as some sort of performance metric. This post is a reflection of my thinking around this area.

Like most other nations, Australian universities are increasingly required to justify the expenditure of public funds and to demonstrate ‘value for money’. Since the financial crisis first began in mid-1995, the emphasis on overt management has increased and the extent of the organizational loose-coupling has been reduced (Deem, 1998). This appears to be an attempt to use increased emphasis on the management of academic performances and cultures as a panacea that compensates for having considerably reduced resources. It has also been suggested that the increase in overt management, control and regulation of academic labour seem to have replaced collegiality, trust and professional discretion (Deem, 1998). Along with the increase in overt management comes the tendency to measure anything and everything.

“Just because we have the desire and arguably the means and methodology to measure everything, is this incessant focus on detail and analysis dragging us down? Causing us to lose focus on the key intangible factors which allow us to optimize our collaborative Company goals and cohesion?” (Frankel, 2011)

The point is that the tendency to measure anything and everything makes academic analytics an area of interest in this era of increasing overt management. Therefore it will be likely to attract the attention of management and government. The trouble I have with this is in terms of academic analytics is twofold. Firstly it adds to the entrenchment of the learning management system (LMS) as the dominant paradigm further removing flexibility from the system in the overall sense. Secondly, it seems to ignore the nature of the system that is being measured. I would argue that online learning via an LMS is a complex system that needs to be treated differently to other IT systems such as accounting systems, student records systems and other academic systems. In other words it is not necessarily a linear system where cause (proportionally) follows effect.

“A complex system is one that is adaptive to changes in its local environment, is composed of other complex systems, and behaves in a non-linear fashion where changes in outcomes are not proportional to changes in input” (Shiell, Hawe, & Gold, 2008)

“[complexity] concerns itself with environments, organisations, or systems that are complex in the sense that very large numbers of constituent elements or agents are connected to and interacting with each other in many different ways”  (Mason, 2008).

Essentially, the point I make about academic analytics is that it is data that results from interactions within a complex system and as such, is highly contextual at the micro level. For example, there are statistical and mathematical models that describe the behaviour of traffic in some cities. At the macro level, traffic flows can be predicted and enacted upon by city officials wishing to improve the efficiency of the system. This is analogous to the whole of LMS statistics that we have been extracting as part of the Indicators project. I would suggest that the same traffic system would struggle in terms of its predictive ability if the total number of cars in the city were reduced to less than 100. There would simply not be enough cars to generate a critical mass by which statistical predictability could be ascertained. This is analogous to the course level use of academic analytics where the sample size is quite small in comparison to the whole of LMS example.

In terms of the practical application of academic analytics, I maintain that it is most useful when used by the teacher, or student at their point and time of need. At the course level tactical data is required whereas at the whole of LMS level, the data is mainly strategic. The teacher’s conceptions of learning and teaching, their experience with teaching online courses, their technical aptitude and a whole bunch of other things contribute to the student experience. So better tools that can tactically demonstrate how online courses are being utilised by the staff and students can only help if applied at that level. David alludes to this in one of his recent posts.

Any thoughts or comments?

Deem, R. (1998). ‘New Managerialism’ and higher education: the management of performances and cultures in universities in the United Kingdom. [Journal]. International Studies in Sociology of Education, 8(1), 23.

Frankel, E. (2011). Do We Have to Measure Everything?   , from

Mason, M. (2008). Complexity theory and the philosophy of education. Educational Philosophy and Theory, 40(1), 15.

Shiell, A., Hawe, P., & Gold, L. (2008). Complex interventions or complex systems? Implications for health economic evaluation. British Medical Journal, 336, 3.


2 thoughts on “Measurement and complexity”

  1. Col,

    I think it’s time that a statement like this

    “I maintain that it is most useful when used by the teacher, or student at their point and time of need.”

    Get’s some back up. For me, this is where your PhD should. THink of some way to prove this. Or something else you think is important or central to your view of analytics.

    Do some fail-safe probes with different models of using analytics to make interventions in real courses, programs or faculties. Research what the results of those interventions are.

    Though I imagine you’ll find it hard to do the sort of contextual, responsive intervention that we think would end up with the best results.


  2. I heard a funny comment today to describe the quality assurance and compliance approaches of measuring learning and teaching quality.

    “You don’t fatten a pig by weighing it”

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s