Educational data mining, learning analytics, academic analytics and complexity

Associated with the dominance of the LMS in higher education is a burgeoning interest in educational data mining which is the harvesting and analysis of user activity and interaction data captured by the LMS (Macfadyen & Dawson, 2010). George Siemens (2011) blogged recently about the three distinct terms used when talking about the application of educational data mining.

• Educational data mining is concerned with developing methods for exploring the unique types of data that come from educational settings.
• Learning analytics is the measurement, collection, analysis and reporting of data about learners and their context.
• Academic analytics a mixture. More aligned with traditional business intelligence in higher education.

From my perspective there could be another dimension to these definitions and that is the difference between tactical analytics and strategic analytics. Tactical analytics is a subset of learning analytics designed to assist the student or teacher at their point of need and at the time of need. It is about what is happening within this context right now. Academic analytics is more strategic analytics in that it is (perhaps aggregated) data that is analysed retrospectively. Actually I think it may have been George who alluded to something like this in one of his Slideshare presentations.

On another note, here at CQUniversity we have been tinkering with learning analytics for some time now. The trouble I am beginning to appreciate is that analytics is a retrospective indicator of what has happened in a complex adaptive system and consequently only provides limited insight into the here and now. As the interdependent systems and their agents that constitute an e-learning environment evolve and adapt, the measurements of their behaviors and interactions within the e-learning environment will also change and inhibit their predictive value. Based on this and in my particular context, I am beginning to think that the bigger picture is how educational data mining, learning analytics and academic analytics contribute to creating interventions in e-learning when e-learning is considered as a complex system. Especially when considered against a backdrop of universities being managed like a ‘machine’ with replaceable parts and a belief that problems can be solved by rational and reducible deduction (Plsek & Greenhalgh, 2001).

 References

Macfadyen, L. P., & Dawson, S. (2010). Mining LMS data to develop an “early warning system” for educators: A proof of concept. [Article]. Computers & Education, 54(2), 588-599.

Plsek, P. E., & Greenhalgh, T. (2001). Complexity science: The challenge of complexity in health care. BMJ (Clinical Research Ed.), 323(7313), 625-628.

Siemens, G. (2011). Learning and Knowledge Analytics.   Retrieved 1/11/2011, 2011, from http://www.learninganalytics.net/?p=131

 

 

Rewarding high performing teachers

The media is reporting today that the Australian federal government has set in place a scheme that rewards high performing teachers with bonuses. On the surface, rewarding high performing teachers with bonuses sounds like a great idea. The troubles I perceive with this scheme is that it assumes that motivating good teaching is simply a matter of money and the criteria the government is going to use to identify high performing teachers are inadequate.

According to this news article, “teachers will be assessed through student performance data, lesson observations, parental feedback and teacher qualifications” as well as the controversial and compulsory NAPLAN literacy and numeracy tests. The main issues I have with this are:

• There is enormous scope in the performance criteria for task corruption, not to mention the inherit inaccuracy of the criteria themselves. For example they seem to be correlating parental feedback and teacher qualifications with teacher quality when there are obvious flaws with this approach. I should imagine the private schools, who are often more concerned about their public image than their curriculum, are already planning on how to corrupt the underlying intent of this task to their marketing advantage.
• Stick and carrot does not work with any task that is cognitive in function. Motivation for improvement for workers with cognitive functions comes from autonomy, mastery and purpose, not financial reward.
• The problems inherit in the NAPLAN process are well known and publicized. Reinforcing a questionable process like NAPLAN by linking it to a measure of teaching quality seems somewhat bizarre to me.

Self and Peer Assessment Tool

CQUniversity recently undertook a mapping project that evaluated the levels at which the university’s eight graduate attributes were represented in all undergraduate courses. This process was part of a systematic approach to embed its nominated generic skills and attributes into the curriculum, teaching and assessment practices of the institution (Fleming, Donovan, Beer, & Clark, 2010). With the mapping process almost completed, it has become apparent that the teamwork graduate attribute is the most under-represented graduate attribute across the curriculum.  As part of a strategy to address this deficit, the educational development team looked at ways that teamwork could be introduced into the university’s online learning spaces. Self and peer assessment was discussed as one approach that may contribute to the promotion of teamwork in the online curricula.

Extant literature espouses the virtues of self and peer assessment in higher education and in particular online education. It has been said that assessment should play a vital part in the learning process itself and the act of self assessment can be a force pushing students to engage more actively in their own learning (Roberts, 2006). Self and peer assessment can help students develop lifelong learning skills and can aid in students’ critical reflections of their own, and other’s work in parallel (RMIT, 2008). Given that CQUniversity has recently adopted Moodle as its single LMS, there existed limited technological means by which teachers could implement self and peer assessment for its online students.

One of the comments often made with CQUniversity’s LMS implementation project was the need to keep Moodle as ‘vanilla’ as possible to maximise maintainability and security. While the basis for this decision is arguable, it limited the opportunity for teaching staff to introduce functionality into their online learning that did not exist in the default Moodle system. Recognising this deficiency and the gap with the teamwork graduate attribute, the educational development team set about developing a small, simple web based system to assist the teaching staff in facilitating self and peer assessment.

CQUniversity’s Moodle LMS has a simple process whereby teaching staff can assign students to groups. These groups can then access resources and activities based on their group membership. For example, students posting to a forum that has been configured for group-work will only be able to see the posts and replies made by their team members. The process of assigning students to groups is quite simple and there is even a process whereby group allocation can be automated. The issue faced by the educational development team was that there was no opportunity within Moodle to leverage these groups to create a self and peer assessment process.

It was decided to develop version 1 of our self and peer assessment tool outside of Moodle and, by using access to a read only copy of the Moodle database, we could extract a range of information useful for the self and peer assessment web system (known as SPA). Information such as course and group membership, email addresses and names could be extracted from the Moodle database via an update script created for this purpose. AJAX drag and drop functionality was utilised where possible to maximise the usability of the system and a simple, clean user interface was the goal. Authentication is handled via standard LDAP libraries that interface with the university’s authentication systems. While we recognise that this self and peer assessment tool lacks the features of more mature systems designed specifically for self and peer assessment, it does integrate with our existing systems and makes it very easy for the teaching staff to implement self and peer assessment in their courses within the CQUniversity context. Our thinking is that by making it easier for teaching staff to assess group-work, they will be more likely to experiment with assessment strategies that fall outside their ‘comfort zone’. The following are some screenshots taken from the system.

The standard login screen where users are authenticated.

SPA Login Screen

The following screen is the main SPA page where the users are presented with a list of their courses taken from the Moodle system. They can add a self and peer assessment to their course by selecting the Add SPA link or review the status of existing self and peer assessments in the SPA status tab.

 

SPA Main Page

The next screen shows the SPA status screen where the teacher can monitor the number of student responses for existing self and peer assessments.

SPA status

On the SPA configuration screen there are three tabs; questions, groups and emailing. Questions are added to the self and peer assessment by dragging them to the right hand column.

Question selection

Groups from Moodle are selected in the same way. Note that on this screen it will show students who are not included in the selected groups and will therefore require the teacher to go to the Moodle site and fix up their group allocations.

Moodle group selection

Once the configuration has been completed and the students have finished their group task within Moodle. All the teacher has to do is to click send in the email tab and a link will be sent out to each student. The link is unique to each student and they will asked the selected questions about themselves and the other members of their groups. The following is an example of what the students will see with the names blanked out to protect the innocent. Note that there is no submit button as their selections are saved automatically. The students can revisit their selections at any time up until the due date expires.

Student Screen

This system is being piloted this term with a small group of students (31). We will expand the pilot next term to cater for all the interest we are getting in the system. Most of the credit for this system is attributable to Rolley for his awesome visual design skills and Damo for both his patience with teaching this old dog new programming tricks, and his ability to make complex code look simple.

Our todo list remains long and includes leveraging some of the better charting libraries to develop the reports that staff require. This is more complex than first thought as there are least two different interpretations of self and peer assessment among the two teaching staff assisting the trial. The first person wishes to use the system to identify discrepancies between  what the student self evaluated and how their peers evaluated them. The second is to give the students the raw feedback from their peers along with a grade based on the average responses. Both are applicable in their own context and the reporting functionality needs to reflect these requirements. Any thoughts or comments would be most welcome.

Fleming, J., Donovan, R., Beer, C., & Clark, D. (2010). A whole of university approach to embedding graduate attributes: A reflection. Paper presented at the DEHUB Education 2011 Summit. Retrieved from http://beerc.files.wordpress.com/2010/11/graduateattributespaper.pdf

RMIT. (2008). Self and Peer assessment. Retrieved from http://mams.rmit.edu.au/71ra0k9io8yzz.pdf

Roberts, T. S. (2006). Self, Peer and Group Assessment in E-Learning. Bundaberg, QLD, Australia: Information Science Publishing.

Measurement and complexity

In my previous post I talked about a potential danger associated with the burgeoning interest in academic analytics whereby there may be a temptation to use it as some sort of performance metric. This post is a reflection of my thinking around this area.

Like most other nations, Australian universities are increasingly required to justify the expenditure of public funds and to demonstrate ‘value for money’. Since the financial crisis first began in mid-1995, the emphasis on overt management has increased and the extent of the organizational loose-coupling has been reduced (Deem, 1998). This appears to be an attempt to use increased emphasis on the management of academic performances and cultures as a panacea that compensates for having considerably reduced resources. It has also been suggested that the increase in overt management, control and regulation of academic labour seem to have replaced collegiality, trust and professional discretion (Deem, 1998). Along with the increase in overt management comes the tendency to measure anything and everything.

“Just because we have the desire and arguably the means and methodology to measure everything, is this incessant focus on detail and analysis dragging us down? Causing us to lose focus on the key intangible factors which allow us to optimize our collaborative Company goals and cohesion?” (Frankel, 2011)

The point is that the tendency to measure anything and everything makes academic analytics an area of interest in this era of increasing overt management. Therefore it will be likely to attract the attention of management and government. The trouble I have with this is in terms of academic analytics is twofold. Firstly it adds to the entrenchment of the learning management system (LMS) as the dominant paradigm further removing flexibility from the system in the overall sense. Secondly, it seems to ignore the nature of the system that is being measured. I would argue that online learning via an LMS is a complex system that needs to be treated differently to other IT systems such as accounting systems, student records systems and other academic systems. In other words it is not necessarily a linear system where cause (proportionally) follows effect.

“A complex system is one that is adaptive to changes in its local environment, is composed of other complex systems, and behaves in a non-linear fashion where changes in outcomes are not proportional to changes in input” (Shiell, Hawe, & Gold, 2008)

“[complexity] concerns itself with environments, organisations, or systems that are complex in the sense that very large numbers of constituent elements or agents are connected to and interacting with each other in many different ways”  (Mason, 2008).

Essentially, the point I make about academic analytics is that it is data that results from interactions within a complex system and as such, is highly contextual at the micro level. For example, there are statistical and mathematical models that describe the behaviour of traffic in some cities. At the macro level, traffic flows can be predicted and enacted upon by city officials wishing to improve the efficiency of the system. This is analogous to the whole of LMS statistics that we have been extracting as part of the Indicators project. I would suggest that the same traffic system would struggle in terms of its predictive ability if the total number of cars in the city were reduced to less than 100. There would simply not be enough cars to generate a critical mass by which statistical predictability could be ascertained. This is analogous to the course level use of academic analytics where the sample size is quite small in comparison to the whole of LMS example.

In terms of the practical application of academic analytics, I maintain that it is most useful when used by the teacher, or student at their point and time of need. At the course level tactical data is required whereas at the whole of LMS level, the data is mainly strategic. The teacher’s conceptions of learning and teaching, their experience with teaching online courses, their technical aptitude and a whole bunch of other things contribute to the student experience. So better tools that can tactically demonstrate how online courses are being utilised by the staff and students can only help if applied at that level. David alludes to this in one of his recent posts.

Any thoughts or comments?

Deem, R. (1998). ‘New Managerialism’ and higher education: the management of performances and cultures in universities in the United Kingdom. [Journal]. International Studies in Sociology of Education, 8(1), 23.

Frankel, E. (2011). Do We Have to Measure Everything?   , from http://www.humanresourcesiq.com/metrics/articles/do-we-have-to-measure-everything/

Mason, M. (2008). Complexity theory and the philosophy of education. Educational Philosophy and Theory, 40(1), 15.

Shiell, A., Hawe, P., & Gold, L. (2008). Complex interventions or complex systems? Implications for health economic evaluation. British Medical Journal, 336, 3.