Col's Weblog

Educational data mining, learning analytics, academic analytics and complexity

beerc — Wed, 19 Oct 2011 03:35:56 +0000

Associated with the dominance of the LMS in higher education is a burgeoning interest in educational data mining which is the harvesting and analysis of user activity and interaction data captured by the LMS (Macfadyen & Dawson, 2010). George Siemens (2011) blogged recently about the three distinct terms used when talking about the application of educational data mining.

Educational data mining is concerned with developing methods for exploring the unique types of data that come from educational settings.
Learning analytics is the measurement, collection, analysis and reporting of data about learners and their context.
Academic analytics a mixture. More aligned with traditional business intelligence in higher education.

From my perspective there could be another dimension to these definitions and that is the difference between tactical analytics and strategic analytics. Tactical analytics is a subset of learning analytics designed to assist the student or teacher at their point of need and at the time of need. It is about what is happening within this context right now. Academic analytics is more strategic analytics in that it is (perhaps aggregated) data that is analysed retrospectively. Actually I think it may have been George who alluded to something like this in one of his Slideshare presentations.

On another note, here at CQUniversity we have been tinkering with learning analytics for some time now. The trouble I am beginning to appreciate is that analytics is a retrospective indicator of what has happened in a complex adaptive system and consequently only provides limited insight into the here and now. As the interdependent systems and their agents that constitute an e-learning environment evolve and adapt, the measurements of their behaviors and interactions within the e-learning environment will also change and inhibit their predictive value. Based on this and in my particular context, I am beginning to think that the bigger picture is how educational data mining, learning analytics and academic analytics contribute to creating interventions in e-learning when e-learning is considered as a complex system. Especially when considered against a backdrop of universities being managed like a ‘machine’ with replaceable parts and a belief that problems can be solved by rational and reducible deduction (Plsek & Greenhalgh, 2001).

References

Macfadyen, L. P., & Dawson, S. (2010). Mining LMS data to develop an ‚Äúearly warning system‚Äù for educators: A proof of concept. [Article]. Computers & Education, 54(2), 588-599.

Plsek, P. E., & Greenhalgh, T. (2001). Complexity science: The challenge of complexity in health care. BMJ (Clinical Research Ed.), 323(7313), 625-628.

Siemens, G. (2011). Learning and Knowledge Analytics. Retrieved 1/11/2011, 2011, from http://www.learninganalytics.net/?p=131

Rewarding high performing teachers

beerc — Mon, 02 May 2011 03:12:29 +0000

The media is reporting today that the Australian federal government has set in place a scheme that rewards high performing teachers with bonuses. On the surface, rewarding high performing teachers with bonuses sounds like a great idea. The troubles I perceive with this scheme is that it assumes that motivating good teaching is simply a matter of money and the criteria the government is going to use to identify high performing teachers are inadequate.

According to this news article, “teachers will be assessed through student performance data, lesson observations, parental feedback and teacher qualifications” as well as the controversial and compulsory NAPLAN literacy and numeracy tests. The main issues I have with this are:

There is enormous scope in the performance criteria for task corruption, not to mention the inherit inaccuracy of the criteria themselves. For example they seem to be correlating parental feedback and teacher qualifications with teacher quality when there are obvious flaws with this approach. I should imagine the private schools, who are often more concerned about their public image than their curriculum, are already planning on how to corrupt the underlying intent of this task to their marketing advantage.
Stick and carrot does not work with any task that is cognitive in function. Motivation for improvement for workers with cognitive functions comes from autonomy, mastery and purpose, not financial reward.
The problems inherit in the NAPLAN process are well known and publicized. Reinforcing a questionable process like NAPLAN by linking it to a measure of teaching quality seems somewhat bizarre to me.

Self and Peer Assessment Tool

beerc — Wed, 06 Apr 2011 08:28:46 +0000

CQUniversity recently undertook a mapping project that evaluated the levels at which the university’s eight graduate attributes were represented in all undergraduate courses. This process was part of a systematic approach to embed its nominated generic skills and attributes into the curriculum, teaching and assessment practices of the institution (Fleming, Donovan, Beer, & Clark, 2010). With the mapping process almost completed, it has become apparent that the teamwork graduate attribute is the most under-represented graduate attribute across the curriculum. As part of a strategy to address this deficit, the educational development team looked at ways that teamwork could be introduced into the university’s online learning spaces. Self and peer assessment was discussed as one approach that may contribute to the promotion of teamwork in the online curricula.

Extant literature espouses the virtues of self and peer assessment in higher education and in particular online education. It has been said that assessment should play a vital part in the learning process itself and the act of self assessment can be a force pushing students to engage more actively in their own learning (Roberts, 2006). Self and peer assessment can help students develop lifelong learning skills and can aid in students’ critical reflections of their own, and other’s work in parallel (RMIT, 2008). Given that CQUniversity has recently adopted Moodle as its single LMS, there existed limited technological means by which teachers could implement self and peer assessment for its online students.

One of the comments often made with CQUniversity’s LMS implementation project was the need to keep Moodle as ‘vanilla’ as possible to maximise maintainability and security. While the basis for this decision is arguable, it limited the opportunity for teaching staff to introduce functionality into their online learning that did not exist in the default Moodle system. Recognising this deficiency and the gap with the teamwork graduate attribute, the educational development team set about developing a small, simple web based system to assist the teaching staff in facilitating self and peer assessment.

CQUniversity’s Moodle LMS has a simple process whereby teaching staff can assign students to groups. These groups can then access resources and activities based on their group membership. For example, students posting to a forum that has been configured for group-work will only be able to see the posts and replies made by their team members. The process of assigning students to groups is quite simple and there is even a process whereby group allocation can be automated. The issue faced by the educational development team was that there was no opportunity within Moodle to leverage these groups to create a self and peer assessment process.

It was decided to develop version 1 of our self and peer assessment tool outside of Moodle and, by using access to a read only copy of the Moodle database, we could extract a range of information useful for the self and peer assessment web system (known as SPA). Information such as course and group membership, email addresses and names could be extracted from the Moodle database via an update script created for this purpose. AJAX drag and drop functionality was utilised where possible to maximise the usability of the system and a simple, clean user interface was the goal. Authentication is handled via standard LDAP libraries that interface with the university’s authentication systems. While we recognise that this self and peer assessment tool lacks the features of more mature systems designed specifically for self and peer assessment, it does integrate with our existing systems and makes it very easy for the teaching staff to implement self and peer assessment in their courses within the CQUniversity context. Our thinking is that by making it easier for teaching staff to assess group-work, they will be more likely to experiment with assessment strategies that fall outside their ‘comfort zone’. The following are some screenshots taken from the system.

The standard login screen where users are authenticated.

SPA Login Screen

The following screen is the main SPA page where the users are presented with a list of their courses taken from the Moodle system. They can add a self and peer assessment to their course by selecting the Add SPA link or review the status of existing self and peer assessments in the SPA status tab.

SPA Main Page

The next screen shows the SPA status screen where the teacher can monitor the number of student responses for existing self and peer assessments.

SPA status

On the SPA configuration screen there are three tabs; questions, groups and emailing. Questions are added to the self and peer assessment by dragging them to the right hand column.

Question selection

Groups from Moodle are selected in the same way. Note that on this screen it will show students who are not included in the selected groups and will therefore require the teacher to go to the Moodle site and fix up their group allocations.

Moodle group selection

Once the configuration has been completed and the students have finished their group task within Moodle. All the teacher has to do is to click send in the email tab and a link will be sent out to each student. The link is unique to each student and they will asked the selected questions about themselves and the other members of their groups. The following is an example of what the students will see with the names blanked out to protect the innocent. Note that there is no submit button as their selections are saved automatically. The students can revisit their selections at any time up until the due date expires.

Student Screen

This system is being piloted this term with a small group of students (31). We will expand the pilot next term to cater for all the interest we are getting in the system. Most of the credit for this system is attributable to Rolley for his awesome visual design skills and Damo for both his patience with teaching this old dog new programming tricks, and his ability to make complex code look simple.

Our todo list remains long and includes leveraging some of the better charting libraries to develop the reports that staff require. This is more complex than first thought as there are least two different interpretations of self and peer assessment among the two teaching staff assisting the trial. The first person wishes to use the system to identify discrepancies between what the student self evaluated and how their peers evaluated them. The second is to give the students the raw feedback from their peers along with a grade based on the average responses. Both are applicable in their own context and the reporting functionality needs to reflect these requirements. Any thoughts or comments would be most welcome.

Fleming, J., Donovan, R., Beer, C., & Clark, D. (2010). A whole of university approach to embedding graduate attributes: A reflection. Paper presented at the DEHUB Education 2011 Summit. Retrieved from http://beerc.files.wordpress.com/2010/11/graduateattributespaper.pdf

RMIT. (2008). Self and Peer assessment. Retrieved from http://mams.rmit.edu.au/71ra0k9io8yzz.pdf

Roberts, T. S. (2006). Self, Peer and Group Assessment in E-Learning. Bundaberg, QLD, Australia: Information Science Publishing.

Measurement and complexity

beerc — Sun, 03 Apr 2011 01:08:04 +0000

In my previous post I talked about a potential danger associated with the burgeoning interest in academic analytics whereby there may be a temptation to use it as some sort of performance metric. This post is a reflection of my thinking around this area.

Like most other nations, Australian universities are increasingly required to justify the expenditure of public funds and to demonstrate ‘value for money’. Since the financial crisis first began in mid-1995, the emphasis on overt management has increased and the extent of the organizational loose-coupling has been reduced (Deem, 1998). This appears to be an attempt to use increased emphasis on the management of academic performances and cultures as a panacea that compensates for having considerably reduced resources. It has also been suggested that the increase in overt management, control and regulation of academic labour seem to have replaced collegiality, trust and professional discretion (Deem, 1998). Along with the increase in overt management comes the tendency to measure anything and everything.

“Just because we have the desire and arguably the means and methodology to measure everything, is this incessant focus on detail and analysis dragging us down? Causing us to lose focus on the key intangible factors which allow us to optimize our collaborative Company goals and cohesion?” (Frankel, 2011)

The point is that the tendency to measure anything and everything makes academic analytics an area of interest in this era of increasing overt management. Therefore it will be likely to attract the attention of management and government. The trouble I have with this is in terms of academic analytics is twofold. Firstly it adds to the entrenchment of the learning management system (LMS) as the dominant paradigm further removing flexibility from the system in the overall sense. Secondly, it seems to ignore the nature of the system that is being measured. I would argue that online learning via an LMS is a complex system that needs to be treated differently to other IT systems such as accounting systems, student records systems and other academic systems. In other words it is not necessarily a linear system where cause (proportionally) follows effect.

“A complex system is one that is adaptive to changes in its local environment, is composed of other complex systems, and behaves in a non-linear fashion where changes in outcomes are not proportional to changes in input” (Shiell, Hawe, & Gold, 2008)

“[complexity] concerns itself with environments, organisations, or systems that are complex in the sense that very large numbers of constituent elements or agents are connected to and interacting with each other in many different ways” (Mason, 2008).

Essentially, the point I make about academic analytics is that it is data that results from interactions within a complex system and as such, is highly contextual at the micro level. For example, there are statistical and mathematical models that describe the behaviour of traffic in some cities. At the macro level, traffic flows can be predicted and enacted upon by city officials wishing to improve the efficiency of the system. This is analogous to the whole of LMS statistics that we have been extracting as part of the Indicators project. I would suggest that the same traffic system would struggle in terms of its predictive ability if the total number of cars in the city were reduced to less than 100. There would simply not be enough cars to generate a critical mass by which statistical predictability could be ascertained. This is analogous to the course level use of academic analytics where the sample size is quite small in comparison to the whole of LMS example.

In terms of the practical application of academic analytics, I maintain that it is most useful when used by the teacher, or student at their point and time of need. At the course level tactical data is required whereas at the whole of LMS level, the data is mainly strategic. The teacher’s conceptions of learning and teaching, their experience with teaching online courses, their technical aptitude and a whole bunch of other things contribute to the student experience. So better tools that can tactically demonstrate how online courses are being utilised by the staff and students can only help if applied at that level. David alludes to this in one of his recent posts.

Any thoughts or comments?

Deem, R. (1998). ‘New Managerialism’ and higher education: the management of performances and cultures in universities in the United Kingdom. [Journal]. International Studies in Sociology of Education, 8(1), 23.

Frankel, E. (2011). Do We Have to Measure Everything? , from http://www.humanresourcesiq.com/metrics/articles/do-we-have-to-measure-everything/

Mason, M. (2008). Complexity theory and the philosophy of education. Educational Philosophy and Theory, 40(1), 15.

Shiell, A., Hawe, P., & Gold, L. (2008). Complex interventions or complex systems? Implications for health economic evaluation. British Medical Journal, 336, 3.

A potential danger of academic analytics

beerc — Wed, 23 Mar 2011 01:17:28 +0000

A member of my family works for a large semi-industrial company that expends extraordinary effort to ensuring worker safety. They have a system where any sort of safety issue, right down to a paper cut, is logged into a central system and plans are developed to prevent the issue’s reoccurrence. On the surface this sounds great, a semi-government semi-industrial company that encourages employee safety over all else. Unfortunately the implementation of their safety strategy has two serious flaws.

The divisional managers of this company are responsible for ensuring every issue relating to safety is logged into the central safety system.
One of the key performance indicators (KPI) of these divisional managers is their safety record which is calculated from the number of issues logged into the central safety system.

There was a minor incident at this workplace recently that, while not leading to an injury, demonstrated a potential safety hazard that need to be rectified. As the manager responsible for this area has a KPI linked to their safety record, the incident did not get logged with the central system. This is a classic example of task corruption and Goodhart’s law, both of which, I think, are going to be issues associated with the burgeoning interest in academic analytics.

At our institution, the Indicators project has recently attracted the attention of upper management who would like us to do a series of presentations to staff showing the link between student behaviours within the learning management system (LMS) and the student’s resulting grade. In one respect this a good thing, as I firmly believe that the path to better learning and teaching starts with informing and improving the teaching staff. However it is the potential of task corruption and Goodhart’s law that worries me when management takes an interest in these things.

The predictive value of academic analytics is not a precision instrument. Comparing the current term’s student behaviours with previous terms is really only useful as a guide as it disregards context to some extent. For example every student will use the LMS in different ways and even the same student will use the LMS differently depending on the type of course, type of assessment and what is happening in their lives at that particular point in the term. Another example is a student who is completely lost in a particular course will make a large number of clicks within this course, equivalent to a high-achieving student and still fail even though their ‘measurable’ effort indicates a much higher grade. This is why we espouse academic analytics as a tool for the teacher and, perhaps a tool for the students, so that the information they are provided with, can be contextualized by them into their situation. However there is potential for task corruption and Goodhart’s law to become an issue in this situation.

Given that one of the correlations identified by the Indicators project shows that there appears to be a relationship between the teacher’s level of engagement in the LMS course and the student’s level of engagement in the same course, I worry about a KPI being introduced around a teacher’s level of engagement in online courses. In such a circumstance, the pragmatic educator would spend a short while making random clicks within the system, whereby achieving the metric based on their LMS activity, and then getting on with their work for the rest of the term having satisfied the compliance police. Task corruption 101. Ok. This is extreme and is probably not (I hope) a realistic scenario, but it does highlight the dangers of taking complex, context dependent data and using it as a performance metric.

One of the dangers as I see it, is that while academic analytics provides some great data for educators on what is happening in learning situations facilitated by an LMS, there may be a temptation to use it as some sort of performance measurement. Dave Snowden talks about complex and complicated systems where complicated systems are receptive to the concept of good practice whereas complex systems are not due to their ever-changing context. The point here is that academic analytics is data that results from activity occurring in a complex system and not a complicated system. Therefore ‘best practice’ and a one-size-fits-all approach to measuring learning using academic analytics is not likely to work.

Quick reponse to A strange result

beerc — Thu, 10 Mar 2011 00:09:23 +0000

My previous post ‘A strange response’ generated some questions around the difference between high and low achieving students and how they use the learning management system (LMS) discussion forum. David suggested looking for the difference in complexity in the postings between high and low achieving students. The following chart is the start of that process. Note if your having trouble understanding the context of this post, have a read of the previous post and its comments. This is the average number of characters in student posts and student replies on the LMS discussion forum grouped by their resulting grades.

I know the next chart is very ‘dodgy’, but I quickly wanted to know whether high achieving students asked more questions than low achieving students. So I counted the number of question marks in the posts and replies for all the students in my sample. Note the sample size is over 8000 student courses. This is the average number of questions marks per post and reply for each of the student grade groups.

Obviously these results are not scientific, but they certainly create a path for further investigation. Please feel free to share your thoughts on what these charts might mean and perhaps avenues for further investigation.

A strange result

beerc — Wed, 09 Mar 2011 10:37:28 +0000

The folk involved in the Indicators project are always on the lookout for interesting correlations between student activity on the learning management system (LMS) and their resulting grade. One of the first correlations we found was the following, that shows a relationship between the number of clicks students make on the LMS and their resulting grade. It also shows number of clicks each grade group made on discussion forums. Note that flex students in the CQUniversity context typically refers to students who are pursuing their studies via distance education and this is often facilitated via the Moodle LMS.

The ‘Y’ axis is the number of clicks (hits) on Moodle while the ‘X’ axis is the different grades the students received. While the sample size is somewhat small by Indicators project standards (5872), it does show that, on average, the more students use the LMS, the better their results. Of course, and like most of the correlations we have found, they are quite distinct when looked at by average over large sample sizes, but become far less clearly defined when the number of students being sampled is reduced.

One of the things I am particularly interested in, are the patterns around student utilization of LMS discussion forums. At a basic level there are three different types of interaction in any online learning situation:

Interactions with content. This is represented by clicks on web pages, links and files such as PDF and PPT.
Interactions with other learners. Often this is represented by forum posts, replies and reads amongst the student group.
Interactions with the instructor. This is also often represented by forum posts, replies and reads between the students and the instructor.

The following chart shows the relationship between the number of flex student clicks within the LMS discussion forum and their resulting grade. Note that n=8928.

As you will notice, the pattern is similar to what we saw before with the overall hits on the LMS; the more the flex student interacts with the LMS discussion, the better their result. To further expand on this the following chart shows the relationship between the number of forum posts and replies the flex students make and their resulting grade.

Again the trend is fairly clear. The students receiving the higher grades have a tendency to make more posts on the LMS discussion forums. None of this should really come as any great surprise to anyone involved in online education. There is ample research that shows the link between student engagement and their resulting grades and one of my earlier research projects suggested that there is some value in using hits on the LMS as a proxy measure for student engagement. This is based on the fact that online students are ‘invisible’ to the instructor in that the instructor cannot see the ‘glint in their eye’ and this makes it hard to determine if their motivation is waning.

Over the weekend, I was trying to develop a script that produced ratios of the three engagement types that I mentioned earlier. Just to see what correlations, if any, were present. One of the first results from the script is in the following chart and shows something that is both very interesting and also quite perplexing. It shows what percentage of each grade group of flex students are hits on the discussion forum.

Why on earth do the failing students spend more energy (for the want of a better word) on discussion forums than pass or even credit students. Sure the number of clicks the failing students make on discussion forums is less than both pass and credit students, but in terms of the proportion of their hits that are forum based, they exceed the pass and credit students. So give that, in the past, we have shown that instructor interaction with students on discussion forums has a significant influence the level of effort that online students make, I compared this result with students from courses where instructors demonstrated high levels of forum against students from courses where instructors demonstrated low levels of forum activity.

Interestingly, aside from showing the same pattern as the previous chart where failing students exhibited a higher proportion of forum hits than passing and credit students, this chart also suggests that higher instructor activity on the forums leads to proportionally higher student activity on the forums. Not a great surprise I guess but I am really curious as to the dip in the chart lines when looking at student forum activity in relation to their overall activity. Any ideas anyone?

Instructor participation and evaluation response rates

beerc — Sun, 20 Feb 2011 03:46:19 +0000

David raised an interesting point in a recent blog post where he wondered about the possible correlation between academic participation in a course and the response rates in end of term student evaluations.

I used the average level of course coordinator ‘views’ of the discussion forums to distinguish between below average and above average instructor participation. With these two groups I compared the number of distinct students submitting feedback against the total number of students in the course.

While there are probably some variables that I have neglected to account for, there appears to be a 4% increase in response rates for courses where the academic participation on the discussion forums was above average. The same trend applied when I used the number of instructor discussion posts and replies as the method of distinguishing between below average and above average instructor participation. I found this result a little surprising. I would have expected a great variation than 4% considering that active participation in discussion forums by the instructor would tend to hint at a greater effort of relationship building. I guess the fact that the 4% variation is across the entire student cohort regardless of campus and the next step would be to look at the response rates of online students only.

Context Context

beerc — Thu, 17 Feb 2011 10:57:31 +0000

A colleague of mine today blogged about learning theories and context. He believes that the purist approach that says, “learning theory ‘x’ is right and learning theory ‘y’ is wrong” is misguided as it disregards context. I thoroughly agree with him. I also believe that this observation is not just limited to learning theories but could also apply to learning styles. To ‘pigeon hole’ someone as possessing a particular learning style irrespective of the context of the learning situation seems a bit silly to me. Or am I just showing my ignorance?

In the same blog post Damien talks about Connectivism and he describes a context where there is limited access to the technologies that may be utilized to develop a learning network. My over simplistic view is that Connectivism can still apply to learning situations prior to the information age. Take a master-apprentice situation for example. The apprentice does form a network albeit with far fewer nodes than the Internet affords, but it is still a network. In this case they have formed a link with external knowledge (the Master), as we might have a link to a particular website. With a memory like mine I believe that know-where trumps know-how. Of course I have been reading a book on Actor-Network theory and it is leaving me a little confused

Start of a small project

beerc — Sun, 13 Feb 2011 01:50:18 +0000

I am currently working on a small project to help some course coordinators facilitate self and peer assessment of group-work in their courses. The concept is very simple and it is quite surprising that there appear to be very few tools available to course coordinators in this space. Basically, for a given group-work task, the students are asked a series of questions about how they, and their team-mates performed in regards to teamwork. This allows the learners to critically reflect upon their own learning which has long been shown to positively enhance the learning process. The students are provided with an opportunity to consider their own learning along with how they have learned and unfortunately these elements are often overlooked when designing assessment.

Essentially this project is extracting group membership information from the Moodle learning management system and emailing students a link that is personalized for their group. The link is a simple survey that asks the students to answer a series of questions based on their experience of the task the group had to perform. While not ideal in that it is a separate system from the LMS, it is hoped that this trial will help inform the development of a Moodle module that will perform the same function albeit in a more integrated and seamless way. Fortunately a colleague at my university has edited a book about self and peer assessment in E-Learning and I hope to make contact with him at some stage to get some feedback on the project.

Roberts, T. S. (2006). Self, Peer and Group Assessment in E-Learning. Bundaberg, QLD, Australia: Information Science Publishing.