Bridging the data driven decision-making gap in Universities.

This “waffle” was conceived after watching the following video that one of my colleagues distributed. It raised some interesting points that could potentially relate to the indicators project. Another video relating to the same thing can be found here at TED talks.

Student engagement is well represented in the literature and can be summarised as the behavioural intensity and emotional quality of a person’s active involvement during a task (Reeve, Jang, Carrell, Jeon, & Barch, 2004). We know that students’ engagement in the learning process is a key performance indicator (KPI) when trying to predict student success and some of the Indicators project’s data tends to confirm this. Universities essentially provide the students with an environment that facilitates interactions with between the students, the course content and the course instructor. They are also given opportunities to reflect upon what they have learned in the form of summative and formative assessment tasks and activities. So I guess you could summarise learning as a series of cycles each containing a process of interacting and reflecting where engagement is the student’s degree of participation in these processes. Knowledge creation 101 I guess.

Is the process any different for teaching staff in terms of improving their practice?  They still require feedback and reflection in order to improve their practice don’t they? If learning from a student’s perspective is comprised of interactions and reflection, than I don’t see how this is vastly different for the teacher or any worker in any trade if it comes to it. I am often surprised at how little credence is given to the role of the teacher in a learning situation. Given that the teacher is usually responsible for developing the course content, course activities and facilitates the social discourse, I’d suggest that they are the capstone upon which the improvement of teaching and learning occurs. Trigwell’s (2001) model of teaching seemingly confirms this.

So I am saying that staff engagement in the process of improving teaching and learning is important, yet it seems to me to be underrepresented in both the literature and organizations. For example a search on Google scholar for the term “staff engagement” returns 1650 results while a search for “student engagement” returns 33400 results. I find this interesting as how do you achieve student engagement without engaged staff? Ok I realize that it is vastly more complex than this and I tend to over-generalize/over-simplify but can you see my point? Teaching staff are very important when you are trying to improve teaching and learning at a university and like the students, they require feedback and opportunities for reflection in order to improve.

A possibly related and interesting point raised by the video mentioned previously, was that the stick and carrot approach to managing workers, works only when the task is mechanical or algorithmic in nature (by stick and carrot I mean reward the good and ignore, or punish the bad). As soon as cognitive function is required for a task, the stick and carrot approach no longer works. This means that workers involved in tasks that are more than rudimentarily cognitive in nature (such as teaching), reward no longer works as a motivator for improvement and creativity in the worker. They say that, in this situation, worker motivation is, instead, linked to three factors:

• Autonomy. The ability to be self directed.
• Mastery. The desire to improve.
• Purpose. The sense of purpose in what they do.

So where am I going with this and how does it relate to the Indicators project? The indicators project is broadly about maximising the potential of data that is already being collected by the university for the purpose of improving teaching and learning. That is, data that is currently either not being used or not being used to its full potential. We know that learning management systems (LMS) and student administration systems contain a wealth of data that can be scrutinised for correlations relating to teaching and learning. Information such as student activity counts versus results from previous offerings can be given to the teacher, at the point of need, in order provide them with a point of reflection on how their current student cohort is performing. This could give the teacher a sense of ownership over the evaluation of their course in context. So its not a stick or carrot but more a vessel for autonomy and mastery and allows the academic to analyse their LMS course during the course of a term.

The use of data mining and data analysis is nothing new to universities and most have some sort of business intelligence unit dedicated to the extraction and analysis of corporate data. Where they have gone wrong in my opinion is that these units are often focused on providing data specifically for management. Don’t get me wrong, strategic data on enrolment numbers, student demographics, fail rates, pass rates etcetera is excellent and highly useful information for everyone working for a university and possibly links university staff with a greater sense of purpose as mentioned before. However there is a single point upon which the university depends, that is the teacher. They are the “point of contact” between the university’s “product” and the student. I’d argue that teachers need ‘tactical’ data as well as ‘strategic’ data.

CQUniversity recently produced an academic dashboard that provides the teachers with an interface into the university’s strategic data. They define the dashboard as an interface that displays an organisation’s strategic data and trends. It is an excellent tool that provides an enormous amount of information to teaching staff and gives the teaching staff an idea of how their courses relate to other courses and programs in terms of fail rates, pass rates, campus performance etc.

However while it may have a limited ability to identify potential issues with a specific course or program, it doesn’t help the teacher rectify the problem. A nice analogy is that the strategic data is required to fight the war whereas tactical data is required to fight the battles that make up a war. I’m not arguing that strategic data is more or less important than tactical data, but rather they are both a critical element of data driven decision-making for universities.

Another point of difference between the Indicators project and business intelligence data relates to the point of need. Strategic data can generally be viewed in isolation from the day-to-day activities of staff and students. It is strategic data that cannot generally be used in the running of a particular course day-to-day. The Indicators data works best when placed at the point of need. Using the example of the Indicators ‘at risk’ student system, we know the average level of activity for students who received passing grades at this point in the term for previous offerings of this course. This is compared to the activity levels of current students to give the teacher some ‘tactical’ information that can be used proactively to address potentially lagging students. The point of need for this ‘tactical’ data is, in the case of CQUniversity, the Moodle LMS.

So business intelligence areas tend to produce abstracted and strategic data. I believe that teachers need additional data, that is tactical and contextual.

Reeve, J., Jang, H., Carrell, D., Jeon, S., & Barch, J. (2004). Enhancing Students’ engagement by increasing teachers’ autonomy support. [Journal Paper]. Motivation and Emotion, 28(2).

Trigwell, K. (2001). “Judging university teaching.” The International Journal for Academic Development 6(1): 65-73.

Ethical issues around data mining learning management systems

Web servers are the computers that host the various web pages that make up the Internet. Web servers also log a range of information about people who visit their web pages such as:

• IP address
• Type of operating system and browser
• Operating characteristics such as screen resolution and colour depth
• The URL that referred the visitor to the site
• The country the visitor is from
• Where this visitor clicks while visiting the site

It is common practice for web maintainers to analyse web activity log data to generate marketing intelligence by analysing visitor’s online behaviour and turning this information into marketing knowledge. It is a similar story in higher education where Learning management systems like Moodle and Blackboard aggregate these logs into database tables where the records can be analysed by the institution. For example, our Indicators project is looking at ways that this recorded data can be converted into information that can inform and improve university teaching and learning. We have delivered several presentations based on the Indicators project and a common question that arises is about the ethics of using web server logs for this purpose.

There appear to be two main themes or ethical concerns around the use of web data mining and these relate to privacy and individuality. According to a paper that discusses ethical issues in web data mining;  “Web mining does, however, pose a threat to some important ethical values like privacy and individuality. Web mining makes it difficult for an individual to autonomously control the unveiling and dissemination of data about his/her private life” (Wel & Royakkers, 2004). They go on to say that web usage mining raises privacy concerns when web users are traced and their actions are analysed without their knowledge.

Privacy is a conceptually fragile and enigmatic term but in the context of web data mining it is commonly referred to as the control of information about oneself. In terms of the Indicators project we are de-identifying individuals and courses as well as aggregating data to look at patterns of activity across student groups that consist of thousands of students. This, I suspect, does not present any privacy concerns, as it is impossible to identify individuals within the data sets.

De-individualisation has been defined as a tendency of judging and treating people on the basis of group characteristics instead of on their own individual characteristics and merits. It has been said that when group profiles are used as a basis for decision-making and formulation of policy, or if profiles somehow become public knowledge, the individuality of people is threatened. My interpretation of this is that it more relates to how the collected data is used rather than how the data is collected and I can think of many situations where the use of such information could be deemed unethical.

In terms of the Indicators project, where we are endeavouring to provide research on how students are using the LMS, I do not see any issues relating to privacy as the identity of individuals is not disclosed and cannot be inferred. The argument about Individuality is more complex as it relates to how the information is used. The following is taken from the privacy statement on the Australian Privacy Commissioner’s web site:

When an individual looks at our website, our internet service provider (WebCentral) makes a record of the individual’s visit and logs (in server logs) the following information for statistical purposes:

• the individual’s server address
• the individual’s top level domain name (for example .com, .gov, .org, .au, etc)
• the pages the individual accessed and documents downloaded
• the previous site the individual visited and
• the type of browser being used.

We do not identify users or their browsing activities except, in the event of an investigation, where a law enforcement agency may exercise a warrant to inspect the Internet service provider’s server logs. (http://www.privacy.gov.au/component/content/article/545#mozTocId230471)

This seems to be a standard inclusion into the privacy statements of most governments and organisations including CQUniversity. There appears to be very few privacy statements attached to web sites that actually spell out how the information will be used but I suspect that most will be used to either improve services and process or simply as a marketing intelligence tool.

What do you think?

Wel, L. v., & Royakkers, L. (2004). Ethical Issues in web data mining. Ehtics and Information Technology, 6, 11.

Slides used in today’s Moodlemoot presentation

A few people have asked for a link to the slides I used into today’s presentation. Here it is.

Using the Indicators project data to identify at risk students

It is well known in the literature that early intervention for ‘at risk’ students can often help the individual student overcome, or get assistance with, the issues that lead them to fall into this category. Particularly in Australia, there is an institutional imperative to respond to rates of student attrition due to the negative effect that attrition has on the funding the institution receives from the government (Hinton, 2007).

It is also widely known that dropout rates tend to be higher in distance-learning contexts than in face-to-face programs (Robai, 2002). Learning via distance in the modern era is typically facilitated online by learning management systems (LMS) and research has suggested that the rates of attrition for online students can range between 20-50% higher than for on-campus students (Dawson, Macfadyen, & Lockyer, 2009). In an online learning environment the Instructor’s visibility over the student’s engagement level is limited when compared to a face-to-face environment where they can see the ‘glint in their eyes’ and can tell at a glance how the student is engaging in the lesson.

Meanwhile, it has been said that the fundamental measure of student experience with a LMS is the degree to which students use the system (Caruso, 2006) which appears to align with the historical precedent where class attendance has been used a simple metric for measuring face-to-face student engagement (Douglas & Alemanne, 2007). A fortuitous feature of most LMS is their ability to track and store vast amounts of data on student and designer behaviour (Heathcoate & Dawson, 2005). This data can be used during the course by the instructor to monitor student activity during the term and most LMS provide a basic interface that facilitates this process.

However while LMS collect vast amounts of data on staff and students, the interfaces they provide for analysis of the data is often basic and does not contribute to converting this data to information or knowledge. Additionally, it is common for universities to use a separate system to the LMS for tracking student information and results. It could be said that this restricts the instructor’s ability to compare current student activity data with data from passing students in previous course offerings using only the tools supplied by the LMS.

There is a popular saying that states “a hammer sees the world as a nail”. My ‘hammer’ for the last couple of years has been the Indicators project and I have been thinking about how we can harness the data gathering potential of an LMS to assist in the prediction of ‘at risk’ students. To date, the Indicators project has been looking at ‘lag’ indicators in the data. This means we have been looking at what has happened rather than using this information to try and predict what might happen. Lately I have been working on a simple script that might help assist CQUniversity teaching staff in the early identification and intervention of ‘at risk’ distance students based on data extracted from our local Moodle LMS. The plan is to trial the script with some courses during the next term with the intent to improve its functionality and generate some research output. The following is an explanation of how it works keeping firmly in mind that it is in its very early stages of development.

For any particular course hosted on Moodle:

• It gathers the FLEX students from the previous offering who received a PASS grade.
• It takes the current day of the current term and calculates the same day from the previous offering.
• The activity count or hitcount average from the passing FLEX students is generated for this point in the previous term.
• The current FLEX students activity counts are compared with the average from the previous offering and the current students are place into three groups; below average, about average and above average.
• These groups are displayed on the webpage and the teacher has the option to either email individual students or they can mailmerge the students as a group.
• Clicking on the student’s name takes the view directly to a page that displays the student’s profile.

A sample webpage generated by the script can be viewed here and is based on a current live course. Note that course and student details have been removed.

Some points to note:

• The data is extracted from a copy of the Moodle backend database that is at least 24 hours old.
• It only works for courses who have a previous offering on Moodle where the student grades have been posted.

This is only the first version and is really only a test of the concept. Although initial testing of the script with a couple of live courses has been encouraging it is by no means a solution to the broader problem of student attrition. However, it might be another useful tool in the teacher’s toolkit when delivering an online course.

References:

Caruso, J. B. (2006). Measuring Student Experiences with Course Management Systems [Electronic Version]. Educause, 2006, from http://net.educause.edu/ir/library/pdf/ERB0619.pdf

Dawson, S., Macfadyen, L., & Lockyer, L. (2009). Learning or performance: Predicting drivers of student motivation. Paper presented at the Same places, different spaces. Proceedings ascilite Auckland 2009, Auckland.

Douglas, I., & Alemanne, N. D. (2007). Measuring Student Participation and Effort. Paper presented at the International Conference on Cognition and Exploratory Learning in Digital Age, Algarve, Portugal.

Hinton, L. (2007). Causes of attrition in first year students in science foundation courses and recommendations for intervention. Studies in Learning, Evaluation, Innovation and Development, 4(2), 13.

Rovai, A. (2002). Building Sense of Community at a Distance. International Review of Research in Open and Distance Learning, 3(1).