This is a continuation from part of the recent webinar I was involved in.
Building a community of participants is a first step, but how can student engagement be measured?
First a disclaimer! Measuring student engagement is very difficult if not impossible. I liken it to long-range weather forecasting where you look at a stack of historical data and try to model what is going to happen in the future, except student engagement introduces much more randomness into the system than occurs with the weather. All we can do with regards to measuring engagement is look for indicators that correlate with student success. Our approach with the indicators project is to look for correlations between LMS data and student results knowing that results are not necessarily indicative of learning. Luckily, we have at CQUniversity, a cohort of students who are typically reliant on the LMS for the interactions around their learning so by focusing on this cohort we can ‘filter’ out the influences of face-to-face and blended teaching. As we are only best-guessing with regards to student engagement, our approach is to present the data to the instructors who are in the best position to interpret an act on the data we produce. Additionally, while measuring student engagement is potentially useful for online students where we can’t see the glint in their eyes, better course and program design that promotes engagement is a vastly more important pursuit.
Most students engage to be successful?
I would suggest yes and from what we have seen so far in our foray into academic analytics would confirm this. We have found that there is a distinct correlation between the quantity of clicks within the Moodle site and the student’s grade. This, however, needs to be tempered by the fact that our project is looking at aggregates and averages and not individuals; and student grade is by no means a perfect representation of learning. We also notice that engagement (as indicated by Moodle activity) greatly increased after course redesigns that centred around an authentic objective.
How effectively are those engagements being measured now?
When looking at online courses, LMS activity counts seems to be a reasonable indicator of engagement from what we are seeing here. Most LMS provide a basic interface into the data they collect, however they often don’t interface into student administration systems where the student behaviours can be compared with student grades. There are however pockets of folk such as Purdue University and in particular Shane Dawson from UBC. Shane is looking at visualizing the social interactions and content analysis of LMS discussion forums. While there are also limitations to this, it appears to be one of the better approaches that I know about.
Once we have the tools to measure student engagement, how can the collected data be effectively analysed?
A lot of the analytical technologies have a sense of “replace the human” about them. This is an evolutionary dead end that could be better utilized by combining the technology with the human mind. Also in Australia, the course coordinators design and deliver the course, which means their conception of L&T influences the way that students engage with the course. Additionally, there is a great deal of diversity in course designs and delivery strategies. Consequently this makes it difficult to develop a consistent metric for student engagement, as there is so much variation to be considered. So we are aiming to use the analytics information to inform teachers and students so that they can evaluate the information in terms of their context.
What are we looking for in the data?
At the stage we are at with the Indicators project, we are simply looking for correlations between student behaviours and their resulting grades. For example we recently developed an ‘at risk’ student identification tool for Moodle that looks at patterns of behaviour for students for each grade group from previous course offerings and provide forecasts to teaching staff on students who are falling below the pass level at this point in the term based on student behaviours in previous terms. It then provides email and mail-merge links for students where the teacher can intervene earlier than was previously possible. We are currently working on building this tool into a Moodle block and using the same code-based, producing an indicators for the students along the lines of what Purdue have done with their signals project.
Are there effective tools available now for analyzing that sort of academic data?
This is where universities are behind the business world. Businesses have been using data warehouses and delving into the predictive value of the data that they accumulate for some time now. Universities have an array of systems that gather all sorts of data relating to their students, staff and how they engage with these systems and its only quite recently that there are moves to mine this data for informing and improving teaching and learning. I think in terms of tools available right now, SNAPP is the best of a developing bunch.
What role does the LMS play in implementing academic based analytical technologies? What data should be collected? Analysed?
For better or worse most universities are using an LMS of one sort or another to meet their eLearning requirements. For L&T related data mining this provides a single point by which student activity can be monitored, measured and analysed. The LMS is also where course interactions are facilitated and is the point of need for any predictive data if it is to assist the teachers or students. At least for the moment it is the place where both the data is being accumulated and the place where staff and students are interacting. LMS tend to gather basic data on individual staff and student clicks in the system in much the same way as web servers do. I would like to see LMS activity recording pay more attention to click stream recording so as to ascertain how students are navigating around the sites and when they are using particular features in relation to their assessment schedules. There are some issues relating to the recording of user behaviour within the LMS that needs to be considered. There are privacy and ethical considerations for universities tracking staff and student behaviour within the LMS although the online world seems to ‘get away’ with more privacy infringements that the offline world I suspect. Another couple of things from my perspective that are important when talking about student engagement relates to university contexts. For example it could be said that some of the academic workload models could be said to be ‘discouraging’ of engagement and reflective practice. Of course, the other issue that arises with relation to the use of social media and new things in general, is the resistance of staff to change.
What are your thoughts on the place of social media in today’s learning environment?
I love the connective nature of social media but there appears to be a disconnect between the closed nature of the typical LMS and the open nature of social media. Open standards like RSS can help with products like BIM that integrate social media with the LMS.
How effectively is social media software being used today? How can that be changes?
It’s currently limited due to the closed nature of the LMS although this is changing, albeit very slowly.
Col's Weblog 
