Identifying ‘at risk’ students is the easy bit

We are currently trialling a simple system designed to help teaching academics identify students who may be at risk of failing. We are hoping to identify struggling students earlier than we previously could so as to more efficiently target support and academic interventions. Basically we are amalgamating data from the student information system (SIS) and the learning management system (LMS) which in our case, is Moodle. At my university we have a large array of academic disciplines and students can study via a number of modes other than online or distance learning. This makes predictions based on previous student results, student demographics and student patterns of online behavior from the LMS very difficult at best. Our thinking, based in complexity science, is the information needs to be directed to the point and time where it can best be used to influence the outcome. In our case, and for this particular trial, the point of need sits with the academic teacher.

The following is a ‘dummy’ screen grab from the system which is explained further down.

Image

  • Mail merge, simply allows the teaching academic to select multiple students in order to send a personalized email.
  • Prior fail indicates whether or not this student has failed this particular course previously.
  • Pass rate is the number of courses this student has passed out of the number they have attempted.
  • Load is the number of courses that this student is attempting this term.
  • GPA is a self explanatory.
  • Clicking on the student name takes the teacher to another page with more details on the students academic history.
  • Week. This is arranged by weeks of the term. (Moodle courses at my university are made available to students two weeks prior to the official term start). The numbers in these columns are simply the number of clicks that these students have made within the course site during that week.

As the title of this post suggests, this is the easy part. Once a student has been identified by the teaching academic as potentially being at risk of failing, then what? The reasons that students fall into the ‘at risk’ category are as extraordinarily diverse as our students. Some may be struggling academically, some financially or personally, and some (like me) are just struggling for time. Every student’s situation is different and a one-size-fits-all approach to intervention is not going to work. This is why I continue to be fascinated by the extraordinary effort universities are putting into refining their ‘at risk’ student identification algorithms. Accurate statistical models are all well and good, but  all rather pointless without an intervention strategy that is capable of dealing with diversity and complexity.

Advertisements

LMS and natural selection.

In a previous post I wrote about limitations of Learning Management Systems in general. It’s quite easy to find criticisms of such systems, but far more difficult to find solutions to the previously noted issues that they bring to a university but that’s what I believe is required. Concepts like personal learning environments (PLEs), while potentially a vision for the future are probably someway off for the average organization when you consider they tend to be focused on centralizing services and technologies for reasons such as risk mitigation and other commercial pressures. Business and organizational issues, rightly or wrongly, dictate that a LMS be used and it could be said that this comes at the cost of innovation and experimentation into to other avenues of teaching and learning. The organization mandates and LMS be used and associated policy ensures this is adhered to. Unfortunately this has the side effect of creating organizational resistance to any changes that may be seen to “compete” with single chosen system such as emerging teaching and learning innovations like PLEs. I believe the issue is one of balance between commercial realities and the need to research and innovate. It appears to me to be currently weighted disproportionately away from research and innovations around teaching and learning.

That said, a pragmatic approach needs to be taken by folk such as myself who are embedded in a context over which they have little influence which, for me, means to have a closer look at the problems of the LMS and seek solutions and alternatives to these issues within the confines of the situation I’m presented with. When time permits (if ever) I’d like to generate a list of well known criticisms of learning management systems and fit these into particular categories in order to define spheres of responsibility so that limitations can be assessed and addressed within our context. Should PLEs or another emerging concept start to become mainstream I doubt that it will be via a revolution but will be more of a Darwinian process of evolution.

Limitations of the LMS.

While we are considering the best ways to move forward in a project that mines the activity data stored by CQUniversity’s Learning Mangement System (LMS) in order to identify aspects and behaviors that impact on teaching and learning, we must remain mindful of the limitations of both our approach and, probably more importantly, the LMS itself.

The limitations of an LMS in a typical university context have been widely documented. Some of these limitations as detailed by Beer & Jones (2008) include:

  • Organizational and instructor focus. They tend to meet the needs of the organization and the instructor more so than the learner.
  • IT Culture. Centralize and control approach to educational technology.
  • Limited informal learning possibilities.
  • Course based model that limits community development.

While others such as Coates,James and Baldwin (2005) look at the influences an LMS has on teaching and learning such as:

  • LMS features aren’t as important as the educational application of the features.
  • They are based on an overly simplistic understanding of the relationship between teachers, knowledge and student learning.
  • They can rely too heavily on automatic forms of assessment.
  • “Codified” teaching can lack the flexibility and nuances essential to effective teaching.
  • LMS are not pedagogically neutral but by their very design guide and influence teaching.
  • LMS have uncertain effects on student engagement.
  • LMS create new and complex divisions of labour between administrators and teachers.
  • LMS vendor’s appear to be seeking commercialization of content. An example is deals with publishing houses.

For me learning is about interactions and there are three categories of interactions involved.
Learner – Content. This is the interaction between the learner and the content.
Learner – Instructor. Conversations and correspondance between the learner and the instructor.
Learner – Learner. Conversations, collaboration and correspondence between learners.

Most of my learning is, I believe,  based on the last two or the social aspects of learning and it’s these two that I believe are lacking in your typical LMS. For example. Imagine your at school and have a question for your teacher. You enter the building, navigate your way to the only room where questions can asked, its pitch black, you don’t know if there is anyone else there, you ask a question and then repeat the process later on to see if the question has been answered.  Now imagine IT have designed the building, the hallways, the room and the conversation protocols and you may have an understanding of how an LMS limits the scope of social discourse within it’s boundaries.

So what does all of this mean to the indicators project? We need to know what the limitations are of the system we are measuring to ensure we are measuring the right things. I believe the three levels of interaction are a place to start and the seven principles is a framework that we can use to interpret our results.

References.

Beer, C. & Jones, D. (2008). Learning networks: harnessing the power of online communities for discipline and lifelong learning. In D. Orr, P.A. Danaher, G. Danaher & R.E. Harreveld (Eds.), Lifelong Learning: reflecting on successes and framing futures. Keynote and refereed papers from the 5th International Lifelong Learning Conference (pp. 66-71). Rockhampton: Central Queensland University Press. http://hdl.cqu.edu.au/10018/13162

Coates, H. J., Richard.Baldwin,Gabrielle. (2005). “A critical examination of the effects of learning management systems on university teaching and learning.” Tertiary education and management 11(2005): 19-36.

LMS. Education’s swiss army knife

I find it interesting the way that Universities think about their learning management systems (LMS) which are often the primary channel of interaction with their students who coincidentally are the primary source of income for said universities. Take my opinions with a grain of salt as I am probably tainted by my prior experience which was 17 years in the IT service delivery industry where we delivered a tangible product or service that was relatively easy to quantify and therefore easier to rate our effectiveness. Not so with teaching and learning which is a process that is difficult to measure and has a great deal more variables and complexity than IT service delivery.

Learing Styles
Learning Styles

People have different learning styles, different motivations, varying preferred learning environments and vastly different backgrounds and experiences. Then Universities attempt to provide a cookie cutter approach by providing an online learning environment in which the student must learn what they are there to learn within the confines of a single tool. Add to this that every course is different in the way that it can/is/should be delivered online will vary greatly depending on the course’s context. By this I mean Physics 101 will apply a different approach to the pedagogical construction of the course than will Rail Signaling 101. I’m not saying what Universities are doing is wrong by selecting single LMS products, there are valid reasons for it, but I do believe they are probably over simplifying what is an incredibly complex process by restricting the environment in which occurs to a single product with a single underlying paradigm that is one of knowledge dissemination. The problem is further complicated by the fact that that university courses have much more of a commercialized nature than has historically been the case and I suspect that this will only get worse as universities will be pushed further by financial restrictions.

Its the paradigm of dissemination that has links to the Indicators project that forms part of my masters. The indicators takes the three categories of interaction (learner-learner, learner-content, learner-instructor) and looks at (online only) user behavior within the LMS against the seven principles of effective under graduate education. Through the Indicators project we hope to provide contextual evidence that the LMS does provide effective content interaction but doesn’t necessarily facilitate well the higher levels of learning such as can be found in the Blooms taxonomy.

Blooms Taxonomy

The learner-content is the one interaction where the LMS appears to perform very well however the knowledge gained by content interaction can be quite low level and arguably forgotten the soonest. Perhaps this is due to the limitations whereby the content or theoretical knowledge can’t be applied to a particular context or situation as it is learned and therefore can’t be internalized as efficiency as a real situation may allow. Nona has done some work in the area of situated congnition that is worth revisiting and perhaps looking more closely to Snowden’s work, particularly pattern matching, could be worthwhile.

Incorrect quote removed. (Sorry. It was getting late)

There is some research that indicates it may even be more effective than traditional methods like face to face (Ladyshewsky 2004)

So assuming that the LMS is good at the learner-content interaction but is not as good at the learner-learner and learner-instructor interactions, how can we improve the balance? I suspect the answers are in the nature of the social set of interactions and what is learned from other people. My current thinking is that it relates to experience. Working with someone with vast experience on a particular task is, for me, my favorite way of learning which begs the question how can we embed this sort of rich exposure to experience within the confines of the LMS? I’ve no idea at this stage but we are hoping our investigations in to the LMS usage will provide insights into how the students are using the existing resources and then inform a direction we can take to assist with the social aspects that appear to be lacking in the LMS.

Siemens, G. (2004). “Learning Management Systems: The wrong place to start learning.” retrieved 20 February from http://www.elearnspace.org/Articles/lms.htm.

Ladyshewsky, R. K. (2004). “Online learning versus face to face learning: What is the difference?” Teaching and Learning Forum 2004. Curtin University of Technology. Retrieved 20 February from http://lsn.curtin.edu.au/tlf/tlf2004/ladyshewsky.html