Expected and unexpected results

Even though we have a long way to go on the Indicators project with regard to organizing our data sets I jumped the gun the other night and ran some queries that pulled data from a variety of sources into Perl and created a temporary database table on my Mac. I then ran some more scripts on this processed data to produce some results that I thought would be fairly obvious and would validate the methods we are using to extract the data from their original data sources. As with all of this early data, it should be taken with a grain of salt as I’m yet to consult a statistics expert on the validity of the methods I’m using.

I was of the opinion, based on my readings and converstions with other folk, that student success would be linked to the participation rates of both themselves and their instructors so I ran a couple of scripts to reasure myself that this was indeed the case.

Avg Hits per Grade

Firstly I looked at student participation rates and linked that to their success in their courses and the result was fairly conclusive. Note that this was done quickly prior to our consultation with a statistical expert and is subject to my limited experience in statistical analysis. I grabbed the LMS hitcounts of close to 30,000 online students and averaged these out for each grade across close to 400 online courses and the result was pretty much what I expected. The more the students accessed the online courseware the better their result would be when averaged over such a large number of students and courses. There will always be exceptions and I hope to have a look at some in the course of this project.

The second thing I looked at was the effect staff participation had on the students results. Naturally I expected to see student

Staff Activity vs Student Performance
Staff Activity vs Student Performance

results improving relatively to the participation rates of the teaching staff. How ever this wasn’t the case and I could discern no pattern from the resulting data which was a surprise for me. I sent the results to a colleague of mine who responded with a list of possibilities for the result

* The methodology used wasn’t sound from a mathamatical, statistical perspective.

* Increasingly the larger courses have an established set of resources that resused each term and aare good enough that the impact of academic intervention isn’t significant.

* LMS activity simply doesn’t have any casual link with student performance.

Note. To explain the graph the vertical axis is staff hit counts while the horizontal is a rating of student performance based on grade. The more I think about it the less I like the rating system I chose.

I suspect, like most research projects, the initial results aren’t quite what was expected and now a great amount of time will be expended to try and make sense of the unexpected result.