The following is a piece of writing that may form the basis of a paper sometime next year as time permits. Its needs a great deal of work.
Learning management systems (LMS) have become almost ubiquitous in higher education in Australia with most universities using either a commercial or an open source variant. Whilst they have several widely published weaknesses, such as the ability to effectively provide participant interactions, their popularity among higher education institutions remains high and further research into their impact on students is required. The following paper draws upon data from the CQUniversity Indicators project to look at the effects of LMS hosted discussion forums on fully online (flex) students.
The adoption of LMS as an approach to eLearning is almost universal (Coates, James, & Baldwin, 2005) and they have become perhaps the most widely used educational technologies used in higher education, only ranking behind the Internet and common office applications (West, Waddoups, & Graham, 2006). Some have said that the LMS dominance in the higher education sector is due to institutions adopting LMS simply because “everyone else is” (Northover, 2002) rather than on their educational merit. Whether this situation has developed for the benefit of the learner as opposed to the organization or instructor is arguable (Beer & Jones, 2008) but also somewhat irrelevant, considering that most institutions are now using an LMS to deliver eLearning.
A contributing factor to the fact that LMS are the dominant technology used for eLearning in universities could be the increase in demand for flexibility by students (Hinton & Bradshaw, 2004; Northover, 2002). There are increasing numbers, of adults who are employed in the workforce and are returning to university for interest or career advancement and life circumstances often prevent people in this category from committing to full time study (Hinton & Bradshaw, 2004). As a result, they often choose to undertake their studies via distance or flexible delivery modes that most universities now offer (Hinton & Bradshaw, 2004). These flexible modes are often delivered entirely via the LMS and are independent of place or time as opposed to face-to-face university courses where lectures and tutorials are held in specific locations at specific times and the learner is required to be present to participate. When undertaking a flexible course offering, the LMS is often the only mechanism by which the student can interact with course content, fellow students and their instructors. As the LMS is the only system a student in a flexible learning situation will be exposed to, it is important that the LMS effectively deliver the range of interactions that learners require in order to achieve their learning goals.
From the perspective of a student, learning online or otherwise can be described by three fundamental interactions (Moore, 1989).
Moore (1989) states that learner-content interaction is a defining characteristic of education and is the process of intellectually interacting with content that changes the learner’s understanding, the learner’s perspective, or the cognitive structures of the learner’s mind. Whilst framed as a criticism for being too content focused, most LMS effectively provide learner-content interaction (Siemens, 2004b) and others have said that they may even be more effective at facilitating learner-content interaction than traditional methods like face-to-face (Ladyshewsky, 2004). A potential explanation for this is that the learners can survey the content at their leisure and at their own pace in a manner that matches their individual learning style. A typical LMS provides many features by which an instructor can sequence and present content and its only recently that LMS vendors have started extending tools and offerings beyond simple content sequencing (Siemens, 2004b). LMS features that facilitate learner-content interactions vary between systems but broadly include features like announcements, gradebooks and access to instructor-uploaded files.
Learner-learner interaction can be one to one or one to many, can occur through group work, and can occur in or out of real time and is most often represented within an LMS by discussion forums that allow threaded, text based discussions to occur within a specific course. Learner-instructor interaction is where the learner interacts with the teacher or expert and this interaction is facilitated by the LMS via email, asynchronous discussion forums or synchronous chat.
Moore (1989) describes the three basic interactions required for a given learning situation and it has been suggested that in general, LMS can effectively provide learner-content interactions. The main feature used by LMS to provide learner-learner and learner-instructor interactions is the discussion forum, although most LMS provide an email feature that can perform a similar function. However due to the difficulties in the day-to-day moderation of course emails in courses with large student numbers the email feature is not widely used by the Blackboard LMS at CQUniversity and is not included in this study. Similarly, the asynchronous communication tool provided by CQUniversity’s current version of Blackboard, is not widely used due to poor performance resulting from technical limitations with the tool.
Learning is social
Social constructivism as a learning theory emphasizes the importance of culture and context in understanding (Kim, 2006) and therefore suggests the importance of interpersonal communication in learning. Similarly, connectivism, which is a relatively new theory in education born from the internet age, suggests that learning and knowledge rests in a diversity of opinions and learning is a process of connecting specialized nodes or information sources (Siemens, 2004a). When considering connectivism as a learning theory Siemens (2004) says that a central tenet of most learning theories is that learning occurs inside a person and these theories do not address learning that occurs outside of people such as learning that is stored in, and manipulated by technology. Computer mediated communications or technology supported social networking is a key aspect to connectivism. The point being, that connectivism, is a learning theory born in the Internet age and greatly values technology supported interactions between participants in a learning situation.
It has been said that for deeper learning to occur, learning must be social, active, contextual, engaging and student owned (Carmean & Haefner, 2002). Others have suggested that the learning process is transitioning from cognitive theories based on the individual to theories that stress the importance of the social nature of learning (Dawson, 2006) . Dawson et al (2006) states, numerous authors have espoused the importance for developing social learning opportunities where students actively debate, exchange and clarify ideas with other peers. Irrespective of learning theory, there is little doubt that social interaction plays an important part in learning and for the fully online student in a university course, the LMS discussion forum is the main enabler for social discourse and interaction during their course.
LMS discussion forums.
Like most LMS hosted by universities, discussion forums are frequently used in online courses at CQUniversity, as a means of extending the class conversations beyond the time and place restrictions of the classroom. Other researchers have espoused the benefits of discussion forums, as they are ideal teaching tools for a student population who choose to study via distance because of their life circumstances. They can foster collaborative learning and provide the opportunity for students to interact with each other and the teachers. They engage students in debate and discourse that would normally not be available via distance learning (Hinton & Bradshaw, 2004). Others have said that the use of asynchronous online discussion forums can provide online learning communities with unprecedented learning opportunities (Hew & Cheung, 2003). Oliver (2000) goes on to say that creating collaborative and cooperative settings for learners provides many advantages for the designer and learners. As learners collaborate, they articulate their ideas and thinking and this contributes in large ways to developing their understanding.
An important point to note is that the emergence of community in the educational context has been demonstrated to enhance student learning through the implementation of an overarching pedagogical framework (Dawson, 2006). The LMS makes it very easy for a teacher to include a course discussion board and the “good practice” guidelines at CQUniversity even mandate that each course that is to be delivered by the LMS must include a space for spontaneous student communication (CQUniversity, 2009). Whilst there appears to be some benefit by providing students with such a space, a more valuable use of the technology would be to incorporate the discussion space into the overall course design where the conversation is supporting overall course design strategies.
Teachers and teaching.
An LMS allows the teaching staff to add resources and activities to an online course via a web interface with little or no knowledge of underlying web (html) technologies. Typically, course design and development is the responsibility of the teaching staff and this is an important to consideration when looking at the adoption of features with the Blackboard LMS at CQUniversity. If the teaching staff are responsible for the addition of resources and activities such as discussion forums, then the underlying pedagogy of the course is influenced by their experience with the teaching medium and their conception of teaching and learning. Therefore, LMS are not pedagogically neutral technologies, but rather through their very design, they influence and guide teaching (Coates et al., 2005). Another related point often overlooked by universities when conducting an LMS evaluation prior to implementation is it’s not the provision of features [in an LMS] but their uptake and use that really determines their educational value (Coates et al., 2005) and therefore their value to the university. Additionally, the fundamental measure of student experience with an LMS is the degree to which students use the system (Caruso, 2006).
The Indicators project overview
Academic analytics describes a method for mining and analyzing institutional data, such as the data captured by an LMS, for informing decision making and reporting purposes (Campbell & Oblinger, 2007). It is claimed that the use of academic analytics in higher education has the potential to improve teaching, learning and student success by a combination of awareness of patterns in the data by teaching staff and other predictive modeling techniques. The Indicators project at CQUniversity is aggregating data collected by the Blackboard LMS and the PeopleSoft student administration system. It is proposed that this will reveal patterns in the data that can link LMS features and patterns of behavior with student grades. Note that the Indicators project uses quantitative and automatic methods to extract and aggregate the data and this approach has shortcomings that need to be considered.
There are significant limitations in a purely quantitative analysis of captured data and this is especially true in a complex educational setting. Data mining can help reveal patterns and relationships but does not tell the user the value or significance of these patterns (Seifert, Updated 2004). Additionally, a systems scan of designer and user behavior within an LMS can never describe in full how staff and students are engaging in the use of online environments for teaching and learning (Heathcoate & Dawson, 2005). However it has been suggested that using captured LMS data can aid teachers in decisions about their courses (Campbell & Oblinger, 2007) and others have suggested that there are benefits in exposing the people ‘at the coal face’ to raw data as it allows them to move from an abstract representation of large data sets and spot patterns and anomalies (Snowden, 2008).
CQUniversity is especially complex in that a typical LMS course could be delivered via three different methods, on-campus, off-campus and multi-modal delivery, and delivered to three distinct student cohorts regional campus students, international students and flex students. These different delivery modes and cohorts will influence both the LMS features chosen by the teacher, user behavior within the system and consequently the data captured by the LMS. The project name, the Indicators, is an acknowledgement of the limitations of captured LMS data especially in the complex multi-campus teaching environment like CQUniversity. The data displayed in the following sections has been take from the Indicators database using a variety of technical means such as SQL and Perl. It includes data from over 4700 courses and over 500,000 students over a six-year period.
CQUniversity Flex Students and the LMS.
So far it has been established that there are three fundamental interactions required in a given learning situation, learner-content, learner-learner and learner-instructor, and that effective learning requires a degree of social interaction. Often, the LMS is the single mechanism via which a flex student will access their course and its associated interactions. The literature suggests that discussion forums are an important part of online courses, especially if they are embedded into the underlying pedagogy of the course (Marra, Moore, & Klimczak, 2004). LMS features have an effect on pedagogy and as the teaching staff are responsible for course design and feature adoption, their conceptions of teaching and learning also affect the course’s underlying pedagogy.
CQUniversity has three distinct student cohorts. Rockhampton and regional students are enrolled in courses that are delivered face-to-face at the Rockhampton or other regional campuses such as Mackay, Bundaberg and Gladstone. This paper refers to these students as regional students. Flex students are students who are enrolled in courses that are delivered either partly or wholly online. These students typically only interact with their course and colleagues via the LMS. The third group of students is international students studying at one of the international campuses such as Brisbane, Sydney or Melbourne. These students are not part of this study.
The following sections use student activity data from CQUniversity’s Blackboard LMS as gathered by the Indicators project, to test some of the concepts previously identified from the literature. As the flex students typically only interact via the LMS, the data on their activity is more accurate as it is not as polluted by face-to-face influences when compared to face-to-face students. In order to understand the effects that discussion forums have on flex students, it is important to obtain an overview of the flex student population and its importance to CQUniversity.
Figure 1. Flex students as a percentage of overall student population.
Figure 1 suggests that the proportion of CQUniversity students who are flex students is increasing.
Figure 2. The number of flex student-course units.
While Figure 1 shows that the proportion of flex students to overall student population is increasing, figure 2 shows the the overall flex student numbers is also increasing, albeit slowly. This trend is indicative of the increasing importance of flex students to CQUniversity and therefore the increasing importance of research into how flex students interact with the LMS.
Figure 3. Grade distribution of flex students at CQUniversity.
Figure 3 shows the distribution of grades across the flex student population at CQUniversity. Note that in this paper and for purposes of brevity and clarity, grades refer to a refined version of the official CQUniversity grading system (CQUniversity, 2007) and uses only the most commonly used grades which are High Distinction (HD), Distinction (D), Credit (C), Pass (P), Fail (F).
Figures 1 and 2 showed the increasing importance of flex students to CQUniversity and as a result, the importance of the LMS in delivering the range of interactions required by flex students to meet their learning goals. The following sections address three points of discussion around the use of LMS by flex students. It does this by drawing upon data extracted from the Blackboard LMS at CQUniversity by the Indicators project. While these points are often attributed anecdotally, the Indicators data can now be used to provide empirical evidence to support the following statements.
- Flex students are much more dependent on the LMS than face-to-face students.
- LMS are content focused (Siemens, 2004b).
- LMS discussion boards are particularly important for flex students.
Point 1. Flex students are much more dependent on the LMS than face-to-face students.
Figure 4. Average hits, forum posts and replies for flex students against grade.
These three tables show average flex student hits (red) on the LMS associated with grade and compared to face-to-face (blue) CQUniversity non-flex students. Activity is determined by measuring the amount of hits mouse clicks within a course site. For example the overall average hitcount in Figure 4 shows that a HD student will, on average, make 730 clicks within a course website while a HD non-flex (face-to-face) student makes 250 clicks. Figure 4 shows that on average, flex students access the course sites approximately four times more than a non-flex student and are more likely to post and reply to discussion forums hosted by the LMS than equivalent face-to-face students with the same grade.
This systems scan of flex student behavior within CQUniversity’s Blackboard LMS tends to confirm the hypothesis that flex students are more reliant on the LMS than are students studying via face-to-face modes.
Point 2. LMS are content focused.
As mentioned above, it is not the provision of LMS features, but their uptake that really determines their educational value (Coates et al., 2005). As the teaching staff are responsible for building and maintaining LMS courses at CQUniversity, they have a significant influence on how LMS features will be implemented and therefore used. So whilst feature adoption may at first glance appear outside the scope of this document, it requires mention due to its important influence on LMS feature presence and utilization. While Siemens (2004) suggests that LMS are content focused, it is actually the staff who are responsible for the adoption of features and therefore influence the apparent tendency towards features that facilitate content interaction.
Figure 5. Feature adoption evolution of features supporting content and communication by LMS courses.
In this case, LMS features such as files, announcements and gradebook are regarded as content dissemination features while discussion forums, email and synchronous chat are features that support communication. Figure 5 shows the overall percentage of courses that use features supporting of learner-content interactions compared to features that support leaner-learner and learner-instructor interactions. Of note is the fact that features supporting learner-content interactions are, on average, adopted more than features that support the other interactions and this would tend to add weight to the previous argument by Siemens (2004) who suggests that LMS tend to provide more features that support content sequencing and presentation. The overall trend for both elements is increasing which could be indicative of teaching staff becoming more comfortable with the LMS and using more features or simple more courses having an online presence. However, figure 5 shows only the features that are present in the courses and does not indicate how those features are being used or their pedagogical application within the course.
Most studies into how LMS are being used have tended to focus on specific technical details with little consideration about the pedagogical implications that the LMS usage data indicates (Malikowski, Thompton, & Theis, 2007). Malikowski et al (2007) propose a model that equally considers technical features and research into how people learn. The model looks at LMS feature adoption over time and suggests that the teacher will tend to adopt LMS features in a linear fashion starting with features associated with content transmission, moving to more complex features as their experience with the teaching medium increases (Malikowski et al., 2007). While the detail of this model is outside the scope of this paper and is the subject for further research, it requires mention as teaching staff are responsible for LMS feature adoption and therefore influence the notion that LMS are content focused by their choice of LMS features. The following figure (figure 6) demonstrates the rate of adoption of the various LMS features, over time at CQUniversity and shows the dominance of content dissemination features such as files, announcements and gradebook.
Figure 6. LMS feature adoption for the Blackboard LMS at CQUniversity.
With CQUniversity’s Blackboard LMS there is an apparent bias towards features associated with information dissemination. This may be a concern as it is contrary to research showing that effective learning occurs when students learn with computers rather than from computers (Oliver, 2005).
Point 3. LMS discussion boards are important for flex students in particular.
Figure 7. Flex student usage across 61 million hits.
Figure 7 is showing us that 31% of flex student activity occurs within the discussion forums while the remaining 69% is activity on content such as files, folders and other LMS content tools. Whilst this does not tell us a great deal except that, on average a third of flex student activity within the LMS is spent in the discussion forum, it starts to expose how important the discussion forums are to flex students when compared to non-flex students as figure 8 indicates.
Figure 8. Non-flex student usage across 50 million hits.
The contrast between figures 7 and 8 is significant in that flex students would appear to visit discussion forums at more than twice the rate of their face-to-face equivalents. This is despite anecdotal evidence that suggests that most discussion forums in Blackboard are not necessarily utilized to meet specific pedagogical goals. The following figure is representative of four teachers who are known by the author to often use discussion forums in their courses to meet discrete pedagogical objectives and show a higher than average bias towards forum activity.
Figure 9. Flex student usage for teachers who promote discussion forum participation.
The previous section indicates an increased reliance on discussion forums for flex students. What hasn’t been established is if this reliance has an impact on both student success as indicated by grade, and the students’ perception of community.
The following two figures require some explanation as they use the notion of an average grade and low, medium and high forum use. Average grade is loosely based on a weighted gradepoint average calculation that is an internationally recognized measure for establishing equivalence across different grading systems (Monash University, 2009). The accuracy or validity of the average grade calculation used in the following two tables isn’t very important as it is the trend demonstrated by the data that supplies the reader with a clue to what the data represents. The number of HD grades is multiplied by seven, D grades by six, C grades by five, P grades by four and F grades by 1.5 and the total divided by the number of students in the course. The low, medium and high activity is simply calculated by looking at the total number of forum messages for each courses and grouping the courses by their message quantities. Low is the bottom 33%, medium is the middle 33% and high is the top 33%. Whilst not a statistically valid grade point average, it does illuminate a pattern in the data that indicates a relationship between student grade averages and the level of forum use within courses. The following grade averages were created using the gradepoint average described above for each group of students based on the levels of forum use in their courses.
Figure 10. Indicator of discussion forum usage on grade averages by course.
Figure 11, while appearing similar to figure 10, is different. Figure 10 looked at the grade averages of students in courses with high, medium and low degrees of forum activity. Figure 11 is looking at the grade averages of students where their individual forum activity is high, medium or low.
Figure 11. Indicator of discussion forum usage on grade averages by students.
The results in figure 10 and figure 11 are crucial in determining the value of LMS discussion forums to flex students. Figure 10 shows that, on average, courses that promote and encourage student participation in discussion forums generally demonstrate higher than average grades for flex students than courses that do not. Figure 11 shows that, again on average, flex students who tend to be more active on LMS discussion forums also tend to have higher grade averages that those that do not. This could simply be explained by saying students who are more likely to participate and visit the discussion forums are students who are on the whole, more engaged and motivated. This was checked and confirmed by using the same students as in figure 11 and looking at their overall hitcounts on the LMS that could be used as an indicator of engagement.
Figure 12. Overall hitcounts of students based on discussion forum engagement.
While being potentially indicative, Figure 12 tells us that the link between discussion forum participation and student success is inconclusive in that, while students who participated in courses that encourage forum participation and students who demonstrate a propensity to engage in forums tend to have higher grade averages, the same students are demonstrating a higher degree of engagement across all aspects of the LMS.
LMS have become an almost ubiquitous technology in universities as an eLearning solution. Their dominance in higher education is attributable to an increasing demand for eLearning that is being driven by students requiring more flexible study options from universities. LMS have been described as being too content focused (Siemens, 2004b) and data from CQUniversity showing how LMS features are adopted and utilized adds weight to this argument by indicating that LMS features that support learner-content interactions are adopted more quickly and used the most. Student use of LMS features follows a similar trend with most activity occurring on LMS features that support content dissemination as opposed to other LMS elements that support participant communication and interaction. There is an important social element to learning and this is especially important for flex or distance students whose only source of interaction with their course and fellow learners is via the LMS discussion forum feature. However, it is not the presence of these discussion forums in an LMS, but the way in which they are used that determines their value to the learner.
This study has shown that there appears to be some correlation between student activity on discussion forums and their resulting grades. However, the purely quantitative approach used by this study showed that students with higher than average activity on discussion forums also tended to demonstrate higher levels of activity on the course overall. This prevents an accurate determination into the exact effects of forums on student grades. Despite this, the general trend is that, in courses that encourage student discussion forum participation, students will generally have a higher average grade than courses that do not.
In summary, LMS discussion forums are important to flex students and there are indications that encouraging and facilitating participation leads to improved outcomes for the student. However further research is required to determine effective ways of embedding and facilitating discussion forums within course curriculums as well as identifying other elements of the LMS that impact flex student results.
Beer, C., & Jones, D. (2008). Learning Networks: Harnessing the power of online communities for discipline and lifelong learning. Paper presented at the 2008 Lifelong Learning Conference. from http://hdl.cqu.edu.au/10018/13162
Campbell, J. P., & Oblinger, D. G. (2007). Academic Analytics. Educause Article.
Carmean, C., & Haefner, J. (2002). Transforming Course Management Systems into Effective Learning Environments [Electronic Version]. Educause, November/December 2002, from http://net.educause.edu/ir/library/pdf/ERM0261.pdf
Caruso, J. B. (2006). Measuring Student Experiences with Course Management Systems [Electronic Version]. Educause, 2006, from http://net.educause.edu/ir/library/pdf/ERB0619.pdf
Coates, H., James, R., & Baldwin, G. (2005). A critical examination of the effects of learning management systems on university teaching and learning. Tertiary education and management, 11(2005), 19-36.
CQUniversity. (2007). Grades and Results Policy (version 11). Retrieved 29th September 2009, 2009, from http://policy.cqu.edu.au/Policy/policy.jsp?policyid=437
CQUniversity. (2009). CQUniversity’s Minimum Service Standards. Retrieved 28/9/2009, 2009, from http://lmsip.cqu.edu.au/FCWViewer/getFile.do?id=28098
Dawson, S. (2006). Online forum discussion interactions as an indicator of student community. Australian Journal of Educational Technology, 22(4), 495-510.
Heathcoate, L., & Dawson, S. (2005). Data Mining for Evaluation, Benchmarking and Reflective Practice in a LMS. E-Learn 2005: World conference on E-Learning in corporate, government, healthcare and higher education.
Hew, K. F., & Cheung, W. S. (2003). Models to evaluate online learning communities of asynchronous discussion forums. Australian Journal of Educational Technology, 19(2), 241-259.
Hinton, L., & Bradshaw, J. (2004). Benefits of An Online Discussion List in A Traditional Distance Education Course. Turkish Online Journal of Distance Education – TOJDE, 5(3).
Kim, B. (2006). Social Constructivism. Retrieved 20th September 2009, 2009, from http://projects.coe.uga.edu/epltt/index.php?title=Social_Constructivism
Ladyshewsky, R. K. (2004). Online learning versus face to face learning: What is the difference? Paper presented at the Teaching and Learning Forum.
Malikowski, S., Thompton, M., & Theis, J. (2007). A model for research into course management systems: bridging technology and learning theory. Journal of educational computing research, 36(2)(2007), 24.
Marra, R. M., Moore, J. L., & Klimczak, A. K. (2004). Content analysis of online discussion forums: A comparative analysis of protocols. Educational Technology Research and Development, 52(2), 23-40.
Monash University. (2009). Grade Point Average and Weighted Average Mark. Retrieved 2nd October, 2009, 2009, from http://www.adm.monash.edu.au/service-centre/gpa-wam.html
Moore, M. G. (1989). Three types of interaction. The American Journal of Distance Educaton, 3(2).
Northover, M. (2002). Online discussion boards – friend or foe? Paper presented at the Ascilite 2002. from http://www.ascilite.org.au/conferences/auckland02/proceedings/papers/193.pdf
Oliver, R. (2005). Using blended learning approaches to enhance teaching and learning outcomes in higher education. Paper presented at the International Association of University Presidents’ Teaching Showcase, Joondalup, WA.
Seifert, J. W. (Updated 2004). Data Mining: An Overview. Retrieved. from http://www.fas.org/irp/crs/RL31798.pdf.
Siemens, G. (2004a). Connectivism: A learning theory for the digital age. from http://www.elearnspace.org/Articles/connectivism.htm
Siemens, G. (2004b). The wrong place to start learning. LMS. elearnspace. everything elearning, from http://www.elearnspace.org/Articles/lms.htm
Snowden, D. (2008). The dogmas of the quiet past. Cognitive Edge Retrieved 4/8/2009, 2009, from http://www.cognitive-edge.com/blogs/dave/2008/10/the_dogmas_of_the_quiet_past_1.php
West, R. E., Waddoups, G., & Graham, C. R. (2006). Understanding the experiences of instructors as they adopt a course management system. Educational Technology Research and Development, 55(1), 1-26.
 Hits are mouse clicks within the LMS.
 Hitcount is the total number of hits.