Introduction
The research methodology that a researcher cho
Overview of chosen paper
The research paper in question is Accessing online learning material: Quantitative behaviour patterns and their effects on motivation and learning performance (Li & Tsai, 2017). In this study, the researchers looked closely at what types of learning materials students spent the most time accessing through the learning management system (LMS) and the relationship of that data to learning performance and motivation.
Learning management systems are widely used to support teachers in conducting a variety of activities (Li & Tsai, 2017). Over the years, researchers have conducting studies analyzing the LMS system logs to help understand engagement levels, discover behaviour patterns, and explore the relationship between behaviour and learning performance (Li & Tsai, 2017). Li and Tsai’s 2017 paper aimed to further the knowledge surrounding LMS’s by looking at the types of learning materials (lecture slides, video lectures, shared assignments and posted messages) students accessed. They asked four specific research questions:
- What online materials did students spend more time on?
- What kinds of behavioural patterns exist when students viewed online learning materials?
- How did the different behavioural patterns relate to students’ learning performance?
- How did the different behaviour patters relate to students’ learning motivation (Li & Tsai, 2017)?
Li and Tsai (2017) conducted a quantitative study to answer their four research questions, studying a group of students enrolled in a blended course called Mobile Phone Programming. Data about LMS usage was collected via the LMS system logs, while learning performance in the course was measured via multiple homework assignments each week and a final examination. Motivation was measured through the use of the Motivated Strategies for Learning Questionnaire (Li & Tsai, 2017).
What if they used a mixed methods approach?
As evidenced by the methodology outlined above, this study was decidedly quantitative. But what would happen if the researchers had chosen to undertake a mixed methods study instead? How would this impact the four R’s of research (Thom, J., 2019): the research, the researcher, the researched and the reader?.
For Li and Tsai’s 2017 paper, to use a mixed methods research methodology, they would need to add a qualitative component to their study. This qualitative component would likely consist of interviews or focus groups with the students in the class to learn more about their online choices and behaviours, their offline choices and behaviours, and their motivations.
The research
In the current study, using the quantitative research methodology, the researchers were able to come to several conclusions about how students were using the course and how behavioural patterns correlated with learning performance. They were able to: determine which learning material was used the most; identify three different behaviour patterns; link those behaviour patterns to learning performance; and link those behaviour patterns to motivation (Li & Tsai, 2017).
If this study had been conducted using a mixed methods methodology, the above findings would continue to exist, as the quantitative components of the study likely would not change. However, the researchers would be able to identify the why behind the findings and provide a more comprehensive picture of what is happening
Learning the why behind online learning material choices
While the study found that students spent more time on the lecture slides than other learning materials (Li & Tsai, 2017), it provides no insight into why students made that choice. The mixed methods approach would allow the researchers to explore the reasons why students made the choices they did when engaging with online learning material. While it is possible that students spent the most time on the content they found most useful, it is also possible that some content simply took less time to extract value from (this seems possible particularly in the case of posted messages) or that the LMS was set-up to encourage the use of one learning material over the others (i.e. which learning material could be found at the top of the page vs. the bottom of the page?).
Learning the why behind the behaviour patterns
Similar to the argument above, while the quantitative data showed that there were three different behaviour patterns in terms of LMS usage (Li & Tsai, 2017), there is no data to be able to explain why these different behavioural patterns existed. Why did “less use” students access the LMS less frequently? Were they simply unmotivated and uninterested, as the study seems to suggest, or did outside factors, such as work commitments, overall course load or family concerns prevent them from engaging as frequently? Or perhaps, these “less use” students had confidence in their ability to master the material so did not feel the need to access the materials as often.
Validate the motivational finding
One of the strengths of mixed methods research is that it is possible to validate some of the study findings within the study itself (O’Cathain, 2010). Using mixed methods, researchers could have validated the connections between motivation and behavourial patterns. While correlations were found between motivation levels and frequency and duration of use, connecting qualitative data about LMS usage and choices with the motivational data could strengthen the correlational finding.
Explore how students are using the material
In the paper, the researchers state: “it cannot be determined if a student is actively and effectively learning from a material once they have accessed it” (Li & Tsai, 2017). The researchers have no idea what students are doing with the learning material. Are they opening the slides and staring at them, or are they studying them using effective study methods? Are they simply reading through the shared assignments, or are they comparing them to their own assignments? Answers to these questions could provide additional value in connecting accessing to learning performance.
The researcher
The research methodology chosen can have an impact on the researcher, and conversely, the research often influences the research methodology chosen. This study was carried out by two researchers at two different universities in Taiwan. Liang-Yi Li is a professor in the Department of Computer Science and Engineering at National Central University, working in two fields that rely heavily on numbers and data (“ResearchGate: Liang-Yi Li”, n.d.). Chin-Chuang Tsai is a professor and chair with the Program of Learning Sciences at National Taiwan Normal University, and has an educational background in both science and education (“Chin-Chung Tsai Curriculum Vitae,” n.d.). Both researchers appear to focus primarily on quantitative research methods. If they were to adopt a mixed methods approach for this study, it is likely they would want to bring in additional collaborators; folks who have expertise in mixed methods research and could assist with the delivery of interviews or focus groups and the analysis of the resulting data. The additional collaborator(s) would also help in distributing the labour, as the mixed methods approach which require additional time and monetary resources.
The researched
Switching to a mixed methods approach would not change who is being researched; the study would continue to focus on the students enrolled in Mobile Phone Programming. However, the impact on the research participants would change. Currently, the participants don’t have to devote any time or effort to the research study beyond what they would already be investing in the coursework. Using a mixed methods approach would require students to dedicate additional time to participating in interviews or focus groups.
The reader
The lack of rationale in the current study makes it difficult for the reader to understand the value of the research and be able to apply it in useful ways. Since the data as presented do not show the full picture, it is difficult to use in determining action, such as what materials to develop for a course, how to set up a course, how to motivate students or how to increase student learning performance. As the mixed methods approach would show a fuller picture, the reader would be better able to determine possible action.
Conclusion
Examining a quantitative research study through the lens of a mixed methods approach highlights the advantages and challenges of the mixed methods approach. The mixed methods approach helps to fill the gap left behind by the major weakness of quantitative research: a limitation in the understanding of context (“Overview of Mixed Methods,” n.d.). This leads to a more comprehensive answer to the larger research question. However, this approach would also require more resources in terms of time, people and money. As evidenced by this paper, the research methodology chosen can impact all 4 R’s: the research, the researcher, the researched and the reader, and these four aspects should all be carefully considered when choosing a research methodology.
References
Chin-Chung Tsai Curriculum Vitae. (n.d.). Retrieved July 10, 2019, from http://www.cctsai.net/eng/curriculum_vita.html
Li, L.-Y., & Tsai, C.-C. (2017). Accessing online learning material: Quantitative behavior patterns and their effects on motivation and learning performance. Computers & Education, 114, 286–297. https://doi.org/10.1016/j.compedu.2017.07.007
O’Cathain, A. (2010). Assessing the Quality of Mixed Methods Research: Toward a Comprehensive Framework. In A. Tashakkori & C. Teddlie, SAGE Handbook of Mixed Methods in Social & Behavioral Research (pp. 531–556). https://doi.org/10.4135/9781506335193.n21
Overview of Mixed Methods. (n.d.). Retrieved July 10, 2019, from Centre for Innovation in Research and Teaching website: https://cirt.gcu.edu/research/developmentresources/research_ready/mixed_methods/overview
ResearchGate: Liang-Li Yi. (n.d.). Retrieved July 10,
Thom J., (2019). EDCI 515: E-Research: Harnessing and Understanding Technology in Research, July 4 lecture notes. [Course lecture].
Featured image by João Silas on Unsplash
