Misc,
Engaging stakeholders in the learning analytics design process
(2020)
Abstract
University of Technology Sydney. Connected Intelligence Centre., Learning Analytics (LA) is a new promising field that is attracting the attention of education providers and a range of stakeholders including teachers, learning designers, academic directors and data scientists. Researchers and practitioners are interested in learning analytics as it can provide insights from student data about learning processes, learners who may need more help, and learners’ behaviours and strategies. However, problems such as low educator satisfaction, steep learning curves, misalignment between the analytics and pedagogical approaches, lack of engagement with learning technologies and other barriers to learning analytics development have already been reported. From a human-centred design perspective, these problems can be explained due to the lack of stakeholders’ involvement in the design of the LA tools. In particular, learners and teachers are commonly not considered as active agents of the LA design process. Including teachers, learners, developers and other stakeholders as collaborators in the 𝘤𝘰-𝘥𝘦𝘴𝘪𝘨𝘯 of LA innovations can bring promising benefits in democratising the LA design process, aligning analytics and pedagogy, and meeting stakeholders’ expectations. Yet, working in collaboration with stakeholders to design LA innovations opens a series of questions that are addressed in this thesis in order to contribute to closing the gap for effective co-design of LA innovations. The questions addressed in this thesis are the following:
1. How can co-design techniques assist in the integration of diverse stakeholders in the LA design process?
2. What are the roles of the co-design practitioner/researcher in the LA design process?
3. What are the challenges in engaging stakeholders in the LA design process?
Based on co-design principles, and following a Design-Based Research process, this thesis explores the critical challenge of engaging educators and students, the non-technical stakeholders who are often neglected, but who should ultimately be the main beneficiaries of LA innovations. In this research work, three case studies have been used to test, analyse and verify various co-design techniques in diverse learning contexts across a university to generate a co-design toolkit and recommendations for other co-design practitioners: i) learners and educators engaged in simulation-based healthcare scenarios, ii) learners, educators and other stakeholders in a Data Science Masters program , and iii) educators interested in providing personalised feedback at scale.
This thesis presents three contributions to knowledge for effectively collaborating with educational stakeholders in the LA co-design process:
1. Inspired by archetypal challenges reported in classic and contemporary co-design literature, and in current LA research, the thesis identifies, exemplifies and reflects on five key challenges for LA co-design: power relationships, surveillance, learning design dependencies, asymmetric teaching/learning expertise, and data literacy.
2. By adopting and adapting well established co-design techniques, across the three case studies, the thesis provides empirical evidence of how these techniques can be used in LA co-design, reflecting on their affordances, and providing guidance on their usage. These detailed findings are distilled into a 𝘓𝘦𝘢𝘳𝘯𝘪𝘯𝘨 𝘈𝘯𝘢𝘭𝘺𝘵𝘪𝘤𝘴 𝘊𝘰-𝘥𝘦𝘴𝘪𝘨𝘯 𝘗𝘭𝘢𝘺𝘣𝘰𝘰𝘬, published under an open license to assist adoption and improvements.
3. Recognising the importance of the co-design practitioner in ensuring that the design process is participatory, the thesis documents and discusses the key functions and skills that this position requires. The role is further complicated when the practitioner is not only a 𝘧𝘢𝘤𝘪𝘭𝘪𝘵𝘢𝘵𝘰𝘳 serving a project, but also a 𝘳𝘦𝘴𝘦𝘢𝘳𝘤𝘩𝘦𝘳 of co-design. This motivates guidelines on the role of the co-design practitioner/researcher when working with stakeholders, and simultaneously studying the LA co-design process, tools and methods.
