I recently attended a workshop facilitated by SICSA entitled Learning analytics for improving evidence-based teaching. There were around 20 delegates in all, representing several universities across Scotland. Pavlos Andreadis was also representing the University of Edinburgh. Discussion sessions were led by Kassim Terzic.

The invited guest speaker was Andrew Cormack, Chief Regulatory Adviser at Jisc Technologies, who gave an informative talk on Learning Improvement, Ethics and Law. You can read more about Andrew’s thoughts on GDPR and education on his blog.

For most of the day, we formed smaller groups to discuss various aspects of learning analytics data points at our respective institutions. Summaries of our discussions can be found below.

What learning-related data is currently gathered and stored in your institution? 

– Grades (exam, coursework)

– Entry Requirements

– NSS evaluation (likert scale + free text)

– Engagement

– Attendance (eg Glasgow Caledonian require swipe access to lecture rooms)

– Lecture capture

– Forums

– Reading lists (different access methods)

St Andrews also conducts exit type interviews with its students, measuring longer term feedback. Pavlos also noted that feedback is also given in the form of complaints, either to the student union or director of studies (or both). This kind of feedback is less likely to be logged in a central system.

How is impact of changes on learning currently measured in your institution?

– Scatter plots (modules vs all)

– Average / standard deviation (final grade / exam only / per exam question)

– Student evaluation (module boards)

– Trends per student

– Averages / histograms over years

– Student background (how do students with a programming background compare with those who don’t re final classification)

– Selection of exam question.

How can statistical and data-focussed approaches help evidence learning outcomes?

Consistent data points across all courses and all institutions:

– Learning outcomes

– Delivery method

– Assessment method

A proper experiment in this area would require an active intervention and a control group. This, however, would be ethically difficult. It was suggested we look instead at ‘passive’ interventions. For example, could we use text analysis of discussion boards to demonstrate understanding of key concepts in the lecture? Or look backwards for evidence of a student request for change and if the resulting change had the desired effect?

We also discussed how machine learning could be useful in predicting students who are ‘at risk’. A word of caution here though:  the University of Edinburgh has a Learning Analytics Policy and has developed seven principles to sit alongside this policy. One of these principles states:

“Our vision is that learning analytics can benefit all students in reaching their full academic potential. While we recognise that some of the insights from learning analytics may be directed more at some students than others, we do not propose a deficit model targeted only at supporting students at risk of failure.”

What data-driven approaches could be applied in a study across Scottish Universities?

We then discussed how we could evaluate a passive intervention across multiple HE institutions. One suggestion was to find a course common to all Computer Science programmes (eg a second year Database course) which has remained relatively consistent across 7-10 years, to use in the data study.  We could introduce a guest lecture to this course, which could be delivered remotely. We could then measure:

– Engagement

– Satisfaction

– Learning (grades)

and compare these data points to previous cohorts.

A personal note

Without a clear understanding of what ‘gap’ this guest lecture was aiming to fill, I remain unsure as to what such an experiment would tell us. I preferred (my own) suggestion of introducing more (and different) types of assessment into a stable, common course across programmes. How we assess, how often we assess, why we assess and the feedback loop is ripe material for a lot of research into education just now, and it is my personal opinion that we could contribute to this research in a meaningful way.

Finally, a cautionary tale: on 29 May I attended a talk given by Joel Smith from Carnegie Mellon University entitled: The Eye of the Needle: New Understandings of the Complex Barriers to Instructional Innovation with Technology. In this talk, Professor Smith talked about the twin challenges of any TEL project:

– effectiveness in improving learning outcomes

– effectiveness in terms of adoption and sustainability (ie the implementation of innovations).

Demonstrating the former does not necessarily lead to the latter. We should always keep this in mind when looking to evidence better teaching.

I’m pleased to say Informatics now has a ‘mobile capture kit’ which can be booked out by colleagues. Perhaps you want to record a lecture in a space which currently doesn’t have the hardware installed? Or a research seminar? Or a video answering a common question posed by your students?

You may already have some equipment which can support you in this. However, the Learning Technology service has packaged this up together for ease of use to support you in your teaching. It consists of the following:

– Windows laptop (HP Elitebook G3)

– USB Document Camera (HoverCam Solo 8Plus)

– Webcam (Logitech C920)

– USB mic (Snowball)

– Graphics tablet and pen (Wacom intuos)

– Tripod.

The laptop has the Echo360 Classroom Capture client installed which supports recording your screen, audio (via the Snowball mic) + video (via the Logitech webcam or HoverCam document camera).

It is worth noting that the mobile capture kit could also act as a mobile solution for video conferencing. Conferencing could be managed by a web browser (such as Blackboard Collaborate) or via an application such as Skype for Business.

If you’d like to borrow the kit, please get in touch with the learning technology service.

The Learning, Teaching and Web directorate within Information Services is starting a project that aims to deliver an Academic Blogging Service for the University. The service will support research and teaching and provide students and staff with a user-friendly blogging environment for sharing knowledge, reflecting on learning, and representing individual and group identity online.

They are currently in the user consultation phase. If you missed the workshops, you can still contribute your thoughts and ideas via the wiki.

Links are:

https://www.wiki.ed.ac.uk/display/ABS/Academic+Blogging+Service

https://www.wiki.ed.ac.uk/display/ABS/User+Consultation

For what it’s worth, my main suggestion was to remove the word ‘academic’ from the service.

Background

Edinburgh College of Art (ECA) frequently assesses to multiple learning outcomes (LOs). This means that for each assignment, a student will receive multiple grades and sections of feedback. In the past, ECA used a custom VLE (the Portal) to support this kind of assessment. Tutors would select grades for each learning outcome which was being assessed (often three) and feedback for each learning outcome. Students would then need to leave a piece of self-evaluation and ‘submit’ in order to access their tutor’s feedback.  As we migrated from this legacy VLE (the Portal) to the centrally supported VLE (Blackboard Learn) I went about trying to identify how best we could leverage the tools of the new system for a workflow which was familiar, and favoured, by teaching staff and students at ECA.

The challenge was as follows:

– Allow markers to leave multiple grades and feedback for each assignment

– Allow support staff to download these grades for upload to the central assessment and progression software with minimum intervention (and risk of mistakes)

– Allow students to receive their multiple grades and feedback with as few clicks as possible, and represented in a clear and consistent way.

I identified the rubric tool within Blackboard as the most useful tool we could use to replicate this workflow. While this addressed point 1, it did not address points 2 and 3:

– as it stood, administrators could only download the aggregated grade from the Blackboard Learn Grade Centre.

– students would have several clicks (many of which would not be intuitive) in order to access their feedback. Because of the complexity of this task, and the multitude of ways a student could give up half way through, communicating grades and feedback (essential to the learning process) would be severely compromised.

I proposed that ECA use some if its Information Services (IS) Apps development budget for investigating ways of addressing these issues. I met with a project manager from IS and we blocked out the best part of a day for exploring what the current challenges were, what the risks were for continuing with the current system, and the opportunities for developing something which could improve the experience.

I then took these initial findings to the development team in IS at a subsequent meeting. After ruling out some options, we proposed building a new building block for Blackboard which would provide a different view into the Grade Centre. This would have to be accessed via a new tool, rather than a different stylesheet for the existing tool (My Grades). This would introduce a potential cause for confusion, but I balanced this against the potential confusion from the existing workflow and argued the benefits would outweigh the challenges.

The project then moved into the development phase. I was tasked with testing the Beta tool. The new tool went live for ECA staff and students in September 2016. After a year of successful deployment, the tool was made available to the rest of the University in September 2017.

Project documentation.

My name is Alex Burford and I’m the new learning technologist for the School. If you’re not sure what that means, my job is “to support the development and delivery of high quality and innovative teaching materials and methods for both on-site and distance-learning students”.

I hope to develop this site into a useful support tool for colleagues as and when you have a questions about learning technology. Initially, the site will focus on centrally supported tools but as I get to know the School better, this may well widen out into more bespoke tools. Similarly, if you’d like this site to cover a particular product or service, please don’t hesitate to get in touch.

I shall also be offering regular informal drop-in sessions and some more targeted training events. Please keep an eye on the Training page for further information on this. Meantime, if you think I may be able to help you in your teaching, please don’t hesitate to get in touch.