The fifth eAssessment Scotland conference was held this year on 23rd August at the University of Dundee. Billed as “the UK’s largest conference dedicated to exploring the best examples of eAssessment in the world today”, this year’s theme focused on the potential role of social and participatory technologies in teaching and assessment.

First off, this is a very well attended event, with representatives from all sectors of Scottish education. It is also very well organised, with some 45 speakers involved in the single day conference and a week’s worth of online events running alongside it. To put the value of the event in perspective, it is free to all participants, fully funded by sponsors, the headline sponsor being QuestionMark. Three of our team attended this year – Stuart Nicol, Robert Chmielewski and Mark Wetton – so we were able to cover a good spread of the workshops and seminars which ran concurrently in the morning and afternoon between the three keynotes. See the full programme for the day event here.

Keynote 1: Assessment in Open Spaces

Catherine Cronin, The National University of Ireland (Abstract | Slides | Twitter: @catherinecronin)

Unfortunately for Catherine, who was delivering the opening keynote in the main lecture theatre, the start of her presentation was badly hampered by audio problems – some bizarre gremlin sitting in the cable between what looked like a well-travelled MacBook and the theatre’s AV system. But as a colleague commented, glitches like this tend to make presentations stick in your mind – and Catherine’s ability to work through the tech issues whilst still putting across a powerful and engaging keynote is testament to her skills as a presenter.

Catherine began in her exploration of open education by drawing on the book Networked by Rainie and Wellman which suggests that the influence of social networks, mobile devices and the internet have given rise to the notion of “networked individualism” – individuals (be it teacher or student) existing outwith their traditional ‘local’ groups. As Joi Ito of MIT puts it “I don’t think education is about centralised instruction anymore; rather, it is the process [of] establishing oneself as a node in a broad network of distributed creativity” (from Catherine’s blog on “distributed creativity”).

Networked educators / networked students

This networked space is characterised as the third space for learning: where the physical classroom is the first; and the ‘bounded’ online space (e.g. the VLE) is the second. Dannah Boyd talks of social network sites as “networked publics”. These are spaces constructed by networked technologies; where “imagined collectives” emerge from the interaction of people, technologies and practice. Catherine argues that this open space is often underused in education: why don’t we make more use of the affordances of available technologies (e.g. mobile devices) and networks (e.g. social media) in teaching and learning? This open online space enables a level of identity play which stands in contrast to those imposed by VLEs like Blackboard. See Bonnie Stewart’s work on digital identities. Whole new types of interaction could be opened up between student and teacher.

Clearly, the use of social media in education, and especially assessment, throws up questions about safety and ethics. Catherine references Howard Rheingold’s notion of networked literacies. This is about showing students how to become responsible and safe “digital citizens”. Given that one of the stated aims of the school education system is to nurture citizenship, surely ‘digital citizenship’ – responsibility to yourself and others – is a part of that.

So, what does this mean in terms of assessment in open spaces? Catherine draws on the work of Jonathan Kozol (1992) which emphasises the importance of ‘student voice’, along with a move to bring ‘joy’ back to education: meaning, enjoyment and connection. She provides some examples for various levels of education:

• In the primary school context she showed us “Ms O’Keefe’s Class”. An experiment in bringing student voice to the fore by means of open, student-owned learning blogs from a fifth year primary class in Galway, Ireland. Each student has their own blog space as part of the class to blog about activities of interest. They follow and engage with other primary level class blogs from around the world.
• Chalfonts Community College Media Studies Blog provides a secondary school example. An open set of blogs for teachers and students to interact and discuss media studies, as well as to develop and present coursework.
• Catherine’s own 2nd year Professional Skills course at NUI Galway provides a HE example. Again a blog is used as the main platform for feeding discussion and coursework, but with an emphasis on building a network of social peers, primarily via Twitter. The aim is that students should leave University with a “personal learning network” – an essential personal/professional learning support network to take forward into further education or the workplace.

The concept of “personal learning networks”, and its relation to employability, is very interesting – how important is evidence of social network building skills to employers (it’s surely key to effective learning in the workplace)? How could an individual demonstrate this? Is evidence of network building a key part of a student’s wider portfolio of achievement in HE? And therefore is it something schools should be building the foundation for (that’s exactly what Ms O’Keefe seems to be doing in her primary school classroom)?

To what extent should students be demonstrating meaningful and productive networking, i.e. moving beyond the gathering of ephemeral contacts to tangible collaboration and reflective assessment of the productivity of such collaborations?

Catherine highlighted that learning should be ‘life-long’ and hence the importance of keeping networks open post graduation. She wound up her talk by suggesting four contributions to the dialogue around assessment in open spaces:

• connect & build networks, because learning itself is social and connected;
• enable student voice and choice;
• share;
• and choose open.

Keynote 2: Structured Chaos, Learning Webs – and Assessment

Helen Keegan, University of Salford (Abstract | Twitter: @heloukee)

Helen’s keynote overlapped nicely with Catherine Cronin’s in discussing how students and teachers can make use of open learning spaces and personal learning networks – making effective connections both within and outwith their institution. She also touched on the convergence of technologies and practice, and highlighted Rheingold’s Net Smart approach to digital literacies and digital citizenship. Helen mentioned various projects with which she has been involved where technologies and practice converge: iCollab; ELVSS12 (Entertainment Lab for the Very Small Screen) – shared, cross-institutional film and video making using mobile phones; crowd sourcing lecture notes – much like blogging/tweeting from a conference but from a lecture, and then using Storify to create a complete, shared set of lecture notes (assessment for learning); crowd sourced bibliographies, where each student is asked to find a paper and blog to the others about it, given a basic bibliography as a starting point – the merging learning and assessment.

However, Helen’s set piece in this keynote was really the story of how she embedded an Augmented Reality Game (ARG) into a module she was teaching. The aim was twofold: firstly ARGs were part of the curriculum, so she aimed to teach it by living it; but more importantly it was to help students identify the ‘hidden curriculum’ by engendering an environment of mystery, curiosity and intrigue. By creating a story based around the made-up character ‘Rufi Franzen’ she let the mystery unfold gradually –from the ‘rabbit hole’ of seemingly random numbers mailed to random members of the module group. The students were never explicitly told that this was part of the course, or even that she was behind it. They began to learn and problem-solve in a social and active manner, collaborating and working in teams, pulling them deeper into Helen’s ‘cloaked curriculum’ than they would normally venture. To this end the experiment seems to have been successful with students beginning to ‘drive’ her curriculum, but not without a cost. Issues of ethics and power relations emerged as students began to take the ‘game’ too seriously and become emotionally involved: some becoming scared; others even reacting aggressively.

This was such an engaging example of going beyond assessment, driving the genuine curiosity of learners – and it seems to have caused quite a stir amongst the students at Salford. However, one might question how scalable such an approach can be when applied in wider contexts. The ‘story’ probably couldn’t be repeated year-on-year as one cohort tells the next about it – and with extensive ‘setup’ time cost it doesn’t seem feasible to create a new plot each year. Would such an approach pass the ethics committees it required to become more than just an experiment in learning and assessment? These are big issues, and big questions, to answer – but given the reaction and engagement shown by Helen’s students they seem to be worth pursuing. To find out more read Helen’s blog post about running a module as an ARG.

Keynote 3: Anyone Need a Can Opener?

Fiona Leteney, BUPA (Abstract)

Our initial expectation was that this keynote would be least interesting – partly because of the BUPA connection and the commercial learning provider connotation. However, this was absolutely not the case. The session turned out to be an extremely insightful introduction to the Tin Can API, from the perspective of a learning provider supporting 18 to 26 thousand users worldwide. BUPA Learn is the platform used – an installation of Blackboard Learn (which is Tin Can compliant). The aim of Tin Can is to liberate learning content from the LMS – it’s the descendant of SCORM, a standard which aimed to keep content platform independent. But Tin Can represents a complete rethink in approach: the focus is not on the content, but rather on activity; and the implementation is as an API that enables a range of compliant technologies to speak to a student’s Learning Record Store (LRS). Using this model, content can ‘live’ anywhere, enabling learning and assessment to be transferable.

The LRS is at the core of Tin Can and it’s the place where a learner’s activities are stored. Having this in Blackboard would allow student activity – for example assessment exercises – to be tracked by the institution. But where this becomes more interesting is where it’s mixed with the idea of an individual’s “Personal Data Locker” – a student-owned store of their own activity. If there was a way of importing/exporting between the personal store and the institutional store then a student could capture both formal, and informal, learning events. And here’s where the concept overlaps with portfolios and the ability to capture, on an ongoing basis, your own ‘learning events’.

Clearly, another overlap is with Mozilla Badges. These technologies have been developed independently, with no foresight to work together – but the LRS does sound a heck of a lot like a backpack, and badges sound like a way of evidencing completion of activities. This post on “Tin Badges” discusses the potential cross-over in more detail. BUPA are investing heavily in assessing the impact of learning on performance, data structures and analysis, which also closely relate to our interest in Learning and Educational Analytics. These technologies, and approaches, seem like ones to keep an eye on.

Seminar 1: Inclusive Assessment: From aspiration to daily practice

Alistair McNaught (JISC TechDis)

Alistair highlighted the importance of this topic by giving the context that 10.3 % of exam candidates in Scotland have accessibility arrangements, and the legal compliance requirement. Despite this, the need for supportive readers or scribes in assessment has fallen in recent years due to increasing use of technology and digital question papers.

Alistair took a practical perspective, explaining that as you scale up either the audience size, the stakes involved, or longevity of engagement, then accessibility becomes more significant. His advice was to simply “keep it real” suggesting some simple benchmarks to early assessment of mode accessibility:

• Can I change colours
• Can I change fonts
• Can I use it without a mouse
• Will it work with speech technologies
• Will it work with a screen reader

Alistair showed a flexible and accessible assessment tool that has been developed by the University of Nottingham, called “Xerte” which provides an online assessment toolkit:

The software itself is a freely hosted service, although local copies can be downloaded for trial. It runs over the web using html5, and its main features are:

• Accessibility (tested via JAWS, NVDA and SuperNova)
• Mobile device friendly
• Enables feedback on answers
• Drag and drop image based quizzes
  • E.g. Labeling an image
  • Select a zone on an image
  • Works well across browsers

JISC’s TechDis unit provides free voices at for screen readers, even providing Scottish accents, via ‘Heather and Stuart’.

Alistair went on to describe the benefits of e-assessment such as the ability to test on demand and avoid some seasonable fluctuations in specific disabilities, but explained that the rapid feedback and increasing variety of digital modes of assessment benefitted all. He concluded the seminar with 4 principles of accessible e-assessment:

1. Anticipate disability
2. Make reasonable accommodation for the context, starting with what can be done easily
3. Recognise new technologies e.g. iPads that may open opportunities
4. Understand corporate responsibility and Equality legislative requirements
5. User testing is critical

The Xerte tool looked like a really flexible yet easy to navigate tool, and one that we should explore further.

Seminar 2: Video Assessment of Psychomotor Skills in Online Learning: Professional Fitness Practice

Lon Kilgore, Anita Neilson, Ann Kinchloe-Skinner & Kim MacIntyre, University of the West of Scotland and Elise Brown, Early College High School at Brookhaven College

How can the teaching of psychomotor skills – the ability to effectively teach motor skills – be objectively assessed in a fully online course? Lon Kilgore’s team have worked with CAPDM to build an add-on to Moodle to facilitate the upload and marking of videos demonstrating physical education students teaching various motor skills. Version 1 of the plugin relies on the YouTube API for video upload and playback, and a PDF download as marking rubric. Version 2 will integrate a closed video platform and fully online marking workflow. One question that came up a couple of times is how to ensure students have the hardware and knowledge to effectively film themselves and process the video file for upload. The team have put together a simple, single page guide, and students (so far) have been able to achieve this without much bother. I guess this relates back to Helen Keegan’s ELVSS project, and student’s ability to produce good quality video from ready-to-hand technologies like mobile phones/devices.

Seminar 3: Can your assessment platform do this – using QTI

Mary McVey Glasgow University

Mary presented the Uniqurate editor, an assessment editing tool that has been developed by Kingston University. The tool enables drag and drop question construction and looks particularly suited for math based assessments. The tool itself is free, and outputs QTI (Test interoperability Standard), XML or IMS Content Package.

Mary described key features of the tool in a positive manner:

• Handles multiple inputs, graphics
• Feedback can result from a combination of multiple student inputs
• Expert mode editor enables XML editing of then creation of assessments, with a rich graphic interface
• Questions can be randomised
• Works within Blackboard, Moodle and Sakai, so no risk of loss of investment if there is a VLE change.
• There is a QTI support website, with a link for converting Blackboard questions into QTI
• Students can work in groups to designed Multiple Choice Questions, such as in PeerWise that can then be imported into Uniqurate
• Electronic marking of questions using Intelligent Character Recognition (ICR)
• Integral Rubrics
• Latex and mathML compliant

Seminar 4: Beyond Me-portfolios: Embedding Personal Learning Tools in the Assessable Curriculum

Shane Sutherland, Pebble Learning

When opening his session, Shane introduced himself as ‘evil vendor’ (Shane’s one of the directors of PebbleLearning, the company behind the PebblePad e-portfolio system). Yet, despite those close links between the topic of his presentation and the product he promotes, Shane managed not to endorse PebblePad too directly when detailing his fresh e-portfolio ideas.

The freshness of Shane’s approach begins with dividing learner e-portfolio systems into:

• me-portfolios (focused on the qualities of the owner);
• and t-portfolios (or task-portfolios; aiming at recording specific experiences or tasks situated on a timeline).

According to Shane, there is a large volume of evidence which suggests that many educational e-portfolio implementations are pursuing mainly the me-portfolio route. And yet, today’s technology allows for much segmented e-portfolio customisation, where complex templates are timely introduced to allow learners to collect artefacts more coherently. For those who used PebblePad3, the idea of designing such t-portfolios, might draw parallels with the PebblePad’s ‘Workbook builder’ tool.

In Shane’s vision, t-portfolios are presented as multi-chapter collections, owned by the learner and combining both: the learning content and the personalised online forms. Although the forms are completed and owned by the learner, it is the instructor/designer who retains the ultimate control over the structure of the t-portfolio. Depending on the nature of narrated experiences, t-portfolios can be sub-divided into sub-categories: Process, Project, Placement, Production and Pedagogical.

Closing Comments

This is just a modest sample of what was presented at the event. Between the sessions, the refreshing scent of hot coffee and the noise of involved conversations made the conference area buzzing with activity. Those who preferred solitude would certainly not find it in the poster corner or the exhibitor’s hall, both of which proved to be popular with lunching participants.

Summing up, eAssessment Scotland is an event that attracts consistently good interest and is well organised. Both JISC and HEA are actively involved, and the event itself provides good networking and sharing opportunities. We would certainly recommend attendance whether you are new to the subject or an old hand, as there is a broad agenda to suit all needs. I expect we’ll be there in force again next year.

eAssessment Scotland 2013 / Educational Design and Engagement by is licensed under a Creative Commons Attribution CC BY 3.0

Share