The Electronic Management of Assessment (EMA) project is a Jisc project which is working with partners (the Heads of e-Learning Forum (HeLF) and the Universities and Colleges Information Systems Association (UCISA) to support institutions with the electronic management of assessment. The term EMA is increasingly being used to describe the way that technology can be used to support the management of the assessment and feedback lifecycle, including the electronic submission of assignments, e-marking and e-feedback. This project seeks to maximise the benefits technology can offer.
The recording of our webinar ‘Online exams: migration or transformation’ run jointly between Jisc and EUNIS http://www.eunis.org/ is now available.
Listen to the recording on our YouTube site https://www.youtube.com/watch?v=MjMNanbMpMQ to hear the views of:
- Stuart Allan, who recently completed an MSc on the subject at the University of Edinburgh
- David Parmentier, Sogn og Fjordane University College, Norway
- Annette Peet, SURF , Netherlands
- Martyn Roads, consultant specialising in assessment in the UK FE and skills sector
You can find out more about the background to the webinar in this blog post https://ema.jiscinvolve.org/wp/2016/08/01/online-exams-migration-or-transformation/
The session was held using Blackboard Collaborate and the recording is made available on YouTube to maximise its accessibility. This means you don’t have access to the chat that was taking place so here I have jotted down a few comments and observations from the chat as well as links to some of the key resources suggested by the presenters.
If you would like to access the full version recorded in Blackboard Collaborate you can find it here: https://ca-sas.bbcollab.com/site/external/recording/playback/link/meeting.jnlp?suid=M.12D0DC0F57D3390C63CFE5388C03F8&sid=2009077
We were using the term ‘online’ in a broad sense: really this was all about summative testing in digital format not necessarily implying that there needs to be full Internet access at all times.
‘Pedagogic’ reasons for moving to digital exams include enhancing the curriculum and better preparing students for employment.
People are often worried about the cost of migrating to digital without having any real idea of costs/benchmarks for existing paper processes.
Interesting point about how students think when writing vs typing. Mogey and Fluck (2015) found that in computer-based exams students are sometimes more concerned with maximising the number of words they can include in their response than with the construction of academic arguments. Mogey N. and Fluck A. (2015) ‘Factors inﬂuencing student preference when comparing handwriting and typing for essay style examinations’, British Journal of Educational Technology, 46 (4), pp. 793–802
In relation to collaboration on item banks – technical incompatibility between systems seems to be an issue. Manchester University’s UKCDR study looked at this – medical education boards share successfully across institutions. If the on-screen exam/ item building system is QTI compliant, that helps with migration of items and exam templates. https://www.imsglobal.org/question/index.html There was also a comment that the education sector needs to look beyond QTI and more at API based content integration.
UK-based awarding bodies and commercial education companies in professional education outside of Ofqual’s regulation have deployed onboard remote invigilation. Service support and cultural acceptability (especially if the remote invigilation provider is non-UK based) are proving the main barriers to adoption.
‘Surely if students are taught in the same method of delivery as online assessment then they would be more in tune.’
‘The digital literacy of staff is something we are also experiencing as a challenge. The students are way ahead of anyone, so not a problem. Assessors seem to be the least prepared for an entirely digital exam.’
English government policy has retrenched to terminal summative assessment. Scottish government keen to use digital evidence in qualifications. http://www.gov.scot/Resource/0050/00505855.pdf
Resources from the presenters
You can find Stuarts blog at: https://stuartallanblog.wordpress.com/dissertation-abstract/
Here’s link to Myyry’s and Joutsenvirta’s publication as recommended by Stuart: http://alh.sagepub.com/content/early/2015/03/24/1469787415574053.full.pdf
White Paper on online proctoring (in English)
Innovation in digital assessment: assessment themed issue (April 2016 in English)
Assessment Security Selection Model (in English)
Transforming assessment and feedback with technology guide: http://ji.sc/transforming-assessment-feedback-guide
EMA processes and system guide: http://ji.sc/ema-processes-systems-guide
Supplier responses to system requirements: http://ji.sc/supplier-responses-ema
2012 UK Landscape report: bit.ly/jisc-ema
Resources suggested by participants
Surpass Paper+ assessment system as a means from migrating from paper to digital exams https://www.youtube.com/watch?v=ENAmkPWVN6k&feature=em-subs_digest
Case study on Saxion University, Netherlands (who used Surpass) and their implementation of e-assessment to deliver over 50k exams last year http://www.btl.com/case-studies/
GeoGebra for digitising mathematical equations, drawings etc https://www.geogebra.org/
Unidoodle for allowing students to submit sketch style answers http://www.unidoodle.com/
Jisc guide to digitising learning and teaching materials sustainably https://www.jisc.ac.uk/guides/digitising-your-collections-sustainably
Resources from the e-assessment in mathematical sciences conference at Newcastle University, September 2016 http://eams.ncl.ac.uk/
KioWare used in the financial services education field for browser lockdown http://www.kioware.com/kwb.aspx
Systems produced by Inspera and Uniwise were mentioned and further information about these can be found in the Jisc overview of systems supporting electronic management of assessment https://ema.jiscinvolve.org/wp/2016/02/24/supplier-responses-to-uk-he-ema-system-requirements/
Case studies on the e-assessment association website http://www.e-assessment.com/
It’s a couple of years now since we had a flurry of activity around people sharing and discussing product comparisons (search on product features/comparisons if you want to look at the archives).
Since then we have published a set of supplier responses to the sector-wide specification (spring 2016) and this was updated in September 2016. As many Jisc and other resources link to the original blog post on this topic we have decided it is logical to keep adding updates at the bottom of that page. To see the original and updated supplier responses go to https://ema.jiscinvolve.org/wp/2016/02/24/supplier-responses-to-uk-he-ema-system-requirements/
As the products are continuing to evolve, your views on how well various combinations meet your needs are still of interest to others so please keep sharing. For those using Moodle and Turnitin the comparison on the UCL wiki is regularly updated https://wiki.ucl.ac.uk/display/MoodleResourceCentre/Comparing+Moodle+Assignment+and+Turnitin
Following our guest blog post where Stuart Allan talked about his research into online exams we are running a webinar on the topic in collaboration with EUNIS (European University Information Systems organisation).
The webinar will take place on Wednesday 21st September at 12.00 UK time. Book your place here: https://www.jisc.ac.uk/events/online-exams-migration-or-transformation-21-sep-2016
Following pilot testing by 12 institutions we are offering a wider range of people the opportunity to try out our self-assessment tool. The tool is still in beta version so we know there will be functionality you would like to see added. The tool has so far been tested with universities so we are particularly interested in feedback from the FE and Skills sector as to whether adaptations are necessary to meet your needs.
The tool can be found at: http://ji.sc/emaready Please read these guidance notes before using it.
You can give feedback by using the comment functionality on this blog or by contacting firstname.lastname@example.org
About the tool
The self-assessment is designed to support institution-wide development of electronic management of assessment (EMA). It assumes that any college or university will benefit from a strategic approach to managing assessment and feedback (and hence EMA). Such an approach requires a good overview of practice and a clear set of policies, regardless of the extent to which institutional policy leans towards promoting conformity or supporting diversity of practice.
At the end of the assessment you will be able to able to estimate your current maturity and develop an action plan. Our accompanying guide, Transforming assessment and feedback with technology provides ideas and resources to help you enhance the entire assessment and feedback lifecycle.
There are five stages:
|Researching||You are at an early stage of EMA. You do not seem to have a comprehensive view of organisational activity overall; policy, process and systems seem fragmented. Ensure you have senior management support to undertake further investigation. Start by defining the principles that underpin assessment and feedback in your organisation and find the areas of good practice you can build on.|
|Exploring||You are probably aware of pockets of good practice but have not really begun to try to scale this up. You will need to be clear about expected benefits in order to effect the cultural change needed.|
|Embedding||You are at a tipping point where fairly widespread experimentation is close to becoming mainstream practice. A key issue will be ensuring that business processes are sufficiently consistent to support a more holistic approach.|
|Enhancing||You are probably already supporting the core of the assessment and feedback life cycle with technology. You are looking to fill gaps and find more elegant solutions to existing workarounds.|
|Pioneering||You are looking to go beyond automation, standardisation and efficiency gains to ensuring that EMA has a truly transformative impact on learning and teaching in your organisation. Your organisation is probably a provider of many of the resources in our accompanying guide but we can still provide some inspiration and support.|
Why do the self-assessment?
For those in the early stages of EMA adoption the self-assessment will serve as a development tool and point you to resources that may be useful. For those further down the line you may find that your experience to date lies in particular aspects of EMA and the self-assessment gives a more rounded view of where you might benefit.
For those at a more advanced stage it may serve as reassurance that what you are doing is regarded as good practice but there may still be some areas where there are opportunities to do things differently.
Who should do the self-assessment?
The self-assessment aims to support whole institution development while recognising that different parts of the institution may be at different starting points and working towards different goals. It can therefore be completed from different perspectives:
- A central support team may complete the evaluation from a whole institution perspective
- Individual departments, schools or faculties may respond in the way that reflects their own practice
- Course or programme teams may use the tool to give an even more localised view of practice
Completing the questions
Because the self-assessment is designed to work at different levels we ask you to define what happens in your ‘organisation’ in many of the questions. Organisation in this case refers to the coherent entity on behalf of which you are responding so it may be your programme team, your department or your whole institution. We also ask about consistency across your ‘component areas’ and practices at ‘local level’. At institutional level component areas/ local level will generally be schools or faculties, at department level they may be programmes of study and at programme/ course level they may be individual modules.
In many cases it may not be possible for an individual to answer all the questions. Indeed, we suggest that the self-assessment should be done as a group exercise because the dialogue that ensues is the first stage in the change process.
What will the self-assessment tell you?
The self-assessment tool will give you a report rating you at one of five levels against a range of headings:
- Strategy/policy and quality assurance
- Curriculum data
- Processes and working practices
- Student experience
A diagrammatic representation will serve as a quick overview of strengths and weaknesses.
An onscreen report will give you a set of suggested actions intended to help you build on your strengths and address your limitations and it will also provide links to resources that might help you carry out the suggested actions. The resources may be in the form of practical guidance, checklists and templates from other institutions or case studies and examples of practice.
To obtain an email copy of the report scroll to the bottom of the screen and enter your email address.
How should you use the self-assessment outcomes?
The tool has been tested with a range of experts from different institutions. They reported that the outcomes matched their own understanding of their strengths and weaknesses and that the suggested actions fitted with their experience of how to make progress in each of these areas.
Every institution is of course unique and your own local knowledge will be needed to put our generic guidance into context.
Usually you will be offered a range of suggested actions and resources for each development area. You can have confidence that all of these approaches have worked for others but it will be up to you to decide which are best suited to your particular circumstances. As well as using the specific resources suggested you will find additional support in our guide to Transforming assessment and feedback with technology.
Depending on where you sit within the institution not all of the suggested actions will be within your remit to implement. Some actions such as those requiring clearer definition of policy or improvements to business processes may need support from a higher level of authority. Nevertheless the self-assessment outcomes will provide useful evidence of need for you to begin the necessary conversations.
There will always be some concerns about the validity of data from a self-reporting exercise such as this. Different people may have a different interpretation of some of the questions and responses and it is only through dialogue that such differences can be explored and enhance the institutional knowledge base. Most of those involved in the collaborative development of this tool found the dialogue instigated by the self-assessment process to be the most valuable aspect of the activity.
We suggest the tool is best suited to supporting continuous improvement rather than any kind of benchmarking.
Whether your approach to developing EMA capability is top-down or bottom-up and whether you are a policy maker or a practitioner, you will probably find that you want to compare results from different parts of your institution. This will help you target staff development, select areas to pilot new approaches and identify good practice that can be built upon.
What should you be aiming for?
Our five level scale reflects the increasing use of EMA bringing further benefits for students and institutions. It should however be viewed with a number of caveats.
The scale is not a simple linear one. The first two levels are quite similar in terms of the user experience. You may correspond to the researcher level because your institutional practice is disjointed and people do not have a clear idea what others are doing. However, the overall user experience may not be significantly different to that of institutions at the explorer level.
Progress through levels
Institutions have also reported that the amount of effort needed to move between levels is not equally distributed. The most significant amount of effort is needed to get from the early stages to the embedding and enhancing levels. Once there, further progress is proportionately easier to achieve.
Progress through the levels is associated with evidence of greater benefits but that is not to say that every institution will necessarily be aiming to reach the highest level. In some cases institutions may provide an excellent student assessment experience in spite of constraints on how much they can invest in supporting information systems.
A co-design initiative
The self-assessment tool was developed using our co-design approach and we are particularly grateful for participation from the following institutions:
|Anglia Ruskin University|
|Birmingham City University|
|Manchester Metropolitan University|
|University of Bradford|
|University of Edinburgh|
|University of Hull|
|University of Nottingham|
|University of Sheffield|
|University of Southampton|
|University of York|
Our latest blog post is a guest contribution by Stuart Allan who has just completed an MSc in Digital Education at the University of Edinburgh. Stuart has been undertaking research into online exams. This is a short reflection on his research and Stuart will be joining us for a webinar on the topic in September so watch this space for further details.
I’ve been interested in exams and how they might relate to learning ever since my undergraduate days, when my degree was decided by nine finals in the space of ten days. (I still have nightmares about them…)
So when I was thinking of a topic for my (MSc in digital education) dissertation research, I wondered how useful digital technologies might be in final, high-stakes exams. As I read more, I discovered that the published literature on the specifically educational (as opposed to administrative or technical) implications of online exams was actually very small. (Myyry and Joutsenvirta (2015) is an interesting place to start.)
I found that the use of online exams often follows one of two main approaches:
- migration (transposing traditional exams to digital environments in order to achieve organisational gains, e.g. improved efficiency), and
- transformation (using digital technologies as a catalyst to redefine summative assessment and align it with the perceived needs of contemporary students).
My main focus was on how the migration and transformation approaches translated into educational practice in particular contexts. I interviewed eight higher-education staff involved in designing, developing and delivering online exams across four countries. They talked at length about their experiences, beliefs, aspirations and frustrations.
Instead of finding one approach to be better than the other, I concluded that both the migration and transformation approaches had significant shortcomings. The migration view seems to assume that online exam environments are instruments that we can use to achieve pre-ordained aims (such as improved efficiency); however, in my interviews I found examples of technologies interacting with, and having significant implications for, educational practice. The sociomaterial perspective was very useful here (see Bayne 2015 and Hannon 2013).
I also found the transformation view to be problematic in its own ways. For instance I began to question the validity of claims that online exams are a logical response to society’s changing needs, and to suggest that a more detailed understanding of the ways in which online exams might be qualitatively different to traditional exams is required.
Moreover, I discovered a potentially hazardous assumption that traditional exams could be migrated online (or be ‘a little bit digitalised’, to borrow one interviewee’s expression) as a prelude to more ambitious and educationally motivated changes further down the line. This transition appears not to be as straightforward as some might believe, and the migration stage often requires practitioners to overcome challenges that are unexpectedly time-consuming and financially draining.
One of the things I found most interesting was the apparent strength of some university professionals’ conviction that online exams must comply with exactly the same conditions – in terms of invigilation, the types of questions asked and candidates’ access to course materials, notes etc – as traditional pen-and-paper tests. To a large extent these assumptions set the tone for how the participants in my research used online exams.
With this in mind, I produced a number of questions that practitioners working with online exams might wish to consider:
- In your institution, what motivations exist for pursuing online exams, understood particularly in terms of how educational goals are defined at institutional and programme-specific levels?
- What assumptions are being made about what is meant by an ‘online exam’ within your context, and what can be done to support a constructive dialogue around these?
- To what extent does the dialogue between educational practice and the material contexts of particular digital environments result in online exams that are qualitatively different from traditional tests? For example, do online exams actively support, alter or proscribe particular types of student responses?
- In what ways might online exams be used to support increased assessment authenticity, in terms of both the context and content of examination tasks?
Lastly, I’d argue that the term ‘online exam’ itself – and all the assumptions about technology, education and assessment that seem to underpin it – might constrain the potential for developing practice to an unacceptable degree (see Gillespie 2010). Do we need to invent a new term to describe the summative assessment activities of the future? If so, what might that term be?
Bayne S. (2015) ‘What’s the matter with “technology-enhanced learning”?’, Learning, Media and Technology, 40 (1), pp. 5–20.
Gillespie T. (2010) ‘The politics of “platforms”’, New Media and Society, 12 (3), pp. 347–364.
Hannon J. (2013) ‘Incommensurate practices: sociomaterial entanglements of learning technology implementation’, Journal of Computer Assisted Learning, 29, pp. 168–178.
Myyry L. and Joutsenvirta T. (2015) ‘Open-book, open-web online examinations: developing examination practices to support university students’ learning and self-efficacy’, Active Learning in Higher Education, 16 (2), pp. 119–132.
Link to dissertation abstract: https://stuartallanblog.wordpress.com/dissertation-abstract/
Find me on Twitter: https://twitter.com/GeeWhizzTime
We’re pleased to announce the launch of a new guide on technology-enhanced assessment and feedback which has a focus on FE and skills. This new guide complements our existing suite of resources on technology in assessment and feedback, providing an up-to-date snapshot of practice across this diverse sector. The guide contains a rich variety of case studies and includes a podcast by Jayne Holt, assistant principal of Walsall College, on integrating college systems to track learners’ progress.
The message from the guide is clear: organisations, teachers, trainers and learners can all benefit from a technology enhanced approach to assessment and feedback. However, our e-assessment survey report (May 2016) reveals that some are yet to appreciate the full benefits. The overall picture, particularly for tracking systems and e-portfolios, is mixed, as is the ability of organisations to integrate their various technologies to maximise potential efficiency gains. For this reason, our latest guide to assessment and feedback focuses specifically on the needs of the FE and skills sector, with our assessment and feedback lifecycle model as its starting point.
You may also like to know our report on the evolution of FELTAG (June 2016) is now available. This offers further examples of effective technology-enhanced practice from colleagues across FE and skills sector.
We are delighted that 16 suppliers have so far responded to the Jisc and UCISA invitation to complete our EMA system requirements template.
The template was designed to clarify the core requirements of UK He providers. The responses will help universities and colleges better understand the ways in which particular systems can support their needs and to see how particular combinations of systems might work together. The template is based on our assessment and feedback lifecycle and is intended to be used in conjunction with our guide to EMA processes and systems.
Download the attached Excel spreadsheet (see below) to view a summary table and all of the individual responses.
Please note the following caveats:
- The data is based on self-reporting by the suppliers. Neither Jisc nor UCISA has tested the products and inclusion in this listing should not be taken as an endorsement of particular products
- The individual responses are recorded under supplier name not product name. If you are having trouble finding a product check you have the correct supplier. For example: Banner = Ellucian; Canvas = Instructure; Moodle Coursework = University of London; URKUND = Prioinfocenter
- The list of requirements relates solely to EMA and may not represent the full functionality of the systems included here eg student record systems cover many functions other than assessment
- This listing includes products intended to cover most of the EMA lifecycle as well as some more niche products. It is intended as a means of identifying which combination of products could meet your needs. It is not a like-for-like comparison of similar systems.
We will continue to update this information from time to time to cover additional products or new releases of the featured products, and will list any updates along with the original. Suppliers wishing to submit new or updated information should contact Lisa Gray email@example.com
You are welcome to use the comment facility on this blog to discuss and share information about the effective use of all of these products and particularly their interoperability to help others.
February 2016: EMA System Requirements Supplier Responses 2016 02 V1
Not all universities or colleges are currently implementing EMA organisation-wide. Some of you have told us you are looking for guidance on how to get started with EMA on a smaller scale.
In this podcast Bryony Olney from the University of Sheffield talks about how she went about organising an EMA pilot in her department.
You can also download a transcript of the case study.
Jisc and UCISA have been working with universities to develop a statement of requirements to help suppliers understand the needs of UK higher education in relation to the electronic management of assessment (EMA).
We have created a template for suppliers to respond to those requirements to help universities better understand the ways in which particular systems can support their needs and to see how particular combinations of systems might work together. The template is based on our assessment and feedback lifecycle.
We are now issuing an open invitation for suppliers who support the electronic management of assessment to respond and complete this template, by Friday 5th February. The downloadable template is attached to this blog post.
The template is being shared with all of the major suppliers of student record systems and learning platforms in the UK as well as suppliers of the most commonly used assessment related products. We welcome participation by any other interested suppliers. Please email your responses back to firstname.lastname@example.org. We hope the template is self-explanatory but feel free to contact us with any queries.
Responses will be published on this blog to help universities making system decisions. The full responses will be available to download and we will also compile an overview table which will pull out all supplier responses to the ‘included’ column.
Suppliers will be encouraged to update their initial responses as new product versions are released.
Download the template here:
This is a draft version of content for a new Jisc guide on EMA processes and systems that will complement the existing guide. Here on the blog we have split the two to help readability. This post should be viewed in conjunction with that on processes. The drafts will be open for feedback to help us improve the guide until 5th Jan 2016 so please do share your thoughts using the ‘comment’ function below.
Because the assessment and feedback lifecycle covers so many different functions most institutions need a range of systems to support all of their activities. The key areas covered by information systems are generally:
- course and module information including assessment details
- student records including marks, feedback and final grades
- submission of assignments
- marking and feedback
- academic integrity checking
- online testing and examinations
Ideally these systems should be able to exchange data readily so that institutions can mix and match technologies based on needs, preferences and making best use of the systems they already have. Currently however, interoperability between systems remains a key problem area. The expectation is that modern IT systems should have good APIs (application programming interfaces) ie a set of routines, protocols, and tools that describe each component of the system (data or function) serving as building blocks to create a plug and play architecture. In practice though the emphasis is still very much on creating a set of interfaces to move data around between systems on a point-to-point basis. This is complex to achieve and brings with it a maintenance overhead as whenever a particular system is changed, a series of interfaces must be rewritten to update the links to all of the other systems.
The systems are not the only problem. System integration often throws up a host of issues around institutional business processes, workflows, data definitions and data quality. This is why we have tackled the two topics in tandem. You need to ensure your data and processes are not an obstacle to making best use of your existing systems or to effective implementation of new and better systems.
Through collaborating with a working group of c.30 universities and the membership of UCISA we have identified the core requirements that UK higher education institutions have for information systems to support assessment and feedback practice.
The requirements are presented in a downloadable format that maps to the assessment and feedback lifecycle and which has supporting user stories to illustrate why the functionality is necessary. They are also viewable as embedded pop-ups as part of our EMA process maps.
Because we have concentrated on what is fundamentally important to all HEIs, all of the requirements should be considered as being ‘Must have’ priority.
Download requirements list as an Excel template
See the requirements embedded in our process maps
The specification has been publicised via Jisc and UCISA channels and suppliers of products of relevance to the EMA lifecycle are invited to use our template to highlight which of the requirements are supported by their product. Supplier responses are published on our EMA blog and customers of those suppliers are invited to use the blog for comment and discussion. The idea is that by sharing knowledge about effective use of a particular product, or about integration between a particular set of products, we can help institutions to get the most out of their existing investments.
As a supplier we suggest you continue to:
- consider the specification when preparing your product roadmaps
- update your response as new versions of your product are launched
- Respond to customer discussion on the blog so that the wider community can develop a better understanding of your product.
The requirements specification can be used as a basis for developing an ITT to select a new system for your institution. This will not only save you work; you can also have confidence that the major system suppliers will be familiar with the requirements expressed in this way so you have a better chance of getting accurate and meaningful responses.
Using this list as a starting point you can select the parts that are relevant to your particular procurement exercise and add features that are desirable for your institution as well as further detail about your existing product set that will need to interoperate with the new system.
For more guidance on how to go about choosing new technologies to meet your needs see our guide to selecting technologies. This will take you through managing a selection project, defining your requirements and conducting supplier evaluation.