By Gill Ferrell December 14, 2015
Uncategorized

This is a draft version of content for a new Jisc guide on EMA processes and systems that will complement the existing guide. Here on the blog we have split the two to help readability.  This post should be viewed in conjunction with that on processes. The drafts will be open for feedback to help us improve the guide until 5th Jan 2016 so please do share your thoughts using the ‘comment’ function below.

Selecting EMA systems

Because the assessment and feedback lifecycle covers so many different functions most institutions need a range of systems to support all of their activities. The key areas covered by information systems are generally:

• course and module information including assessment details
• student records including marks, feedback and final grades
• submission of assignments
• marking and feedback
• academic integrity checking
• online testing and examinations

Integrating systems

Ideally these systems should be able to exchange data readily so that institutions can mix and match technologies based on needs, preferences and making best use of the systems they already have. Currently however, interoperability between systems remains a key problem area. The expectation is that modern IT systems should have good APIs (application programming interfaces) ie a set of routines, protocols, and tools that describe each component of the system (data or function) serving as building blocks to create a plug and play architecture. In practice though the emphasis is still very much on creating a set of interfaces to move data around between systems on a point-to-point basis. This is complex to achieve and brings with it a maintenance overhead as whenever a particular system is changed, a series of interfaces must be rewritten to update the links to all of the other systems.

The systems are not the only problem. System integration often throws up a host of issues around institutional business processes, workflows, data definitions and data quality. This is why we have tackled the two topics in tandem. You need to ensure your data and processes are not an obstacle to making best use of your existing systems or to effective implementation of new and better systems.

System requirements

Through collaborating with a working group of c.30 universities and the membership of UCISA we have identified the core requirements that UK higher education institutions have for information systems to support assessment and feedback practice.

The requirements are presented in a downloadable format that maps to the assessment and feedback lifecycle and which has supporting user stories to illustrate why the functionality is necessary. They are also viewable as embedded pop-ups as part of our EMA process maps.

Because we have concentrated on what is fundamentally important to all HEIs, all of the requirements should be considered as being ‘Must have’ priority.

Download requirements list as an Excel template

EMA System Requirements Template for supplier responses i3

See the requirements embedded in our process maps

Guidance for suppliers on using the requirements specification

The specification has been publicised via Jisc and UCISA channels and suppliers of products of relevance to the EMA lifecycle are invited to use our template to highlight which of the requirements are supported by their product. Supplier responses are published on our EMA blog and customers of those suppliers are invited to use the blog for comment and discussion. The idea is that by sharing knowledge about effective use of a particular product, or about integration between a particular set of products, we can help institutions to get the most out of their existing investments.

As a supplier we suggest you continue to:

• consider the specification when preparing your product roadmaps
• update your response as new versions of your product are launched
• Respond to customer discussion on the blog so that the wider community can develop a better understanding of your product.

Guidance for universities on using the requirements specification

The requirements specification can be used as a basis for developing an ITT to select a new system for your institution. This will not only save you work; you can also have confidence that the major system suppliers will be familiar with the requirements expressed in this way so you have a better chance of getting accurate and meaningful responses.

Using this list as a starting point you can select the parts that are relevant to your particular procurement exercise and add features that are desirable for your institution as well as further detail about your existing product set that will need to interoperate with the new system.

For more guidance on how to go about choosing new technologies to meet your needs see our guide to selecting technologies. This will take you through managing a selection project, defining your requirements and conducting supplier evaluation.

By Gill Ferrell
Uncategorized

This is a draft version of content for a new Jisc guide on EMA processes and systems that will complement the existing guide. Here on the blog we have split the two to help readability. The drafts will be open for feedback to help us improve the guide until 5th Jan 2016, so please do share your thoughts using the ‘comment’ function below.

EMA in higher education: processes and systems 

This guide forms part of our suite of Assessment resources. It is intended to help those of you in HE review and improve the business processes that support your assessment and feedback practice and make the right choices as regards supporting information systems. We have created this guide for two main reasons:

• Our research has shown that many institutions struggle to get the most out of their information systems due to the variability and complexity of their business processes
• Both suppliers and institutions tells us there is a need to see core UK wide system requirements clearly expressed

What’s in this guide?

This guide contains resources to help people who manage assessment processes from both the academic and the administrative point of view, people who manage the supporting EMA systems and suppliers of EMA systems (ie systems supporting online submission, marking and feedback of assignments). It includes:

• A simple overview of the submission, feedback and marking process concentrating on the core tasks that all institutions need to carry out
• A set of requirements that specify what EMA systems need to be able to do to meet these core UK requirements
• A set of prompts to help you compare your own processes to our model and analyse where improvements could be made

The guide should be used in conjunction with our description of the assessment and feedback lifecycle in the guide to Transforming assessment and feedback with technology.

What are we trying to achieve?

The purpose of this guide is to help you implement processes that meet all necessary quality standards but which are no more complex than they need to be. Principles underlying the approach are:

• Ensuring tasks are carried out by the right people. We have identified tasks as being: a learner responsibility, an administrative matter or a task requiring academic expertise
• Automating any routine administrative tasks that can be automated. Digital information immediately opens up possibilities for streamlining the manual workload associated with dealing with vast quantities of paper
• Ensuring processes are undertaken for sound academic reasons. We hope you will use these resources to challenge complexity that exists for no better reason than “We’ve always done it that way”.

We can help you identify what a process needs to achieve, what skill set is needed and where information systems can help. What we can’t do is design standard workflows that will suit your institution. Individual workflows will depend on how your organisation is structured and what information systems it uses. You may well have different workflows for different assessment situations and this is fine so long as you are sure there is a valid reason why each variant exists.

What are the common problems?

Variation in approach across the institution

Research for our 2014 EMA landscape review[1] showed that responsibility for assessment and feedback policy and procedure is often devolved to local level within institutions. What this means in practice is that large institutions rarely carry out a particular function by means of a single, institution-wide, business process. Different faculties, schools, departments or even programmes, each have their own ways of doing things. This level of process variation is an inhibitor to achieving the efficiencies and benefits possible through the application of EMA technology because a series of time-consuming and cumbersome workarounds are likely to be needed to adapt the system to many different ways of carrying out the same activity.

Technology ‘bolted on’

Participants in our research frequently commented on the extent to which new technologies are ‘bolted on’ to old processes without people really taking the time to stand back and consider what the process is really intended to achieve.

In some cases poor process design is due to lack of time and appropriate skills. During the Jisc assessment and feedback programme, a concern was voiced that academic staff often find themselves on a ‘treadmill’ due to poorly designed processes. Their workload is such that they cannot pause to think about doing things differently. They also recognise that they do not have the skills to undertake process review and redesign without some more specialist support yet they know that they cannot improve their pedagogy without better designed processes.

Organisational myths

In other cases a significant part of the problem is the persistence of ‘organisational myths’ surrounding policy and process. The tendency to do things the way they have always been done is perpetuated by a belief that this is somehow enshrined in local or institutional policy. When challenged on existing approaches, academics are often surprised to find that many characteristics of the process are matters of historic choice rather than regulatory issues and, indeed, often surprised at how few regulations there actually are or how easy it is to make changes to perceived blocks and barriers in the regulatory frameworks. Variation in the application of assessment policy across an institution is often down to such myths about what actually constitutes the policy in the first place.

Example: Impact on student experience

Different approaches to carrying out the same task impact not only staff workload but also the student experience. As an example one institution that had the capacity to accept e-submission of all assignments based on the written word noted the following variations:

• One faculty accepted e-submission for postgraduates only but then printed out the assignments for marking
• Some course teams were happy to accept and mark submissions electronically but students were still required to submit a paper copy to meet the requirements of the coursework receipting system
• One department required students to submit a hard copy for marking and also an electronic copy to be submitted through the plagiarism detection system

Submission process detailed view

Below is an example of the submission phase broken down into more detail. This time we have identified the role of the EMA system to show which tasks can be fully automated. If you follow the link to the interactive version and hover over the question marks you will see hints and tips about managing the task.

http://ovod.net/wilbert/sundry/ema/submissionSwimLaneModded.html#

N.B. The guide should ultimately offer options to download as an interactive PDF and as a Visio file to edit

Marking and feedback process detailed view

N.B. Interactive view similar to the above to be added

N.B. The guide should ultimately offer options to download as an interactive PDF and as a Visio file to edit Marking and feedback process i2

Improving EMA processes

We suggest you use our 10 step model (and the detailed breakdowns) to compare against your own practice and ask yourself the following questions:

• Are you doing additional tasks – if so, why?
• Are the tasks being done by the right people eg do you have academic staff undertaking administrative duties that do not require academic judgement?
• Do you have systems that could carry out some of the tasks you are doing manually?
• Do you have multiple ways of performing the same task – if so, why?

You can find out more about ways to help improve your processes in our guide to process improvement.

Our guidance on process mapping includes some simple and user friendly techniques that can be used with all types of stakeholders to analyse issues with current approaches and suggest and evaluate ideas for change. In particular some of the techniques you might consider are: flow charts with swim lanes, rich pictures and RAEW (Responsibility, Authority, Expertise and Work) analysis and details on all of these can be found at the end of the process mapping guide.

You can also find out about how some universities have benefited from so-called Lean approaches at the end of our section looking at what is a process?

Examples from across the sector

Here are some further example of how different universities have approached the mapping of assessment and feedback processes to help them review practice:

N.B. Examples will be linked from the guide.

• The University of Oxford, department of continuing education used a swim lane approach with all actors identified developed in Excel. CASCADE workflows
• The University of Hull has created a more detailed version of our overview with its individual systems identified. Hull Ideal process
• The University of Sheffield, department of town and regional planning, used before and after process maps to review assignment submission. This is an example where the department identified a continued need to support marking on paper. Sheffield TRP Assignment Submission Process    Sheffield TRP Assignment Submission Process – To Be
• University X has used our lifecycle to produce a roadmap for an ambitious development programme over a three year period.

[1] Ferrell, G. (2014) Electronic Management of Assessment: a landscape review

By Gill Ferrell December 11, 2015
Uncategorized

Work is progressing well on a diagnostic tool that will help you assess the state of progress with electronic management of assessment in your area of work. At the end of the assessment you will be able to able to estimate your current state of maturity and develop an action plan.

The tool uses five stages of maturity:

The tool has been tested with a pilot group of institutions. As a result of their feedback we are making improvements so that the tool better meets the needs of those working at a more localised level than whole institutional change.

We hope to be launching the final version early in the New Year so keep an eye out for more updates.

The self-assessment tool is being developed using the Jisc co-design approach and we are particularly grateful for participation from the following institutions:

By Lisa Gray October 19, 2015
Uncategorized

As you all know we’ve been working on the development of an online toolkit to bring together examples, guidance and resources to support universities and colleges with their use of technology for assessment and feedback. We’re pleased to say that the first element of that suite of resources is now available, the ‘Transforming assessment and feedback with technology‘ online guide, with other elements due for launch in March 2016.

We’re keen to gain your thoughts on this resource, and have created a short googleform to capture these. So please do have a browse, and tell us what you think! This feedback will help to inform the final shape of the resource (the form will be open till the end of December).

The full suite of resources when launched fully in March will also include:

• an EMA self-assessment tool (a development requested by the working group) leading to resources within the guide (currently being tested);
• and also the EMA workflows / systems specification resources embedded within the guide (see previous blog post on where we are up to with those).

By Gill Ferrell October 5, 2015
Uncategorized

This post is intended to support an activity during the Jisc Learning and Teaching Experts Practice Group meeting on 14th October.

One of the aims of our EMA work is to help you implement processes that meet all necessary quality standards but which are no more complex than they need to be. Principles underlying the approach are:

• Ensuring tasks are carried out by the right people. We have identified tasks as being: a learner responsibility, an administrative matter or a task requiring academic expertise
• Automating any routine administrative tasks that can be automated. Digital information immediately opens up possibilities for streamlining the manual workload associated with dealing with vast quantities of paper
• Ensuring processes are undertaken for sound academic reasons. We hope you will use the resources we produce to challenge complexity that exists for no better reason than “We’ve always done it that way”

Our resources are intended to help you identify what a process needs to achieve, what skill set is needed and where information systems can help. What we can’t do is design standard workflows that will suit your institution. Individual workflows will depend on how your organisation is structured and what information systems it uses. You may well have different workflows for different assessment situations and this is fine so long as you are sure there is a valid reason why each variant exists.

Shown below is our 10 step overview of the submission, marking and feedback process. Click on the image to enlarge it.

This model covers all types of summative assessment where there is a mark given as well as feedback. It also covers iterative processes where students might undertake formative checking of their own work using text matching tools to review academic integrity, might undertake self or peer review and might be required to show evidence that they have engaged with their feedback before their mark is released.

Why are there 11 task boxes rather than 10? Call it cheating but we decided that ‘Apply penalty or mitigation’ is single task but might occur at different times depending for example on whether the penalty was for late submission or academic misconduct.

This is a high level overview. There is no right or wrong way to draw a process map – you need to choose the level that is right for you. For example you might choose to break down some of the sub processes further.

At the moment we are working on how to present process maps and associated system requirements in a way that helps both institutions and suppliers.

Below is an example of the submission phase broken down into more detail. This time we have identified the role of the EMA system to show which tasks can be fully automated. Hovering over a box will show the system requirements associated with the task and clicking a question mark will highlight common issues for institutions.

We suggest you use this model to compare against your own practice and ask yourself the following questions:

• Are you doing additional tasks – if so, why?
• Are the tasks being done by the right people eg do you have academic staff undertaking administrative duties that do not require academic judgement?
• Do you have systems that could carry out some of the tasks you are doing manually?
• Do you have multiple ways of performing the same task – if so, why?

We invite the Jisc Learning and Teaching Experts Practice Group to provide feedback on how the models are presented. There are currently 3 options:

1. By clicking on the image to go to an interactive diagram in a separate window.

2. A link to the image in an interactive diagramming tool.

https://www.lucidchart.com/documents/view/20f69d27-9b02-42ad-8bf7-e49b7bc083f4

3. A link to an interactive PDF. We think this works very well in Adobe but may render less well if you are using a different PDF reader.

Submission swim lanes i4

By Lisa Gray August 28, 2015
Uncategorized

We’re starting to think about a name for our EMA toolkit, which will be the home for many of the outputs of this project (see previous blog post ‘Solution development is underway‘). So, as a bit of August Bank Holiday fun, if you have any suggestions please do add them into the mix – we’ve created a googledoc to collate any thoughts at: http://bit.ly/1NDZt12. By Tuesday 1st Sept noon please!