E-assessment – the high stakes strategy

E-assessment is becoming an increasingly hot topic, with an increasing number of governments around the world taking their first steps in this area. Whilst e-assessment has alway been an option in Learning Management Systems, formalised testing at national scale is a relatively new phenomenon. This article explores the opportunities, risks and architectures associated with delivering e-assessment at scale.

For the clarity, the term “e-assessment” is used here as the collective noun for electronic delivery of High Stakes and Low Stakes testing, diagnostics and examinations. The term also covers both summative and formative testing.

Norway – with over 800,000 school-age students – was the first country to implement national level e-assessment. As part of a national programme for improving education, and after successful trials in 2009 where students took examinations on their laptops, all the national tests for Reading, English and Math are now digital. A large part of Norway’s exams are also conducted digitally.

Log-on screen for students taking national e-assessments in Norway

Students enrol in the exam at least one week before they sit it. On the day of the exam they are given a user-name and password. The PCs that they take the assessments on are owned by the students but provided by the school, so there is a minimum specification for the hardware and browser (HTML 5). It’s acceptable for students to use materials that they have stored on their Hard Drive or a USB but not to gain help over the internet. Schools are 100% responsible for ensuring compliance with the rules, and E-assessments are monitored by a combination of teachers and software.

Arild Stangeland from The Directorate for Education and Training at the Norway Ministry of Education explains that the Norwegian system breaks down into the following components:

• Administration of examinations, registration and results/reports
• Electronic national tests and diagnostic tests
• E-examinations
• Collaboration Solution for preparing exams and tests
• E-processes for preparation of materials for exams for the students

Each of these components have a separate technical architecture supported by a large stack of applications written in .NET, Java, and Flash, and maintained by the The Directorate for Education and Training. Several hundred servers are used, and BizTalk Server is at the centre of the architecture to co-ordinate traffic between different systems. A locally produced Learning Management System is used to deliver the assessments.

E-assessments in Georgia

Another country that has implemented a national level e-examination system is Georgia, in Eastern Europe. Microsoft’s Shota Murtskhvaladze reports that school graduation exams are now delivered through a “Computer Adaptive Testing” (CAT) system. Last year, 50,000 schoolchildren took the school-leaving exams in 8 subjects in 1520 public and private schools within an eight-day-long timeframe. The solution was developed by an agency of the MoE’s National Examinations Center.

There are a number or drivers behind the move towards e-examination:

1. Cost – the English examination system cost ~ $1bn in 2009. Much of this is tied up in paper-based processes – printing, delivering, collecting and scanning papers.
2. Flexibility – the potential for going beyond what students can physically write on a paper.
3. Speed and accuracy - the time from sitting the assessment to getting an accurate the result in front of those who need to know is compressed with e-assessment.

Whilst the benefits of moving entirely to electronic assessment are clear, some countries are using technology to manage individual component parts.

The assessment division of British company RM Education handles a range of tasks for a large number of UK and international examination and assessment boards. They deliver authoring, delivery, marking and results services. For example, the company carries out on-screen marking of scanned paper scripts for the International Baccalaureate.

RM Assessement have a range of service offerings

Since 2009, RM Assessment has been working in partnership with Cambridge Assessment, the University’s international exams group, to enable e-assessment in more than 3,000 test venues across 18 countries.

In Romania in 2011, SIVECO built a solution to publish the results of National High School exams. The solution produced 30 reports showing the results for 200,000 candidates and had to deal with high peak usage in a small time-frame – just 2 days.

To handle the peaks, SIVECO used Cloud technologies – Windows Azure in particular.  In this project the Romanian Ministry of Education gained ample processing power, eliminated downtime, and avoided spending $100,000 for a comparable on-premises infrastructure. Romania is far from alone in experiencing peaks in data generation and process - the whole assessment industry experiences significant peaks in demand and load during one or two months of the year, which makes Cloud technologies an ideal candidate for e-assessment solutions.

SIVECO used Azure to handle peak data loads

Cloud technologies are also being used to support e-assessment in Columbia. There, the Instituto Colombiano para la Evaluación de la Educación (ICFES) administers standardised tests to students and has used Cloud technologies to reduce costs and better manage online queries when scores are posted. ICFES moved to a Windows Azure platform in partnership with Asesoftware, and has cut costs by 80% and provided students a faster and more reliable solution.

Taking this a step further, the New South Wales Department of Education and Culture - the largest School District in the Southern Hemisphere – has moved to a complete cloud based e-assessment system for Year 9 Science Standards diagnostic testing (ESSA tests). Working in partnership with Australian company Janison, 65,000 students were tested last year in a comprehensive diagnostic assessment.

Part of New South Wales’s ESSA tests – c/o NSW DEC

Tests online revealed much more about how students were thinking, enabling the NSW DEC to provide high quality advice on how to improve teaching and learning. There were other benefits too - saving $200,000 on server infrastructure costs, saving printing and distribution costs, and gaining a week on marking time over previous years.

Risks

So if it’s that easy to do, why aren’t more countries doing it? The main barrier is risk. An assessment system failing during the critical period is headline news, as is inequity and inaccuracy. Many of these risks, however, are inherent in paper based systems too. There are plenty of examples of the wrong papers being delivered to schools, and papers getting lost on return to the examination centres. Like all mission critical IT systems, the key is to architect the system with risk mitigation as a top priority.

Architectures

A basic building-block view of an e-assessment system looks like this:

Key functions include:

A simplified Azure enabled workflow looks like this:

Using Azure as a key component in delivering e-assessment at scale. This is the kind of approach used by Janison for the NSW DEC ESSEA assessments.

1. Exam/Assessment Board produces and signs-off assessment content collaboratively.
2. Assessment content is pushed into the Cloud and distributed via a Content Delivery Network
3. Assessment content is cached at school/exam center level after the first student has viewed a particular resource. As candidates enter the examination centre, they are given a username and password on a card.
4. Just before the assessment starts, policies are enforced on the candidate’s client computer, and the assessment content is cached either in a dedicated application or on the browser. The candidate’s response data is cached locally and periodically sent to the Cloud via the school level cache.
5. In the Cloud, the candidate’s data sits in a queue, and is then stored in flat tables.
6. Encrypted data from the Cloud is sent to a data center for longer-term storage and processing and in relational databases. Once all the candidate’s response data is taken from the Cloud to the data warehouse, and the Cloud application is stopped.
7. Markers grade the work and ensure leveling and normalisation.
8. Results are collated, reported and analysed.
9. Results are passed on to relevant agencies for recognition and certificate distribution.

Security and Equity

It’s crucial that candidates are all able to use devices of the same minimal specification, which makes a straight BYOD policy – where any device is acceptable – a difficult proposition.

Enforcing policies on the client computer is a key component. Until recently, attaining ‘lock-down’ would have required each computer to join a domain. Whilst having a Domain and Active Directory joined client computer has many advantages, there is another approach -  a solution developed by FullArmor called GPAnywhere. This allows “portable” policies to be created from Group Policy Objects and be applied to any end point including a Virtual Application. This means that any device running Windows can have an Assessment policy applied to them.

FullArmor’s GPAnywhere

What next?

Another approach to delivery being considered by some is VDI. The ability to be able to push a virtual assessment desktop to a device and lock it down is appealing as it is potentially a simpler approach. However, there are continuity of service risks with VDI which have yet to be fully tested.

assessment is in its infancy, but many leading examination and assessment authorities are looking carefully into what’s next in this space.

CAT

Computerised Adaptive Testing (CAT) is a form of computer-based test that adapts to the examinee’s ability level. Medical students at St George’s, University of London using CAT based e-assessment tools are asked to make decisions along a branched narrative in which information and choices available at a later stage depend on the choices the student made earlier.

ACARA - the Australian Curriculum and Assessment and Reporting Authority – takes this a step further and are talking about how to provide candidates with branched routes through the assessment so they get appropriate recognition for what they have learned. A student who struggles with a question or task can be routed along a less demanding pathway, whilst a more able or better prepared student can be routed along a more demanding pathway – both are able to get the best out of the assessment process. Test-takers also do not waste their time attempting items that are too hard or trivially easy.

Simulations

The New South Wales DEC were able to exploit interactivity when they ran their science tests online. Being able to use interactivity in an assessment opens up a wide range of testing options – for example, asking candidates to build or construct something, conduct virtual experiments, use haptics to test dexterity, or develop an animated scenario. None of these options are practical in a paper and pencil assessment.

21st Century Skills

Whilst we will see paper-based assessment for a long time yet, the pressure is on to find ways of assessing 21st Century skills such as creativity, problem solving, communication and collaboration. Problem Solving is now part of PISA 2012 framework Also, ATC21 – the 21st Century Skills assessment project – is doing some very interesting work in the area of collaborative assessment – www.youtube.com/atc21s One thing is certain – pencil and paper testing won’t help much in diagnosing and assessing whether students have acquired 21st Century Skills or not, so its reasonable to conclude that assessment has a big future.

Conclusion

E-assessment has come a long way in a very short time and is one of the last main barriers to the wider adoption of ICT in schooling. It’s clear that Cloud technology is changing the game here - not only enabling lower cost of service, but also opening the possibility of global e-assessment, with assessment and Examination Boards being able to offer their services to anyone on the planet. With the advent of better biometrics, and new ways of supervising assessments remotely perhaps the most exciting prospect is the notion of assessments being available at any point in one’s lifetime, not just at specified times in the calendar.

Practically everyone on the planet takes many examinations and assessment over their lifetime, so the prospects of this age-old process being made more fair, accurate, helpful, available and engaging are very exciting indeed.

Additional Information

• Janison blog on ESSA: http://blogs.janison.com.au/janison-blogs/essa/Video
• Case study on ESSA: http://www.microsoft.com/casestudies/Case_Study_Detail.aspx?CaseStudyID=4000009354
• Janison’s CAFe dashboard shows the statistics when running the ESSA assessments: http://cafemetrics.cloudapp.net/essalive/run

• More information about national testing http://www.udir.no/Vurdering/Nasjonale-prover/
• Sample tasks: http://www.udir.no/Vurdering/Nasjonale-prover/Innhold-NP/Eksempeloppgaver/Eksempeloppgaver-nasjonale-prover.
• User manual: http://www.udir.no/PageFiles/46582/Brukerveiledning_PAS_PGS_Nasjonaleprover_Vers_1-00_Nynorsk.pdf?epslanguage=no

• http://www.xplora.org/ww/en/pub/insight/thematic_dossiers/articles/e_assessment/eassessment2.htm
• http://www.cambridgeassessment.org.uk/ca/digitalAssets/198405_MH_ALT-C_Poster.pdf

Azure

http://www.windowsazure.com

Thanks to:

Arild Stangeland, The Directorate for Education and Training, Norway Ministry of Education

Wayne Houlden, Aaron Wittman, Caroline Thompson and Niels Grootscholten, Janison, Australia

Eric Jamieson, Robert Cordaiy, Joanne Sim, Jim Sturgiss, and Penny Gill, from New South Wales DEC, Australia

Peter Adams, ACARA, Australia

Steve Harrington and Dave Patrick, RM Assessment

Alexandru Cosbuc and Florian Ciolacu, Siveco

Bob Chung, FullArmor

Horng Shya Chua and Puay San Ng, Microsoft Singapore; Bjørnar Hovemoen, Microsoft Norway; Shota Murtskhvaladze, Microsoft Georgia; Teo Milev and Ksenia Filippova, Microsoft Central and Eastern Europe; and Brad Tipp, Corporate HQ.