Squaring the circle: a new alternative to alternative-assessment

Many quality assurance systems rely on high-stakes assessment for course certification. Such methods are not as objective as they might appear; they can have detrimental effects on student motivation and may lack relevance to the needs of degree courses increasingly oriented to vocational utility. Alternative assessment methods can show greater formative and motivational value for students but are not well suited to the demands of course certification. The widespread use of virtual learning environments and electronic portfolios generates substantial learner activity data to enable new ways of monitoring and assessing students through Learning Analytics. These emerging practices have the potential to square the circle by generating objective, summative reports for course certification while at the same time providing formative assessment to personalise the student experience. This paper introduces conceptual models of assessment to explore how traditional reliance on numbers and grades might be displaced by new forms of evidence-intensive student profiling and engagement.


Introduction: the changing context of higher education
Trends in applications to English universities show a changing pattern.The undergraduate population grew by 44% in the decade from 1999 but only a 10% increase is expected over the next (OECD, 2010).The average age of students is increasing, as is the number of applicants with non-traditional entry qualifications (Coleman and Bekhradnia, 2011).In 2010 the trebling of tuition fees for many undergraduate programmes may have contributed to the increased demand for parttime and vocationally-oriented degrees (UCAS, 2013) such that a third of UK students now study part-time.In Flexible Learning: Wrapping Higher Education Around the Needs Of Part-Time Students, Maguire (2013) presents a clear analysis of this developing concern.
Universities must adapt to this changing landscape in provision and practice.
Vocationally-oriented courses involve work placements and curriculum linkage between academic and professional components.Some external professional bodies demand assessment that is criterion-referenced and situated in context -in contrast to the norm-referenced practice of higher education, where summative assessment often takes place outside and after the experience of learning.The Assessment & Teaching of 21st Century Skills, an international organisation based at the University of Melbourne, identifies collaborative problem solving and learning in digital networks as key skills for the future (ATC21S, 2013).Similarly, employers for graduate occupations in the UK look for the 'soft skills' of initiative, interpersonal facility, communication, problem solving and flexibility (Prospects, 2012).Zepke and Leach (2010) represent these soft outcomes associated with student success in a matrix of factors showing the importance of engagement.However, such emerging priorities have yet to be reflected in the assessment practices of many universities.

Beliefs about assessment
Assessment has been identified as the single most influential factor in student learning (Gibbs and Simpson, 2004) and is a complex and contested activity.The concepts of validity and reliability reflect positivist assumptions that it is possible to arrive at a unitary measurement of what is being tested as if it were a physical property such as length; Yorke (2011) refers to this as the measurement fallacy.Biggs (1996) argues that a more appropriate way to view assessment is as being in constructive alignment with learning and teaching: as a set of procedures with intrinsic value to the learner as well as providing reporting functions for the teacher.Boud (2007) takes a similar view, that assessment should inform judgement, for the learner's own self-evaluation as well as for others' external evaluations.This constructivist position is usually referred to as assessment for learning, as distinct from the objective assessment of learning, contending that as there is no supposed entity to be measured, methods of assessment should reflect the diversity of pedagogical approaches (Sambell et al., 2013).
These contrasting belief systems resonate with the quantitative-qualitative debate familiar (if somewhat wearily) to researchers in the social sphere.Bryman (1988) anticipated a blended approach -latterly called mixed-methods -in which evidence from both traditions might be employed to present a fuller picture of the object of research.This paper will go on to explore the possibility of a mixed-methods approach to assessment.

Assessment in higher education
Currently in higher education the assessment of learning predominates over assessment for learning, as the quality control of certified awards and the demands of external professional bodies is a major concern of the strategic managerialism of universities (Preston, 2001).Gibbs (2006) notes that the modularisation of degree courses has increased the frequency of high-stakes summative assessments and has narrowed their focus, from integrative and processual to discrete and content-bound.
He concludes that as student-to-staff ratios have fallen and assessment loads have grown, the opportunities for formative assessment are increasingly constrained.

Disadvantages of conventional assessment
From a pedagogical perspective the high-stakes summative assessment typified by conventional examinations has three significant disadvantages.First is the 'backwash' effect (Biggs, 1999), whereby the content of what is being assessed influences 'strategic learners' to focus only on what will gain them higher grades.Their teachers, pressured to perform by managers and league tables, 'teach to the test' (for example, Klein et al., 2000).A second significant disadvantage is the limited use of feedback.
Feedback to students on their examination performance is typically brief and delivered after the learning cycle has ended.However, well-constructed formative feedback has been shown to have high motivational value to enhance learning (Black and Wiliam, 1998;Taras, 2002;Brennan and Williams, 2004).The wider context within which feedback is provided is a concern of Bailey and Garner (2010).In their study of academic staff attitudes to the provision of written feedback they note 'serious problems' in the tension between the formative and institutional purposes of feedback -the latter being quality assurance requirements for standardised approaches.Hence, tutors feel they are required to provide feedback that may not fully meet the needs of the student.Palmer et al. (2009) studied the development of first-year students' sense of belonging and identity as undergraduates, noting the anxiety of 'first feedback' and the strongly negative effect of critical comments.A third significant disadvantage of high-stakes summative assessment is that conventional methods are better suited to assessing propositional than procedural knowledge (Schön, 1983).The vocationally oriented courses discussed earlier are more likely to include the application of procedural 'know-how' in simulated and work-based environments that, as Williams (2008) argues, provide more relevant contexts than handwritten exercises in examination halls.
Institutional resistance to reforming assessment practice remains high.Elton and Johnston (2002) note a lack of evidence for the supposed validity and reliability of high-stakes assessment, with widespread and repeated calls for reform since the 1960s (and latterly, Brown, 2010).In the view of Knight (2002, 275) "… high stakes assessment in first degrees is in such disarray that it is difficult to know what grades or classifications mean and risky to treat them as reliable".Universities' resistance to change may be explained in part by the treatment of numerical marks as if they were valid and reliable indicators to provide the quality assurance confidence demanded for course certification.Such confidence is misplaced, however, and wide variations exist across British universities in the proportions of 'good honours' degrees awarded in different academic subjects (Yorke, 2009, 8).Price et al. (2008) call for a shift in emphasis from summative to formative assessment, away from marks and grades towards evaluative feedback focused on intended learning outcomes.They argue also for students to become more actively engaged and take greater ownership of their learning.Similar recommendations are made by Boud and Associates (2010) to place assessment for learning at the centre of course design.An institution-wide approach at Stellenbosch University reported by van Schalkwyk (2010) shows this is practically realisable.Its online Early Assessment System is used to collate the outcomes of formative assessment on all first-year students in order to target support interventions.A longitudinal impact study is currently being conducted, but early data indicate improved academic success and student retention.

Alternative assessment
What is known as alternative assessment provides a sharp epistemological and practical contrast to conventional approaches.There are so many resonances between alternative assessment and assessment for learning that the two ideas seem different expressions of the same epistemological stance towards education.Both view assessment not as summative measurement but as a formative, dialogic process by which the learner constructs knowledge on the basis of evidence from peers and teachers (Biggs and Tang, 2007).It is linked to the notion of authentic assessment (Torrance, 1995;Gulikers et al., 2004) and can variously: include formative assessments in stages over time rather than just summatively; involve mastery learning (Kulik et al., 1990); involve students more actively in collaborating and assessing; employ portfolios and reflective logs (Nomathamsanqa, 2008;Dyment and O'Connell, 2011); be more likely to employ problem solving and enquiry based learning (Deignan, 2009), and be contextualised in real-world or closely-simulated applications.This latter feature would seem to make it well suited to vocationally oriented courses.Its advocates point to the frequent opportunities for students to have ready access to meaningful feedback and they identify the learning and motivational benefits, including improved student retention, of this way of working (Gulikers et al., 2004;Savin-Baden, 2003;Brew, 2003;Waterfield and Parker, 2003).
Alternative assessment has a number of limitations that make it unsuitable as a direct replacement for conventional practice; Maclellan (2004) sees the major ones to be task specification and consistency of marking.The framing of appropriate tasks (for example, problems and simulations) is hampered by the difficulty of tuning out nonrelevant variables such as generic skills and knowledge, and by the difficulty of separating judgements of task outcome from those of student performance.
Consistency of marking is hampered by the difficulty of determining optimal assessment criteria and of judging across the variety of complex factors that make up real or simulated situations.Her conclusion is that alternative assessment is 'not a particularly convincing form for high stakes assessment' (ibid., 319).Maclellan's critique implies there is little middle ground between conventional and alternative assessment.Alternative methods offer well-documented benefits for student engagement, learning and retention, but as Kandlbinder (2007) describes, can be very time consuming and labour intensive to implement.They are not well suited (or even intended) as replacements for the discrete outcomes required by quality control systems for certified awards.It may be useful at this point to relate the discussion to a conceptual model.

A conceptual model for assessment
Two versions of a conceptual model for assessment have been devised as tools for viewing the relationship between learning and assessment.The first is in the form of a scale comprising five assessment activities with different 'distances' between the act of learning and the act of assessment; these are summarised in Table 1.

Table 1: Five assessment activities located along a dimension of Learning--to--Assessment Distance
At one end of the scale lies the conventional, closed-book examination consisting of content-specific exercises completed after the end of a learning period.This activity has the greatest conceptual distance -in terms of time and situation -between the act of learning and the act of assessment.Coursework assignments are similar, but may involve more of a balance of content-specific and transferable skills.Scheduling them towards the end of the learning period means it is unlikely that feedback will be available in time for these activities to have formative value.Alternative assessment activities such as enquiry based learning or problem based learning (typically involving simulations) are next, scheduled during the learning period and designed to involve transferable skills of applying knowledge content; they have greater potential to be formative, but this is dependent on the time available to tutors for this purpose.
Portfolios developed by the student through the process of learning record and evidence a variety of achievements and outcomes; sharing with tutors gives them considerable formative potential, but again is dependent on the time available.
Finally, continuous monitoring of the process of learning provides a fine-grained log of the actions and interactions of the learner.At the conclusion it is a summative record, but when sampled during the learning period it has the potential to be a powerful formative tool.As learning and its logging are contemporaneous and colocated the conceptual distance between them is zero.
Strongly implicit in this conceptual model is the hypothesis that the more frequent, fine-grained and coincident the assessment, the greater its formative potential -in providing timely feedback -to promote effective learning and effective learners.If formative potential for learning is added as an orthogonal dimension, a second version of the model can be created, as presented in Table 2.The version of the model shown in Table 2 employs the learning-to-assessment distance scale on the vertical axis, along which three regions of increasing distance have been identified, from coincident to remote.The horizontal axis represents formative potential for learning and has three regions of increasing potential, from weak to strong.In the first of these regions lie educational activities offering little or no formative potential for learning; in the final region are activities to which assessment is fully integrated with learning.The five assessment activities defined in Table 1 have been mapped to the resulting nine-cell matrix of Table 2 to reveal a linear sequence from examinations to continuous monitoring.
Until recently, the continuous monitoring of a learner's activities and interactions was impractical to achieve at scale and was confined to experimental studies.Rapid growth in the use of virtual learning environments (VLEs -or course management systems in North America) and electronic portfolios (ePortfolios) in higher education has generated substantial learner activity data that are currently under-used.Parallel advances have been made in the analysis and extraction of information from very large data sets (Ferguson, 2012).Linking these developments is the emerging practice of Learning Analytics.

Student profiling with Learning Analytics
Long and Siemens (2011, 34) define Learning Analytics as 'the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs'.The data are extant and machine-readable, drawn from students' online activities, coursework and formal assessments, but available at a scale that makes manual tracking impractical.The analysis of what is known as big data is well established in other walks of life but relatively new to Education.Business analytics has for many years employed complex computational techniques on data from customer and market behaviours to inform future strategy.These knowledge discovery techniques include data mining: to extract semantically meaningful information from patterns in very large sets of data in order to create probabilistic, predictive models.Comparable techniques are used in educational data mining, the term used for this algorithmic process, with Learning Analytics being its educational application (Bienkowski et al., 2012).Baepler and Murdoch (2010)  Long and Siemens (2011) describe Learning Analytics in higher education as operating at two levels.The first is course level, benefiting learners and their teachers through the analysis of social networks, learner discourse and conceptual development and informing 'intelligent curricula' that adapt to the ways in which they are used.The second level is departmental, where patterns of successful study behaviour are used to predict student success or failure.Three major effects of Learning Analytics position it at the intersection of student-centred pedagogy and evidence-based practice and each is potentially transformative; they will be examined after considering some examples of Learning Analytics in practice.

Implementing Learning Analytics
Monitoring students' academic progress by examining and profiling their VLE usage is a major application of Learning Analytics.Data visualisation systems summarise this information on 'dashboards' in simple ways, making it available to teachers and students.In addition to regular formative feedback, 'at risk' alerts can be generated,  Fritz (2011) found a strong relationship between online activity and grade achievement.Students gaining D and F grades used the VLE 39% less than those gaining a C or higher.
Student success algorithms are also used at Purdue University in the Course Signals system, employing a simple traffic lights dashboard to show whether a student's progress is satisfactory, at mild risk or urgent risk, and where interventions are triggered as early as the second week of a course.A study conducted by Arnold and Pistilli (2012) showed that most students thought Course Signals more personal, inclusive and motivating; they became more proactive in meeting course targets and the university recorded significant improvements in student retention.
The online Khan Academy ( 2012) is an open educational resource providing over 3,300 freely available video tutorials and mastery learning exercises on a range of academic topics.It employs extensive Learning Analytics in tracking progress and performance, available in dashboard form to students and teachers, with the latter able to review summaries of class activity and to replay the study logs of underperforming students.

Potentially transformative effects of Learning Analytics
Three major effects of Learning Analytics can now be examined; in combination they have transformative potential for teaching, learning and assessment in higher education.There is an implication here that learner activity data will be available at sufficient scale, but Prineas and Cini (2011) note a blurring of boundaries between conventional and online courses, which typically share the same VLEs and employ blended learning formats.Sharpe et al. (2006) report a similar trend in British higher education.
The first transformative effect is the provision of detailed and frequent feedback on students' learning progress and performance.The examples above show that when this is available to both learners and teachers successful students obtain personalised, motivation-building confirmations and their less-successful peers receive early interventions from teachers.Evidence available to date indicates the net effect is beneficial, especially for those first-year students and non-conventional entrants particularly at risk (Crozier et al., 2008;Palmer et al. 2009).This echoes the Stellenbosch study (van Schalkwyk, 2010) mentioned earlier, where an institutionwide approach to monitoring first-year students significantly improved student satisfaction and retention.
The second major effect of Learning Analytics is a potential to transform pedagogic process.Traditional pedagogy leads with teacher input -typically a lecture -in which subject content is presented, and is followed by consolidation activities such as seminars and private study.Traditionally, a sizeable proportion of teacher-student contact time is taken by the didactic delivery of information to large groups, where as noted earlier there are restricted opportunities for formative assessment and feedback.
There is some evidence that Learning Analytics may be used to inform the development of an adaptive curriculum in which educational resources can be iteratively shaped to better meet students' needs.For example, Western Governors University now pays publishers for online learning materials by their effectiveness in helping students achieve a B grade or better (Kolowich, 2012).In parallel is the opportunity for intelligent, adaptive assessment that matches to a student's level of achievement -in contrast to the linear, one-size-fits-all approach of traditional assessment.Prineas and Cini (2011, 10) believe this will reverse the traditional pedagogic process in the following way.Students will first work in their own time through interactive online course materials based upon mastery learning.Data on their progress and performance will be analysed and summarised for course tutors.Finally, face-to-face time in the classroom will be used for personalised activities targeting areas of need.The pioneering Open Learning Initiative at Carnegie Mellon University (CMU, 2012) provides some indication of how this might work in practice.
The third major effect of Learning Analytics might be upon the working practices and professional identity of university teachers.In the same way as the roles of medical professionals are changing from sole reliance on personal knowledge and experience to the greater use of evidence bases, so educators might increasingly employ data from analytics to inform their judgements and guide their interventions.Prineas and Cini (2011, 13) anticipate that such a transition would not be easy, but identify the opportunities for teachers, relieved of much marking and grading, to engage with the more rewarding provision of individualised attention and support for their students.
A wider view of the potential for evidence-based assessment is taken by Redecker and Johannessen (2013), who see Learning Analytics as one component of a 30-year transition through continuous, integrated assessment to personalised feedback and tutoring.From this perspective, assessment would develop from being a separate and periodic adjunct to an integral and continuous part of learning.These ideas are reflected in the conceptual model presented in Table 3.

Criticisms of Learning Analytics
In its current technological alignment and application, Learning Analytics seems far removed from the constructivist orientation of alternative assessment and assessment for learning.It has a behaviourist/cognitivist concern with the optimal structuring of learning materials based on the recordable actions students take, rather than with what they think and feel -ignoring affective aspects of education such as personal identity, self-worth and autonomy.In Analytics Examined (Educause, 2012) Clifford Lynch expresses concerns that present implementations, such as comparing individuals' activity profiles to the group's and making prompt interventions for those with identified problems, might jeopardise students' responsibility and ownership of their studies, and so restrict their opportunities to be different and to take risks.Gardner Campbell (ibid.) is also concerned that a mechanistic focus on observable activity neglects students' personal and affective development and their shared making of meaning.These are certainly the limitations of a crude and simplistic implementation of Learning Analytics but may be characteristic of the early days of this emerging field.What is needed is a more sensitive and teacher-mediated approach to interpret the outputs of analysis and support human judgements based on wider factors.The use of Learning Analytics in such a way, as an important component of comprehensive and mixed-methods student profiling, is the subject of the remainder of the paper.

Evidence-intensive student profiles
The adoption of Personal Development Planning (PDP) in the UK reflects a growing trend towards student profiling that goes beyond academic transcripts to include soft skills and wider achievements.PDP is defined by the Higher Education Agency as 'a structured and supported process undertaken by an individual to reflect upon their own learning, performance and/or achievement and to plan for their personal, educational and career development' (HEA, 2012).PDP is closely related to the use ePortfolios, and a network of British universities has evaluated PDP and ePortfolios for the purpose of creating graduate profiles (NARN, 2012).Some assessment methods have greater affinity than others for profiling, and this is represented in a variant of the conceptual model introduced earlier.
In the model illustrated in Table 3 the vertical scale has been replaced by evidence granularity: the extent and detail of assessment evidence sampling, from coarsegrained and performance-centric to fine-grained and activity-centric.Examinations, especially high-stakes ones, are conducted infrequently so are rated on this three-point scale as coarse-grained.At the other end is learning monitoring -now defined as Learning Analytics -in which frequent, fine-grained evidence is collected.On the horizontal scale is profile affinity: the suitability of data generated in the process of learning and assessment for representation in a learner's comprehensive record of achievement.As mentioned earlier, examinations would be a poor way to assess the soft skills demanded by employers, so their usefulness to graduate profiles is rated as low.Portfolios and Learning Analytics by their nature and function have a great affinity to profiling so have a high rating.

Conclusion: squaring the circle?
The growing vocational orientation of higher education is likely to make universities more accountable to external bodies for evidence-based, contextualised assessment for professional certification.If, as it seems, learning becomes more situated, then traditional high-stakes methods may be seen as less appropriate.Graduate employers looking for proficiency in soft skills should be able to go beyond norm-referenced ratings and methods of warranting (Knight, 2007) to extended transcripts, PDP and graduate profiles.Alternative assessment has the potential to provide a more relevant evidence base of students' holistic performance but has proved unwieldy and ill suited to the demands of professional certification.Learning Analytics could square this circle -but much work remains to be done in integrating its use to ensure sensitive interventions providing formative feedback to motivate and empower students.
Improvements in social network analysis and the data mining of ePortfolio evidence hold the potential to generate detailed and comprehensive summaries for inclusion in graduate profiles.The objectivity of such evidence could meet the needs of employers and professional certification, but could also be used by universities, not only to enhance the student learning experience but to inform the improvement of course management, learning materials and curricula.Learning Analytics is still at a very early stage in its development but universities should take careful note of its potential for use alongside conventional methods of student support to achieve substantial improvements in the practice of higher education.
see the former as more open and exploratory, with the latter more focused and hypothesis-driven.Commenting on the arrival of these techniques in higher education, Long and Siemens (2011) make a comparison with healthcare, noting the shift in focus from clinical practice -where medical professionals made decisions based upon their knowledge and experience -towards evidence-based medicine in which decisions are guided by far larger knowledge bases.Educational data mining increasingly employs techniques modelled on learning theory, and Ferguson (2012) notes how this has influenced the growth of social Learning Analytics, including discourse analytics and content analytics.Affective aspects of the learning process are manifested and can be identified through student interactions, captured by dispositions analytics and learning in social contexts including mobile learning, which relates social and geographical data.For example, the SNAPP (2011) software tool developed at the University of Wollongong provides a social network diagram to visualise interactions in the online discussion forums of commonly used VLEs.Teachers are able to see which students are central to discussions and which are disconnected.As such techniques advance it will be possible to construct increasingly comprehensive and sophisticated models of individual learners' progress.
students can receive guidance on what actions to take to improve their performance and study support staff can receive detailed diagnostic evidence on which to base interventions.Overviews are provided here of three examples of Learning Analytics applications.One of the first large scale implementations began in 2008 at the University of Maryland, where students use a 'Check My Activity' tool accessed through the VLE to compare their own online study performance with that of an anonymous sample of their peers.Research by

Table 2 :
Five assessment activities located in a matrix of Learning--to--Assessment Distance and Formative Potential for Learning

Table 3 :
Five assessment activities located in a matrix of Evidence Granularity and Profile Affinity