The main reason for my visit to Uruguay was to attend the First International Workshop on New Metrics for Evaluation: Towards Innovation in Learning. This event was organised by the Centre for Research at the Ceibal Foundation in collaboration with INEEd, the ICT4V centre and the education division of the Inter-American Development Bank.
The workshop had four objectives, which the organisers framed as:
1. Using data for research and evaluation: towards an open and collaborative process for analysis, research and improving education.
2. Presenting experiences in the use of information systems for improving learning outcomes.
3. Presenting innovative approaches for evaluation and assessment of learning outcomes.
4. Policies, projects and programs for technology integration and data use in education.
It was a fascinating event, with representatives from countries across South and Central America, including speakers from Brazil, Chile, Colombia, Ecuador, Mexico, Nicaragua and Uruguay. Other speakers from outside the continent were Dragan Gasevic from Edinburgh, Neil Selwyn from Monash in Australia and Gilles Dowek from France.
I was particularly interested to find that Uruguay runs a ‘One Laptop per Child’ programme based on premises of equality and justice. Uruguay sees access to computers and the Internet as a right. You should have them in your classroom, just as you should have electricity in your classroom. Plan Ceibal has supplied 600,000 people (a fifth of the population) with laptops or tablets. Every child gets one when they start school, and they get a replacement every three years, with secondary school children now receiving Chromebooks. Internet is available nationwide – no one should be more than 400 metres from the Internet. There is a maintenance programme and a disposal programme, a teacher training programme, a learning management system, a suite of software, and a programme of video-conferenced English lessons, arranged in conjunction with the British Council.
- What are the potential gains, and what are the potential losses?
- What are the unintended consequences or second-order effects?
- What underlying values and agendas are implicit?
- In whose interests is this working? Who benefits, and in what ways?
- What are the social problems that data is being presented as a solution to?
- How responsive to a ‘data fix’ are these problems likely to be?
These wider questions of politics and power have not yet been taken up to any extent by the learning analytics community, but they look set to be bigger issues as the field matures.
My talk was on learning analytics, the state of the art and what the future might look like.
I also took part in a round-table discussion with Neil, Gilles and Dragan on issues related to learning analytics.
The back channel – mostly in Spanish – used the hashtag #edumetricas
During a visit to Uruguay, I was lucky enough to be invited to visit the Institute of Education at the ORT University in Montevideo. There, I gave a presentation to faculty members and postgraduate students on Innovating Pedagogy.
For the past four years, The Open University has produced an Innovating Pedagogy report annually. This series explores new forms of teaching, learning and assessment for an interactive world, to guide educators in productive innovation. As one of the report authors, I presented a quality enhancement lunchtime seminar on 23 March 2016 (part of the QELS series). In the seminar, I introduced the themes that have emerged from this series of reports – scale, connectivity, reflection, extension, embodiment and personalisation – and how these connect with modules (courses) run by the OU. The seminar included examples of innovative pedagogies in use at the OU, and identified others that could be used in future.
Fifty people attended the workshop, including invited experts (expert presentations), representatives of current European-funded projects in the field of learning analytics (project presentations), and representatives of the European Commission.
The workshop dealt with the current state of the art in learning analytics, the prospects for the implementation of learning analytics in the next decade, and the potential for European policy to guide and support the take-up and adaptation of learning analytics to enhance education.
The workshop began with a review of current learning analytics work by participants and went on to consider how learning analytics work can be taken forward in Europe (presentation on the LAEP project).
Participants at the workshop identified immediate issues for learning analytics in Europe. They set out considerations to be taken into account when developing learning analytics, made recommendations for learning analytics work in Europe and then identified both short- and long-term policy priorities in the area.Immediate issues for LA in Europe
Framework for development: A European roadmap for learning analytics development would help us to build and develop a set of interoperable learning analytics tools that are tailored for the needs of Europe and that have been shown to work in practice.
Stakeholder involvement: There is a need to bring different people and stakeholders on board by reaching out to groups including teachers, students, staff, employers and parents. Our current engagement with stakeholders is too limited.
Data protection and surveillance: As legislation changes and individuals become more aware of data use, institutions need to understand their responsibilities and obligations with regard to data privacy and data protection
Empirical evidence and quality assurance: More empirical evidence is needed about the effects of learning analytics, in order to support a process of quality assurance.Considerations for the development of LA
- Learning analytics can change or reinforce the status quo
- Learning analytics should enhance teaching, not replace it
- It is our duty to act upon the data we possess
- Desirable learning outcomes must be identified
- Be clear why we are collecting and analysing data
- Bring the data back to the learner
- Intelligent systems need human and cultural awareness
- Impressive data are not enough
- Undertake qualitative studies to understand how learning analytics can be aligned with the perceived purpose of education in different contexts, and which aspects of different educational contexts will support or constrain the use of learning analytics.
- Publicise existing evaluation frameworks for learning analytics and develop case studies that can be used to enrich and refine these frameworks
- Develop forms of quality assurance for learning analytics tools and for the evidence that is shared about these tools.
- Identify the limitations of different datasets and analytics and share this information clearly with end users.
- Explore ways of combining different datasets to increase the value of learning analytics for learners and teachers.
- Extend to different sectors of education the work currently being carried out in the higher education sector to identify the different elements that need to be taken into account when deploying learning analytics.
- Develop analytics, and uses for analytics, that delight and empower users.
Innovative pedagogy: Top priority is the need for novel, innovative pedagogy that drives innovation and the use of data to solve practical problems.
Evidence hub: Second priority is to secure continuing funding for a site that brings together evidence of what works and what does not in the field of learning analytics.
Data privacy: Participants considered that a clear statement is needed from privacy commissioners about controls to protect learners, teachers and society.
Orchestration of grants: The European grants system could better support the development of learning analytics if grants were orchestrated around an agreed reference model.
Crowd-sourced funding support: Set up a system for crowd-sourcing funding of tools teachers need, with EU top-up funding available for successful candidates.
21st-century skills: Focus on developing learning analytics for important skills and competencies that are difficult to measure, particularly 21st-century skills.
Open access standards: Standards need to be put into practice for analytics across Europe, with an open access forum that will enable the creation of standards from practice.
Ambassadors: We need more outreach, with ministries and politicians spreading the word and encouraging local communities and schools to engage.Long-term policy priorities
Teacher education: Top priority in the longer term was for media competencies and learning analytics knowledge to be built into training for both new and existing teachers.
Decide which problems we want to solve: In order to develop the field of learning analytics we need to have collective discussions on the directions in which we want to go.
Facilitate data amalgamation: More consideration is needed of how to combine data sources to provide multi-faceted insights into the problems we seek to solve.
Identify success cases and methodologies that give us a solid foundation: We need a coordinated approach to quality assurance and to the identification of successful work.
Several accounts of the workshop are available online, dealing with the morning of day one, the afternoon of day one, day one as a whole, the morning of day two, the afternoon of day two and day two as a whole.
I’ve been listening to educational technology hype recently with an eyebrow raised particularly in respect to the ideas being expressed around artificial intelligence and the role of intelligent agents to replace humans. One of the most recent examples of this is Mark Zuckerberg at F8 conference saying ““Our goal with AI is to build systems that are better than people at perception.” The Telegraph provides a summary of his keynote and the F8 conference.
Sit back and reflect on his statement for a moment.perception pəˈsɛpʃ(ə)n/ noun
- the ability to see, hear, or become aware of something through the senses. “the normal limits to human perception”
- the way in which something is regarded, understood, or interpreted. “Hollywood’s perception of the tastes of the American public”
What is perception? – a personal view of the world? – shaped by our emotional state and environment? – An entirely subjective reality. What do we mean by better perception? is this seeing the world logically without the trappings of emotion? – is it about the ‘wisdom of crowds? – If it’s the latter then we know that this is being gradually debunked because we are seeing greater confirmation bias within social media circles, I referred to this in a previous post as ripples in the pond, and there is evidence of the undermining effect of social influence. However there is no doubt that artificial intelligence will have access to a greater dataset and will have the ability to interpret data in ways that would be impossible to humans. My question though might be is that going to translate into better outcomes?
Invention comes from creative friction, discourse, questioning. In a world where we are all synthesized down within a crucible above the flame of artificial intelligence what happens to inspiration. interpretation. challenge? – this is of course a dyspotian future that people in the AI world are keen to promote because it creates a big dream of the future and a strong emotional connection.
But we do need to be concerned because at a minimum a possible future predicted by Gartner may see smart machines replacing millions of humans but at the same time we should be rational because we must recognize the Myths around AI’s and their usefulness is in support human endeavours, especially around tackling big data challenges.
…so what of humanity?
Eric and I introduce the group to our social media session (That’s me on the right) – Image courtesy of Ian Roddis .
Several months of planning, and a few nights waking up in a sweat, have led to a successful one day social media event which I co-chaired with colleagues from Learning and Teaching Solutions (LTS) on “Embedding Social Media to effectively support OU learners”.
There were two reasons that it’s taken so long to arrange:
- I wanted to introduce external perspectives to the topic to refresh our thinking. To this end my fantastic co-chair Beccy Dresden got in touch with Eric Stoller and we brought him to work with us. You’ll get a sense of his work from his blog. The thing I most like about Eric is his passion and enthusiasm for effective knowledge of, and use of, social media (more on that later).
- I wanted to tackle this problem at three levels in order to get actionable outputs and from both a top-down and bottom-up perspective, by that I mean (i) the Vice Chancellor, (ii) the people at Director/AD level responsible for learning & teaching, communications and marketing and (iii) the people who work directly in support of academic practice around module production and presentation.
I structured the day to begin with a conversation with the Vice Chancellor about the Open University and use of social media for a variety of strategic purposes, then we held a wider conversation which I chaired with a group of senior OU staff, from both academic and non-academic areas on “Embedding social media to effectively support OU learners facilitated by Eric Stoller”, then in the afternoon Beccy chaired sessions with academic support staff which began with a Keynote by Eric followed by parallel sessions around Social media for professional development with Eric and Lawrie Phipps (JISC) and Exploring the possibilities for social media within distance learning material hosted by Beccy Dresden and Steve Parkinson from Learning and Teaching Solutions (OU) and concluded with a plenary/roundup.
I began the morning session by introducing four provocations:
Provocation 1 – “Do we need a social media strategy for learning?”
Provocation 2 -“How and when do we embed social media practice within our modules and across the curriculum?”
Provocation 3 – “What can we learn from others?”
Provocation 4 – “Can we use social media to bridge the informal/formal divide?”
We then has an introductory chat about our different perspectives with social media and Eric followed this up by giving a talk which went into more detail starting with why does social media matter?
We kept the presentations short to allow plenty of time for discussion and the session has a lot of stimulating and interesting perspectives thanks in large part to Eric’s facilitation. Eric asked me before the session what type of conversation should we expect “..sometimes it’s a conversation about org culture and daring to dream/experiment that is needed…sometimes it’s more about choosing which tools are relevant right now and how to apply them in strategic / worthwhile ways.” I said that it was a bit of both and that turned out to be the case. Eric was also interested in the variety of perspectives and knowledge, for example some people in the room, such as Ian Fribbance, have used social media effectively in their practice for some time. The OU has some examples of great use of social media within pockets of the curriculum but there are also pockets of skepticism around social media and particularly about its relevance within formal learning and teaching. In fact one person at the meeting had never used social media and didn’t want to try it, to which Eric exclaimed “This is 2016! – I’ll not force you to use social media but we will talk later!”. The OU is also a place where practice is diverse and where OU academics don’t necessarily engage directly with students but that aspect is managed through tutors (or ALs) so there can be a disconnect.
Here are my key takeaways from the session:
- We aren’t using social media consistently and effectively to support and facilitate our discourse within the Open University and that has consequences for our engagement with our learners and more widely within our teaching communities.
- Things are improving. Examples of use of social media which have in the past been treated as ‘renegade’ are now being seen as exemplars of good practice, which is encouraging. e.g. the use of FaceBook within Social Science to support 26,000 learners
- It sounds like assessment may be the key to unlocking a bit of a cultural shift towards using social media more effectively…that and the push by certain individuals at the senior level is crucial. (this was echoed by Eric)
- We don’t need a formal strategy (considered to be constricting) and LTS have created a “manifesto” already as a grass roots approach so what the group thought would be most valuable was an enabling framework within which people could experiment with optionally using social media within their contexts.
- We need to ensure that academic staff are developed and supported to be digital scholars, which includes using social media effectively, so we see a need to build this into the “academic excellence” objective that is currently being formulated.
- We need to ensure that we consider appropriate platforms and risks when using social media so we see a need to build these elements into the “leadership in digital innovation” objective that is currently being formulated.
- We need to provide greater support for ‘grass roots’ initiatives and to remove barriers to adoption, this includes advocacy at senior level but also enabling through joined up thinking and grass roots initiatives such as the special interest group for social media.
- We need to continue to engage with external perspectives to help us to see how we compare, and to ensure that we are leading the way around social learning.
Eric is reporting back his thoughts to the Vice Chancellor, and we are now exploring how we can work with the Pro-Vice Chancellor (Learning and Teaching Innovation), the Pro-Vice Chancellor (Research and Academic Strategy) and the Head of Digital Engagement in particular to form an action plan to take this work forward – with thanks to Simon Horrocks, Beccy Dresden and The LTS team in particular who are supporting this work and considering the next steps.
Watch this space.
New paper out in the Journal of Learning Analytics Research, building on our previous papers dealing with how learners engage with MOOCs.Abstract
Massive open online courses (MOOCs) are being used across the world to provide millions of learners with access to education. Many who begin these courses complete them successfully, or to their own satisfaction, but the high numbers who do not finish remain a subject of concern. In 2013, a team from Stanford University analysed engagement patterns on three MOOCs run on the Coursera platform. They found four distinct patterns of engagement that emerged from MOOCs based on videos and assessments. Subsequent studies on the FutureLearn platform, which is underpinned by social-constructivist pedagogy, indicate that patterns of engagement in these massive learning environments are influenced by decisions about pedagogy and learning design. This paper reports on two of these studies of learner engagement with FutureLearn courses. Study One first tries, not wholly successfully, to replicate the findings of the Coursera study in a new context. It then uses the same methodological approach to identify patterns of learner engagement on the FutureLearn platform, and indicates how these patterns are influenced by pedagogy and elements of learning design. Study Two investigates whether these patterns of engagement are stable on subsequent presentations of the same courses. Two patterns are found consistently in this and other work: samplers who visit briefly, and completers who fully engage with the course. The paper concludes by exploring the implications for both research and practice.
Ferguson, Rebecca, & Clow, Doug. (2016). Consistent commitment: patterns of engagement across time in massive open online courses (MOOCs). Journal of Learning Analytics, 2(3), 63-88.