Rigour in journal articles

Published on Friday, April 7th, 2017

These points are taken from a talk by Sara Hennessy about assessing rigour in papers submitted to the British Journal of Educational Technology.

  • Is the account analytical or purely descriptive?
  • Do the authors critique the literature and present a balanced account or does the review gloss over known issues with educational technology initiatives?
  • How do the theoretical assumptions and explanations of the case compare with alternative explanations?
  • Are links made in the discussion/conclusions back to the conceptual framework?
  • Are the findings critically interpreted in relation to the existing literature?
  • Do the research questions convey genuine inquiry or do they assume that educational technology is a Good Thing?
  • What is the theory of change?
  • Were there any potential sources of bias?
  • What measures were taken to counter them? eg were any counter examples sought when collecting and analysing data?
  • Does reporting seem selective? Are the analyses systematic and explicit? Is the account reflective and evaluative?
  • Was there any kind of control in a technology intervention design?
  • Are there threats to validity and reliability? Does inadequate control of extraneous factors threaten validity of theoretical inferences from data?
  • Are limitations acknowledged?
  • Could reactivity have played a role? Did novelty value of shiny new tools increase motivation?
  • How robust are the claims – how adequately is the argument supported by the evidence provided?
  • Does the sampling strategy permit empirical generalization to a larger population?
  • Who is the audience? Is the work relevant and current?
  • Are there clear conclusions that generalized beyond specific case/context? Do they apply in other institutions?
  • Is the work applicable / of interest in other countries? Is the focus overly parochial?
  • Is educational technology serving only the privileged in developing countries who already have access?
  • SES and gender inequity, rural/urban divide, language, computer literacy, bandwidth and intermittent connectivity / electricity maintenance, technical support, gatekeepers / stakeholders, culturally appropriate content….
  • Is the research innovative or, at least, original
  • What is the significance and contribution to existing knowledge – theory? methodology? empirical data?
  • Has the innovation been tested with real users?
  • Are there any convincing learning outcomes?
  • Are there explicit implications for practice? Policy?
  • Many reports of interventions motivated by ‘how can I use…’ rather than educational need
  • How is it used, by and with whom, how often, for what purpose, under what conditions, with what support…
  • What is the role of the teacher? Has pedagogy-focused professional development been offered? What cultural shift in teacher and learner roles is necessary?

 


Giving a conference presentation

Published on Thursday, March 23rd, 2017

I’m just back from the LAK17 conference in Vancouver. While I was there, I talked to a professor who had given feedback to doctoral students on their presentations in previous years. In most cases, it was the first time they had ever had feedback on their presenting skills. Thinking back to some of the talks I have heard from senior academics, I would guess that some have never had any help iat all in this area.

For me, an important part of presenting is the interactive aspect. If I see people looking puzzled, I try to go into more depth. If I see people looking bored, I’ll try to shift the pace. If a group of people arrives late, I’ll do a brief recap. If something of note happens in the room (loud noises outside, a phone going off , a butterfly circling overhead) I’ll try to react. A presentation is an opportunity to engage an audience and also to engage with an audience. I try to achieve that. I’m not claiming that I always succeed, but it’s always something I aim for.

The opposite of this approach is someone who reads a prepared script. I can see the attraction – particularly if you are nervous or if you are presenting in a language you don’t speak well. However, it’s usually far more difficult for an audience to follow. You put in more words. You use longer words. You start adding in citations. Your audience may be extremely familiar with the major works of every single person you cite – but it’s unlikely. If you cite two or three people in a single sentence, even the most skilled audience will be struggling to keep up. If everything you want to say is in the paper, people could just read your paper. Your presentation needs to offer something extra.

Small text doesn’t work. If you want your audience to read it, it needs to be big. If you are presenting to a large audience in a large room, your text needs to be very big. Unless it’s there as an example of illegible text, then small text serves no purpose. Long text has similar drawbacks. Either you have to pause awkwardly while the audience reads it, or you have to read it to the audience while they are reading it themselves, or you talk over it and the audience isn’t listening because they are reading the text.

You can run into the same problem with inexplicable images. For example, you put up a picture of a bear because you are talking about a big problem and you see a bear as an example of a big problem. While you talk about the issue, your audience is busy running through possible explanations for your choice of image. Woodland creature? About to hibernate? Likes honey? Lives in Canada? I once spent 20 minutes trying to figure out the use of an image on the second slide of a presentation. It was a presentation about a strong piece of work – in fact, it later won the prize for best paper – but I didn’t hear a word of it, because I was so distracted by the image.

Images also routinely get used without accreditation. People who scrupulously attribute the sources of their ideas use pictures without any reference to their creators. Sometimes, they even use ones that are have watermarks making it clear they are commercially available and should not be used without permission. It doesn’t take much effort to find images licensed using Creative Commons (you can filter your searches on both Flickr and Google Images in order to return these images). And when you use the images, credit them. Credit the work that went into them. Creative commons usually requires the author’s name and a link to the appropriate licence. This isn’t arduous, and it’s acknowledgment you would give to others whose work you have used.

A final thought. Your final slide. This is the one that stays up through the questions and answers that follow your talk. It is the slide that the audience spends most time looking at. Don’t waste it by filling it with ‘Thanks!’ Reiterate your key point. Provide a link to your work. Offer your contact details. Pose a question. Help the audience to engage with you once your presentation has ended.


Writing a methodology chapter

Published on Wednesday, January 27th, 2016

There’s a difference between ‘method’ and ‘methodology’ – and it isn’t easy to grasp.

Partly that’s because the methodology chapter of a thesis contains your method and. when you’re grappling with the chapter, it’s difficult to see what you can add to that. You know how you’re going to collect your data. You probably even know how you’re planning to analyse your data. So how do you phrase that in the theorised way that your supervisor is asking for? At first sight, it does look as if your supervisor is trying to make a simple matter needlessly complex.

So let’s take a simple example. Imagine you meet a woman who owns a grocery shop, and she says she’ll pay you to find out how much fruit he has in her shop. At first sight, this too looks simple. You’re going to count the fruit. And, for the benefit of your supervisor (you are a doctoral student, after all), you note that this will be a quantitative approach.

You take in your notebook and pen, and you carry out a count and tally up your results. Because you’re a doctoral student with a little time on your hands you count them twice. You bring in your friend, and she counts them as well. A really reliable result. You go and tell the shopkeeper that she has 100 apples, 79 oranges, and 82 bananas. She tells you that isn’t the answer you wanted.

You’re a doctoral student, so you go and sit down and have a coffee and complain to a friend. And the friend, who happens to have been watching QI on television recently, says she thinks a banana isn’t rechnically a fruit. And maybe an apple is a fruit and maybe it’s a vegetable. Oh, and did you take pumpkins and cucumbers into account? So you grumpily stomp back to the shop and go through every type of produce with the shopkeeper and ask whether she defines it as a fruit. Then you count all the items defined as fruit. This time, the answer is 785. Or, annoyingly, 784 when you count a second time. The shopkeeper rejects both answers.

Time for another coffee, and another chat to your friend. How can you be expected to now what the shopkeeper wants? Well – your friend points out – you could ask why the shopkeeper wants this information and in what form she wants it. And it turns out the government is taxing fruit (which is defined in a particular governmental way) by the kilo. So you adopt a new system of classification and a new measure, and you tell the shopkeeper the answer is 350kg. She’s happy, and she pays you – which will keep you in coffee for a while.

Too give a meaningful answer you had to define your terms, and take the context and environment into account, and produce an answer that would be useful to the end user. Those are some of the things that you need to do in a methodology chapter.

 


A PhD is more than a thesis

Published on Monday, June 29th, 2015

Inspired by a Tweet I read recently about the distinction between a thesis and a PhD, I have been thinking about the difference between the two.

The university really focuses on the thesis, which must :

  • be of good presentation and style
  • be a significant contribution to knowledge and/or to understanding
  • demonstrate capacity to pursue further research without supervision
  • contain a significant amount of material worthy of publication or public presentation.

What else? Well, our university specifies you must be a registered student, you must live in the UK, you must pass your probationary period, you must spend a minimum amount of time as a registered student, you must make satisfactory progress, you must have a viva, you must make any specified corrections and you must present your thesis according to the guidelines.

All very thesis focused.

Vitae has a Researcher Development Framework that covers knowledge and intellectual abilities;  personal effectiveness;  research governance and organisation; engagement, influence and impact. The university encourages students to engage with this but, apart from reporting satisfactory progress at probationary review, it isn’t enforced or assessed.

Typically, students are assessment focused. They learn what they will be assessed on. It’s not surprising, then, that many doctoral students focus their entire attention on the thesis. That is the centre of their activity – everything else that takes place at the university is a distraction and has lower priority. In extreme cases, they only visit the university for supervision sessions, talk to nobody but their supervisors about their research, and focus totally on putting their thesis together and passing their viva.

But what then? A PhD is one line in a CV – perhaps five or six if you bulk it out with a description of your research. Permanent academic jobs in the UK and in many other countries may not be as rare as hens’ teeth, but they come pretty close. Even fixed-term contracts are difficult to get.

For employers, the PhD is not just one line in a CV, it’s also one line in a long job specification.

Academic employers want to know that you can publish papers, put together grant proposals, attract funding, increase impact via social media, create course materials, teach, mentor, work as part of a team, initiate projects, provide connections to a wider academic community and work on several projects at the same time.

The people getting the academic jobs are the people who can produce evidence that they can do all those things, and that they have already done those things. The people who treated their PhD as a period of academic apprenticeship, when the thesis is just one activity among many. The people struggling to get a toehold in the academic sector are the ones who have simply written a thesis.


Literature reviews

Published on Tuesday, May 19th, 2015

Link to a useful article by the Thesis Whisperer, aimed at doctoral students beginning to work on their literature review.

How to become a literature searching ninja


Capturing an online student feedback history to enable ipsative assessment and sustained motivation

Published on Thursday, May 7th, 2015

CALRG seminar by Dr Gwyneth Hughes, Reader in Higher Education, Institute of Education, UCL

‘Ipsative’ assessment is about comparing your current performance with your past perfomance. It comes from the Latin word ‘ipse’, meaning herself or himself

Assessment is predominantly taken to be a measurement of learning, and can be considered to be one of the cornerstones of a meritocracy.

A focus on marks, grades and performance distracts attention from the learning process. It can reduce the motivation of students who consistently receive low marks.

Ipsative assessment distinguishes between learning and attainment, it also helps to build motivation and self-esteem. It involves feedback on how a learner has progressed. One approach is the use of learning portfolios in which students provide evidence of how they have learned and developed.

A funded project on assessment found that students were rarely given written feedback on progress. However, assessors found it very difficult, because they did not know what feedback students had been given in the past – particularly if the feedback had been provided by other educators. Feedback is not stored centrally.

They developed a Moodle plug-in that provides a reports dashboard, bringing together all previous feedback.

Assessment Careers: Enhancing Learning Pathways through Assessment: funded project

Ipsative Assessment: book by Gwyneth, published by Palgrave

 


Scrum management framework

Published on Thursday, December 11th, 2014

With its scrums, sprints and stories, Scrum Management always sounds intriguing. I’ve been involved with several teams who have either used this system knowingly, or have employed elements from it. However, I’ve never seen the process formalised until I spotted it in the January 2015 edition of Wired magazine (where they had compressed a version of Jeff Sutherland’s book). Wikipedia tells me that this style of software development emerged in 1986 – so I guess I’ve been slow in investigating the approach.

Wired describes it as a seven-step process:

1. Small  teams. These should include the product owner, who has the vision and decides on the order in which things should be done, and the scrum master who facilitates communication and removes obstacles.

2. Tell stories. Each new features should be associated with a short story about the user and why the feature will add value for the user.

3. Assign effort points. Compare the stories and give them points for effort involved (or T-shirt sizes: small, medium, large and extra large).

4. Prioritise features. Each sprint should end with something that can be demoed, so make the chunks of work small enough to fit into a sprint.

5. Sprint. A sprint should be 1-4 weeks long – long enough to deal with a set amount of effort points.

6. Scrum. A 15-minute meeting every morning, standing up, so you’re not tempted to settle in. Three questions. What did you do yesterday to help finish the sprint? What will you do today to help finish the sprint? What obstacles need to be overcome?

7. Sprint review. At the end of the sprint the team meets to discuss what has been achieved, and to improve working practices for the next sprint.

 


I’m back!

Published on Tuesday, November 25th, 2014

I managed to lock myself out of this blog for over a year. First I forgot my password – then I forgot that I had created an in-box rule in Outlook that automatically junked any messages from my blog (I kept getting messages about moderating spam). So my password resets have all been vanishing into the ether.

Today I set a new in-box rule and spotted/deleted the old one. I’m back in.

First action – delete more than 10,000 spam comments that have arrived on the site while I have been away.


How to structure a literature review

Published on Monday, September 9th, 2013

It’s difficult to structure a literature review – you have read tens, or even hundreds, of articles, chapters, blog posts and presentations, and it appears almost impossible to pull them into shape and relate them to your own work. As a PhD student, or an early-career researcher, it is difficult to know how your contribution fits in.

One way forward is to treat your literature review as an art gallery (I guess a science museum would be a good alternative if you are not from an arts background).

You first welcome your reader / visitor to the art gallery and briefly point out that it deals with art and not science and, specifically, paintings. If they are looking for geological specimens, 19th-cenury curios or medieval tapestries, they are in the wrong place.

You walk them through the doorway – pointing out the names of famous painters engraved above the door, thus situating what you are showing them in the context of a tradition. You don’t need to dwell at the entrance, just show that you are aware of some of the greats who have gone before.

Next, you walk them down the corridor into the gallery of (for example) European art, pointing out the 16th-century gallery and the 19th-century gallery, taking them in more detail past the expressionists and the cubists. Here you are beginning to relate your work to some broad subject areas, showing awareness of how these have developed over time.

You pause to look closely at a series of paintings by Monet and Picasso, focusing the attention of your audience on two specific paintings. Here you are introducing the work most closely related to your own, drawing attention to salient points and identifying the gap that your work will fill.

Finally, you lead them into the new alcove you have constructed, to look at the contents of that alcove. This is the work that you will describe and explore in the following sections or chapters.

The route you have taken helps your audience to understand what they see in the new alcove. People coming straight to the alcove wouldn’t really understand what was going on there, and certainly wouldn’t be able to understand it in terms of what had gone before. People choosing their own path through the gallery might miss the significance of your alcove, or understand it in a completely different way.

Your tour guides your audience through the environment to your work. They may already know that environment very well, and be looking out for key landmarks, or even for their own work, but it is only you who can create for them the route that shows your work to its best advantage.


Learning analytics and ethics

Published on Tuesday, July 2nd, 2013

Way back when, I did some thinking about the differences between approaches to ethics in the Arts and the Social Sciences. To generalise, the Social Sciences treat the Internet as space, whereas the Arts treat the Internet as text. As I noted at the time: if you view the Internet as a virtual space populated by human actors, then you need a human subject approach to ethics, with informed consent a big issue. If, on the other hand, you see the Internet as an accumulation of texts, then your concern is with data protection, copyright and intellectual property rights. One practical example of this is that giving the real name of a data source is typically unethical behaviour in the Social Sciences, while failing to give the real name of a data source is typically unethical behaviour in the Arts.

So ethical behaviour is not a given – it is context dependent.

Extending this to learning analytics, a one-size-fits-all approach to ethics won’t necessarily work. Ethical research behaviour depends on what we are doing, on what we are trying to do and on what those involved in the research believe we are trying to do. The ethics discussion at #lasi13 suggested many of us are trying to different things – so maybe our approach to ethics will need to vary according to context.

Much of the discussion about the ethics of learning analytics this morning was framed in terms of learning as an economic transaction. The student contributes time, effort and money to an educational institution and, if this transaction is successfully completed, the student should emerge with a proxy for learning in the form of a certificate.

This view of learning is associated with  a view of data as a commodity to be owned and exchanged. In order for this transaction to be successfully completed, some exchange of data (attendance figures, grades, etc) is essential, and each party to the contract has rights and responsibilities in relation to the data.

So that implies a contractual perspective on ethics. My own work is in a different context – in informal or barely formal learning settings. Learning through social media, open educational resources, MOOCs, virtual worlds… The transaction is not overtly economic, the outcomes are more varied, the data have a different role. There is less sense of an obligation on either side. I suspect this means that the ethical concerns and priorities will be different, and that negotiating them will take us in different directions.

So one ethical code for learning analytics may prove impossible, we may need to shift from one to another according to context.