This second ‘rant of the day’ focuses on practice which, I think, arises from the fact that most people are not as fortunate(?) as me in having data from hundreds and thousands of students on each module each year. It also stems from a very human wish for our data to show what we want them to show.
The first problem that arises is akin to that shown in the photograph (which, I hasten to add, has nothing to do with students, at the Open University or elsewhere – it is just an image I’ve found on XPert, which seems to be about the number of people who have looked at a particular photo). Wow yes, there has been a marked increase, of …ah yes…three! (probably the photographer’s mum, sister and wife…) – and look at all those months when the photo was not viewed (I suspect because it had not been uploaded then…). This example may relate to photographs, but I have seen similarly small data sets used to ‘prove’ changes in student behaviour.
The second type of problem is slightly more complicated to explain – but I saw it in a published peer-reviewed paper that I read last week. Basically, you are likely to need a reasonably large data set in order to show a statistically significant difference between the different behaviour of different groups of students. So if your numbers are on the small side and no significant difference is shown, you can’t conclude that there isn’t a difference, just that you don’t have evidence of one.