Evaluation using PROMPT

Young female student looking at books in book shop and surrounded by lots of books
The ability to critically evaluate information is an essential postgraduate research skill. This skill is particularly pertinent to the production of literature reviews, where a critical appraisal or analysis of the literature is required.

We recommend you use the PROMPT mnemonic (Presentation, Relevance, Objectivity, Method, Provenance, Timeliness) outlined in this document to provide a structured approach to critical evaluation of information.

Presentation

The way in which information is presented has a profound effect on the way we receive and perceive it. There are many aspects of presentation, any of which, if badly applied, can create a barrier between the message and the audience.

Poor presentation and inappropriate or confusing use of language will hinder your ability to critically evaluate the academic content. Try not to let poor presentation stop you from using what might otherwise be good quality, relevant information.

Things to be aware of

Choice of colour
Choice of font type and size
Use of diagrams and images
Lack of or illogical structure
Confusing layout
Poor use of language
Inappropriate or ineffectual writing style
Poor reproduction

Relevance

Relevance is an important aspect of information quality. It is not a property of the information itself, but rather of its relationship to the need you have identified. It may be a piece of high-quality information, but not relevant to the question you are asking or the scope of your search. There are a number of ways in which the information may or may not be relevant to your needs.

Geographical: It may relate to countries or areas which you are not interested in.

Level: It may be too detailed/specialised or too general/simple for your needs.

Emphasis: It may not contain the kind of information you are seeking, which may not be identifiable from the abstract.

To determine relevance

Be clear about your requirements. This will help you to be ruthless in discarding things on the basis of relevance.
Try to avoid having to read things in full. Look at the title, abstract or summary, keywords and descriptors.
If you are evaluating a large body of material, learn to skim read and/or scan information to get a quick indication of what it is about.
For more details on reading techniques see the Reading efficiently pages within OU skills for study.
Consider research in context.
Do the research results provide a unique insight into an aspect of your subject?
Do they confirm or refute the findings of other researchers?

Objectivity

In an ideal world, ‘objective’ or ‘balanced’ information would present all the evidence and all the arguments and leave you to weigh this up and draw conclusions. In the real world, however, we recognise that all information is presented from a position of interest, although this may not necessarily be intentional. Objectivity, therefore, may be an unachievable ideal.

This means that the onus is on you, the reader, to develop a critical awareness of the positions represented in what you read, and to take account of this when you interpret the information. It is also important to recognise that your own belief systems and opinions will influence your ability to be dispassionate and objectively evaluate information.

In some cases, authors may be explicitly expressing a particular viewpoint. This is perfectly valid, as long as they are explicit about the perspective they represent. Hidden bias or errors of omission, whether or not it is deliberate, can be misleading.

When producing a literature review, there is a particular onus on you to recognise any selective interpretation of data. You will need to comment on any significant omissions or biases that you may encounter in other people’s findings.

Things to watch out for

Perspectives: Do the authors state clearly the viewpoint they are taking? Are diverse perspectives presented?
Opinions: Academic articles will often present unsubstantiated theories for debate. Look out for opinion presented as if it were fact.
Language: This can be a useful danger sign. Look out for language that is either emotionally charged or vague.
Sponsorship (commercial, political, personal): For example, academic research may be sponsored by industry or government. This does not necessarily make the research less objective, but it may make its interpretation selective. Make sure that all potential vested interests are clearly identified and that the sponsors are happy to give access to the actual research data.

Method

For this aspect of PROMPT we do not refer to the evaluation of research methods per se, but to the information produced as a result of using particular methods. With your knowledge of the methods used in your subject area there are a few things you should think about.

Things to consider

Is it clear how the research was carried out?
Were the methods appropriate?
Ask some basic questions about sample size and nature, use of control groups, questionnaire design.
Are the results produced consistent with the methods stated?
Are the methods suitable for your needs?
Do you need the methods to be the same as, or different, to your own?

Do not assume that because a research report has been accepted for publication, it is error-free and meets a certain standard. There have been cases of fraudulent research that have successfully fooled the research establishment and been published in high profile journals.

Provenance

The provenance of a piece of information (i.e., who produced it, and where it came from) may provide a useful clue to its reliability. It represents the ‘credentials’ of a piece of information that support its status and perceived value.

It is, therefore, very important to be able to identify the author, sponsoring body or source of your information.

The ‘stable theory’ suggests that academic work is often valued highly just because it emanates from a prestigious research group or is published in a prestigious journal. So we should judge information on its own merits.

However, provenance can be an indirect clue to the reliability of information – a safety net that gives you the opportunity to check things out. Provenance can affect other people’s confidence in the sources you are citing.

Provenance: Things to think about

Authors

Are they acknowledged experts in the subject area? Are they respected and reliable sources?
Are their views controversial? Have they been frequently cited by other authors in the field?
To find out whether material has been frequently cited requires either prior knowledge or a citation search.
Are they known to have a particular perspective on the topic?

Sponsoring organisations

What type of organisation is it? Commercial company, voluntary organisation, statutory body, research organisation?
How well established is the organisation?
Does the organisation have any vested interests in the subject area being researched?

Publication methods

Any individual can publish anything on the web or post to a discussion list.
This must be judged on its own merit and with reference to the author’s credentials.
If published within a journal, what do you know of the editor and/or the editorial board? How might editorial policy influence what is published?
Is the journal well regarded?
Does it have a high rating in the Journal Citation Reports? Does this matter? Is the information peer reviewed? Many electronic journals do not have a peer review process.

Timeliness

The date when information was produced or published can be an important aspect of quality. This is not quite as simple as saying that ‘good’ information must be up to date; it depends on your information need.