critical-reading.png

Artwork past Oliver Hoeller.

In the latest "How to…" guide, TIR offers a few tips and recommendations on the critical evaluation of scientific papers.

Introduction:
Reading scientific papers is a vital office of every researcher's regular activity. It'south a means of keeping up with the latest developments in your field, learning nearly new discoveries elsewhere, and getting to grips with novel methodologies and assays.

However, a primal element of this activity is the disquisitional cess of the cloth existence consumed. It's of import to apply the same rigour, scepticism, and discipline to reading others' piece of work every bit you would to assay of your own data. A paper should never, ever, exist taken at confront value, and given the growing popularity of preprints and the possibility of significant change in the fashion scientific information is disseminated, it'southward more of import than ever that young scientists are capable of objective scrutiny of outside work.

Critical assessment of a paper doesn't cease when it's published; arguably, it never ends. This is how y'all can contribute to that procedure.

Order of reading:
The best way to critically read a paper is:

  • Abstract
  • Figures + Figure legends;
  • Abstract
  • Introduction (plus References)
  • Results (plus Figures/Figure legends)
  • Word
  • Materials & Methods.

The key thing nigh this approach is that it places the data kickoff, before the authors have a chance to justify/explain themselves. A well-constructed paper should be intelligible from the Figures and Figure legends solitary, and this is often a useful source of first impressions. A classic shortcoming of published scientific work is a disconnect between the data (in the Figures), and the interpretation of that information (in the text). By going over the Figures offset (using only the Abstruse to guide your sense of what the overall story is about), you will be able to draw your own conclusions and get a sense of the overall quality before you lot come across what estimation the authors are favouring.

Plainly this approach is far more time-consuming that just reading the whole newspaper in publication society (Abstract, Introduction, Results+Figures, Discussion, Materials & Methods), and may not be practical if you are just reading a paper for interest.

If, however, a proper appreciation of the paper is important (for example, for a Periodical Club), and so the "Figures first" arroyo is definitely recommended. If possible, it's likewise worth spreading out the process over several days, so that you tin can keep returning to it with a fresh heed. If you have the time, and so allocating 3-4 days (not full time!) is good.

This would mean something like:

  • 24-hour interval ane: skim the Abstract, Figures, Effigy legends to get a flavour of the story.
  • Day 2: become over the Abstract, Figures, Figure legends in particular.
  • Mean solar day iii: quickly skim the Abstract, Figures, Effigy legends over again to refresh your retentiveness, then exercise the other sections in detail. Annotation the structure of the argument as you continue.
  • 24-hour interval 4: skim over the whole thing again, then write your study.

The key reward of spreading the task over several days is that it helps to take the emotion out of the process. If at that place'due south something that's annoying in the data presentation or writing, you have time to calm down before you lot put the report together. Equally, there's time to think over the authors' statement carefully and decide what you agree and don't agree with.

In chronological social club:

Office 1: Abstract
Read this first to give you a sense of the bailiwick, organism, and methodological arroyo used in the paper. At this stage, it'south simply a crude guide and then you lot tin can put the data in context; you will read it once more afterward y'all've finished going through the information.

Office two: Figures + figure legends
Get through the effigy legend for each figure LINE By LINE.

– Does the championship of the figure legend explain the master question addressed in the figure?

For each console/assay:

– Have the authors explained what assay is being used?

– Accept they provided a positive and a negative control?

– Practice the choices of positive and negative command seem appropriate?

– Take they quantified their data?

– Is the quantified data better than presenting all of the raw data? (this is the zippo hypothesis for quantification – if the quantified data are harder to follow or less informative than the raw data would take been, and so something is awry. Example: presenting a mean and an fault bar when only two independent experiments have been carried out)

– Take they explained their measure of reproducibility? (technical replicates, biological replicates, independent experiments – are YOU sure of the difference between these things? Come across TIR's guide if non.)

– Do yous agree with their reproducibility measures, or do yous retrieve the authors are unclear on the difference between technical/biological replicates and independent experiments?

– If they have carried out significance tests, do those tests seem justified? (for depression sample numbers, significance tests are unlikely to be meaningful)

– Are at that place scale bars on their micrographs?

– In blots, have they provided an input sample? Is the % value of the input disclosed? Are the % values of the other samples disclosed? (a loftier number of published fractionation/immunoprecipitation/pulldown figures are very vague on these points)

– How much of the data presented relies on "representative examples"? Ideally, a representative example should be accompanied by quantification of a larger sample.

– Conversely, when quantification of large samples has been carried out, accept examples of exactly what was quantified been provided? (Otherwise, "garbage in, garbage out" may apply)

–  (Plus point) Accept they provided schematics or similar to explicate what they did for not-specialists?

In one case you've gone through the figures and fatigued your own conclusions, yous can now come across what story the authors have constructed around the same data and if their interpretations are consistent with yours…

Part 3: Abstruse (accept ii)
– Does the abstract explain what the main question is, and what the master finding is?

– Practice you have a rough sense of the field of study expanse, organism, and technical approaches?

Part four: Introduction + References:
(Generally, the justification for the inquiry can exist skimmed over – this is often just framing)

Tip: copy and paste the references list into a new document and examine that alongside the manuscript text. This volition let you to bank check the references on the go.

– (For specialists) Have they cited the right papers for each particular claim? Tin You lot call back of any relevant articles that are missing?

– Is there an overreliance on citation of reviews? Papers should cite other chief literature whenever possible.

– Have the authors explained the background of the field? What is the big question or subject area?

– Have the authors defined what the extent of current knowledge is?

– Have the authors identified where at that place is a cognition gap, which they will attempt to address?

– (optional) Have they briefly outlined how they attempted to address information technology?

– (optional) Have they provided the principal conclusion in advance? (rhetorical device – helps prepare the reader to accept what they say)

Part 5: Results
*** Cross-cheque your cess of the results with your impressions from the figures+figure legends alone***

For each figure:
– Have they explained the rationale backside the experiment(s)?

– Have they explained what their positive and negative controls were?

– Are the positive and negative controls appropriate? (negative controls are sometimes misleading)

– What controls would YOU utilise if you were doing this experiment?

– Have the authors quantified the data?

– Is information technology possible/easy to relate the quantified data to the raw data?

– How would You have quantified the information? Tin can you recall of a better presentation? (e.g. bar vs line charts, pie charts, tables, image galleries, scatter plots, Venn diagrams, Euler diagrams)

– What do the data show?

– What is the authors' interpretation of the information? What is YOUR interpretation of the data?

– Does the authors' interpretation of the information seem valid? Does the estimation go beyond what the information accept really shown?

– Have the authors considered alternative interpretations of their data? (Or is it a merely-so story?)

– If so, have they tested whatever of these alternative interpretations?

– Can YOU call up of an culling interpretation of their data?

– Would You conclude the same thing they did from the data?

– Can Y'all recall of a more precise interpretation of their information? What have they actually shown?

– Have the authors strengthened their conclusion by obtaining the same conclusion using a different assay or different approach?

– What alternative assay would You lot apply to strengthen their conclusion? (How many tin can you think of?)

– (style point) Does the text go in figure order (A, B, C, D, E) or does it jump effectually (A, C, B, E, D) and is harder to follow?

– Have they explained why they did the adjacent experiment?

Repeat until the end of the Results section is reached.

– Could you follow the newspaper without consulting the supplemental figures, or do the supplements actually contain essential data? (This is a particular problem for journals that impose restrictive word/figure limits on authors)

Function 6: Discussion
– (style signal) Do the primary discussion points occur in the same order they do in the Results? (much easier to follow)

– Accept the authors placed their results in the context of existing knowledge? Take they highlighted areas of agreement/disagreement?

– Accept they acknowledged where alternative interpretations of their data are possible? Take they explained why they were not able to resolve these issues?

– Have they synthesised their results into a model of what they think is happening?

– Do YOU agree with the model? Does it accurately convey the information, or is it too speculative?

– Have they suggested how elements of their model could be tested?

– Do You agree with their conclusions?

Role 7: Materials and Methods
– Could You echo their experiments based on the information provided? If not, what information is missing?

– Have they used appropriate terminology ("separated by centrifugation", "amplified past PCR", "leaner were transformed using…") or slang? ("spun down", "PCRed", "plasmid 10 was transformed into bacteria")

– Have they indicated where they purchased or obtained not-standard reagents?

– Have they explained how data were quantified? (blinding, bias, conclusion of sample size, choice and justification of statistical tests, if used – almost NOBODY does this last point)

– Have they named the counterion for each pH buffer? (e.grand. Tris-HCl pH nine, not Tris pH9) Are there any other minor errors?

Part eight: Overall
– Do YOU trust the paper? Would y'all be happy to put your name to it?

– If you do non concur with the authors' conclusions or are unhappy with the data quality, what would You conclude about what has been learnt?

– How would Yous summarise the paper in three-4 sentences?

– A cardinal affair throughout is transparency. Yous should be able to look at the data, draw your own conclusions, and so check if your conclusions friction match those of the authors. If y'all depict different conclusions, or if y'all're not able to draw conclusions, this is a sign that the paper is not optimally written/presented (once again, this might exist partly due to limits imposed by journal).

Finally:
Recall that no paper is perfect, and if yous want to discover fault yous always can do. Skilful critical reading is not most trashing other people'due south piece of work, it's almost trying to establish what the objective truth is, and either helping people improve or helping yourself improve past considering how things tin/could be done.

Never, ever forget that the authors have invested a great bargain of fourth dimension, energy, and thought into their piece of work – in well-nigh cases, lower-quality work is a result of inadequate training or depression internal standards, non deliberate laziness. Your job as a disquisitional reader is to identify areas of weakness, and suggest how they could be strengthened. Always care for authors with respect, even if you experience that they've wasted your time.

(This document originally written by Brooke Morriswood in May 2018 for students taking the Molecular Parasitology course at the University of Würzburg)