For more than ten years now, scientists have been discussing the so-called reproducibility crisis: often scientific results cannot be reproduced later and/or in other laboratories, despite the studies being conducted under highly standardized conditions. . Standardization includes, for example, using genetically identical animals, keeping animals in identically equipped cages and performing experiments in exactly the same way. To uncover the sources of poor reproducibility, researchers typically try to identify potential confounders under the experimental conditions. The first confounding factor is the experimenter, that is, the person conducting the experiment. A team led by behavioral biologists Dr Vanessa von Kortzfleisch and Prof Helene Richter from the University of Münster (Germany) has now studied precisely this factor in behavioral experiments on mice carried out simultaneously at three different locations. Their study has just been published in the journal PLOS Biology.
To the researchers’ surprise, the influence of different experimenters on test results was not as pronounced as previous studies suggested. On the other hand, the researchers detected other confounding factors. So what plays a much bigger role than the experimenter was the “laboratory” factor. More importantly, however, most of the variations were explained by inexplicable differences between individual mice. Specifically, this proportion of “unexplained variance” in the data was between 41 and 72 percent. “It’s particularly surprising,” says lead author Vanessa von Kortzfleisch, “when you consider that the animals were tested under highly standardized conditions within the same testing cohort, in other words, by the same experimenter in the same laboratory and under exactly the same conditions.
The results certainly do not mean that the experimenter is not a decisive factor. What they do indicate, however, is that different testing conditions in labs – despite standardized conditions – have a considerably greater influence on the outcome than the experimenter. These conditions can include, for example, small differences in ambient sounds or smells. “But what our results most importantly show is that biological variation plays a key role in animal research, even when animals come from inbred lines. In the future, we will need better strategies to incorporate this variation in controlled manner in the experimental design,” says Vanessa von Kortzfleisch.
Twelve experimenters on three sites
The background: Contrary to the dogma of strict standardization, there are alternative suggestions to systematically incorporate variation into experimental design to improve reproducibility. In order to determine whether involving multiple experimenters in a single study can increase external validity, and therefore improve the reproducibility of results, this latest study was conducted by twelve different experimenters in Münster, Osnabrück and Bern, all performing the same test. behavioral. battery with mice from two inbred strains. Such phenotyping experiments are widely used in biomedical research to study the effects of different genotypes on animal behavior and, thus, to draw conclusions about the genetic basis of certain human diseases. For example, in a so-called Open-Field test, researchers check whether a mouse is more or less anxious when exploring a new environment.
Specifically, the research team investigated whether a strictly standardized experimental design, in which all animals are tested by one experimenter, differs in reproducibility from an experimental design in which animals are tested by multiple experimenters. The team compared the experimental plans to see which of them gave the most consistent results across the three different labs. Additionally, the researchers investigated what other influencing factors might account for the variation in the data. One result was that at all three locations, researchers were unable to replicate some of the results, whether the experiment was conducted by a single experimenter or multiple experimenters.
Animal-based research: New experimental design for better reproducibility
Vanessa Tabea von Kortzfleisch et al, Do multiple experimenters improve the reproducibility of animal studies?, PLOS Biology (2022). DOI: 10.1371/journal.pbio.3001564
Provided by the University of Münster
Quote: Animal research: Experimenter influence on results weaker than expected (May 10, 2022) retrieved June 12, 2022 from https://phys.org/news/2022-05-animal-experimenters-results-strong.html
This document is subject to copyright. Except for fair use for purposes of private study or research, no part may be reproduced without written permission. The content is provided for information only.