An analysis of the quality of experimental design and reliability of results in tribology research

Watson, Michael; Christoforou, Peter; Herrera, Paulo; Preece, Daniel; Carrell, Julia; Harmon, Matthew; Krier, Peter; Lewis, Stephen; Maiti, Raman; Skipper, William; Taylor, Ellis; Walsh, Jonathan; Zalzalah, Mohanad; Alhadeff, Lisa; Kempka, Reuben; Lanigan, Joseph; Lee, Zing Siang; White, Benjamin; Ishizaka, Kei; Lewis, Roger; Slatter, Tom; Dwyer-Joyce, Rob; Marshall, Matthew

Abstract

In recent years several high profile projects have questioned the repeatability and validity of scientific research in the fields of psychology and medicine. In general, these studies have shown or estimated that less than 50% of published research findings are true or replicable even when no breaches of ethics are made. This high percentage stems from widespread poor study design; either through the use of underpowered studies or designs that allow the introduction of bias into the results. In this work, we have aimed to assess, for the first time, the prevalence of good study design in the field of tribology. A set of simple criteria for factors such as randomisation, blinding, use of control and repeated tests has been made. These criteria have been used in a mass review of the output of five highly regarded tribology journals for the year 2017. In total 379 papers were reviewed by 26 reviewers, 28% of the total output of the journals selected for 2017. Our results show that the prevalence of these simple aspects of study design is poor. Out of 290 experimental studies, 2.2% used any form of blinding, 3.2% used randomisation of either the tests or the test samples, while none randomised both. 30% repeated experiments 3 or more times and 86% of those who repeated tests used single batches of test materials. 4.4% completed statistical tests on their data. Due to the low prevalence of repeated tests and statistical analysis it is impossible to give a realistic indication of the percentage of the published works that are likely to be false positives, however these results compare poorly to other more well studied fields. Finally, recommendations for improved study design for researchers and group design for research group leaders are given.

Citation

Watson, M., Christoforou, P., Herrera, P., Preece, D., Carrell, J., Harmon, M., Krier, P., Lewis, S., Maiti, R., Skipper, W., Taylor, E., Walsh, J., Zalzalah, M., Alhadeff, L., Kempka, R., Lanigan, J., Lee, Z. S., White, B., Ishizaka, K., Lewis, R., …Marshall, M. (2019). An analysis of the quality of experimental design and reliability of results in tribology research. Wear, 426-427, 1712-1718. https://doi.org/10.1016/j.wear.2018.12.028

• Journal Article Type: Article
• Acceptance Date: Dec 12, 2018
• Online Publication Date: Apr 10, 2019
• Publication Date: Apr 30, 2019
• Deposit Date: Oct 3, 2022
• Journal: Wear
• Print ISSN: 0043-1648
• Publisher: Elsevier
• Peer Reviewed: Peer Reviewed
• Volume: 426-427
• Pages: 1712-1718
• DOI: https://doi.org/10.1016/j.wear.2018.12.028
• Keywords: Tribology; Replication; Experimental design
• Public URL: https://hull-repository.worktribe.com/output/4086942
• Related Public URLs: https://eprints.whiterose.ac.uk/140294/