Genomic reproducibility in the bioinformatics era

doi:10.1186/s13059-024-03343-2

Review

. 2024 Aug 9;25(1):213.

doi: 10.1186/s13059-024-03343-2.

Genomic reproducibility in the bioinformatics era

Pelin Icer Baykal ^{1

2}, Paweł Piotr Łabaj ^{3

4}, Florian Markowetz ^{5

6}, Lynn M Schriml ⁷, Daniel J Stekhoven ^{2

8}, Serghei Mangul ^#^{9

10}, Niko Beerenwinkel ^#^{11

12}

Affiliations

¹ Department of Biosystems Science and Engineering, ETH Zurich, 4058, Basel, Switzerland.
² SIB Swiss Institute of Bioinformatics, 4058, Basel, Switzerland.
³ Małopolska Centre of Biotechnology, Jagiellonian University, 30-387, Gronostajowa 7A, Krakow, Poland.
⁴ Department of Biotechnology, Boku University Vienna, Muthgasse 18, 1190, Vienna, Austria.
⁵ Cancer Research UK Cambridge Research Institute, Cambridge, CB2 0RE, UK.
⁶ Department of Oncology, University of Cambridge, Cambridge, CB2 2XZ, UK.
⁷ Institute for Genome Sciences, University of Maryland School of Medicine, HSFIII, 670 W. Baltimore St, Baltimore, MD, 21201, USA.
⁸ NEXUS Personalized Health Technologies, ETH Zurich, 8952, Zurich, Switzerland.
⁹ Titus Family Department of Clinical Pharmacy, USC Alfred E. Mann School of Pharmacy and Pharmaceutical Sciences, University of Southern California, 1540 Alcazar Street, Los Angeles, CA, 90033, USA. serghei.mangul@gmail.com.
¹⁰ Department of Quantitative and Computational Biology, University of Southern California Dornsife College of Letters, Arts, and Sciences, Los Angeles, CA, 90089, USA. serghei.mangul@gmail.com.
¹¹ Department of Biosystems Science and Engineering, ETH Zurich, 4058, Basel, Switzerland. niko.beerenwinkel@bsse.ethz.ch.
¹² SIB Swiss Institute of Bioinformatics, 4058, Basel, Switzerland. niko.beerenwinkel@bsse.ethz.ch.

^# Contributed equally.

PMID: 39123217
PMCID: PMC11312195
DOI: 10.1186/s13059-024-03343-2

Review

Genomic reproducibility in the bioinformatics era

Pelin Icer Baykal et al. Genome Biol. 2024.

. 2024 Aug 9;25(1):213.

doi: 10.1186/s13059-024-03343-2.

Authors

Pelin Icer Baykal ^{1

2}, Paweł Piotr Łabaj ^{3

4}, Florian Markowetz ^{5

6}, Lynn M Schriml ⁷, Daniel J Stekhoven ^{2

8}, Serghei Mangul ^#^{9

10}, Niko Beerenwinkel ^#^{11

12}

Affiliations

¹ Department of Biosystems Science and Engineering, ETH Zurich, 4058, Basel, Switzerland.
² SIB Swiss Institute of Bioinformatics, 4058, Basel, Switzerland.
³ Małopolska Centre of Biotechnology, Jagiellonian University, 30-387, Gronostajowa 7A, Krakow, Poland.
⁴ Department of Biotechnology, Boku University Vienna, Muthgasse 18, 1190, Vienna, Austria.
⁵ Cancer Research UK Cambridge Research Institute, Cambridge, CB2 0RE, UK.
⁶ Department of Oncology, University of Cambridge, Cambridge, CB2 2XZ, UK.
⁷ Institute for Genome Sciences, University of Maryland School of Medicine, HSFIII, 670 W. Baltimore St, Baltimore, MD, 21201, USA.
⁸ NEXUS Personalized Health Technologies, ETH Zurich, 8952, Zurich, Switzerland.
⁹ Titus Family Department of Clinical Pharmacy, USC Alfred E. Mann School of Pharmacy and Pharmaceutical Sciences, University of Southern California, 1540 Alcazar Street, Los Angeles, CA, 90033, USA. serghei.mangul@gmail.com.
¹⁰ Department of Quantitative and Computational Biology, University of Southern California Dornsife College of Letters, Arts, and Sciences, Los Angeles, CA, 90089, USA. serghei.mangul@gmail.com.
¹¹ Department of Biosystems Science and Engineering, ETH Zurich, 4058, Basel, Switzerland. niko.beerenwinkel@bsse.ethz.ch.
¹² SIB Swiss Institute of Bioinformatics, 4058, Basel, Switzerland. niko.beerenwinkel@bsse.ethz.ch.

^# Contributed equally.

PMID: 39123217
PMCID: PMC11312195
DOI: 10.1186/s13059-024-03343-2

Abstract

In biomedical research, validating a scientific discovery hinges on the reproducibility of its experimental results. However, in genomics, the definition and implementation of reproducibility remain imprecise. We argue that genomic reproducibility, defined as the ability of bioinformatics tools to maintain consistent results across technical replicates, is essential for advancing scientific knowledge and medical applications. Initially, we examine different interpretations of reproducibility in genomics to clarify terms. Subsequently, we discuss the impact of bioinformatics tools on genomic reproducibility and explore methods for evaluating these tools regarding their effectiveness in ensuring genomic reproducibility. Finally, we recommend best practices to improve genomic reproducibility.

Keywords: Reproducibility; bioinformatics tools; genomics; synthetic replicates; technical replicates.

PubMed Disclaimer

Conflict of interest statement

The authors declare that they have no competing interests.

Figures

Fig. 1

Fig. 1

Schematic representation of three key concepts: technical replicates, methods reproducibility, and genomic reproducibility. The same sample is processed (library preparation) and sequenced multiple times, possibly in different laboratories, but using the same experimental protocols and sequencing platform. The output of these sequencing runs are technical replicates represented as FASTQ files. Data analysis is performed for each technical replicate multiple times to assess consistency of genomic results, which refers to methods reproducibility. Genomic reproducibility, on the other hand, evaluates the consistency of genomic results across technical replicates

Fig. 2

Fig. 2

Schematic representation of generating synthetic replicates. Based on a given dataset consisting of five reads R1, ..., R5 (left) four different types of synthetic replicates (right) are created by either randomly shuffling the order of the five reads (a), or by taking the reverse complement of each read (b), or by bootstrapping, i.e., resampling of the five reads with replacement (c), or by subsampling, i.e., selecting a subset consisting of three reads from the original five reads (d)

See this image and copyright information in PMC

References

1. Leipzig J, Nüst D, Hoyt CT, Ram K, Greenberg J. The role of metadata in reproducible computational research. Patterns (N Y). 2021;2:100322. 10.1016/j.patter.2021.100322 - DOI - PMC - PubMed
1. Bakinam T Essawy, Jonathan L. Goodall, Daniel Voce, Mohamed M. Morsy, Jeffrey M. Sadler, Young Don Choi, David G. Tarboton, Tanu Malik. A taxonomy for reproducible and replicable research in environmental modelling. Environmental Modelling and Software. 2020;134:104753.
1. Arnold, B. et al. The Turing Way: A Handbook for Reproducible Data Science. 10.5281/zenodo.3233986.
1. Goodman, S. N., Fanelli, D. & Ioannidis, J. P. A. What does research reproducibility mean? Sci. Transl. Med. 8, 341ps12 (2016). - PubMed
1. Whitaker, K. Showing Your Working: A Guide to Reproducible Neuroimaging Analyses. (figshare, 2016). 10.6084/M9.FIGSHARE.4244996.V1.

Publication types

Actions

MeSH terms

Actions
Actions
Actions
Actions

Grants and funding

LinkOut - more resources

Full Text Sources
- BioMed Central
- PubMed Central

[1] Leipzig J, Nüst D, Hoyt CT, Ram K, Greenberg J. The role of metadata in reproducible computational research. Patterns (N Y). 2021;2:100322. 10.1016/j.patter.2021.100322 - DOI - PMC - PubMed

[2] Leipzig J, Nüst D, Hoyt CT, Ram K, Greenberg J. The role of metadata in reproducible computational research. Patterns (N Y). 2021;2:100322. 10.1016/j.patter.2021.100322 - DOI - PMC - PubMed

[3] Bakinam T Essawy, Jonathan L. Goodall, Daniel Voce, Mohamed M. Morsy, Jeffrey M. Sadler, Young Don Choi, David G. Tarboton, Tanu Malik. A taxonomy for reproducible and replicable research in environmental modelling. Environmental Modelling and Software. 2020;134:104753.

[4] Bakinam T Essawy, Jonathan L. Goodall, Daniel Voce, Mohamed M. Morsy, Jeffrey M. Sadler, Young Don Choi, David G. Tarboton, Tanu Malik. A taxonomy for reproducible and replicable research in environmental modelling. Environmental Modelling and Software. 2020;134:104753.

[5] Arnold, B. et al. The Turing Way: A Handbook for Reproducible Data Science. 10.5281/zenodo.3233986.

[6] Arnold, B. et al. The Turing Way: A Handbook for Reproducible Data Science. 10.5281/zenodo.3233986.

[7] Goodman, S. N., Fanelli, D. & Ioannidis, J. P. A. What does research reproducibility mean? Sci. Transl. Med. 8, 341ps12 (2016). - PubMed

[8] Goodman, S. N., Fanelli, D. & Ioannidis, J. P. A. What does research reproducibility mean? Sci. Transl. Med. 8, 341ps12 (2016). - PubMed

[9] Whitaker, K. Showing Your Working: A Guide to Reproducible Neuroimaging Analyses. (figshare, 2016). 10.6084/M9.FIGSHARE.4244996.V1.

[10] Whitaker, K. Showing Your Working: A Guide to Reproducible Neuroimaging Analyses. (figshare, 2016). 10.6084/M9.FIGSHARE.4244996.V1.

Account

Save citation to file

Email citation

Add to Collections

Add to My Bibliography

Your saved search

Create a file for external citation management software

Your RSS Feed

Genomic reproducibility in the bioinformatics era

Affiliations

Genomic reproducibility in the bioinformatics era

Authors

Affiliations

Abstract

Conflict of interest statement

Figures

References

Publication types

MeSH terms

Grants and funding

LinkOut - more resources

Full Text Sources