Computational methods for benchmarking and development of best practices for virus discovery and characterization in the HVP consortium
openNIDCR - National Institute of Dental and Craniofacial Research
The Human Virome Program (HVP) seeks to advance our understanding of the human virome’s role in health
and disease, but the field faces persistent challenges in virus discovery and characterization due to
inconsistencies in analytic methods and uneven access to tools and benchmarking datasets. Currently, a range
of bioinformatic tools exist for viral genome assembly, taxonomic classification, and novel virus identification,
although many require specialized computing environments or expertise. Moreover, there is a lack of
standardized benchmarks for assessing tool performance, leading to fragmented evaluations that make it
difficult for researchers to identify the most effective methods for their specific needs. This absence of uniform
evaluation criteria undermines reproducibility and reduces the efficiency of methodological innovation.
To address these challenges, we propose the development of a platform to provide access to: 1) high-priority
analytic tools, 2) standardized benchmarking datasets, and 3) an environment for continuous tool evaluation.
Leveraging our expertise in genomic data analysis and interoperable computing, we will first establish a
common HVP tool registry, prioritizing bioinformatic methods critical for virome research, including viral
genome assembly, taxonomic assignment, and novel RNA virus identification. These tools will be ported to the
Dockstore tool registry to maximize availability to all HVP researchers. This collection will be continuously
maintained based on feedback from the HVP Bioinformatics Working Group and tool authors.
To support standardized evaluation, we will construct comprehensive benchmarking datasets that reflect the
complexity and diversity of human virome samples. These datasets will include both real and synthetic data,
spanning a range of virome compositions, host backgrounds, and sequencing technologies. Synthetic datasets
will be designed to test specific tool capabilities under controlled conditions, incorporating known viral
genomes, engineered mutations, and varying levels of host and microbial contamination. Real datasets will be
curated from validated sources, providing examples for benchmarking common analytic tasks. These
resources will allow for standardized, reproducible comparisons of tool performance across multiple metrics,
including accuracy, sensitivity, precision, recall, computational efficiency, and resource utilization.
Finally, we will develop a benchmarking infrastructure using shared workspaces on the Terra cloud compute
platform that integrate benchmarking datasets with analytic workflows, enabling users to test and compare
tools directly. Researchers will have the flexibility to customize benchmarking parameters and compare
performance across different use cases. To foster community engagement and continuous improvement, we
will coordinate two benchmarking challenges officiated by the HVP Bioinformatics Working Group, the first
involving a single use case of known virus identification, and the second spanning three additional priority use
cases: 1) novel virus identification, 2) phage profiling, and 3) host prediction.
Up to $563K
health research