GPU Acceleration of Molecular Modeling Applications

GPU-accelerated ion placement
Modern graphics processing units (GPUs) contain hundreds of arithmetic units and can be harnessed to provide tremendous acceleration for numerically intensive scientific applications such as molecular modeling. The increased capabilities and flexibility of recent GPU hardware combined with high level GPU programming languages such as CUDA and OpenCL has unlocked this computational power and made it accessible to computational scientists. The key to effective GPU computing is the design and implementation of data-parallel algorithms that scale to hundreds of tightly coupled processing units. Many molecular modeling applications are well suited to GPUs, due to their extensive computational requirements, and because they lend themselves to data-parallel implementations. Several exemplary results from our GPU computing work are presented in Klaus Schulten's Keynote Lecture from the 2010 GPU Technology Conference.

Elusive HIV-1 Capsid Structure Determination Accelerated by GPUs

Human immunodeficiency virus type 1 (HIV-1) is the major cause of AIDS, for which treatments need to be developed continuously as the virus becomes quickly resistant to new drugs. When the virus infects a human cell it releases into the cell its capsid, a closed, stable container protecting the viral genetic material. However, interaction with the cell triggers at some point an instability of the capsid, leading to a well timed release of the genetic material that merges then with the cell's genes and begins to control the cell. The dual role of the capsid, to be functionally both stable and unstable, makes it in principle an ideal target for antiviral drugs and, in fact, treatments of other viral infections successfully target the respective capsids. The size of the HIV-1 capsid (about 1,300 proteins), and its irregular shape had prevented so far the resolution of a full capsid atomic-level structure. However, in a tour de force effort, groups of experimental and computational scientists have now resolved the capsid's chemical structure (deposited to the protein data bank under the accession codes 3J3Q and 3J3Y). As reported recently (see also journal cover), the researchers combined NMR structure analysis, electron microscopy and data-guided molecular dynamics simulations utilizing VMD to prepare and analyze simulations performed using NAMD on one of the most powerful computers worldwide, Blue Waters, to obtain and characterize the HIV-1 capsid. The discovery can guide now the design of novel drugs for enhanced antiviral therapy.

Molecular Dynamics

Continuing increases in high performance computing technology have rapidly expanded the domain of biomolecular simulation from isolated proteins in solvent to complex aggregates, often in a lipid environment. Such systems routinely comprise 100,000 atoms, and several published NAMD simulations have exceeded 10,000,000 atoms. Studying the function of even the simplest biomolecular machines requires simulations of 100 ns or longer, even when employing simulation techniques for accelerating processes of interest. One of the most time consuming calculations in a typical molecular dynamics simulation is the evaluation of forces between atoms that do not share bonds. The high degree of parallelism and floating point arithmetic capability of GPUs can attain performance levels twenty times that of a single CPU core. The twenty-fold acceleration provided by the GPU decreases the runtime for the non-bonded force evaluations such that it can be overlapped with bonded forces and PME long-range force calculations on the CPU. These and other CPU-bound operations must be ported to the GPU before further acceleration of the entire NAMD application can be realized.


Power Profiling

 Our team has recently developed hardware and software for measuring and optimizing the power efficiency of VMD and other computational biology applications on mobile computing hardware, ensuring that VMD runs well on battery-powered devices. By measuring power consumption of both the host computer and either integrated or discrete GPUs, one can measure the power consumed during different phases of application execution, and optimize the trade-off between performance and energy efficiency.

Multi-Resolution Molecular Surface Visualization

Molecular surface visualization allows researchers to see where structures are exposed to solvent, where structures come into contact, and to view the overall architecture of large biomolecular complexes such as trans-membrane channels and virus capsids. Recently, we have developed a new GPU-accelerated multi-resolution molecular surface representation, enabling smooth interactive animation of moderate sized biomolecular complexes consisting of a few hundred thousand to one million atoms, and interactive display of molecular surfaces for multi-million atom complexes, e.g. large virus capsids. The GPU-accelerated QuickSurf representation in VMD achieves performance orders of magnitude faster than the conventional Surf and MSMS representations, and makes VMD the first molecular visualization tool capable of achieving smooth animations of surface representations for systems of up to one million atoms.

Molecular Orbital Display

Visualization of molecular orbitals (MOs) is important for analyzing the results of quantum chemistry simulations. The functions describing the MOs are computed on a three-dimensional lattice, and the resulting data can then be used for plotting isocontours or isosurfaces for visualization as well as for other types of analyses. Existing software packages that render MOs perform calculations on the CPU and require runtimes of tens to hundreds of seconds depending on the complexity of the molecular system.

We have developed present novel data-parallel algorithms for computing MOs on modern graphics processing units (GPUs) using CUDA. As recently reported, the fastest GPU algorithm achieves up to a 125-fold speedup over an optimized CPU implementation running on one CPU core. We have implemented these algorithms within the popular molecular visualization program VMD, which can now produce high quality MO renderings for large systems in less than a second, and achieves the first-ever interactive animations of quantum chemistry simulation trajectories using only on-the-fly calculation.


Ion Placement

To best reproduce physiological conditions, molecular dynamics simulations must be run in the presence of appropriate ions. Generally such simulations are performed in the presence of sodium chloride, although in some cases (such as simulations including nucleic acid) other ions such as magnesium are necessary. Although many tools such as the VMD Autoionize plugin can place a random distribution of ions, molecules requiring counterions for their stability are better treated using ion placement methods which take the electrostatics of the solute into account. One method for doing this is to place important counterions at minima in the electrostatic potential field generated by the biomolecule of interest, iteratively updating the potential field after each ion is placed.

While this method of ion placement is simple and computes ion positions matched to the specific target molecule, it can be very computationally demanding for large structures because it requires calculation of the electrostatic potential at all points on a high-resolution 3-D lattice in the neighborhood of the solute. Coulomb-based ionization of very large structures such as viruses could require several days even using moderately sized clusters of computers. However, the calculation of a function on a lattice where all points are independent is an ideal application for GPU acceleration, and as recently reported in the Journal of Computational Chemistry, the use of GPUs to accelerate Coulomb-based ion placement leads to speedups of 100 times or more, allowing large structures to be properly ionized in less than an hour on a single desktop computer.

GPU accelerated ion placement for large bacterial ribosome and STMV virus structures

The direct summation of the Coulomb potential from all atoms to every lattice point requires computational work that grows quadratically, proportional to the product of the number of atoms and the number of lattice points. An algorithmic enhancement known as multilevel summation uses hierarchical interpolation of softened pairwise potentials from lattices of increasing coarseness to compute an approximation to the Coulomb potential. The amount of computational work for multilevel summation grows linearly, proportional to the sum of the number of atoms and the number of lattice points. Our reported GPU-assisted implementation of this method further reduces the time of obtaining large ionized structures to just a few minutes on a single desktop computer. The accuracy of the implementation is sufficient (with an average difference from the direct approach demonstrated to be in the range of 0.025% to 0.037%) to permit identical ion placement as the direct summation approach for small test molecules and nearly identical results for the ribosome.

The GPU-accelerated Coulomb potential calculation can be directly applied to calculate time-averaged electrostatic potentials from molecular dynamics simulations. As we reported, a VMD calculation of the electrostatic potential for one frame of a molecular dynamics simulation of the ribosome takes 529 seconds on a single GPU, as opposed to 5.24 hours on a single CPU core. A multilevel summation calculation for a single frame requires 67 seconds on one GPU.


Multi-GPU Coulomb Summation

Just as scientific computing can be done on clusters composed of a large number of CPU cores, in some cases problems can be decomposed and run in parallel on multiple GPUs within a single host machine, achieving correspondingly higher levels of performance. One of the drawbacks to the use of multi-core CPUs for scientific computing has been the limited amount of memory bandwidth available to each CPU socket, often severely limiting the performance of bandwidth-intensive scientific codes. Recently this problem has been further exacerbated since the memory bandwidth available to each CPU socket hasn't kept pace with the increasing number of cores in current CPUs. Since GPUs contain their own on-board high performance memory, the available memory bandwidth available for computational kernels scales as the number of GPUs is increased. This property can allow single-system multi-GPU codes to scale much better than their multi-core CPU based counterparts. Highly data-parallel and memory bandwidth intensive problems are often excellent candidates for such multi-GPU performance scaling.

The direct Coulomb summation algorithm implemented in VMD is an exemplary case for multi-GPU acceleration. The scaling efficiency for direct summation across multiple GPUs is nearly perfect -- the use of 4 GPUs delivers almost exactly 4X performance increase. A single GPU evaluates up to 39 billion atom potentials per second, performing 290 GFLOPS of floating point arithmetic. With the use of four GPUs, total performance increases to 157 billion atom potentials per second and 1.156 TFLOPS of floating point arithmetic, for a multi-GPU speedup of 3.99 and a scaling efficiency of 99.7%, as recently reported. To match this level of performance using CPUs, hundreds of state-of-the-art CPU cores would be required, along with their attendant cabling, power, and cooling requirements. While only one of the first steps in our exploration of the use of multiple GPUs, this result clearly demonstrates that it is possible to harness multiple GPUs in a single system with high efficiency.


Fluorescence Microphotolysis

Fluorescence microphotolysis is a non-invasive method of studying dynamics of cellular components using optical microscopy. In its framework, a small area of a fluorescent specimen is illuminated by a focused laser beam, and the fluorescence of the illuminated spot is recorded. Analyzing the change of the fluorescence signal with time, one can extract diffusion constants of the fluorescent molecules. However, such an analysis of experimental data often requires numerical calculations, namely, a diffusion-reaction equation (a partial differential equation in time and 2D or 3D space) has to be solved. Numerical schemes for solving this equation on a grid feature a significant degree of parallelism; indeed, the scheme can be represented as a vector-matrix multiplication problem, which is common for graphics applications and can easily be computed on a GPU. On the other hand, the computation of the fluorescent molecules concentration at a given point depends on the concentration at other points, introducing interdependencies that limit parallelism. Nevertheless, it has been demonstrated recently that one can achieve a significant speed-up with the GPU-accelerated computation of the fluorescence microphotolysis signals, as compared to the CPU computation. The computation that took about 8 minutes on a CPU has been shown to run in 38 seconds on a GPU. Given that experimentalists need to perform multiple computation runs with various parameters to match the observed fluorescence signals, this 12-times speed-up is very welcome. As we reported, the GPUs accelerated computation of fluorescence measurements opens new possibilities for experiments that employ new high-resolution microscopes (such as the so-called 4Pi microscope), because, due to the intricate pattern of light distribution in such microscopes, numerical solution is necessary to analyze experimental data. Further information on this topic is available here.

Software

Book Chapters

"Interactive Ray Tracing Techniques for High-Fidelity Scientific Visualization"
John E. Stone.
In, Eric Haines and Tomas Akenine-Möller, editors, Ray Tracing Gems, Apress, Chapter 27, pp. 493-515, 2019.
Book home pages: Apress | Raytracinggems.com
"A Planetarium Dome Master Camera"
John E. Stone.
In, Eric Haines and Tomas Akenine-Möller, editors, Ray Tracing Gems, Apress, Chapter 4, pp. 49-60, 2019.
Book home pages: Apress | Raytracinggems.com
"Omnidirectional Stereoscopic Projections for VR"
John E. Stone.
In, William R. Sherman, editor, VR Developer Gems, Taylor and Francis / CRC Press, Chapter 24, 2019.
Book home pages: Amazon
"Application Case Study — Molecular Visualization and Analysis"
John E. Stone.
In, David Kirk, Wen-mei Hwu, Programming Massively Parallel Processors: A Hands-on Approach (Third Edition), Morgan Kaufmann, Chapter 15, pp. 331-344, Cambridge, MA, 2017.
Book home pages: Amazon | Elsevier | Online full text at ScienceDirect
"GPU-Accelerated Molecular Dynamics Clustering Analysis with OpenACC"
John E. Stone, Juan R. Perilla, C. Keith Cassidy, and Klaus Schulten.
In, Robert Farber, editor, Parallel Programming with OpenACC, Morgan Kaufmann, pp. 215-240, Cambridge, MA, 2016.
Book home pages: Amazon | Elsevier
"GPU-Accelerated Computation and Interactive Display of Molecular Orbitals"
John E. Stone, David J. Hardy, Jan Saam, Kirby L. Vandivort, and Klaus Schulten.
In, Wen-Mei Hwu, editor, GPU Computing Gems, Chapter 1, pp. 5-18, 2011.
Book home pages: Amazon | Elsevier
"Fast Molecular Electrostatics Algorithms on GPUs"
David J. Hardy, John E. Stone, Kirby L. Vandivort, David Gohara, Christopher Rodrigues, and Klaus Schulten.
In, Wen-Mei Hwu, editor, GPU Computing Gems, Chapter 4, pp. 43-58, 2011.
Book home pages: Amazon | Elsevier
"GPU Algorithms for Molecular Modeling"
John E. Stone, David J. Hardy, Barry Isralewitz, and Klaus Schulten.
In Jack Dongarra, David A. Bader, and Jakub Kurzak editors, Scientific Computing with Multicore and Accelerators, Chapman & Hall / CRC Press, Chapter 16, pp. 351-371, 2010.
Book home pages: Amazon | CRC Press

Publications

AltStyle によって変換されたページ (->オリジナル) /

  • The OLCF GPU Hackathon Series: The Story Behind Advancing Scientific Applications with a Sustained Impact. Sunita Chandrasekaran, Guido Juckeland, Meifeng Ling, Matthew Otten, Dirk Pleiter, John E. Stone, Juan Lucio-Vega, Michael Zingale, and Fernanda Foertter. The 2017 Workshop on Education for High Performance Computing (EduHPC'17), 2017.
  • Challenges of Integrating Stochastic Dynamics and Cryo-electron Tomograms in Whole-cell Simulations. Tyler M. Earnest, Reika Watanabe, John E. Stone, Julia Mahamid, Wolfgang Baumeister, Elizabeth Villa, and Zaida Luthey-Schulten. J. Physical Chemistry B, 121(15): 3871-3881, 2017. Publications Database
  • Early experiences porting the NAMD and VMD molecular simulation and analysis software to GPU-accelerated OpenPOWER platforms. John E. Stone, Antti-Pekka Hynninen, James C. Phillips, and Klaus Schulten. International Workshop on OpenPOWER for HPC (IWOPH'16), pp. 188-206, 2016.
  • Atomic detail visualization of photosynthetic membranes with GPU-accelerated ray tracing. John E. Stone, Melih Sener, Kirby L. Vandivort, Angela Barragan, Abhishek Singharoy, Ivan Teo, Joao V. Ribeiro, Barry Isralewitz, Bo Liu, Boon Chong Goh, James C. Phillips, Craig MacGregor-Chatwin, Matthew P. Johnson, Lena F. Kourkoutis, C. Neil Hunter, and Klaus Schulten. Parallel Computing, 55:17-27, 2016.
  • Immersive molecular visualization with omnidirectional stereoscopic ray tracing and remote rendering. John E. Stone, William R. Sherman, and Klaus Schulten. 2016 IEEE International Parallel and Distributed Processing Symposium Workshop (IPDPSW), pp. 1048-1057, 2016.
  • High performance molecular visualization: In-situ and parallel rendering with EGL. John E. Stone, Peter Messmer, Robert Sisneros, and Klaus Schulten. 2016 IEEE International Parallel and Distributed Processing Symposium Workshop (IPDPSW), pp. 1014-1023, 2016.
  • Evaluation of emerging energy-efficient heterogeneous computing platforms for biomolecular and cellular simulation workloads. John E. Stone, Michael J. Hallock, James C. Phillips, Joseph R. Peterson, Zaida Luthey-Schulten, and Klaus Schulten. 2016 IEEE International Parallel and Distributed Processing Symposium Workshop (IPDPSW), pp. 89-100, 2016.
  • Chemical visualization of human pathogens: the Retroviral Capsids. Juan R. Perilla, Boon Chong Goh, John Stone, and Klaus Schulten. Proceedings of the 2015 ACM/IEEE Conference on Supercomputing, 2015. (4 pages).
  • Multilevel summation method for electrostatic force evaluation. David J. Hardy, Zhe Wu, James C. Phillips, John E. Stone, Robert D. Skeel, and Klaus Schulten. Journal of Chemical Theory and Computation, 11:766-779, 2015.
  • Runtime and Architecture Support for Efficient Data Exchange in Multi-Accelerator Applications, Javier Cabezas, Isaac Gelado, John E. Stone, Nacho Navarro, David B. Kirk, and Wen-mei Hwu. IEEE Transactions on Parallel and Distributed Systems, 26(5):1405-1418, 2015. Publications Database
  • Visualization of energy conversion processes in a light harvesting organelle at atomic detail. Melih Sener, John E. Stone, Angela Barragan, Abhishek Singharoy, Ivan Teo, Kirby L. Vandivort, Barry Isralewitz, Bo Liu, Boon Chong Goh, James C. Phillips, Lena F. Kourkoutis, C. Neil Hunter, and Klaus Schulten. In Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis, SC '14. IEEE Press, 2014. (4 pages).
  • GPU-accelerated analysis and visualization of large structures solved by molecular dynamics flexible fitting. John E. Stone, Ryan McGreevy, Barry Isralewitz, and Klaus Schulten. Faraday Discussions, 169:265-283, 2014.
  • Simulation of reaction diffusion processes over biologically relevant size and time scales using multi-GPU workstations. Michael J. Hallock, John E. Stone, Elijah Roberts, Corey Fry, and Zaida Luthey-Schulten. Parallel Computing, 40:86-99, 2014.
  • Unlocking the full potential of the Cray XK7 accelerator. Mark D. Klein and John E. Stone. In Cray User Group Conference. Cray, May 2014.
  • GPU-accelerated molecular visualization on petascale supercomputing platforms. John E. Stone, Kirby L. Vandivort, and Klaus Schulten. In Proceedings of the 8th International Workshop on Ultrascale Visualization, UltraVis '13, pp. 6:1-6:8, New York, NY, USA, 2013. ACM.
  • Early experiences scaling VMD molecular visualization and analysis jobs on Blue Waters. John E. Stone, Barry Isralewitz, and Klaus Schulten. In Extreme Scaling Workshop (XSW), 2013, pp. 43-50, Aug. 2013.
  • Lattice microbes: High-performance stochastic simulation method for the reaction-diffusion master equation. Elijah Roberts, John E. Stone, and Zaida Luthey-Schulten. Journal of Computational Chemistry, 34:245-255, 2013.
  • Fast visualization of Gaussian density surfaces for molecular dynamics and particle system trajectories. Michael Krone, John E. Stone, Thomas Ertl, and Klaus Schulten. In EuroVis - Short Papers 2012, pp. 67-71, 2012.
  • Fast analysis of molecular dynamics trajectories with graphics processing units-radial distribution function histogramming. Benjamin G. Levine, John E. Stone, and Axel Kohlmeyer. Journal of Computational Physics, 230:3556-3569, 2011.
  • Immersive molecular visualization and interactive modeling with commodity hardware. John E. Stone, Axel Kohlmeyer, Kirby L. Vandivort, and Klaus Schulten. Lecture Notes in Computer Science, 6454:382-393, 2010.
  • Quantifying the impact of GPUs on performance and energy efficiency in HPC clusters. Jeremy Enos, Craig Steffen, Joshi Fullop, Michael Showerman, Guochun Shi, Kenneth Esler, Volodymyr Kindratenko, John E. Stone, and James C. Phillips. In International Conference on Green Computing, pp. 317-324, 2010.
  • GPU-accelerated molecular modeling coming of age. John E. Stone, David J. Hardy, Ivan S. Ufimtsev, and Klaus Schulten. Journal of Molecular Graphics and Modelling, 29:116-125, 2010.
  • OpenCL: A parallel programming standard for heterogeneous computing systems. John E. Stone, David Gohara, and Guochun Shi. Computing in Science and Engineering, 12:66-73, 2010.
  • An Asymmetric Distributed Shared Memory Model for Heterogeneous Parallel Systems. Isaac Gelado, John E. Stone, Javier Cabezas, Sanjay Patel, Nacho Navarro, and Wen-mei W. Hwu. ASPLOS '10: Proceedings of the 15th International Conference on Architectural Support for Programming Languages and Operating Systems, pp. 347-358, 2010.
  • GPU Clusters for High Performance Computing. Volodymyr Kindratenko, Jeremy Enos, Guochun Shi, Michael Showerman, Galen Arnold, John E. Stone, James Phillips, Wen-mei Hwu. Cluster Computing and Workshops, 2009. CLUSTER '09. IEEE International Conference on. pp. 1-8, Aug. 2009.
  • Long time-scale simulations of in vivo diffusion using GPU hardware. Elijah Roberts, John E. Stone, Leonardo Sepulveda, Wen-mei W. Hwu, and Zaida Luthey-Schulten. In IPDPS '09: Proceedings of the 2009 IEEE International Symposium on Parallel & Distributed Processing, pp. 1-8, 2009 Publications Database
  • Probing biomolecular machines with graphics processors. James C. Phillips and John E. Stone. Communications of the ACM, 52:34-41, 2009.
  • High performance computation and interactive display of molecular orbitals on GPUs and multi-core CPUs. John E. Stone, Jan Saam, David J. Hardy, Kirby L. Vandivort, Wen-mei W. Hwu, and Klaus Schulten. In Proceedings of the 2nd Workshop on General-Purpose Processing on Graphics Processing Units, ACM International Conference Proceeding Series, volume 383, pp. 9-18, New York, NY, USA, 2009. ACM.
  • Multilevel summation of electrostatic potentials using graphics processing units. David J. Hardy, John E. Stone, and Klaus Schulten. Journal of Parallel Computing, 35:164-177, 2009.
  • Adapting a message-driven parallel application to GPU-accelerated clusters. James C. Phillips, John E. Stone, and Klaus Schulten. In SC '08: Proceedings of the 2008 ACM/IEEE Conference on Supercomputing, Piscataway, NJ, USA, 2008. IEEE Press. (9 pages).
  • GPU acceleration of cutoff pair potentials for molecular modeling applications. Christopher I. Rodrigues, David J. Hardy, John E. Stone, Klaus Schulten, and Wen-mei W. Hwu. In CF'08: Proceedings of the 2008 Conference on Computing Frontiers, pp. 273-282, New York, NY, USA, 2008. ACM.
  • GPU computing. John D. Owens, Mike Houston, David Luebke, Simon Green, John E. Stone, and James C. Phillips. Proceedings of the IEEE, 96:879-899, 2008.
  • Continuous fluorescence microphotolysis and correlation spectroscopy using 4Pi microscopy. Anton Arkhipov, Jana Hüve, Martin Kahms, Reiner Peters, and Klaus Schulten. Biophysical Journal, 93:4006-4017, 2007.
  • Accelerating molecular modeling applications with graphics processors. John E. Stone, James C. Phillips, Peter L. Freddolino, David J. Hardy, Leonardo G. Trabuco, and Klaus Schulten. Journal of Computational Chemistry, 28:2618-2640, 2007.

    • Presentations

    Class lectures, workshop materials, and sample source code:

    Investigators

    Collaborators

    Our Research in the News