Web Release | Division of Theoretical Astronomy, National Astronomical Observatory of Japan

The world’s most extensive simulation of the Milky Way galaxy

2014年12月16日 Released

Abstract

An international research team including Jeroen Bédorf, Simon Portegies Zwart (Leiden University, the Netherlands), Michiko Fujii (NAOJ, Japan) and others developed a new application "Bonsai" for numerical simulations of galaxies. Using this, they performed the world’s most extensive simulation of the Milky Way galaxy using 18600 GPUs¹. Around 240 billion particles are used in this simulation, and it provides detailed enough data for direct comparison to the observed stars in the Milky Way. In addition, the simulation achieved a computational speed of 24.77 Pflops (Peta Floating-point Operations Per Second), which is currently the world record. Because of these results, the research team is one of five finalists nominated for this year’s Gordon Bell Prize.

Fig 1: Face-on view of a simulated galaxy. Green grids show the domain decomposition for the gravity calculation. (Credit: Bédorf et al. (2014))

Research team

Jeroen Bédorf (Leiden University, Centrum Wiskunde & Informatica)

Evghenii Gaburov (SURFsara)

Michiko S. Fujii (Division of Theoretical Astronomy, National Astronomical Observatory of Japan)

Keigo Nitadori (Advanced Institute for Computational Science, RIKEN)

Tomoaki Ishiyama (Center for Computational Sciences, University of Tsukuba)

Simon Portegies Zwart (Leiden University)

Paper

Jeroen Bédorf, Evghenii Gaburov, Michiko S. Fujii, Keigo Nitadori, Tomoaki Ishiyama, Simon Portegies Zwart, "24.77 Pflops on a Gravitational Tree-Code to Simulate the Milky Way Galaxy with 18600 GPUs", 2014 ACM/IEEE conference on Supercomputing (SC14), New Orleans, Louisiana, USA, Nov. 2014
http://dl.acm.org/citation.cfm?id=2683600

Research Background

Our Solar System is located in the disk of the Milky Way galaxy. Since we live in the disk, we can only see the edge-on view of the disk. Unfortunately, we cannot yet travel out of the Milky Way and look back to see the entire galaxy at once. However, we can determine the structure of the Milky Way, such as the presence of a bar and spiral arms, from measuring the positions and velocities of stars. The Gaia spacecraft was just launched by the European Space Agency (ESA) in 2013; in Japan, the JASMINE satellite is also planned. These satellites will measure the precise positions and velocities of stars in the Milky Way, providing us with a huge amount of data in the near future. In order to understand such observational data, we have to understand the relation between the motions of individual stars and the structure of the Milky Way. Simulations are expected to clarify this relation. The comparison of simulations to observations is expected to yield a better understanding of the structure of the Milky Way.

Fig 2: Fig 2: UGC12158 (barred galaxy) observed by the Hubble Space Telescope. This galaxy has a shape similar to the Milky Way, which also has a central bar and surrounding spiral arms. (Credit: ESA/NASA)

The number of stars in the Milky Way disk is estimated to be 200 – 400 billion. For simulations of galaxies, the gravitational interactions between all of the stars and dark matter (which surrounds the galactic disk) must be calculated. Then the motions of all the stars can be obtained. This process requires a huge amount of computational cost. Previously, galaxies have been modeled by grouping the stars into "particles" containing roughly 1000 stars each. However, these particle simulations do not provide the spatial resolution needed to compare to observations. In order to interpret the data expected in the near future, more detailed simulations of the Milky Way are needed.

The world’s most extensive simulation of the Milky Way galaxy using GPUs

The research team has performed the world’s most extensive simulation of the Milky Way galaxy using the supercomputers Piz Daint (The Swiss National Supercomputing Centre) and Titan (Oak Ridge National Laboratory), which are CPU²-GPU hybrid supercomputers. This simulation used the maximum available, 18600 GPUs. The research team made a great effort to use this large number of GPUs efficiently. In general, the calculation efficiency decreases as the number of GPUs increases because of the burden of communicating between the individual GPUs. In other words, even if we use twice as many as GPUs, the calculation speed does not double. The new application, "Bonsai," developed by the research team solved this problem by sharing the work between the CPUs and GPUs. In "Bonsai," GPUs are used for calculations and CPUs are used for communications. In this way the calculations and communications can be conducted simultaneously. This method enabled high efficiency even for 18600 GPUs, allowing this simulation to achieve the effective computational speed of 24.77 Pflops (24.77 million-billion floating-point operations per second), the current world record.
The Milky Way simulation performed by the team handled 20 billion particles representing groups of stars. Including dark matter particles, the total number reached 242 billion particles. This is 100 to 1000 times more particles than in previous simulations, making it the first simulation to provide enough data to enable a direct comparison to observations.

[フレーム]

Movie:Visualization of this simulation. (Credit : SURFsara, Bédorf et al. 2014 and NVIDIA)

Figure 3: (Top) Simulated Milky Way galaxy. Stars are shown in blue and white; dark matter is not shown. Galactic structures such as spiral arms are seen. (Credit : SURFsara, Bédorf et al. 2014 and NVIDIA) (Bottom) Velocity maps obtained from previous simulations (a), this research (b), and observations (c). Horizontal and vertical axes indicate the radial and tangential velocities in the Milky Way disk. Panel (b) shows the result obtained from one of this project’s simulations using 50 billion particles (500 times more particles than previous research.) Stellar velocity patterns are much clearer than in previous simulations (a). Similar stellar velocity patterns are also seen in observations (c). (Credit : Fig(a,b): Bédorf et al. (2014), Fig(c): T. Antoja et al., "Kinematic groups beyond the solar neighbourhood with RAVE", MNRAS (2012), 426(1):L1-L5, Figure 2.(a))

Nomination for the Gordon Bell Prize

The research team was nominated as one of the five finalists for this year’s Gordon Bell Prize, because of the high performance of this simulation using a large number of GPUs and the scientific importance of the Milky Way simulation.

Michiko Fujii, NAOJ Fellow in the Division of Theoretical Astronomy NAOJ, worked on the modeling of the simulated Milky Way and also on the data analysis. She chose the best model for this simulation, taking into account her own recent research results, and evaluated the results of the simulation. She is now working on the data analysis of the results in order to compare them to observations. From Japan, Keigo Nitadori (RIKEN AICS) and Tomoaki Ishiyama (University of Tsukuba) also contributed, especially for developing the application. In the Netherlands, Simon Portegies Zwart (Leiden University, team leader), Jeroen Bedorf (Leiden University/CWI), and Evghenii Gaburov (SURFsara) contributed to the project.

Michiko Fujii, NAOJ Fellow at Division of Theoretical Astronomy, NAOJ Michiko Fujii says "I was in Leiden University until September of 2013. I have worked on this project since last year. This is our second attempt for the Gordon Bell prize, and this year we were able to become a finalist for the first time. The scale of supercomputers in the Netherlands is not as large as the scale in Japan. It was a big problem for us to test our code using many GPUs. We therefore developed our code using supercomputers in Japan such as HA-PACS (GPU supercomputer) in the University of Tsukuba and ATERUI (CPU supercomputer) in NAOJ. The chances we had to use Titan, with which we performed the largest simulation, were also very limited. We are very happy to be nominated as a finalist after overcoming these difficulties. We still have to work on the data analysis. We hope that our results will become a big step in the study of the Milky Way."

Future Prospects

Compared with previous simulations, this simulation handled a large number of star particles and therefore it enables a more detailed comparison between simulations and observations. The research team expects that the comparison between this simulation and the observation by Gaia will bring a new understanding of structures in the Milky Way and their formation processes. The results of the data analysis will be released in the near future.

This work was supported in part by MEXT SPIRE Field 5 and JICFuS.

Notes:
1) GPU (Graphics Processing Unit) is auxiliary processors specialized for computing graphics. Because graphics also require many floating-point operations, like those used in gravity simulations, GPUs provide better performance for simulations than general purpose CPUs.
2) CPU (Central Processing Units) is the centralized, general purpose processors of a computer.

Use Policy of Images and Videos:
Contents are to be used with a clear indication of credits.
If you would like to use images found on this web site, please follow "Terms of Use of the Website of NAOJ".

Contact
PR Office of DTA
Phone: +81-422-34-3836
Email: fukushi.hinako ATM nao.ac.jp (ATM→@)