Fig. 1

3.1.

JAXA Supercomputer System Generation 3

The JAXA Supercomputer System Generation 3 (JSS3),⁶ which began operating in 2020, is the infrastructure for numerical simulations and large-scale data analyses. It consists of

There is also a 0.4-PB storage system called TOKI-TFS, which connects to TOKI-TRURI. For TOKI-SORA and TOKI-RURI, TOKI-FS serves as the sole hot storage; all input and output files must reside in TOKI-FS. It should be noted that these systems are completely isolated from the Internet. To use them, one should log in to one of the login nodes through a virtual private network (VPN). Furthermore, only SSH communication from the Internet to the login node is permitted. The firewall prohibits communication in the reverse direction (see the bottom half of Fig. 2).

TOKI-RURI is the best choice as each node can be considered a standard personal computer (PC) running Linux. TOKI-RURI is further divided into four types as summarized in Table 1. Thus, the TOKI-RURI system has 14,976 physical CPU cores and a total of 104.8 TiB of memory; MPI enables parallel computations across multiple nodes, even if they are of different types. The login nodes for TOKI-RURI consist of eight servers, each equipped with dual Intel Xeon Gold 6240 processors, 384 GiB of memory, and one NVIDIA Quadro P4000. These login nodes are also used for small-scale testing and debugging. The sequential write to an HDD region in TOKI-FS on a login node is $\simeq 140\mathrm{MB}/\mathrm{s}$.

Table 1

Specifications of the TOKI-RURI system.

To execute commands (or programs) in parallel on TOKI-RURI, users need to write and submit a bash script, often referred to as a "job script," that defines necessary computing resources such as the number of CPU cores, memory capacity, and computation time (Code 1). By considering the resource requirements outlined in the job script and ensuring fairness among users, a job scheduler allocates the appropriate computing nodes from the four types of TOKI-RURI listed in Table 1 and determines the execution timing (fair-share scheduling). This ensures that the job runs without being affected by performance degradation caused by other jobs. However, the job will be deferred when sufficient resources are not available at that moment. This implies that the resource requirements of the job script must always exceed the actual usage for the task. If the actual usage slightly exceeds that defined in the job script, the job is forcibly terminated. In addition, the user does not know exactly when the job will start ($\simeq 20$ jobs started execution immediately after submitting the job scripts in our case). Unless an exception request for special circumstances is submitted, the computing time is limited to 24 h.

Because PPL does not have parallel execution parts, we can reliably request a single CPU core as a constant value for the job script. To adapt PPL for TOKI-RURI, we need to derive formulas that estimate the memory capacity and computing time required for each OBSID based on its observation duration and prepare a metascript to incorporate these values into a job script template.

3.2.

Virtualization Technologies on Linux and Singularity

VMs such as the Oracle VirtualBox¹⁵ are among the most well-known VTs. VMs were invented in the late 1960s and are categorized into type 1 and 2 hypervisors.¹⁶ The former runs directly on PC hardware, whereas the latter runs on a host operating system (OS). Both hypervisors can emulate commonly used PC hardware to run arbitrary OSes. Although PC users can simultaneously execute both Linux and Windows applications, for instance, on their laptops by employing VMs, running multiple VMs on the respective high-performance servers increases their density in a data center (Reformatter is the case), thus maximizing the hardware resource usage. In return, there are overheads due to hardware emulations, which are significant in file input/output (I/O) to virtual disks.

Another VT isolates a process, an execution unit of programs, and its subprocesses from the others on an OS. It originates from the chroot command/system call of Version 7 Unix, released in 1979, leading to the jail mechanism of FreeBSD introduced in 2000.¹⁷ The same feature was implemented as a set of patches, "Linux-VServer," to the Linux kernel, announced in 2001.¹⁸ Regarding Linux, the process isolation technology known today as "containers" is based on the two features of the Linux kernel cgroups¹⁹ and namespaces.²⁰ The former hierarchically groups the respective processes and limits their hardware usage, such as CPU time, I/O bandwidth, and memory capacity, and the latter constrains their visibilities of software resources such as mount points, hostname, and user/group IDs. As the program inside a container instance runs directly on the host kernel, there is marginal overhead, which is an advantage over VMs. On the other hand, note that a container on a Linux system accepts only Linux programs, unlike VMs.

Container platforms, such as Docker²¹ and Singularity (and its descendant Apptainer²²), wrap the cgroups and namespaces into an easy-to-use form and package all necessary software into a container image by resolving their mutual dependencies. Both enable users to build their original container image from a seed image on Docker Hub,²³ which covers major Linux distributions and creates package management tools such as apt, dnf, pacman, and zypper commands available inside the container instance. Hence, VMs and containers are similar in terms of practical usage. Because a Docker instance occupies its image, we can create just one instance from one Docker image; Docker is frequently used to develop and deploy server applications. Singularity mounts a container image as a read-only image, which means that multiple instances can be spawned from a single container image. Singularity is often used in HPC systems, and its Version 3.10 is also available in TOKI-RURI.

3.3.

Obstacles to the HPC Version of PPL and the Key to the Solution

In the early design phase of PPL, we fixed all software versions, including the operating system (RedHat Enterprise Linux 7 with glibc 2.17), compilers (GCC 4.8 and 8), and scripting languages (Perl 5.16, Python 3.8, and Ruby 2.6). We also decided that it should only run on Reformatter and is not guaranteed to work in any other environment. Although these compromises in the PPL specifications clarified and simplified the development goals, the latest version of PPL, its related software, and the auxiliary files have been deployed in a subdirectory named after the eight-digit release date in the appropriate directory and identified as "latest" by a symbolic link (note that we mention deployment here, and the source codes are naturally version-controlled by Git). There were also cross-references between the directories through symbolic links. Although everything worked perfectly inside Reformatter, these tricks seemed to hinder the execution of PPL on the TOKI-RURI system because it was time consuming to manually build all the software that PPL depends on and to fit PPL into the system’s directory structure.

However, based on our knowledge of containers and their technological backgrounds, we provide an elegant solution for porting PPL to TOKI-RURI: manipulating the mount namespace and creating everything inside a container instance identical to Reformatter. If this is possible, all we have to do is copy the binaries compiled on Reformatter into a container image and other materials onto TOKI-FS; Singularity accepts the "--bind" option. For example,

3.4.

Estimating Computing Resources on Reformatter

During the commissioning period, we needed to profile PPL on Reformatter. Because no profilers were installed there and we did not have root privileges for Reformatter, we wrote a small profiler in Python. The uses and functions of the script are as follows.

We start two terminals and execute the top-level script of PPL in one of them. Immediately afterward, we run the ps command in the other terminal to find the process ID of the top-level script and execute the profiler by specifying the process ID as an argument. The profiler runs the following shell one-liner every minute, where pid_parent represents the process ID of the top-level script: Obtain a list of all the descendant processes’ process IDs

pstree -p (pid_parent) | perl -ne 'print "1ドル " while /\((\d+)\)/g'

Next, the profiler immediately executes the following commands to obtain a list of each process’s physical memory usage (resident set size; RSS), reserved memory size (virtual memory size; VSZ), and command name with its arguments. The variable pid_list corresponds to a comma-separated list of the aforementioned process IDs

ps -p (pid_list) -o pid,rss,vsz,command

The profiler appends these values and the time required for each process to a log file as tab-delimited text (one line for each process). The profiler also records the totals of RSS and VSZ at that moment as the consumption of a virtual "TOTAL" command (the name has no other meaning than as a flag) in the log file. When the top-level script terminates, the profiler loads the log file, groups the data using the command names, computes the maxima of RSS and VSZ for each command, and stores the results in a summary file. We consider the maximum VSZ value of the "TOTAL" command in the summary file to represent the memory consumption of PPL; we define the execution time of PPL as the last time recorded in the log file minus the first time.

We selected five OBSIDs that were likely to cover a range of future observation durations and measured the execution time ($T_{\mathrm{Reformatter}}$) and memory usage ($C_{\mathrm{Reformatter}}$) of PPL for each OBSID using the aforementioned method. Note that Reformatter is noisy regarding these measurements because programs related to XRISM other than PPL are running. By linear fitting, we obtained

C-SODA provides a program in the PPL chain that converts telemetry packets into raw values in a FITS file. We observed its anomalous behavior in 2020. An analysis using Valgrind revealed that the initial memory size and increment were not optimized for the typical size of an XRISM RPT file, wasting approximately half the execution time of the program ($\simeq 400\mathrm{s}$) and $\simeq 4\mathrm{GiB}$ of memory. Although we reported this issue to C-SODA, which was later resolved, the update was not reflected in PPL because we had already fixed the software version at that time and did not require the absolute performance. Considering that the program is called multiple times in the PPL chain, the constants in Eqs. (1) and (2) can be attributed to this overhead.

3.5.

Porting Strategies

Figures 2 and 3 illustrate the schematic network setup and sequence diagram of the HPC version of PPL, respectively. Reformatter has a $\simeq 90\mathrm{MB}/\mathrm{s}$ (sequential write) network-attached storage (NAS) that stores all input and output data. We created a directory on the NAS to accumulate all the materials necessary for PPL processing on TOKI-RURI (the "delivery directory" hereafter). Because TOKI-RURI cannot communicate with SIRIUS, we wrote a small patch (25 lines) to PPL that pauses the PPL processing when the corresponding RPT file is generated on Reformatter and resumes the remaining processing on TOKI-RURI.

Because we do not have root privileges for Reformatter, we created a container image on a PC other than Reformatter, where Ubuntu 24.04 is running and Version 3.10.5 of the Singularity Community Edition is installed to match the version with TOKI-RURI. In the definition file, a recipe for creating the container image, a seed of CentOS 7 on Docker Hub is obtained, and all software updates are applied; all software packages installed on Reformatter with the yum command are also installed similarly on the container. At the end of the definition file, the binaries of the patched PPL built on Reformatter are copied to the container image, and all mount points for the "--bind" option are created Code 2). The image ($\simeq 7$ GiB) is transferred to the delivery directory on Reformatter via the scp command.

On Reformatter, the time calibration table, orbit, and attitude files are copied to the delivery directory. A small Python script reads the list of OBSIDs, sequentially creates the respective RPT files, and copies them to the delivery directory. The directory is archived into a tarball with Zstandard compression ($\sim \mathrm{TB}$) and copied to a TOKI-RURI login node using the rsync command, allowing file transfer to resume in case of unexpected network interruptions. The bandwidth of rsync was $\simeq 50\mathrm{MiB}/\mathrm{s}$.

When the file transfer is completed, it is manually extracted into the working directory of TOKI-FS (not SSDs but HDDs). As described in Sec. 3.1, we require a job script that defines the computing resources necessary to run the containerized PPL as a batch job on TOKI-RURI. The key point here is not that the estimation is correct but rather that the actual resource consumption does not exceed the estimation. By considering the difference in the CPU clock frequencies between Reformatter and TOKI-RURI, the overhead of the Lustre FS (Sec. 4.3), and margins, we estimate the computing time $T_{\mathrm{TOKI}}$ and memory requirement $C_{\mathrm{TOKI}}$ on TOKI-RURI as follows:

A Python script reads the list of OBSIDs with their observation start and end times, computes the respective $(T_{\mathrm{TOKI}},C_{\mathrm{TOKI}})$, builds the argument for the "--bind" option, which gives the mappings between the native FS and inside the container on TOKI-RURI, and writes them into the corresponding job scripts. The Python script automatically submits these job scripts. When all jobs are completed, the resultant FFFs and log files are manually archived into a tarball using Zstandard compression. The tarball is sent back to Reformatter with the rsync command and extracted into a specified directory on Reformatter.

4.1.

Reprocessing with HPC PPL Version 1

In March 2024, we reprocessed the 80 OBSIDs observed during the commissioning period using the initial version of the HPC PPL. The results are summarized in the first row of Table 2. It was estimated to take 218.3 h to complete the reprocessing tasks on the Reformatter with no parallelization; the time required to create the RPT files was estimated to be $\simeq 3\%$ of the total. The HPC PPL completed them in 26.3 h (the latest log file update time among all 80 jobs minus the earliest creation time), although it failed to process 11 OBSIDs (Sec. 4.2 for the cause), which were later reprocessed on Reformatter. When we define speedup $S$ as

Table 2

Number of OBSIDs for reprocessing and the statistics.

4.2.

Cause of the Failures

Shortly after completing the reprocessing tasks, we investigated the causes of the failures in detail. We successfully reproduced these errors on a PC for container image creation. To convert the CCSDS packets in an RPT file into raw values, we used a set of programs provided by C-SODA. For some of them, no source code was accessible for SOC, and some of these without the source code were 32-bit binaries. One of these 32-bit programs invokes the others and communicates with them through standard input/output. Because these 32-bit programs are unaware of the 64-bit inode numbers, which is the case for XFS and Lustre FS, they fail to read their configuration files, open the input and output files, and spawn others onto these FSes when their inode numbers exceed the 32-bit range. One of the advantages of containers is that we can perform detailed analyses on our own PCs, in which various debugging tools are installed.

4.3.

Improvements Toward Version 2

Our solution to this problem is straightforward: we used disk images formatted to ext3, where everything was 32-bit, as working areas. We attached two working disk images to each container instance with the "--bind" option: one 64 GiB for the PPL software and another 512 GiB for the RPT and FFFs because the option does not accept the mapping from a disk image on the host onto two different directories inside the container instance. Although copying the PPL software inside the container image onto an ext3 disk image for each container instance would be wasteful, this reduces human errors during container image creation. Because PPL is required to process continuous observations for up to 10 days and we have never experienced such a condition, we could not determine whether 512 GiB is excessive.

Using a disk image as the working area is also expected to improve the I/O performance of the Lustre FS, which consists of MetaData servers (MDSes) and object storage servers (OSSes). When an append operation occurs in an existing file, the MDSes search for adequate blocks among the storage devices mounted by OSSes and update the metadata shared among the MDSes; when small appends occur extensively in a single file, which is the case for log and FITS file manipulations, the performance of Lustre FS is significantly degraded.²⁴ However, when partial changes occur within a file for which space has already been allocated, the computation for block allocations shall be omitted, leading to minimal performance degradation.

The dd command can create a disk image with general user permission, formatting the image to ext3 with the mkfs.ext3 command requires root permission. Hence, we format the images on the local PC and compress them with the xz command with the "?9" option; this reduces the 64 and 512 GiB images to 10 and 77 MiB, respectively. The compressed working disks and container images are transferred to the delivery directory on the NAS attached to Reformatter. When a container instance is invoked on TOKI-RURI, the PPL software inside the container image and the RPT file on the host are copied to their respective ext3 working disks. After the PPL process is completed and immediately before the container instance is terminated, the FFFs as products and log files are synchronized with the host FS. Considering the time required for synchronization, Eq. (3) is modified as

4.4.

Reprocessing with HPC PPL Version 2

In September 2024, soon after the PV period, we reprocessed the 161 OBSIDs observed during the commissioning and PV periods, respectively, using the second version of the HPC PPL. Because running 161 jobs at once could reach the disk quota of the working directory on TOKI-RURI, we submitted them in two batches: 100 + 61; the latter batch was submitted when 90% of the former was completed. The results are summarized in the second row of Table 2. Although it was estimated to take 515.5 h to complete the reprocessing tasks on Reformatter with no parallelization, the HPC PPL completed them within 15.2 h; it is noteworthy that the operator was not constantly monitoring the execution status. The speedup $S$ is

No errors were observed. From this, the following conclusions were drawn:

4.5.

Bottleneck: Bandwidth of the File Transfers

We mentioned file transfers during reprocessing in September. The size of the tarball included all RPT files created on the Reformatter, as well as the Singularity and working disk images, was 1.6 TiB. The size of the tarball containing all the products on TOKI-RURI was 1.2 TiB. When the bandwidth of 50 MiB/s is assumed, we obtain 9.3 and 7.0 h for the file transfers, respectively. Even if these are added to the computing time of TOKI-RURI, we could still conclude that the 3-week jobs were reduced to 2 days. On the other hand, a file transfer with parallel TCP streams using the bbcp²⁵ command will shorten this time. However, these connections are now completely blocked by the firewall on the JSS3 side. Reducing the file transfer time will require coordination between departments within JAXA, which is expected to be realized in the future.

References