-
Notifications
You must be signed in to change notification settings - Fork 34
Running the melt pool 3D example #192
Unanswered
alimostafavi24
asked this question in
Q&A General
Hello,
When I run the melt pool 3D case in MALAMUTE examples, I got the following error on HPC using 92 CPUs. Any suggestion to tackle this issue is highly appreciated.
Thanks,
Ali
�[36mreinit0: �[39m
�[36mreinit0: �[39mCommand Line Argument(s):
�[36mreinit0: �[39m --allow-unused
�[36mreinit0: �[39m
�[36mreinit0: �[39mParallelism:
�[36mreinit0: �[39m Num Processors: 92
�[36mreinit0: �[39m Num Threads: 1
�[36mreinit0: �[39m
�[36mreinit0: �[39mMesh:
�[36mreinit0: �[39m Parallel Type: replicated
�[36mreinit0: �[39m Mesh Dimension: 3
�[36mreinit0: �[39m Spatial Dimension: 3
�[36mreinit0: �[39m Nodes:
�[36mreinit0: �[39m Total: 132651
�[36mreinit0: �[39m Local: 1823
�[36mreinit0: �[39m Min/Max/Avg: 974/1823/1441
�[36mreinit0: �[39m Elems:
�[36mreinit0: �[39m Total: 125000
�[36mreinit0: �[39m Local: 1383
�[36mreinit0: �[39m Min/Max/Avg: 1319/1399/1358
�[36mreinit0: �[39m Num Subdomains: 1
�[36mreinit0: �[39m Num Partitions: 92
�[36mreinit0: �[39m Partitioner: metis
�[36mreinit0: �[39m
�[36mreinit0: �[39mNonlinear System:
�[36mreinit0: �[39m Num DOFs: 530604
�[36mreinit0: �[39m Num Local DOFs: 7292
�[36mreinit0: �[39m Variables: "ls" "grad_ls"
�[36mreinit0: �[39m Finite Element Types: "LAGRANGE" "LAGRANGE_VEC"
�[36mreinit0: �[39m Approximation Orders: "FIRST" "FIRST"
�[36mreinit0: �[39m
�[36mreinit0: �[39mAuxiliary System:
�[36mreinit0: �[39m Num DOFs: 257651
�[36mreinit0: �[39m Num Local DOFs: 3206
�[36mreinit0: �[39m Variables: "ls_0" "box"
�[36mreinit0: �[39m Finite Element Types: "LAGRANGE" "MONOMIAL"
�[36mreinit0: �[39m Approximation Orders: "FIRST" "CONSTANT"
�[36mreinit0: �[39m
�[36mreinit0: �[39mExecution Information:
�[36mreinit0: �[39m Executioner: Transient
�[36mreinit0: �[39m TimeStepper: ConstantDT
�[36mreinit0: �[39m TimeIntegrator(s): ImplicitEuler
�[36mreinit0: �[39m Solver Mode: NEWTON
�[36mreinit0: �[39m PETSc Preconditioner: lu mat_superlu_dist_fact: SamePattern mat_superlu_dist_replacetinypivot: true
�[36mreinit0: �[39m MOOSE Preconditioner: SMP (auto)
�[36mreinit0: �[39m
�[36mreinit0: �[39m�[31mLEGACY MODES ENABLED:�[39m
�[36mreinit0: �[39m This application uses the legacy initial residual evaluation behavior. The legacy behavior performs an often times redundant residual evaluation before the solution modifying objects are executed prior to the initial (0th nonlinear iteration) residual evaluation. The new behavior skips that redundant residual evaluation unless the parameter Executioner/use_pre_smo_residual is set to true. To remove this message and enable the new behavior, set the parameter 'use_legacy_initial_residual_evaluation_behavior' to false in *App.C. Some tests that rely on the side effects of the legacy behavior may fail/diff and should be re-golded.
�[36mreinit0: �[39m�[39m
�[36mreinit0: �[39mCurrently Performing Initial Setup�[36mreinit0: �[39m
�[36mreinit0: �[39m Performing Initial Adaptivity�[36mreinit0: �[39m
�[36mreinit0: �[39m Handling Mesh Changes [�[33m 10.34 s�[39m] [�[33m 3064 MB�[39m]
�[36mreinit0: �[39m Adapting Mesh.... [�[33m 33.41 s�[39m] [�[33m 3677 MB�[39m]
�[36mreinit0: �[39m Handling Mesh Changes........... [�[33m 71.38 s�[39m] [�[33m 4298 MB�[39m]
�[36mreinit0: �[39m Finished Performing Initial Adaptivity [�[33m123.94 s�[39m] [�[33m 4298 MB�[39m]
�[36mreinit0: �[39m Building SemiLocalElemMap.... [�[33m 32.70 s�[39m] [�[33m 4298 MB�[39m]
�[36mreinit0: �[39mFinished Performing Initial Setup [�[33m157.01 s�[39m] [�[33m 4298 MB�[39m]
Finished Initializing MultiApps [�[33m157.01 s�[39m] [�[33m 4298 MB�[39m]
Building SemiLocalElemMap.... [�[33m 32.53 s�[39m] [�[33m 4298 MB�[39m]
Still Performing Initial Setup..
Finished Performing Initial Setup [�[33m356.38 s�[39m] [�[33m 4298 MB�[39m]
Time Step 0, time = 0
Currently Outputting
Outputting Step. [�[33m 15.83 s�[39m] [�[33m 4411 MB�[39m]
Finished Outputting [�[33m 15.84 s�[39m] [�[33m 4411 MB�[39m]
Time Step 1, time = 0.0001, dt = 0.0001
* Nonlinear |R| = �[32m1.617244e-07�[39m (Before preset BCs, predictors, correctors, and constraints)
Performing automatic scaling calculation
Computing Automatic Scaling
Computing Jacobian......... [�[33m 52.11 s�[39m] [�[33m 4413 MB�[39m]
Finished Computing Automatic Scaling [�[33m 52.48 s�[39m] [�[33m 4413 MB�[39m]
0 Nonlinear |R| = �[32m3.988573e-08�[39m
Finished Solving [�[33m 61.02 s�[39m] [�[33m 4413 MB�[39m]
�[32m Solve Converged!�[39m
Still Executing.Solving Reinitialization problem.
�[36mreinit0: �[39m
�[36mreinit0: �[39mTime Step 0, time = 0
�[36mreinit0: �[39m
�[36mreinit0: �[39mTime Step 1, time = 1e-05, dt = 1e-05
�[36mreinit0: �[39m * Nonlinear |R| = �[32m1.738155e-07�[39m (Before preset BCs, predictors, correctors, and constraints)
�[36mreinit0: �[39m 0 Nonlinear |R| = �[32m1.738155e-07�[39m
�[36mreinit0: �[39m
�[36mreinit0: �[39m Computing Jacobian [�[33m 10.08 s�[39m] [�[33m 4489 MB�[39m]
[r3u05n1:3207054:0:3207054] cma_ep.c:85 process_vm_readv(pid=3207053 {0x14a99e9efa38,524288}-->{0x1490ca5f9a38,524288}) returned -1: No such process
[r3u05n1:3207047:0:3207047] cma_ep.c:85 process_vm_readv(pid=3207048 {0x14cbbcdf5030,524288}-->{0x14cd6db0a030,524288}) returned -1: No such process
[r3u05n1:3207052:0:3207052] cma_ep.c:85 process_vm_readv(pid=3207050 {0x15265252d140,524288}-->{0x1549d2c8a140,524288}) returned -1: No such process
--------------------------------------------------------------------------
prterun noticed that process rank 69 with PID 3207048 on node r3u05n1 exited on
signal 9 (Killed).
--------------------------------------------------------------------------
Apr 20 15:31:37.599787 3206920 slurmstepd 0x15041958cfc0: error: Detected 4 oom_kill events in StepId=12751509.batch. Some of the step tasks have been OOM Killed.
All reactions
Replies: 1 comment
Regarding my previous comment, the following preconditioner is too expensive, and could explain why I got out of memory error.
petsc_options_iname = '-pc_type -pc_factor_shift_type -pc_factor_mat_solver_package -ksp_type'
petsc_options_value = 'lu NONZERO superlu_dist preonly'
Are there any alternatives that known to work well for this problem? for example I tried the following and faced this error:
petsc_options_iname = '-pc_type -pc_asm_overlap -sub_pc_type -sub_pc_factor_levels -ksp_gmres_restart'
petsc_options_value = 'bjacobi 2 ilu 3 200'
0 Nonlinear |R| = �[32m1.476174e-07�[39m
Computing Jacobian. [�[33m 10.51 s�[39m] [�[33m 608 MB�[39m]
Linear solve did not converge due to DIVERGED_PC_FAILED iterations 0
PC failed due to SUBPC_ERROR
Nonlinear solve did not converge due to DIVERGED_FNORM_NAN iterations 0
�[31m Solve Did NOT Converge!�[39m
All reactions
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment