-
Notifications
You must be signed in to change notification settings - Fork 648
When I used the finetuned model for MD simulation, an "out-of-memory" error occurred #5167
Dear Developer
When I used the finetuned model for MD simulation, an "out-of-memory" error occurred.
deepmd.utils.errors.OutOfMemoryError: The callable still throws an out-of-memory (OOM) error even when batch size is 1!
version:DeePMD-kit v3.1.2
installation way:conda
input.json
ASE_md.py
If you could offer me some suggestions, I would be very grateful.
All reactions
Replies: 3 comments 1 reply
Hi @Yancy-Luke! I'm Dosu and I’m helping the deepmd-kit team.
Out-of-memory errors in DeePMD-kit MD simulations with finetuned models can happen for several reasons. If batch size is already 1, the model or system may be too large for your available memory. Common causes include a large model network size, a high number of atoms, or using descriptor types like se_e3 which consume much more memory than others such as se_e2_r—switching descriptor type or reducing the cutoff radius can help reduce memory usage [discussion].
If you are running inference or model deviation in a loop, make sure you are not repeatedly loading the same model, as TensorFlow or PyTorch may not release memory until the process shuts down [docs].
For large systems, consider using an external neighbor list compatible with ASE's NewPrimitiveNeighborList to avoid the native O(N2) complexity [docs]. If you are using a DPA-2/3 finetuned model, distilling it to a compressible DPA-1 model can significantly reduce memory requirements [discussion].
If the error occurs at a specific simulation step, it may indicate a memory leak or accumulation. Evaluating descriptors for each system in a loop (rather than all at once) can also help avoid OOM errors [discussion].
Check your hardware specs (RAM/GPU memory), confirm whether the simulation is running on CPU or GPU, and try running a smaller system or simpler model to isolate the issue. For more details on tuning model/network size and other memory-related settings, see the troubleshooting guide.
If you can share your hardware specs, model architecture, and system size, I can help with more targeted suggestions.
To reply, just mention @dosu.
Share context across your team and agents. Try Dosu.
Leave Feedback Learn about deepmd-kit with Dosu Add Dosu to your team
All reactions
@dosu
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.105.08 Driver Version: 580.105.08 CUDA Version: 13.0 |
+-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GeForce RTX 3060 Off | 00000000:A8:00.0 Off | N/A |
| 30% 46C P0 43W / 170W | 0MiB / 12288MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
+-----------------------------------------------------------------------------------------+
CPU(s): 208
NUMA node0 CPU(s): 0-51,104-155
NUMA node1 CPU(s): 52-103,156-207
total used free shared buff/cache available
Mem: 251Gi 59Gi 95Gi 130Mi 97Gi 191Gi
Swap: 4.0Gi 0B 4.0Gi
the descriptor is dpa3,the model-branch is H2O_H2O_PD,and the system size is 3000 atoms
All reactions
Your out-of-memory error is due to the extremely high memory requirements of DPA-3 models—on GPUs, even 40GB cards can only handle ~400 atoms with DPA-3, so 3000 atoms on a 12GB RTX 3060 is far beyond feasible [discussion]. This is not related to your batch size or available system RAM; it's a fundamental limitation of the DPA-3 architecture and GPU memory usage.
Here’s what you can do:
- For large-scale MD, the recommended solution is to distill your DPA-3 model to a compressible DPA-1 model. DPA-1 is much more memory-efficient and can run large systems on GPUs with limited memory. The distillation method is described in this Nature Communications paper, and there’s an example notebook here [discussion].
- If you must use DPA-3, you’ll need to drastically reduce your system size (to <400 atoms for your GPU), or run on CPU (which is slower but can handle larger memory footprints).
- DPA-3 does not support model compression, so you cannot reduce its memory usage that way [docs].
- You can try tuning DPA-3 descriptor settings (like reducing n_dim, e_dim, a_dim, e_sel, a_sel, or using angular message compression), but these changes have limited impact for very large systems and may affect accuracy [docs].
- Make sure you are not repeatedly loading the model in a loop, as this can cause memory leaks in TensorFlow/PyTorch [issue].
If you want help with the DPA-2/3 to DPA-1 distillation workflow or advanced descriptor tuning, let me know your priorities and I can walk you through the steps.
To reply, just mention @dosu.
Docs are dead. Just use Dosu.
Leave Feedback Ask Dosu about deepmd-kit Share Dosu with your team
All reactions
The broad diagnosis (this DPA3/system combination does not fit in 12 GB) is reasonable, but a few details in the Dosu replies need qualification.
batch size is 1 means DeePMD has already reduced inference to one frame. That does not split the 3,000 atoms within that frame, so lowering the inference batch size cannot solve this particular OOM. There is also no universal DPA3 limit of "400 atoms on 40 GB": memory depends strongly on density, neighbor selections/cutoffs, message dimensions and layers, precision, and the inference path. The cited issue reports one user's configuration, not an architectural limit.
For DeePMD-kit 3.1.2, the external ASE NewPrimitiveNeighborList path documented by DeePMD is TensorFlow-only, so it is not a valid workaround for this PyTorch DPA3 model. Practical options are:
- use the LAMMPS interface and MPI domain decomposition, preferably across multiple GPUs for a system this large;
- use CPU inference or a larger-memory GPU;
- reduce the simulated system size;
- retrain a smaller DPA3 configuration or distill the DPA3 teacher to a more deployment-efficient student such as DPA1.
Post-training dp compress tabulation is unsupported for DPA3 in v3.1.2. DPA3's angular-message compression settings are architecture/training-time choices and do not turn an already trained checkpoint into a smaller model.
Coding agent: Codex
Codex version: codex-cli 0.149.0
Model: gpt-5.6-sol
Reasoning effort: xhigh