-
Notifications
You must be signed in to change notification settings - Fork 249
MacOS test falsely uses MPS, fails and is misreported as passing #1416
Description
🐛 Describe the bug
#1415, #1404 and all other PRs fail on test-readme-macos when torchchat apparently falsely tries to load the model to MPS.
Due to #1315 , the test don't report as failed, so things get committed anyway.
I don't know why test-readme-macos would try to load into MPS. There's a multi-layered story here, where virtualized Mac does not support MPS (I think because most likely there's no MMU for MPS, so you can't virtualize MPS). MPS is however still reported as available by the OS, and hence a simple check torch.backends.mps.is_available(): is not sufficient because pytorch thinks that MPS is actually available (but any and all memory allocations should fail).
We're trying to fix this by doing an allocation of a tensor in MPS memory and see if that succeeds or fails in is_mps_available() here => https://github.com/pytorch/torchchat/blob/main/torchchat/utils/build_utils.py#L269 when get_device_str() is looking to check if MPS is available, and the fastest device should be returned as MPS.
Ideally, this should be fixed in the expansion get_device_str() and is_mps_available() functions, together with #1315.
Fail example is here => https://github.com/pytorch/torchchat/actions/runs/12243820522/job/34154220414?pr=1404
## Running via PyTorch
Downloading https://huggingface.co/karpathy/tinyllamas/resolve/main/stories15M.pt...
Downloading https://github.com/karpathy/llama2.c/raw/master/tokenizer.model...
NumExpr defaulting to 6 threads.
PyTorch version 2.6.0.dev20241013 available.
Moving model to /Users/runner/.torchchat/model-cache/stories15M.
Downloading builder script: 0%| | 0.00/5.67k [00:00<?, ?B/s]
Downloading builder script: 100%|██████████| 5.67k/5.67k [00:00<00:00, 5.30MB/s]
Traceback (most recent call last):
File "/Users/runner/work/torchchat/torchchat/torchchat.py", line 96, in <module>
Using device=mps
Loading model...
generate_main(args)
File "/Users/runner/work/torchchat/torchchat/torchchat/generate.py", line 1235, in main
gen = Generator(
File "/Users/runner/work/torchchat/torchchat/torchchat/generate.py", line 293, in __init__
self.model = _initialize_model(self.builder_args, self.quantize, self.tokenizer)
File "/Users/runner/work/torchchat/torchchat/torchchat/cli/builder.py", line 603, in _initialize_model
model = _load_model(builder_args)
File "/Users/runner/work/torchchat/torchchat/torchchat/cli/builder.py", line 465, in _load_model
model = _load_model_default(builder_args)
File "/Users/runner/work/torchchat/torchchat/torchchat/cli/builder.py", line 427, in _load_model_default
checkpoint = _load_checkpoint(builder_args)
File "/Users/runner/work/torchchat/torchchat/torchchat/cli/builder.py", line 412, in _load_checkpoint
checkpoint = torch.load(
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/torch/serialization.py", line 1359, in load
return _load(
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/torch/serialization.py", line 1856, in _load
result = unpickler.load()
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/torch/_weights_only_unpickler.py", line 388, in load
self.append(self.persistent_load(pid))
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/torch/serialization.py", line 1820, in persistent_load
Time to load model: 0.10 seconds
typed_storage = load_tensor(
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/torch/serialization.py", line 1792, in load_tensor
wrap_storage=restore_location(storage, location),
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/torch/serialization.py", line 1693, in restore_location
return default_restore_location(storage, map_location)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/torch/serialization.py", line 601, in default_restore_location
result = fn(storage, location)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/torch/serialization.py", line 467, in _mps_deserialize
return obj.mps()
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/torch/storage.py", line 260, in mps
return torch.UntypedStorage(self.size(), device="mps").copy_(self, False)
RuntimeError: MPS backend out of memory (MPS allocated: 1.02 GB, other allocations: 0 bytes, max allowed: 15.87 GB). Tried to allocate 256 bytes on shared pool. Use PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.0 to disable upper limit for memory allocations (may cause system failure).
+ echo ::group::Completion
Versions
problem occurs in github ci/cd