Anurag1734/cuda-error-resolution-analysis
07
1[2 {3 "post_stream": {4 "posts": [5 {6 "id": 464429,7 "name": "",8 "username": "pierisk",9 "avatar_template": "/user_avatar/discuss.pytorch.org/pierisk/{size}/54972_2.png",10 "created_at": "2025-01-31T18:30:16.375Z",11 "cooked": "<p>I’m using <a href=\"https://pytorch.org/docs/stable/multiprocessing.html#sharing-cuda-tensors\" rel=\"noopener nofollow ugc\">this documentation guide</a> to share a CUDA tensor between 2 python mp processes using <a href=\"https://pytorch.org/docs/stable/generated/torch.Tensor.share_memory_.html#torch.Tensor.share_memory_\" rel=\"noopener nofollow ugc\">share_memory_()</a> and a torch.mp.Queue.<br>\nThe tensors in the source arrive from an external source in streams as CUDA buffers, so I don’t have full control apart from converting them to a Tensor allocated on CUDA device (if you want details I can provide them in a reply).</p>\n<p>I wanted to pass some <strong>metadata</strong> along with the CUDA tensor e.g. a string name/key for each image in the batch tensor (i.e. the first dimension indices of the tensor).<br>\nI was wondering if the same shared memory (zero copy) between processes can work when:</p>\n<ol>\n<li>Wrapping the CUDA tensor to be shared in a python class with these metadata as additional attributes.</li>\n<li>Somehow attaching metadata within the tensor itself. I saw <a href=\"https://dev-discuss.pytorch.org/t/is-there-a-place-for-storing-custom-data-within-pytorch-tensor/1085/8\" rel=\"noopener nofollow ugc\">this topic discussion</a> but it seems non-standard and non-trivial at least.</li>\n</ol>\n<p>Is there a way to send metadata along with each tensor in any way?<br>\nThank you very much for all your help!</p>",12 "post_number": 1,13 "post_type": 1,14 "posts_count": 1,15 "updated_at": "2025-01-31T18:30:16.375Z",16 "reply_count": 0,17 "reply_to_post_number": null,18 "quote_count": 0,19 "incoming_link_count": 5,20 "reads": 3,21 "readers_count": 2,22 "score": 25.6,23 "yours": false,24 "topic_id": 216101,25 "topic_slug": "torch-multiprocessing-pass-metadata-or-class-wrapper-for-shared-memory-cuda-tensor",26 "display_username": "",27 "primary_group_name": null,28 "flair_name": null,29 "flair_url": null,30 "flair_bg_color": null,31 "flair_color": null,32 "flair_group_id": null,33 "badges_granted": [],34 "version": 1,35 "can_edit": false,36 "can_delete": false,37 "can_recover": false,38 "can_see_hidden_post": false,39 "can_wiki": false,40 "link_counts": [41 {42 "url": "https://pytorch.org/docs/stable/multiprocessing.html#sharing-cuda-tensors",43 "internal": false,44 "reflection": false,45 "title": "Multiprocessing package - torch.multiprocessing — PyTorch 2.6 documentation",46 "clicks": 047 },48 {49 "url": "https://pytorch.org/docs/stable/generated/torch.Tensor.share_memory_.html#torch.Tensor.share_memory_",50 "internal": false,51 "reflection": false,52 "title": "torch.Tensor.share_memory_ — PyTorch 2.6 documentation",53 "clicks": 054 },55 {56 "url": "https://dev-discuss.pytorch.org/t/is-there-a-place-for-storing-custom-data-within-pytorch-tensor/1085/8",57 "internal": false,58 "reflection": false,59 "title": "Is there a place for storing custom data within PyTorch Tensor - #8 by ppiskorski - hardware-backends - PyTorch Developer Mailing List",60 "clicks": 061 }62 ],63 "read": true,64 "user_title": null,65 "bookmarked": false,66 "actions_summary": [],67 "moderator": false,68 "admin": false,69 "staff": false,70 "user_id": 61091,71 "hidden": false,72 "trust_level": 1,73 "deleted_at": null,74 "user_deleted": false,75 "edit_reason": null,76 "can_view_edit_history": true,77 "wiki": false,78 "post_url": "/t/torch-multiprocessing-pass-metadata-or-class-wrapper-for-shared-memory-cuda-tensor/216101/1",79 "can_accept_answer": false,80 "can_unaccept_answer": false,81 "accepted_answer": false,82 "topic_accepted_answer": null,83 "can_vote": false84 }85 ],86 "stream": [87 46442988 ]89 },90 "timeline_lookup": [91 [92 1,93 26794 ]95 ],96 "suggested_topics": [97 {98 "fancy_title": "Ubuntu 24.04 NCCL Seemingly Randomly Timing Out on All Reduce",99 "id": 215440,100 "title": "Ubuntu 24.04 NCCL Seemingly Randomly Timing Out on All Reduce",101 "slug": "ubuntu-24-04-nccl-seemingly-randomly-timing-out-on-all-reduce",102 "posts_count": 1,103 "reply_count": 0,104 "highest_post_number": 1,105 "image_url": null,106 "created_at": "2025-01-15T23:33:54.535Z",107 "last_posted_at": "2025-01-15T23:33:54.578Z",108 "bumped": true,109 "bumped_at": "2025-01-15T23:33:54.578Z",110 "archetype": "regular",111 "unseen": false,112 "pinned": false,113 "unpinned": null,114 "visible": true,115 "closed": false,116 "archived": false,117 "bookmarked": null,118 "liked": null,119 "tags_descriptions": {},120 "like_count": 0,121 "views": 228,122 "category_id": 12,123 "featured_link": null,124 "has_accepted_answer": false,125 "posters": [126 {127 "extras": "latest single",128 "description": "Original Poster, Most Recent Poster",129 "user": {130 "id": 78912,131 "username": "Tanay_Arora",132 "name": "Tanay Arora",133 "avatar_template": "/user_avatar/discuss.pytorch.org/tanay_arora/{size}/72747_2.png",134 "trust_level": 1135 }136 }137 ]138 },139 {140 "fancy_title": "How to share Dataset among different processes in Pytorch？",141 "id": 213998,142 "title": "How to share Dataset among different processes in Pytorch？",143 "slug": "how-to-share-dataset-among-different-processes-in-pytorch",144 "posts_count": 1,145 "reply_count": 0,146 "highest_post_number": 1,147 "image_url": null,148 "created_at": "2024-12-09T14:17:51.942Z",149 "last_posted_at": "2024-12-09T14:17:52.005Z",150 "bumped": true,151 "bumped_at": "2024-12-09T14:17:52.005Z",152 "archetype": "regular",153 "unseen": false,154 "pinned": false,155 "unpinned": null,156 "visible": true,157 "closed": false,158 "archived": false,159 "bookmarked": null,160 "liked": null,161 "tags_descriptions": {},162 "like_count": 0,163 "views": 30,164 "category_id": 12,165 "featured_link": null,166 "has_accepted_answer": false,167 "posters": [168 {169 "extras": "latest single",170 "description": "Original Poster, Most Recent Poster",171 "user": {172 "id": 81406,173 "username": "forestbat",174 "name": "forestbat",175 "avatar_template": "/user_avatar/discuss.pytorch.org/forestbat/{size}/74440_2.png",176 "trust_level": 1177 }178 }179 ]180 },181 {182 "fancy_title": "DDP barrier has no effect when more than one is used",183 "id": 212518,184 "title": "DDP barrier has no effect when more than one is used",185 "slug": "ddp-barrier-has-no-effect-when-more-than-one-is-used",186 "posts_count": 1,187 "reply_count": 0,188 "highest_post_number": 1,189 "image_url": null,190 "created_at": "2024-11-04T21:22:19.733Z",191 "last_posted_at": "2024-11-04T21:22:19.799Z",192 "bumped": true,193 "bumped_at": "2024-11-04T21:27:58.272Z",194 "archetype": "regular",195 "unseen": false,196 "pinned": false,197 "unpinned": null,198 "visible": true,199 "closed": false,200 "archived": false,201 "bookmarked": null,202 "liked": null,203 "tags_descriptions": {},204 "like_count": 0,205 "views": 39,206 "category_id": 12,207 "featured_link": null,208 "has_accepted_answer": false,209 "posters": [210 {211 "extras": "latest single",212 "description": "Original Poster, Most Recent Poster",213 "user": {214 "id": 47099,215 "username": "seer_mer",216 "name": "seer mer",217 "avatar_template": "/user_avatar/discuss.pytorch.org/seer_mer/{size}/40173_2.png",218 "trust_level": 1219 }220 }221 ]222 },223 {224 "fancy_title": "Behavior of wait() on async CUDA collectives",225 "id": 214543,226 "title": "Behavior of wait() on async CUDA collectives",227 "slug": "behavior-of-wait-on-async-cuda-collectives",228 "posts_count": 2,229 "reply_count": 0,230 "highest_post_number": 2,231 "image_url": null,232 "created_at": "2024-12-22T15:28:02.279Z",233 "last_posted_at": "2024-12-26T22:56:53.306Z",234 "bumped": true,235 "bumped_at": "2024-12-26T22:56:53.306Z",236 "archetype": "regular",237 "unseen": false,238 "pinned": false,239 "unpinned": null,240 "visible": true,241 "closed": false,242 "archived": false,243 "bookmarked": null,244 "liked": null,245 "tags_descriptions": {},246 "like_count": 0,247 "views": 169,248 "category_id": 12,249 "featured_link": null,250 "has_accepted_answer": false,251 "posters": [252 {253 "extras": null,254 "description": "Original Poster",255 "user": {256 "id": 80214,257 "username": "x41lakazam",258 "name": "",259 "avatar_template": "/user_avatar/discuss.pytorch.org/x41lakazam/{size}/73332_2.png",260 "trust_level": 1261 }262 },263 {264 "extras": "latest",265 "description": "Most Recent Poster",266 "user": {267 "id": 39542,268 "username": "H-Huang",269 "name": "Howard Huang",270 "avatar_template": "/user_avatar/discuss.pytorch.org/h-huang/{size}/35598_2.png",271 "trust_level": 2272 }273 }274 ]275 },276 {277 "fancy_title": "How to handle RAM OOM in DDP?",278 "id": 219245,279 "title": "How to handle RAM OOM in DDP?",280 "slug": "how-to-handle-ram-oom-in-ddp",281 "posts_count": 4,282 "reply_count": 2,283 "highest_post_number": 4,284 "image_url": null,285 "created_at": "2025-04-19T08:33:38.442Z",286 "last_posted_at": "2025-04-19T14:08:42.350Z",287 "bumped": true,288 "bumped_at": "2025-04-19T14:08:42.350Z",289 "archetype": "regular",290 "unseen": false,291 "pinned": false,292 "unpinned": null,293 "visible": true,294 "closed": false,295 "archived": false,296 "bookmarked": null,297 "liked": null,298 "tags_descriptions": {},299 "like_count": 0,300 "views": 128,301 "category_id": 12,302 "featured_link": null,303 "has_accepted_answer": false,304 "posters": [305 {306 "extras": null,307 "description": "Original Poster",308 "user": {309 "id": 82744,310 "username": "Maria_Djeblahi",311 "name": "Maria Djeblahi",312 "avatar_template": "/user_avatar/discuss.pytorch.org/maria_djeblahi/{size}/75704_2.png",313 "trust_level": 1314 }315 },316 {317 "extras": "latest",318 "description": "Most Recent Poster",319 "user": {320 "id": 3534,321 "username": "ptrblck",322 "name": "",323 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",324 "admin": true,325 "moderator": true,326 "trust_level": 2327 }328 }329 ]330 }331 ],332 "tags_descriptions": {},333 "fancy_title": "Torch.multiprocessing: pass metadata or class wrapper for shared memory CUDA tensor?",334 "id": 216101,335 "title": "Torch.multiprocessing: pass metadata or class wrapper for shared memory CUDA tensor?",336 "posts_count": 1,337 "created_at": "2025-01-31T18:30:16.330Z",338 "views": 23,339 "reply_count": 0,340 "like_count": 0,341 "last_posted_at": "2025-01-31T18:30:16.375Z",342 "visible": true,343 "closed": false,344 "archived": false,345 "has_summary": false,346 "archetype": "regular",347 "slug": "torch-multiprocessing-pass-metadata-or-class-wrapper-for-shared-memory-cuda-tensor",348 "category_id": 12,349 "word_count": 220,350 "deleted_at": null,351 "user_id": 61091,352 "featured_link": null,353 "pinned_globally": false,354 "pinned_at": null,355 "pinned_until": null,356 "image_url": null,357 "slow_mode_seconds": 0,358 "draft": null,359 "draft_key": "topic_216101",360 "draft_sequence": null,361 "unpinned": null,362 "pinned": false,363 "current_post_number": 1,364 "highest_post_number": 1,365 "deleted_by": null,366 "actions_summary": [367 {368 "id": 4,369 "count": 0,370 "hidden": false,371 "can_act": false372 },373 {374 "id": 8,375 "count": 0,376 "hidden": false,377 "can_act": false378 },379 {380 "id": 10,381 "count": 0,382 "hidden": false,383 "can_act": false384 },385 {386 "id": 7,387 "count": 0,388 "hidden": false,389 "can_act": false390 }391 ],392 "chunk_size": 20,393 "bookmarked": false,394 "topic_timer": null,395 "message_bus_last_id": 0,396 "participant_count": 1,397 "show_read_indicator": false,398 "thumbnails": null,399 "slow_mode_enabled_until": null,400 "can_vote": false,401 "vote_count": 0,402 "user_voted": false,403 "discourse_zendesk_plugin_zendesk_id": null,404 "discourse_zendesk_plugin_zendesk_url": "https://your-url.zendesk.com/agent/tickets/",405 "details": {406 "can_edit": false,407 "notification_level": 1,408 "participants": [409 {410 "id": 61091,411 "username": "pierisk",412 "name": "",413 "avatar_template": "/user_avatar/discuss.pytorch.org/pierisk/{size}/54972_2.png",414 "post_count": 1,415 "primary_group_name": null,416 "flair_name": null,417 "flair_url": null,418 "flair_color": null,419 "flair_bg_color": null,420 "flair_group_id": null,421 "trust_level": 1422 }423 ],424 "created_by": {425 "id": 61091,426 "username": "pierisk",427 "name": "",428 "avatar_template": "/user_avatar/discuss.pytorch.org/pierisk/{size}/54972_2.png"429 },430 "last_poster": {431 "id": 61091,432 "username": "pierisk",433 "name": "",434 "avatar_template": "/user_avatar/discuss.pytorch.org/pierisk/{size}/54972_2.png"435 }436 },437 "bookmarks": []438 },439 {440 "post_stream": {441 "posts": [442 {443 "id": 440042,444 "name": "",445 "username": "vortec",446 "avatar_template": "/user_avatar/discuss.pytorch.org/vortec/{size}/66340_2.png",447 "created_at": "2024-04-21T15:25:46.367Z",448 "cooked": "<p>Hi, I’m trying to get torchrun to work on my M1 Pro Mac. I saw the other forum posts on this topic, but development happens rapidly and I didn’t get it to work. So, I downloaded Llama 3, ran pip install -e setup.py. Trying to run torchrun with the following command:</p>\n<pre><code class=\"lang-auto\">torchrun --nproc_per_node 1 example_chat_completion.py \\\n --no_cuda=True --ckpt_dir Meta-Llama-3-8B-Instruct/ \\\n --tokenizer_path Meta-Llama-3-8B-Instruct/tokenizer.model \\\n --max_seq_len 512 --max_batch_size 6\n</code></pre>\n<p>Then I get the following error:</p>\n<pre><code class=\"lang-auto\">Traceback (most recent call last):\n File \"/Users/vortec/workspace/llm/llama3/example_chat_completion.py\", line 84, in <module>\n fire.Fire(main)\n File \"/Users/vortec/workspace/instances/llama3/lib/python3.11/site-packages/fire/core.py\", line 143, in Fire\n component_trace = _Fire(component, args, parsed_flag_args, context, name)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/Users/vortec/workspace/instances/llama3/lib/python3.11/site-packages/fire/core.py\", line 477, in _Fire\n component, remaining_args = _CallAndUpdateTrace(\n ^^^^^^^^^^^^^^^^^^^^\n File \"/Users/vortec/workspace/instances/llama3/lib/python3.11/site-packages/fire/core.py\", line 693, in _CallAndUpdateTrace\n component = fn(*varargs, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^\n File \"/Users/vortec/workspace/llm/llama3/example_chat_completion.py\", line 31, in main\n generator = Llama.build(\n ^^^^^^^^^^^^\n File \"/Users/vortec/workspace/llm/llama3/llama/generation.py\", line 68, in build\n torch.distributed.init_process_group(\"nccl\")\n File \"/Users/vortec/workspace/instances/llama3/lib/python3.11/site-packages/torch/distributed/c10d_logger.py\", line 86, in wrapper\n func_return = func(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^\n File \"/Users/vortec/workspace/instances/llama3/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py\", line 1184, in init_process_group\n default_pg, _ = _new_process_group_helper(\n ^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/Users/vortec/workspace/instances/llama3/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py\", line 1302, in _new_process_group_helper\n raise RuntimeError(\"Distributed package doesn't have NCCL built in\")\nRuntimeError: Distributed package doesn't have NCCL built in\n</code></pre>\n<p>Here’s the output of collect_env:</p>\n<pre><code class=\"lang-auto\">Collecting environment information...\nPyTorch version: 2.2.2\nIs debug build: False\nCUDA used to build PyTorch: None\nROCM used to build PyTorch: N/A\n\nOS: macOS 14.2.1 (arm64)\nGCC version: Could not collect\nClang version: 15.0.0 (clang-1500.1.0.2.5)\nCMake version: Could not collect\nLibc version: N/A\n\nPython version: 3.11.4 (main, Jun 20 2023, 17:23:00) [Clang 14.0.3 (clang-1403.0.22.14.1)] (64-bit runtime)\nPython platform: macOS-14.2.1-arm64-arm-64bit\nIs CUDA available: False\nCUDA runtime version: No CUDA\nCUDA_MODULE_LOADING set to: N/A\nGPU models and configuration: No CUDA\nNvidia driver version: No CUDA\ncuDNN version: No CUDA\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nApple M1 Pro\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] torch==2.2.2\n[pip3] torchaudio==2.2.2\n[pip3] torchvision==0.17.2\n[conda] Could not collect\n</code></pre>\n<p>What can I do to fix it?</p>",449 "post_number": 1,450 "post_type": 1,451 "posts_count": 4,452 "updated_at": "2024-04-21T15:25:46.367Z",453 "reply_count": 1,454 "reply_to_post_number": null,455 "quote_count": 0,456 "incoming_link_count": 1104,457 "reads": 10,458 "readers_count": 9,459 "score": 5472.0,460 "yours": false,461 "topic_id": 201315,462 "topic_slug": "apple-silicon-torchrun-distributed-package-doesnt-have-nccl-built-in",463 "display_username": "",464 "primary_group_name": null,465 "flair_name": null,466 "flair_url": null,467 "flair_bg_color": null,468 "flair_color": null,469 "flair_group_id": null,470 "badges_granted": [],471 "version": 1,472 "can_edit": false,473 "can_delete": false,474 "can_recover": false,475 "can_see_hidden_post": false,476 "can_wiki": false,477 "read": true,478 "user_title": null,479 "bookmarked": false,480 "actions_summary": [],481 "moderator": false,482 "admin": false,483 "staff": false,484 "user_id": 75191,485 "hidden": false,486 "trust_level": 0,487 "deleted_at": null,488 "user_deleted": false,489 "edit_reason": null,490 "can_view_edit_history": true,491 "wiki": false,492 "post_url": "/t/apple-silicon-torchrun-distributed-package-doesnt-have-nccl-built-in/201315/1",493 "can_accept_answer": false,494 "can_unaccept_answer": false,495 "accepted_answer": false,496 "topic_accepted_answer": null,497 "can_vote": false498 },499 {500 "id": 440045,501 "name": "",502 "username": "ptrblck",503 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",504 "created_at": "2024-04-21T15:30:44.849Z",505 "cooked": "<aside class=\"quote no-group\" data-username=\"vortec\" data-post=\"1\" data-topic=\"201315\">\n<div class=\"title\">\n<div class=\"quote-controls\"></div>\n<img loading=\"lazy\" alt=\"\" width=\"24\" height=\"24\" src=\"https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vortec/48/66340_2.png\" class=\"avatar\"> vortec:</div>\n<blockquote>\n<p>What can I do to fix it?</p>\n</blockquote>\n</aside>\n<p>Don’t use NCCL for distributed operations, as it’s not supported on Mac since it needs an NVIDIA GPU.<br>\nI’m also unsure if your system even has multiple GPUs (MPS devices), so you might want to disable the distributed usage.</p>",506 "post_number": 2,507 "post_type": 1,508 "posts_count": 4,509 "updated_at": "2024-04-21T15:30:44.849Z",510 "reply_count": 1,511 "reply_to_post_number": null,512 "quote_count": 1,513 "incoming_link_count": 6,514 "reads": 9,515 "readers_count": 8,516 "score": 36.8,517 "yours": false,518 "topic_id": 201315,519 "topic_slug": "apple-silicon-torchrun-distributed-package-doesnt-have-nccl-built-in",520 "display_username": "",521 "primary_group_name": null,522 "flair_name": null,523 "flair_url": null,524 "flair_bg_color": null,525 "flair_color": null,526 "flair_group_id": null,527 "badges_granted": [],528 "version": 1,529 "can_edit": false,530 "can_delete": false,531 "can_recover": false,532 "can_see_hidden_post": false,533 "can_wiki": false,534 "read": true,535 "user_title": "",536 "bookmarked": false,537 "actions_summary": [],538 "moderator": true,539 "admin": true,540 "staff": true,541 "user_id": 3534,542 "hidden": false,543 "trust_level": 2,544 "deleted_at": null,545 "user_deleted": false,546 "edit_reason": null,547 "can_view_edit_history": true,548 "wiki": false,549 "post_url": "/t/apple-silicon-torchrun-distributed-package-doesnt-have-nccl-built-in/201315/2",550 "can_accept_answer": false,551 "can_unaccept_answer": false,552 "accepted_answer": false,553 "topic_accepted_answer": null554 },555 {556 "id": 440046,557 "name": "",558 "username": "vortec",559 "avatar_template": "/user_avatar/discuss.pytorch.org/vortec/{size}/66340_2.png",560 "created_at": "2024-04-21T15:31:39.281Z",561 "cooked": "<p>I think my question is: What do I need to do to change this?</p>",562 "post_number": 3,563 "post_type": 1,564 "posts_count": 4,565 "updated_at": "2024-04-21T15:31:39.281Z",566 "reply_count": 1,567 "reply_to_post_number": 2,568 "quote_count": 0,569 "incoming_link_count": 6,570 "reads": 9,571 "readers_count": 8,572 "score": 36.8,573 "yours": false,574 "topic_id": 201315,575 "topic_slug": "apple-silicon-torchrun-distributed-package-doesnt-have-nccl-built-in",576 "display_username": "",577 "primary_group_name": null,578 "flair_name": null,579 "flair_url": null,580 "flair_bg_color": null,581 "flair_color": null,582 "flair_group_id": null,583 "badges_granted": [],584 "version": 1,585 "can_edit": false,586 "can_delete": false,587 "can_recover": false,588 "can_see_hidden_post": false,589 "can_wiki": false,590 "read": true,591 "user_title": null,592 "reply_to_user": {593 "id": 3534,594 "username": "ptrblck",595 "name": "",596 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png"597 },598 "bookmarked": false,599 "actions_summary": [],600 "moderator": false,601 "admin": false,602 "staff": false,603 "user_id": 75191,604 "hidden": false,605 "trust_level": 0,606 "deleted_at": null,607 "user_deleted": false,608 "edit_reason": null,609 "can_view_edit_history": true,610 "wiki": false,611 "post_url": "/t/apple-silicon-torchrun-distributed-package-doesnt-have-nccl-built-in/201315/3",612 "can_accept_answer": false,613 "can_unaccept_answer": false,614 "accepted_answer": false,615 "topic_accepted_answer": null616 },617 {618 "id": 464423,619 "name": "Scott",620 "username": "scottnzuk",621 "avatar_template": "/letter_avatar_proxy/v4/letter/s/e47c2d/{size}.png",622 "created_at": "2025-01-31T16:12:57.674Z",623 "cooked": "<p>Here’s a concise solution for using <strong>PyTorch Distributed (torchrun)</strong> on Apple Silicon (M1/M2) where NCCL is unavailable:</p>\n<hr>\n<h3><a name=\"p-464423-problem-1\" class=\"anchor\" href=\"#p-464423-problem-1\"></a><strong>Problem</strong></h3>\n<p>NCCL is not supported on macOS, so distributed training with <code>torchrun</code> fails with errors like:</p>\n<pre data-code-wrap=\"bash\"><code class=\"lang-bash\">RuntimeError: Distributed package doesn't have NCCL built in\n</code></pre>\n<hr>\n<h3><a name=\"p-464423-solution-2\" class=\"anchor\" href=\"#p-464423-solution-2\"></a><strong>Solution</strong></h3>\n<p>Use the <strong><code>gloo</code> backend</strong> instead of NCCL. Follow these steps:</p>\n<h4><a name=\"p-464423-h-1-install-pytorch-with-mps-support-3\" class=\"anchor\" href=\"#p-464423-h-1-install-pytorch-with-mps-support-3\"></a>1. <strong>Install PyTorch with MPS Support</strong></h4>\n<p>Ensure you have PyTorch ≥ 2.0 (with Apple Silicon MPS support):</p>\n<pre data-code-wrap=\"bash\"><code class=\"lang-bash\">pip3 install torch torchvision torchaudio\n</code></pre>\n<h4><a name=\"p-464423-h-2-modify-your-training-script-4\" class=\"anchor\" href=\"#p-464423-h-2-modify-your-training-script-4\"></a>2. <strong>Modify Your Training Script</strong></h4>\n<p>Explicitly set the backend to <code>gloo</code> and use <code>device=\"mps\"</code> for GPU acceleration:</p>\n<pre data-code-wrap=\"python\"><code class=\"lang-python\">import os\nimport torch\nimport torch.distributed as dist\nfrom torch.nn.parallel import DistributedDataParallel as DDP\n\ndef setup(rank, world_size):\n # Use \"gloo\" backend for CPU/MPS\n os.environ[\"MASTER_ADDR\"] = \"localhost\"\n os.environ[\"MASTER_PORT\"] = \"12355\"\n dist.init_process_group(\"gloo\", rank=rank, world_size=world_size)\n\ndef main(rank, world_size):\n setup(rank, world_size)\n model = YourModel().to(rank) # Use `device=\"mps\"` if needed\n ddp_model = DDP(model, device_ids=[rank])\n # ... training logic ...\n dist.destroy_process_group()\n\nif __name__ == \"__main__\":\n world_size = 2 # Match your CPU cores or MPS capability\n torch.multiprocessing.spawn(main, args=(world_size,), nprocs=world_size)\n</code></pre>\n<h4><a name=\"p-464423-h-3-launch-with-torchrun-5\" class=\"anchor\" href=\"#p-464423-h-3-launch-with-torchrun-5\"></a>3. <strong>Launch with <code>torchrun</code></strong></h4>\n<p>Force the <code>gloo</code> backend and specify processes:</p>\n<pre data-code-wrap=\"bash\"><code class=\"lang-bash\">torchrun \\\n --nproc_per_node=2 \\ # Match your CPU cores\n --nnodes=1 \\\n --node_rank=0 \\\n --master_addr=localhost \\\n --master_port=12355 \\\n your_script.py\n</code></pre>\n<hr>\n<h3><a name=\"p-464423-key-notes-6\" class=\"anchor\" href=\"#p-464423-key-notes-6\"></a><strong>Key Notes</strong></h3>\n<ul>\n<li><strong>MPS Limitations</strong>: While <code>device=\"mps\"</code> uses Apple Silicon GPUs, <code>gloo</code> runs distributed communication on CPU. This hybrid setup may not be optimal but works for basic tasks.</li>\n<li><strong>CPU Fallback</strong>: If MPS causes issues, use <code>device=\"cpu\"</code> for full compatibility.</li>\n<li><strong>Performance</strong>: Expect slower speeds compared to NCCL on NVIDIA GPUs.</li>\n</ul>\n<hr>\n<h3><a name=\"p-464423-alternatives-7\" class=\"anchor\" href=\"#p-464423-alternatives-7\"></a><strong>Alternatives</strong></h3>\n<p>For advanced distributed training:</p>\n<ol>\n<li><strong>Linux + NVIDIA GPU</strong>: Use NCCL in a cloud/remote server.</li>\n<li><strong>MLX Framework</strong>: Optimized for Apple Silicon (e.g., <a href=\"https://github.com/ml-explore/mlx-examples/tree/main/whisper\" rel=\"noopener nofollow ugc\">MLX-Whisper</a>).</li>\n</ol>",624 "post_number": 4,625 "post_type": 1,626 "posts_count": 4,627 "updated_at": "2025-01-31T16:12:57.674Z",628 "reply_count": 0,629 "reply_to_post_number": 3,630 "quote_count": 0,631 "incoming_link_count": 27,632 "reads": 4,633 "readers_count": 3,634 "score": 135.8,635 "yours": false,636 "topic_id": 201315,637 "topic_slug": "apple-silicon-torchrun-distributed-package-doesnt-have-nccl-built-in",638 "display_username": "Scott",639 "primary_group_name": null,640 "flair_name": null,641 "flair_url": null,642 "flair_bg_color": null,643 "flair_color": null,644 "flair_group_id": null,645 "badges_granted": [],646 "version": 1,647 "can_edit": false,648 "can_delete": false,649 "can_recover": false,650 "can_see_hidden_post": false,651 "can_wiki": false,652 "link_counts": [653 {654 "url": "https://github.com/ml-explore/mlx-examples/tree/main/whisper",655 "internal": false,656 "reflection": false,657 "title": "mlx-examples/whisper at main · ml-explore/mlx-examples · GitHub",658 "clicks": 15659 }660 ],661 "read": true,662 "user_title": null,663 "reply_to_user": {664 "id": 75191,665 "username": "vortec",666 "name": "",667 "avatar_template": "/user_avatar/discuss.pytorch.org/vortec/{size}/66340_2.png"668 },669 "bookmarked": false,670 "actions_summary": [],671 "moderator": false,672 "admin": false,673 "staff": false,674 "user_id": 82420,675 "hidden": false,676 "trust_level": 0,677 "deleted_at": null,678 "user_deleted": false,679 "edit_reason": null,680 "can_view_edit_history": true,681 "wiki": false,682 "post_url": "/t/apple-silicon-torchrun-distributed-package-doesnt-have-nccl-built-in/201315/4",683 "can_accept_answer": false,684 "can_unaccept_answer": false,685 "accepted_answer": false,686 "topic_accepted_answer": null687 }688 ],689 "stream": [690 440042,691 440045,692 440046,693 464423694 ]695 },696 "timeline_lookup": [697 [698 1,699 552700 ],701 [702 4,703 267704 ]705 ],706 "suggested_topics": [707 {708 "fancy_title": "Having trouble getting the mps version of torch to install",709 "id": 216103,710 "title": "Having trouble getting the mps version of torch to install",711 "slug": "having-trouble-getting-the-mps-version-of-torch-to-install",712 "posts_count": 2,713 "reply_count": 0,714 "highest_post_number": 2,715 "image_url": null,716 "created_at": "2025-01-31T19:04:49.418Z",717 "last_posted_at": "2025-02-28T16:01:18.379Z",718 "bumped": true,719 "bumped_at": "2025-02-28T16:01:18.379Z",720 "archetype": "regular",721 "unseen": false,722 "pinned": false,723 "unpinned": null,724 "visible": true,725 "closed": false,726 "archived": false,727 "bookmarked": null,728 "liked": null,729 "tags_descriptions": {},730 "like_count": 0,731 "views": 707,732 "category_id": 32,733 "featured_link": null,734 "has_accepted_answer": false,735 "posters": [736 {737 "extras": null,738 "description": "Original Poster",739 "user": {740 "id": 82422,741 "username": "voxvox142",742 "name": "",743 "avatar_template": "/user_avatar/discuss.pytorch.org/voxvox142/{size}/74704_2.png",744 "trust_level": 0745 }746 },747 {748 "extras": "latest",749 "description": "Most Recent Poster",750 "user": {751 "id": 76547,752 "username": "roach",753 "name": "",754 "avatar_template": "/user_avatar/discuss.pytorch.org/roach/{size}/61570_2.png",755 "trust_level": 1756 }757 }758 ]759 },760 {761 "fancy_title": "Building PyTorch 2.6.0 from source on macOS x86_64 (Intel)",762 "id": 216283,763 "title": "Building PyTorch 2.6.0 from source on macOS x86_64 (Intel)",764 "slug": "building-pytorch-2-6-0-from-source-on-macos-x86-64-intel",765 "posts_count": 7,766 "reply_count": 2,767 "highest_post_number": 7,768 "image_url": null,769 "created_at": "2025-02-05T17:07:14.219Z",770 "last_posted_at": "2025-09-24T15:35:05.044Z",771 "bumped": true,772 "bumped_at": "2025-09-24T15:35:05.044Z",773 "archetype": "regular",774 "unseen": false,775 "pinned": false,776 "unpinned": null,777 "visible": true,778 "closed": false,779 "archived": false,780 "bookmarked": null,781 "liked": null,782 "tags_descriptions": {},783 "like_count": 3,784 "views": 3877,785 "category_id": 32,786 "featured_link": null,787 "has_accepted_answer": false,788 "posters": [789 {790 "extras": null,791 "description": "Original Poster",792 "user": {793 "id": 82511,794 "username": "uwu-420",795 "name": "Uwu 420",796 "avatar_template": "/user_avatar/discuss.pytorch.org/uwu-420/{size}/75489_2.png",797 "trust_level": 0798 }799 },800 {801 "extras": null,802 "description": "Frequent Poster",803 "user": {804 "id": 64351,805 "username": "aisven",806 "name": "",807 "avatar_template": "/user_avatar/discuss.pytorch.org/aisven/{size}/58504_2.png",808 "trust_level": 1809 }810 },811 {812 "extras": "latest",813 "description": "Most Recent Poster",814 "user": {815 "id": 85615,816 "username": "Casey_KimScripts",817 "name": "",818 "avatar_template": "/user_avatar/discuss.pytorch.org/casey_kimscripts/{size}/78035_2.png",819 "trust_level": 1820 }821 }822 ]823 },824 {825 "fancy_title": "MAC inter processors cannot install torch2.5.1",826 "id": 215100,827 "title": "MAC inter processors cannot install torch2.5.1",828 "slug": "mac-inter-processors-cannot-install-torch2-5-1",829 "posts_count": 10,830 "reply_count": 6,831 "highest_post_number": 10,832 "image_url": null,833 "created_at": "2025-01-08T06:02:41.764Z",834 "last_posted_at": "2025-05-31T06:28:44.746Z",835 "bumped": true,836 "bumped_at": "2025-05-31T06:28:44.746Z",837 "archetype": "regular",838 "unseen": false,839 "pinned": false,840 "unpinned": null,841 "visible": true,842 "closed": false,843 "archived": false,844 "bookmarked": null,845 "liked": null,846 "tags_descriptions": {},847 "like_count": 5,848 "views": 2122,849 "category_id": 32,850 "featured_link": null,851 "has_accepted_answer": false,852 "posters": [853 {854 "extras": null,855 "description": "Original Poster",856 "user": {857 "id": 81936,858 "username": "tmac-weylin",859 "name": "Yuezhipeng",860 "avatar_template": "/user_avatar/discuss.pytorch.org/tmac-weylin/{size}/74959_2.png",861 "trust_level": 0862 }863 },864 {865 "extras": null,866 "description": "Frequent Poster",867 "user": {868 "id": 82103,869 "username": "kannangce",870 "name": "Kannan Ramamoorthy",871 "avatar_template": "/user_avatar/discuss.pytorch.org/kannangce/{size}/75115_2.png",872 "trust_level": 0873 }874 },875 {876 "extras": null,877 "description": "Frequent Poster",878 "user": {879 "id": 81170,880 "username": "Naeem_Khoshnevis",881 "name": "Naeem Khoshnevis",882 "avatar_template": "/user_avatar/discuss.pytorch.org/naeem_khoshnevis/{size}/74236_2.png",883 "trust_level": 1884 }885 },886 {887 "extras": null,888 "description": "Frequent Poster",889 "user": {890 "id": 3534,891 "username": "ptrblck",892 "name": "",893 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",894 "admin": true,895 "moderator": true,896 "trust_level": 2897 }898 },899 {900 "extras": "latest",901 "description": "Most Recent Poster",902 "user": {903 "id": 84527,904 "username": "Jing-Jing_Li",905 "name": "Jing-Jing Li",906 "avatar_template": "/user_avatar/discuss.pytorch.org/jing-jing_li/{size}/77224_2.png",907 "trust_level": 0908 }909 }910 ]911 },912 {913 "fancy_title": "Cuda installation on M3",914 "id": 215620,915 "title": "Cuda installation on M3",916 "slug": "cuda-installation-on-m3",917 "posts_count": 11,918 "reply_count": 8,919 "highest_post_number": 11,920 "image_url": null,921 "created_at": "2025-01-20T11:23:16.903Z",922 "last_posted_at": "2025-01-20T23:48:31.539Z",923 "bumped": true,924 "bumped_at": "2025-01-20T23:48:31.539Z",925 "archetype": "regular",926 "unseen": false,927 "pinned": false,928 "unpinned": null,929 "visible": true,930 "closed": false,931 "archived": false,932 "bookmarked": null,933 "liked": null,934 "tags_descriptions": {},935 "like_count": 0,936 "views": 2414,937 "category_id": 32,938 "featured_link": null,939 "has_accepted_answer": false,940 "posters": [941 {942 "extras": null,943 "description": "Original Poster",944 "user": {945 "id": 82193,946 "username": "djdoogle123",947 "name": "Djdoogle123",948 "avatar_template": "/user_avatar/discuss.pytorch.org/djdoogle123/{size}/75207_2.png",949 "trust_level": 1950 }951 },952 {953 "extras": "latest",954 "description": "Most Recent Poster",955 "user": {956 "id": 3534,957 "username": "ptrblck",958 "name": "",959 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",960 "admin": true,961 "moderator": true,962 "trust_level": 2963 }964 }965 ]966 },967 {968 "fancy_title": "Segmentation error on MacOS because of OpenMP",969 "id": 221141,970 "title": "Segmentation error on MacOS because of OpenMP",971 "slug": "segmentation-error-on-macos-because-of-openmp",972 "posts_count": 1,973 "reply_count": 0,974 "highest_post_number": 1,975 "image_url": null,976 "created_at": "2025-06-29T18:23:50.152Z",977 "last_posted_at": "2025-06-29T18:23:50.199Z",978 "bumped": true,979 "bumped_at": "2025-06-29T18:23:50.199Z",980 "archetype": "regular",981 "unseen": false,982 "pinned": false,983 "unpinned": null,984 "visible": true,985 "closed": false,986 "archived": false,987 "bookmarked": null,988 "liked": null,989 "tags_descriptions": {},990 "like_count": 0,991 "views": 74,992 "category_id": 32,993 "featured_link": null,994 "has_accepted_answer": false,995 "posters": [996 {997 "extras": "latest single",998 "description": "Original Poster, Most Recent Poster",999 "user": {1000 "id": 84874,1001 "username": "lorentzenchr",1002 "name": "Christian Lorentzen",1003 "avatar_template": "/user_avatar/discuss.pytorch.org/lorentzenchr/{size}/77530_2.png",1004 "trust_level": 01005 }1006 }1007 ]1008 }1009 ],1010 "tags_descriptions": {},1011 "fancy_title": "Apple Silicon & torchrun: Distributed package doesn’t have NCCL built in",1012 "id": 201315,1013 "title": "Apple Silicon & torchrun: Distributed package doesn't have NCCL built in",1014 "posts_count": 4,1015 "created_at": "2024-04-21T15:25:46.259Z",1016 "views": 1198,1017 "reply_count": 2,1018 "like_count": 0,1019 "last_posted_at": "2025-01-31T16:12:57.674Z",1020 "visible": true,1021 "closed": false,1022 "archived": false,1023 "has_summary": false,1024 "archetype": "regular",1025 "slug": "apple-silicon-torchrun-distributed-package-doesnt-have-nccl-built-in",1026 "category_id": 32,1027 "word_count": 838,1028 "deleted_at": null,1029 "user_id": 75191,1030 "featured_link": null,1031 "pinned_globally": false,1032 "pinned_at": null,1033 "pinned_until": null,1034 "image_url": null,1035 "slow_mode_seconds": 0,1036 "draft": null,1037 "draft_key": "topic_201315",1038 "draft_sequence": null,1039 "unpinned": null,1040 "pinned": false,1041 "current_post_number": 1,1042 "highest_post_number": 4,1043 "deleted_by": null,1044 "actions_summary": [1045 {1046 "id": 4,1047 "count": 0,1048 "hidden": false,1049 "can_act": false1050 },1051 {1052 "id": 8,1053 "count": 0,1054 "hidden": false,1055 "can_act": false1056 },1057 {1058 "id": 10,1059 "count": 0,1060 "hidden": false,1061 "can_act": false1062 },1063 {1064 "id": 7,1065 "count": 0,1066 "hidden": false,1067 "can_act": false1068 }1069 ],1070 "chunk_size": 20,1071 "bookmarked": false,1072 "topic_timer": null,1073 "message_bus_last_id": 0,1074 "participant_count": 3,1075 "show_read_indicator": false,1076 "thumbnails": null,1077 "slow_mode_enabled_until": null,1078 "can_vote": false,1079 "vote_count": 0,1080 "user_voted": false,1081 "discourse_zendesk_plugin_zendesk_id": null,1082 "discourse_zendesk_plugin_zendesk_url": "https://your-url.zendesk.com/agent/tickets/",1083 "details": {1084 "can_edit": false,1085 "notification_level": 1,1086 "participants": [1087 {1088 "id": 75191,1089 "username": "vortec",1090 "name": "",1091 "avatar_template": "/user_avatar/discuss.pytorch.org/vortec/{size}/66340_2.png",1092 "post_count": 2,1093 "primary_group_name": null,1094 "flair_name": null,1095 "flair_url": null,1096 "flair_color": null,1097 "flair_bg_color": null,1098 "flair_group_id": null,1099 "trust_level": 01100 },1101 {1102 "id": 3534,1103 "username": "ptrblck",1104 "name": "",1105 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",1106 "post_count": 1,1107 "primary_group_name": null,1108 "flair_name": null,1109 "flair_url": null,1110 "flair_color": null,1111 "flair_bg_color": null,1112 "flair_group_id": null,1113 "admin": true,1114 "moderator": true,1115 "trust_level": 21116 },1117 {1118 "id": 82420,1119 "username": "scottnzuk",1120 "name": "Scott",1121 "avatar_template": "/letter_avatar_proxy/v4/letter/s/e47c2d/{size}.png",1122 "post_count": 1,1123 "primary_group_name": null,1124 "flair_name": null,1125 "flair_url": null,1126 "flair_color": null,1127 "flair_bg_color": null,1128 "flair_group_id": null,1129 "trust_level": 01130 }1131 ],1132 "created_by": {1133 "id": 75191,1134 "username": "vortec",1135 "name": "",1136 "avatar_template": "/user_avatar/discuss.pytorch.org/vortec/{size}/66340_2.png"1137 },1138 "last_poster": {1139 "id": 82420,1140 "username": "scottnzuk",1141 "name": "Scott",1142 "avatar_template": "/letter_avatar_proxy/v4/letter/s/e47c2d/{size}.png"1143 },1144 "links": [1145 {1146 "url": "https://github.com/ml-explore/mlx-examples/tree/main/whisper",1147 "title": "mlx-examples/whisper at main · ml-explore/mlx-examples · GitHub",1148 "internal": false,1149 "attachment": false,1150 "reflection": false,1151 "clicks": 15,1152 "user_id": 82420,1153 "domain": "github.com",1154 "root_domain": "github.com"1155 }1156 ]1157 },1158 "bookmarks": []1159 },1160 {1161 "post_stream": {1162 "posts": [1163 {1164 "id": 464361,1165 "name": null,1166 "username": "Dirk10001",1167 "avatar_template": "/letter_avatar_proxy/v4/letter/d/c37758/{size}.png",1168 "created_at": "2025-01-30T14:25:47.929Z",1169 "cooked": "<p>Hello all,</p>\n<p>I want to subtract tensor1 [1,384] from tensor2 [ 1, 384, 56, 56]. The images of tensor2 should be subtracted by the scalar in tensor1.</p>\n<pre><code class=\"lang-auto\">result= tensor2 - tensor1\nresult= at::sub(tensor2-tensor1)\n</code></pre>\n<p>Both failed. I do not want to blow tensor1 to [1,384,56,56].<br>\nMany thanks for your help.</p>\n<h1><a name=\"p-464361-tensor1view-1-doesnt-work-1\" class=\"anchor\" href=\"#p-464361-tensor1view-1-doesnt-work-1\"></a>tensor1.view(-1) doesn’t work</h1>",1170 "post_number": 1,1171 "post_type": 1,1172 "posts_count": 4,1173 "updated_at": "2025-01-30T14:28:23.979Z",1174 "reply_count": 0,1175 "reply_to_post_number": null,1176 "quote_count": 0,1177 "incoming_link_count": 8,1178 "reads": 5,1179 "readers_count": 4,1180 "score": 36.0,1181 "yours": false,1182 "topic_id": 216053,1183 "topic_slug": "tensor-subtraction",1184 "display_username": null,1185 "primary_group_name": null,1186 "flair_name": null,1187 "flair_url": null,1188 "flair_bg_color": null,1189 "flair_color": null,1190 "flair_group_id": null,1191 "badges_granted": [],1192 "version": 1,1193 "can_edit": false,1194 "can_delete": false,1195 "can_recover": false,1196 "can_see_hidden_post": false,1197 "can_wiki": false,1198 "read": true,1199 "user_title": null,1200 "bookmarked": false,