Anurag1734/cuda-error-resolution-analysis
07
1[2 {3 "post_stream": {4 "posts": [5 {6 "id": 360871,7 "name": "",8 "username": "danielmanu93",9 "avatar_template": "/letter_avatar_proxy/v4/letter/d/a88e4f/{size}.png",10 "created_at": "2022-08-09T17:32:12.629Z",11 "cooked": "<p>Hi All I have data from a dataloader which I pass to my model for prediction. I am trying to normalize my prediction within a certain range using <code>T.tonumpy_denormalize</code>, however I get this error <code>RuntimeError: Can't call numpy() on Tensor that requires grad. Use tensor.detach().numpy() instead</code>. I have tried to use <code>.detach().numpy()</code> on my prediction tensor but I get this error afterwards <code>AttributeError: 'numpy.ndarray' object has no attribute 'cpu'</code>. Initially I wasn’t getting any of these error but I changed my code structure and started getting this error, even though its the same code as before. Below is my sample code for demonstration:</p>\n<pre><code class=\"lang-auto\">val = \"./data.txt\"\nval_data = torch.load(val)\ndataloader = torch.utils.data.DataLoader(val_data, batch_size=1, shuffle=True)\n\ndef evaluate(model):\n label_min = 1500\n label_max = 4500\n model.eval()\n data, label = iter(dataloader).next()\n pred = model(data)\n pred_np = T.tonumpy_denormalize(pred, label_min, label_max, exp=False) # I get the error on this line. #\n# previously wasn't getting this error, I just changed the code structure by moving some parts upwards and got this error. #\n\n# full error is below #\npred_np = T.tonumpy_denormalize(pred, label_min, label_max, exp=False)\n File \"/home/pi/Desktop/exp/fcnvmb/transforms.py\", line 97, in tonumpy_denormalize\n vid = minmax_denormalize(vid.cpu().numpy(), vmin, vmax, scale)\nRuntimeError: Can't call numpy() on Tensor that requires grad. Use tensor.detach().numpy() instead.\n</code></pre>",12 "post_number": 1,13 "post_type": 1,14 "posts_count": 4,15 "updated_at": "2022-08-09T17:35:32.809Z",16 "reply_count": 0,17 "reply_to_post_number": null,18 "quote_count": 0,19 "incoming_link_count": 7446,20 "reads": 29,21 "readers_count": 28,22 "score": 37070.8,23 "yours": false,24 "topic_id": 158743,25 "topic_slug": "runtimeerror-cant-call-numpy-on-tensor-that-requires-grad-use-tensor-detach-numpy-instead",26 "display_username": "",27 "primary_group_name": null,28 "flair_name": null,29 "flair_url": null,30 "flair_bg_color": null,31 "flair_color": null,32 "flair_group_id": null,33 "badges_granted": [],34 "version": 1,35 "can_edit": false,36 "can_delete": false,37 "can_recover": false,38 "can_see_hidden_post": false,39 "can_wiki": false,40 "read": true,41 "user_title": null,42 "bookmarked": false,43 "actions_summary": [],44 "moderator": false,45 "admin": false,46 "staff": false,47 "user_id": 40263,48 "hidden": false,49 "trust_level": 1,50 "deleted_at": null,51 "user_deleted": false,52 "edit_reason": null,53 "can_view_edit_history": true,54 "wiki": false,55 "post_url": "/t/runtimeerror-cant-call-numpy-on-tensor-that-requires-grad-use-tensor-detach-numpy-instead/158743/1",56 "can_accept_answer": false,57 "can_unaccept_answer": false,58 "accepted_answer": false,59 "topic_accepted_answer": true,60 "can_vote": false61 },62 {63 "id": 360885,64 "name": "",65 "username": "ptrblck",66 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",67 "created_at": "2022-08-09T18:13:31.460Z",68 "cooked": "<p>You are trying to call <code>numpy()</code> on a tensor, which is still attached to a computation graph, and which is disallowed without calling <code>detach()</code> explicitly. This error is used to make sure the users are aware that this tensor is being detached (and thus they have to use the <code>detach()</code> call explicitly) in the <code>numpy()</code> operation.<br>\nI don’t know why the error wasn’t raised previously, but <code>pred</code> should have been attached to a computation graph before as well.</p>",69 "post_number": 2,70 "post_type": 1,71 "posts_count": 4,72 "updated_at": "2022-08-09T18:13:31.460Z",73 "reply_count": 0,74 "reply_to_post_number": null,75 "quote_count": 0,76 "incoming_link_count": 76,77 "reads": 24,78 "readers_count": 23,79 "score": 384.8,80 "yours": false,81 "topic_id": 158743,82 "topic_slug": "runtimeerror-cant-call-numpy-on-tensor-that-requires-grad-use-tensor-detach-numpy-instead",83 "display_username": "",84 "primary_group_name": null,85 "flair_name": null,86 "flair_url": null,87 "flair_bg_color": null,88 "flair_color": null,89 "flair_group_id": null,90 "badges_granted": [],91 "version": 1,92 "can_edit": false,93 "can_delete": false,94 "can_recover": false,95 "can_see_hidden_post": false,96 "can_wiki": false,97 "read": true,98 "user_title": "",99 "bookmarked": false,100 "actions_summary": [],101 "moderator": true,102 "admin": true,103 "staff": true,104 "user_id": 3534,105 "hidden": false,106 "trust_level": 2,107 "deleted_at": null,108 "user_deleted": false,109 "edit_reason": null,110 "can_view_edit_history": true,111 "wiki": false,112 "post_url": "/t/runtimeerror-cant-call-numpy-on-tensor-that-requires-grad-use-tensor-detach-numpy-instead/158743/2",113 "can_accept_answer": false,114 "can_unaccept_answer": false,115 "accepted_answer": true,116 "topic_accepted_answer": true117 },118 {119 "id": 360889,120 "name": "",121 "username": "danielmanu93",122 "avatar_template": "/letter_avatar_proxy/v4/letter/d/a88e4f/{size}.png",123 "created_at": "2022-08-09T18:34:38.170Z",124 "cooked": "<p>Okay I figured. I just used <code>pred.detach()</code> and it worked. Thanks.</p>",125 "post_number": 4,126 "post_type": 1,127 "posts_count": 4,128 "updated_at": "2022-08-09T18:34:38.170Z",129 "reply_count": 0,130 "reply_to_post_number": 3,131 "quote_count": 0,132 "incoming_link_count": 21,133 "reads": 23,134 "readers_count": 22,135 "score": 109.6,136 "yours": false,137 "topic_id": 158743,138 "topic_slug": "runtimeerror-cant-call-numpy-on-tensor-that-requires-grad-use-tensor-detach-numpy-instead",139 "display_username": "",140 "primary_group_name": null,141 "flair_name": null,142 "flair_url": null,143 "flair_bg_color": null,144 "flair_color": null,145 "flair_group_id": null,146 "badges_granted": [],147 "version": 1,148 "can_edit": false,149 "can_delete": false,150 "can_recover": false,151 "can_see_hidden_post": false,152 "can_wiki": false,153 "read": true,154 "user_title": null,155 "reply_to_user": {156 "id": 40263,157 "username": "danielmanu93",158 "name": "",159 "avatar_template": "/letter_avatar_proxy/v4/letter/d/a88e4f/{size}.png"160 },161 "bookmarked": false,162 "actions_summary": [],163 "moderator": false,164 "admin": false,165 "staff": false,166 "user_id": 40263,167 "hidden": false,168 "trust_level": 1,169 "deleted_at": null,170 "user_deleted": false,171 "edit_reason": null,172 "can_view_edit_history": true,173 "wiki": false,174 "post_url": "/t/runtimeerror-cant-call-numpy-on-tensor-that-requires-grad-use-tensor-detach-numpy-instead/158743/4",175 "can_accept_answer": false,176 "can_unaccept_answer": false,177 "accepted_answer": false,178 "topic_accepted_answer": true179 },180 {181 "id": 437993,182 "name": "Sishou Zhou",183 "username": "Sishou_Zhou",184 "avatar_template": "/user_avatar/discuss.pytorch.org/sishou_zhou/{size}/61201_2.png",185 "created_at": "2024-04-02T12:03:59.966Z",186 "cooked": "<p>I try to use a neural network to estimate parameters. When I loop my program 10000 times, and it runs without error. But the result is rather bad, Then I try to loop the program 100,000,000 times, it shows this error, who knows the reason? I feel very wired.</p>",187 "post_number": 5,188 "post_type": 1,189 "posts_count": 4,190 "updated_at": "2024-04-02T12:03:59.966Z",191 "reply_count": 0,192 "reply_to_post_number": null,193 "quote_count": 0,194 "incoming_link_count": 10,195 "reads": 12,196 "readers_count": 11,197 "score": 52.4,198 "yours": false,199 "topic_id": 158743,200 "topic_slug": "runtimeerror-cant-call-numpy-on-tensor-that-requires-grad-use-tensor-detach-numpy-instead",201 "display_username": "Sishou Zhou",202 "primary_group_name": null,203 "flair_name": null,204 "flair_url": null,205 "flair_bg_color": null,206 "flair_color": null,207 "flair_group_id": null,208 "badges_granted": [],209 "version": 1,210 "can_edit": false,211 "can_delete": false,212 "can_recover": false,213 "can_see_hidden_post": false,214 "can_wiki": false,215 "read": true,216 "user_title": null,217 "bookmarked": false,218 "actions_summary": [],219 "moderator": false,220 "admin": false,221 "staff": false,222 "user_id": 74640,223 "hidden": false,224 "trust_level": 1,225 "deleted_at": null,226 "user_deleted": false,227 "edit_reason": null,228 "can_view_edit_history": true,229 "wiki": false,230 "post_url": "/t/runtimeerror-cant-call-numpy-on-tensor-that-requires-grad-use-tensor-detach-numpy-instead/158743/5",231 "can_accept_answer": false,232 "can_unaccept_answer": false,233 "accepted_answer": false,234 "topic_accepted_answer": true235 }236 ],237 "stream": [238 360871,239 360885,240 360889,241 437993242 ]243 },244 "timeline_lookup": [245 [246 1,247 1173248 ],249 [250 4,251 571252 ]253 ],254 "suggested_topics": [255 {256 "fancy_title": "Torch.onnx.export() - No ONNX function for OpOverload",257 "id": 216624,258 "title": "Torch.onnx.export() - No ONNX function for OpOverload",259 "slug": "torch-onnx-export-no-onnx-function-for-opoverload",260 "posts_count": 3,261 "reply_count": 0,262 "highest_post_number": 3,263 "image_url": null,264 "created_at": "2025-02-13T10:07:22.506Z",265 "last_posted_at": "2025-02-13T20:11:56.474Z",266 "bumped": true,267 "bumped_at": "2025-02-13T20:11:56.474Z",268 "archetype": "regular",269 "unseen": false,270 "pinned": false,271 "unpinned": null,272 "visible": true,273 "closed": false,274 "archived": false,275 "bookmarked": null,276 "liked": null,277 "tags_descriptions": {},278 "like_count": 0,279 "views": 455,280 "category_id": 1,281 "featured_link": null,282 "has_accepted_answer": false,283 "posters": [284 {285 "extras": "latest",286 "description": "Original Poster, Most Recent Poster",287 "user": {288 "id": 82662,289 "username": "MSK28",290 "name": "",291 "avatar_template": "/letter_avatar_proxy/v4/letter/m/ce7236/{size}.png",292 "trust_level": 0293 }294 },295 {296 "extras": null,297 "description": "Frequent Poster",298 "user": {299 "id": 29009,300 "username": "take-cheeze",301 "name": "Takeshi Watanabe",302 "avatar_template": "/user_avatar/discuss.pytorch.org/take-cheeze/{size}/21811_2.png",303 "trust_level": 1304 }305 }306 ]307 },308 {309 "fancy_title": "Checkpoint() doesn’t work with use_reentrant=False",310 "id": 213226,311 "title": "Checkpoint() doesn't work with use_reentrant=False",312 "slug": "checkpoint-doesnt-work-with-use-reentrant-false",313 "posts_count": 2,314 "reply_count": 0,315 "highest_post_number": 2,316 "image_url": null,317 "created_at": "2024-11-20T19:13:39.494Z",318 "last_posted_at": "2024-12-03T14:29:05.733Z",319 "bumped": true,320 "bumped_at": "2024-12-03T14:29:05.733Z",321 "archetype": "regular",322 "unseen": false,323 "pinned": false,324 "unpinned": null,325 "visible": true,326 "closed": false,327 "archived": false,328 "bookmarked": null,329 "liked": null,330 "tags_descriptions": {},331 "like_count": 0,332 "views": 629,333 "category_id": 1,334 "featured_link": null,335 "has_accepted_answer": false,336 "posters": [337 {338 "extras": "latest single",339 "description": "Original Poster, Most Recent Poster",340 "user": {341 "id": 81024,342 "username": "kwohlfahrt",343 "name": "Kai Wohlfahrt",344 "avatar_template": "/letter_avatar_proxy/v4/letter/k/c77e96/{size}.png",345 "trust_level": 1346 }347 }348 ]349 },350 {351 "fancy_title": "What is the use of tensor.share_memory_()?",352 "id": 213481,353 "title": "What is the use of tensor.share_memory_()?",354 "slug": "what-is-the-use-of-tensor-share-memory",355 "posts_count": 1,356 "reply_count": 0,357 "highest_post_number": 1,358 "image_url": null,359 "created_at": "2024-11-26T16:17:26.814Z",360 "last_posted_at": "2024-11-26T16:17:26.866Z",361 "bumped": true,362 "bumped_at": "2024-11-26T16:17:26.866Z",363 "archetype": "regular",364 "unseen": false,365 "pinned": false,366 "unpinned": null,367 "visible": true,368 "closed": false,369 "archived": false,370 "bookmarked": null,371 "liked": null,372 "tags_descriptions": {},373 "like_count": 0,374 "views": 45,375 "category_id": 1,376 "featured_link": null,377 "has_accepted_answer": false,378 "posters": [379 {380 "extras": "latest single",381 "description": "Original Poster, Most Recent Poster",382 "user": {383 "id": 81147,384 "username": "dcusmeb",385 "name": "",386 "avatar_template": "/user_avatar/discuss.pytorch.org/dcusmeb/{size}/72677_2.png",387 "trust_level": 0388 }389 }390 ]391 },392 {393 "fancy_title": "Attempted to use an uninitialized parameter in <method ‘element_size’ of ‘torch._C._TensorBase’ objects>",394 "id": 215970,395 "title": "Attempted to use an uninitialized parameter in <method 'element_size' of 'torch._C._TensorBase' objects>",396 "slug": "attempted-to-use-an-uninitialized-parameter-in-method-element-size-of-torch-c-tensorbase-objects",397 "posts_count": 1,398 "reply_count": 0,399 "highest_post_number": 1,400 "image_url": null,401 "created_at": "2025-01-28T02:56:22.218Z",402 "last_posted_at": "2025-01-28T02:56:22.259Z",403 "bumped": true,404 "bumped_at": "2025-01-28T03:03:12.718Z",405 "archetype": "regular",406 "unseen": false,407 "pinned": false,408 "unpinned": null,409 "visible": true,410 "closed": false,411 "archived": false,412 "bookmarked": null,413 "liked": null,414 "tags_descriptions": {},415 "like_count": 0,416 "views": 157,417 "category_id": 1,418 "featured_link": null,419 "has_accepted_answer": false,420 "posters": [421 {422 "extras": "latest single",423 "description": "Original Poster, Most Recent Poster",424 "user": {425 "id": 31826,426 "username": "acmilannesta",427 "name": "",428 "avatar_template": "/user_avatar/discuss.pytorch.org/acmilannesta/{size}/24428_2.png",429 "trust_level": 1430 }431 }432 ]433 },434 {435 "fancy_title": "Question About the Design of Serialization Function Signature in BackendMeta.",436 "id": 216745,437 "title": "Question About the Design of Serialization Function Signature in BackendMeta.",438 "slug": "question-about-the-design-of-serialization-function-signature-in-backendmeta",439 "posts_count": 1,440 "reply_count": 0,441 "highest_post_number": 1,442 "image_url": null,443 "created_at": "2025-02-16T15:50:51.238Z",444 "last_posted_at": "2025-02-16T15:50:51.270Z",445 "bumped": true,446 "bumped_at": "2025-02-16T15:50:51.270Z",447 "archetype": "regular",448 "unseen": false,449 "pinned": false,450 "unpinned": null,451 "visible": true,452 "closed": false,453 "archived": false,454 "bookmarked": null,455 "liked": null,456 "tags_descriptions": {},457 "like_count": 0,458 "views": 20,459 "category_id": 1,460 "featured_link": null,461 "has_accepted_answer": false,462 "posters": [463 {464 "extras": "latest single",465 "description": "Original Poster, Most Recent Poster",466 "user": {467 "id": 82722,468 "username": "Seungchul_Han",469 "name": "Seungchul Han",470 "avatar_template": "/user_avatar/discuss.pytorch.org/seungchul_han/{size}/74567_2.png",471 "trust_level": 0472 }473 }474 ]475 }476 ],477 "tags_descriptions": {},478 "fancy_title": "RuntimeError: Can’t call numpy() on Tensor that requires grad. Use tensor.detach().numpy() instead",479 "id": 158743,480 "title": "RuntimeError: Can't call numpy() on Tensor that requires grad. Use tensor.detach().numpy() instead",481 "posts_count": 4,482 "created_at": "2022-08-09T17:32:12.568Z",483 "views": 6659,484 "reply_count": 2,485 "like_count": 0,486 "last_posted_at": "2024-04-02T12:03:59.966Z",487 "visible": true,488 "closed": false,489 "archived": false,490 "has_summary": false,491 "archetype": "regular",492 "slug": "runtimeerror-cant-call-numpy-on-tensor-that-requires-grad-use-tensor-detach-numpy-instead",493 "category_id": 1,494 "word_count": 374,495 "deleted_at": null,496 "user_id": 40263,497 "featured_link": null,498 "pinned_globally": false,499 "pinned_at": null,500 "pinned_until": null,501 "image_url": null,502 "slow_mode_seconds": 0,503 "draft": null,504 "draft_key": "topic_158743",505 "draft_sequence": null,506 "unpinned": null,507 "pinned": false,508 "current_post_number": 1,509 "highest_post_number": 5,510 "deleted_by": null,511 "actions_summary": [512 {513 "id": 4,514 "count": 0,515 "hidden": false,516 "can_act": false517 },518 {519 "id": 8,520 "count": 0,521 "hidden": false,522 "can_act": false523 },524 {525 "id": 10,526 "count": 0,527 "hidden": false,528 "can_act": false529 },530 {531 "id": 7,532 "count": 0,533 "hidden": false,534 "can_act": false535 }536 ],537 "chunk_size": 20,538 "bookmarked": false,539 "topic_timer": null,540 "message_bus_last_id": 0,541 "participant_count": 3,542 "show_read_indicator": false,543 "thumbnails": null,544 "slow_mode_enabled_until": null,545 "accepted_answer": {546 "post_number": 2,547 "username": "ptrblck",548 "name": "",549 "excerpt": "You are trying to call numpy() on a tensor, which is still attached to a computation graph, and which is disallowed without calling detach() explicitly. This error is used to make sure the users are aware that this tensor is being detached (and thus they have to use the detach() call explicitly) in …"550 },551 "can_vote": false,552 "vote_count": 0,553 "user_voted": false,554 "discourse_zendesk_plugin_zendesk_id": null,555 "discourse_zendesk_plugin_zendesk_url": "https://your-url.zendesk.com/agent/tickets/",556 "details": {557 "can_edit": false,558 "notification_level": 1,559 "participants": [560 {561 "id": 40263,562 "username": "danielmanu93",563 "name": "",564 "avatar_template": "/letter_avatar_proxy/v4/letter/d/a88e4f/{size}.png",565 "post_count": 2,566 "primary_group_name": null,567 "flair_name": null,568 "flair_url": null,569 "flair_color": null,570 "flair_bg_color": null,571 "flair_group_id": null,572 "trust_level": 1573 },574 {575 "id": 3534,576 "username": "ptrblck",577 "name": "",578 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",579 "post_count": 1,580 "primary_group_name": null,581 "flair_name": null,582 "flair_url": null,583 "flair_color": null,584 "flair_bg_color": null,585 "flair_group_id": null,586 "admin": true,587 "moderator": true,588 "trust_level": 2589 },590 {591 "id": 74640,592 "username": "Sishou_Zhou",593 "name": "Sishou Zhou",594 "avatar_template": "/user_avatar/discuss.pytorch.org/sishou_zhou/{size}/61201_2.png",595 "post_count": 1,596 "primary_group_name": null,597 "flair_name": null,598 "flair_url": null,599 "flair_color": null,600 "flair_bg_color": null,601 "flair_group_id": null,602 "trust_level": 1603 }604 ],605 "created_by": {606 "id": 40263,607 "username": "danielmanu93",608 "name": "",609 "avatar_template": "/letter_avatar_proxy/v4/letter/d/a88e4f/{size}.png"610 },611 "last_poster": {612 "id": 74640,613 "username": "Sishou_Zhou",614 "name": "Sishou Zhou",615 "avatar_template": "/user_avatar/discuss.pytorch.org/sishou_zhou/{size}/61201_2.png"616 }617 },618 "bookmarks": []619 },620 {621 "post_stream": {622 "posts": [623 {624 "id": 400182,625 "name": "nitish kumar",626 "username": "nitish_kumar",627 "avatar_template": "/user_avatar/discuss.pytorch.org/nitish_kumar/{size}/59841_2.png",628 "created_at": "2023-04-30T04:43:53.011Z",629 "cooked": "<p>My use case requires me to train a list of models together at the same time. This is a heavily simplified version of my loss function</p>\n<pre><code class=\"lang-auto\">def loss_function(inputs, targets, models):\n for model in models:\n outputs = model(inputs)\n loss_std += F.cross_entropy(outputs, targets)\n loss_std = loss_std/len(models)\n return loss_std\n</code></pre>\n<p>I was attempting to vectorize the model using something like this</p>\n<pre><code class=\"lang-auto\">from functorch import combine_state_for_ensemble, vmap\n\ndef loss_function(inputs, targets, models):\n fmodel, params, buffers = combine_state_for_ensemble(models)\n outputs = vmap(fmodel, in_dims=(0, 0, None))(params, buffers, inputs)\n loss_std = F.cross_entropy(outputs.mean(dim=0), targets)\n return loss_std\n</code></pre>\n<p>I’m training using resnet18 models, and it contains batch norm layers</p>\n<p>when tried to run the new vectorized loss function, I got the following error</p>\n<pre><code class=\"lang-auto\">Traceback (most recent call last):\n File \"/home/nitish/projects/Fast-Adv-Ensemble/train_adv_ensemble.py\", line 362, in <module>\n run()\n File \"/home/nitish/projects/Fast-Adv-Ensemble/train_adv_ensemble.py\", line 359, in run\n torch.multiprocessing.spawn(main_worker, nprocs=ngpus_per_node, join=True)\n File \"/home/nitish/mambaforge/envs/fast/lib/python3.9/site-packages/torch/multiprocessing/spawn.py\", line 240, in spawn\n return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')\n File \"/home/nitish/mambaforge/envs/fast/lib/python3.9/site-packages/torch/multiprocessing/spawn.py\", line 198, in start_processes\n while not context.join():\n File \"/home/nitish/mambaforge/envs/fast/lib/python3.9/site-packages/torch/multiprocessing/spawn.py\", line 160, in join\n raise ProcessRaisedException(msg, error_index, failed_process.pid)\ntorch.multiprocessing.spawn.ProcessRaisedException: \n\n-- Process 0 terminated with the following error:\nTraceback (most recent call last):\n File \"/home/nitish/mambaforge/envs/fast/lib/python3.9/site-packages/torch/multiprocessing/spawn.py\", line 69, in _wrap\n fn(i, *args)\n File \"/home/nitish/projects/Fast-Adv-Ensemble/train_adv_ensemble.py\", line 349, in main_worker\n train(nets, ema_nets, trainloader, optimizer, lr_scheduler, scaler, attack)\n File \"/home/nitish/projects/Fast-Adv-Ensemble/train_adv_ensemble.py\", line 137, in train\n loss, _, _ = trades_loss(\n File \"/home/nitish/projects/Fast-Adv-Ensemble/utils/loss.py\", line 75, in trades_loss\n nat_loss, nat_outputs = adp_loss(\n File \"/home/nitish/projects/Fast-Adv-Ensemble/utils/loss.py\", line 50, in adp_loss\n outputs = vmap(fmodel, in_dims=(0, 0, None))(params, buffers, inputs)\n File \"/home/nitish/mambaforge/envs/fast/lib/python3.9/site-packages/functorch/_src/vmap.py\", line 362, in wrapped\n return _flat_vmap(\n File \"/home/nitish/mambaforge/envs/fast/lib/python3.9/site-packages/functorch/_src/vmap.py\", line 35, in fn\n return f(*args, **kwargs)\n File \"/home/nitish/mambaforge/envs/fast/lib/python3.9/site-packages/functorch/_src/vmap.py\", line 489, in _flat_vmap\n batched_outputs = func(*batched_inputs, **kwargs)\n File \"/home/nitish/mambaforge/envs/fast/lib/python3.9/site-packages/torch/nn/modules/module.py\", line 1194, in _call_impl\n return forward_call(*input, **kwargs)\n File \"/home/nitish/mambaforge/envs/fast/lib/python3.9/site-packages/functorch/_src/make_functional.py\", line 282, in forward\n return self.stateless_model(*args, **kwargs)\n File \"/home/nitish/mambaforge/envs/fast/lib/python3.9/site-packages/torch/nn/modules/module.py\", line 1194, in _call_impl\n return forward_call(*input, **kwargs)\n File \"/home/nitish/mambaforge/envs/fast/lib/python3.9/site-packages/torch/nn/parallel/distributed.py\", line 1040, in forward\n output = self._run_ddp_forward(*inputs, **kwargs)\n File \"/home/nitish/mambaforge/envs/fast/lib/python3.9/site-packages/torch/nn/parallel/distributed.py\", line 1000, in _run_ddp_forward\n return module_to_run(*inputs[0], **kwargs[0])\n File \"/home/nitish/mambaforge/envs/fast/lib/python3.9/site-packages/torch/nn/modules/module.py\", line 1194, in _call_impl\n return forward_call(*input, **kwargs)\n File \"/home/nitish/projects/Fast-Adv-Ensemble/models/resnet.py\", line 235, in forward\n return self._forward_impl(x)\n File \"/home/nitish/projects/Fast-Adv-Ensemble/models/resnet.py\", line 219, in _forward_impl\n x = self.bn1(x)\n File \"/home/nitish/mambaforge/envs/fast/lib/python3.9/site-packages/torch/nn/modules/module.py\", line 1194, in _call_impl\n return forward_call(*input, **kwargs)\n File \"/home/nitish/mambaforge/envs/fast/lib/python3.9/site-packages/torch/nn/modules/batchnorm.py\", line 740, in forward\n return F.batch_norm(\n File \"/home/nitish/mambaforge/envs/fast/lib/python3.9/site-packages/torch/nn/functional.py\", line 2450, in batch_norm\n return torch.batch_norm(\nRuntimeError: NYI: querying is_contiguous inside of vmap for memory_format other than torch.contiguous_format\n</code></pre>\n<p>I’m guessing NYI means Not Yet Implemented. Is there a way to make it work for my case? I am currently using version 1.13.1. Is it implemented in 2.0.0 or in the nightly builds?</p>\n<p>Please let me know if there is a fix for this or an alternative approach I could try out to vectorize/ speedup the function.</p>",630 "post_number": 1,631 "post_type": 1,632 "posts_count": 10,633 "updated_at": "2023-04-30T04:43:53.011Z",634 "reply_count": 0,635 "reply_to_post_number": null,636 "quote_count": 0,637 "incoming_link_count": 1422,638 "reads": 45,639 "readers_count": 44,640 "score": 7049.0,641 "yours": false,642 "topic_id": 178854,643 "topic_slug": "using-vmap-to-train-an-ensemble-of-models-together-at-the-same-time",644 "display_username": "nitish kumar",645 "primary_group_name": null,646 "flair_name": null,647 "flair_url": null,648 "flair_bg_color": null,649 "flair_color": null,650 "flair_group_id": null,651 "badges_granted": [],652 "version": 1,653 "can_edit": false,654 "can_delete": false,655 "can_recover": false,656 "can_see_hidden_post": false,657 "can_wiki": false,658 "read": true,659 "user_title": null,660 "bookmarked": false,661 "actions_summary": [],662 "moderator": false,663 "admin": false,664 "staff": false,665 "user_id": 65558,666 "hidden": false,667 "trust_level": 1,668 "deleted_at": null,669 "user_deleted": false,670 "edit_reason": null,671 "can_view_edit_history": true,672 "wiki": false,673 "post_url": "/t/using-vmap-to-train-an-ensemble-of-models-together-at-the-same-time/178854/1",674 "can_accept_answer": false,675 "can_unaccept_answer": false,676 "accepted_answer": false,677 "topic_accepted_answer": null,678 "can_vote": false679 },680 {681 "id": 400401,682 "name": "",683 "username": "ptrblck",684 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",685 "created_at": "2023-05-02T14:46:42.863Z",686 "cooked": "<p>I don’t know if this method is already implemented but it’s certainly a good idea to try it out using the latest nightly release. In case you want to keep your older PyTorch installation you could create a new virtual environment and install the binaries there.</p>",687 "post_number": 2,688 "post_type": 1,689 "posts_count": 10,690 "updated_at": "2023-05-02T14:46:42.863Z",691 "reply_count": 1,692 "reply_to_post_number": null,693 "quote_count": 0,694 "incoming_link_count": 7,695 "reads": 37,696 "readers_count": 36,697 "score": 47.4,698 "yours": false,699 "topic_id": 178854,700 "topic_slug": "using-vmap-to-train-an-ensemble-of-models-together-at-the-same-time",701 "display_username": "",702 "primary_group_name": null,703 "flair_name": null,704 "flair_url": null,705 "flair_bg_color": null,706 "flair_color": null,707 "flair_group_id": null,708 "badges_granted": [],709 "version": 1,710 "can_edit": false,711 "can_delete": false,712 "can_recover": false,713 "can_see_hidden_post": false,714 "can_wiki": false,715 "read": true,716 "user_title": "",717 "bookmarked": false,718 "actions_summary": [],719 "moderator": true,720 "admin": true,721 "staff": true,722 "user_id": 3534,723 "hidden": false,724 "trust_level": 2,725 "deleted_at": null,726 "user_deleted": false,727 "edit_reason": null,728 "can_view_edit_history": true,729 "wiki": false,730 "post_url": "/t/using-vmap-to-train-an-ensemble-of-models-together-at-the-same-time/178854/2",731 "can_accept_answer": false,732 "can_unaccept_answer": false,733 "accepted_answer": false,734 "topic_accepted_answer": null735 },736 {737 "id": 401087,738 "name": "nitish kumar",739 "username": "nitish_kumar",740 "avatar_template": "/user_avatar/discuss.pytorch.org/nitish_kumar/{size}/59841_2.png",741 "created_at": "2023-05-08T00:57:20.120Z",742 "cooked": "<p>I have tested it out with the latest version, but the error persists. I am not familiar with what contiguous means in this context. My understanding was that classical format is contiguous, where data of the same channel are sequentially ordered, making it channel first memory and channel last the exact opposite. Can you help me understand what the error means in this context?</p>\n<p>Things I am also using along with the vmap function:</p>\n<ol>\n<li>Mixed precision training</li>\n<li>channel last memory format ( I have tried switching this off but i still run into the same error)</li>\n<li>DistributedDataParallel</li>\n</ol>",743 "post_number": 4,744 "post_type": 1,745 "posts_count": 10,746 "updated_at": "2023-05-08T00:57:46.749Z",747 "reply_count": 1,748 "reply_to_post_number": 2,749 "quote_count": 0,750 "incoming_link_count": 53,751 "reads": 37,752 "readers_count": 36,753 "score": 272.4,754 "yours": false,755 "topic_id": 178854,756 "topic_slug": "using-vmap-to-train-an-ensemble-of-models-together-at-the-same-time",757 "display_username": "nitish kumar",758 "primary_group_name": null,759 "flair_name": null,760 "flair_url": null,761 "flair_bg_color": null,762 "flair_color": null,763 "flair_group_id": null,764 "badges_granted": [],765 "version": 1,766 "can_edit": false,767 "can_delete": false,768 "can_recover": false,769 "can_see_hidden_post": false,770 "can_wiki": false,771 "read": true,772 "user_title": null,773 "reply_to_user": {774 "id": 3534,775 "username": "ptrblck",776 "name": "",777 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png"778 },779 "bookmarked": false,780 "actions_summary": [],781 "moderator": false,782 "admin": false,783 "staff": false,784 "user_id": 65558,785 "hidden": false,786 "trust_level": 1,787 "deleted_at": null,788 "user_deleted": false,789 "edit_reason": null,790 "can_view_edit_history": true,791 "wiki": false,792 "post_url": "/t/using-vmap-to-train-an-ensemble-of-models-together-at-the-same-time/178854/4",793 "can_accept_answer": false,794 "can_unaccept_answer": false,795 "accepted_answer": false,796 "topic_accepted_answer": null797 },798 {799 "id": 424823,800 "name": "",801 "username": "vani",802 "avatar_template": "/user_avatar/discuss.pytorch.org/vani/{size}/65683_2.png",803 "created_at": "2023-11-22T13:27:00.399Z",804 "cooked": "<p>Hello, are there any further developments on this? I have a very similar problem where I want to train a dozen of smaller networks on the same data, which should be done in parallel. jax offers a vmap function for this and a nice tutorial on how to do it on gpu, but what about pytorch? I’ve scanned the internet and it seems there are no clear solutions if you want to do it with pytorch… Any help would be appreciated!</p>",805 "post_number": 5,806 "post_type": 1,807 "posts_count": 10,808 "updated_at": "2023-11-22T13:27:00.399Z",809 "reply_count": 1,810 "reply_to_post_number": 4,811 "quote_count": 0,812 "incoming_link_count": 58,813 "reads": 27,814 "readers_count": 26,815 "score": 300.4,816 "yours": false,817 "topic_id": 178854,818 "topic_slug": "using-vmap-to-train-an-ensemble-of-models-together-at-the-same-time",819 "display_username": "",820 "primary_group_name": null,821 "flair_name": null,822 "flair_url": null,823 "flair_bg_color": null,824 "flair_color": null,825 "flair_group_id": null,826 "badges_granted": [],827 "version": 1,828 "can_edit": false,829 "can_delete": false,830 "can_recover": false,831 "can_see_hidden_post": false,832 "can_wiki": false,833 "read": true,834 "user_title": null,835 "reply_to_user": {836 "id": 65558,837 "username": "nitish_kumar",838 "name": "nitish kumar",839 "avatar_template": "/user_avatar/discuss.pytorch.org/nitish_kumar/{size}/59841_2.png"840 },841 "bookmarked": false,842 "actions_summary": [],843 "moderator": false,844 "admin": false,845 "staff": false,846 "user_id": 71170,847 "hidden": false,848 "trust_level": 1,849 "deleted_at": null,850 "user_deleted": false,851 "edit_reason": null,852 "can_view_edit_history": true,853 "wiki": false,854 "post_url": "/t/using-vmap-to-train-an-ensemble-of-models-together-at-the-same-time/178854/5",855 "can_accept_answer": false,856 "can_unaccept_answer": false,857 "accepted_answer": false,858 "topic_accepted_answer": null859 },860 {861 "id": 424905,862 "name": "",863 "username": "ptrblck",864 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",865 "created_at": "2023-11-22T22:51:55.952Z",866 "cooked": "<p>Training model ensembling via <code>vmap</code> should be supported in stable releases now as <a href=\"https://pytorch.org/tutorials/intermediate/ensembling.html\">this tutorial</a> indicates.</p>",867 "post_number": 6,868 "post_type": 1,869 "posts_count": 10,870 "updated_at": "2023-11-22T22:51:55.952Z",871 "reply_count": 2,872 "reply_to_post_number": 5,873 "quote_count": 0,874 "incoming_link_count": 9,875 "reads": 25,876 "readers_count": 24,877 "score": 75.0,878 "yours": false,879 "topic_id": 178854,880 "topic_slug": "using-vmap-to-train-an-ensemble-of-models-together-at-the-same-time",881 "display_username": "",882 "primary_group_name": null,883 "flair_name": null,884 "flair_url": null,885 "flair_bg_color": null,886 "flair_color": null,887 "flair_group_id": null,888 "badges_granted": [],889 "version": 1,890 "can_edit": false,891 "can_delete": false,892 "can_recover": false,893 "can_see_hidden_post": false,894 "can_wiki": false,895 "link_counts": [896 {897 "url": "https://pytorch.org/tutorials/intermediate/ensembling.html",898 "internal": false,899 "reflection": false,900 "title": "Model ensembling — PyTorch Tutorials 2.1.1+cu121 documentation",901 "clicks": 315902 }903 ],904 "read": true,905 "user_title": "",906 "reply_to_user": {907 "id": 71170,908 "username": "vani",909 "name": "",910 "avatar_template": "/user_avatar/discuss.pytorch.org/vani/{size}/65683_2.png"911 },912 "bookmarked": false,913 "actions_summary": [914 {915 "id": 2,916 "count": 1917 }918 ],919 "moderator": true,920 "admin": true,921 "staff": true,922 "user_id": 3534,923 "hidden": false,924 "trust_level": 2,925 "deleted_at": null,926 "user_deleted": false,927 "edit_reason": null,928 "can_view_edit_history": true,929 "wiki": false,930 "post_url": "/t/using-vmap-to-train-an-ensemble-of-models-together-at-the-same-time/178854/6",931 "can_accept_answer": false,932 "can_unaccept_answer": false,933 "accepted_answer": false,934 "topic_accepted_answer": null935 },936 {937 "id": 426265,938 "name": "",939 "username": "vani",940 "avatar_template": "/user_avatar/discuss.pytorch.org/vani/{size}/65683_2.png",941 "created_at": "2023-12-05T08:45:35.738Z",942 "cooked": "<p>Thanks, it works very good!</p>",943 "post_number": 7,944 "post_type": 1,945 "posts_count": 10,946 "updated_at": "2023-12-05T08:45:35.738Z",947 "reply_count": 0,948 "reply_to_post_number": 6,949 "quote_count": 0,950 "incoming_link_count": 5,951 "reads": 22,952 "readers_count": 21,953 "score": 29.4,954 "yours": false,955 "topic_id": 178854,956 "topic_slug": "using-vmap-to-train-an-ensemble-of-models-together-at-the-same-time",957 "display_username": "",958 "primary_group_name": null,959 "flair_name": null,960 "flair_url": null,961 "flair_bg_color": null,962 "flair_color": null,963 "flair_group_id": null,964 "badges_granted": [],965 "version": 1,966 "can_edit": false,967 "can_delete": false,968 "can_recover": false,969 "can_see_hidden_post": false,970 "can_wiki": false,971 "read": true,972 "user_title": null,973 "reply_to_user": {974 "id": 3534,975 "username": "ptrblck",976 "name": "",977 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png"978 },979 "bookmarked": false,980 "actions_summary": [],981 "moderator": false,982 "admin": false,983 "staff": false,984 "user_id": 71170,985 "hidden": false,986 "trust_level": 1,987 "deleted_at": null,988 "user_deleted": false,989 "edit_reason": null,990 "can_view_edit_history": true,991 "wiki": false,992 "post_url": "/t/using-vmap-to-train-an-ensemble-of-models-together-at-the-same-time/178854/7",993 "can_accept_answer": false,994 "can_unaccept_answer": false,995 "accepted_answer": false,996 "topic_accepted_answer": null997 },998 {999 "id": 427644,1000 "name": "",1001 "username": "vani",1002 "avatar_template": "/user_avatar/discuss.pytorch.org/vani/{size}/65683_2.png",1003 "created_at": "2023-12-19T12:47:57.068Z",1004 "cooked": "<p>Thanks once again for that. Do you have any idea how to backpropagate and update the gradients in a similar manner? When training my ensemble, the forward call is indeed much faster when using vmap, with respect to classical looping over models, but it seems that the <em>loss.backward()</em> and especially <em>optimizer.step()</em> calls are equally slow as before (to be expected). Is there any way to accelerate this using functional optimizers maybe? Thank you!</p>",1005 "post_number": 8,1006 "post_type": 1,1007 "posts_count": 10,1008 "updated_at": "2023-12-19T12:47:57.068Z",1009 "reply_count": 1,1010 "reply_to_post_number": 6,1011 "quote_count": 0,1012 "incoming_link_count": 13,1013 "reads": 20,1014 "readers_count": 19,1015 "score": 74.0,1016 "yours": false,1017 "topic_id": 178854,1018 "topic_slug": "using-vmap-to-train-an-ensemble-of-models-together-at-the-same-time",1019 "display_username": "",1020 "primary_group_name": null,1021 "flair_name": null,1022 "flair_url": null,1023 "flair_bg_color": null,1024 "flair_color": null,1025 "flair_group_id": null,1026 "badges_granted": [],1027 "version": 1,1028 "can_edit": false,1029 "can_delete": false,1030 "can_recover": false,1031 "can_see_hidden_post": false,1032 "can_wiki": false,1033 "read": true,1034 "user_title": null,1035 "reply_to_user": {1036 "id": 3534,1037 "username": "ptrblck",1038 "name": "",1039 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png"1040 },1041 "bookmarked": false,1042 "actions_summary": [],1043 "moderator": false,1044 "admin": false,1045 "staff": false,1046 "user_id": 71170,1047 "hidden": false,1048 "trust_level": 1,1049 "deleted_at": null,1050 "user_deleted": false,1051 "edit_reason": null,1052 "can_view_edit_history": true,1053 "wiki": false,1054 "post_url": "/t/using-vmap-to-train-an-ensemble-of-models-together-at-the-same-time/178854/8",1055 "can_accept_answer": false,1056 "can_unaccept_answer": false,1057 "accepted_answer": false,1058 "topic_accepted_answer": null1059 },1060 {1061 "id": 427679,1062 "name": "Yu Feng",1063 "username": "Yu_Feng",1064 "avatar_template": "/user_avatar/discuss.pytorch.org/yu_feng/{size}/66369_2.png",1065 "created_at": "2023-12-19T21:03:20.276Z",1066 "cooked": "<p>I haven’t tried it myself as I just start working on vmap ensemble model idea. But maybe combine the loss of each individual model into a big loss function and do backward pass for all models together?</p>",1067 "post_number": 9,1068 "post_type": 1,1069 "posts_count": 10,1070 "updated_at": "2023-12-19T21:03:20.276Z",1071 "reply_count": 0,1072 "reply_to_post_number": 8,1073 "quote_count": 0,1074 "incoming_link_count": 11,1075 "reads": 19,1076 "readers_count": 18,1077 "score": 58.8,1078 "yours": false,1079 "topic_id": 178854,1080 "topic_slug": "using-vmap-to-train-an-ensemble-of-models-together-at-the-same-time",1081 "display_username": "Yu Feng",1082 "primary_group_name": null,1083 "flair_name": null,1084 "flair_url": null,1085 "flair_bg_color": null,1086 "flair_color": null,1087 "flair_group_id": null,1088 "badges_granted": [],1089 "version": 1,1090 "can_edit": false,1091 "can_delete": false,1092 "can_recover": false,1093 "can_see_hidden_post": false,1094 "can_wiki": false,1095 "read": true,1096 "user_title": null,1097 "reply_to_user": {1098 "id": 71170,1099 "username": "vani",1100 "name": "",1101 "avatar_template": "/user_avatar/discuss.pytorch.org/vani/{size}/65683_2.png"1102 },1103 "bookmarked": false,1104 "actions_summary": [],1105 "moderator": false,1106 "admin": false,1107 "staff": false,1108 "user_id": 71852,1109 "hidden": false,1110 "trust_level": 0,1111 "deleted_at": null,1112 "user_deleted": false,1113 "edit_reason": null,1114 "can_view_edit_history": true,1115 "wiki": false,1116 "post_url": "/t/using-vmap-to-train-an-ensemble-of-models-together-at-the-same-time/178854/9",1117 "can_accept_answer": false,1118 "can_unaccept_answer": false,1119 "accepted_answer": false,1120 "topic_accepted_answer": null1121 },1122 {1123 "id": 427716,1124 "name": "",1125 "username": "vani",1126 "avatar_template": "/user_avatar/discuss.pytorch.org/vani/{size}/65683_2.png",1127 "created_at": "2023-12-20T08:58:15.036Z",1128 "cooked": "<p>Hi Yu, I have done this, but I wonder if this could be accelerated somehow… I observe that optimizer.step() call takes quite some time (especially for wider ensemble member architectures). I know that functional optimizers exist (from torchopt) but I do not have any idea if this would be useful here or if they could vectorize optimizer.step() function… Btw, if you have troubles using vmap to train an ensemble, feel free to ask, I have implemented a working version of this…</p>",1129 "post_number": 10,1130 "post_type": 1,1131 "posts_count": 10,1132 "updated_at": "2023-12-20T08:58:15.036Z",1133 "reply_count": 1,1134 "reply_to_post_number": null,1135 "quote_count": 0,1136 "incoming_link_count": 5,1137 "reads": 19,1138 "readers_count": 18,1139 "score": 33.8,1140 "yours": false,1141 "topic_id": 178854,1142 "topic_slug": "using-vmap-to-train-an-ensemble-of-models-together-at-the-same-time",1143 "display_username": "",1144 "primary_group_name": null,1145 "flair_name": null,1146 "flair_url": null,1147 "flair_bg_color": null,1148 "flair_color": null,1149 "flair_group_id": null,1150 "badges_granted": [],1151 "version": 1,1152 "can_edit": false,1153 "can_delete": false,1154 "can_recover": false,1155 "can_see_hidden_post": false,1156 "can_wiki": false,1157 "read": true,1158 "user_title": null,1159 "bookmarked": false,1160 "actions_summary": [],1161 "moderator": false,1162 "admin": false,1163 "staff": false,1164 "user_id": 71170,1165 "hidden": false,1166 "trust_level": 1,1167 "deleted_at": null,1168 "user_deleted": false,1169 "edit_reason": null,1170 "can_view_edit_history": true,1171 "wiki": false,1172 "post_url": "/t/using-vmap-to-train-an-ensemble-of-models-together-at-the-same-time/178854/10",1173 "can_accept_answer": false,1174 "can_unaccept_answer": false,1175 "accepted_answer": false,1176 "topic_accepted_answer": null1177 },1178 {1179 "id": 437981,1180 "name": "Vignesh Gopakumar",1181 "username": "gitvicky",1182 "avatar_template": "/user_avatar/discuss.pytorch.org/gitvicky/{size}/34794_2.png",1183 "created_at": "2024-04-02T10:44:57.548Z",1184 "cooked": "<p>Hi, Would you be able to share your code that looks at this ? I am a bit unsure as to how to deploy the weights within the optimizer when using the ensembles.</p>",1185 "post_number": 11,1186 "post_type": 1,1187 "posts_count": 10,1188 "updated_at": "2024-04-02T10:49:17.616Z",1189 "reply_count": 0,1190 "reply_to_post_number": 10,1191 "quote_count": 0,1192 "incoming_link_count": 3,1193 "reads": 13,1194 "readers_count": 12,1195 "score": 17.6,1196 "yours": false,1197 "topic_id": 178854,1198 "topic_slug": "using-vmap-to-train-an-ensemble-of-models-together-at-the-same-time",1199 "display_username": "Vignesh Gopakumar",1200 "primary_group_name": null,