Anurag1734/cuda-error-resolution-analysis
07
1[2 {3 "post_stream": {4 "posts": [5 {6 "id": 167839,7 "name": "",8 "username": "ntenenz",9 "avatar_template": "/letter_avatar_proxy/v4/letter/n/258eb7/{size}.png",10 "created_at": "2020-02-21T04:46:04.743Z",11 "cooked": "<p>In the following code snippet, the two assertions hold:</p>\n<pre><code class=\"lang-auto\">import torch\nimport torch.nn as nn\n\nconv = nn.Conv2d(8, 8, 3)\nd = {k: v for k, v in conv.named_parameters()}\n\n# torch.__future__.set_overwrite_module_params_on_conversion(True)\nassert all(d[k] is v for k, v in conv.named_parameters())\nconv.to('cuda:0')\nassert all(d[k] is v for k, v in conv.named_parameters())\n</code></pre>\n<p>However, if the commented line is uncommented, they fail. Such a change would obviously have a significant impact on the future of lots of PyTorch code, e.g., any code that <a href=\"https://discuss.pytorch.org/t/effect-of-calling-model-cuda-after-constructing-an-optimizer/15165\">moves a model after creating an optimizer</a>. In general, storing references to parameters is often more efficient than recursively traversing a module, which generates one such use case.</p>\n<pre><code class=\"lang-auto\">import copy\nimport torch.nn as nn\n\nconv = nn.Conv2d(8, 8, 3)\nseq = nn.Sequential(*[copy.deepcopy(conv) for _ in range(5)])\nfor _ in range(5):\n seq = nn.Sequential(*[copy.deepcopy(seq) for _ in range(5)])\n\nd = {k: v for k, v in seq.named_parameters()}\n\n%timeit list(d.items())\n%timeit list(seq.named_parameters())\n\n%timeit list(d.values())\n%timeit list(seq.parameters())\n</code></pre>\n<p>The internet doesn’t seem to offer much on the future of when this change will be rolled out despite the fact that it has been merged in master <a href=\"https://github.com/pytorch/pytorch/pull/21613\" rel=\"nofollow noopener\">for quite some time</a>. Can the PyTorch team provide any insights into the rollout plan?</p>",12 "post_number": 1,13 "post_type": 1,14 "posts_count": 3,15 "updated_at": "2020-02-21T04:52:45.539Z",16 "reply_count": 0,17 "reply_to_post_number": null,18 "quote_count": 0,19 "incoming_link_count": 83,20 "reads": 14,21 "readers_count": 13,22 "score": 417.8,23 "yours": false,24 "topic_id": 70564,25 "topic_slug": "future-of-references-to-module-parameters",26 "display_username": "",27 "primary_group_name": null,28 "flair_name": null,29 "flair_url": null,30 "flair_bg_color": null,31 "flair_color": null,32 "flair_group_id": null,33 "badges_granted": [],34 "version": 2,35 "can_edit": false,36 "can_delete": false,37 "can_recover": false,38 "can_see_hidden_post": false,39 "can_wiki": false,40 "link_counts": [41 {42 "url": "https://github.com/pytorch/pytorch/pull/21613",43 "internal": false,44 "reflection": false,45 "clicks": 146 },47 {48 "url": "https://discuss.pytorch.org/t/effect-of-calling-model-cuda-after-constructing-an-optimizer/15165",49 "internal": true,50 "reflection": false,51 "title": "Effect of calling model.cuda() after constructing an optimizer",52 "clicks": 153 }54 ],55 "read": true,56 "user_title": null,57 "bookmarked": false,58 "actions_summary": [],59 "moderator": false,60 "admin": false,61 "staff": false,62 "user_id": 27927,63 "hidden": false,64 "trust_level": 1,65 "deleted_at": null,66 "user_deleted": false,67 "edit_reason": null,68 "can_view_edit_history": true,69 "wiki": false,70 "post_url": "/t/future-of-references-to-module-parameters/70564/1",71 "can_accept_answer": false,72 "can_unaccept_answer": false,73 "accepted_answer": false,74 "topic_accepted_answer": null,75 "can_vote": false76 },77 {78 "id": 167994,79 "name": "Thomas V",80 "username": "tom",81 "avatar_template": "/user_avatar/discuss.pytorch.org/tom/{size}/3162_2.png",82 "created_at": "2020-02-21T21:31:05.479Z",83 "cooked": "<p>To the best of my knowledge, the advice has always been to move your model first as the fact that the parameters don’t change in this has been more an implementation detail than part of the design.</p>\n<p>It would seem that in most use cases, the traversing of the modules upon setup is rather small compared to running or training the models.</p>\n<p>Best regards</p>\n<p>Thomas</p>",84 "post_number": 2,85 "post_type": 1,86 "posts_count": 3,87 "updated_at": "2020-02-21T21:31:05.479Z",88 "reply_count": 1,89 "reply_to_post_number": null,90 "quote_count": 0,91 "incoming_link_count": 1,92 "reads": 12,93 "readers_count": 11,94 "score": 12.4,95 "yours": false,96 "topic_id": 70564,97 "topic_slug": "future-of-references-to-module-parameters",98 "display_username": "Thomas V",99 "primary_group_name": null,100 "flair_name": null,101 "flair_url": null,102 "flair_bg_color": null,103 "flair_color": null,104 "flair_group_id": null,105 "badges_granted": [],106 "version": 1,107 "can_edit": false,108 "can_delete": false,109 "can_recover": false,110 "can_see_hidden_post": false,111 "can_wiki": false,112 "read": true,113 "user_title": null,114 "bookmarked": false,115 "actions_summary": [],116 "moderator": false,117 "admin": false,118 "staff": false,119 "user_id": 616,120 "hidden": false,121 "trust_level": 2,122 "deleted_at": null,123 "user_deleted": false,124 "edit_reason": null,125 "can_view_edit_history": true,126 "wiki": false,127 "post_url": "/t/future-of-references-to-module-parameters/70564/2",128 "can_accept_answer": false,129 "can_unaccept_answer": false,130 "accepted_answer": false,131 "topic_accepted_answer": null132 },133 {134 "id": 168006,135 "name": "",136 "username": "ntenenz",137 "avatar_template": "/letter_avatar_proxy/v4/letter/n/258eb7/{size}.png",138 "created_at": "2020-02-21T22:08:24.065Z",139 "cooked": "<p>My apologies, I wasn’t entirely clear regarding my use case. I would like to be able to stably store references to parameters rather than recursively iterate over the entire set of parameters. This will be done frequently, as opposed to during the initialization of training.</p>",140 "post_number": 3,141 "post_type": 1,142 "posts_count": 3,143 "updated_at": "2020-02-21T22:08:24.065Z",144 "reply_count": 0,145 "reply_to_post_number": 2,146 "quote_count": 0,147 "incoming_link_count": 4,148 "reads": 11,149 "readers_count": 10,150 "score": 22.2,151 "yours": false,152 "topic_id": 70564,153 "topic_slug": "future-of-references-to-module-parameters",154 "display_username": "",155 "primary_group_name": null,156 "flair_name": null,157 "flair_url": null,158 "flair_bg_color": null,159 "flair_color": null,160 "flair_group_id": null,161 "badges_granted": [],162 "version": 1,163 "can_edit": false,164 "can_delete": false,165 "can_recover": false,166 "can_see_hidden_post": false,167 "can_wiki": false,168 "read": true,169 "user_title": null,170 "reply_to_user": {171 "id": 616,172 "username": "tom",173 "name": "Thomas V",174 "avatar_template": "/user_avatar/discuss.pytorch.org/tom/{size}/3162_2.png"175 },176 "bookmarked": false,177 "actions_summary": [],178 "moderator": false,179 "admin": false,180 "staff": false,181 "user_id": 27927,182 "hidden": false,183 "trust_level": 1,184 "deleted_at": null,185 "user_deleted": false,186 "edit_reason": null,187 "can_view_edit_history": true,188 "wiki": false,189 "post_url": "/t/future-of-references-to-module-parameters/70564/3",190 "can_accept_answer": false,191 "can_unaccept_answer": false,192 "accepted_answer": false,193 "topic_accepted_answer": null194 }195 ],196 "stream": [197 167839,198 167994,199 168006200 ]201 },202 "timeline_lookup": [203 [204 1,205 2074206 ],207 [208 2,209 2073210 ]211 ],212 "suggested_topics": [213 {214 "fancy_title": "Model.eval() causes all predictions to be identical",215 "id": 212664,216 "title": "Model.eval() causes all predictions to be identical",217 "slug": "model-eval-causes-all-predictions-to-be-identical",218 "posts_count": 1,219 "reply_count": 0,220 "highest_post_number": 1,221 "image_url": null,222 "created_at": "2024-11-07T16:01:06.996Z",223 "last_posted_at": "2024-11-07T16:01:07.056Z",224 "bumped": true,225 "bumped_at": "2024-11-07T16:01:07.056Z",226 "archetype": "regular",227 "unseen": false,228 "pinned": false,229 "unpinned": null,230 "visible": true,231 "closed": false,232 "archived": false,233 "bookmarked": null,234 "liked": null,235 "tags_descriptions": {},236 "like_count": 0,237 "views": 123,238 "category_id": 1,239 "featured_link": null,240 "has_accepted_answer": false,241 "posters": [242 {243 "extras": "latest single",244 "description": "Original Poster, Most Recent Poster",245 "user": {246 "id": 80751,247 "username": "TallMoose",248 "name": "",249 "avatar_template": "/user_avatar/discuss.pytorch.org/tallmoose/{size}/73834_2.png",250 "trust_level": 0251 }252 }253 ]254 },255 {256 "fancy_title": "Build torch from source for jetson",257 "id": 212183,258 "title": "Build torch from source for jetson",259 "slug": "build-torch-from-source-for-jetson",260 "posts_count": 2,261 "reply_count": 0,262 "highest_post_number": 2,263 "image_url": null,264 "created_at": "2024-10-28T02:26:59.172Z",265 "last_posted_at": "2024-10-28T13:29:44.901Z",266 "bumped": true,267 "bumped_at": "2024-10-28T13:29:44.901Z",268 "archetype": "regular",269 "unseen": false,270 "pinned": false,271 "unpinned": null,272 "visible": true,273 "closed": false,274 "archived": false,275 "bookmarked": null,276 "liked": null,277 "tags_descriptions": {},278 "like_count": 0,279 "views": 125,280 "category_id": 1,281 "featured_link": null,282 "has_accepted_answer": false,283 "posters": [284 {285 "extras": null,286 "description": "Original Poster",287 "user": {288 "id": 80526,289 "username": "wangyitao42",290 "name": "小明 王",291 "avatar_template": "/user_avatar/discuss.pytorch.org/wangyitao42/{size}/73607_2.png",292 "trust_level": 1293 }294 },295 {296 "extras": "latest",297 "description": "Most Recent Poster",298 "user": {299 "id": 3534,300 "username": "ptrblck",301 "name": "",302 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",303 "admin": true,304 "moderator": true,305 "trust_level": 2306 }307 }308 ]309 },310 {311 "fancy_title": "Questions Regarding the Direction of Half Padding for Odd-Sized Inputs in PyTorch Conv2D and ConvTransposed2D",312 "id": 214237,313 "title": "Questions Regarding the Direction of Half Padding for Odd-Sized Inputs in PyTorch Conv2D and ConvTransposed2D",314 "slug": "questions-regarding-the-direction-of-half-padding-for-odd-sized-inputs-in-pytorch-conv2d-and-convtransposed2d",315 "posts_count": 2,316 "reply_count": 0,317 "highest_post_number": 2,318 "image_url": "https://discuss.pytorch.org/uploads/default/optimized/3X/b/b/bbbaac4cedae04ffa60904b9c583ad66ddfa697c_2_1024x449.png",319 "created_at": "2024-12-15T07:28:33.352Z",320 "last_posted_at": "2024-12-17T13:51:47.078Z",321 "bumped": true,322 "bumped_at": "2024-12-17T13:51:47.078Z",323 "archetype": "regular",324 "unseen": false,325 "pinned": false,326 "unpinned": null,327 "visible": true,328 "closed": false,329 "archived": false,330 "bookmarked": null,331 "liked": null,332 "tags_descriptions": {},333 "like_count": 0,334 "views": 91,335 "category_id": 1,336 "featured_link": null,337 "has_accepted_answer": false,338 "posters": [339 {340 "extras": "latest single",341 "description": "Original Poster, Most Recent Poster",342 "user": {343 "id": 81511,344 "username": "mx34kryce",345 "name": "",346 "avatar_template": "/user_avatar/discuss.pytorch.org/mx34kryce/{size}/74521_2.png",347 "trust_level": 1348 }349 }350 ]351 },352 {353 "fancy_title": "Checking termination condition is exceptionally slow",354 "id": 214324,355 "title": "Checking termination condition is exceptionally slow",356 "slug": "checking-termination-condition-is-exceptionally-slow",357 "posts_count": 3,358 "reply_count": 1,359 "highest_post_number": 3,360 "image_url": null,361 "created_at": "2024-12-17T16:53:47.241Z",362 "last_posted_at": "2024-12-18T08:55:09.146Z",363 "bumped": true,364 "bumped_at": "2024-12-18T08:55:09.146Z",365 "archetype": "regular",366 "unseen": false,367 "pinned": false,368 "unpinned": null,369 "visible": true,370 "closed": false,371 "archived": false,372 "bookmarked": null,373 "liked": null,374 "tags_descriptions": {},375 "like_count": 1,376 "views": 218,377 "category_id": 1,378 "featured_link": null,379 "has_accepted_answer": false,380 "posters": [381 {382 "extras": "latest",383 "description": "Original Poster, Most Recent Poster",384 "user": {385 "id": 81163,386 "username": "ViktorAJStein",387 "name": "Viktor AJ Stein",388 "avatar_template": "/letter_avatar_proxy/v4/letter/v/47e85d/{size}.png",389 "trust_level": 0390 }391 },392 {393 "extras": null,394 "description": "Frequent Poster",395 "user": {396 "id": 3534,397 "username": "ptrblck",398 "name": "",399 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",400 "admin": true,401 "moderator": true,402 "trust_level": 2403 }404 }405 ]406 },407 {408 "fancy_title": "H100 vs A100 Memory Usage Difference",409 "id": 214869,410 "title": "H100 vs A100 Memory Usage Difference",411 "slug": "h100-vs-a100-memory-usage-difference",412 "posts_count": 8,413 "reply_count": 5,414 "highest_post_number": 8,415 "image_url": null,416 "created_at": "2025-01-02T05:02:00.260Z",417 "last_posted_at": "2025-02-05T21:15:17.407Z",418 "bumped": true,419 "bumped_at": "2025-02-05T21:15:17.407Z",420 "archetype": "regular",421 "unseen": false,422 "pinned": false,423 "unpinned": null,424 "visible": true,425 "closed": false,426 "archived": false,427 "bookmarked": null,428 "liked": null,429 "tags_descriptions": {},430 "like_count": 1,431 "views": 722,432 "category_id": 1,433 "featured_link": null,434 "has_accepted_answer": false,435 "posters": [436 {437 "extras": null,438 "description": "Original Poster",439 "user": {440 "id": 75268,441 "username": "Ty4Reading",442 "name": "Ty",443 "avatar_template": "/user_avatar/discuss.pytorch.org/ty4reading/{size}/69516_2.png",444 "trust_level": 1445 }446 },447 {448 "extras": null,449 "description": "Frequent Poster",450 "user": {451 "id": 3534,452 "username": "ptrblck",453 "name": "",454 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",455 "admin": true,456 "moderator": true,457 "trust_level": 2458 }459 },460 {461 "extras": null,462 "description": "Frequent Poster",463 "user": {464 "id": 41997,465 "username": "bdhirsh",466 "name": "Brian Hirsh",467 "avatar_template": "/user_avatar/discuss.pytorch.org/bdhirsh/{size}/74038_2.png",468 "trust_level": 2469 }470 },471 {472 "extras": "latest",473 "description": "Most Recent Poster",474 "user": {475 "id": 82492,476 "username": "ekomp",477 "name": "",478 "avatar_template": "/user_avatar/discuss.pytorch.org/ekomp/{size}/75513_2.png",479 "trust_level": 1480 }481 }482 ]483 }484 ],485 "tags_descriptions": {},486 "fancy_title": "Future of References to Module Parameters",487 "id": 70564,488 "title": "Future of References to Module Parameters",489 "posts_count": 3,490 "created_at": "2020-02-21T04:46:04.668Z",491 "views": 699,492 "reply_count": 1,493 "like_count": 0,494 "last_posted_at": "2020-02-21T22:08:24.065Z",495 "visible": true,496 "closed": false,497 "archived": false,498 "has_summary": false,499 "archetype": "regular",500 "slug": "future-of-references-to-module-parameters",501 "category_id": 1,502 "word_count": 365,503 "deleted_at": null,504 "user_id": 27927,505 "featured_link": null,506 "pinned_globally": false,507 "pinned_at": null,508 "pinned_until": null,509 "image_url": null,510 "slow_mode_seconds": 0,511 "draft": null,512 "draft_key": "topic_70564",513 "draft_sequence": null,514 "unpinned": null,515 "pinned": false,516 "current_post_number": 1,517 "highest_post_number": 3,518 "deleted_by": null,519 "actions_summary": [520 {521 "id": 4,522 "count": 0,523 "hidden": false,524 "can_act": false525 },526 {527 "id": 8,528 "count": 0,529 "hidden": false,530 "can_act": false531 },532 {533 "id": 10,534 "count": 0,535 "hidden": false,536 "can_act": false537 },538 {539 "id": 7,540 "count": 0,541 "hidden": false,542 "can_act": false543 }544 ],545 "chunk_size": 20,546 "bookmarked": false,547 "topic_timer": null,548 "message_bus_last_id": 0,549 "participant_count": 2,550 "show_read_indicator": false,551 "thumbnails": null,552 "slow_mode_enabled_until": null,553 "can_vote": false,554 "vote_count": 0,555 "user_voted": false,556 "discourse_zendesk_plugin_zendesk_id": null,557 "discourse_zendesk_plugin_zendesk_url": "https://your-url.zendesk.com/agent/tickets/",558 "details": {559 "can_edit": false,560 "notification_level": 1,561 "participants": [562 {563 "id": 27927,564 "username": "ntenenz",565 "name": "",566 "avatar_template": "/letter_avatar_proxy/v4/letter/n/258eb7/{size}.png",567 "post_count": 2,568 "primary_group_name": null,569 "flair_name": null,570 "flair_url": null,571 "flair_color": null,572 "flair_bg_color": null,573 "flair_group_id": null,574 "trust_level": 1575 },576 {577 "id": 616,578 "username": "tom",579 "name": "Thomas V",580 "avatar_template": "/user_avatar/discuss.pytorch.org/tom/{size}/3162_2.png",581 "post_count": 1,582 "primary_group_name": null,583 "flair_name": null,584 "flair_url": null,585 "flair_color": null,586 "flair_bg_color": null,587 "flair_group_id": null,588 "trust_level": 2589 }590 ],591 "created_by": {592 "id": 27927,593 "username": "ntenenz",594 "name": "",595 "avatar_template": "/letter_avatar_proxy/v4/letter/n/258eb7/{size}.png"596 },597 "last_poster": {598 "id": 27927,599 "username": "ntenenz",600 "name": "",601 "avatar_template": "/letter_avatar_proxy/v4/letter/n/258eb7/{size}.png"602 },603 "links": [604 {605 "url": "https://discuss.pytorch.org/t/effect-of-calling-model-cuda-after-constructing-an-optimizer/15165",606 "title": "Effect of calling model.cuda() after constructing an optimizer",607 "internal": true,608 "attachment": false,609 "reflection": false,610 "clicks": 1,611 "user_id": 27927,612 "domain": "discuss.pytorch.org",613 "root_domain": "pytorch.org"614 },615 {616 "url": "https://github.com/pytorch/pytorch/pull/21613",617 "title": null,618 "internal": false,619 "attachment": false,620 "reflection": false,621 "clicks": 1,622 "user_id": 27927,623 "domain": "github.com",624 "root_domain": "github.com"625 }626 ]627 },628 "bookmarks": []629 },630 {631 "post_stream": {632 "posts": [633 {634 "id": 167114,635 "name": "Maral",636 "username": "maralm",637 "avatar_template": "/letter_avatar_proxy/v4/letter/m/e36b37/{size}.png",638 "created_at": "2020-02-19T00:34:33.073Z",639 "cooked": "<p>Hi,</p>\n<p>I wrote a custom backward function for my model. I want to use the DataParallel package. However I have a problem. If I use</p>\n<pre><code class=\"lang-auto\">model = torch.nn.DataParallel(model, device_ids=[0,1])\n</code></pre>\n<p>I get the following error:</p>\n<p>“‘DataParallel’ object has no attribute ‘backward’”</p>\n<p>I know this can be solved by using model.module.backward, but then it will only use one gpu. Is there a way to use the torch.nn.DataParallel with custom backward and attributes?</p>",640 "post_number": 1,641 "post_type": 1,642 "posts_count": 6,643 "updated_at": "2020-02-19T00:35:24.110Z",644 "reply_count": 0,645 "reply_to_post_number": null,646 "quote_count": 0,647 "incoming_link_count": 134,648 "reads": 13,649 "readers_count": 12,650 "score": 672.6,651 "yours": false,652 "topic_id": 70250,653 "topic_slug": "multi-gpu-with-custom-backward-and-attributes",654 "display_username": "Maral",655 "primary_group_name": null,656 "flair_name": null,657 "flair_url": null,658 "flair_bg_color": null,659 "flair_color": null,660 "flair_group_id": null,661 "badges_granted": [],662 "version": 1,663 "can_edit": false,664 "can_delete": false,665 "can_recover": false,666 "can_see_hidden_post": false,667 "can_wiki": false,668 "read": true,669 "user_title": null,670 "bookmarked": false,671 "actions_summary": [],672 "moderator": false,673 "admin": false,674 "staff": false,675 "user_id": 19822,676 "hidden": false,677 "trust_level": 1,678 "deleted_at": null,679 "user_deleted": false,680 "edit_reason": null,681 "can_view_edit_history": true,682 "wiki": false,683 "post_url": "/t/multi-gpu-with-custom-backward-and-attributes/70250/1",684 "can_accept_answer": false,685 "can_unaccept_answer": false,686 "accepted_answer": false,687 "topic_accepted_answer": null,688 "can_vote": false689 },690 {691 "id": 167204,692 "name": "",693 "username": "ptrblck",694 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",695 "created_at": "2020-02-19T07:27:19.086Z",696 "cooked": "<p>Would it be possible to return the outputs in your <code>forward</code> method and calculate the loss on the default device?<br>\nThis would be the vanilla use case, while it seems you’ve implemented <code>backward</code> as a class function?</p>",697 "post_number": 2,698 "post_type": 1,699 "posts_count": 6,700 "updated_at": "2020-02-19T07:27:19.086Z",701 "reply_count": 1,702 "reply_to_post_number": null,703 "quote_count": 0,704 "incoming_link_count": 1,705 "reads": 12,706 "readers_count": 11,707 "score": 12.4,708 "yours": false,709 "topic_id": 70250,710 "topic_slug": "multi-gpu-with-custom-backward-and-attributes",711 "display_username": "",712 "primary_group_name": null,713 "flair_name": null,714 "flair_url": null,715 "flair_bg_color": null,716 "flair_color": null,717 "flair_group_id": null,718 "badges_granted": [],719 "version": 1,720 "can_edit": false,721 "can_delete": false,722 "can_recover": false,723 "can_see_hidden_post": false,724 "can_wiki": false,725 "read": true,726 "user_title": "",727 "bookmarked": false,728 "actions_summary": [],729 "moderator": true,730 "admin": true,731 "staff": true,732 "user_id": 3534,733 "hidden": false,734 "trust_level": 2,735 "deleted_at": null,736 "user_deleted": false,737 "edit_reason": null,738 "can_view_edit_history": true,739 "wiki": false,740 "post_url": "/t/multi-gpu-with-custom-backward-and-attributes/70250/2",741 "can_accept_answer": false,742 "can_unaccept_answer": false,743 "accepted_answer": false,744 "topic_accepted_answer": null745 },746 {747 "id": 167375,748 "name": "Maral",749 "username": "maralm",750 "avatar_template": "/letter_avatar_proxy/v4/letter/m/e36b37/{size}.png",751 "created_at": "2020-02-19T17:19:17.786Z",752 "cooked": "<p>Thanks for the reply. No the backward is not a separate class. It is a function inside the model class. Here is how I define it:</p>\n<pre><code class=\"lang-auto\">Class myModel():\n def __init__(self, config):\n ....\n def forward(...):\n ....\n def backward(...):\n ....\n</code></pre>\n<p>And I call it this way:</p>\n<pre><code class=\"lang-auto\">outputs = model(....)\nloss = outputs[0] \nif args.n_gpu > 1:\n loss = loss.mean()\nmodel = model.backward(...)\n</code></pre>\n<p>but nn.DataParallel is not recognizing the backward and some other attributes without using module.</p>",753 "post_number": 3,754 "post_type": 1,755 "posts_count": 6,756 "updated_at": "2020-02-19T17:19:17.786Z",757 "reply_count": 1,758 "reply_to_post_number": 2,759 "quote_count": 0,760 "incoming_link_count": 2,761 "reads": 11,762 "readers_count": 10,763 "score": 17.2,764 "yours": false,765 "topic_id": 70250,766 "topic_slug": "multi-gpu-with-custom-backward-and-attributes",767 "display_username": "Maral",768 "primary_group_name": null,769 "flair_name": null,770 "flair_url": null,771 "flair_bg_color": null,772 "flair_color": null,773 "flair_group_id": null,774 "badges_granted": [],775 "version": 1,776 "can_edit": false,777 "can_delete": false,778 "can_recover": false,779 "can_see_hidden_post": false,780 "can_wiki": false,781 "read": true,782 "user_title": null,783 "reply_to_user": {784 "id": 3534,785 "username": "ptrblck",786 "name": "",787 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png"788 },789 "bookmarked": false,790 "actions_summary": [],791 "moderator": false,792 "admin": false,793 "staff": false,794 "user_id": 19822,795 "hidden": false,796 "trust_level": 1,797 "deleted_at": null,798 "user_deleted": false,799 "edit_reason": null,800 "can_view_edit_history": true,801 "wiki": false,802 "post_url": "/t/multi-gpu-with-custom-backward-and-attributes/70250/3",803 "can_accept_answer": false,804 "can_unaccept_answer": false,805 "accepted_answer": false,806 "topic_accepted_answer": null807 },808 {809 "id": 167454,810 "name": "",811 "username": "ptrblck",812 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",813 "created_at": "2020-02-20T00:43:00.591Z",814 "cooked": "<p>Thanks for the information.</p>\n<p>What’s the design decision to put the <code>backward</code> call inside your model?<br>\nAre you using some internal parameters?<br>\nIf so, how are these parameters updated/used inside the model?</p>",815 "post_number": 4,816 "post_type": 1,817 "posts_count": 6,818 "updated_at": "2020-02-20T00:43:00.591Z",819 "reply_count": 1,820 "reply_to_post_number": 3,821 "quote_count": 0,822 "incoming_link_count": 3,823 "reads": 7,824 "readers_count": 6,825 "score": 21.4,826 "yours": false,827 "topic_id": 70250,828 "topic_slug": "multi-gpu-with-custom-backward-and-attributes",829 "display_username": "",830 "primary_group_name": null,831 "flair_name": null,832 "flair_url": null,833 "flair_bg_color": null,834 "flair_color": null,835 "flair_group_id": null,836 "badges_granted": [],837 "version": 1,838 "can_edit": false,839 "can_delete": false,840 "can_recover": false,841 "can_see_hidden_post": false,842 "can_wiki": false,843 "read": true,844 "user_title": "",845 "reply_to_user": {846 "id": 19822,847 "username": "maralm",848 "name": "Maral",849 "avatar_template": "/letter_avatar_proxy/v4/letter/m/e36b37/{size}.png"850 },851 "bookmarked": false,852 "actions_summary": [],853 "moderator": true,854 "admin": true,855 "staff": true,856 "user_id": 3534,857 "hidden": false,858 "trust_level": 2,859 "deleted_at": null,860 "user_deleted": false,861 "edit_reason": null,862 "can_view_edit_history": true,863 "wiki": false,864 "post_url": "/t/multi-gpu-with-custom-backward-and-attributes/70250/4",865 "can_accept_answer": false,866 "can_unaccept_answer": false,867 "accepted_answer": false,868 "topic_accepted_answer": null869 },870 {871 "id": 167786,872 "name": "Maral",873 "username": "maralm",874 "avatar_template": "/letter_avatar_proxy/v4/letter/m/e36b37/{size}.png",875 "created_at": "2020-02-20T21:22:14.119Z",876 "cooked": "<p>I needed to access the activations and activation gradients in backward. I collect activations in forward pass and access to them in backward. I use autograd backward function to calculate each layer’s backward and make the changes that I want in the process.</p>",877 "post_number": 5,878 "post_type": 1,879 "posts_count": 6,880 "updated_at": "2020-02-20T21:22:14.119Z",881 "reply_count": 1,882 "reply_to_post_number": 4,883 "quote_count": 0,884 "incoming_link_count": 1,885 "reads": 7,886 "readers_count": 6,887 "score": 11.4,888 "yours": false,889 "topic_id": 70250,890 "topic_slug": "multi-gpu-with-custom-backward-and-attributes",891 "display_username": "Maral",892 "primary_group_name": null,893 "flair_name": null,894 "flair_url": null,895 "flair_bg_color": null,896 "flair_color": null,897 "flair_group_id": null,898 "badges_granted": [],899 "version": 1,900 "can_edit": false,901 "can_delete": false,902 "can_recover": false,903 "can_see_hidden_post": false,904 "can_wiki": false,905 "read": true,906 "user_title": null,907 "reply_to_user": {908 "id": 3534,909 "username": "ptrblck",910 "name": "",911 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png"912 },913 "bookmarked": false,914 "actions_summary": [],915 "moderator": false,916 "admin": false,917 "staff": false,918 "user_id": 19822,919 "hidden": false,920 "trust_level": 1,921 "deleted_at": null,922 "user_deleted": false,923 "edit_reason": null,924 "can_view_edit_history": true,925 "wiki": false,926 "post_url": "/t/multi-gpu-with-custom-backward-and-attributes/70250/5",927 "can_accept_answer": false,928 "can_unaccept_answer": false,929 "accepted_answer": false,930 "topic_accepted_answer": null931 },932 {933 "id": 168004,934 "name": "Maral",935 "username": "maralm",936 "avatar_template": "/letter_avatar_proxy/v4/letter/m/e36b37/{size}.png",937 "created_at": "2020-02-21T22:02:51.032Z",938 "cooked": "<p>I tried the distributed data parallel instead of data parallel and it is working.</p>",939 "post_number": 6,940 "post_type": 1,941 "posts_count": 6,942 "updated_at": "2020-02-21T22:02:51.032Z",943 "reply_count": 0,944 "reply_to_post_number": 5,945 "quote_count": 0,946 "incoming_link_count": 3,947 "reads": 5,948 "readers_count": 4,949 "score": 16.0,950 "yours": false,951 "topic_id": 70250,952 "topic_slug": "multi-gpu-with-custom-backward-and-attributes",953 "display_username": "Maral",954 "primary_group_name": null,955 "flair_name": null,956 "flair_url": null,957 "flair_bg_color": null,958 "flair_color": null,959 "flair_group_id": null,960 "badges_granted": [],961 "version": 1,962 "can_edit": false,963 "can_delete": false,964 "can_recover": false,965 "can_see_hidden_post": false,966 "can_wiki": false,967 "read": true,968 "user_title": null,969 "reply_to_user": {970 "id": 19822,971 "username": "maralm",972 "name": "Maral",973 "avatar_template": "/letter_avatar_proxy/v4/letter/m/e36b37/{size}.png"974 },975 "bookmarked": false,976 "actions_summary": [],977 "moderator": false,978 "admin": false,979 "staff": false,980 "user_id": 19822,981 "hidden": false,982 "trust_level": 1,983 "deleted_at": null,984 "user_deleted": false,985 "edit_reason": null,986 "can_view_edit_history": true,987 "wiki": false,988 "post_url": "/t/multi-gpu-with-custom-backward-and-attributes/70250/6",989 "can_accept_answer": false,990 "can_unaccept_answer": false,991 "accepted_answer": false,992 "topic_accepted_answer": null993 }994 ],995 "stream": [996 167114,997 167204,998 167375,999 167454,1000 167786,1001 1680041002 ]1003 },1004 "timeline_lookup": [1005 [1006 1,1007 20761008 ],1009 [1010 3,1011 20751012 ],1013 [1014 5,1015 20741016 ],1017 [1018 6,1019 20731020 ]1021 ],1022 "suggested_topics": [1023 {1024 "fancy_title": "NVLS support in pytorch",1025 "id": 217616,1026 "title": "NVLS support in pytorch",1027 "slug": "nvls-support-in-pytorch",1028 "posts_count": 3,1029 "reply_count": 0,1030 "highest_post_number": 3,1031 "image_url": null,1032 "created_at": "2025-03-09T07:44:49.033Z",1033 "last_posted_at": "2025-03-11T18:12:08.034Z",1034 "bumped": true,1035 "bumped_at": "2025-03-11T18:12:08.034Z",1036 "archetype": "regular",1037 "unseen": false,1038 "pinned": false,1039 "unpinned": null,1040 "visible": true,1041 "closed": false,1042 "archived": false,1043 "bookmarked": null,1044 "liked": null,1045 "tags_descriptions": {},1046 "like_count": 1,1047 "views": 247,1048 "category_id": 12,1049 "featured_link": null,1050 "has_accepted_answer": false,1051 "posters": [1052 {1053 "extras": null,1054 "description": "Original Poster",1055 "user": {1056 "id": 83151,1057 "username": "rajagond",1058 "name": "",1059 "avatar_template": "/user_avatar/discuss.pytorch.org/rajagond/{size}/76053_2.png",1060 "trust_level": 11061 }1062 },1063 {1064 "extras": null,1065 "description": "Frequent Poster",1066 "user": {1067 "id": 81105,1068 "username": "HyperHyper",1069 "name": "HyperHyper",1070 "avatar_template": "/user_avatar/discuss.pytorch.org/hyperhyper/{size}/74168_2.png",1071 "trust_level": 11072 }1073 },1074 {1075 "extras": "latest",1076 "description": "Most Recent Poster",1077 "user": {1078 "id": 3534,1079 "username": "ptrblck",1080 "name": "",1081 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",1082 "admin": true,1083 "moderator": true,1084 "trust_level": 21085 }1086 }1087 ]1088 },1089 {1090 "fancy_title": "PyTorch using both GPUs even when after setting explictly",1091 "id": 218422,1092 "title": "PyTorch using both GPUs even when after setting explictly",1093 "slug": "pytorch-using-both-gpus-even-when-after-setting-explictly",1094 "posts_count": 3,1095 "reply_count": 1,1096 "highest_post_number": 3,1097 "image_url": null,1098 "created_at": "2025-03-31T00:11:55.521Z",1099 "last_posted_at": "2025-04-01T22:02:28.424Z",1100 "bumped": true,1101 "bumped_at": "2025-04-01T22:02:28.424Z",1102 "archetype": "regular",1103 "unseen": false,1104 "pinned": false,1105 "unpinned": null,1106 "visible": true,1107 "closed": false,1108 "archived": false,1109 "bookmarked": null,1110 "liked": null,1111 "tags_descriptions": {},1112 "like_count": 0,1113 "views": 68,1114 "category_id": 12,1115 "featured_link": null,1116 "has_accepted_answer": false,1117 "posters": [1118 {1119 "extras": "latest",1120 "description": "Original Poster, Most Recent Poster",1121 "user": {1122 "id": 83395,1123 "username": "Malitha96",1124 "name": "",1125 "avatar_template": "/letter_avatar_proxy/v4/letter/m/eb9ed0/{size}.png",1126 "trust_level": 01127 }1128 },1129 {1130 "extras": null,1131 "description": "Frequent Poster",1132 "user": {1133 "id": 39542,1134 "username": "H-Huang",1135 "name": "Howard Huang",1136 "avatar_template": "/user_avatar/discuss.pytorch.org/h-huang/{size}/35598_2.png",1137 "trust_level": 21138 }1139 }1140 ]1141 },1142 {1143 "fancy_title": "How to solve the issue with getting free ports in Pytorch DDP?",1144 "id": 213921,1145 "title": "How to solve the issue with getting free ports in Pytorch DDP?",1146 "slug": "how-to-solve-the-issue-with-getting-free-ports-in-pytorch-ddp",1147 "posts_count": 1,1148 "reply_count": 0,1149 "highest_post_number": 1,1150 "image_url": null,1151 "created_at": "2024-12-06T19:39:09.953Z",1152 "last_posted_at": "2024-12-06T19:39:09.998Z",1153 "bumped": true,1154 "bumped_at": "2024-12-06T19:39:09.998Z",1155 "archetype": "regular",1156 "unseen": false,1157 "pinned": false,1158 "unpinned": null,1159 "visible": true,1160 "closed": false,1161 "archived": false,1162 "bookmarked": null,1163 "liked": null,1164 "tags_descriptions": {},1165 "like_count": 0,1166 "views": 100,1167 "category_id": 12,1168 "featured_link": null,1169 "has_accepted_answer": false,1170 "posters": [1171 {1172 "extras": "latest single",1173 "description": "Original Poster, Most Recent Poster",1174 "user": {1175 "id": 81360,1176 "username": "Shataneek",1177 "name": "",1178 "avatar_template": "/letter_avatar_proxy/v4/letter/s/57b2e6/{size}.png",1179 "trust_level": 01180 }1181 }1182 ]1183 },1184 {1185 "fancy_title": "How to create a DistributedSampler based on my own Sampler",1186 "id": 215244,1187 "title": "How to create a DistributedSampler based on my own Sampler",1188 "slug": "how-to-create-a-distributedsampler-based-on-my-own-sampler",1189 "posts_count": 3,1190 "reply_count": 1,1191 "highest_post_number": 4,1192 "image_url": null,1193 "created_at": "2025-01-11T01:52:41.394Z",1194 "last_posted_at": "2025-01-11T07:09:01.838Z",1195 "bumped": true,1196 "bumped_at": "2025-01-11T07:09:01.838Z",1197 "archetype": "regular",1198 "unseen": false,1199 "pinned": false,1200 "unpinned": null,