Anurag1734/cuda-error-resolution-analysis
07
1[2 {3 "post_stream": {4 "posts": [5 {6 "id": 153044,7 "name": "Silviu",8 "username": "smu226",9 "avatar_template": "/user_avatar/discuss.pytorch.org/smu226/{size}/9063_2.png",10 "created_at": "2019-12-16T06:53:59.131Z",11 "cooked": "<p>Hello! I am having some issues on using batch norm. I am in the beginning of building my NN so for now I am using 100 samples for training and I want to overfit to it, just to make sure that the network can learn. The input and output are bout 1500 each. Here is my network:</p>\n<pre><code class=\"lang-auto\">class GW_NN(nn.Module):\n def __init__(self):\n super().__init__()\n self.linear1 = nn.Linear(n_inp, 2000)\n self.linear2 = nn.Linear(2000, 2000)\n self.linear3 = nn.Linear(2000, 2000)\n self.linear4 = nn.Linear(2000, 2000)\n self.linear5 = nn.Linear(2000,n_out)\n self.bn = nn.BatchNorm1d(2000)\n \n def forward(self, x):\n x = F.softplus(self.bn(self.linear1(x)))\n x = F.softplus(self.bn(self.linear2(x)))\n x = F.softplus(self.bn(self.linear3(x)))\n x = F.softplus(self.bn(self.linear4(x)))\n x = self.linear5(x) \n return x\n\nmodel_gw = GW_NN().cuda()\n\nlrs = 1e-2\noptimizer_gw = optim.Adam(model_gw.parameters(), lr = lrs)\n\nfor epoch in range(10001):\n model_gw.train()\n for i, dtt in enumerate(my_dataloader):\n optimizer_gw.zero_grad()\n\n inp = dtt[0].float().cuda()\n output = dtt[1].float().cuda()\n\n loss = F.mse_loss(model_gw(inp),output)\n\n loss.backward()\n optimizer_gw.step()\n\n if epoch%100==0:\n print(loss.data.cpu().numpy())\n</code></pre>\n<p>The loss goes down okish:</p>\n<pre><code class=\"lang-auto\">29418.57\n20.279129\n11.549426\n8.563468\n8.235117\n8.161551\n9.561671\n7.5749683\n7.60303\n7.609553\n7.265949\n7.824227\n10.810941\n7.803124\n7.6215243\n7.977992\n7.9355087\n7.574047\n7.326716\n</code></pre>\n<p>But when I want to try the trained NN (on the same data used for training) it fails:</p>\n<pre><code class=\"lang-auto\">idx = 10\ny_real = output_data_phi[idx].data.cpu().numpy()\nmodel_gw.eval()\ny_pred = model_gw(input_data)[idx].data.cpu().numpy()\n\nprint(((y_pred-y_real)**2).mean())\n</code></pre>\n<p>I am getting <code>169114.33</code>. I assume that the problem is with using batch norm in eval mode, but I am not sure what to do. Can someone help me? Thank you!</p>",12 "post_number": 1,13 "post_type": 1,14 "posts_count": 5,15 "updated_at": "2019-12-16T06:53:59.131Z",16 "reply_count": 0,17 "reply_to_post_number": null,18 "quote_count": 0,19 "incoming_link_count": 20,20 "reads": 7,21 "readers_count": 6,22 "score": 101.4,23 "yours": false,24 "topic_id": 64161,25 "topic_slug": "issues-with-batch-norm",26 "display_username": "Silviu",27 "primary_group_name": null,28 "flair_name": null,29 "flair_url": null,30 "flair_bg_color": null,31 "flair_color": null,32 "flair_group_id": null,33 "badges_granted": [],34 "version": 1,35 "can_edit": false,36 "can_delete": false,37 "can_recover": false,38 "can_see_hidden_post": false,39 "can_wiki": false,40 "read": true,41 "user_title": null,42 "bookmarked": false,43 "actions_summary": [],44 "moderator": false,45 "admin": false,46 "staff": false,47 "user_id": 15365,48 "hidden": false,49 "trust_level": 1,50 "deleted_at": null,51 "user_deleted": false,52 "edit_reason": null,53 "can_view_edit_history": true,54 "wiki": false,55 "post_url": "/t/issues-with-batch-norm/64161/1",56 "can_accept_answer": false,57 "can_unaccept_answer": false,58 "accepted_answer": false,59 "topic_accepted_answer": null,60 "can_vote": false61 },62 {63 "id": 153048,64 "name": "",65 "username": "ptrblck",66 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",67 "created_at": "2019-12-16T07:07:22.030Z",68 "cooked": "<p>You are currently reusing the batchnorm layer in each step, which might throw the running stats as well as the trainable parameters off.<br>\nCould you try to use a separate layer and run the code again?</p>",69 "post_number": 2,70 "post_type": 1,71 "posts_count": 5,72 "updated_at": "2019-12-16T07:07:22.030Z",73 "reply_count": 1,74 "reply_to_post_number": null,75 "quote_count": 0,76 "incoming_link_count": 1,77 "reads": 7,78 "readers_count": 6,79 "score": 11.4,80 "yours": false,81 "topic_id": 64161,82 "topic_slug": "issues-with-batch-norm",83 "display_username": "",84 "primary_group_name": null,85 "flair_name": null,86 "flair_url": null,87 "flair_bg_color": null,88 "flair_color": null,89 "flair_group_id": null,90 "badges_granted": [],91 "version": 1,92 "can_edit": false,93 "can_delete": false,94 "can_recover": false,95 "can_see_hidden_post": false,96 "can_wiki": false,97 "read": true,98 "user_title": "",99 "bookmarked": false,100 "actions_summary": [],101 "moderator": true,102 "admin": true,103 "staff": true,104 "user_id": 3534,105 "hidden": false,106 "trust_level": 2,107 "deleted_at": null,108 "user_deleted": false,109 "edit_reason": null,110 "can_view_edit_history": true,111 "wiki": false,112 "post_url": "/t/issues-with-batch-norm/64161/2",113 "can_accept_answer": false,114 "can_unaccept_answer": false,115 "accepted_answer": false,116 "topic_accepted_answer": null117 },118 {119 "id": 153051,120 "name": "Silviu",121 "username": "smu226",122 "avatar_template": "/user_avatar/discuss.pytorch.org/smu226/{size}/9063_2.png",123 "created_at": "2019-12-16T07:20:46.733Z",124 "cooked": "<p>Thank you for your reply. Do you mean instead of repeating <code>self.bn</code> 4 times to use <code>self.bn1, self.bn2, self.bn3 self.bn4,</code> or something like this (different for each layer)?</p>",125 "post_number": 3,126 "post_type": 1,127 "posts_count": 5,128 "updated_at": "2019-12-16T07:20:46.733Z",129 "reply_count": 1,130 "reply_to_post_number": 2,131 "quote_count": 0,132 "incoming_link_count": 0,133 "reads": 6,134 "readers_count": 5,135 "score": 6.2,136 "yours": false,137 "topic_id": 64161,138 "topic_slug": "issues-with-batch-norm",139 "display_username": "Silviu",140 "primary_group_name": null,141 "flair_name": null,142 "flair_url": null,143 "flair_bg_color": null,144 "flair_color": null,145 "flair_group_id": null,146 "badges_granted": [],147 "version": 1,148 "can_edit": false,149 "can_delete": false,150 "can_recover": false,151 "can_see_hidden_post": false,152 "can_wiki": false,153 "read": true,154 "user_title": null,155 "reply_to_user": {156 "id": 3534,157 "username": "ptrblck",158 "name": "",159 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png"160 },161 "bookmarked": false,162 "actions_summary": [],163 "moderator": false,164 "admin": false,165 "staff": false,166 "user_id": 15365,167 "hidden": false,168 "trust_level": 1,169 "deleted_at": null,170 "user_deleted": false,171 "edit_reason": null,172 "can_view_edit_history": true,173 "wiki": false,174 "post_url": "/t/issues-with-batch-norm/64161/3",175 "can_accept_answer": false,176 "can_unaccept_answer": false,177 "accepted_answer": false,178 "topic_accepted_answer": null179 },180 {181 "id": 153052,182 "name": "",183 "username": "ptrblck",184 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",185 "created_at": "2019-12-16T07:22:58.121Z",186 "cooked": "<p>Yes, I meant the creation of 4 different layers as <code>self.bn1</code>, <code>self.bn2</code> etc. <img src=\"https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9\" title=\":wink:\" class=\"emoji\" alt=\":wink:\"></p>",187 "post_number": 4,188 "post_type": 1,189 "posts_count": 5,190 "updated_at": "2019-12-16T07:22:58.121Z",191 "reply_count": 1,192 "reply_to_post_number": 3,193 "quote_count": 0,194 "incoming_link_count": 0,195 "reads": 6,196 "readers_count": 5,197 "score": 6.2,198 "yours": false,199 "topic_id": 64161,200 "topic_slug": "issues-with-batch-norm",201 "display_username": "",202 "primary_group_name": null,203 "flair_name": null,204 "flair_url": null,205 "flair_bg_color": null,206 "flair_color": null,207 "flair_group_id": null,208 "badges_granted": [],209 "version": 1,210 "can_edit": false,211 "can_delete": false,212 "can_recover": false,213 "can_see_hidden_post": false,214 "can_wiki": false,215 "read": true,216 "user_title": "",217 "reply_to_user": {218 "id": 15365,219 "username": "smu226",220 "name": "Silviu",221 "avatar_template": "/user_avatar/discuss.pytorch.org/smu226/{size}/9063_2.png"222 },223 "bookmarked": false,224 "actions_summary": [],225 "moderator": true,226 "admin": true,227 "staff": true,228 "user_id": 3534,229 "hidden": false,230 "trust_level": 2,231 "deleted_at": null,232 "user_deleted": false,233 "edit_reason": null,234 "can_view_edit_history": true,235 "wiki": false,236 "post_url": "/t/issues-with-batch-norm/64161/4",237 "can_accept_answer": false,238 "can_unaccept_answer": false,239 "accepted_answer": false,240 "topic_accepted_answer": null241 },242 {243 "id": 153066,244 "name": "Silviu",245 "username": "smu226",246 "avatar_template": "/user_avatar/discuss.pytorch.org/smu226/{size}/9063_2.png",247 "created_at": "2019-12-16T07:50:29.513Z",248 "cooked": "<p>Awesome, it’s working! Thank you!</p>",249 "post_number": 5,250 "post_type": 1,251 "posts_count": 5,252 "updated_at": "2019-12-16T07:50:29.513Z",253 "reply_count": 0,254 "reply_to_post_number": 4,255 "quote_count": 0,256 "incoming_link_count": 0,257 "reads": 5,258 "readers_count": 4,259 "score": 1.0,260 "yours": false,261 "topic_id": 64161,262 "topic_slug": "issues-with-batch-norm",263 "display_username": "Silviu",264 "primary_group_name": null,265 "flair_name": null,266 "flair_url": null,267 "flair_bg_color": null,268 "flair_color": null,269 "flair_group_id": null,270 "badges_granted": [],271 "version": 1,272 "can_edit": false,273 "can_delete": false,274 "can_recover": false,275 "can_see_hidden_post": false,276 "can_wiki": false,277 "read": true,278 "user_title": null,279 "reply_to_user": {280 "id": 3534,281 "username": "ptrblck",282 "name": "",283 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png"284 },285 "bookmarked": false,286 "actions_summary": [],287 "moderator": false,288 "admin": false,289 "staff": false,290 "user_id": 15365,291 "hidden": false,292 "trust_level": 1,293 "deleted_at": null,294 "user_deleted": false,295 "edit_reason": null,296 "can_view_edit_history": true,297 "wiki": false,298 "post_url": "/t/issues-with-batch-norm/64161/5",299 "can_accept_answer": false,300 "can_unaccept_answer": false,301 "accepted_answer": false,302 "topic_accepted_answer": null303 }304 ],305 "stream": [306 153044,307 153048,308 153051,309 153052,310 153066311 ]312 },313 "timeline_lookup": [314 [315 1,316 2141317 ]318 ],319 "suggested_topics": [320 {321 "fancy_title": "I met strange error when train rwkv",322 "id": 214360,323 "title": "I met strange error when train rwkv",324 "slug": "i-met-strange-error-when-train-rwkv",325 "posts_count": 9,326 "reply_count": 7,327 "highest_post_number": 9,328 "image_url": null,329 "created_at": "2024-12-18T13:10:15.024Z",330 "last_posted_at": "2024-12-19T03:10:04.653Z",331 "bumped": true,332 "bumped_at": "2024-12-19T03:10:04.653Z",333 "archetype": "regular",334 "unseen": false,335 "pinned": false,336 "unpinned": null,337 "visible": true,338 "closed": false,339 "archived": false,340 "bookmarked": null,341 "liked": null,342 "tags_descriptions": {},343 "like_count": 0,344 "views": 165,345 "category_id": 1,346 "featured_link": null,347 "has_accepted_answer": false,348 "posters": [349 {350 "extras": null,351 "description": "Original Poster",352 "user": {353 "id": 81570,354 "username": "ddddewang0425",355 "name": "ddddewang0425",356 "avatar_template": "/user_avatar/discuss.pytorch.org/ddddewang0425/{size}/74590_2.png",357 "trust_level": 0358 }359 },360 {361 "extras": "latest",362 "description": "Most Recent Poster",363 "user": {364 "id": 3534,365 "username": "ptrblck",366 "name": "",367 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",368 "admin": true,369 "moderator": true,370 "trust_level": 2371 }372 }373 ]374 },375 {376 "fancy_title": "Educational Framework Inspired by PyTorch",377 "id": 215288,378 "title": "Educational Framework Inspired by PyTorch",379 "slug": "educational-framework-inspired-by-pytorch",380 "posts_count": 1,381 "reply_count": 0,382 "highest_post_number": 1,383 "image_url": null,384 "created_at": "2025-01-12T00:56:41.683Z",385 "last_posted_at": "2025-01-12T00:56:41.721Z",386 "bumped": true,387 "bumped_at": "2025-01-12T00:56:41.721Z",388 "archetype": "regular",389 "unseen": false,390 "pinned": false,391 "unpinned": null,392 "visible": true,393 "closed": false,394 "archived": false,395 "bookmarked": null,396 "liked": null,397 "tags_descriptions": {},398 "like_count": 1,399 "views": 87,400 "category_id": 1,401 "featured_link": null,402 "has_accepted_answer": false,403 "posters": [404 {405 "extras": "latest single",406 "description": "Original Poster, Most Recent Poster",407 "user": {408 "id": 82031,409 "username": "joseortizm",410 "name": "",411 "avatar_template": "/user_avatar/discuss.pytorch.org/joseortizm/{size}/75060_2.png",412 "trust_level": 0413 }414 }415 ]416 },417 {418 "fancy_title": "SGD with momentum pseudocode error?",419 "id": 218178,420 "title": "SGD with momentum pseudocode error?",421 "slug": "sgd-with-momentum-pseudocode-error",422 "posts_count": 2,423 "reply_count": 0,424 "highest_post_number": 2,425 "image_url": "https://discuss.pytorch.org/uploads/default/optimized/3X/6/2/629513af133b3b796dd1e2ea0beb07388c42d8ef_2_1024x941.png",426 "created_at": "2025-03-23T18:03:26.991Z",427 "last_posted_at": "2025-03-24T21:13:14.876Z",428 "bumped": true,429 "bumped_at": "2025-03-24T21:13:14.876Z",430 "archetype": "regular",431 "unseen": false,432 "pinned": false,433 "unpinned": null,434 "visible": true,435 "closed": false,436 "archived": false,437 "bookmarked": null,438 "liked": null,439 "tags_descriptions": {},440 "like_count": 0,441 "views": 92,442 "category_id": 1,443 "featured_link": null,444 "has_accepted_answer": false,445 "posters": [446 {447 "extras": null,448 "description": "Original Poster",449 "user": {450 "id": 83429,451 "username": "belsten",452 "name": "afb",453 "avatar_template": "/user_avatar/discuss.pytorch.org/belsten/{size}/76304_2.png",454 "trust_level": 1455 }456 },457 {458 "extras": "latest",459 "description": "Most Recent Poster",460 "user": {461 "id": 18088,462 "username": "KFrank",463 "name": "K. Frank",464 "avatar_template": "/letter_avatar_proxy/v4/letter/k/ecb155/{size}.png",465 "trust_level": 2466 }467 }468 ]469 },470 {471 "fancy_title": "[Solved] Maskrcnn_benchmark + sm_90 (or Cuda > 11.8 or torch > 1.10 )",472 "id": 218350,473 "title": "[Solved] Maskrcnn_benchmark + sm_90 (or Cuda > 11.8 or torch > 1.10 )",474 "slug": "solved-maskrcnn-benchmark-sm-90-or-cuda-11-8-or-torch-1-10",475 "posts_count": 2,476 "reply_count": 0,477 "highest_post_number": 2,478 "image_url": null,479 "created_at": "2025-03-27T23:49:09.573Z",480 "last_posted_at": "2025-03-28T01:19:51.783Z",481 "bumped": true,482 "bumped_at": "2025-03-28T18:33:02.825Z",483 "archetype": "regular",484 "unseen": false,485 "pinned": false,486 "unpinned": null,487 "visible": true,488 "closed": false,489 "archived": false,490 "bookmarked": null,491 "liked": null,492 "tags_descriptions": {},493 "like_count": 0,494 "views": 101,495 "category_id": 1,496 "featured_link": null,497 "has_accepted_answer": true,498 "posters": [499 {500 "extras": "latest single",501 "description": "Original Poster, Most Recent Poster, Accepted Answer",502 "user": {503 "id": 26909,504 "username": "Priyank_Pathak",505 "name": "Priyank Pathak",506 "avatar_template": "/user_avatar/discuss.pytorch.org/priyank_pathak/{size}/19974_2.png",507 "trust_level": 1508 }509 }510 ]511 },512 {513 "fancy_title": "Model Agnostic Approach to create a 2nd output head?",514 "id": 213427,515 "title": "Model Agnostic Approach to create a 2nd output head?",516 "slug": "model-agnostic-approach-to-create-a-2nd-output-head",517 "posts_count": 3,518 "reply_count": 2,519 "highest_post_number": 4,520 "image_url": null,521 "created_at": "2024-11-25T21:31:06.923Z",522 "last_posted_at": "2024-11-27T13:54:32.512Z",523 "bumped": true,524 "bumped_at": "2024-11-27T13:54:32.512Z",525 "archetype": "regular",526 "unseen": false,527 "pinned": false,528 "unpinned": null,529 "visible": true,530 "closed": false,531 "archived": false,532 "bookmarked": null,533 "liked": null,534 "tags_descriptions": {},535 "like_count": 1,536 "views": 39,537 "category_id": 1,538 "featured_link": null,539 "has_accepted_answer": false,540 "posters": [541 {542 "extras": null,543 "description": "Original Poster",544 "user": {545 "id": 32995,546 "username": "sad_robot",547 "name": "",548 "avatar_template": "/user_avatar/discuss.pytorch.org/sad_robot/{size}/25510_2.png",549 "trust_level": 2550 }551 },552 {553 "extras": "latest",554 "description": "Most Recent Poster",555 "user": {556 "id": 3534,557 "username": "ptrblck",558 "name": "",559 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",560 "admin": true,561 "moderator": true,562 "trust_level": 2563 }564 }565 ]566 }567 ],568 "tags_descriptions": {},569 "fancy_title": "Issues with batch norm",570 "id": 64161,571 "title": "Issues with batch norm",572 "posts_count": 5,573 "created_at": "2019-12-16T06:53:59.076Z",574 "views": 328,575 "reply_count": 3,576 "like_count": 0,577 "last_posted_at": "2019-12-16T07:50:29.513Z",578 "visible": true,579 "closed": false,580 "archived": false,581 "has_summary": false,582 "archetype": "regular",583 "slug": "issues-with-batch-norm",584 "category_id": 1,585 "word_count": 405,586 "deleted_at": null,587 "user_id": 15365,588 "featured_link": null,589 "pinned_globally": false,590 "pinned_at": null,591 "pinned_until": null,592 "image_url": null,593 "slow_mode_seconds": 0,594 "draft": null,595 "draft_key": "topic_64161",596 "draft_sequence": null,597 "unpinned": null,598 "pinned": false,599 "current_post_number": 1,600 "highest_post_number": 5,601 "deleted_by": null,602 "actions_summary": [603 {604 "id": 4,605 "count": 0,606 "hidden": false,607 "can_act": false608 },609 {610 "id": 8,611 "count": 0,612 "hidden": false,613 "can_act": false614 },615 {616 "id": 10,617 "count": 0,618 "hidden": false,619 "can_act": false620 },621 {622 "id": 7,623 "count": 0,624 "hidden": false,625 "can_act": false626 }627 ],628 "chunk_size": 20,629 "bookmarked": false,630 "topic_timer": null,631 "message_bus_last_id": 0,632 "participant_count": 2,633 "show_read_indicator": false,634 "thumbnails": null,635 "slow_mode_enabled_until": null,636 "can_vote": false,637 "vote_count": 0,638 "user_voted": false,639 "discourse_zendesk_plugin_zendesk_id": null,640 "discourse_zendesk_plugin_zendesk_url": "https://your-url.zendesk.com/agent/tickets/",641 "details": {642 "can_edit": false,643 "notification_level": 1,644 "participants": [645 {646 "id": 15365,647 "username": "smu226",648 "name": "Silviu",649 "avatar_template": "/user_avatar/discuss.pytorch.org/smu226/{size}/9063_2.png",650 "post_count": 3,651 "primary_group_name": null,652 "flair_name": null,653 "flair_url": null,654 "flair_color": null,655 "flair_bg_color": null,656 "flair_group_id": null,657 "trust_level": 1658 },659 {660 "id": 3534,661 "username": "ptrblck",662 "name": "",663 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",664 "post_count": 2,665 "primary_group_name": null,666 "flair_name": null,667 "flair_url": null,668 "flair_color": null,669 "flair_bg_color": null,670 "flair_group_id": null,671 "admin": true,672 "moderator": true,673 "trust_level": 2674 }675 ],676 "created_by": {677 "id": 15365,678 "username": "smu226",679 "name": "Silviu",680 "avatar_template": "/user_avatar/discuss.pytorch.org/smu226/{size}/9063_2.png"681 },682 "last_poster": {683 "id": 15365,684 "username": "smu226",685 "name": "Silviu",686 "avatar_template": "/user_avatar/discuss.pytorch.org/smu226/{size}/9063_2.png"687 }688 },689 "bookmarks": []690 },691 {692 "post_stream": {693 "posts": [694 {695 "id": 136408,696 "name": "Yaroslav Bulatov",697 "username": "Yaroslav_Bulatov",698 "avatar_template": "/user_avatar/discuss.pytorch.org/yaroslav_bulatov/{size}/7017_2.png",699 "created_at": "2019-09-23T21:16:15.359Z",700 "cooked": "<p>I’m wondering if it’s safe to assume that users will not call my gradient library from multiple Python threads simultaneously</p>\n<p>IE, are there reasons to support this use-case?</p>\n<p>It seems current PyTorch doesn’t fully support this use case either, thread-global storage for .grad means that two different Python threads can interfere with each other, example <a href=\"https://gist.github.com/yaroslavvb/e53c83c40c8385cd90cdc15c7c61fa63\" rel=\"nofollow noopener\">https://gist.github.com/yaroslavvb/e53c83c40c8385cd90cdc15c7c61fa63</a></p>",701 "post_number": 1,702 "post_type": 1,703 "posts_count": 4,704 "updated_at": "2019-09-23T21:16:15.359Z",705 "reply_count": 0,706 "reply_to_post_number": null,707 "quote_count": 0,708 "incoming_link_count": 793,709 "reads": 48,710 "readers_count": 47,711 "score": 3949.6,712 "yours": false,713 "topic_id": 56666,714 "topic_slug": "assumptions-around-autograd-and-python-multi-threading",715 "display_username": "Yaroslav Bulatov",716 "primary_group_name": null,717 "flair_name": null,718 "flair_url": null,719 "flair_bg_color": null,720 "flair_color": null,721 "flair_group_id": null,722 "badges_granted": [],723 "version": 1,724 "can_edit": false,725 "can_delete": false,726 "can_recover": false,727 "can_see_hidden_post": false,728 "can_wiki": false,729 "link_counts": [730 {731 "url": "https://gist.github.com/yaroslavvb/e53c83c40c8385cd90cdc15c7c61fa63",732 "internal": false,733 "reflection": false,734 "title": "Example of Python multi-threading giving a mix of .grad from different backward calls · GitHub",735 "clicks": 19736 }737 ],738 "read": true,739 "user_title": null,740 "bookmarked": false,741 "actions_summary": [],742 "moderator": false,743 "admin": false,744 "staff": false,745 "user_id": 11965,746 "hidden": false,747 "trust_level": 2,748 "deleted_at": null,749 "user_deleted": false,750 "edit_reason": null,751 "can_view_edit_history": true,752 "wiki": false,753 "post_url": "/t/assumptions-around-autograd-and-python-multi-threading/56666/1",754 "can_accept_answer": false,755 "can_unaccept_answer": false,756 "accepted_answer": false,757 "topic_accepted_answer": null,758 "can_vote": false759 },760 {761 "id": 136629,762 "name": "Alban D",763 "username": "albanD",764 "avatar_template": "/user_avatar/discuss.pytorch.org/alband/{size}/215_2.png",765 "created_at": "2019-09-24T20:07:23.686Z",766 "cooked": "<p>I don’t think we took any particular attention to this case appart from: “It should not deadlock”, “It should compute correct gradients”.<br>\nFor the absence of thread-local storage, my feeling as a python user was that, unless you redeclare something in your new thread, everything is global to all threads.<br>\nIn particular if you want to do learning in multiple threads (not that it is a useful thing to do), you should have two different models to do that.<br>\nYou think we should document / change this behavior?</p>",767 "post_number": 2,768 "post_type": 1,769 "posts_count": 4,770 "updated_at": "2019-09-24T20:07:23.686Z",771 "reply_count": 1,772 "reply_to_post_number": null,773 "quote_count": 0,774 "incoming_link_count": 29,775 "reads": 48,776 "readers_count": 47,777 "score": 154.6,778 "yours": false,779 "topic_id": 56666,780 "topic_slug": "assumptions-around-autograd-and-python-multi-threading",781 "display_username": "Alban D",782 "primary_group_name": null,783 "flair_name": null,784 "flair_url": null,785 "flair_bg_color": null,786 "flair_color": null,787 "flair_group_id": null,788 "badges_granted": [],789 "version": 1,790 "can_edit": false,791 "can_delete": false,792 "can_recover": false,793 "can_see_hidden_post": false,794 "can_wiki": false,795 "read": true,796 "user_title": "",797 "bookmarked": false,798 "actions_summary": [],799 "moderator": true,800 "admin": true,801 "staff": true,802 "user_id": 211,803 "hidden": false,804 "trust_level": 4,805 "deleted_at": null,806 "user_deleted": false,807 "edit_reason": null,808 "can_view_edit_history": true,809 "wiki": false,810 "post_url": "/t/assumptions-around-autograd-and-python-multi-threading/56666/2",811 "can_accept_answer": false,812 "can_unaccept_answer": false,813 "accepted_answer": false,814 "topic_accepted_answer": null815 },816 {817 "id": 136638,818 "name": "Yaroslav Bulatov",819 "username": "Yaroslav_Bulatov",820 "avatar_template": "/user_avatar/discuss.pytorch.org/yaroslav_bulatov/{size}/7017_2.png",821 "created_at": "2019-09-24T20:50:13.805Z",822 "cooked": "<p>Documenting this would be useful. In particular the fact that there’s global autograd engine shared among Python threads, and that a call into <code>backward</code> will block until all concurrent <code>.backward</code> calls complete. Perhaps here? <a href=\"https://pytorch.org/docs/stable/autograd.html#torch.Tensor.backward\" rel=\"nofollow noopener\">https://pytorch.org/docs/stable/autograd.html#torch.Tensor.backward</a></p>\n<p>Disallowing concurrent backward calls in a process seems fine since Python support for concurrency is not great.</p>",823 "post_number": 3,824 "post_type": 1,825 "posts_count": 4,826 "updated_at": "2019-09-24T20:50:13.805Z",827 "reply_count": 0,828 "reply_to_post_number": 2,829 "quote_count": 0,830 "incoming_link_count": 27,831 "reads": 49,832 "readers_count": 48,833 "score": 144.8,834 "yours": false,835 "topic_id": 56666,836 "topic_slug": "assumptions-around-autograd-and-python-multi-threading",837 "display_username": "Yaroslav Bulatov",838 "primary_group_name": null,839 "flair_name": null,840 "flair_url": null,841 "flair_bg_color": null,842 "flair_color": null,843 "flair_group_id": null,844 "badges_granted": [],845 "version": 1,846 "can_edit": false,847 "can_delete": false,848 "can_recover": false,849 "can_see_hidden_post": false,850 "can_wiki": false,851 "link_counts": [852 {853 "url": "https://pytorch.org/docs/stable/autograd.html#torch.Tensor.backward",854 "internal": false,855 "reflection": false,856 "title": "Automatic differentiation package - torch.autograd — PyTorch master documentation",857 "clicks": 38858 }859 ],860 "read": true,861 "user_title": null,862 "reply_to_user": {863 "id": 211,864 "username": "albanD",865 "name": "Alban D",866 "avatar_template": "/user_avatar/discuss.pytorch.org/alband/{size}/215_2.png"867 },868 "bookmarked": false,869 "actions_summary": [],870 "moderator": false,871 "admin": false,872 "staff": false,873 "user_id": 11965,874 "hidden": false,875 "trust_level": 2,876 "deleted_at": null,877 "user_deleted": false,878 "edit_reason": null,879 "can_view_edit_history": true,880 "wiki": false,881 "post_url": "/t/assumptions-around-autograd-and-python-multi-threading/56666/3",882 "can_accept_answer": false,883 "can_unaccept_answer": false,884 "accepted_answer": false,885 "topic_accepted_answer": null886 },887 {888 "id": 153041,889 "name": "creigelde",890 "username": "creigelde",891 "avatar_template": "/letter_avatar_proxy/v4/letter/c/6bbea6/{size}.png",892 "created_at": "2019-12-16T06:28:25.022Z",893 "cooked": "<p>The Python Global Interpreter Lock or GIL, in simple words, is a mutex (or a lock) that allows only one thread to hold the control of the Python interpreter. All the GIL does is make sure only one thread is executing Python code at a time; control still switches between threads. What the GIL prevents then, is making use of more than one CPU core or separate CPUs to run threads in parallel.</p>\n<p>Python threading is great for creating a responsive GUI, or for handling multiple short web requests where I/O is the bottleneck more than the Python code. It is not suitable for parallelizing computationally intensive Python code, stick to the multiprocessing module for such tasks or delegate to a dedicated external library. For actual parallelization in Python, you should use the multiprocessing module to fork multiple processes that execute in parallel (due to the global interpreter lock, Python threads provide interleaving, but they are in fact executed serially, not in parallel, and are only useful when interleaving I/O operations). However, <a href=\"http://net-informations.com/python/pro/threading.htm\" rel=\"nofollow noopener\">threading</a> is still an appropriate model if you want to run multiple I/O-bound tasks simultaneously.</p>",894 "post_number": 4,895 "post_type": 1,896 "posts_count": 4,897 "updated_at": "2019-12-30T06:46:08.800Z",898 "reply_count": 0,899 "reply_to_post_number": null,900 "quote_count": 0,901 "incoming_link_count": 6,902 "reads": 45,903 "readers_count": 44,904 "score": 39.0,905 "yours": false,906 "topic_id": 56666,907 "topic_slug": "assumptions-around-autograd-and-python-multi-threading",908 "display_username": "creigelde",909 "primary_group_name": null,910 "flair_name": null,911 "flair_url": null,912 "flair_bg_color": null,913 "flair_color": null,914 "flair_group_id": null,915 "badges_granted": [],916 "version": 1,917 "can_edit": false,918 "can_delete": false,919 "can_recover": false,920 "can_see_hidden_post": false,921 "can_wiki": false,922 "link_counts": [923 {924 "url": "http://net-informations.com/python/pro/threading.htm",925 "internal": false,926 "reflection": false,927 "title": "Threads and Threading in Python",928 "clicks": 5929 }930 ],931 "read": true,932 "user_title": null,933 "bookmarked": false,934 "actions_summary": [],935 "moderator": false,936 "admin": false,937 "staff": false,938 "user_id": 24840,939 "hidden": false,940 "trust_level": 0,941 "deleted_at": null,942 "user_deleted": false,943 "edit_reason": null,944 "can_view_edit_history": true,945 "wiki": false,946 "post_url": "/t/assumptions-around-autograd-and-python-multi-threading/56666/4",947 "can_accept_answer": false,948 "can_unaccept_answer": false,949 "accepted_answer": false,950 "topic_accepted_answer": null951 }952 ],953 "stream": [954 136408,955 136629,956 136638,957 153041958 ]959 },960 "timeline_lookup": [961 [962 1,963 2224964 ],965 [966 2,967 2223968 ],969 [970 4,971 2141972 ]973 ],974 "suggested_topics": [975 {976 "fancy_title": "How to pinpoint NaN grads?",977 "id": 213942,978 "title": "How to pinpoint NaN grads?",979 "slug": "how-to-pinpoint-nan-grads",980 "posts_count": 3,981 "reply_count": 1,982 "highest_post_number": 3,983 "image_url": null,984 "created_at": "2024-12-07T10:27:15.422Z",985 "last_posted_at": "2024-12-12T20:50:29.568Z",986 "bumped": true,987 "bumped_at": "2024-12-12T20:50:29.568Z",988 "archetype": "regular",989 "unseen": false,990 "pinned": false,991 "unpinned": null,992 "visible": true,993 "closed": false,994 "archived": false,995 "bookmarked": null,996 "liked": null,997 "tags_descriptions": {},998 "like_count": 1,999 "views": 248,1000 "category_id": 7,1001 "featured_link": null,1002 "has_accepted_answer": true,1003 "posters": [1004 {1005 "extras": null,1006 "description": "Original Poster, Accepted Answer",1007 "user": {1008 "id": 79050,1009 "username": "CesareMontresor",1010 "name": "Cesare Montresor",1011 "avatar_template": "/user_avatar/discuss.pytorch.org/cesaremontresor/{size}/72884_2.png",1012 "trust_level": 11013 }1014 },1015 {1016 "extras": "latest",1017 "description": "Most Recent Poster",1018 "user": {1019 "id": 41396,1020 "username": "soulitzer",1021 "name": "",1022 "avatar_template": "/letter_avatar_proxy/v4/letter/s/839c29/{size}.png",1023 "trust_level": 21024 }1025 }1026 ]1027 },1028 {1029 "fancy_title": "Unexpected behavior when using torch.autograd.functional.jacobian with multiple inputs/outputs neural network",1030 "id": 215580,1031 "title": "Unexpected behavior when using torch.autograd.functional.jacobian with multiple inputs/outputs neural network",1032 "slug": "unexpected-behavior-when-using-torch-autograd-functional-jacobian-with-multiple-inputs-outputs-neural-network",1033 "posts_count": 4,1034 "reply_count": 1,1035 "highest_post_number": 5,1036 "image_url": "https://discuss.pytorch.org/uploads/default/optimized/3X/6/d/6dffd4799040f9c82e2216c6f8cb0d6ff91d3cff_2_1024x576.jpeg",1037 "created_at": "2025-01-19T09:33:34.505Z",1038 "last_posted_at": "2025-01-21T01:09:44.326Z",1039 "bumped": true,1040 "bumped_at": "2025-01-21T01:09:44.326Z",1041 "archetype": "regular",1042 "unseen": false,1043 "pinned": false,1044 "unpinned": null,1045 "visible": true,1046 "closed": false,1047 "archived": false,1048 "bookmarked": null,1049 "liked": null,1050 "tags_descriptions": {},1051 "like_count": 1,1052 "views": 52,1053 "category_id": 7,1054 "featured_link": null,1055 "has_accepted_answer": true,1056 "posters": [1057 {1058 "extras": null,1059 "description": "Original Poster",1060 "user": {1061 "id": 81904,1062 "username": "ziv_chen",1063 "name": "ziv",1064 "avatar_template": "/letter_avatar_proxy/v4/letter/z/49beb7/{size}.png",1065 "trust_level": 11066 }1067 },1068 {1069 "extras": "latest",1070 "description": "Most Recent Poster, Accepted Answer",1071 "user": {1072 "id": 18088,1073 "username": "KFrank",1074 "name": "K. Frank",1075 "avatar_template": "/letter_avatar_proxy/v4/letter/k/ecb155/{size}.png",1076 "trust_level": 21077 }1078 }1079 ]1080 },1081 {1082 "fancy_title": "Autograd error on VIT",1083 "id": 214433,1084 "title": "Autograd error on VIT",1085 "slug": "autograd-error-on-vit",1086 "posts_count": 4,1087 "reply_count": 2,1088 "highest_post_number": 4,1089 "image_url": null,1090 "created_at": "2024-12-20T04:02:29.092Z",1091 "last_posted_at": "2024-12-23T00:55:00.806Z",1092 "bumped": true,1093 "bumped_at": "2024-12-23T00:55:00.806Z",1094 "archetype": "regular",1095 "unseen": false,1096 "pinned": false,1097 "unpinned": null,1098 "visible": true,1099 "closed": false,1100 "archived": false,1101 "bookmarked": null,1102 "liked": null,1103 "tags_descriptions": {},1104 "like_count": 2,1105 "views": 240,1106 "category_id": 7,1107 "featured_link": null,1108 "has_accepted_answer": true,1109 "posters": [1110 {1111 "extras": null,1112 "description": "Original Poster",1113 "user": {1114 "id": 81595,1115 "username": "Nikhil_Dugar",1116 "name": "ND",1117 "avatar_template": "/user_avatar/discuss.pytorch.org/nikhil_dugar/{size}/74621_2.png",1118 "trust_level": 11119 }1120 },1121 {1122 "extras": "latest",1123 "description": "Most Recent Poster, Accepted Answer",1124 "user": {1125 "id": 41396,1126 "username": "soulitzer",1127 "name": "",1128 "avatar_template": "/letter_avatar_proxy/v4/letter/s/839c29/{size}.png",1129 "trust_level": 21130 }1131 }1132 ]1133 },1134 {1135 "fancy_title": "How does Pytorch Backward functions handle batched inputs?",1136 "id": 214837,1137 "title": "How does Pytorch Backward functions handle batched inputs?",1138 "slug": "how-does-pytorch-backward-functions-handle-batched-inputs",1139 "posts_count": 5,1140 "reply_count": 3,1141 "highest_post_number": 5,1142 "image_url": null,1143 "created_at": "2025-01-01T02:18:15.016Z",1144 "last_posted_at": "2025-01-04T05:04:34.586Z",1145 "bumped": true,1146 "bumped_at": "2025-01-04T05:04:34.586Z",1147 "archetype": "regular",1148 "unseen": false,1149 "pinned": false,1150 "unpinned": null,1151 "visible": true,1152 "closed": false,1153 "archived": false,1154 "bookmarked": null,1155 "liked": null,1156 "tags_descriptions": {},1157 "like_count": 0,1158 "views": 131,1159 "category_id": 7,1160 "featured_link": null,1161 "has_accepted_answer": true,1162 "posters": [1163 {1164 "extras": "latest",1165 "description": "Original Poster, Most Recent Poster",1166 "user": {1167 "id": 77182,1168 "username": "Siddhanth_Ramani",1169 "name": "Siddhanth Ramani",1170 "avatar_template": "/user_avatar/discuss.pytorch.org/siddhanth_ramani/{size}/62072_2.png",1171 "trust_level": 11172 }1173 },1174 {1175 "extras": null,1176 "description": "Frequent Poster, Accepted Answer",1177 "user": {1178 "id": 18088,1179 "username": "KFrank",1180 "name": "K. Frank",1181 "avatar_template": "/letter_avatar_proxy/v4/letter/k/ecb155/{size}.png",1182 "trust_level": 21183 }1184 }1185 ]1186 },1187 {1188 "fancy_title": "Monitor optimizer step - Adam",1189 "id": 219589,1190 "title": "Monitor optimizer step - Adam",1191 "slug": "monitor-optimizer-step-adam",1192 "posts_count": 2,1193 "reply_count": 0,1194 "highest_post_number": 3,1195 "image_url": null,1196 "created_at": "2025-04-29T10:34:48.814Z",1197 "last_posted_at": "2025-04-30T10:10:17.457Z",1198 "bumped": true,1199 "bumped_at": "2025-04-30T10:10:17.457Z",1200 "archetype": "regular",