Anurag1734/cuda-error-resolution-analysis
07
1[2 {3 "post_stream": {4 "posts": [5 {6 "id": 5171,7 "name": "Andy Brock",8 "username": "ajbrock",9 "avatar_template": "/user_avatar/discuss.pytorch.org/ajbrock/{size}/139_2.png",10 "created_at": "2017-04-09T20:37:22.630Z",11 "cooked": "<p>Hi guys,</p>\n<p>I’ve been running into the sudden appearance of NaNs when I attempt to train using Adam and Half (float16) precision; my nets train just fine on half precision with SGD+nesterov momentum, and they train just fine with single precision (float32) and Adam, but switching them over to half seems to cause numerical instability. I’ve fiddled with the hyperparams a bit; upping epsilon helps a <em>tiny</em> bit but doesn’t fix the issue.</p>\n<p>Is this something anyone else has info on? If not I can throw together a reproduction script and dig into the issue.</p>\n<p>Thanks again! Been a good while since I’ve had to post on account of hitting no issues otherwise.</p>",12 "post_number": 1,13 "post_type": 1,14 "posts_count": 15,15 "updated_at": "2017-04-09T20:37:22.630Z",16 "reply_count": 0,17 "reply_to_post_number": null,18 "quote_count": 0,19 "incoming_link_count": 13106,20 "reads": 472,21 "readers_count": 471,22 "score": 65563.4,23 "yours": false,24 "topic_id": 1765,25 "topic_slug": "adam-half-precision-nans",26 "display_username": "Andy Brock",27 "primary_group_name": null,28 "flair_name": null,29 "flair_url": null,30 "flair_bg_color": null,31 "flair_color": null,32 "flair_group_id": null,33 "badges_granted": [],34 "version": 1,35 "can_edit": false,36 "can_delete": false,37 "can_recover": false,38 "can_see_hidden_post": false,39 "can_wiki": false,40 "link_counts": [41 {42 "url": "https://discuss.pytorch.org/t/the-problematic-adam-stability-in-automatic-mixed-precision-training/137434",43 "internal": true,44 "reflection": true,45 "title": "The problematic Adam stability in Automatic Mixed Precision training",46 "clicks": 847 }48 ],49 "read": true,50 "user_title": null,51 "bookmarked": false,52 "actions_summary": [53 {54 "id": 2,55 "count": 156 }57 ],58 "moderator": false,59 "admin": false,60 "staff": false,61 "user_id": 259,62 "hidden": false,63 "trust_level": 2,64 "deleted_at": null,65 "user_deleted": false,66 "edit_reason": null,67 "can_view_edit_history": true,68 "wiki": false,69 "post_url": "/t/adam-half-precision-nans/1765/1",70 "can_accept_answer": false,71 "can_unaccept_answer": false,72 "accepted_answer": false,73 "topic_accepted_answer": true,74 "can_vote": false75 },76 {77 "id": 5173,78 "name": "",79 "username": "smth",80 "avatar_template": "/user_avatar/discuss.pytorch.org/smth/{size}/13_2.png",81 "created_at": "2017-04-09T22:08:42.629Z",82 "cooked": "<p>half precision is super finicky during training, so I’m not surprised.</p>\n<p>One thing I recommend trying is to do the forward + backward in half precision, but the optimizer step in float precision.<br>\nTo do this, you might have to clone your parameters, and cast them to float32 and once forward+backward is over, you copy over the param .data and .grad into this float32 copy (and call optimizer.step on this float32 copy) and then copy back…</p>\n<p>Other than that, I dont have a good idea of why adam + half is giving NaNs.</p>",83 "post_number": 2,84 "post_type": 1,85 "posts_count": 15,86 "updated_at": "2017-04-09T22:08:42.629Z",87 "reply_count": 2,88 "reply_to_post_number": null,89 "quote_count": 0,90 "incoming_link_count": 198,91 "reads": 469,92 "readers_count": 468,93 "score": 1133.8,94 "yours": false,95 "topic_id": 1765,96 "topic_slug": "adam-half-precision-nans",97 "display_username": "",98 "primary_group_name": null,99 "flair_name": null,100 "flair_url": null,101 "flair_bg_color": null,102 "flair_color": null,103 "flair_group_id": null,104 "badges_granted": [],105 "version": 1,106 "can_edit": false,107 "can_delete": false,108 "can_recover": false,109 "can_see_hidden_post": false,110 "can_wiki": false,111 "read": true,112 "user_title": "PyTorch Dev, Facebook AI Research",113 "title_is_group": false,114 "bookmarked": false,115 "actions_summary": [116 {117 "id": 2,118 "count": 3119 }120 ],121 "moderator": true,122 "admin": true,123 "staff": true,124 "user_id": 1,125 "hidden": false,126 "trust_level": 2,127 "deleted_at": null,128 "user_deleted": false,129 "edit_reason": null,130 "can_view_edit_history": true,131 "wiki": false,132 "post_url": "/t/adam-half-precision-nans/1765/2",133 "can_accept_answer": false,134 "can_unaccept_answer": false,135 "accepted_answer": false,136 "topic_accepted_answer": true137 },138 {139 "id": 5266,140 "name": "Andy Brock",141 "username": "ajbrock",142 "avatar_template": "/user_avatar/discuss.pytorch.org/ajbrock/{size}/139_2.png",143 "created_at": "2017-04-11T20:43:22.965Z",144 "cooked": "<p>Thanks, that’s just the answer I was looking for–will try out the precision swaps and report back.</p>",145 "post_number": 3,146 "post_type": 1,147 "posts_count": 15,148 "updated_at": "2017-04-11T20:43:22.965Z",149 "reply_count": 0,150 "reply_to_post_number": 2,151 "quote_count": 0,152 "incoming_link_count": 134,153 "reads": 440,154 "readers_count": 439,155 "score": 758.0,156 "yours": false,157 "topic_id": 1765,158 "topic_slug": "adam-half-precision-nans",159 "display_username": "Andy Brock",160 "primary_group_name": null,161 "flair_name": null,162 "flair_url": null,163 "flair_bg_color": null,164 "flair_color": null,165 "flair_group_id": null,166 "badges_granted": [],167 "version": 1,168 "can_edit": false,169 "can_delete": false,170 "can_recover": false,171 "can_see_hidden_post": false,172 "can_wiki": false,173 "read": true,174 "user_title": null,175 "reply_to_user": {176 "id": 1,177 "username": "smth",178 "name": "",179 "avatar_template": "/user_avatar/discuss.pytorch.org/smth/{size}/13_2.png"180 },181 "bookmarked": false,182 "actions_summary": [],183 "moderator": false,184 "admin": false,185 "staff": false,186 "user_id": 259,187 "hidden": false,188 "trust_level": 2,189 "deleted_at": null,190 "user_deleted": false,191 "edit_reason": null,192 "can_view_edit_history": true,193 "wiki": false,194 "post_url": "/t/adam-half-precision-nans/1765/3",195 "can_accept_answer": false,196 "can_unaccept_answer": false,197 "accepted_answer": false,198 "topic_accepted_answer": true199 },200 {201 "id": 6663,202 "name": "Andy Brock",203 "username": "ajbrock",204 "avatar_template": "/user_avatar/discuss.pytorch.org/ajbrock/{size}/139_2.png",205 "created_at": "2017-04-29T14:08:15.578Z",206 "cooked": "<p>EDIT: see below, it looks like eps was the culprit after all, no need for this solution.</p>\n<p>Alright, got this working by just hanging onto fp32 copies of the parameters and keeping all of the Adam values in fp32 as well, as shown in <a href=\"https://gist.github.com/ajbrock/075c0ca4036dc4d8581990a6e76e07a3\" rel=\"nofollow noopener\">This Gist</a>. I suspect you could get the desired stability and do this even more efficiently by just keeping the Adam values in fp32 (I think there’s a divide-by-0 happening somewhere) but this gives the desired memory reduction without any loss of speed over fp32.</p>",207 "post_number": 4,208 "post_type": 1,209 "posts_count": 15,210 "updated_at": "2017-04-29T15:02:03.450Z",211 "reply_count": 0,212 "reply_to_post_number": 2,213 "quote_count": 0,214 "incoming_link_count": 378,215 "reads": 419,216 "readers_count": 418,217 "score": 1983.8,218 "yours": false,219 "topic_id": 1765,220 "topic_slug": "adam-half-precision-nans",221 "display_username": "Andy Brock",222 "primary_group_name": null,223 "flair_name": null,224 "flair_url": null,225 "flair_bg_color": null,226 "flair_color": null,227 "flair_group_id": null,228 "badges_granted": [],229 "version": 2,230 "can_edit": false,231 "can_delete": false,232 "can_recover": false,233 "can_see_hidden_post": false,234 "can_wiki": false,235 "link_counts": [236 {237 "url": "https://gist.github.com/ajbrock/075c0ca4036dc4d8581990a6e76e07a3",238 "internal": false,239 "reflection": false,240 "title": "FP16 Adam for PyTorch · GitHub",241 "clicks": 388242 }243 ],244 "read": true,245 "user_title": null,246 "reply_to_user": {247 "id": 1,248 "username": "smth",249 "name": "",250 "avatar_template": "/user_avatar/discuss.pytorch.org/smth/{size}/13_2.png"251 },252 "bookmarked": false,253 "actions_summary": [254 {255 "id": 2,256 "count": 1257 }258 ],259 "moderator": false,260 "admin": false,261 "staff": false,262 "user_id": 259,263 "hidden": false,264 "trust_level": 2,265 "deleted_at": null,266 "user_deleted": false,267 "edit_reason": null,268 "can_view_edit_history": true,269 "wiki": false,270 "post_url": "/t/adam-half-precision-nans/1765/4",271 "can_accept_answer": false,272 "can_unaccept_answer": false,273 "accepted_answer": false,274 "topic_accepted_answer": true275 },276 {277 "id": 6664,278 "name": "Adam Paszke",279 "username": "apaszke",280 "avatar_template": "/user_avatar/discuss.pytorch.org/apaszke/{size}/21_2.png",281 "created_at": "2017-04-29T14:38:26.901Z",282 "cooked": "<p>It’s probably a 0 division somewhere. Have you tried using a much larger eps (say 1e-4)? The default 1e-8 is rounded to 0 in half precision.</p>",283 "post_number": 5,284 "post_type": 1,285 "posts_count": 15,286 "updated_at": "2021-05-13T07:30:45.388Z",287 "reply_count": 3,288 "reply_to_post_number": null,289 "quote_count": 0,290 "incoming_link_count": 399,291 "reads": 397,292 "readers_count": 396,293 "score": 2359.4,294 "yours": false,295 "topic_id": 1765,296 "topic_slug": "adam-half-precision-nans",297 "display_username": "Adam Paszke",298 "primary_group_name": null,299 "flair_name": null,300 "flair_url": null,301 "flair_bg_color": null,302 "flair_color": null,303 "flair_group_id": null,304 "badges_granted": [],305 "version": 1,306 "can_edit": false,307 "can_delete": false,308 "can_recover": false,309 "can_see_hidden_post": false,310 "can_wiki": false,311 "read": true,312 "user_title": "Chief Crazy Person @ PyTorch",313 "title_is_group": false,314 "bookmarked": false,315 "actions_summary": [316 {317 "id": 2,318 "count": 18319 }320 ],321 "moderator": true,322 "admin": true,323 "staff": true,324 "user_id": 2,325 "hidden": false,326 "trust_level": 4,327 "deleted_at": null,328 "user_deleted": false,329 "edit_reason": null,330 "can_view_edit_history": true,331 "wiki": false,332 "post_url": "/t/adam-half-precision-nans/1765/5",333 "can_accept_answer": false,334 "can_unaccept_answer": false,335 "accepted_answer": true,336 "topic_accepted_answer": true337 },338 {339 "id": 6673,340 "name": "Andy Brock",341 "username": "ajbrock",342 "avatar_template": "/user_avatar/discuss.pytorch.org/ajbrock/{size}/139_2.png",343 "created_at": "2017-04-29T15:00:42.795Z",344 "cooked": "<p>I had previously tried upping epsilon after tagging it as the culprit, but I can’t recall to exactly what values–as of right now I’m training with eps=1e-4 and it’s working just fine. Guess I should have dug into that further, thanks!</p>",345 "post_number": 6,346 "post_type": 1,347 "posts_count": 15,348 "updated_at": "2017-04-29T15:00:42.795Z",349 "reply_count": 1,350 "reply_to_post_number": 5,351 "quote_count": 0,352 "incoming_link_count": 515,353 "reads": 388,354 "readers_count": 387,355 "score": 2732.6,356 "yours": false,357 "topic_id": 1765,358 "topic_slug": "adam-half-precision-nans",359 "display_username": "Andy Brock",360 "primary_group_name": null,361 "flair_name": null,362 "flair_url": null,363 "flair_bg_color": null,364 "flair_color": null,365 "flair_group_id": null,366 "badges_granted": [],367 "version": 1,368 "can_edit": false,369 "can_delete": false,370 "can_recover": false,371 "can_see_hidden_post": false,372 "can_wiki": false,373 "read": true,374 "user_title": null,375 "reply_to_user": {376 "id": 2,377 "username": "apaszke",378 "name": "Adam Paszke",379 "avatar_template": "/user_avatar/discuss.pytorch.org/apaszke/{size}/21_2.png"380 },381 "bookmarked": false,382 "actions_summary": [383 {384 "id": 2,385 "count": 3386 }387 ],388 "moderator": false,389 "admin": false,390 "staff": false,391 "user_id": 259,392 "hidden": false,393 "trust_level": 2,394 "deleted_at": null,395 "user_deleted": false,396 "edit_reason": null,397 "can_view_edit_history": true,398 "wiki": false,399 "post_url": "/t/adam-half-precision-nans/1765/6",400 "can_accept_answer": false,401 "can_unaccept_answer": false,402 "accepted_answer": false,403 "topic_accepted_answer": true404 },405 {406 "id": 6710,407 "name": "Michael Klachko",408 "username": "michaelklachko",409 "avatar_template": "/user_avatar/discuss.pytorch.org/michaelklachko/{size}/541_2.png",410 "created_at": "2017-04-30T17:28:20.473Z",411 "cooked": "<p>You might want to look at this paper:<br>\n<a href=\"https://arxiv.org/abs/1609.07061\" class=\"onebox\" target=\"_blank\" rel=\"nofollow noopener\">https://arxiv.org/abs/1609.07061</a><br>\nif you’re willing to keep a copy of weights/gradients in FP32 you might be able to reduce the precision of forward/backward step much further than FP16.</p>",412 "post_number": 7,413 "post_type": 1,414 "posts_count": 15,415 "updated_at": "2017-04-30T17:28:20.473Z",416 "reply_count": 0,417 "reply_to_post_number": 6,418 "quote_count": 0,419 "incoming_link_count": 66,420 "reads": 360,421 "readers_count": 359,422 "score": 417.0,423 "yours": false,424 "topic_id": 1765,425 "topic_slug": "adam-half-precision-nans",426 "display_username": "Michael Klachko",427 "primary_group_name": null,428 "flair_name": null,429 "flair_url": null,430 "flair_bg_color": null,431 "flair_color": null,432 "flair_group_id": null,433 "badges_granted": [],434 "version": 1,435 "can_edit": false,436 "can_delete": false,437 "can_recover": false,438 "can_see_hidden_post": false,439 "can_wiki": false,440 "link_counts": [441 {442 "url": "https://arxiv.org/abs/1609.07061",443 "internal": false,444 "reflection": false,445 "title": "[1609.07061] Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations",446 "clicks": 429447 }448 ],449 "read": true,450 "user_title": null,451 "reply_to_user": {452 "id": 259,453 "username": "ajbrock",454 "name": "Andy Brock",455 "avatar_template": "/user_avatar/discuss.pytorch.org/ajbrock/{size}/139_2.png"456 },457 "bookmarked": false,458 "actions_summary": [459 {460 "id": 2,461 "count": 1462 }463 ],464 "moderator": false,465 "admin": false,466 "staff": false,467 "user_id": 1161,468 "hidden": false,469 "trust_level": 2,470 "deleted_at": null,471 "user_deleted": false,472 "edit_reason": null,473 "can_view_edit_history": true,474 "wiki": false,475 "post_url": "/t/adam-half-precision-nans/1765/7",476 "can_accept_answer": false,477 "can_unaccept_answer": false,478 "accepted_answer": false,479 "topic_accepted_answer": true480 },481 {482 "id": 82188,483 "name": "",484 "username": "iidsample",485 "avatar_template": "/letter_avatar_proxy/v4/letter/i/3d9bf3/{size}.png",486 "created_at": "2018-12-15T21:17:00.538Z",487 "cooked": "<p>For anybody who arrives here through a google search -<br>\nThis is a paper by Nvidia which sheds more light on training in FP16.<br>\n<a href=\"https://arxiv.org/abs/1710.03740\" class=\"onebox\" target=\"_blank\" rel=\"nofollow noopener\">https://arxiv.org/abs/1710.03740</a></p>\n<p>Also they have been nice to provide implemented code -<br>\n<aside class=\"onebox whitelistedgeneric\">\n <header class=\"source\">\n <img src=\"https://github.githubassets.com/favicon.ico\" class=\"site-icon\" width=\"16\" height=\"16\">\n <a href=\"https://github.com/NVIDIA/apex\" target=\"_blank\" rel=\"nofollow noopener\">GitHub</a>\n </header>\n <article class=\"onebox-body\">\n <img src=\"https://avatars0.githubusercontent.com/u/1728152?s=400&v=4\" class=\"thumbnail\" width=\"\" height=\"\">\n\n<h3><a href=\"https://github.com/NVIDIA/apex\" target=\"_blank\" rel=\"nofollow noopener\">NVIDIA/apex</a></h3>\n\n<p>A PyTorch Extension: Tools for easy mixed precision and distributed training in Pytorch - NVIDIA/apex</p>\n\n\n </article>\n <div class=\"onebox-metadata\">\n \n \n </div>\n <div style=\"clear: both\"></div>\n</aside>\n</p>",488 "post_number": 8,489 "post_type": 1,490 "posts_count": 15,491 "updated_at": "2018-12-15T21:17:00.538Z",492 "reply_count": 0,493 "reply_to_post_number": null,494 "quote_count": 0,495 "incoming_link_count": 104,496 "reads": 273,497 "readers_count": 272,498 "score": 574.6,499 "yours": false,500 "topic_id": 1765,501 "topic_slug": "adam-half-precision-nans",502 "display_username": "",503 "primary_group_name": null,504 "flair_name": null,505 "flair_url": null,506 "flair_bg_color": null,507 "flair_color": null,508 "flair_group_id": null,509 "badges_granted": [],510 "version": 1,511 "can_edit": false,512 "can_delete": false,513 "can_recover": false,514 "can_see_hidden_post": false,515 "can_wiki": false,516 "link_counts": [517 {518 "url": "https://arxiv.org/abs/1710.03740",519 "internal": false,520 "reflection": false,521 "title": "[1710.03740] Mixed Precision Training",522 "clicks": 445523 },524 {525 "url": "https://github.com/NVIDIA/apex",526 "internal": false,527 "reflection": false,528 "title": "GitHub - NVIDIA/apex: A PyTorch Extension: Tools for easy mixed precision and distributed training in Pytorch",529 "clicks": 173530 }531 ],532 "read": true,533 "user_title": null,534 "bookmarked": false,535 "actions_summary": [],536 "moderator": false,537 "admin": false,538 "staff": false,539 "user_id": 2227,540 "hidden": false,541 "trust_level": 2,542 "deleted_at": null,543 "user_deleted": false,544 "edit_reason": null,545 "can_view_edit_history": true,546 "wiki": false,547 "post_url": "/t/adam-half-precision-nans/1765/8",548 "can_accept_answer": false,549 "can_unaccept_answer": false,550 "accepted_answer": false,551 "topic_accepted_answer": true552 },553 {554 "id": 236099,555 "name": "Lin Jia",556 "username": "Lin_Jia",557 "avatar_template": "/user_avatar/discuss.pytorch.org/lin_jia/{size}/22799_2.png",558 "created_at": "2020-10-14T03:50:58.788Z",559 "cooked": "<p>This is a very old post, and google search took me here. For future references, it seems Adam is now adapted to do half-precision training with tuning on hyperparameters:</p>\n<aside class=\"onebox allowlistedgeneric\">\n <header class=\"source\">\n <a href=\"https://www.programmersought.com/article/22245145270/\" target=\"_blank\" rel=\"noopener nofollow ugc\">programmersought.com</a>\n </header>\n <article class=\"onebox-body\">\n <img src=\"\" class=\"thumbnail\" width=\"\" height=\"\">\n\n<h3><a href=\"https://www.programmersought.com/article/22245145270/\" target=\"_blank\" rel=\"noopener nofollow ugc\">pytorch1.1 half-precision training Adam RMSprop optimizer Nan problem -...</a></h3>\n\n\n\n </article>\n <div class=\"onebox-metadata\">\n \n \n </div>\n <div style=\"clear: both\"></div>\n</aside>\n",560 "post_number": 9,561 "post_type": 1,562 "posts_count": 15,563 "updated_at": "2020-10-14T03:50:58.788Z",564 "reply_count": 0,565 "reply_to_post_number": null,566 "quote_count": 0,567 "incoming_link_count": 66,568 "reads": 134,569 "readers_count": 133,570 "score": 356.8,571 "yours": false,572 "topic_id": 1765,573 "topic_slug": "adam-half-precision-nans",574 "display_username": "Lin Jia",575 "primary_group_name": null,576 "flair_name": null,577 "flair_url": null,578 "flair_bg_color": null,579 "flair_color": null,580 "flair_group_id": null,581 "badges_granted": [],582 "version": 1,583 "can_edit": false,584 "can_delete": false,585 "can_recover": false,586 "can_see_hidden_post": false,587 "can_wiki": false,588 "link_counts": [589 {590 "url": "https://www.programmersought.com/article/22245145270/",591 "internal": false,592 "reflection": false,593 "title": "pytorch1.1 half-precision training Adam RMSprop optimizer Nan problem - Programmer Sought",594 "clicks": 492595 }596 ],597 "read": true,598 "user_title": null,599 "bookmarked": false,600 "actions_summary": [],601 "moderator": false,602 "admin": false,603 "staff": false,604 "user_id": 30054,605 "hidden": false,606 "trust_level": 2,607 "deleted_at": null,608 "user_deleted": false,609 "edit_reason": null,610 "can_view_edit_history": true,611 "wiki": false,612 "post_url": "/t/adam-half-precision-nans/1765/9",613 "can_accept_answer": false,614 "can_unaccept_answer": false,615 "accepted_answer": false,616 "topic_accepted_answer": true617 },618 {619 "id": 248462,620 "name": "Alexander Mathiasen",621 "username": "alexmath",622 "avatar_template": "/user_avatar/discuss.pytorch.org/alexmath/{size}/16393_2.png",623 "created_at": "2020-12-01T17:39:40.112Z",624 "cooked": "<p>I had a similar issue. Changing BCELoss to BCEWithLogitsLoss and Adam epsilon from 10**(-8) to 10**(-4) worked for me.</p>\n<p>I found it useful to inspect the computed gradients when debugging.</p>\n<pre><code class=\"lang-auto\">print([p.grad for p in model.parameters()])\n</code></pre>",625 "post_number": 10,626 "post_type": 1,627 "posts_count": 15,628 "updated_at": "2020-12-01T17:39:40.112Z",629 "reply_count": 0,630 "reply_to_post_number": null,631 "quote_count": 0,632 "incoming_link_count": 77,633 "reads": 112,634 "readers_count": 111,635 "score": 407.4,636 "yours": false,637 "topic_id": 1765,638 "topic_slug": "adam-half-precision-nans",639 "display_username": "Alexander Mathiasen",640 "primary_group_name": null,641 "flair_name": null,642 "flair_url": null,643 "flair_bg_color": null,644 "flair_color": null,645 "flair_group_id": null,646 "badges_granted": [],647 "version": 1,648 "can_edit": false,649 "can_delete": false,650 "can_recover": false,651 "can_see_hidden_post": false,652 "can_wiki": false,653 "read": true,654 "user_title": null,655 "bookmarked": false,656 "actions_summary": [],657 "moderator": false,658 "admin": false,659 "staff": false,660 "user_id": 23045,661 "hidden": false,662 "trust_level": 1,663 "deleted_at": null,664 "user_deleted": false,665 "edit_reason": null,666 "can_view_edit_history": true,667 "wiki": false,668 "post_url": "/t/adam-half-precision-nans/1765/10",669 "can_accept_answer": false,670 "can_unaccept_answer": false,671 "accepted_answer": false,672 "topic_accepted_answer": true673 },674 {675 "id": 260446,676 "name": "",677 "username": "gingsi",678 "avatar_template": "/user_avatar/discuss.pytorch.org/gingsi/{size}/34033_2.png",679 "created_at": "2021-01-30T17:54:16.108Z",680 "cooked": "<p>For me the backward function died before the optimizer was even called. My loss was very high and dividing the loss by 1e5 as last step before the backward pass helped to get rid of the NaN. Of course, performance could suffer, learning rate may need to be tuned etc.</p>",681 "post_number": 11,682 "post_type": 1,683 "posts_count": 15,684 "updated_at": "2021-01-30T17:54:16.108Z",685 "reply_count": 0,686 "reply_to_post_number": null,687 "quote_count": 0,688 "incoming_link_count": 37,689 "reads": 95,690 "readers_count": 94,691 "score": 204.0,692 "yours": false,693 "topic_id": 1765,694 "topic_slug": "adam-half-precision-nans",695 "display_username": "",696 "primary_group_name": null,697 "flair_name": null,698 "flair_url": null,699 "flair_bg_color": null,700 "flair_color": null,701 "flair_group_id": null,702 "badges_granted": [],703 "version": 1,704 "can_edit": false,705 "can_delete": false,706 "can_recover": false,707 "can_see_hidden_post": false,708 "can_wiki": false,709 "read": true,710 "user_title": null,711 "bookmarked": false,712 "actions_summary": [],713 "moderator": false,714 "admin": false,715 "staff": false,716 "user_id": 41558,717 "hidden": false,718 "trust_level": 1,719 "deleted_at": null,720 "user_deleted": false,721 "edit_reason": null,722 "can_view_edit_history": true,723 "wiki": false,724 "post_url": "/t/adam-half-precision-nans/1765/11",725 "can_accept_answer": false,726 "can_unaccept_answer": false,727 "accepted_answer": false,728 "topic_accepted_answer": true729 },730 {731 "id": 298557,732 "name": "Matt Raymond",733 "username": "m4ttr4ymond",734 "avatar_template": "/user_avatar/discuss.pytorch.org/m4ttr4ymond/{size}/40764_2.png",735 "created_at": "2021-07-29T19:31:11.147Z",736 "cooked": "<p>I had this problem with TensorFlow, but this is the only post I found discussing it, so I might as well share my solution. Float16s can only represent numbers as small as 10e-5, but the default adam epsilons for TensorFlow and PyTorch are lower than this (10e-7 and 10e-8 respectively). This seems to cause underflow errors when using float16s. Changing the epsilon to 10e-4 solved the problem for me.</p>",737 "post_number": 12,738 "post_type": 1,739 "posts_count": 15,740 "updated_at": "2021-07-29T19:31:48.781Z",741 "reply_count": 0,742 "reply_to_post_number": null,743 "quote_count": 0,744 "incoming_link_count": 66,745 "reads": 71,746 "readers_count": 70,747 "score": 359.2,748 "yours": false,749 "topic_id": 1765,750 "topic_slug": "adam-half-precision-nans",751 "display_username": "Matt Raymond",752 "primary_group_name": null,753 "flair_name": null,754 "flair_url": null,755 "flair_bg_color": null,756 "flair_color": null,757 "flair_group_id": null,758 "badges_granted": [],759 "version": 3,760 "can_edit": false,761 "can_delete": false,762 "can_recover": false,763 "can_see_hidden_post": false,764 "can_wiki": false,765 "read": true,766 "user_title": null,767 "bookmarked": false,768 "actions_summary": [769 {770 "id": 2,771 "count": 1772 }773 ],774 "moderator": false,775 "admin": false,776 "staff": false,777 "user_id": 47664,778 "hidden": false,779 "trust_level": 0,780 "deleted_at": null,781 "user_deleted": false,782 "edit_reason": null,783 "can_view_edit_history": true,784 "wiki": false,785 "post_url": "/t/adam-half-precision-nans/1765/12",786 "can_accept_answer": false,787 "can_unaccept_answer": false,788 "accepted_answer": false,789 "topic_accepted_answer": true790 },791 {792 "id": 415640,793 "name": "Yi, Hongkyu",794 "username": "Zeratul",795 "avatar_template": "/user_avatar/discuss.pytorch.org/zeratul/{size}/44616_2.png",796 "created_at": "2023-08-29T06:40:13.524Z",797 "cooked": "<p>Wow, thank you so much !!<img src=\"https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12\" title=\":slight_smile:\" class=\"emoji\" alt=\":slight_smile:\" loading=\"lazy\" width=\"20\" height=\"20\"> Your solution works really well !!!</p>",798 "post_number": 13,799 "post_type": 1,800 "posts_count": 15,801 "updated_at": "2023-08-29T06:40:13.524Z",802 "reply_count": 0,803 "reply_to_post_number": 5,804 "quote_count": 0,805 "incoming_link_count": 24,806 "reads": 25,807 "readers_count": 24,808 "score": 140.0,809 "yours": false,810 "topic_id": 1765,811 "topic_slug": "adam-half-precision-nans",812 "display_username": "Yi, Hongkyu",813 "primary_group_name": null,814 "flair_name": null,815 "flair_url": null,816 "flair_bg_color": null,817 "flair_color": null,818 "flair_group_id": null,819 "badges_granted": [],820 "version": 1,821 "can_edit": false,822 "can_delete": false,823 "can_recover": false,824 "can_see_hidden_post": false,825 "can_wiki": false,826 "read": true,827 "user_title": null,828 "reply_to_user": {829 "id": 2,830 "username": "apaszke",831 "name": "Adam Paszke",832 "avatar_template": "/user_avatar/discuss.pytorch.org/apaszke/{size}/21_2.png"833 },834 "bookmarked": false,835 "actions_summary": [836 {837 "id": 2,838 "count": 1839 }840 ],841 "moderator": false,842 "admin": false,843 "staff": false,844 "user_id": 51312,845 "hidden": false,846 "trust_level": 1,847 "deleted_at": null,848 "user_deleted": false,849 "edit_reason": null,850 "can_view_edit_history": true,851 "wiki": false,852 "post_url": "/t/adam-half-precision-nans/1765/13",853 "can_accept_answer": false,854 "can_unaccept_answer": false,855 "accepted_answer": false,856 "topic_accepted_answer": true857 },858 {859 "id": 422608,860 "name": "Mark Hamazaspyan",861 "username": "Mark_Hamazaspyan",862 "avatar_template": "/user_avatar/discuss.pytorch.org/mark_hamazaspyan/{size}/65173_2.png",863 "created_at": "2023-11-02T06:48:12.996Z",864 "cooked": "<p>Another solution, use bitsandbytes. It has Adam8bit optimizer.</p>\n<pre><code class=\"lang-auto\">import bitsandbytes as bnb\n\n# adam = torch.optim.Adam(model.parameters(), lr=0.001, betas=(0.9, 0.995)) # comment out old optimizer\nadam = bnb.optim.Adam8bit(model.parameters(), lr=0.001, betas=(0.9, 0.995)) # add bnb optimizer\n</code></pre>\n<aside class=\"onebox allowlistedgeneric\" data-onebox-src=\"https://github.com/TimDettmers/bitsandbytes\">\n <header class=\"source\">\n <img src=\"https://github.githubassets.com/favicons/favicon.svg\" class=\"site-icon\" width=\"32\" height=\"32\">\n\n <a href=\"https://github.com/TimDettmers/bitsandbytes\" target=\"_blank\" rel=\"noopener nofollow ugc\">GitHub</a>\n </header>\n\n <article class=\"onebox-body\">\n <div class=\"aspect-image\" style=\"--aspect-ratio:690/344;\"><img src=\"https://opengraph.githubassets.com/4bd24347f5962bdaea38eed0dc568f503e4d4372055cda966b576564c807a83a/TimDettmers/bitsandbytes\" class=\"thumbnail\" width=\"690\" height=\"345\"></div>\n\n<h3><a href=\"https://github.com/TimDettmers/bitsandbytes\" target=\"_blank\" rel=\"noopener nofollow ugc\">GitHub - TimDettmers/bitsandbytes: 8-bit CUDA functions for PyTorch</a></h3>\n\n <p>8-bit CUDA functions for PyTorch. Contribute to TimDettmers/bitsandbytes development by creating an account on GitHub.</p>\n\n\n </article>\n\n <div class=\"onebox-metadata\">\n \n \n </div>\n\n <div style=\"clear: both\"></div>\n</aside>\n",865 "post_number": 14,866 "post_type": 1,867 "posts_count": 15,868 "updated_at": "2023-11-02T06:48:12.996Z",869 "reply_count": 0,870 "reply_to_post_number": null,871 "quote_count": 0,872 "incoming_link_count": 34,873 "reads": 23,874 "readers_count": 22,875 "score": 174.6,876 "yours": false,877 "topic_id": 1765,878 "topic_slug": "adam-half-precision-nans",879 "display_username": "Mark Hamazaspyan",880 "primary_group_name": null,881 "flair_name": null,882 "flair_url": null,883 "flair_bg_color": null,884 "flair_color": null,885 "flair_group_id": null,886 "badges_granted": [],887 "version": 1,888 "can_edit": false,889 "can_delete": false,890 "can_recover": false,891 "can_see_hidden_post": false,892 "can_wiki": false,893 "link_counts": [894 {895 "url": "https://github.com/TimDettmers/bitsandbytes",896 "internal": false,897 "reflection": false,898 "title": "GitHub - TimDettmers/bitsandbytes: 8-bit CUDA functions for PyTorch",899 "clicks": 53900 }901 ],902 "read": true,903 "user_title": null,904 "bookmarked": false,905 "actions_summary": [],906 "moderator": false,907 "admin": false,908 "staff": false,909 "user_id": 70628,910 "hidden": false,911 "trust_level": 1,912 "deleted_at": null,913 "user_deleted": false,914 "edit_reason": null,915 "can_view_edit_history": true,916 "wiki": false,917 "post_url": "/t/adam-half-precision-nans/1765/14",918 "can_accept_answer": false,919 "can_unaccept_answer": false,920 "accepted_answer": false,921 "topic_accepted_answer": true922 },923 {924 "id": 451601,925 "name": "",926 "username": "Nth1",927 "avatar_template": "/user_avatar/discuss.pytorch.org/nth1/{size}/63959_2.png",928 "created_at": "2024-08-10T20:26:34.836Z",929 "cooked": "<p>Thank you very much <img src=\"https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12\" title=\":slight_smile:\" class=\"emoji\" alt=\":slight_smile:\" loading=\"lazy\" width=\"20\" height=\"20\"></p>",930 "post_number": 15,931 "post_type": 1,932 "posts_count": 15,933 "updated_at": "2024-08-10T20:26:34.836Z",934 "reply_count": 0,935 "reply_to_post_number": 5,936 "quote_count": 0,937 "incoming_link_count": 10,938 "reads": 12,939 "readers_count": 11,940 "score": 52.4,941 "yours": false,942 "topic_id": 1765,943 "topic_slug": "adam-half-precision-nans",944 "display_username": "",945 "primary_group_name": null,946 "flair_name": null,947 "flair_url": null,948 "flair_bg_color": null,949 "flair_color": null,950 "flair_group_id": null,951 "badges_granted": [],952 "version": 1,953 "can_edit": false,954 "can_delete": false,955 "can_recover": false,956 "can_see_hidden_post": false,957 "can_wiki": false,958 "read": true,959 "user_title": null,960 "reply_to_user": {961 "id": 2,962 "username": "apaszke",963 "name": "Adam Paszke",964 "avatar_template": "/user_avatar/discuss.pytorch.org/apaszke/{size}/21_2.png"965 },966 "bookmarked": false,967 "actions_summary": [],968 "moderator": false,969 "admin": false,970 "staff": false,971 "user_id": 77928,972 "hidden": false,973 "trust_level": 1,974 "deleted_at": null,975 "user_deleted": false,976 "edit_reason": null,977 "can_view_edit_history": true,978 "wiki": false,979 "post_url": "/t/adam-half-precision-nans/1765/15",980 "can_accept_answer": false,981 "can_unaccept_answer": false,982 "accepted_answer": false,983 "topic_accepted_answer": true984 }985 ],986 "stream": [987 5171,988 5173,989 5266,990 6663,991 6664,992 6673,993 6710,994 82188,995 236099,996 248462,997 260446,998 298557,999 415640,1000 422608,1001 4516011002 ]1003 },1004 "timeline_lookup": [1005 [1006 1,1007 31211008 ],1009 [1010 3,1011 31191012 ],1013 [1014 4,1015 31011016 ],1017 [1018 7,1019 31001020 ],1021 [1022 8,1023 25061024 ],1025 [1026 9,1027 18381028 ],1029 [1030 10,1031 17891032 ],1033 [1034 11,1035 17291036 ],1037 [1038 12,1039 15491040 ],1041 [1042 13,1043 7881044 ],1045 [1046 14,1047 7231048 ],1049 [1050 15,1051 4411052 ]1053 ],1054 "suggested_topics": [1055 {1056 "fancy_title": "Distributed app examples is crashing with error",1057 "id": 213442,1058 "title": "Distributed app examples is crashing with error",1059 "slug": "distributed-app-examples-is-crashing-with-error",1060 "posts_count": 2,1061 "reply_count": 0,1062 "highest_post_number": 2,1063 "image_url": null,1064 "created_at": "2024-11-26T06:00:28.350Z",1065 "last_posted_at": "2024-11-26T18:54:26.852Z",1066 "bumped": true,1067 "bumped_at": "2024-11-26T18:54:26.852Z",1068 "archetype": "regular",1069 "unseen": false,1070 "pinned": false,1071 "unpinned": null,1072 "visible": true,1073 "closed": false,1074 "archived": false,1075 "bookmarked": null,1076 "liked": null,1077 "tags_descriptions": {},1078 "like_count": 0,1079 "views": 54,1080 "category_id": 1,1081 "featured_link": null,1082 "has_accepted_answer": false,1083 "posters": [1084 {1085 "extras": null,1086 "description": "Original Poster",1087 "user": {1088 "id": 70312,1089 "username": "jury_duty",1090 "name": "jury duty",1091 "avatar_template": "/user_avatar/discuss.pytorch.org/jury_duty/{size}/64824_2.png",1092 "trust_level": 01093 }1094 },1095 {1096 "extras": "latest",1097 "description": "Most Recent Poster",1098 "user": {1099 "id": 3534,1100 "username": "ptrblck",1101 "name": "",1102 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",1103 "admin": true,1104 "moderator": true,1105 "trust_level": 21106 }1107 }1108 ]1109 },1110 {1111 "fancy_title": "Need help pivoting this code such that it handles a dataset with 2d output",1112 "id": 215346,1113 "title": "Need help pivoting this code such that it handles a dataset with 2d output",1114 "slug": "need-help-pivoting-this-code-such-that-it-handles-a-dataset-with-2d-output",1115 "posts_count": 1,1116 "reply_count": 0,1117 "highest_post_number": 1,1118 "image_url": null,1119 "created_at": "2025-01-14T00:43:33.562Z",1120 "last_posted_at": "2025-01-14T00:43:33.600Z",1121 "bumped": true,1122 "bumped_at": "2025-01-14T00:43:33.600Z",1123 "archetype": "regular",1124 "unseen": false,1125 "pinned": false,1126 "unpinned": null,1127 "visible": true,1128 "closed": false,1129 "archived": false,1130 "bookmarked": null,1131 "liked": null,1132 "tags_descriptions": {},1133 "like_count": 0,1134 "views": 22,1135 "category_id": 1,1136 "featured_link": null,1137 "has_accepted_answer": false,1138 "posters": [1139 {1140 "extras": "latest single",1141 "description": "Original Poster, Most Recent Poster",1142 "user": {1143 "id": 82067,1144 "username": "TrifeLife",1145 "name": "",1146 "avatar_template": "/letter_avatar_proxy/v4/letter/t/6f9a4e/{size}.png",1147 "trust_level": 01148 }1149 }1150 ]1151 },1152 {1153 "fancy_title": "Best Practices: Using a buffer for the latent space?",1154 "id": 215390,1155 "title": "Best Practices: Using a buffer for the latent space?",1156 "slug": "best-practices-using-a-buffer-for-the-latent-space",1157 "posts_count": 1,1158 "reply_count": 0,1159 "highest_post_number": 1,1160 "image_url": null,1161 "created_at": "2025-01-14T19:34:36.122Z",1162 "last_posted_at": "2025-01-14T19:34:36.184Z",1163 "bumped": true,1164 "bumped_at": "2025-01-14T19:34:36.184Z",1165 "archetype": "regular",1166 "unseen": false,1167 "pinned": false,1168 "unpinned": null,1169 "visible": true,1170 "closed": false,1171 "archived": false,1172 "bookmarked": null,1173 "liked": null,1174 "tags_descriptions": {},1175 "like_count": 0,1176 "views": 24,1177 "category_id": 1,1178 "featured_link": null,1179 "has_accepted_answer": false,1180 "posters": [1181 {1182 "extras": "latest single",1183 "description": "Original Poster, Most Recent Poster",1184 "user": {1185 "id": 82090,1186 "username": "jsully83",1187 "name": "Jonathan",1188 "avatar_template": "/user_avatar/discuss.pytorch.org/jsully83/{size}/75105_2.png",1189 "trust_level": 11190 }1191 }1192 ]1193 },1194 {1195 "fancy_title": "Run two models in parallel using torch.multiprocessing on ONE GPU is slower than running in sequential",1196 "id": 217102,1197 "title": "Run two models in parallel using torch.multiprocessing on ONE GPU is slower than running in sequential",1198 "slug": "run-two-models-in-parallel-using-torch-multiprocessing-on-one-gpu-is-slower-than-running-in-sequential",1199 "posts_count": 1,1200 "reply_count": 0,