Anurag1734/cuda-error-resolution-analysis
07
1[2 {3 "post_stream": {4 "posts": [5 {6 "id": 219528,7 "name": "",8 "username": "cindybrain",9 "avatar_template": "/letter_avatar_proxy/v4/letter/c/9fc348/{size}.png",10 "created_at": "2020-08-10T06:22:35.001Z",11 "cooked": "<p>When I trained resnet18 on ImageNet, I stop it at epoch 30. Then in a later period, i train it again resuming from the pretrained model(epoch 30). However, during the training process, the training loss always turns to nan half way. And I try three times, it always has the same problem(loss nan at different iteration of epoch 30)<br>\nThe training code is the official training example at <a href=\"https://github.com/pytorch/examples/blob/master/imagenet/main.py\" class=\"inline-onebox\" rel=\"noopener nofollow ugc\">examples/imagenet/main.py at main · pytorch/examples · GitHub</a><br>\nI don’t know how the solve the problem. If someone can help me, I will be quite grateful!<br>\n<div class=\"lightbox-wrapper\"><a class=\"lightbox\" href=\"https://discuss.pytorch.org/uploads/default/original/3X/8/1/8139631c126584d21679370cdead5c4c6af5f4ea.png\" data-download-href=\"https://discuss.pytorch.org/uploads/default/8139631c126584d21679370cdead5c4c6af5f4ea\" title=\"loss_nan_pic\"><img src=\"https://discuss.pytorch.org/uploads/default/original/3X/8/1/8139631c126584d21679370cdead5c4c6af5f4ea.png\" alt=\"loss_nan_pic\" data-base62-sha1=\"iraxNiFmaZ7V60ByXROzF7hf1yO\" width=\"690\" height=\"114\" data-dominant-color=\"1D1D1D\"><div class=\"meta\"><svg class=\"fa d-icon d-icon-far-image svg-icon\" aria-hidden=\"true\"><use href=\"#far-image\"></use></svg><span class=\"filename\">loss_nan_pic</span><span class=\"informations\">1431×237 21.6 KB</span><svg class=\"fa d-icon d-icon-discourse-expand svg-icon\" aria-hidden=\"true\"><use href=\"#discourse-expand\"></use></svg></div></a></div></p>",12 "post_number": 1,13 "post_type": 1,14 "posts_count": 9,15 "updated_at": "2020-08-10T06:26:04.063Z",16 "reply_count": 0,17 "reply_to_post_number": null,18 "quote_count": 0,19 "incoming_link_count": 1171,20 "reads": 41,21 "readers_count": 40,22 "score": 5853.2,23 "yours": false,24 "topic_id": 92234,25 "topic_slug": "loss-nan-when-resuming-from-a-pretrained-model",26 "display_username": "",27 "primary_group_name": null,28 "flair_name": null,29 "flair_url": null,30 "flair_bg_color": null,31 "flair_color": null,32 "flair_group_id": null,33 "badges_granted": [],34 "version": 1,35 "can_edit": false,36 "can_delete": false,37 "can_recover": false,38 "can_see_hidden_post": false,39 "can_wiki": false,40 "link_counts": [41 {42 "url": "https://github.com/pytorch/examples/blob/master/imagenet/main.py",43 "internal": false,44 "reflection": false,45 "title": "examples/main.py at master · pytorch/examples · GitHub",46 "clicks": 647 },48 {49 "url": "https://discuss.pytorch.org/uploads/default/original/3X/8/1/8139631c126584d21679370cdead5c4c6af5f4ea.png",50 "internal": true,51 "reflection": false,52 "clicks": 053 }54 ],55 "read": true,56 "user_title": null,57 "bookmarked": false,58 "actions_summary": [],59 "moderator": false,60 "admin": false,61 "staff": false,62 "user_id": 33076,63 "hidden": false,64 "trust_level": 1,65 "deleted_at": null,66 "user_deleted": false,67 "edit_reason": null,68 "can_view_edit_history": true,69 "wiki": false,70 "post_url": "/t/loss-nan-when-resuming-from-a-pretrained-model/92234/1",71 "can_accept_answer": false,72 "can_unaccept_answer": false,73 "accepted_answer": false,74 "topic_accepted_answer": null,75 "can_vote": false76 },77 {78 "id": 219932,79 "name": "",80 "username": "ptrblck",81 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",82 "created_at": "2020-08-11T08:24:09.290Z",83 "cooked": "<p>Could you check the input for invalid values via:</p>\n<pre><code class=\"lang-python\">print(torch.isfinite(input), torch.isfinite(target))\n</code></pre>\n<p>Did you change anything else in the code?</p>",84 "post_number": 2,85 "post_type": 1,86 "posts_count": 9,87 "updated_at": "2020-08-11T08:24:09.290Z",88 "reply_count": 1,89 "reply_to_post_number": null,90 "quote_count": 0,91 "incoming_link_count": 5,92 "reads": 39,93 "readers_count": 38,94 "score": 37.8,95 "yours": false,96 "topic_id": 92234,97 "topic_slug": "loss-nan-when-resuming-from-a-pretrained-model",98 "display_username": "",99 "primary_group_name": null,100 "flair_name": null,101 "flair_url": null,102 "flair_bg_color": null,103 "flair_color": null,104 "flair_group_id": null,105 "badges_granted": [],106 "version": 1,107 "can_edit": false,108 "can_delete": false,109 "can_recover": false,110 "can_see_hidden_post": false,111 "can_wiki": false,112 "read": true,113 "user_title": "",114 "bookmarked": false,115 "actions_summary": [],116 "moderator": true,117 "admin": true,118 "staff": true,119 "user_id": 3534,120 "hidden": false,121 "trust_level": 2,122 "deleted_at": null,123 "user_deleted": false,124 "edit_reason": null,125 "can_view_edit_history": true,126 "wiki": false,127 "post_url": "/t/loss-nan-when-resuming-from-a-pretrained-model/92234/2",128 "can_accept_answer": false,129 "can_unaccept_answer": false,130 "accepted_answer": false,131 "topic_accepted_answer": null132 },133 {134 "id": 220202,135 "name": "",136 "username": "cindybrain",137 "avatar_template": "/letter_avatar_proxy/v4/letter/c/9fc348/{size}.png",138 "created_at": "2020-08-12T04:47:33.928Z",139 "cooked": "<p>The values all show true by using</p>\n<pre><code class=\"lang-auto\">print(torch.isfinite(input), torch.isfinite(target))\n</code></pre>\n<p>I change the code a little only for recording the accuracy and not showing the Data time. I think it doesnot effect the training process.</p>\n<p>And now, when I train it from scratch by using the official code(not change anything this time), I always get nan at iteration 2</p>\n<p><div class=\"lightbox-wrapper\"><a class=\"lightbox\" href=\"https://discuss.pytorch.org/uploads/default/original/3X/c/a/caeb9e453f079e91743415c6d594c4993d75736f.png\" data-download-href=\"https://discuss.pytorch.org/uploads/default/caeb9e453f079e91743415c6d594c4993d75736f\" title=\"error\"><img src=\"https://discuss.pytorch.org/uploads/default/original/3X/c/a/caeb9e453f079e91743415c6d594c4993d75736f.png\" alt=\"error\" data-base62-sha1=\"sX7g2N2k7YlRvzd6xxS9DhtTJev\" width=\"690\" height=\"96\" data-dominant-color=\"121212\"><div class=\"meta\"><svg class=\"fa d-icon d-icon-far-image svg-icon\" aria-hidden=\"true\"><use href=\"#far-image\"></use></svg><span class=\"filename\">error</span><span class=\"informations\">1633×228 16.6 KB</span><svg class=\"fa d-icon d-icon-discourse-expand svg-icon\" aria-hidden=\"true\"><use href=\"#discourse-expand\"></use></svg></div></a></div></p>\n<p>after using this script</p>\n<pre><code class=\"lang-auto\">torch.autograd.set_detect_anomaly(True)\nwith torch.autograd.detect_anomaly():\n\tloss.backward()\n</code></pre>\n<p>I got the error message as follows:</p>\n<pre><code class=\"lang-auto\">(pytorch) shu@hec02:~/GANs/Image Classification/ImageNet$ python main_official.py /nfs/home/Imagenet_ILSVRC2012\n=> creating model 'resnet18'\n\n\n ...,\n\n\n [[[True, True, True, ..., True, True, True],\n [True, True, True, ..., True, True, True],\n [True, True, True, ..., True, True, True],\n ...,\n [True, True, True, ..., True, True, True],\n [True, True, True, ..., True, True, True],\n [True, True, True, ..., True, True, True]],\n\n [[True, True, True, ..., True, True, True],\n [True, True, True, ..., True, True, True],\n [True, True, True, ..., True, True, True],\n ...,\n [True, True, True, ..., True, True, True],\n [True, True, True, ..., True, True, True],\n [True, True, True, ..., True, True, True]],\n\n [[True, True, True, ..., True, True, True],\n [True, True, True, ..., True, True, True],\n [True, True, True, ..., True, True, True],\n ...,\n [True, True, True, ..., True, True, T\n [True, True, True, ..., True, True, True],\n [True, True, True, ..., True, True, True]]]]) tensor([True, True, True, True, True, True, True, True, True, True, True, True,\n True, True, True, True, True, True, True, True, True, True, True, True,\n True, True, True, True, True, True, True, True, True, True, True, True,\n True, True, True, True, True, True, True, True, True, True, True, True,\n True, True, True, True, True, True, True, True, True, True, True, True,\n True, True, True, True, True, True, True, True, True, True, True, True,\n True, True, True, True, True, True, True, True, True, True, True, True,\n True, True, True, True, True, True, True, True, True, True, True, True,\n True, True, True, True, True, True, True, True, True, True, True, True,\n True, True, True, True, True, True, True, True, True, True, True, True,\n True, True, True, True, True, True, True, True, True, True, True, True,\n True, True, True, True, True, True, True, True, True, True, True, True,\n True, True, True, True, True, True, True, True, True, True, True, True,\n True, True, True, True, True, True, True, True, True, True, True, True,\n True, True, True, True, True, True, True, True, True, True, True, True,\n True, True, True, True, True, True, True, True, True, True, True, True,\n True, True, True, True, True, True, True, True, True, True, True, True,\n True, True, True, True, True, True, True, True, True, True, True, True,\n True, True, True, True, True, True, True, True, True, True, True, True,\n True, True, True, True, True, True, True, True, True, True, True, True,\n True, True, True, True, True, True, True, True, True, True, True, True,\n True, True, True, True])\n/nfs/home/shu/.local/lib/python3.6/site-packages/torch/autograd/anomaly_mode.py:70: UserWarning: Anomaly Detection has been enabled. This mode will increase the runtime and should only be enabled for debugging.\n warnings.warn('Anomaly Detection has been enabled. '\nEpoch: [0][ 0/5005] Time 5.869 ( 5.869) Data 4.310 ( 4.310) Loss 7.0919e+00 (7.0919e+00) Acc@1 0.78 ( 0.78) Acc@5 0.78 ( 0.78)\n\n\n ...,\n\n\n [[[True, True, True, ..., True, True, True],\n [True, True, True, ..., True, True, True],\n [True, True, True, ..., True, True, True],\n ...,\n [True, True, True, ..., True, True, True],\n [True, True, True, ..., True, True, True],\n [True, True, True, ..., True, True, True]],\n\n [[True, True, True, ..., True, True, True],\n [True, True, True, ..., True, True, True],\n [True, True, True, ..., True, True, True],\n ...,\n [True, True, True, ..., True, True, True],\n [True, True, True, ..., True, True, True],\n [True, True, True, ..., True, True, True]],\n\n [[True, True, True, ..., True, True, True],\n [True, True, True, ..., True, True, True],\n [True, True, True, ..., True, True, True],\n ...,\n [True, True, True, ..., True, True, T\n [True, True, True, ..., True, True, True],\n [True, True, True, ..., True, True, True]]]]) tensor([True, True, True, True, True, True, True, True, True, True, True, True,\n True, True, True, True, True, True, True, True, True, True, True, True,\n True, True, True, True, True, True, True, True, True, True, True, True,\n True, True, True, True, True, True, True, True, True, True, True, True,\n True, True, True, True, True, True, True, True, True, True, True, True,\n True, True, True, True, True, True, True, True, True, True, True, True,\n True, True, True, True, True, True, True, True, True, True, True, True,\n True, True, True, True, True, True, True, True, True, True, True, True,\n True, True, True, True, True, True, True, True, True, True, True, True,\n True, True, True, True, True, True, True, True, True, True, True, True,\n True, True, True, True, True, True, True, True, True, True, True, True,\n True, True, True, True, True, True, True, True, True, True, True, True,\n True, True, True, True, True, True, True, True, True, True, True, True,\n True, True, True, True, True, True, True, True, True, True, True, True,\n True, True, True, True, True, True, True, True, True, True, True, True,\n True, True, True, True, True, True, True, True, True, True, True, True,\n True, True, True, True, True, True, True, True, True, True, True, True,\n True, True, True, True, True, True, True, True, True, True, True, True,\n True, True, True, True, True, True, True, True, True, True, True, True,\n True, True, True, True, True, True, True, True, True, True, True, True,\n True, True, True, True, True, True, True, True, True, True, True, True,\n True, True, True, True])\nWarning: Error detected in LogSoftmaxBackward. Traceback of forward call that caused the error:\n File \"main_official.py\", line 439, in <module>\n main()\n File \"main_official.py\", line 118, in main\n main_worker(args.gpu, ngpus_per_node, args)\n File \"main_official.py\", line 251, in main_worker\n train(train_loader, model, criterion, optimizer, epoch, args)\n File \"main_official.py\", line 299, in train\n loss = criterion(output, target)\n File \"/nfs/home/shu/.local/lib/python3.6/site-packages/torch/nn/modules/module.py\", line 550, in __call__\n result = self.forward(*input, **kwargs)\n File \"/nfs/home/shu/.local/lib/python3.6/site-packages/torch/nn/modules/loss.py\", line 932, in forward\n ignore_index=self.ignore_index, reduction=self.reduction)\n File \"/nfs/home/shu/.local/lib/python3.6/site-packages/torch/nn/functional.py\", line 2317, in cross_entropy\n return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)\n File \"/nfs/home/shu/.local/lib/python3.6/site-packages/torch/nn/functional.py\", line 1535, in log_softmax\n ret = input.log_softmax(dim)\n (print_stack at /pytorch/torch/csrc/autograd/python_anomaly_mode.cpp:60)\nTraceback (most recent call last):\n File \"main_official.py\", line 439, in <module>\n main()\n File \"main_official.py\", line 118, in main\n main_worker(args.gpu, ngpus_per_node, args)\n File \"main_official.py\", line 251, in main_worker\n train(train_loader, model, criterion, optimizer, epoch, args)\n File \"main_official.py\", line 310, in train\n loss.backward()\n File \"/nfs/home/shu/.local/lib/python3.6/site-packages/torch/tensor.py\", line 198, in backward\n torch.autograd.backward(self, gradient, retain_graph, create_graph)\n File \"/nfs/home/shu/.local/lib/python3.6/site-packages/torch/autograd/__init__.py\", line 100, in backward\n allow_unreachable=True) # allow_unreachable flag\nRuntimeError: Function 'LogSoftmaxBackward' returned nan values in its 0th output.\n</code></pre>\n<p>I do not know what is wrong with it.</p>",140 "post_number": 3,141 "post_type": 1,142 "posts_count": 9,143 "updated_at": "2020-08-12T04:47:33.928Z",144 "reply_count": 1,145 "reply_to_post_number": 2,146 "quote_count": 0,147 "incoming_link_count": 14,148 "reads": 37,149 "readers_count": 36,150 "score": 82.4,151 "yours": false,152 "topic_id": 92234,153 "topic_slug": "loss-nan-when-resuming-from-a-pretrained-model",154 "display_username": "",155 "primary_group_name": null,156 "flair_name": null,157 "flair_url": null,158 "flair_bg_color": null,159 "flair_color": null,160 "flair_group_id": null,161 "badges_granted": [],162 "version": 1,163 "can_edit": false,164 "can_delete": false,165 "can_recover": false,166 "can_see_hidden_post": false,167 "can_wiki": false,168 "link_counts": [169 {170 "url": "https://discuss.pytorch.org/uploads/default/original/3X/c/a/caeb9e453f079e91743415c6d594c4993d75736f.png",171 "internal": true,172 "reflection": false,173 "clicks": 0174 }175 ],176 "read": true,177 "user_title": null,178 "reply_to_user": {179 "id": 3534,180 "username": "ptrblck",181 "name": "",182 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png"183 },184 "bookmarked": false,185 "actions_summary": [],186 "moderator": false,187 "admin": false,188 "staff": false,189 "user_id": 33076,190 "hidden": false,191 "trust_level": 1,192 "deleted_at": null,193 "user_deleted": false,194 "edit_reason": null,195 "can_view_edit_history": true,196 "wiki": false,197 "post_url": "/t/loss-nan-when-resuming-from-a-pretrained-model/92234/3",198 "can_accept_answer": false,199 "can_unaccept_answer": false,200 "accepted_answer": false,201 "topic_accepted_answer": null202 },203 {204 "id": 220214,205 "name": "",206 "username": "ptrblck",207 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",208 "created_at": "2020-08-12T06:34:06.506Z",209 "cooked": "<p>Thanks for the update. Could you add an <code>.all()</code> operation to the check and also check the <code>output</code> tensor as:</p>\n<pre><code class=\"lang-python\">print(torch.isfinite(input).all(), torch.isfinite(target).all(), torch.isfinite(output).all())\n</code></pre>\n<p>It seems that the loss is getting a NaN value, so I guess that the model might output NaNs.</p>",210 "post_number": 4,211 "post_type": 1,212 "posts_count": 9,213 "updated_at": "2020-08-12T06:34:06.506Z",214 "reply_count": 1,215 "reply_to_post_number": 3,216 "quote_count": 0,217 "incoming_link_count": 6,218 "reads": 34,219 "readers_count": 33,220 "score": 41.8,221 "yours": false,222 "topic_id": 92234,223 "topic_slug": "loss-nan-when-resuming-from-a-pretrained-model",224 "display_username": "",225 "primary_group_name": null,226 "flair_name": null,227 "flair_url": null,228 "flair_bg_color": null,229 "flair_color": null,230 "flair_group_id": null,231 "badges_granted": [],232 "version": 1,233 "can_edit": false,234 "can_delete": false,235 "can_recover": false,236 "can_see_hidden_post": false,237 "can_wiki": false,238 "read": true,239 "user_title": "",240 "reply_to_user": {241 "id": 33076,242 "username": "cindybrain",243 "name": "",244 "avatar_template": "/letter_avatar_proxy/v4/letter/c/9fc348/{size}.png"245 },246 "bookmarked": false,247 "actions_summary": [],248 "moderator": true,249 "admin": true,250 "staff": true,251 "user_id": 3534,252 "hidden": false,253 "trust_level": 2,254 "deleted_at": null,255 "user_deleted": false,256 "edit_reason": null,257 "can_view_edit_history": true,258 "wiki": false,259 "post_url": "/t/loss-nan-when-resuming-from-a-pretrained-model/92234/4",260 "can_accept_answer": false,261 "can_unaccept_answer": false,262 "accepted_answer": false,263 "topic_accepted_answer": null264 },265 {266 "id": 220329,267 "name": "Zds Alpha",268 "username": "ZdsAlpha",269 "avatar_template": "/user_avatar/discuss.pytorch.org/zdsalpha/{size}/51282_2.png",270 "created_at": "2020-08-12T14:58:58.484Z",271 "cooked": "<p>Did you make any changes to the model. It might have something to do with final activation layer.</p>",272 "post_number": 5,273 "post_type": 1,274 "posts_count": 9,275 "updated_at": "2020-08-12T14:58:58.484Z",276 "reply_count": 1,277 "reply_to_post_number": null,278 "quote_count": 0,279 "incoming_link_count": 3,280 "reads": 33,281 "readers_count": 32,282 "score": 26.6,283 "yours": false,284 "topic_id": 92234,285 "topic_slug": "loss-nan-when-resuming-from-a-pretrained-model",286 "display_username": "Zds Alpha",287 "primary_group_name": null,288 "flair_name": null,289 "flair_url": null,290 "flair_bg_color": null,291 "flair_color": null,292 "flair_group_id": null,293 "badges_granted": [],294 "version": 1,295 "can_edit": false,296 "can_delete": false,297 "can_recover": false,298 "can_see_hidden_post": false,299 "can_wiki": false,300 "read": true,301 "user_title": null,302 "bookmarked": false,303 "actions_summary": [],304 "moderator": false,305 "admin": false,306 "staff": false,307 "user_id": 29118,308 "hidden": false,309 "trust_level": 2,310 "deleted_at": null,311 "user_deleted": false,312 "edit_reason": null,313 "can_view_edit_history": true,314 "wiki": false,315 "post_url": "/t/loss-nan-when-resuming-from-a-pretrained-model/92234/5",316 "can_accept_answer": false,317 "can_unaccept_answer": false,318 "accepted_answer": false,319 "topic_accepted_answer": null320 },321 {322 "id": 220465,323 "name": "",324 "username": "cindybrain",325 "avatar_template": "/letter_avatar_proxy/v4/letter/c/9fc348/{size}.png",326 "created_at": "2020-08-13T04:52:04.919Z",327 "cooked": "<p>I did not change the model, the model is created in the code below</p>\n<pre><code class=\"lang-auto\">print(\"=> creating model '{}'\".format(args.arch))\n model = models.__dict__[args.arch]()\n</code></pre>\n<p>and because this problem happens sometimes, and sometimes not. Now, it seems to work fine, therefore, I will update the error message if I meet nan again. Thanks very much! <a class=\"mention\" href=\"/u/ptrblck\">@ptrblck</a> <a class=\"mention\" href=\"/u/zdsalpha\">@ZdsAlpha</a> <img src=\"https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9\" title=\":slight_smile:\" class=\"emoji\" alt=\":slight_smile:\"></p>",328 "post_number": 6,329 "post_type": 1,330 "posts_count": 9,331 "updated_at": "2020-08-13T04:52:04.919Z",332 "reply_count": 1,333 "reply_to_post_number": 5,334 "quote_count": 0,335 "incoming_link_count": 5,336 "reads": 31,337 "readers_count": 30,338 "score": 36.2,339 "yours": false,340 "topic_id": 92234,341 "topic_slug": "loss-nan-when-resuming-from-a-pretrained-model",342 "display_username": "",343 "primary_group_name": null,344 "flair_name": null,345 "flair_url": null,346 "flair_bg_color": null,347 "flair_color": null,348 "flair_group_id": null,349 "badges_granted": [],350 "version": 1,351 "can_edit": false,352 "can_delete": false,353 "can_recover": false,354 "can_see_hidden_post": false,355 "can_wiki": false,356 "read": true,357 "user_title": null,358 "reply_to_user": {359 "id": 29118,360 "username": "ZdsAlpha",361 "name": "Zds Alpha",362 "avatar_template": "/user_avatar/discuss.pytorch.org/zdsalpha/{size}/51282_2.png"363 },364 "bookmarked": false,365 "actions_summary": [],366 "moderator": false,367 "admin": false,368 "staff": false,369 "user_id": 33076,370 "hidden": false,371 "trust_level": 1,372 "deleted_at": null,373 "user_deleted": false,374 "edit_reason": null,375 "can_view_edit_history": true,376 "wiki": false,377 "post_url": "/t/loss-nan-when-resuming-from-a-pretrained-model/92234/6",378 "can_accept_answer": false,379 "can_unaccept_answer": false,380 "accepted_answer": false,381 "topic_accepted_answer": null382 },383 {384 "id": 220470,385 "name": "",386 "username": "cindybrain",387 "avatar_template": "/letter_avatar_proxy/v4/letter/c/9fc348/{size}.png",388 "created_at": "2020-08-13T05:25:03.450Z",389 "cooked": "<pre><code class=\"lang-auto\">Epoch: [1][2120/5005] Loss 3.5405e+00 (3.6588e+00) Acc@1 25.00 ( 25.53) Acc@5 51.56 ( 49.25)\ntensor(True) tensor(True, device='cuda:0') tensor(True, device='cuda:0')\ntensor(True) tensor(True, device='cuda:0') tensor(True, device='cuda:0')\ntensor(True) tensor(True, device='cuda:0') tensor(True, device='cuda:0')\ntensor(True) tensor(True, device='cuda:0') tensor(True, device='cuda:0')\ntensor(True) tensor(True, device='cuda:0') tensor(True, device='cuda:0')\ntensor(True) tensor(True, device='cuda:0') tensor(True, device='cuda:0')\ntensor(True) tensor(True, device='cuda:0') tensor(True, device='cuda:0')\ntensor(True) tensor(True, device='cuda:0') tensor(True, device='cuda:0')\ntensor(True) tensor(True, device='cuda:0') tensor(True, device='cuda:0')\ntensor(True) tensor(True, device='cuda:0') tensor(True, device='cuda:0')\nEpoch: [1][2130/5005] Loss 3.7294e+00 (3.6584e+00) Acc@1 23.05 ( 25.54) Acc@5 50.39 ( 49.26)\ntensor(True) tensor(True, device='cuda:0') tensor(True, device='cuda:0')\ntensor(True) tensor(True, device='cuda:0') tensor(True, device='cuda:0')\ntensor(True) tensor(True, device='cuda:0') tensor(True, device='cuda:0')\ntensor(True) tensor(True, device='cuda:0') tensor(True, device='cuda:0')\ntensor(True) tensor(True, device='cuda:0') tensor(True, device='cuda:0')\ntensor(True) tensor(True, device='cuda:0') tensor(True, device='cuda:0')\ntensor(True) tensor(True, device='cuda:0') tensor(False, device='cuda:0')\ntensor(True) tensor(True, device='cuda:0') tensor(False, device='cuda:0')\ntensor(True) tensor(True, device='cuda:0') tensor(False, device='cuda:0')\ntensor(True) tensor(True, device='cuda:0') tensor(False, device='cuda:0')\nEpoch: [1][2140/5005] Loss nan (nan) Acc@1 0.00 ( 25.49) Acc@5 1.17 ( 49.18)\ntensor(True) tensor(True, device='cuda:0') tensor(False, device='cuda:0')\ntensor(True) tensor(True, device='cuda:0') tensor(False, device='cuda:0')\ntensor(True) tensor(True, device='cuda:0') tensor(False, device='cuda:0')\ntensor(True) tensor(True, device='cuda:0') tensor(False, device='cuda:0')\ntensor(True) tensor(True, device='cuda:0') tensor(False, device='cuda:0')\ntensor(True) tensor(True, device='cuda:0') tensor(False, device='cuda:0')\ntensor(True) tensor(True, device='cuda:0') tensor(False, device='cuda:0')\ntensor(True) tensor(True, device='cuda:0') tensor(False, device='cuda:0')\ntensor(True) tensor(True, device='cuda:0') tensor(False, device='cuda:0')\ntensor(True) tensor(True, device='cuda:0') tensor(False, device='cuda:0')\nEpoch: [1][2150/5005] Loss nan (nan) Acc@1 0.00 ( 25.38) Acc@5 0.78 ( 48.96)\ntensor(True) tensor(True, device='cuda:0') tensor(False, device='cuda:0')\ntensor(True) tensor(True, device='cuda:0') tensor(False, device='cuda:0')\ntensor(True) tensor(True, device='cuda:0') tensor(False, device='cuda:0')\ntensor(True) tensor(True, device='cuda:0') tensor(False, device='cuda:0')\ntensor(True) tensor(True, device='cuda:0') tensor(False, device='cuda:0')\ntensor(True) tensor(True, device='cuda:0') tensor(False, device='cuda:0')\ntensor(True) tensor(True, device='cuda:0') tensor(False, device='cuda:0')\ntensor(True) tensor(True, device='cuda:0') tensor(False, device='cuda:0')\ntensor(True) tensor(True, device='cuda:0') tensor(False, device='cuda:0')\ntensor(True) tensor(True, device='cuda:0') tensor(False, device='cuda:0')\nEpoch: [1][2160/5005] Loss nan (nan) Acc@1 0.00 ( 25.26) Acc@5 0.78 ( 48.73)\ntensor(True) tensor(True, device='cuda:0') tensor(False, device='cuda:0')\ntensor(True) tensor(True, device='cuda:0') tensor(False, device='cuda:0')\ntensor(True) tensor(True, device='cuda:0') tensor(False, device='cuda:0')\ntensor(True) tensor(True, device='cuda:0') tensor(False, device='cuda:0')\ntensor(True) tensor(True, device='cuda:0') tensor(False, device='cuda:0')\ntensor(True) tensor(True, device='cuda:0') tensor(False, device='cuda:0')\ntensor(True) tensor(True, device='cuda:0') tensor(False, device='cuda:0')\ntensor(True) tensor(True, device='cuda:0') tensor(False, device='cuda:0')\ntensor(True) tensor(True, device='cuda:0') tensor(False, device='cuda:0')\ntensor(True) tensor(True, device='cuda:0') tensor(False, device='cuda:0')\n</code></pre>\n<p>I got this message when add this code.</p>",390 "post_number": 7,391 "post_type": 1,392 "posts_count": 9,393 "updated_at": "2020-08-13T05:25:03.450Z",394 "reply_count": 1,395 "reply_to_post_number": 4,396 "quote_count": 0,397 "incoming_link_count": 13,398 "reads": 30,399 "readers_count": 29,400 "score": 76.0,401 "yours": false,402 "topic_id": 92234,403 "topic_slug": "loss-nan-when-resuming-from-a-pretrained-model",404 "display_username": "",405 "primary_group_name": null,406 "flair_name": null,407 "flair_url": null,408 "flair_bg_color": null,409 "flair_color": null,410 "flair_group_id": null,411 "badges_granted": [],412 "version": 1,413 "can_edit": false,414 "can_delete": false,415 "can_recover": false,416 "can_see_hidden_post": false,417 "can_wiki": false,418 "read": true,419 "user_title": null,420 "reply_to_user": {421 "id": 3534,422 "username": "ptrblck",423 "name": "",424 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png"425 },426 "bookmarked": false,427 "actions_summary": [],428 "moderator": false,429 "admin": false,430 "staff": false,431 "user_id": 33076,432 "hidden": false,433 "trust_level": 1,434 "deleted_at": null,435 "user_deleted": false,436 "edit_reason": null,437 "can_view_edit_history": true,438 "wiki": false,439 "post_url": "/t/loss-nan-when-resuming-from-a-pretrained-model/92234/7",440 "can_accept_answer": false,441 "can_unaccept_answer": false,442 "accepted_answer": false,443 "topic_accepted_answer": null444 },445 {446 "id": 220494,447 "name": "",448 "username": "ptrblck",449 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",450 "created_at": "2020-08-13T07:50:47.338Z",451 "cooked": "<p>For some reason the output is getting NaN values.<br>\nCould you <code>break</code> the training loop, after you’ve encountered the first NaN and check all parameters of the model?<br>\nE.g. you could print their <code>abs().max()</code> via:</p>\n<pre><code class=\"lang-python\">for name, param in model.named_parameters():\n print(name, param.abs().max())\n</code></pre>\n<p>If this looks alright, you could repeat the last forward iteration (since the input contains valid values) and check all intermediate activations to narrow down, which layer creates the NaN outputs using forward hooks as described <a href=\"https://discuss.pytorch.org/t/how-can-l-load-my-best-model-as-a-feature-extractor-evaluator/17254/6\">here</a>.</p>",452 "post_number": 8,453 "post_type": 1,454 "posts_count": 9,455 "updated_at": "2020-08-13T07:50:47.338Z",456 "reply_count": 0,457 "reply_to_post_number": 7,458 "quote_count": 0,459 "incoming_link_count": 16,460 "reads": 28,461 "readers_count": 27,462 "score": 85.6,463 "yours": false,464 "topic_id": 92234,465 "topic_slug": "loss-nan-when-resuming-from-a-pretrained-model",466 "display_username": "",467 "primary_group_name": null,468 "flair_name": null,469 "flair_url": null,470 "flair_bg_color": null,471 "flair_color": null,472 "flair_group_id": null,473 "badges_granted": [],474 "version": 1,475 "can_edit": false,476 "can_delete": false,477 "can_recover": false,478 "can_see_hidden_post": false,479 "can_wiki": false,480 "link_counts": [481 {482 "url": "https://discuss.pytorch.org/t/how-can-l-load-my-best-model-as-a-feature-extractor-evaluator/17254/6",483 "internal": true,484 "reflection": false,485 "title": "How can l load my best model as a feature extractor/evaluator?",486 "clicks": 45487 }488 ],489 "read": true,490 "user_title": "",491 "reply_to_user": {492 "id": 33076,493 "username": "cindybrain",494 "name": "",495 "avatar_template": "/letter_avatar_proxy/v4/letter/c/9fc348/{size}.png"496 },497 "bookmarked": false,498 "actions_summary": [],499 "moderator": true,500 "admin": true,501 "staff": true,502 "user_id": 3534,503 "hidden": false,504 "trust_level": 2,505 "deleted_at": null,506 "user_deleted": false,507 "edit_reason": null,508 "can_view_edit_history": true,509 "wiki": false,510 "post_url": "/t/loss-nan-when-resuming-from-a-pretrained-model/92234/8",511 "can_accept_answer": false,512 "can_unaccept_answer": false,513 "accepted_answer": false,514 "topic_accepted_answer": null515 },516 {517 "id": 437482,518 "name": "Chandrashekar Konda",519 "username": "chandrak1907",520 "avatar_template": "/letter_avatar_proxy/v4/letter/c/b9e5f3/{size}.png",521 "created_at": "2024-03-28T04:07:15.716Z",522 "cooked": "<p>Hi <a class=\"mention\" href=\"/u/cindybrain\">@cindybrain</a> , Could you please advise if you found a solution to this? I’m also encountering a similar situation, where model runs fine on some occasions and gives nan on other occasions.</p>",523 "post_number": 9,524 "post_type": 1,525 "posts_count": 9,526 "updated_at": "2024-03-28T04:07:15.716Z",527 "reply_count": 0,528 "reply_to_post_number": 6,529 "quote_count": 0,530 "incoming_link_count": 4,531 "reads": 6,532 "readers_count": 5,533 "score": 21.2,534 "yours": false,535 "topic_id": 92234,536 "topic_slug": "loss-nan-when-resuming-from-a-pretrained-model",537 "display_username": "Chandrashekar Konda",538 "primary_group_name": null,539 "flair_name": null,540 "flair_url": null,541 "flair_bg_color": null,542 "flair_color": null,543 "flair_group_id": null,544 "badges_granted": [],545 "version": 1,546 "can_edit": false,547 "can_delete": false,548 "can_recover": false,549 "can_see_hidden_post": false,550 "can_wiki": false,551 "read": true,552 "user_title": null,553 "reply_to_user": {554 "id": 33076,555 "username": "cindybrain",556 "name": "",557 "avatar_template": "/letter_avatar_proxy/v4/letter/c/9fc348/{size}.png"558 },559 "bookmarked": false,560 "actions_summary": [],561 "moderator": false,562 "admin": false,563 "staff": false,564 "user_id": 10183,565 "hidden": false,566 "trust_level": 1,567 "deleted_at": null,568 "user_deleted": false,569 "edit_reason": null,570 "can_view_edit_history": true,571 "wiki": false,572 "post_url": "/t/loss-nan-when-resuming-from-a-pretrained-model/92234/9",573 "can_accept_answer": false,574 "can_unaccept_answer": false,575 "accepted_answer": false,576 "topic_accepted_answer": null577 }578 ],579 "stream": [580 219528,581 219932,582 220202,583 220214,584 220329,585 220465,586 220470,587 220494,588 437482589 ]590 },591 "timeline_lookup": [592 [593 1,594 1903595 ],596 [597 2,598 1901599 ],600 [601 4,602 1900603 ],604 [605 8,606 1899607 ],608 [609 9,610 577611 ]612 ],613 "suggested_topics": [614 {615 "fancy_title": "Faster Vit Hierarchical Attention",616 "id": 214037,617 "title": "Faster Vit Hierarchical Attention",618 "slug": "faster-vit-hierarchical-attention",619 "posts_count": 1,620 "reply_count": 0,621 "highest_post_number": 1,622 "image_url": null,623 "created_at": "2024-12-10T08:36:29.626Z",624 "last_posted_at": "2024-12-10T08:36:29.684Z",625 "bumped": true,626 "bumped_at": "2024-12-10T08:36:29.684Z",627 "archetype": "regular",628 "unseen": false,629 "pinned": false,630 "unpinned": null,631 "visible": true,632 "closed": false,633 "archived": false,634 "bookmarked": null,635 "liked": null,636 "tags_descriptions": {},637 "like_count": 0,638 "views": 127,639 "category_id": 5,640 "featured_link": null,641 "has_accepted_answer": false,642 "posters": [643 {644 "extras": "latest single",645 "description": "Original Poster, Most Recent Poster",646 "user": {647 "id": 81424,648 "username": "hussainmir05",649 "name": "Hussain mir",650 "avatar_template": "/user_avatar/discuss.pytorch.org/hussainmir05/{size}/72721_2.png",651 "trust_level": 0652 }653 }654 ]655 },656 {657 "fancy_title": "Exporting pth file to onnx with torch",658 "id": 212289,659 "title": "Exporting pth file to onnx with torch",660 "slug": "exporting-pth-file-to-onnx-with-torch",661 "posts_count": 3,662 "reply_count": 0,663 "highest_post_number": 3,664 "image_url": null,665 "created_at": "2024-10-29T17:44:55.606Z",666 "last_posted_at": "2025-04-10T07:02:27.139Z",667 "bumped": true,668 "bumped_at": "2025-04-10T07:02:27.139Z",669 "archetype": "regular",670 "unseen": false,671 "pinned": false,672 "unpinned": null,673 "visible": true,674 "closed": false,675 "archived": false,676 "bookmarked": null,677 "liked": null,678 "tags_descriptions": {},679 "like_count": 1,680 "views": 256,681 "category_id": 5,682 "featured_link": null,683 "has_accepted_answer": false,684 "posters": [685 {686 "extras": null,687 "description": "Original Poster",688 "user": {689 "id": 62480,690 "username": "PedroC",691 "name": "Pedro Carvalho",692 "avatar_template": "/letter_avatar_proxy/v4/letter/p/d6d6ee/{size}.png",693 "trust_level": 1694 }695 },696 {697 "extras": "latest",698 "description": "Most Recent Poster",699 "user": {700 "id": 83738,701 "username": "Divya932",702 "name": "",703 "avatar_template": "/letter_avatar_proxy/v4/letter/d/a5b964/{size}.png",704 "trust_level": 0705 }706 }707 ]708 },709 {710 "fancy_title": "Using TransformerEncoderLayer for image classification",711 "id": 215758,712 "title": "Using TransformerEncoderLayer for image classification",713 "slug": "using-transformerencoderlayer-for-image-classification",714 "posts_count": 1,715 "reply_count": 0,716 "highest_post_number": 1,717 "image_url": null,718 "created_at": "2025-01-23T10:50:41.594Z",719 "last_posted_at": "2025-01-23T10:50:41.655Z",720 "bumped": true,721 "bumped_at": "2025-01-23T11:01:38.629Z",722 "archetype": "regular",723 "unseen": false,724 "pinned": false,725 "unpinned": null,726 "visible": true,727 "closed": false,728 "archived": false,729 "bookmarked": null,730 "liked": null,731 "tags_descriptions": {},732 "like_count": 0,733 "views": 40,734 "category_id": 5,735 "featured_link": null,736 "has_accepted_answer": false,737 "posters": [738 {739 "extras": "latest single",740 "description": "Original Poster, Most Recent Poster",741 "user": {742 "id": 68222,743 "username": "amy2",744 "name": "amy",745 "avatar_template": "/user_avatar/discuss.pytorch.org/amy2/{size}/62675_2.png",746 "trust_level": 1747 }748 }749 ]750 },751 {752 "fancy_title": "Multiclass semantic segmentation giving zero in all output channels except for the first channel",753 "id": 215788,754 "title": "Multiclass semantic segmentation giving zero in all output channels except for the first channel",755 "slug": "multiclass-semantic-segmentation-giving-zero-in-all-output-channels-except-for-the-first-channel",756 "posts_count": 6,757 "reply_count": 4,758 "highest_post_number": 6,759 "image_url": null,760 "created_at": "2025-01-23T20:54:22.497Z",761 "last_posted_at": "2025-01-23T21:05:06.364Z",762 "bumped": true,763 "bumped_at": "2025-01-23T21:05:06.364Z",764 "archetype": "regular",765 "unseen": false,766 "pinned": false,767 "unpinned": null,768 "visible": true,769 "closed": false,770 "archived": false,771 "bookmarked": null,772 "liked": null,773 "tags_descriptions": {},774 "like_count": 0,775 "views": 49,776 "category_id": 5,777 "featured_link": null,778 "has_accepted_answer": false,779 "posters": [780 {781 "extras": null,782 "description": "Original Poster",783 "user": {784 "id": 82180,785 "username": "Amit_Sur",786 "name": "Amit Sur",787 "avatar_template": "/user_avatar/discuss.pytorch.org/amit_sur/{size}/75195_2.png",788 "trust_level": 1789 }790 },791 {792 "extras": "latest",793 "description": "Most Recent Poster",794 "user": {795 "id": 3534,796 "username": "ptrblck",797 "name": "",798 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",799 "admin": true,800 "moderator": true,801 "trust_level": 2802 }803 }804 ]805 },806 {807 "fancy_title": "Behavior of Autocontrast on Normalized Images and Default Value for `contrast_factor`",808 "id": 218967,809 "title": "Behavior of Autocontrast on Normalized Images and Default Value for `contrast_factor`",810 "slug": "behavior-of-autocontrast-on-normalized-images-and-default-value-for-contrast-factor",811 "posts_count": 1,812 "reply_count": 0,813 "highest_post_number": 1,814 "image_url": null,815 "created_at": "2025-04-11T10:07:41.312Z",816 "last_posted_at": "2025-04-11T10:07:41.358Z",817 "bumped": true,818 "bumped_at": "2025-04-11T10:07:41.358Z",819 "archetype": "regular",820 "unseen": false,821 "pinned": false,822 "unpinned": null,823 "visible": true,824 "closed": false,825 "archived": false,826 "bookmarked": null,827 "liked": null,828 "tags_descriptions": {},829 "like_count": 0,830 "views": 22,831 "category_id": 5,832 "featured_link": null,833 "has_accepted_answer": false,834 "posters": [835 {836 "extras": "latest single",837 "description": "Original Poster, Most Recent Poster",838 "user": {839 "id": 71430,840 "username": "Fatemeh_A",841 "name": "Fatemeh",842 "avatar_template": "/user_avatar/discuss.pytorch.org/fatemeh_a/{size}/65970_2.png",843 "trust_level": 1844 }845 }846 ]847 }848 ],849 "tags_descriptions": {},850 "fancy_title": "Loss nan when resuming from a pretrained model",851 "id": 92234,852 "title": "Loss nan when resuming from a pretrained model",853 "posts_count": 9,854 "created_at": "2020-08-10T06:22:34.949Z",855 "views": 2979,856 "reply_count": 6,857 "like_count": 0,858 "last_posted_at": "2024-03-28T04:07:15.716Z",859 "visible": true,860 "closed": false,861 "archived": false,862 "has_summary": false,863 "archetype": "regular",864 "slug": "loss-nan-when-resuming-from-a-pretrained-model",865 "category_id": 5,866 "word_count": 2299,867 "deleted_at": null,868 "user_id": 33076,869 "featured_link": null,870 "pinned_globally": false,871 "pinned_at": null,872 "pinned_until": null,873 "image_url": "https://discuss.pytorch.org/uploads/default/optimized/3X/8/1/8139631c126584d21679370cdead5c4c6af5f4ea_2_1024x169.png",874 "slow_mode_seconds": 0,875 "draft": null,876 "draft_key": "topic_92234",877 "draft_sequence": null,878 "unpinned": null,879 "pinned": false,880 "current_post_number": 1,881 "highest_post_number": 9,882 "deleted_by": null,883 "actions_summary": [884 {885 "id": 4,886 "count": 0,887 "hidden": false,888 "can_act": false889 },890 {891 "id": 8,892 "count": 0,893 "hidden": false,894 "can_act": false895 },896 {897 "id": 10,898 "count": 0,899 "hidden": false,900 "can_act": false901 },902 {903 "id": 7,904 "count": 0,905 "hidden": false,906 "can_act": false907 }908 ],909 "chunk_size": 20,910 "bookmarked": false,911 "topic_timer": null,912 "message_bus_last_id": 0,913 "participant_count": 4,914 "show_read_indicator": false,915 "thumbnails": [916 {917 "max_width": null,918 "max_height": null,919 "width": 1431,920 "height": 237,921 "url": "https://discuss.pytorch.org/uploads/default/original/3X/8/1/8139631c126584d21679370cdead5c4c6af5f4ea.png"922 },923 {924 "max_width": 1024,925 "max_height": 1024,926 "width": 1024,927 "height": 169,928 "url": "https://discuss.pytorch.org/uploads/default/optimized/3X/8/1/8139631c126584d21679370cdead5c4c6af5f4ea_2_1024x169.png"929 }930 ],931 "slow_mode_enabled_until": null,932 "can_vote": false,933 "vote_count": 0,934 "user_voted": false,935 "discourse_zendesk_plugin_zendesk_id": null,936 "discourse_zendesk_plugin_zendesk_url": "https://your-url.zendesk.com/agent/tickets/",937 "details": {938 "can_edit": false,939 "notification_level": 1,940 "participants": [941 {942 "id": 33076,943 "username": "cindybrain",944 "name": "",945 "avatar_template": "/letter_avatar_proxy/v4/letter/c/9fc348/{size}.png",946 "post_count": 4,947 "primary_group_name": null,948 "flair_name": null,949 "flair_url": null,950 "flair_color": null,951 "flair_bg_color": null,952 "flair_group_id": null,953 "trust_level": 1954 },955 {956 "id": 3534,957 "username": "ptrblck",958 "name": "",959 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",960 "post_count": 3,961 "primary_group_name": null,962 "flair_name": null,963 "flair_url": null,964 "flair_color": null,965 "flair_bg_color": null,966 "flair_group_id": null,967 "admin": true,968 "moderator": true,969 "trust_level": 2970 },971 {972 "id": 10183,973 "username": "chandrak1907",974 "name": "Chandrashekar Konda",975 "avatar_template": "/letter_avatar_proxy/v4/letter/c/b9e5f3/{size}.png",976 "post_count": 1,977 "primary_group_name": null,978 "flair_name": null,979 "flair_url": null,980 "flair_color": null,981 "flair_bg_color": null,982 "flair_group_id": null,983 "trust_level": 1984 },985 {986 "id": 29118,987 "username": "ZdsAlpha",988 "name": "Zds Alpha",989 "avatar_template": "/user_avatar/discuss.pytorch.org/zdsalpha/{size}/51282_2.png",990 "post_count": 1,991 "primary_group_name": null,992 "flair_name": null,993 "flair_url": null,994 "flair_color": null,995 "flair_bg_color": null,996 "flair_group_id": null,997 "trust_level": 2998 }999 ],1000 "created_by": {1001 "id": 33076,1002 "username": "cindybrain",1003 "name": "",1004 "avatar_template": "/letter_avatar_proxy/v4/letter/c/9fc348/{size}.png"1005 },1006 "last_poster": {1007 "id": 10183,1008 "username": "chandrak1907",1009 "name": "Chandrashekar Konda",1010 "avatar_template": "/letter_avatar_proxy/v4/letter/c/b9e5f3/{size}.png"1011 },1012 "links": [1013 {1014 "url": "https://discuss.pytorch.org/t/how-can-l-load-my-best-model-as-a-feature-extractor-evaluator/17254/6",1015 "title": "How can l load my best model as a feature extractor/evaluator?",1016 "internal": true,1017 "attachment": false,1018 "reflection": false,1019 "clicks": 45,1020 "user_id": 3534,1021 "domain": "discuss.pytorch.org",1022 "root_domain": "pytorch.org"1023 },1024 {1025 "url": "https://github.com/pytorch/examples/blob/master/imagenet/main.py",1026 "title": "examples/main.py at master · pytorch/examples · GitHub",1027 "internal": false,1028 "attachment": false,1029 "reflection": false,1030 "clicks": 6,1031 "user_id": 33076,1032 "domain": "github.com",1033 "root_domain": "github.com"1034 }1035 ]1036 },1037 "bookmarks": []1038 },1039 {1040 "post_stream": {1041 "posts": [1042 {1043 "id": 437361,1044 "name": "chandler-bing",1045 "username": "chandler-bing",1046 "avatar_template": "/user_avatar/discuss.pytorch.org/chandler-bing/{size}/68785_2.png",1047 "created_at": "2024-03-27T09:20:01.976Z",1048 "cooked": "<p>Hi, everyone, I encountered a very simple but weird BUG!</p>\n<pre><code class=\"lang-auto\">import torch\n\nm = torch.load(f'./gate_linear.pt',map_location='cpu')\nprint(m)\n#Linear(in_features=1024, out_features=2816, bias=False)\n\nx1 = torch.load(f'./x1.pt',map_location='cpu')\nx2 = torch.load(f'./x2.pt',map_location='cpu')\nprint(x1.size(),x2.size())\n#torch.Size([1, 128, 1024]) torch.Size([1, 129, 1024])\n\nprint(torch.equal(x1,x2[:,:-1,:]))\n#True\n\no1 = m(x1)\no2 = m(x2)\n\nprint(torch.equal(\n o1,\n o2[:,:-1,:]\n ))\n\n#False\n\n</code></pre>\n<p>the code is simple enough, as the<code> True</code> result of<code> torch.equal</code> suggest, input <code>x1, x2[:,:-1,:]</code> should be same.<br>\nwhy it become different after a simple Linear model?<br>\nI upload my pt files (about 4M size) to reproduce the result, <a href=\"https://drive.google.com/file/d/1E1ShH3j6QrgS8wRRqopz8pLQST-0qZ_T/view?usp=sharing\" rel=\"noopener nofollow ugc\">google drive</a><br>\nany thoughts? thank you guys</p>",1049 "post_number": 1,1050 "post_type": 1,1051 "posts_count": 5,1052 "updated_at": "2024-03-27T09:20:01.976Z",1053 "reply_count": 0,1054 "reply_to_post_number": null,1055 "quote_count": 0,1056 "incoming_link_count": 26,1057 "reads": 8,1058 "readers_count": 7,1059 "score": 131.6,1060 "yours": false,1061 "topic_id": 199684,1062 "topic_slug": "torch-nn-linear-bug",1063 "display_username": "chandler-bing",1064 "primary_group_name": null,1065 "flair_name": null,1066 "flair_url": null,1067 "flair_bg_color": null,1068 "flair_color": null,1069 "flair_group_id": null,1070 "badges_granted": [],1071 "version": 1,1072 "can_edit": false,1073 "can_delete": false,1074 "can_recover": false,1075 "can_see_hidden_post": false,1076 "can_wiki": false,1077 "link_counts": [1078 {1079 "url": "https://drive.google.com/file/d/1E1ShH3j6QrgS8wRRqopz8pLQST-0qZ_T/view?usp=sharing",1080 "internal": false,1081 "reflection": false,1082 "title": "tmp1.zip - Google Drive",1083 "clicks": 11084 }1085 ],1086 "read": true,1087 "user_title": null,1088 "bookmarked": false,1089 "actions_summary": [],1090 "moderator": false,1091 "admin": false,1092 "staff": false,1093 "user_id": 74480,1094 "hidden": false,1095 "trust_level": 1,1096 "deleted_at": null,1097 "user_deleted": false,1098 "edit_reason": null,1099 "can_view_edit_history": true,1100 "wiki": false,1101 "post_url": "/t/torch-nn-linear-bug/199684/1",1102 "can_accept_answer": false,1103 "can_unaccept_answer": false,1104 "accepted_answer": false,1105 "topic_accepted_answer": true,1106 "can_vote": false1107 },1108 {1109 "id": 437362,1110 "name": "chandler-bing",1111 "username": "chandler-bing",1112 "avatar_template": "/user_avatar/discuss.pytorch.org/chandler-bing/{size}/68785_2.png",1113 "created_at": "2024-03-27T09:25:56.182Z",1114 "cooked": "<p><a class=\"mention\" href=\"/u/ptrblck\">@ptrblck</a> could u help me with this? thank u <img src=\"https://discuss.pytorch.org/images/emoji/apple/pray.png?v=12\" title=\":pray:\" class=\"emoji\" alt=\":pray:\" loading=\"lazy\" width=\"20\" height=\"20\"></p>",1115 "post_number": 2,1116 "post_type": 1,1117 "posts_count": 5,1118 "updated_at": "2024-03-27T09:45:01.507Z",1119 "reply_count": 0,1120 "reply_to_post_number": null,1121 "quote_count": 0,1122 "incoming_link_count": 0,1123 "reads": 8,1124 "readers_count": 7,1125 "score": 1.6,1126 "yours": false,1127 "topic_id": 199684,1128 "topic_slug": "torch-nn-linear-bug",1129 "display_username": "chandler-bing",1130 "primary_group_name": null,1131 "flair_name": null,1132 "flair_url": null,1133 "flair_bg_color": null,1134 "flair_color": null,1135 "flair_group_id": null,1136 "badges_granted": [],1137 "version": 2,1138 "can_edit": false,1139 "can_delete": false,1140 "can_recover": false,1141 "can_see_hidden_post": false,1142 "can_wiki": false,1143 "read": true,1144 "user_title": null,1145 "bookmarked": false,1146 "actions_summary": [],1147 "moderator": false,1148 "admin": false,1149 "staff": false,1150 "user_id": 74480,1151 "hidden": false,1152 "trust_level": 1,1153 "deleted_at": null,1154 "user_deleted": false,1155 "edit_reason": null,1156 "can_view_edit_history": true,1157 "wiki": false,1158 "post_url": "/t/torch-nn-linear-bug/199684/2",1159 "can_accept_answer": false,1160 "can_unaccept_answer": false,1161 "accepted_answer": false,1162 "topic_accepted_answer": true1163 },1164 {1165 "id": 437369,1166 "name": "chandler-bing",1167 "username": "chandler-bing",1168 "avatar_template": "/user_avatar/discuss.pytorch.org/chandler-bing/{size}/68785_2.png",1169 "created_at": "2024-03-27T10:34:44.871Z",1170 "cooked": "<p>minimal reproducible code …</p>\n<pre><code class=\"lang-auto\">import torch\ntorch.set_printoptions(precision=8)\nx = torch.tensor([[[ 0.0451, -0.8093],\n [-0.3275, -0.5304]]]\n )\nx1 = x[:,:-1,:]\nx2 = x\nw = torch.tensor( [[ 0.29, 0.05],\n [-0.11, -0.29],\n [ 0.05, 0.29],\n [-0.03, -0.05]], requires_grad=True)\nw = w.transpose(0,1)\n\nprint(x.size(),x.dtype)\nprint(w.size(),w.dtype)\n#torch.Size([1, 2, 2]) torch.float32\n#torch.Size([2, 4]) torch.float32\n\no1 = torch.matmul(x1,w)\no2 = torch.matmul(x2,w)\n\nprint(o1)\nprint(o2)\n# tensor([[[-0.02738600, 0.22973600, -0.23244201, 0.03911200],\n# [-0.12149499, 0.18984099, -0.17019099, 0.03634500]]],\n# grad_fn=<UnsafeViewBackward0>)\nprint(torch.equal(o1,o2[:,:-1,:]))\n#False\n</code></pre>\n<p>seems like a loss of precision?</p>",1171 "post_number": 3,1172 "post_type": 1,1173 "posts_count": 5,1174 "updated_at": "2024-03-27T10:34:44.871Z",1175 "reply_count": 1,1176 "reply_to_post_number": null,1177 "quote_count": 0,1178 "incoming_link_count": 1,1179 "reads": 7,1180 "readers_count": 6,1181 "score": 11.4,1182 "yours": false,1183 "topic_id": 199684,1184 "topic_slug": "torch-nn-linear-bug",1185 "display_username": "chandler-bing",1186 "primary_group_name": null,1187 "flair_name": null,1188 "flair_url": null,1189 "flair_bg_color": null,1190 "flair_color": null,1191 "flair_group_id": null,1192 "badges_granted": [],1193 "version": 1,1194 "can_edit": false,1195 "can_delete": false,1196 "can_recover": false,1197 "can_see_hidden_post": false,1198 "can_wiki": false,1199 "read": true,1200 "user_title": null,