Anurag1734/cuda-error-resolution-analysis
07
1[2 {3 "post_stream": {4 "posts": [5 {6 "id": 138107,7 "name": "Ray",8 "username": "ruian1",9 "avatar_template": "/letter_avatar_proxy/v4/letter/r/2bfe46/{size}.png",10 "created_at": "2019-10-03T19:36:30.815Z",11 "cooked": "<p>Hi Pytorch Community! I was using BCEWithLogintsLoss to train a multi-label network and getting negative loss as shown below. As mentioned in the class documentation, this loss function combines sigmoid and BCELoss…But actually as it shows. I also checked its definition in pytorch/torch/nn/functional.py and I was not able to find the sigmoid operation in this loss. Maybe I used it wrong ?</p>\n<p>I’m using version 1.0.1</p>\n<pre><code class=\"lang-auto\">criterion = nn.BCEWithLogitsLoss()\na = torch.tensor([[1., 1., 1., 0., 0.]])\nb = torch.tensor([[0., 0.0011122, 8.9638, 0., 0.]])\nc = nn.Sigmoid()(b)\nprint(criterion(a,b))\nprint(criterion(a,c))\n\n\"tensor(-0.7278)\"\n\"tensor(0.6652)\"\n</code></pre>\n<p>Should line 598 of pytorch/torch/nn/moduels/loss.py be changed as below ?</p>\n<pre><code class=\"lang-auto\"> return F.binary_cross_entropy_with_logits(nn.Sigmoid()(input), target,</code></pre>",12 "post_number": 1,13 "post_type": 1,14 "posts_count": 5,15 "updated_at": "2019-10-03T19:38:47.788Z",16 "reply_count": 1,17 "reply_to_post_number": null,18 "quote_count": 0,19 "incoming_link_count": 7149,20 "reads": 190,21 "readers_count": 189,22 "score": 35793.0,23 "yours": false,24 "topic_id": 57409,25 "topic_slug": "v1-0-1-nn-bcewithlogitsloss-returns-negative-loss-sigmoid-layer-not-deployed",26 "display_username": "Ray",27 "primary_group_name": null,28 "flair_name": null,29 "flair_url": null,30 "flair_bg_color": null,31 "flair_color": null,32 "flair_group_id": null,33 "badges_granted": [],34 "version": 1,35 "can_edit": false,36 "can_delete": false,37 "can_recover": false,38 "can_see_hidden_post": false,39 "can_wiki": false,40 "read": true,41 "user_title": "",42 "bookmarked": false,43 "actions_summary": [44 {45 "id": 2,46 "count": 147 }48 ],49 "moderator": false,50 "admin": false,51 "staff": false,52 "user_id": 22816,53 "hidden": false,54 "trust_level": 1,55 "deleted_at": null,56 "user_deleted": false,57 "edit_reason": null,58 "can_view_edit_history": true,59 "wiki": false,60 "post_url": "/t/v1-0-1-nn-bcewithlogitsloss-returns-negative-loss-sigmoid-layer-not-deployed/57409/1",61 "can_accept_answer": false,62 "can_unaccept_answer": false,63 "accepted_answer": false,64 "topic_accepted_answer": true,65 "can_vote": false66 },67 {68 "id": 138115,69 "name": "iurii zdebskyi",70 "username": "izdeby",71 "avatar_template": "/letter_avatar_proxy/v4/letter/i/c4cdca/{size}.png",72 "created_at": "2019-10-03T19:52:04.000Z",73 "cooked": "<p>This might be related to this <a href=\"https://github.com/pytorch/pytorch/issues/24933\" rel=\"nofollow noopener\">issue</a>. I will take a look into it</p>",74 "post_number": 2,75 "post_type": 1,76 "posts_count": 5,77 "updated_at": "2019-10-03T19:52:04.000Z",78 "reply_count": 0,79 "reply_to_post_number": null,80 "quote_count": 0,81 "incoming_link_count": 47,82 "reads": 182,83 "readers_count": 181,84 "score": 271.4,85 "yours": false,86 "topic_id": 57409,87 "topic_slug": "v1-0-1-nn-bcewithlogitsloss-returns-negative-loss-sigmoid-layer-not-deployed",88 "display_username": "iurii zdebskyi",89 "primary_group_name": null,90 "flair_name": null,91 "flair_url": null,92 "flair_bg_color": null,93 "flair_color": null,94 "flair_group_id": null,95 "badges_granted": [],96 "version": 1,97 "can_edit": false,98 "can_delete": false,99 "can_recover": false,100 "can_see_hidden_post": false,101 "can_wiki": false,102 "link_counts": [103 {104 "url": "https://github.com/pytorch/pytorch/issues/24933",105 "internal": false,106 "reflection": false,107 "title": "BCELossWithLogits(input) != BCELoss(Sigmoid(input)) · Issue #24933 · pytorch/pytorch · GitHub",108 "clicks": 645109 }110 ],111 "read": true,112 "user_title": null,113 "bookmarked": false,114 "actions_summary": [],115 "moderator": false,116 "admin": false,117 "staff": false,118 "user_id": 23054,119 "hidden": false,120 "trust_level": 1,121 "deleted_at": null,122 "user_deleted": false,123 "edit_reason": null,124 "can_view_edit_history": true,125 "wiki": false,126 "post_url": "/t/v1-0-1-nn-bcewithlogitsloss-returns-negative-loss-sigmoid-layer-not-deployed/57409/2",127 "can_accept_answer": false,128 "can_unaccept_answer": false,129 "accepted_answer": false,130 "topic_accepted_answer": true131 },132 {133 "id": 138125,134 "name": "K. Frank",135 "username": "KFrank",136 "avatar_template": "/letter_avatar_proxy/v4/letter/k/ecb155/{size}.png",137 "created_at": "2019-10-03T22:50:06.698Z",138 "cooked": "<p>Hello Rui An!</p>\n<aside class=\"quote no-group quote-modified\" data-username=\"ruian1\" data-post=\"1\" data-topic=\"57409\" data-full=\"true\">\n<div class=\"title\">\n<div class=\"quote-controls\"></div>\n<img loading=\"lazy\" alt=\"\" width=\"24\" height=\"24\" src=\"https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/2bfe46/48.png\" class=\"avatar\"> ruian1:</div>\n<blockquote>\n<p>I was using BCEWithLogintsLoss to train a multi-label network and getting negative loss as shown below.</p>\n<p>…</p>\n<pre><code class=\"lang-auto\">criterion = nn.BCEWithLogitsLoss()\na = torch.tensor([[1., 1., 1., 0., 0.]])\nb = torch.tensor([[0., 0.0011122, 8.9638, 0., 0.]])\nc = nn.Sigmoid()(b)\nprint(criterion(a,b))\nprint(criterion(a,c))\n\n\"tensor(-0.7278)\"\n\"tensor(0.6652)\"\n</code></pre>\n</blockquote>\n</aside>\n<p>It appears that you have switched the order of your inputs to<br>\n<code>BCEWithLogitsLoss</code>.</p>\n<p><code>BCEWithLogitsLoss</code> (like<br>\n<code>binary_cross_entropy_with_logits()</code>) expects to be<br>\ncalled with predictions that are logits (-infinity to infinity) and<br>\ntargets that are probabilities (0 to 1), in that order.</p>\n<p>Your <code>a</code> are legitimate probabilities, so they are your targets, and<br>\nyour <code>b</code> are legitimate logits, so they are your predictions. Your<br>\ncall should therefore be:</p>\n<pre data-code-wrap=\"python\"><code class=\"lang-python\">print(criterion(b,a))\n</code></pre>\n<p>(In your version of the call your second argument, <code>b</code>, is out of<br>\nrange (not 0 to 1), so the call returns the invalid negative result.<br>\n<code>c</code>, however, being the result of <code>Sigmoid</code>, is in (0, 1), so the<br>\nresult of the call is indeed positive.)</p>\n<aside class=\"quote no-group\">\n<blockquote>\n<p>Should line 598 of pytorch/torch/nn/moduels/loss.py be changed as below ?</p>\n<pre><code class=\"lang-auto\"> return F.binary_cross_entropy_with_logits(nn.Sigmoid()(input), target,\n</code></pre>\n</blockquote>\n</aside>\n<p>No. <code>binary_cross_entropy_with_logits()</code> has built<br>\ninto it (implicitly, in effect) the sigmoid function (just like<br>\n<code>BCEWithLogitsLoss</code>), so you want to pass logits as the first<br>\nargument (<code>input</code>), not probabilities (<code>nn.Sigmoid()(input)</code>).</p>\n<p>You shouldn’t expect to see an explicit call to <code>Sigmoid</code> in the<br>\nsource code for <code>BCEWithLogitsLoss</code>. In effect <code>Sigmoid</code> is<br>\napplied to <code>input</code>, but it’s implicit (using the “log-sum-exp trick”).</p>\n<p>Good luck.</p>\n<p>K. Frank</p>",139 "post_number": 3,140 "post_type": 1,141 "posts_count": 5,142 "updated_at": "2019-10-03T22:51:26.881Z",143 "reply_count": 1,144 "reply_to_post_number": null,145 "quote_count": 1,146 "incoming_link_count": 240,147 "reads": 175,148 "readers_count": 174,149 "score": 1255.0,150 "yours": false,151 "topic_id": 57409,152 "topic_slug": "v1-0-1-nn-bcewithlogitsloss-returns-negative-loss-sigmoid-layer-not-deployed",153 "display_username": "K. Frank",154 "primary_group_name": null,155 "flair_name": null,156 "flair_url": null,157 "flair_bg_color": null,158 "flair_color": null,159 "flair_group_id": null,160 "badges_granted": [],161 "version": 1,162 "can_edit": false,163 "can_delete": false,164 "can_recover": false,165 "can_see_hidden_post": false,166 "can_wiki": false,167 "read": true,168 "user_title": null,169 "bookmarked": false,170 "actions_summary": [171 {172 "id": 2,173 "count": 1174 }175 ],176 "moderator": false,177 "admin": false,178 "staff": false,179 "user_id": 18088,180 "hidden": false,181 "trust_level": 2,182 "deleted_at": null,183 "user_deleted": false,184 "edit_reason": null,185 "can_view_edit_history": true,186 "wiki": false,187 "post_url": "/t/v1-0-1-nn-bcewithlogitsloss-returns-negative-loss-sigmoid-layer-not-deployed/57409/3",188 "can_accept_answer": false,189 "can_unaccept_answer": false,190 "accepted_answer": true,191 "topic_accepted_answer": true192 },193 {194 "id": 138126,195 "name": "Ray",196 "username": "ruian1",197 "avatar_template": "/letter_avatar_proxy/v4/letter/r/2bfe46/{size}.png",198 "created_at": "2019-10-03T22:52:46.654Z",199 "cooked": "<p>Hello KFrank, thanks a lot for a very detailed explanation! Could you also point me to the block where “log-sum-exp” is applied to ? I traced it back to functional.py but still can’t figure out how it is done…</p>",200 "post_number": 4,201 "post_type": 1,202 "posts_count": 5,203 "updated_at": "2019-10-03T23:32:47.964Z",204 "reply_count": 1,205 "reply_to_post_number": 3,206 "quote_count": 0,207 "incoming_link_count": 11,208 "reads": 140,209 "readers_count": 139,210 "score": 88.0,211 "yours": false,212 "topic_id": 57409,213 "topic_slug": "v1-0-1-nn-bcewithlogitsloss-returns-negative-loss-sigmoid-layer-not-deployed",214 "display_username": "Ray",215 "primary_group_name": null,216 "flair_name": null,217 "flair_url": null,218 "flair_bg_color": null,219 "flair_color": null,220 "flair_group_id": null,221 "badges_granted": [],222 "version": 2,223 "can_edit": false,224 "can_delete": false,225 "can_recover": false,226 "can_see_hidden_post": false,227 "can_wiki": false,228 "read": true,229 "user_title": "",230 "reply_to_user": {231 "id": 18088,232 "username": "KFrank",233 "name": "K. Frank",234 "avatar_template": "/letter_avatar_proxy/v4/letter/k/ecb155/{size}.png"235 },236 "bookmarked": false,237 "actions_summary": [],238 "moderator": false,239 "admin": false,240 "staff": false,241 "user_id": 22816,242 "hidden": false,243 "trust_level": 1,244 "deleted_at": null,245 "user_deleted": false,246 "edit_reason": null,247 "can_view_edit_history": true,248 "wiki": false,249 "post_url": "/t/v1-0-1-nn-bcewithlogitsloss-returns-negative-loss-sigmoid-layer-not-deployed/57409/4",250 "can_accept_answer": false,251 "can_unaccept_answer": false,252 "accepted_answer": false,253 "topic_accepted_answer": true254 },255 {256 "id": 138136,257 "name": "K. Frank",258 "username": "KFrank",259 "avatar_template": "/letter_avatar_proxy/v4/letter/k/ecb155/{size}.png",260 "created_at": "2019-10-04T00:37:07.056Z",261 "cooked": "<p>Hi Rui An!</p>\n<aside class=\"quote no-group\" data-username=\"ruian1\" data-post=\"4\" data-topic=\"57409\" data-full=\"true\">\n<div class=\"title\">\n<div class=\"quote-controls\"></div>\n<img loading=\"lazy\" alt=\"\" width=\"24\" height=\"24\" src=\"https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/2bfe46/48.png\" class=\"avatar\"> ruian1:</div>\n<blockquote>\n<p>Could you also point me to the block where “log-sum-exp” is applied to ? I traced it back to functional.py but still can’t figure out how it is done…</p>\n</blockquote>\n</aside>\n<p>I don’t have the <code>binary_cross_entropy_with_logits</code> code<br>\nin front of me, so I can’t give you the specifics.</p>\n<p>The issue is that floating-point error can get amplified when you<br>\ncompute <code>log</code> of an expression containing <code>exp</code>. <code>sigmoid</code> has<br>\nthe <code>exp</code>, and cross-entropy has the <code>log</code>, so you can run into<br>\nthis problem when using <code>sigmoid</code> as input to cross-entropy.<br>\nDealing with this issue is the main reason that<br>\n<code>binary_cross_entropy_with_logits</code> exists.</p>\n<p>See, for example, the comments about “log1p” in the Wikipedia<br>\narticle about <a href=\"https://en.wikipedia.org/wiki/Natural_logarithm\" rel=\"noopener nofollow ugc\">logarithm</a>.</p>\n<p>(I was speaking loosely when I mentioned the related<br>\n“log-sum-exp-trick.” This is more directly relevant to computing<br>\n<code>softmax</code>, but is basically another facet of the same issue. For<br>\nmore on this, see, for example, Wikipedia’s <a href=\"https://en.wikipedia.org/wiki/LogSumExp\" rel=\"noopener nofollow ugc\">LogSumExp</a>)</p>\n<p>Best.</p>\n<p>K. Frank</p>",262 "post_number": 5,263 "post_type": 1,264 "posts_count": 5,265 "updated_at": "2019-10-04T00:37:07.056Z",266 "reply_count": 0,267 "reply_to_post_number": 4,268 "quote_count": 1,269 "incoming_link_count": 80,270 "reads": 132,271 "readers_count": 131,272 "score": 426.4,273 "yours": false,274 "topic_id": 57409,275 "topic_slug": "v1-0-1-nn-bcewithlogitsloss-returns-negative-loss-sigmoid-layer-not-deployed",276 "display_username": "K. Frank",277 "primary_group_name": null,278 "flair_name": null,279 "flair_url": null,280 "flair_bg_color": null,281 "flair_color": null,282 "flair_group_id": null,283 "badges_granted": [],284 "version": 1,285 "can_edit": false,286 "can_delete": false,287 "can_recover": false,288 "can_see_hidden_post": false,289 "can_wiki": false,290 "link_counts": [291 {292 "url": "https://en.wikipedia.org/wiki/LogSumExp",293 "internal": false,294 "reflection": false,295 "title": "LogSumExp - Wikipedia",296 "clicks": 17297 },298 {299 "url": "https://en.wikipedia.org/wiki/Natural_logarithm",300 "internal": false,301 "reflection": false,302 "title": "Natural logarithm - Wikipedia",303 "clicks": 5304 }305 ],306 "read": true,307 "user_title": null,308 "bookmarked": false,309 "actions_summary": [],310 "moderator": false,311 "admin": false,312 "staff": false,313 "user_id": 18088,314 "hidden": false,315 "trust_level": 2,316 "deleted_at": null,317 "user_deleted": false,318 "edit_reason": null,319 "can_view_edit_history": true,320 "wiki": false,321 "post_url": "/t/v1-0-1-nn-bcewithlogitsloss-returns-negative-loss-sigmoid-layer-not-deployed/57409/5",322 "can_accept_answer": false,323 "can_unaccept_answer": false,324 "accepted_answer": false,325 "topic_accepted_answer": true326 }327 ],328 "stream": [329 138107,330 138115,331 138125,332 138126,333 138136334 ]335 },336 "timeline_lookup": [337 [338 1,339 2214340 ]341 ],342 "suggested_topics": [343 {344 "fancy_title": "Parallelize tensor dot-products (different sizes)",345 "id": 212367,346 "title": "Parallelize tensor dot-products (different sizes)",347 "slug": "parallelize-tensor-dot-products-different-sizes",348 "posts_count": 1,349 "reply_count": 0,350 "highest_post_number": 1,351 "image_url": null,352 "created_at": "2024-10-31T13:55:07.835Z",353 "last_posted_at": "2024-10-31T13:55:07.889Z",354 "bumped": true,355 "bumped_at": "2024-10-31T13:55:07.889Z",356 "archetype": "regular",357 "unseen": false,358 "pinned": false,359 "unpinned": null,360 "visible": true,361 "closed": false,362 "archived": false,363 "bookmarked": null,364 "liked": null,365 "tags_descriptions": {},366 "like_count": 0,367 "views": 122,368 "category_id": 1,369 "featured_link": null,370 "has_accepted_answer": false,371 "posters": [372 {373 "extras": "latest single",374 "description": "Original Poster, Most Recent Poster",375 "user": {376 "id": 1879,377 "username": "Zen",378 "name": null,379 "avatar_template": "/letter_avatar_proxy/v4/letter/z/7c8e57/{size}.png",380 "trust_level": 1381 }382 }383 ]384 },385 {386 "fancy_title": "Torch.nn.functional.embedding with output argument?",387 "id": 212632,388 "title": "Torch.nn.functional.embedding with output argument?",389 "slug": "torch-nn-functional-embedding-with-output-argument",390 "posts_count": 1,391 "reply_count": 0,392 "highest_post_number": 1,393 "image_url": null,394 "created_at": "2024-11-06T20:34:34.616Z",395 "last_posted_at": "2024-11-06T20:34:34.723Z",396 "bumped": true,397 "bumped_at": "2024-11-06T20:34:34.723Z",398 "archetype": "regular",399 "unseen": false,400 "pinned": false,401 "unpinned": null,402 "visible": true,403 "closed": false,404 "archived": false,405 "bookmarked": null,406 "liked": null,407 "tags_descriptions": {},408 "like_count": 0,409 "views": 59,410 "category_id": 1,411 "featured_link": null,412 "has_accepted_answer": false,413 "posters": [414 {415 "extras": "latest single",416 "description": "Original Poster, Most Recent Poster",417 "user": {418 "id": 9970,419 "username": "youkaichao1",420 "name": "",421 "avatar_template": "/letter_avatar_proxy/v4/letter/y/f0a364/{size}.png",422 "trust_level": 2423 }424 }425 ]426 },427 {428 "fancy_title": "PyTorch Profiler self cpu time vs cpu time",429 "id": 213646,430 "title": "PyTorch Profiler self cpu time vs cpu time",431 "slug": "pytorch-profiler-self-cpu-time-vs-cpu-time",432 "posts_count": 2,433 "reply_count": 0,434 "highest_post_number": 3,435 "image_url": null,436 "created_at": "2024-11-30T22:10:06.683Z",437 "last_posted_at": "2024-12-03T20:32:58.081Z",438 "bumped": true,439 "bumped_at": "2024-12-03T20:32:58.081Z",440 "archetype": "regular",441 "unseen": false,442 "pinned": false,443 "unpinned": null,444 "visible": true,445 "closed": false,446 "archived": false,447 "bookmarked": null,448 "liked": null,449 "tags_descriptions": {},450 "like_count": 1,451 "views": 193,452 "category_id": 1,453 "featured_link": null,454 "has_accepted_answer": false,455 "posters": [456 {457 "extras": null,458 "description": "Original Poster",459 "user": {460 "id": 72990,461 "username": "m7md_hka",462 "name": "Mohammed Abbadi",463 "avatar_template": "/user_avatar/discuss.pytorch.org/m7md_hka/{size}/67399_2.png",464 "trust_level": 1465 }466 },467 {468 "extras": "latest",469 "description": "Most Recent Poster",470 "user": {471 "id": 3534,472 "username": "ptrblck",473 "name": "",474 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",475 "admin": true,476 "moderator": true,477 "trust_level": 2478 }479 }480 ]481 },482 {483 "fancy_title": "The model trained in PyTorch produces inconsistent predictions for the same image when processed individually versus in a batch.",484 "id": 214051,485 "title": "The model trained in PyTorch produces inconsistent predictions for the same image when processed individually versus in a batch.",486 "slug": "the-model-trained-in-pytorch-produces-inconsistent-predictions-for-the-same-image-when-processed-individually-versus-in-a-batch",487 "posts_count": 5,488 "reply_count": 3,489 "highest_post_number": 5,490 "image_url": null,491 "created_at": "2024-12-10T13:15:22.322Z",492 "last_posted_at": "2024-12-17T08:22:05.979Z",493 "bumped": true,494 "bumped_at": "2024-12-17T08:22:05.979Z",495 "archetype": "regular",496 "unseen": false,497 "pinned": false,498 "unpinned": null,499 "visible": true,500 "closed": false,501 "archived": false,502 "bookmarked": null,503 "liked": null,504 "tags_descriptions": {},505 "like_count": 0,506 "views": 75,507 "category_id": 1,508 "featured_link": null,509 "has_accepted_answer": false,510 "posters": [511 {512 "extras": "latest",513 "description": "Original Poster, Most Recent Poster",514 "user": {515 "id": 81426,516 "username": "Dishant_Dua",517 "name": "Dishant Dua",518 "avatar_template": "/user_avatar/discuss.pytorch.org/dishant_dua/{size}/74450_2.png",519 "trust_level": 0520 }521 },522 {523 "extras": null,524 "description": "Frequent Poster",525 "user": {526 "id": 3534,527 "username": "ptrblck",528 "name": "",529 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",530 "admin": true,531 "moderator": true,532 "trust_level": 2533 }534 }535 ]536 },537 {538 "fancy_title": "Changing the backbone",539 "id": 212284,540 "title": "Changing the backbone",541 "slug": "changing-the-backbone",542 "posts_count": 3,543 "reply_count": 0,544 "highest_post_number": 3,545 "image_url": null,546 "created_at": "2024-10-29T15:27:33.798Z",547 "last_posted_at": "2024-11-08T05:37:31.700Z",548 "bumped": true,549 "bumped_at": "2024-11-08T05:37:31.700Z",550 "archetype": "regular",551 "unseen": false,552 "pinned": false,553 "unpinned": null,554 "visible": true,555 "closed": false,556 "archived": false,557 "bookmarked": null,558 "liked": null,559 "tags_descriptions": {},560 "like_count": 0,561 "views": 97,562 "category_id": 1,563 "featured_link": null,564 "has_accepted_answer": false,565 "posters": [566 {567 "extras": "latest",568 "description": "Original Poster, Most Recent Poster",569 "user": {570 "id": 76804,571 "username": "afnhs-gs",572 "name": "semba",573 "avatar_template": "/letter_avatar_proxy/v4/letter/a/c5a1d2/{size}.png",574 "trust_level": 1575 }576 },577 {578 "extras": null,579 "description": "Frequent Poster",580 "user": {581 "id": 3534,582 "username": "ptrblck",583 "name": "",584 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",585 "admin": true,586 "moderator": true,587 "trust_level": 2588 }589 }590 ]591 }592 ],593 "tags_descriptions": {},594 "fancy_title": "V1.0.1, nn.BCEWithLogitsLoss returns negative loss, Sigmoid layer not deployed",595 "id": 57409,596 "title": "V1.0.1, nn.BCEWithLogitsLoss returns negative loss, Sigmoid layer not deployed",597 "posts_count": 5,598 "created_at": "2019-10-03T19:36:30.762Z",599 "views": 8264,600 "reply_count": 2,601 "like_count": 2,602 "last_posted_at": "2019-10-04T00:37:07.056Z",603 "visible": true,604 "closed": false,605 "archived": false,606 "has_summary": false,607 "archetype": "regular",608 "slug": "v1-0-1-nn-bcewithlogitsloss-returns-negative-loss-sigmoid-layer-not-deployed",609 "category_id": 1,610 "word_count": 659,611 "deleted_at": null,612 "user_id": 22816,613 "featured_link": null,614 "pinned_globally": false,615 "pinned_at": null,616 "pinned_until": null,617 "image_url": null,618 "slow_mode_seconds": 0,619 "draft": null,620 "draft_key": "topic_57409",621 "draft_sequence": null,622 "unpinned": null,623 "pinned": false,624 "current_post_number": 1,625 "highest_post_number": 5,626 "deleted_by": null,627 "actions_summary": [628 {629 "id": 4,630 "count": 0,631 "hidden": false,632 "can_act": false633 },634 {635 "id": 8,636 "count": 0,637 "hidden": false,638 "can_act": false639 },640 {641 "id": 10,642 "count": 0,643 "hidden": false,644 "can_act": false645 },646 {647 "id": 7,648 "count": 0,649 "hidden": false,650 "can_act": false651 }652 ],653 "chunk_size": 20,654 "bookmarked": false,655 "topic_timer": null,656 "message_bus_last_id": 0,657 "participant_count": 3,658 "show_read_indicator": false,659 "thumbnails": null,660 "slow_mode_enabled_until": null,661 "accepted_answer": {662 "post_number": 3,663 "username": "KFrank",664 "name": "K. Frank",665 "excerpt": "Hello Rui An! \n\nIt appears that you have switched the order of your inputs to \nBCEWithLogitsLoss. \nBCEWithLogitsLoss (like \nbinary_cross_entropy_with_logits()) expects to be \ncalled with predictions that are logits (-infinity to infinity) and \ntargets that are probabilities (0 to 1), in that order. \n…"666 },667 "can_vote": false,668 "vote_count": 0,669 "user_voted": false,670 "discourse_zendesk_plugin_zendesk_id": null,671 "discourse_zendesk_plugin_zendesk_url": "https://your-url.zendesk.com/agent/tickets/",672 "details": {673 "can_edit": false,674 "notification_level": 1,675 "participants": [676 {677 "id": 18088,678 "username": "KFrank",679 "name": "K. Frank",680 "avatar_template": "/letter_avatar_proxy/v4/letter/k/ecb155/{size}.png",681 "post_count": 2,682 "primary_group_name": null,683 "flair_name": null,684 "flair_url": null,685 "flair_color": null,686 "flair_bg_color": null,687 "flair_group_id": null,688 "trust_level": 2689 },690 {691 "id": 22816,692 "username": "ruian1",693 "name": "Ray",694 "avatar_template": "/letter_avatar_proxy/v4/letter/r/2bfe46/{size}.png",695 "post_count": 2,696 "primary_group_name": null,697 "flair_name": null,698 "flair_url": null,699 "flair_color": null,700 "flair_bg_color": null,701 "flair_group_id": null,702 "trust_level": 1703 },704 {705 "id": 23054,706 "username": "izdeby",707 "name": "iurii zdebskyi",708 "avatar_template": "/letter_avatar_proxy/v4/letter/i/c4cdca/{size}.png",709 "post_count": 1,710 "primary_group_name": null,711 "flair_name": null,712 "flair_url": null,713 "flair_color": null,714 "flair_bg_color": null,715 "flair_group_id": null,716 "trust_level": 1717 }718 ],719 "created_by": {720 "id": 22816,721 "username": "ruian1",722 "name": "Ray",723 "avatar_template": "/letter_avatar_proxy/v4/letter/r/2bfe46/{size}.png"724 },725 "last_poster": {726 "id": 18088,727 "username": "KFrank",728 "name": "K. Frank",729 "avatar_template": "/letter_avatar_proxy/v4/letter/k/ecb155/{size}.png"730 },731 "links": [732 {733 "url": "https://github.com/pytorch/pytorch/issues/24933",734 "title": "BCELossWithLogits(input) != BCELoss(Sigmoid(input)) · Issue #24933 · pytorch/pytorch · GitHub",735 "internal": false,736 "attachment": false,737 "reflection": false,738 "clicks": 645,739 "user_id": 23054,740 "domain": "github.com",741 "root_domain": "github.com"742 },743 {744 "url": "https://en.wikipedia.org/wiki/LogSumExp",745 "title": "LogSumExp - Wikipedia",746 "internal": false,747 "attachment": false,748 "reflection": false,749 "clicks": 17,750 "user_id": 18088,751 "domain": "en.wikipedia.org",752 "root_domain": "wikipedia.org"753 },754 {755 "url": "https://en.wikipedia.org/wiki/Natural_logarithm",756 "title": "Natural logarithm - Wikipedia",757 "internal": false,758 "attachment": false,759 "reflection": false,760 "clicks": 5,761 "user_id": 18088,762 "domain": "en.wikipedia.org",763 "root_domain": "wikipedia.org"764 }765 ]766 },767 "bookmarks": []768 },769 {770 "post_stream": {771 "posts": [772 {773 "id": 138110,774 "name": "Fiyero P",775 "username": "Fiyero_P",776 "avatar_template": "/user_avatar/discuss.pytorch.org/fiyero_p/{size}/12184_2.png",777 "created_at": "2019-10-03T19:46:24.110Z",778 "cooked": "<p>Hi<br>\nI have a 1080ti and now thinking should I get a 2080ti<br>\nIf I apply DataParallel to train the model in Pytorch with 1080ti and 2080ti, will 1080ti become the bottleneck to the training process?</p>\n<p>Sorry I am not familiarize with the DataParallel process.<br>\nLets say the whole dataset can be divided into 10 batches, will each GPU be assigned with <strong>fixed number of batches</strong> (1080ti and 2080ti each will train 5 batches) ?</p>\n<p>or if one GPU trains faster it will take more batches?</p>\n<p>Thanks</p>",779 "post_number": 1,780 "post_type": 1,781 "posts_count": 3,782 "updated_at": "2019-10-03T19:46:24.110Z",783 "reply_count": 0,784 "reply_to_post_number": null,785 "quote_count": 0,786 "incoming_link_count": 29,787 "reads": 12,788 "readers_count": 11,789 "score": 147.4,790 "yours": false,791 "topic_id": 57410,792 "topic_slug": "question-about-multi-gpu-training",793 "display_username": "Fiyero P",794 "primary_group_name": null,795 "flair_name": null,796 "flair_url": null,797 "flair_bg_color": null,798 "flair_color": null,799 "flair_group_id": null,800 "badges_granted": [],801 "version": 1,802 "can_edit": false,803 "can_delete": false,804 "can_recover": false,805 "can_see_hidden_post": false,806 "can_wiki": false,807 "read": true,808 "user_title": null,809 "bookmarked": false,810 "actions_summary": [],811 "moderator": false,812 "admin": false,813 "staff": false,814 "user_id": 23053,815 "hidden": false,816 "trust_level": 1,817 "deleted_at": null,818 "user_deleted": false,819 "edit_reason": null,820 "can_view_edit_history": true,821 "wiki": false,822 "post_url": "/t/question-about-multi-gpu-training/57410/1",823 "can_accept_answer": false,824 "can_unaccept_answer": false,825 "accepted_answer": false,826 "topic_accepted_answer": null,827 "can_vote": false828 },829 {830 "id": 138128,831 "name": "",832 "username": "ptrblck",833 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",834 "created_at": "2019-10-03T22:57:58.031Z",835 "cooked": "<p>If you are using <code>nn.DataParallel</code>, each GPU will get the same batch size (if possible).<br>\nIf you look at the <a href=\"https://github.com/pytorch/pytorch/blob/74572fc985ace7f024a885218641dbfe0b0dd021/torch/nn/parallel/data_parallel.py#L168\" rel=\"nofollow noopener\">internals</a> you might adapt the code and use a custom chunking approach.</p>\n<p>However, I would generally advice against a system with mixed GPUs.<br>\nE.g. your 2080TI will have TensorCores, which could speedup FP16 calculations, while your 1080Ti won’t be able to do it.</p>",836 "post_number": 2,837 "post_type": 1,838 "posts_count": 3,839 "updated_at": "2019-10-03T22:57:58.031Z",840 "reply_count": 1,841 "reply_to_post_number": null,842 "quote_count": 0,843 "incoming_link_count": 0,844 "reads": 11,845 "readers_count": 10,846 "score": 7.2,847 "yours": false,848 "topic_id": 57410,849 "topic_slug": "question-about-multi-gpu-training",850 "display_username": "",851 "primary_group_name": null,852 "flair_name": null,853 "flair_url": null,854 "flair_bg_color": null,855 "flair_color": null,856 "flair_group_id": null,857 "badges_granted": [],858 "version": 1,859 "can_edit": false,860 "can_delete": false,861 "can_recover": false,862 "can_see_hidden_post": false,863 "can_wiki": false,864 "link_counts": [865 {866 "url": "https://github.com/pytorch/pytorch/blob/74572fc985ace7f024a885218641dbfe0b0dd021/torch/nn/parallel/data_parallel.py#L168",867 "internal": false,868 "reflection": false,869 "title": "pytorch/data_parallel.py at 74572fc985ace7f024a885218641dbfe0b0dd021 · pytorch/pytorch · GitHub",870 "clicks": 0871 }872 ],873 "read": true,874 "user_title": "",875 "bookmarked": false,876 "actions_summary": [],877 "moderator": true,878 "admin": true,879 "staff": true,880 "user_id": 3534,881 "hidden": false,882 "trust_level": 2,883 "deleted_at": null,884 "user_deleted": false,885 "edit_reason": null,886 "can_view_edit_history": true,887 "wiki": false,888 "post_url": "/t/question-about-multi-gpu-training/57410/2",889 "can_accept_answer": false,890 "can_unaccept_answer": false,891 "accepted_answer": false,892 "topic_accepted_answer": null893 },894 {895 "id": 138131,896 "name": "Fiyero P",897 "username": "Fiyero_P",898 "avatar_template": "/user_avatar/discuss.pytorch.org/fiyero_p/{size}/12184_2.png",899 "created_at": "2019-10-03T23:18:51.736Z",900 "cooked": "<p>thanks so much for your reply.<br>\nJust curious for the performance’s point of view, would you replace 1080ti by 2080ti ? or<br>\ntrain model with dual 1080ti gpu?</p>",901 "post_number": 3,902 "post_type": 1,903 "posts_count": 3,904 "updated_at": "2019-10-03T23:20:17.585Z",905 "reply_count": 0,906 "reply_to_post_number": 2,907 "quote_count": 0,908 "incoming_link_count": 0,909 "reads": 11,910 "readers_count": 10,911 "score": 2.2,912 "yours": false,913 "topic_id": 57410,914 "topic_slug": "question-about-multi-gpu-training",915 "display_username": "Fiyero P",916 "primary_group_name": null,917 "flair_name": null,918 "flair_url": null,919 "flair_bg_color": null,920 "flair_color": null,921 "flair_group_id": null,922 "badges_granted": [],923 "version": 1,924 "can_edit": false,925 "can_delete": false,926 "can_recover": false,927 "can_see_hidden_post": false,928 "can_wiki": false,929 "read": true,930 "user_title": null,931 "reply_to_user": {932 "id": 3534,933 "username": "ptrblck",934 "name": "",935 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png"936 },937 "bookmarked": false,938 "actions_summary": [],939 "moderator": false,940 "admin": false,941 "staff": false,942 "user_id": 23053,943 "hidden": false,944 "trust_level": 1,945 "deleted_at": null,946 "user_deleted": false,947 "edit_reason": null,948 "can_view_edit_history": true,949 "wiki": false,950 "post_url": "/t/question-about-multi-gpu-training/57410/3",951 "can_accept_answer": false,952 "can_unaccept_answer": false,953 "accepted_answer": false,954 "topic_accepted_answer": null955 }956 ],957 "stream": [958 138110,959 138128,960 138131961 ]962 },963 "timeline_lookup": [964 [965 1,966 2214967 ]968 ],969 "suggested_topics": [970 {971 "fancy_title": "Help with increasing CNN frame interpolation model accuracy",972 "id": 216197,973 "title": "Help with increasing CNN frame interpolation model accuracy",974 "slug": "help-with-increasing-cnn-frame-interpolation-model-accuracy",975 "posts_count": 1,976 "reply_count": 0,977 "highest_post_number": 1,978 "image_url": "https://discuss.pytorch.org/uploads/default/optimized/3X/d/2/d2917c8eeaf12a4f1e470d6aa48e4e38fe2bc401_2_1024x569.jpeg",979 "created_at": "2025-02-04T02:38:14.644Z",980 "last_posted_at": "2025-02-04T02:38:14.685Z",981 "bumped": true,982 "bumped_at": "2025-02-04T02:38:14.685Z",983 "archetype": "regular",984 "unseen": false,985 "pinned": false,986 "unpinned": null,987 "visible": true,988 "closed": false,989 "archived": false,990 "bookmarked": null,991 "liked": null,992 "tags_descriptions": {},993 "like_count": 0,994 "views": 39,995 "category_id": 1,996 "featured_link": null,997 "has_accepted_answer": false,998 "posters": [999 {1000 "extras": "latest single",1001 "description": "Original Poster, Most Recent Poster",1002 "user": {1003 "id": 81283,1004 "username": "sQuAde",1005 "name": "sQuAde",1006 "avatar_template": "/user_avatar/discuss.pytorch.org/squade/{size}/74332_2.png",1007 "trust_level": 11008 }1009 }1010 ]1011 },1012 {1013 "fancy_title": "RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation [torch==1.9]",1014 "id": 214697,1015 "title": "RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation [torch==1.9]",1016 "slug": "runtimeerror-one-of-the-variables-needed-for-gradient-computation-has-been-modified-by-an-inplace-operation-torch-1-9",1017 "posts_count": 7,1018 "reply_count": 3,1019 "highest_post_number": 7,1020 "image_url": null,1021 "created_at": "2024-12-27T09:40:41.289Z",1022 "last_posted_at": "2025-01-23T20:13:13.910Z",1023 "bumped": true,1024 "bumped_at": "2025-01-23T20:13:13.910Z",1025 "archetype": "regular",1026 "unseen": false,1027 "pinned": false,1028 "unpinned": null,1029 "visible": true,1030 "closed": false,1031 "archived": false,1032 "bookmarked": null,1033 "liked": null,1034 "tags_descriptions": {},1035 "like_count": 1,1036 "views": 94,1037 "category_id": 1,1038 "featured_link": null,1039 "has_accepted_answer": true,1040 "posters": [1041 {1042 "extras": null,1043 "description": "Original Poster",1044 "user": {1045 "id": 81744,1046 "username": "manish_kumar1",1047 "name": "Manish Kumar",1048 "avatar_template": "/user_avatar/discuss.pytorch.org/manish_kumar1/{size}/74750_2.png",1049 "trust_level": 11050 }1051 },1052 {1053 "extras": "latest",1054 "description": "Most Recent Poster, Accepted Answer",1055 "user": {1056 "id": 41396,1057 "username": "soulitzer",1058 "name": "",1059 "avatar_template": "/letter_avatar_proxy/v4/letter/s/839c29/{size}.png",1060 "trust_level": 21061 }1062 },1063 {1064 "extras": null,1065 "description": "Frequent Poster",1066 "user": {1067 "id": 41207,1068 "username": "SumanthRH",1069 "name": "",1070 "avatar_template": "/user_avatar/discuss.pytorch.org/sumanthrh/{size}/74752_2.png",1071 "trust_level": 21072 }1073 }1074 ]1075 },1076 {1077 "fancy_title": "Achieving sth like _foreach_clone?",1078 "id": 213750,1079 "title": "Achieving sth like _foreach_clone?",1080 "slug": "achieving-sth-like-foreach-clone",1081 "posts_count": 1,1082 "reply_count": 0,1083 "highest_post_number": 1,1084 "image_url": null,1085 "created_at": "2024-12-03T15:02:23.889Z",1086 "last_posted_at": "2024-12-03T15:02:23.934Z",1087 "bumped": true,1088 "bumped_at": "2024-12-03T15:02:23.934Z",1089 "archetype": "regular",1090 "unseen": false,1091 "pinned": false,1092 "unpinned": null,1093 "visible": true,1094 "closed": false,1095 "archived": false,1096 "bookmarked": null,1097 "liked": null,1098 "tags_descriptions": {},1099 "like_count": 0,1100 "views": 111,1101 "category_id": 1,1102 "featured_link": null,1103 "has_accepted_answer": false,1104 "posters": [1105 {1106 "extras": "latest single",1107 "description": "Original Poster, Most Recent Poster",1108 "user": {1109 "id": 4154,1110 "username": "sytrus-pytorch",1111 "name": "",1112 "avatar_template": "/letter_avatar_proxy/v4/letter/s/278dde/{size}.png",1113 "trust_level": 21114 }1115 }1116 ]1117 },1118 {1119 "fancy_title": "How to (efficiently) apply a function without a “dim” argument to each row of a 2D tensor?",1120 "id": 215365,1121 "title": "How to (efficiently) apply a function without a \"dim\" argument to each row of a 2D tensor?",1122 "slug": "how-to-efficiently-apply-a-function-without-a-dim-argument-to-each-row-of-a-2d-tensor",1123 "posts_count": 1,1124 "reply_count": 0,1125 "highest_post_number": 1,1126 "image_url": null,1127 "created_at": "2025-01-14T10:42:36.374Z",1128 "last_posted_at": "2025-01-14T10:42:36.448Z",1129 "bumped": true,1130 "bumped_at": "2025-01-14T10:42:36.448Z",1131 "archetype": "regular",1132 "unseen": false,1133 "pinned": false,1134 "unpinned": null,1135 "visible": true,1136 "closed": false,1137 "archived": false,1138 "bookmarked": null,1139 "liked": null,1140 "tags_descriptions": {},1141 "like_count": 0,1142 "views": 83,1143 "category_id": 1,1144 "featured_link": null,1145 "has_accepted_answer": false,1146 "posters": [1147 {1148 "extras": "latest single",1149 "description": "Original Poster, Most Recent Poster",1150 "user": {1151 "id": 82080,1152 "username": "Matt_T1",1153 "name": "Matt T.",1154 "avatar_template": "/user_avatar/discuss.pytorch.org/matt_t1/{size}/75097_2.png",1155 "trust_level": 11156 }1157 }1158 ]1159 },1160 {1161 "fancy_title": "GQA support in scaled_dot_product_attention",1162 "id": 216288,1163 "title": "GQA support in scaled_dot_product_attention",1164 "slug": "gqa-support-in-scaled-dot-product-attention",1165 "posts_count": 3,1166 "reply_count": 0,1167 "highest_post_number": 3,1168 "image_url": null,1169 "created_at": "2025-02-05T19:45:10.249Z",1170 "last_posted_at": "2025-03-29T00:34:36.072Z",1171 "bumped": true,1172 "bumped_at": "2025-03-29T00:34:36.072Z",1173 "archetype": "regular",1174 "unseen": false,1175 "pinned": false,1176 "unpinned": null,1177 "visible": true,1178 "closed": false,1179 "archived": false,1180 "bookmarked": null,1181 "liked": null,1182 "tags_descriptions": {},1183 "like_count": 0,1184 "views": 165,1185 "category_id": 1,1186 "featured_link": null,1187 "has_accepted_answer": false,1188 "posters": [1189 {1190 "extras": null,1191 "description": "Original Poster",1192 "user": {1193 "id": 81725,1194 "username": "mseeger",1195 "name": null,1196 "avatar_template": "/letter_avatar_proxy/v4/letter/m/6bbea6/{size}.png",1197 "trust_level": 11198 }1199 },1200 {