Anurag1734/cuda-error-resolution-analysis
07
1[2 {3 "post_stream": {4 "posts": [5 {6 "id": 259248,7 "name": "",8 "username": "jjhh",9 "avatar_template": "/user_avatar/discuss.pytorch.org/jjhh/{size}/13058_2.png",10 "created_at": "2021-01-24T13:40:29.090Z",11 "cooked": "<p>I made a function to compute the IoU loss of 2 spans</p>\n<pre><code class=\"lang-auto\">def iou_loss(pred_start, pred_end, target_start, target_end):\n \"\"\"\n pred_start, pred_end, target_start, target_end: b, *\n \"\"\"\n\n num_common = (\n torch.minimum(pred_end, target_end)\n - torch.maximum(pred_start, target_start) + 1\n ).clamp_min_(0)\n\n num_pred = (pred_end - pred_start + 1).clamp_min_(0)\n num_target = target_end - target_start + 1\n\n iou = num_common / (num_pred + num_target - num_common)\n\n return 1 - iou\n</code></pre>\n<p>However the gradients are not zero when the loss is at its global minimum.</p>\n<pre><code class=\"lang-auto\">pred_start = torch.tensor([5, 2], dtype=torch.float, requires_grad=True)\npred_end = torch.tensor([5, 4], dtype=torch.float, requires_grad=True)\ntarget_start = torch.tensor([5, 2])\ntarget_end = torch.tensor([5, 4])\n\nl = iou_loss(pred_start, pred_end, target_start, target_end)\nprint(l)\n# tensor([0., 0.], grad_fn=<RsubBackward1>)\n\nl = l.mean()\nl.backward()\n\nprint(pred_start.grad)\n# tensor([-0.5000, -0.1667])\n\nprint(pred_end.grad)\n# tensor([0.5000, 0.1667])\n</code></pre>\n<p>Can anyone help me to understand why? Any help would be appreciated.</p>",12 "post_number": 1,13 "post_type": 1,14 "posts_count": 5,15 "updated_at": "2021-01-24T13:44:05.242Z",16 "reply_count": 1,17 "reply_to_post_number": null,18 "quote_count": 0,19 "incoming_link_count": 189,20 "reads": 19,21 "readers_count": 18,22 "score": 953.8,23 "yours": false,24 "topic_id": 109844,25 "topic_slug": "why-are-gradients-not-zero-at-global-minimum",26 "display_username": "",27 "primary_group_name": null,28 "flair_name": null,29 "flair_url": null,30 "flair_bg_color": null,31 "flair_color": null,32 "flair_group_id": null,33 "badges_granted": [],34 "version": 1,35 "can_edit": false,36 "can_delete": false,37 "can_recover": false,38 "can_see_hidden_post": false,39 "can_wiki": false,40 "read": true,41 "user_title": null,42 "bookmarked": false,43 "actions_summary": [],44 "moderator": false,45 "admin": false,46 "staff": false,47 "user_id": 37390,48 "hidden": false,49 "trust_level": 1,50 "deleted_at": null,51 "user_deleted": false,52 "edit_reason": null,53 "can_view_edit_history": true,54 "wiki": false,55 "post_url": "/t/why-are-gradients-not-zero-at-global-minimum/109844/1",56 "can_accept_answer": false,57 "can_unaccept_answer": false,58 "accepted_answer": false,59 "topic_accepted_answer": true,60 "can_vote": false61 },62 {63 "id": 259278,64 "name": "K. Frank",65 "username": "KFrank",66 "avatar_template": "/letter_avatar_proxy/v4/letter/k/ecb155/{size}.png",67 "created_at": "2021-01-24T19:54:45.123Z",68 "cooked": "<p>Hi jjhh!</p>\n<aside class=\"quote no-group quote-modified\" data-username=\"jjhh\" data-post=\"1\" data-topic=\"109844\" data-full=\"true\">\n<div class=\"title\">\n<div class=\"quote-controls\"></div>\n<img loading=\"lazy\" alt=\"\" width=\"24\" height=\"24\" src=\"https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jjhh/48/13058_2.png\" class=\"avatar\"> jjhh:</div>\n<blockquote>\n<pre><code class=\"lang-auto\"> num_common = (\n torch.minimum(pred_end, target_end)\n - torch.maximum(pred_start, target_start) + 1\n ).clamp_min_(0)\n</code></pre>\n<p>However the gradients are not zero when the loss is at its global minimum.</p>\n</blockquote>\n</aside>\n<p>The short story is that <code>torch.minimum()</code> and <code>torch.maximum()</code> are not<br>\ndifferentiable at the special points where the arguments are equal.</p>\n<p>The function <code>abs (x)</code> offers a simpler example. When <code>x > 0</code>, the derivative<br>\nis <code>+1</code>, while when <code>x < 0</code>, it’s <code>-1</code>. When <code>x = 0</code>, mathematically speaking,<br>\nthe derivative is not defined. (One could try to play some game where you<br>\ndefine the derivative to be <code>0</code> when <code>x = 0</code> – and pytorch does do this for the<br>\n<code>abs()</code> function – but doing so is problematic, and not really worth the bother.)</p>\n<p>Here is an example script that probes your loss function with values where<br>\nthe arguments are close, but not exactly equal:</p>\n<pre data-code-wrap=\"python\"><code class=\"lang-python\">import torch\nprint (torch.__version__)\n\ndef iou_loss(pred_start, pred_end, target_start, target_end):\n \"\"\"\n pred_start, pred_end, target_start, target_end: b, *\n \"\"\"\n\n num_common = (\n torch.minimum(pred_end, target_end)\n - torch.maximum(pred_start, target_start) + 1\n ).clamp_min_(0)\n\n num_pred = (pred_end - pred_start + 1).clamp_min_(0)\n num_target = target_end - target_start + 1\n\n iou = num_common / (num_pred + num_target - num_common)\n\n return 1 - iou\n\n\npred_start = torch.tensor([5, 2], dtype=torch.float, requires_grad=True)\npred_end = torch.tensor([5, 4], dtype=torch.float, requires_grad=True)\ntarget_start = torch.tensor([5, 2])\ntarget_end = torch.tensor([5, 4])\n\n\ndelta = torch.tensor ([1.e-5, 0.0])\n\niou_loss(pred_start, pred_end, target_start, target_end).mean().backward()\nprint ('no delta: pred_start.grad =', pred_start.grad)\n\npred_start.grad *= 0.0\npred_end.grad *= 0.0\niou_loss(pred_start + delta, pred_end, target_start, target_end).mean().backward()\nprint ('plus delta: pred_start.grad =', pred_start.grad)\n\npred_start.grad *= 0.0\npred_end.grad *= 0.0\niou_loss(pred_start - delta, pred_end, target_start, target_end).mean().backward()\nprint ('minus delta: pred_start.grad =', pred_start.grad)\n</code></pre>\n<p>Here is its output:</p>\n<pre><code class=\"lang-plaintext\">1.7.1\nno delta: pred_start.grad = tensor([-0.5000, -0.1667])\nplus delta: pred_start.grad = tensor([ 0.5000, -0.1667])\nminus delta: pred_start.grad = tensor([-0.5000, -0.1667])\n</code></pre>\n<p>You can see the gradient jump as you cross the equality boundary (just<br>\nlike with the <code>abs (x)</code> example).</p>\n<p>Best.</p>\n<p>K. Frank</p>",69 "post_number": 2,70 "post_type": 1,71 "posts_count": 5,72 "updated_at": "2021-01-25T03:24:29.732Z",73 "reply_count": 0,74 "reply_to_post_number": null,75 "quote_count": 1,76 "incoming_link_count": 5,77 "reads": 15,78 "readers_count": 14,79 "score": 88.0,80 "yours": false,81 "topic_id": 109844,82 "topic_slug": "why-are-gradients-not-zero-at-global-minimum",83 "display_username": "K. Frank",84 "primary_group_name": null,85 "flair_name": null,86 "flair_url": null,87 "flair_bg_color": null,88 "flair_color": null,89 "flair_group_id": null,90 "badges_granted": [],91 "version": 1,92 "can_edit": false,93 "can_delete": false,94 "can_recover": false,95 "can_see_hidden_post": false,96 "can_wiki": false,97 "read": true,98 "user_title": null,99 "bookmarked": false,100 "actions_summary": [101 {102 "id": 2,103 "count": 2104 }105 ],106 "moderator": false,107 "admin": false,108 "staff": false,109 "user_id": 18088,110 "hidden": false,111 "trust_level": 2,112 "deleted_at": null,113 "user_deleted": false,114 "edit_reason": null,115 "can_view_edit_history": true,116 "wiki": false,117 "post_url": "/t/why-are-gradients-not-zero-at-global-minimum/109844/2",118 "can_accept_answer": false,119 "can_unaccept_answer": false,120 "accepted_answer": true,121 "topic_accepted_answer": true122 },123 {124 "id": 259309,125 "name": "",126 "username": "jjhh",127 "avatar_template": "/user_avatar/discuss.pytorch.org/jjhh/{size}/13058_2.png",128 "created_at": "2021-01-24T23:52:54.372Z",129 "cooked": "<p>Hi K.Frank,</p>\n<p>Thank you for the explanation.</p>\n<pre><code class=\"lang-auto\">def iou_loss(pred_start, pred_end, target_start, target_end):\n \"\"\"\n pred_start, pred_end, target_start, target_end: b, *\n \"\"\"\n\n num_common = (pred_end - pred_start + 1).clamp_min_(0)\n\n num_pred = (pred_end - pred_start + 1).clamp_min_(0)\n num_target = target_end - target_start + 1\n\n iou = num_common / (num_pred + num_target - num_common)\n\n return 1 - iou\n</code></pre>\n<p>I tested the function without <code>minimum</code> and <code>maximum</code>, but i still got the same gradients.</p>",130 "post_number": 3,131 "post_type": 1,132 "posts_count": 5,133 "updated_at": "2021-01-24T23:53:38.122Z",134 "reply_count": 1,135 "reply_to_post_number": null,136 "quote_count": 0,137 "incoming_link_count": 3,138 "reads": 11,139 "readers_count": 10,140 "score": 22.2,141 "yours": false,142 "topic_id": 109844,143 "topic_slug": "why-are-gradients-not-zero-at-global-minimum",144 "display_username": "",145 "primary_group_name": null,146 "flair_name": null,147 "flair_url": null,148 "flair_bg_color": null,149 "flair_color": null,150 "flair_group_id": null,151 "badges_granted": [],152 "version": 3,153 "can_edit": false,154 "can_delete": false,155 "can_recover": false,156 "can_see_hidden_post": false,157 "can_wiki": false,158 "read": true,159 "user_title": null,160 "bookmarked": false,161 "actions_summary": [],162 "moderator": false,163 "admin": false,164 "staff": false,165 "user_id": 37390,166 "hidden": false,167 "trust_level": 1,168 "deleted_at": null,169 "user_deleted": false,170 "edit_reason": null,171 "can_view_edit_history": true,172 "wiki": false,173 "post_url": "/t/why-are-gradients-not-zero-at-global-minimum/109844/3",174 "can_accept_answer": false,175 "can_unaccept_answer": false,176 "accepted_answer": false,177 "topic_accepted_answer": true178 },179 {180 "id": 259320,181 "name": "K. Frank",182 "username": "KFrank",183 "avatar_template": "/letter_avatar_proxy/v4/letter/k/ecb155/{size}.png",184 "created_at": "2021-01-25T02:57:31.356Z",185 "cooked": "<p>Hi jjhh!</p>\n<aside class=\"quote no-group\" data-username=\"jjhh\" data-post=\"3\" data-topic=\"109844\" data-full=\"true\">\n<div class=\"title\">\n<div class=\"quote-controls\"></div>\n<img loading=\"lazy\" alt=\"\" width=\"24\" height=\"24\" src=\"https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jjhh/48/13058_2.png\" class=\"avatar\"> jjhh:</div>\n<blockquote>\n<p>I tested the function without <code>minimum</code> and <code>maximum</code>, but i still got the same gradients.</p>\n</blockquote>\n</aside>\n<p>First, I do not get the same gradients using your new version of <code>iou_loss()</code>.</p>\n<p>Second, while your new <code>iou_loss()</code> still returns zero for the input parameters<br>\ngiven in your first post, this is no longer the minimum as <code>iou_loss()</code> can now<br>\nbecome negative.</p>\n<p>If you believe otherwise please post a complete, runnable script, together with<br>\nits output, that demonstrates your result.</p>\n<p>Best.</p>\n<p>K. Frank</p>",186 "post_number": 4,187 "post_type": 1,188 "posts_count": 5,189 "updated_at": "2021-01-25T02:57:31.356Z",190 "reply_count": 1,191 "reply_to_post_number": 3,192 "quote_count": 1,193 "incoming_link_count": 0,194 "reads": 9,195 "readers_count": 8,196 "score": 6.8,197 "yours": false,198 "topic_id": 109844,199 "topic_slug": "why-are-gradients-not-zero-at-global-minimum",200 "display_username": "K. Frank",201 "primary_group_name": null,202 "flair_name": null,203 "flair_url": null,204 "flair_bg_color": null,205 "flair_color": null,206 "flair_group_id": null,207 "badges_granted": [],208 "version": 1,209 "can_edit": false,210 "can_delete": false,211 "can_recover": false,212 "can_see_hidden_post": false,213 "can_wiki": false,214 "read": true,215 "user_title": null,216 "bookmarked": false,217 "actions_summary": [],218 "moderator": false,219 "admin": false,220 "staff": false,221 "user_id": 18088,222 "hidden": false,223 "trust_level": 2,224 "deleted_at": null,225 "user_deleted": false,226 "edit_reason": null,227 "can_view_edit_history": true,228 "wiki": false,229 "post_url": "/t/why-are-gradients-not-zero-at-global-minimum/109844/4",230 "can_accept_answer": false,231 "can_unaccept_answer": false,232 "accepted_answer": false,233 "topic_accepted_answer": true234 },235 {236 "id": 259324,237 "name": "",238 "username": "jjhh",239 "avatar_template": "/user_avatar/discuss.pytorch.org/jjhh/{size}/13058_2.png",240 "created_at": "2021-01-25T03:22:21.864Z",241 "cooked": "<p>Hi K.Frank, you are right. the loss becomes <code>1 - num_pred / num_target</code> now and it can be negative. The signs of the gradients switched, too. Sorry it was a stupid question. Thank you for your help.</p>",242 "post_number": 5,243 "post_type": 1,244 "posts_count": 5,245 "updated_at": "2021-01-25T03:22:21.864Z",246 "reply_count": 0,247 "reply_to_post_number": 4,248 "quote_count": 0,249 "incoming_link_count": 1,250 "reads": 8,251 "readers_count": 7,252 "score": 6.6,253 "yours": false,254 "topic_id": 109844,255 "topic_slug": "why-are-gradients-not-zero-at-global-minimum",256 "display_username": "",257 "primary_group_name": null,258 "flair_name": null,259 "flair_url": null,260 "flair_bg_color": null,261 "flair_color": null,262 "flair_group_id": null,263 "badges_granted": [],264 "version": 1,265 "can_edit": false,266 "can_delete": false,267 "can_recover": false,268 "can_see_hidden_post": false,269 "can_wiki": false,270 "read": true,271 "user_title": null,272 "reply_to_user": {273 "id": 18088,274 "username": "KFrank",275 "name": "K. Frank",276 "avatar_template": "/letter_avatar_proxy/v4/letter/k/ecb155/{size}.png"277 },278 "bookmarked": false,279 "actions_summary": [],280 "moderator": false,281 "admin": false,282 "staff": false,283 "user_id": 37390,284 "hidden": false,285 "trust_level": 1,286 "deleted_at": null,287 "user_deleted": false,288 "edit_reason": null,289 "can_view_edit_history": true,290 "wiki": false,291 "post_url": "/t/why-are-gradients-not-zero-at-global-minimum/109844/5",292 "can_accept_answer": false,293 "can_unaccept_answer": false,294 "accepted_answer": false,295 "topic_accepted_answer": true296 }297 ],298 "stream": [299 259248,300 259278,301 259309,302 259320,303 259324304 ]305 },306 "timeline_lookup": [307 [308 1,309 1735310 ]311 ],312 "suggested_topics": [313 {314 "fancy_title": "Why softmax training is more stable (than sigmoid)",315 "id": 213069,316 "title": "Why softmax training is more stable (than sigmoid)",317 "slug": "why-softmax-training-is-more-stable-than-sigmoid",318 "posts_count": 2,319 "reply_count": 0,320 "highest_post_number": 2,321 "image_url": null,322 "created_at": "2024-11-17T09:11:33.052Z",323 "last_posted_at": "2024-11-17T21:05:46.072Z",324 "bumped": true,325 "bumped_at": "2024-11-17T21:05:46.072Z",326 "archetype": "regular",327 "unseen": false,328 "pinned": false,329 "unpinned": null,330 "visible": true,331 "closed": false,332 "archived": false,333 "bookmarked": null,334 "liked": null,335 "tags_descriptions": {},336 "like_count": 1,337 "views": 170,338 "category_id": 1,339 "featured_link": null,340 "has_accepted_answer": false,341 "posters": [342 {343 "extras": null,344 "description": "Original Poster",345 "user": {346 "id": 50872,347 "username": "laro",348 "name": "amit",349 "avatar_template": "/user_avatar/discuss.pytorch.org/laro/{size}/47125_2.png",350 "trust_level": 1351 }352 },353 {354 "extras": "latest",355 "description": "Most Recent Poster",356 "user": {357 "id": 18088,358 "username": "KFrank",359 "name": "K. Frank",360 "avatar_template": "/letter_avatar_proxy/v4/letter/k/ecb155/{size}.png",361 "trust_level": 2362 }363 }364 ]365 },366 {367 "fancy_title": "How to approach a real-life problem while using rainfall data",368 "id": 212728,369 "title": "How to approach a real-life problem while using rainfall data",370 "slug": "how-to-approach-a-real-life-problem-while-using-rainfall-data",371 "posts_count": 1,372 "reply_count": 0,373 "highest_post_number": 1,374 "image_url": null,375 "created_at": "2024-11-09T07:21:57.841Z",376 "last_posted_at": "2024-11-09T07:21:57.888Z",377 "bumped": true,378 "bumped_at": "2024-11-09T07:21:57.888Z",379 "archetype": "regular",380 "unseen": false,381 "pinned": false,382 "unpinned": null,383 "visible": true,384 "closed": false,385 "archived": false,386 "bookmarked": null,387 "liked": null,388 "tags_descriptions": {},389 "like_count": 0,390 "views": 25,391 "category_id": 1,392 "featured_link": null,393 "has_accepted_answer": false,394 "posters": [395 {396 "extras": "latest single",397 "description": "Original Poster, Most Recent Poster",398 "user": {399 "id": 80781,400 "username": "Ritam_Pradhan",401 "name": "Ritam Pradhan",402 "avatar_template": "/user_avatar/discuss.pytorch.org/ritam_pradhan/{size}/73876_2.png",403 "trust_level": 1404 }405 }406 ]407 },408 {409 "fancy_title": "Running PyTorch to use artificial intelligence to generate images with Nvidia GTX 1650Ti",410 "id": 214280,411 "title": "Running PyTorch to use artificial intelligence to generate images with Nvidia GTX 1650Ti",412 "slug": "running-pytorch-to-use-artificial-intelligence-to-generate-images-with-nvidia-gtx-1650ti",413 "posts_count": 15,414 "reply_count": 14,415 "highest_post_number": 15,416 "image_url": null,417 "created_at": "2024-12-16T16:16:36.476Z",418 "last_posted_at": "2024-12-17T16:58:10.513Z",419 "bumped": true,420 "bumped_at": "2024-12-18T00:23:36.853Z",421 "archetype": "regular",422 "unseen": false,423 "pinned": false,424 "unpinned": null,425 "visible": true,426 "closed": false,427 "archived": false,428 "bookmarked": null,429 "liked": null,430 "tags_descriptions": {},431 "like_count": 0,432 "views": 708,433 "category_id": 1,434 "featured_link": null,435 "has_accepted_answer": false,436 "posters": [437 {438 "extras": "latest",439 "description": "Original Poster, Most Recent Poster",440 "user": {441 "id": 81532,442 "username": "Ronalds_Mazitis",443 "name": "Ronalds Mazitis",444 "avatar_template": "/letter_avatar_proxy/v4/letter/r/f08c70/{size}.png",445 "trust_level": 0446 }447 },448 {449 "extras": null,450 "description": "Frequent Poster",451 "user": {452 "id": 41396,453 "username": "soulitzer",454 "name": "",455 "avatar_template": "/letter_avatar_proxy/v4/letter/s/839c29/{size}.png",456 "trust_level": 2457 }458 },459 {460 "extras": null,461 "description": "Frequent Poster",462 "user": {463 "id": 3534,464 "username": "ptrblck",465 "name": "",466 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",467 "admin": true,468 "moderator": true,469 "trust_level": 2470 }471 }472 ]473 },474 {475 "fancy_title": "Performance issue fitting multiple models with CPU in parallel",476 "id": 215318,477 "title": "Performance issue fitting multiple models with CPU in parallel",478 "slug": "performance-issue-fitting-multiple-models-with-cpu-in-parallel",479 "posts_count": 2,480 "reply_count": 0,481 "highest_post_number": 2,482 "image_url": null,483 "created_at": "2025-01-13T07:25:58.523Z",484 "last_posted_at": "2025-01-16T14:03:59.333Z",485 "bumped": true,486 "bumped_at": "2025-01-16T14:03:59.333Z",487 "archetype": "regular",488 "unseen": false,489 "pinned": false,490 "unpinned": null,491 "visible": true,492 "closed": false,493 "archived": false,494 "bookmarked": null,495 "liked": null,496 "tags_descriptions": {},497 "like_count": 0,498 "views": 57,499 "category_id": 1,500 "featured_link": null,501 "has_accepted_answer": false,502 "posters": [503 {504 "extras": "latest single",505 "description": "Original Poster, Most Recent Poster",506 "user": {507 "id": 82048,508 "username": "carusyte",509 "name": "",510 "avatar_template": "/letter_avatar_proxy/v4/letter/c/5fc32e/{size}.png",511 "trust_level": 0512 }513 }514 ]515 },516 {517 "fancy_title": "Download models for sport event detection",518 "id": 219488,519 "title": "Download models for sport event detection",520 "slug": "download-models-for-sport-event-detection",521 "posts_count": 1,522 "reply_count": 0,523 "highest_post_number": 1,524 "image_url": null,525 "created_at": "2025-04-26T14:21:32.760Z",526 "last_posted_at": "2025-04-26T14:21:32.800Z",527 "bumped": true,528 "bumped_at": "2025-04-26T14:23:35.242Z",529 "archetype": "regular",530 "unseen": false,531 "pinned": false,532 "unpinned": null,533 "visible": true,534 "closed": false,535 "archived": false,536 "bookmarked": null,537 "liked": null,538 "tags_descriptions": {},539 "like_count": 0,540 "views": 48,541 "category_id": 1,542 "featured_link": null,543 "has_accepted_answer": false,544 "posters": [545 {546 "extras": "latest single",547 "description": "Original Poster, Most Recent Poster",548 "user": {549 "id": 84024,550 "username": "brainartfu1010",551 "name": "Brain Art",552 "avatar_template": "/user_avatar/discuss.pytorch.org/brainartfu1010/{size}/76799_2.png",553 "trust_level": 0554 }555 }556 ]557 }558 ],559 "tags_descriptions": {},560 "fancy_title": "Why are gradients not zero at global minimum?",561 "id": 109844,562 "title": "Why are gradients not zero at global minimum?",563 "posts_count": 5,564 "created_at": "2021-01-24T13:40:29.033Z",565 "views": 731,566 "reply_count": 2,567 "like_count": 2,568 "last_posted_at": "2021-01-25T03:22:21.864Z",569 "visible": true,570 "closed": false,571 "archived": false,572 "has_summary": false,573 "archetype": "regular",574 "slug": "why-are-gradients-not-zero-at-global-minimum",575 "category_id": 1,576 "word_count": 679,577 "deleted_at": null,578 "user_id": 37390,579 "featured_link": null,580 "pinned_globally": false,581 "pinned_at": null,582 "pinned_until": null,583 "image_url": null,584 "slow_mode_seconds": 0,585 "draft": null,586 "draft_key": "topic_109844",587 "draft_sequence": null,588 "unpinned": null,589 "pinned": false,590 "current_post_number": 1,591 "highest_post_number": 5,592 "deleted_by": null,593 "actions_summary": [594 {595 "id": 4,596 "count": 0,597 "hidden": false,598 "can_act": false599 },600 {601 "id": 8,602 "count": 0,603 "hidden": false,604 "can_act": false605 },606 {607 "id": 10,608 "count": 0,609 "hidden": false,610 "can_act": false611 },612 {613 "id": 7,614 "count": 0,615 "hidden": false,616 "can_act": false617 }618 ],619 "chunk_size": 20,620 "bookmarked": false,621 "topic_timer": null,622 "message_bus_last_id": 0,623 "participant_count": 2,624 "show_read_indicator": false,625 "thumbnails": null,626 "slow_mode_enabled_until": null,627 "accepted_answer": {628 "post_number": 2,629 "username": "KFrank",630 "name": "K. Frank",631 "excerpt": "Hi jjhh! \n\nThe short story is that torch.minimum() and torch.maximum() are not \ndifferentiable at the special points where the arguments are equal. \nThe function abs (x) offers a simpler example. When x > 0, the derivative \nis +1, while when x < 0, it’s -1. When x = 0, mathematically speaking, \nth…"632 },633 "can_vote": false,634 "vote_count": 0,635 "user_voted": false,636 "discourse_zendesk_plugin_zendesk_id": null,637 "discourse_zendesk_plugin_zendesk_url": "https://your-url.zendesk.com/agent/tickets/",638 "details": {639 "can_edit": false,640 "notification_level": 1,641 "participants": [642 {643 "id": 37390,644 "username": "jjhh",645 "name": "",646 "avatar_template": "/user_avatar/discuss.pytorch.org/jjhh/{size}/13058_2.png",647 "post_count": 3,648 "primary_group_name": null,649 "flair_name": null,650 "flair_url": null,651 "flair_color": null,652 "flair_bg_color": null,653 "flair_group_id": null,654 "trust_level": 1655 },656 {657 "id": 18088,658 "username": "KFrank",659 "name": "K. Frank",660 "avatar_template": "/letter_avatar_proxy/v4/letter/k/ecb155/{size}.png",661 "post_count": 2,662 "primary_group_name": null,663 "flair_name": null,664 "flair_url": null,665 "flair_color": null,666 "flair_bg_color": null,667 "flair_group_id": null,668 "trust_level": 2669 }670 ],671 "created_by": {672 "id": 37390,673 "username": "jjhh",674 "name": "",675 "avatar_template": "/user_avatar/discuss.pytorch.org/jjhh/{size}/13058_2.png"676 },677 "last_poster": {678 "id": 37390,679 "username": "jjhh",680 "name": "",681 "avatar_template": "/user_avatar/discuss.pytorch.org/jjhh/{size}/13058_2.png"682 }683 },684 "bookmarks": []685 },686 {687 "post_stream": {688 "posts": [689 {690 "id": 259263,691 "name": "",692 "username": "Loud_BoomBox",693 "avatar_template": "/user_avatar/discuss.pytorch.org/loud_boombox/{size}/34666_2.png",694 "created_at": "2021-01-24T15:48:25.966Z",695 "cooked": "<p>Hi everyone,</p>\n<p>I am a beginner to pytorch and trying to build a custom convolution layer (conv2d) where instead of the default multiplication I replace it with my own multiplication algo. I’ve attached my implementation below.</p><aside class=\"onebox allowlistedgeneric\" data-onebox-src=\"https://github.com/bALAJi-aDItHYa/Custom-conv2d\">\n <header class=\"source\">\n <img src=\"https://github.githubassets.com/favicons/favicon.svg\" class=\"site-icon\" width=\"32\" height=\"32\">\n\n <a href=\"https://github.com/bALAJi-aDItHYa/Custom-conv2d\" target=\"_blank\" rel=\"noopener nofollow ugc\">GitHub</a>\n </header>\n\n <article class=\"onebox-body\">\n <div class=\"aspect-image\" style=\"--aspect-ratio:690/344;\"><img src=\"https://opengraph.githubassets.com/208c2a391d6d30289ca75e280e9995514962fc8205805ae81e1462e716326ec1/bALAJi-aDItHYa/Custom-conv2d\" class=\"thumbnail\" width=\"690\" height=\"345\"></div>\n\n<h3><a href=\"https://github.com/bALAJi-aDItHYa/Custom-conv2d\" target=\"_blank\" rel=\"noopener nofollow ugc\">GitHub - bALAJi-aDItHYa/Custom-conv2d</a></h3>\n\n <p>Contribute to bALAJi-aDItHYa/Custom-conv2d development by creating an account on GitHub.</p>\n\n\n </article>\n\n <div class=\"onebox-metadata\">\n \n \n </div>\n\n <div style=\"clear: both\"></div>\n</aside>\n\n<p>I am getting the following error during backward pass - <strong>Function MBM_conv2dBackward returned an invalid gradient at index 1 - got [16, 32, 28, 28] but expected shape compatible with [32, 1, 3, 3]</strong>:</p>\n<p><div class=\"lightbox-wrapper\"><a class=\"lightbox\" href=\"https://discuss.pytorch.org/uploads/default/original/3X/7/a/7aecadd8b60f90cf553d336678a4d49978b25bca.png\" data-download-href=\"https://discuss.pytorch.org/uploads/default/7aecadd8b60f90cf553d336678a4d49978b25bca\" title=\"image\"><img src=\"https://discuss.pytorch.org/uploads/default/original/3X/7/a/7aecadd8b60f90cf553d336678a4d49978b25bca.png\" alt=\"image\" data-base62-sha1=\"hxrkHU4vrG3YgJ78jfDWMfv8rr4\" width=\"690\" height=\"273\" data-dominant-color=\"3A3B36\"><div class=\"meta\"><svg class=\"fa d-icon d-icon-far-image svg-icon\" aria-hidden=\"true\"><use href=\"#far-image\"></use></svg><span class=\"filename\">image</span><span class=\"informations\">854×338 61.8 KB</span><svg class=\"fa d-icon d-icon-discourse-expand svg-icon\" aria-hidden=\"true\"><use href=\"#discourse-expand\"></use></svg></div></a></div></p>\n<p>There seems to be a problem with the implementation of <strong>grad_weight</strong> calculation in the file:- <strong>custom_conv2d.py</strong></p>\n<p>My doubts are the following:</p>\n<ol>\n<li>Where does needs_input_grad variable come from?</li>\n<li><strong>input</strong> dimensions → [16,1,28,28] - (batch size = 16; 28x28 images)<br>\n<strong>grad_weight</strong> dimensions → [32,1,3,3] - (32 3x3 kernels)<br>\nWhat is the mistake being done during the grad_weight calculation? Or in the backward() implementation in general?</li>\n</ol>\n<p>I couldn’t figure out the problem. Would really appreciate any help! Thanks!</p>",696 "post_number": 1,697 "post_type": 1,698 "posts_count": 1,699 "updated_at": "2021-01-25T03:10:00.588Z",700 "reply_count": 0,701 "reply_to_post_number": null,702 "quote_count": 0,703 "incoming_link_count": 164,704 "reads": 19,705 "readers_count": 18,706 "score": 823.8,707 "yours": false,708 "topic_id": 109857,709 "topic_slug": "backward-in-custom-conv2d",710 "display_username": "",711 "primary_group_name": null,712 "flair_name": null,713 "flair_url": null,714 "flair_bg_color": null,715 "flair_color": null,716 "flair_group_id": null,717 "badges_granted": [],718 "version": 2,719 "can_edit": false,720 "can_delete": false,721 "can_recover": false,722 "can_see_hidden_post": false,723 "can_wiki": false,724 "link_counts": [725 {726 "url": "https://github.com/bALAJi-aDItHYa/Custom-conv2d",727 "internal": false,728 "reflection": false,729 "title": "GitHub - bALAJi-aDItHYa/Custom-conv2d",730 "clicks": 4731 },732 {733 "url": "https://discuss.pytorch.org/uploads/default/original/3X/7/a/7aecadd8b60f90cf553d336678a4d49978b25bca.png",734 "internal": true,735 "reflection": false,736 "clicks": 0737 }738 ],739 "read": true,740 "user_title": "",741 "bookmarked": false,742 "actions_summary": [],743 "moderator": false,744 "admin": false,745 "staff": false,746 "user_id": 41354,747 "hidden": false,748 "trust_level": 1,749 "deleted_at": null,750 "user_deleted": false,751 "edit_reason": null,752 "can_view_edit_history": true,753 "wiki": false,754 "post_url": "/t/backward-in-custom-conv2d/109857/1",755 "can_accept_answer": false,756 "can_unaccept_answer": false,757 "accepted_answer": false,758 "topic_accepted_answer": null,759 "can_vote": false760 }761 ],762 "stream": [763 259263764 ]765 },766 "timeline_lookup": [767 [768 1,769 1735770 ]771 ],772 "suggested_topics": [773 {774 "fancy_title": "Use flexattention with torchrec",775 "id": 215998,776 "title": "Use flexattention with torchrec",777 "slug": "use-flexattention-with-torchrec",778 "posts_count": 2,779 "reply_count": 0,780 "highest_post_number": 2,781 "image_url": null,782 "created_at": "2025-01-28T17:55:21.621Z",783 "last_posted_at": "2025-01-28T19:16:28.133Z",784 "bumped": true,785 "bumped_at": "2025-01-28T19:16:28.133Z",786 "archetype": "regular",787 "unseen": false,788 "pinned": false,789 "unpinned": null,790 "visible": true,791 "closed": false,792 "archived": false,793 "bookmarked": null,794 "liked": null,795 "tags_descriptions": {},796 "like_count": 0,797 "views": 136,798 "category_id": 1,799 "featured_link": null,800 "has_accepted_answer": false,801 "posters": [802 {803 "extras": "latest single",804 "description": "Original Poster, Most Recent Poster",805 "user": {806 "id": 82366,807 "username": "Tpopok",808 "name": "Topopk",809 "avatar_template": "/user_avatar/discuss.pytorch.org/tpopok/{size}/75353_2.png",810 "trust_level": 0811 }812 }813 ]814 },815 {816 "fancy_title": "Setting allow_fp16_reduced_precision_reduction via libtorch",817 "id": 213951,818 "title": "Setting allow_fp16_reduced_precision_reduction via libtorch",819 "slug": "setting-allow-fp16-reduced-precision-reduction-via-libtorch",820 "posts_count": 6,821 "reply_count": 2,822 "highest_post_number": 6,823 "image_url": null,824 "created_at": "2024-12-07T21:00:08.256Z",825 "last_posted_at": "2024-12-08T17:00:40.842Z",826 "bumped": true,827 "bumped_at": "2024-12-08T17:00:40.842Z",828 "archetype": "regular",829 "unseen": false,830 "pinned": false,831 "unpinned": null,832 "visible": true,833 "closed": false,834 "archived": false,835 "bookmarked": null,836 "liked": null,837 "tags_descriptions": {},838 "like_count": 0,839 "views": 96,840 "category_id": 1,841 "featured_link": null,842 "has_accepted_answer": false,843 "posters": [844 {845 "extras": "latest",846 "description": "Original Poster, Most Recent Poster",847 "user": {848 "id": 81380,849 "username": "elvircrn",850 "name": "Elvir Crnčević",851 "avatar_template": "/user_avatar/discuss.pytorch.org/elvircrn/{size}/74418_2.png",852 "trust_level": 1853 }854 },855 {856 "extras": null,857 "description": "Frequent Poster",858 "user": {859 "id": 3534,860 "username": "ptrblck",861 "name": "",862 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",863 "admin": true,864 "moderator": true,865 "trust_level": 2866 }867 }868 ]869 },870 {871 "fancy_title": "Out-of-Memory Error in Multi-GPU Distributed with Torch and Hugging Face Trainer",872 "id": 213548,873 "title": "Out-of-Memory Error in Multi-GPU Distributed with Torch and Hugging Face Trainer",874 "slug": "out-of-memory-error-in-multi-gpu-distributed-with-torch-and-hugging-face-trainer",875 "posts_count": 1,876 "reply_count": 0,877 "highest_post_number": 1,878 "image_url": null,879 "created_at": "2024-11-27T18:27:40.133Z",880 "last_posted_at": "2024-11-27T18:27:40.193Z",881 "bumped": true,882 "bumped_at": "2024-11-27T18:27:40.193Z",883 "archetype": "regular",884 "unseen": false,885 "pinned": false,886 "unpinned": null,887 "visible": true,888 "closed": false,889 "archived": false,890 "bookmarked": null,891 "liked": null,892 "tags_descriptions": {},893 "like_count": 0,894 "views": 117,895 "category_id": 1,896 "featured_link": null,897 "has_accepted_answer": false,898 "posters": [899 {900 "extras": "latest single",901 "description": "Original Poster, Most Recent Poster",902 "user": {903 "id": 37254,904 "username": "enterthevoidf22",905 "name": "",906 "avatar_template": "/user_avatar/discuss.pytorch.org/enterthevoidf22/{size}/29338_2.png",907 "trust_level": 2908 }909 }910 ]911 },912 {913 "fancy_title": "How do I make a C# Console app build and debug in Visual Studio Code? ",914 "id": 215932,915 "title": "How do I make a C# Console app build and debug in Visual Studio Code? ",916 "slug": "how-do-i-make-a-c-console-app-build-and-debug-in-visual-studio-code",917 "posts_count": 2,918 "reply_count": 0,919 "highest_post_number": 2,920 "image_url": null,921 "created_at": "2025-01-27T14:00:02.161Z",922 "last_posted_at": "2025-01-27T14:01:01.767Z",923 "bumped": true,924 "bumped_at": "2025-01-27T14:01:01.767Z",925 "archetype": "regular",926 "unseen": false,927 "pinned": false,928 "unpinned": null,929 "visible": true,930 "closed": false,931 "archived": false,932 "bookmarked": null,933 "liked": null,934 "tags_descriptions": {},935 "like_count": 0,936 "views": 32,937 "category_id": 1,938 "featured_link": null,939 "has_accepted_answer": false,940 "posters": [941 {942 "extras": null,943 "description": "Original Poster",944 "user": {945 "id": 82336,946 "username": "William_Thompson",947 "name": "William Thompson",948 "avatar_template": "/user_avatar/discuss.pytorch.org/william_thompson/{size}/75316_2.png",949 "trust_level": 0950 }951 },952 {953 "extras": "latest",954 "description": "Most Recent Poster",955 "user": {956 "id": 3534,957 "username": "ptrblck",958 "name": "",959 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",960 "admin": true,961 "moderator": true,962 "trust_level": 2963 }964 }965 ]966 },967 {968 "fancy_title": "Embedding model GPU memory usage",969 "id": 216807,970 "title": "Embedding model GPU memory usage",971 "slug": "embedding-model-gpu-memory-usage",972 "posts_count": 2,973 "reply_count": 0,974 "highest_post_number": 2,975 "image_url": null,976 "created_at": "2025-02-18T06:36:18.852Z",977 "last_posted_at": "2025-02-18T21:36:52.485Z",978 "bumped": true,979 "bumped_at": "2025-02-18T21:36:52.485Z",980 "archetype": "regular",981 "unseen": false,982 "pinned": false,983 "unpinned": null,984 "visible": true,985 "closed": false,986 "archived": false,987 "bookmarked": null,988 "liked": null,989 "tags_descriptions": {},990 "like_count": 0,991 "views": 116,992 "category_id": 1,993 "featured_link": null,994 "has_accepted_answer": false,995 "posters": [996 {997 "extras": null,998 "description": "Original Poster",999 "user": {1000 "id": 77693,1001 "username": "Jared4",1002 "name": "KR",1003 "avatar_template": "/letter_avatar_proxy/v4/letter/j/e68b1a/{size}.png",1004 "trust_level": 11005 }1006 },1007 {1008 "extras": "latest",1009 "description": "Most Recent Poster",1010 "user": {1011 "id": 3534,1012 "username": "ptrblck",1013 "name": "",1014 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",1015 "admin": true,1016 "moderator": true,1017 "trust_level": 21018 }1019 }1020 ]1021 }1022 ],1023 "tags_descriptions": {},1024 "fancy_title": "Backward() in custom conv2d",1025 "id": 109857,1026 "title": "Backward() in custom conv2d",1027 "posts_count": 1,1028 "created_at": "2021-01-24T15:48:25.905Z",1029 "views": 635,1030 "reply_count": 0,1031 "like_count": 0,1032 "last_posted_at": "2021-01-24T15:48:25.966Z",1033 "visible": true,1034 "closed": false,1035 "archived": false,1036 "has_summary": false,1037 "archetype": "regular",1038 "slug": "backward-in-custom-conv2d",1039 "category_id": 1,1040 "word_count": 162,1041 "deleted_at": null,1042 "user_id": 41354,1043 "featured_link": null,1044 "pinned_globally": false,1045 "pinned_at": null,1046 "pinned_until": null,1047 "image_url": "https://discuss.pytorch.org/uploads/default/original/3X/7/a/7aecadd8b60f90cf553d336678a4d49978b25bca.png",1048 "slow_mode_seconds": 0,1049 "draft": null,1050 "draft_key": "topic_109857",1051 "draft_sequence": null,1052 "unpinned": null,1053 "pinned": false,1054 "current_post_number": 1,1055 "highest_post_number": 1,1056 "deleted_by": null,1057 "actions_summary": [1058 {1059 "id": 4,1060 "count": 0,1061 "hidden": false,1062 "can_act": false1063 },1064 {1065 "id": 8,1066 "count": 0,1067 "hidden": false,1068 "can_act": false1069 },1070 {1071 "id": 10,1072 "count": 0,1073 "hidden": false,1074 "can_act": false1075 },1076 {1077 "id": 7,1078 "count": 0,1079 "hidden": false,1080 "can_act": false1081 }1082 ],1083 "chunk_size": 20,1084 "bookmarked": false,1085 "topic_timer": null,1086 "message_bus_last_id": 0,1087 "participant_count": 1,1088 "show_read_indicator": false,1089 "thumbnails": [1090 {1091 "max_width": null,1092 "max_height": null,1093 "width": 854,1094 "height": 338,1095 "url": "https://discuss.pytorch.org/uploads/default/original/3X/7/a/7aecadd8b60f90cf553d336678a4d49978b25bca.png"1096 }1097 ],1098 "slow_mode_enabled_until": null,1099 "can_vote": false,1100 "vote_count": 0,1101 "user_voted": false,1102 "discourse_zendesk_plugin_zendesk_id": null,1103 "discourse_zendesk_plugin_zendesk_url": "https://your-url.zendesk.com/agent/tickets/",1104 "details": {1105 "can_edit": false,1106 "notification_level": 1,1107 "participants": [1108 {1109 "id": 41354,1110 "username": "Loud_BoomBox",1111 "name": "",1112 "avatar_template": "/user_avatar/discuss.pytorch.org/loud_boombox/{size}/34666_2.png",1113 "post_count": 1,1114 "primary_group_name": null,1115 "flair_name": null,1116 "flair_url": null,1117 "flair_color": null,1118 "flair_bg_color": null,1119 "flair_group_id": null,1120 "trust_level": 11121 }1122 ],1123 "created_by": {1124 "id": 41354,1125 "username": "Loud_BoomBox",1126 "name": "",1127 "avatar_template": "/user_avatar/discuss.pytorch.org/loud_boombox/{size}/34666_2.png"1128 },1129 "last_poster": {1130 "id": 41354,1131 "username": "Loud_BoomBox",1132 "name": "",1133 "avatar_template": "/user_avatar/discuss.pytorch.org/loud_boombox/{size}/34666_2.png"1134 },1135 "links": [1136 {1137 "url": "https://github.com/bALAJi-aDItHYa/Custom-conv2d",1138 "title": "GitHub - bALAJi-aDItHYa/Custom-conv2d",1139 "internal": false,1140 "attachment": false,1141 "reflection": false,1142 "clicks": 4,1143 "user_id": 41354,1144 "domain": "github.com",1145 "root_domain": "github.com"1146 }1147 ]1148 },1149 "bookmarks": []1150 },1151 {1152 "post_stream": {1153 "posts": [1154 {1155 "id": 259282,1156 "name": "Kabir Nagrecha",1157 "username": "Kabir_Nagrecha",1158 "avatar_template": "/user_avatar/discuss.pytorch.org/kabir_nagrecha/{size}/30493_2.png",1159 "created_at": "2021-01-24T20:23:09.232Z",1160 "cooked": "<p>I am writing a function with attempts to find the upper bound of the possible model size. I do this in a loop which at each iteration “tries” to append a new layer to a moduleList, constructs a model off of that list, then attempts a single forward pass.</p>\n<p>In the exception, I delete the moduleList, the model, and <em>attempt</em> to delete the output of the forward pass. Because the forward pass failed however, the output of course does not exist. The issue is, whatever intermediate computations were attempted on the way to the failure (mid-forward-pass) persist and eat up memory. How do I free them? Is there a way to kill ALL GPU tensors at this moment?</p>\n<p>Example Code (not real):</p>\n<pre><code class=\"lang-auto\">modules = []\nfor i in range(1000):\n try:\n modules.append(nn.Linear(10, 10)).to(device)\n model = nn.Sequential(*modules)\n out = model(input_batch)\n del out\n del model\n except:\n try:\n del out\n except:\n pass\n del model\n del modules\n gc.collect()\n torch.cuda.empty_cache()\n print(torch.cuda.memory_allocated(0)) # this will show that memory is still full!\n</code></pre>",1161 "post_number": 1,1162 "post_type": 1,1163 "posts_count": 2,1164 "updated_at": "2021-01-24T20:32:47.342Z",1165 "reply_count": 0,1166 "reply_to_post_number": null,1167 "quote_count": 0,1168 "incoming_link_count": 616,1169 "reads": 13,1170 "readers_count": 12,1171 "score": 3082.6,1172 "yours": false,1173 "topic_id": 109869,1174 "topic_slug": "gpu-memory-usage-handling-with-try-except",1175 "display_username": "Kabir Nagrecha",1176 "primary_group_name": null,1177 "flair_name": null,1178 "flair_url": null,1179 "flair_bg_color": null,1180 "flair_color": null,1181 "flair_group_id": null,1182 "badges_granted": [],1183 "version": 2,1184 "can_edit": false,1185 "can_delete": false,1186 "can_recover": false,1187 "can_see_hidden_post": false,1188 "can_wiki": false,1189 "read": true,1190 "user_title": null,1191 "bookmarked": false,1192 "actions_summary": [],1193 "moderator": false,1194 "admin": false,1195 "staff": false,1196 "user_id": 38977,1197 "hidden": false,1198 "trust_level": 1,1199 "deleted_at": null,1200 "user_deleted": false,