Anurag1734/cuda-error-resolution-analysis
07
1[2 {3 "post_stream": {4 "posts": [5 {6 "id": 112701,7 "name": "Martin",8 "username": "martinr",9 "avatar_template": "/user_avatar/discuss.pytorch.org/martinr/{size}/30426_2.png",10 "created_at": "2019-05-21T21:31:01.127Z",11 "cooked": "<p>I am trying to expand a [200, 176, 2] binary mask to select from [200, 176, 14] tensor, so that first 7 elements from the tensor’s 3rd dimension (size 14) would be selected by mask[:, :, 0] and last 7 elements by mask[:, :, 1]. E.g. if my mask at third dimension is [0,1] then a selection is made as if it was [0,0,0,0,0,0,0,1,1,1,1,1,1,1]. I managed to solve it by this piece of lengthy code, but I imagine there must be a shorter and more straightforward way (and also without using Numpy as I intend to process this on GPU).</p>\n<p>Goal in short: use [200, 176, 2] binary mask <em><strong>b</strong></em> to select from [200, 176, 14] tensor <em><strong>a</strong></em></p>\n<p>My current code (works as expected, but very lengthy):</p>\n<pre><code class=\"lang-auto\"># tensor to select from\na = torch.rand([200,176,14])\n\n# mask\nb = torch.zeros([200,176,2], dtype=torch.uint8)\n\n# split mask by the last dimension\nmask_parts = torch.split(b, 1, dim=2)\n\n# first part, size torch.Size([200, 176, 1])\nmask1 = mask_parts[0]\n# expand to torch.Size([200, 176, 7])\nmask1 = mask1.expand(-1,-1,7)\n\n# second part, identical processing to the first\nmask2 = mask_parts[1]\nmask2 = mask2.expand(-1,-1,7)\n\n# join masks, get [200, 176, 14]\nmask = torch.cat((mask1, mask2), 2)\n\n# now the goal - use the mask to select elements from a\nresult = a[mask]\n</code></pre>\n<p>Is there a better way to achieve the same result or is it OK?</p>",12 "post_number": 1,13 "post_type": 1,14 "posts_count": 3,15 "updated_at": "2019-05-21T21:31:50.073Z",16 "reply_count": 0,17 "reply_to_post_number": null,18 "quote_count": 0,19 "incoming_link_count": 660,20 "reads": 34,21 "readers_count": 33,22 "score": 3301.8,23 "yours": false,24 "topic_id": 45850,25 "topic_slug": "expanding-multidimensional-tensor-by-non-singleton-dimension",26 "display_username": "Martin",27 "primary_group_name": null,28 "flair_name": null,29 "flair_url": null,30 "flair_bg_color": null,31 "flair_color": null,32 "flair_group_id": null,33 "badges_granted": [],34 "version": 1,35 "can_edit": false,36 "can_delete": false,37 "can_recover": false,38 "can_see_hidden_post": false,39 "can_wiki": false,40 "read": true,41 "user_title": null,42 "bookmarked": false,43 "actions_summary": [],44 "moderator": false,45 "admin": false,46 "staff": false,47 "user_id": 17358,48 "hidden": false,49 "trust_level": 2,50 "deleted_at": null,51 "user_deleted": false,52 "edit_reason": null,53 "can_view_edit_history": true,54 "wiki": false,55 "post_url": "/t/expanding-multidimensional-tensor-by-non-singleton-dimension/45850/1",56 "can_accept_answer": false,57 "can_unaccept_answer": false,58 "accepted_answer": false,59 "topic_accepted_answer": true,60 "can_vote": false61 },62 {63 "id": 112710,64 "name": "Arul",65 "username": "InnovArul",66 "avatar_template": "/user_avatar/discuss.pytorch.org/innovarul/{size}/5282_2.png",67 "created_at": "2019-05-21T23:11:49.022Z",68 "cooked": "<p>To expand [200, 176, 2] mask to size [200, 176, 14], you can do the following:</p>\n<pre><code class=\"lang-auto\">new_mask = b.unsqueeze(-1).repeat(1, 1, 1, 7).view(200, 176, -1)\nprint( torch.all((new_mask.float() - mask.float()) == 0)) # 1\n</code></pre>",69 "post_number": 2,70 "post_type": 1,71 "posts_count": 3,72 "updated_at": "2019-05-22T10:34:31.356Z",73 "reply_count": 1,74 "reply_to_post_number": null,75 "quote_count": 0,76 "incoming_link_count": 1,77 "reads": 30,78 "readers_count": 29,79 "score": 31.0,80 "yours": false,81 "topic_id": 45850,82 "topic_slug": "expanding-multidimensional-tensor-by-non-singleton-dimension",83 "display_username": "Arul",84 "primary_group_name": null,85 "flair_name": null,86 "flair_url": null,87 "flair_bg_color": null,88 "flair_color": null,89 "flair_group_id": null,90 "badges_granted": [],91 "version": 1,92 "can_edit": false,93 "can_delete": false,94 "can_recover": false,95 "can_see_hidden_post": false,96 "can_wiki": false,97 "read": true,98 "user_title": "",99 "bookmarked": false,100 "actions_summary": [101 {102 "id": 2,103 "count": 1104 }105 ],106 "moderator": false,107 "admin": false,108 "staff": false,109 "user_id": 998,110 "hidden": false,111 "trust_level": 2,112 "deleted_at": null,113 "user_deleted": false,114 "edit_reason": null,115 "can_view_edit_history": true,116 "wiki": false,117 "post_url": "/t/expanding-multidimensional-tensor-by-non-singleton-dimension/45850/2",118 "can_accept_answer": false,119 "can_unaccept_answer": false,120 "accepted_answer": true,121 "topic_accepted_answer": true122 },123 {124 "id": 112787,125 "name": "Martin",126 "username": "martinr",127 "avatar_template": "/user_avatar/discuss.pytorch.org/martinr/{size}/30426_2.png",128 "created_at": "2019-05-22T10:37:05.849Z",129 "cooked": "<p>Thank you! Expanding into 4th dimension and back was something I couldn’t think of myself!</p>",130 "post_number": 3,131 "post_type": 1,132 "posts_count": 3,133 "updated_at": "2019-05-22T10:37:05.849Z",134 "reply_count": 0,135 "reply_to_post_number": 2,136 "quote_count": 0,137 "incoming_link_count": 2,138 "reads": 27,139 "readers_count": 26,140 "score": 15.4,141 "yours": false,142 "topic_id": 45850,143 "topic_slug": "expanding-multidimensional-tensor-by-non-singleton-dimension",144 "display_username": "Martin",145 "primary_group_name": null,146 "flair_name": null,147 "flair_url": null,148 "flair_bg_color": null,149 "flair_color": null,150 "flair_group_id": null,151 "badges_granted": [],152 "version": 1,153 "can_edit": false,154 "can_delete": false,155 "can_recover": false,156 "can_see_hidden_post": false,157 "can_wiki": false,158 "read": true,159 "user_title": null,160 "reply_to_user": {161 "id": 998,162 "username": "InnovArul",163 "name": "Arul",164 "avatar_template": "/user_avatar/discuss.pytorch.org/innovarul/{size}/5282_2.png"165 },166 "bookmarked": false,167 "actions_summary": [],168 "moderator": false,169 "admin": false,170 "staff": false,171 "user_id": 17358,172 "hidden": false,173 "trust_level": 2,174 "deleted_at": null,175 "user_deleted": false,176 "edit_reason": null,177 "can_view_edit_history": true,178 "wiki": false,179 "post_url": "/t/expanding-multidimensional-tensor-by-non-singleton-dimension/45850/3",180 "can_accept_answer": false,181 "can_unaccept_answer": false,182 "accepted_answer": false,183 "topic_accepted_answer": true184 }185 ],186 "stream": [187 112701,188 112710,189 112787190 ]191 },192 "timeline_lookup": [193 [194 1,195 2349196 ]197 ],198 "suggested_topics": [199 {200 "fancy_title": "ERROR: Could not find a version that satisfies the requirement torch (from versions: none)",201 "id": 218843,202 "title": "ERROR: Could not find a version that satisfies the requirement torch (from versions: none)",203 "slug": "error-could-not-find-a-version-that-satisfies-the-requirement-torch-from-versions-none",204 "posts_count": 5,205 "reply_count": 3,206 "highest_post_number": 5,207 "image_url": null,208 "created_at": "2025-04-07T21:21:50.654Z",209 "last_posted_at": "2025-04-08T09:19:57.929Z",210 "bumped": true,211 "bumped_at": "2025-04-08T09:19:57.929Z",212 "archetype": "regular",213 "unseen": false,214 "pinned": false,215 "unpinned": null,216 "visible": true,217 "closed": false,218 "archived": false,219 "bookmarked": null,220 "liked": null,221 "tags_descriptions": {},222 "like_count": 0,223 "views": 2901,224 "category_id": 1,225 "featured_link": null,226 "has_accepted_answer": true,227 "posters": [228 {229 "extras": "latest",230 "description": "Original Poster, Most Recent Poster",231 "user": {232 "id": 83690,233 "username": "sputteringPytorch",234 "name": "",235 "avatar_template": "/letter_avatar_proxy/v4/letter/s/35a633/{size}.png",236 "trust_level": 0237 }238 },239 {240 "extras": null,241 "description": "Frequent Poster, Accepted Answer",242 "user": {243 "id": 3534,244 "username": "ptrblck",245 "name": "",246 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",247 "admin": true,248 "moderator": true,249 "trust_level": 2250 }251 }252 ]253 },254 {255 "fancy_title": "Why is there idle time between kernels before and after gpu_memset?",256 "id": 215446,257 "title": "Why is there idle time between kernels before and after gpu_memset?",258 "slug": "why-is-there-idle-time-between-kernels-before-and-after-gpu-memset",259 "posts_count": 1,260 "reply_count": 0,261 "highest_post_number": 1,262 "image_url": "https://discuss.pytorch.org/uploads/default/optimized/3X/6/c/6cea3aff98c501affa2fae5b9c9280fd106c9264_2_1024x460.png",263 "created_at": "2025-01-16T03:46:17.769Z",264 "last_posted_at": "2025-01-16T03:46:17.805Z",265 "bumped": true,266 "bumped_at": "2025-01-16T03:46:17.805Z",267 "archetype": "regular",268 "unseen": false,269 "pinned": false,270 "unpinned": null,271 "visible": true,272 "closed": false,273 "archived": false,274 "bookmarked": null,275 "liked": null,276 "tags_descriptions": {},277 "like_count": 0,278 "views": 76,279 "category_id": 1,280 "featured_link": null,281 "has_accepted_answer": false,282 "posters": [283 {284 "extras": "latest single",285 "description": "Original Poster, Most Recent Poster",286 "user": {287 "id": 82112,288 "username": "Ning_Wang",289 "name": "Ning Wang",290 "avatar_template": "/user_avatar/discuss.pytorch.org/ning_wang/{size}/75120_2.png",291 "trust_level": 1292 }293 }294 ]295 },296 {297 "fancy_title": "torch.OutOfMemoryError Needing Help",298 "id": 215383,299 "title": "torch.OutOfMemoryError Needing Help",300 "slug": "torch-outofmemoryerror-needing-help",301 "posts_count": 2,302 "reply_count": 0,303 "highest_post_number": 2,304 "image_url": "https://discuss.pytorch.org/uploads/default/original/3X/7/1/7177b82c5c3fa982bd5bcb76e82413cb1a9895ee.png",305 "created_at": "2025-01-14T16:13:00.335Z",306 "last_posted_at": "2025-01-14T20:48:39.772Z",307 "bumped": true,308 "bumped_at": "2025-01-14T20:48:39.772Z",309 "archetype": "regular",310 "unseen": false,311 "pinned": false,312 "unpinned": null,313 "visible": true,314 "closed": false,315 "archived": false,316 "bookmarked": null,317 "liked": null,318 "tags_descriptions": {},319 "like_count": 0,320 "views": 72,321 "category_id": 1,322 "featured_link": null,323 "has_accepted_answer": false,324 "posters": [325 {326 "extras": null,327 "description": "Original Poster",328 "user": {329 "id": 82088,330 "username": "meditrust",331 "name": "Yohan Azoulay",332 "avatar_template": "/letter_avatar_proxy/v4/letter/m/f1d935/{size}.png",333 "trust_level": 0334 }335 },336 {337 "extras": "latest",338 "description": "Most Recent Poster",339 "user": {340 "id": 3534,341 "username": "ptrblck",342 "name": "",343 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",344 "admin": true,345 "moderator": true,346 "trust_level": 2347 }348 }349 ]350 },351 {352 "fancy_title": "Unexpected Behavior of torch.median()",353 "id": 218227,354 "title": "Unexpected Behavior of torch.median()",355 "slug": "unexpected-behavior-of-torch-median",356 "posts_count": 2,357 "reply_count": 0,358 "highest_post_number": 2,359 "image_url": null,360 "created_at": "2025-03-25T08:41:52.972Z",361 "last_posted_at": "2025-03-25T13:11:25.772Z",362 "bumped": true,363 "bumped_at": "2025-03-25T13:11:25.772Z",364 "archetype": "regular",365 "unseen": false,366 "pinned": false,367 "unpinned": null,368 "visible": true,369 "closed": false,370 "archived": false,371 "bookmarked": null,372 "liked": null,373 "tags_descriptions": {},374 "like_count": 0,375 "views": 45,376 "category_id": 1,377 "featured_link": null,378 "has_accepted_answer": false,379 "posters": [380 {381 "extras": null,382 "description": "Original Poster",383 "user": {384 "id": 83450,385 "username": "nishantbadhautiya",386 "name": "Nishant",387 "avatar_template": "/user_avatar/discuss.pytorch.org/nishantbadhautiya/{size}/76403_2.png",388 "trust_level": 0389 }390 },391 {392 "extras": "latest",393 "description": "Most Recent Poster",394 "user": {395 "id": 3534,396 "username": "ptrblck",397 "name": "",398 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",399 "admin": true,400 "moderator": true,401 "trust_level": 2402 }403 }404 ]405 },406 {407 "fancy_title": "Question about chain LR scheduler (CosineAnnealingLR + ExponentiaLR)",408 "id": 219147,409 "title": "Question about chain LR scheduler (CosineAnnealingLR + ExponentiaLR)",410 "slug": "question-about-chain-lr-scheduler-cosineannealinglr-exponentialr",411 "posts_count": 1,412 "reply_count": 0,413 "highest_post_number": 1,414 "image_url": "https://discuss.pytorch.org/uploads/default/optimized/3X/a/f/afa0661593930b408a3818112e5b04b7664f390b_2_1024x192.png",415 "created_at": "2025-04-16T08:36:33.463Z",416 "last_posted_at": "2025-04-16T08:36:33.505Z",417 "bumped": true,418 "bumped_at": "2025-04-16T08:36:33.505Z",419 "archetype": "regular",420 "unseen": false,421 "pinned": false,422 "unpinned": null,423 "visible": true,424 "closed": false,425 "archived": false,426 "bookmarked": null,427 "liked": null,428 "tags_descriptions": {},429 "like_count": 0,430 "views": 70,431 "category_id": 1,432 "featured_link": null,433 "has_accepted_answer": false,434 "posters": [435 {436 "extras": "latest single",437 "description": "Original Poster, Most Recent Poster",438 "user": {439 "id": 83846,440 "username": "jesus-333",441 "name": "Alberto Zancanaro",442 "avatar_template": "/user_avatar/discuss.pytorch.org/jesus-333/{size}/76654_2.png",443 "trust_level": 1444 }445 }446 ]447 }448 ],449 "tags_descriptions": {},450 "fancy_title": "Expanding multidimensional tensor by non-singleton dimension",451 "id": 45850,452 "title": "Expanding multidimensional tensor by non-singleton dimension",453 "posts_count": 3,454 "created_at": "2019-05-21T21:31:01.082Z",455 "views": 1544,456 "reply_count": 1,457 "like_count": 1,458 "last_posted_at": "2019-05-22T10:37:05.849Z",459 "visible": true,460 "closed": false,461 "archived": false,462 "has_summary": false,463 "archetype": "regular",464 "slug": "expanding-multidimensional-tensor-by-non-singleton-dimension",465 "category_id": 1,466 "word_count": 308,467 "deleted_at": null,468 "user_id": 17358,469 "featured_link": null,470 "pinned_globally": false,471 "pinned_at": null,472 "pinned_until": null,473 "image_url": null,474 "slow_mode_seconds": 0,475 "draft": null,476 "draft_key": "topic_45850",477 "draft_sequence": null,478 "unpinned": null,479 "pinned": false,480 "current_post_number": 1,481 "highest_post_number": 3,482 "deleted_by": null,483 "actions_summary": [484 {485 "id": 4,486 "count": 0,487 "hidden": false,488 "can_act": false489 },490 {491 "id": 8,492 "count": 0,493 "hidden": false,494 "can_act": false495 },496 {497 "id": 10,498 "count": 0,499 "hidden": false,500 "can_act": false501 },502 {503 "id": 7,504 "count": 0,505 "hidden": false,506 "can_act": false507 }508 ],509 "chunk_size": 20,510 "bookmarked": false,511 "topic_timer": null,512 "message_bus_last_id": 0,513 "participant_count": 2,514 "show_read_indicator": false,515 "thumbnails": null,516 "slow_mode_enabled_until": null,517 "accepted_answer": {518 "post_number": 2,519 "username": "InnovArul",520 "name": "Arul",521 "excerpt": "To expand [200, 176, 2] mask to size [200, 176, 14], you can do the following: \nnew_mask = b.unsqueeze(-1).repeat(1, 1, 1, 7).view(200, 176, -1)\nprint( torch.all((new_mask.float() - mask.float()) == 0)) # 1"522 },523 "can_vote": false,524 "vote_count": 0,525 "user_voted": false,526 "discourse_zendesk_plugin_zendesk_id": null,527 "discourse_zendesk_plugin_zendesk_url": "https://your-url.zendesk.com/agent/tickets/",528 "details": {529 "can_edit": false,530 "notification_level": 1,531 "participants": [532 {533 "id": 17358,534 "username": "martinr",535 "name": "Martin",536 "avatar_template": "/user_avatar/discuss.pytorch.org/martinr/{size}/30426_2.png",537 "post_count": 2,538 "primary_group_name": null,539 "flair_name": null,540 "flair_url": null,541 "flair_color": null,542 "flair_bg_color": null,543 "flair_group_id": null,544 "trust_level": 2545 },546 {547 "id": 998,548 "username": "InnovArul",549 "name": "Arul",550 "avatar_template": "/user_avatar/discuss.pytorch.org/innovarul/{size}/5282_2.png",551 "post_count": 1,552 "primary_group_name": null,553 "flair_name": null,554 "flair_url": null,555 "flair_color": null,556 "flair_bg_color": null,557 "flair_group_id": null,558 "trust_level": 2559 }560 ],561 "created_by": {562 "id": 17358,563 "username": "martinr",564 "name": "Martin",565 "avatar_template": "/user_avatar/discuss.pytorch.org/martinr/{size}/30426_2.png"566 },567 "last_poster": {568 "id": 17358,569 "username": "martinr",570 "name": "Martin",571 "avatar_template": "/user_avatar/discuss.pytorch.org/martinr/{size}/30426_2.png"572 }573 },574 "bookmarks": []575 },576 {577 "post_stream": {578 "posts": [579 {580 "id": 38528,581 "name": "Wei Deng",582 "username": "Wei_Deng",583 "avatar_template": "/user_avatar/discuss.pytorch.org/wei_deng/{size}/3370_2.png",584 "created_at": "2018-03-18T02:39:40.349Z",585 "cooked": "<p>For example, I used his blog to try to get the 2nd derivative [Second order derivatives and inplace gradient “zeroing” ], but it turns out that the grd.grad information is None. Can anyone give me some suggestions?</p>\n<p>import torch<br>\nfrom torch import Tensor<br>\nfrom torch.autograd import Variable<br>\nfrom torch.autograd import grad<br>\nfrom torch import nn</p>\n<p># some toy data<br>\nx = Variable(Tensor([4., 2.]), requires_grad=False)<br>\ny = Variable(Tensor([1.]), requires_grad=False)</p>\n<p># linear model and squared difference loss<br>\nmodel = nn.Linear(2, 1)<br>\nloss = torch.sum((y - model(x))**2)</p>\n<p>optimizer = torch.optim.Adam(model.parameters(), lr=1e-2)</p>\n<p># instead of using loss.backward(), use torch.autograd.grad() to compute gradients<br>\nloss_grads = grad(loss, model.parameters(), create_graph=True)</p>\n<p>gn2 = sum([grd.norm()**2 for grd in loss_grads]) # 2nd derive<br>\nprint(‘loss %f grad norm %f’ % (loss.data, gn2.data))<br>\nmodel.zero_grad()<br>\ngn2.backward()<br>\noptimizer.step()</p>\n<p>for grd in loss_grads:<br>\nprint grd.grad</p>\n<p>The output is None.</p>\n<p>Can any one tell me how to get it?</p>",586 "post_number": 1,587 "post_type": 1,588 "posts_count": 10,589 "updated_at": "2018-03-18T02:39:40.349Z",590 "reply_count": 0,591 "reply_to_post_number": null,592 "quote_count": 0,593 "incoming_link_count": 1910,594 "reads": 132,595 "readers_count": 131,596 "score": 9578.4,597 "yours": false,598 "topic_id": 15093,599 "topic_slug": "how-to-calculate-the-2nd-derivative-of-the-diagonal-of-the-hessian-matrix-from-a-function",600 "display_username": "Wei Deng",601 "primary_group_name": null,602 "flair_name": null,603 "flair_url": null,604 "flair_bg_color": null,605 "flair_color": null,606 "flair_group_id": null,607 "badges_granted": [],608 "version": 1,609 "can_edit": false,610 "can_delete": false,611 "can_recover": false,612 "can_see_hidden_post": false,613 "can_wiki": false,614 "read": true,615 "user_title": null,616 "bookmarked": false,617 "actions_summary": [],618 "moderator": false,619 "admin": false,620 "staff": false,621 "user_id": 6140,622 "hidden": false,623 "trust_level": 1,624 "deleted_at": null,625 "user_deleted": false,626 "edit_reason": null,627 "can_view_edit_history": true,628 "wiki": false,629 "post_url": "/t/how-to-calculate-the-2nd-derivative-of-the-diagonal-of-the-hessian-matrix-from-a-function/15093/1",630 "can_accept_answer": false,631 "can_unaccept_answer": false,632 "accepted_answer": false,633 "topic_accepted_answer": null,634 "can_vote": false635 },636 {637 "id": 38539,638 "name": "Thomas V",639 "username": "tom",640 "avatar_template": "/user_avatar/discuss.pytorch.org/tom/{size}/3162_2.png",641 "created_at": "2018-03-18T03:38:33.831Z",642 "cooked": "<p>You can call <a href=\"http://pytorch.org/docs/0.3.1/autograd.html#torch.autograd.Variable.retain_grad\" rel=\"nofollow noopener\"><code>grd.retain_grad ()</code></a> before backward to keep the grad of a non-leaf variable.</p>\n<p>Best regards</p>\n<p>Thomas</p>",643 "post_number": 2,644 "post_type": 1,645 "posts_count": 10,646 "updated_at": "2018-03-18T03:38:33.831Z",647 "reply_count": 1,648 "reply_to_post_number": null,649 "quote_count": 0,650 "incoming_link_count": 4,651 "reads": 121,652 "readers_count": 120,653 "score": 49.2,654 "yours": false,655 "topic_id": 15093,656 "topic_slug": "how-to-calculate-the-2nd-derivative-of-the-diagonal-of-the-hessian-matrix-from-a-function",657 "display_username": "Thomas V",658 "primary_group_name": null,659 "flair_name": null,660 "flair_url": null,661 "flair_bg_color": null,662 "flair_color": null,663 "flair_group_id": null,664 "badges_granted": [],665 "version": 1,666 "can_edit": false,667 "can_delete": false,668 "can_recover": false,669 "can_see_hidden_post": false,670 "can_wiki": false,671 "link_counts": [672 {673 "url": "http://pytorch.org/docs/0.3.1/autograd.html#torch.autograd.Variable.retain_grad",674 "internal": false,675 "reflection": false,676 "title": "Automatic differentiation package - torch.autograd — PyTorch master documentation",677 "clicks": 40678 }679 ],680 "read": true,681 "user_title": null,682 "bookmarked": false,683 "actions_summary": [],684 "moderator": false,685 "admin": false,686 "staff": false,687 "user_id": 616,688 "hidden": false,689 "trust_level": 2,690 "deleted_at": null,691 "user_deleted": false,692 "edit_reason": null,693 "can_view_edit_history": true,694 "wiki": false,695 "post_url": "/t/how-to-calculate-the-2nd-derivative-of-the-diagonal-of-the-hessian-matrix-from-a-function/15093/2",696 "can_accept_answer": false,697 "can_unaccept_answer": false,698 "accepted_answer": false,699 "topic_accepted_answer": null700 },701 {702 "id": 38600,703 "name": "Wei Deng",704 "username": "Wei_Deng",705 "avatar_template": "/user_avatar/discuss.pytorch.org/wei_deng/{size}/3370_2.png",706 "created_at": "2018-03-18T17:20:04.543Z",707 "cooked": "<p>Thanks Tom, I got the grad, but it is not correct. Like the following example, i want to get the second derivative of (2x)^2 at x0=0.5153, the final result could return the 1st order derivative correctly which is 8*x0=4.12221, but for the second derivative, it is not the expected 8, do you know why?</p>\n<p>import torch<br>\nfrom torch import Tensor<br>\nfrom torch.autograd import Variable<br>\nfrom torch.autograd import grad<br>\nfrom torch import nn</p>\n<p>torch.manual_seed(1)<br>\nx = Variable(Tensor([2.]), requires_grad=False)</p>\n<p>model = nn.Linear(1, 1, bias=False)</p>\n<p>x0 = [par.data for par in model.parameters()][0]<br>\nprint(x0)</p>\n<p>loss = torch.sum(model(x)**2)<br>\noptimizer = torch.optim.Adam(model.parameters(), lr=1e-3)<br>\nloss_grads = grad(loss, model.parameters(), create_graph=True)<br>\ngn2 = sum([grd.norm()**2 for grd in loss_grads]) / 2 # 2nd derive<br>\nprint(‘loss %f grad norm %f’ % (loss.data, gn2.data))</p>\n<p>for grd in loss_grads:<br>\ngrd = grd.retain_grad()</p>\n<p>model.zero_grad()<br>\ngn2.backward(retain_graph=True)</p>\n<p>for grd in loss_grads:<br>\nprint 8 * x0, grd.data[0], grd.grad</p>",708 "post_number": 3,709 "post_type": 1,710 "posts_count": 10,711 "updated_at": "2018-03-18T17:26:57.891Z",712 "reply_count": 1,713 "reply_to_post_number": 2,714 "quote_count": 0,715 "incoming_link_count": 4,716 "reads": 119,717 "readers_count": 118,718 "score": 48.8,719 "yours": false,720 "topic_id": 15093,721 "topic_slug": "how-to-calculate-the-2nd-derivative-of-the-diagonal-of-the-hessian-matrix-from-a-function",722 "display_username": "Wei Deng",723 "primary_group_name": null,724 "flair_name": null,725 "flair_url": null,726 "flair_bg_color": null,727 "flair_color": null,728 "flair_group_id": null,729 "badges_granted": [],730 "version": 2,731 "can_edit": false,732 "can_delete": false,733 "can_recover": false,734 "can_see_hidden_post": false,735 "can_wiki": false,736 "read": true,737 "user_title": null,738 "reply_to_user": {739 "id": 616,740 "username": "tom",741 "name": "Thomas V",742 "avatar_template": "/user_avatar/discuss.pytorch.org/tom/{size}/3162_2.png"743 },744 "bookmarked": false,745 "actions_summary": [],746 "moderator": false,747 "admin": false,748 "staff": false,749 "user_id": 6140,750 "hidden": false,751 "trust_level": 1,752 "deleted_at": null,753 "user_deleted": false,754 "edit_reason": null,755 "can_view_edit_history": true,756 "wiki": false,757 "post_url": "/t/how-to-calculate-the-2nd-derivative-of-the-diagonal-of-the-hessian-matrix-from-a-function/15093/3",758 "can_accept_answer": false,759 "can_unaccept_answer": false,760 "accepted_answer": false,761 "topic_accepted_answer": null762 },763 {764 "id": 38616,765 "name": "Thomas V",766 "username": "tom",767 "avatar_template": "/user_avatar/discuss.pytorch.org/tom/{size}/3162_2.png",768 "created_at": "2018-03-18T21:03:08.122Z",769 "cooked": "<p>This calculated d gn2 / d grd = d (0.5 grd^2) / d grd = grd correctly, but maybe you want something else?</p>\n<p>Best regards</p>\n<p>Thomas</p>",770 "post_number": 4,771 "post_type": 1,772 "posts_count": 10,773 "updated_at": "2018-03-18T21:03:08.122Z",774 "reply_count": 1,775 "reply_to_post_number": 3,776 "quote_count": 0,777 "incoming_link_count": 3,778 "reads": 99,779 "readers_count": 98,780 "score": 39.8,781 "yours": false,782 "topic_id": 15093,783 "topic_slug": "how-to-calculate-the-2nd-derivative-of-the-diagonal-of-the-hessian-matrix-from-a-function",784 "display_username": "Thomas V",785 "primary_group_name": null,786 "flair_name": null,787 "flair_url": null,788 "flair_bg_color": null,789 "flair_color": null,790 "flair_group_id": null,791 "badges_granted": [],792 "version": 1,793 "can_edit": false,794 "can_delete": false,795 "can_recover": false,796 "can_see_hidden_post": false,797 "can_wiki": false,798 "read": true,799 "user_title": null,800 "reply_to_user": {801 "id": 6140,802 "username": "Wei_Deng",803 "name": "Wei Deng",804 "avatar_template": "/user_avatar/discuss.pytorch.org/wei_deng/{size}/3370_2.png"805 },806 "bookmarked": false,807 "actions_summary": [],808 "moderator": false,809 "admin": false,810 "staff": false,811 "user_id": 616,812 "hidden": false,813 "trust_level": 2,814 "deleted_at": null,815 "user_deleted": false,816 "edit_reason": null,817 "can_view_edit_history": true,818 "wiki": false,819 "post_url": "/t/how-to-calculate-the-2nd-derivative-of-the-diagonal-of-the-hessian-matrix-from-a-function/15093/4",820 "can_accept_answer": false,821 "can_unaccept_answer": false,822 "accepted_answer": false,823 "topic_accepted_answer": null824 },825 {826 "id": 38624,827 "name": "Wei Deng",828 "username": "Wei_Deng",829 "avatar_template": "/user_avatar/discuss.pytorch.org/wei_deng/{size}/3370_2.png",830 "created_at": "2018-03-19T00:56:08.164Z",831 "cooked": "<p>Gotcha, that’s why the answers are the same, Thank you so much. Do you know how to calculate the second derivative of (x1)^2 + (2*x2)^2 with respect to x1 and x2, which should be (2, 8)?</p>\n<p>Really appreciate your suggestions. Thanks a lot.</p>",832 "post_number": 5,833 "post_type": 1,834 "posts_count": 10,835 "updated_at": "2018-03-19T00:56:08.164Z",836 "reply_count": 1,837 "reply_to_post_number": 4,838 "quote_count": 0,839 "incoming_link_count": 5,840 "reads": 93,841 "readers_count": 92,842 "score": 48.6,843 "yours": false,844 "topic_id": 15093,845 "topic_slug": "how-to-calculate-the-2nd-derivative-of-the-diagonal-of-the-hessian-matrix-from-a-function",846 "display_username": "Wei Deng",847 "primary_group_name": null,848 "flair_name": null,849 "flair_url": null,850 "flair_bg_color": null,851 "flair_color": null,852 "flair_group_id": null,853 "badges_granted": [],854 "version": 1,855 "can_edit": false,856 "can_delete": false,857 "can_recover": false,858 "can_see_hidden_post": false,859 "can_wiki": false,860 "read": true,861 "user_title": null,862 "reply_to_user": {863 "id": 616,864 "username": "tom",865 "name": "Thomas V",866 "avatar_template": "/user_avatar/discuss.pytorch.org/tom/{size}/3162_2.png"867 },868 "bookmarked": false,869 "actions_summary": [],870 "moderator": false,871 "admin": false,872 "staff": false,873 "user_id": 6140,874 "hidden": false,875 "trust_level": 1,876 "deleted_at": null,877 "user_deleted": false,878 "edit_reason": null,879 "can_view_edit_history": true,880 "wiki": false,881 "post_url": "/t/how-to-calculate-the-2nd-derivative-of-the-diagonal-of-the-hessian-matrix-from-a-function/15093/5",882 "can_accept_answer": false,883 "can_unaccept_answer": false,884 "accepted_answer": false,885 "topic_accepted_answer": null886 },887 {888 "id": 38639,889 "name": "Thomas V",890 "username": "tom",891 "avatar_template": "/user_avatar/discuss.pytorch.org/tom/{size}/3162_2.png",892 "created_at": "2018-03-19T07:23:48.788Z",893 "cooked": "<p>I must admit that I’m confused about how the linear layer fits into what you want to achieve.<br>\nIf you drop the nn.Linear and start with <code>x</code> as requires_grad = True, you get the 2nd derivative in x.grad…</p>",894 "post_number": 6,895 "post_type": 1,896 "posts_count": 10,897 "updated_at": "2018-03-19T07:23:48.788Z",898 "reply_count": 1,899 "reply_to_post_number": 5,900 "quote_count": 0,901 "incoming_link_count": 3,902 "reads": 91,903 "readers_count": 90,904 "score": 38.2,905 "yours": false,906 "topic_id": 15093,907 "topic_slug": "how-to-calculate-the-2nd-derivative-of-the-diagonal-of-the-hessian-matrix-from-a-function",908 "display_username": "Thomas V",909 "primary_group_name": null,910 "flair_name": null,911 "flair_url": null,912 "flair_bg_color": null,913 "flair_color": null,914 "flair_group_id": null,915 "badges_granted": [],916 "version": 1,917 "can_edit": false,918 "can_delete": false,919 "can_recover": false,920 "can_see_hidden_post": false,921 "can_wiki": false,922 "read": true,923 "user_title": null,924 "reply_to_user": {925 "id": 6140,926 "username": "Wei_Deng",927 "name": "Wei Deng",928 "avatar_template": "/user_avatar/discuss.pytorch.org/wei_deng/{size}/3370_2.png"929 },930 "bookmarked": false,931 "actions_summary": [],932 "moderator": false,933 "admin": false,934 "staff": false,935 "user_id": 616,936 "hidden": false,937 "trust_level": 2,938 "deleted_at": null,939 "user_deleted": false,940 "edit_reason": null,941 "can_view_edit_history": true,942 "wiki": false,943 "post_url": "/t/how-to-calculate-the-2nd-derivative-of-the-diagonal-of-the-hessian-matrix-from-a-function/15093/6",944 "can_accept_answer": false,945 "can_unaccept_answer": false,946 "accepted_answer": false,947 "topic_accepted_answer": null948 },949 {950 "id": 38708,951 "name": "Wei Deng",952 "username": "Wei_Deng",953 "avatar_template": "/user_avatar/discuss.pytorch.org/wei_deng/{size}/3370_2.png",954 "created_at": "2018-03-19T17:45:15.760Z",955 "cooked": "<p>Hi, Tom, sorry for not explaining explicitly on my question.</p>\n<p>My ultimate question is that if I got a neural network loss function, can I get the 2nd derivative of the likelihood function with respect to every weight? It doesn’t have to be a hessian matrix, but just the diagonal of it.</p>\n<p>Do you know if we can do it based on the current version?</p>",956 "post_number": 7,957 "post_type": 1,958 "posts_count": 10,959 "updated_at": "2018-03-19T17:45:15.760Z",960 "reply_count": 1,961 "reply_to_post_number": 6,962 "quote_count": 0,963 "incoming_link_count": 6,964 "reads": 90,965 "readers_count": 89,966 "score": 53.0,967 "yours": false,968 "topic_id": 15093,969 "topic_slug": "how-to-calculate-the-2nd-derivative-of-the-diagonal-of-the-hessian-matrix-from-a-function",970 "display_username": "Wei Deng",971 "primary_group_name": null,972 "flair_name": null,973 "flair_url": null,974 "flair_bg_color": null,975 "flair_color": null,976 "flair_group_id": null,977 "badges_granted": [],978 "version": 1,979 "can_edit": false,980 "can_delete": false,981 "can_recover": false,982 "can_see_hidden_post": false,983 "can_wiki": false,984 "read": true,985 "user_title": null,986 "reply_to_user": {987 "id": 616,988 "username": "tom",989 "name": "Thomas V",990 "avatar_template": "/user_avatar/discuss.pytorch.org/tom/{size}/3162_2.png"991 },992 "bookmarked": false,993 "actions_summary": [],994 "moderator": false,995 "admin": false,996 "staff": false,997 "user_id": 6140,998 "hidden": false,999 "trust_level": 1,1000 "deleted_at": null,1001 "user_deleted": false,1002 "edit_reason": null,1003 "can_view_edit_history": true,1004 "wiki": false,1005 "post_url": "/t/how-to-calculate-the-2nd-derivative-of-the-diagonal-of-the-hessian-matrix-from-a-function/15093/7",1006 "can_accept_answer": false,1007 "can_unaccept_answer": false,1008 "accepted_answer": false,1009 "topic_accepted_answer": null1010 },1011 {1012 "id": 38713,1013 "name": "Thomas V",1014 "username": "tom",1015 "avatar_template": "/user_avatar/discuss.pytorch.org/tom/{size}/3162_2.png",1016 "created_at": "2018-03-19T18:23:59.022Z",1017 "cooked": "<p>I don’t think that this is currently possible in geberal (unless iterating over the scalar parameters). The fundamental reason is that backpropagation in PyTorch isn’t prepared take deruvatives of vector-valued functions, so you are limited to taking the derivative of a scalar sum of derivatives, i.o.w. a Hessian-Vector product.<br>\nFor a small number of parameters using torch.autograd.grad will help, but I’m not sure it scales to all parameters of large nets.</p>\n<p>Best regards</p>\n<p>Thomas</p>",1018 "post_number": 8,1019 "post_type": 1,1020 "posts_count": 10,1021 "updated_at": "2018-03-19T18:23:59.022Z",1022 "reply_count": 2,1023 "reply_to_post_number": 7,1024 "quote_count": 0,1025 "incoming_link_count": 9,1026 "reads": 82,1027 "readers_count": 81,1028 "score": 86.4,1029 "yours": false,1030 "topic_id": 15093,1031 "topic_slug": "how-to-calculate-the-2nd-derivative-of-the-diagonal-of-the-hessian-matrix-from-a-function",1032 "display_username": "Thomas V",1033 "primary_group_name": null,1034 "flair_name": null,1035 "flair_url": null,1036 "flair_bg_color": null,1037 "flair_color": null,1038 "flair_group_id": null,1039 "badges_granted": [],1040 "version": 1,1041 "can_edit": false,1042 "can_delete": false,1043 "can_recover": false,1044 "can_see_hidden_post": false,1045 "can_wiki": false,1046 "read": true,1047 "user_title": null,1048 "reply_to_user": {1049 "id": 6140,1050 "username": "Wei_Deng",1051 "name": "Wei Deng",1052 "avatar_template": "/user_avatar/discuss.pytorch.org/wei_deng/{size}/3370_2.png"1053 },1054 "bookmarked": false,1055 "actions_summary": [1056 {1057 "id": 2,1058 "count": 11059 }1060 ],1061 "moderator": false,1062 "admin": false,1063 "staff": false,1064 "user_id": 616,1065 "hidden": false,1066 "trust_level": 2,1067 "deleted_at": null,1068 "user_deleted": false,1069 "edit_reason": null,1070 "can_view_edit_history": true,1071 "wiki": false,1072 "post_url": "/t/how-to-calculate-the-2nd-derivative-of-the-diagonal-of-the-hessian-matrix-from-a-function/15093/8",1073 "can_accept_answer": false,1074 "can_unaccept_answer": false,1075 "accepted_answer": false,1076 "topic_accepted_answer": null1077 },1078 {1079 "id": 38767,1080 "name": "Wei Deng",1081 "username": "Wei_Deng",1082 "avatar_template": "/user_avatar/discuss.pytorch.org/wei_deng/{size}/3370_2.png",1083 "created_at": "2018-03-20T00:59:11.873Z",1084 "cooked": "<p>Got it, thanks a lot.</p>",1085 "post_number": 9,1086 "post_type": 1,1087 "posts_count": 10,1088 "updated_at": "2018-03-20T00:59:11.873Z",1089 "reply_count": 0,1090 "reply_to_post_number": 8,1091 "quote_count": 0,1092 "incoming_link_count": 18,1093 "reads": 76,1094 "readers_count": 75,1095 "score": 105.2,1096 "yours": false,1097 "topic_id": 15093,1098 "topic_slug": "how-to-calculate-the-2nd-derivative-of-the-diagonal-of-the-hessian-matrix-from-a-function",1099 "display_username": "Wei Deng",1100 "primary_group_name": null,1101 "flair_name": null,1102 "flair_url": null,1103 "flair_bg_color": null,1104 "flair_color": null,1105 "flair_group_id": null,1106 "badges_granted": [],1107 "version": 1,1108 "can_edit": false,1109 "can_delete": false,1110 "can_recover": false,1111 "can_see_hidden_post": false,1112 "can_wiki": false,1113 "read": true,1114 "user_title": null,1115 "reply_to_user": {1116 "id": 616,1117 "username": "tom",1118 "name": "Thomas V",1119 "avatar_template": "/user_avatar/discuss.pytorch.org/tom/{size}/3162_2.png"1120 },1121 "bookmarked": false,1122 "actions_summary": [],1123 "moderator": false,1124 "admin": false,1125 "staff": false,1126 "user_id": 6140,1127 "hidden": false,1128 "trust_level": 1,1129 "deleted_at": null,1130 "user_deleted": false,1131 "edit_reason": null,1132 "can_view_edit_history": true,1133 "wiki": false,1134 "post_url": "/t/how-to-calculate-the-2nd-derivative-of-the-diagonal-of-the-hessian-matrix-from-a-function/15093/9",1135 "can_accept_answer": false,1136 "can_unaccept_answer": false,1137 "accepted_answer": false,1138 "topic_accepted_answer": null1139 },1140 {1141 "id": 112777,1142 "name": "Nima Rafiee",1143 "username": "nima_rafiee",1144 "avatar_template": "/user_avatar/discuss.pytorch.org/nima_rafiee/{size}/16289_2.png",1145 "created_at": "2019-05-22T08:51:54.285Z",1146 "cooked": "<p>is there a way to get the full Hessian matrix with w.r.s to the input. calling the backward() function two times only provides me with a diagonal of Hessian matrix but not the full one. I need something like<br>\ntf.Hessian().</p>",1147 "post_number": 10,1148 "post_type": 1,1149 "posts_count": 10,1150 "updated_at": "2019-05-22T08:51:54.285Z",1151 "reply_count": 0,1152 "reply_to_post_number": 8,1153 "quote_count": 0,1154 "incoming_link_count": 4,1155 "reads": 44,1156 "readers_count": 43,1157 "score": 28.8,1158 "yours": false,1159 "topic_id": 15093,1160 "topic_slug": "how-to-calculate-the-2nd-derivative-of-the-diagonal-of-the-hessian-matrix-from-a-function",1161 "display_username": "Nima Rafiee",1162 "primary_group_name": null,1163 "flair_name": null,1164 "flair_url": null,1165 "flair_bg_color": null,1166 "flair_color": null,1167 "flair_group_id": null,1168 "badges_granted": [],1169 "version": 1,1170 "can_edit": false,1171 "can_delete": false,1172 "can_recover": false,1173 "can_see_hidden_post": false,1174 "can_wiki": false,1175 "read": true,1176 "user_title": null,1177 "reply_to_user": {1178 "id": 616,1179 "username": "tom",1180 "name": "Thomas V",1181 "avatar_template": "/user_avatar/discuss.pytorch.org/tom/{size}/3162_2.png"1182 },1183 "bookmarked": false,1184 "actions_summary": [],1185 "moderator": false,1186 "admin": false,1187 "staff": false,1188 "user_id": 12679,1189 "hidden": false,1190 "trust_level": 2,1191 "deleted_at": null,1192 "user_deleted": false,1193 "edit_reason": null,1194 "can_view_edit_history": true,1195 "wiki": false,1196 "post_url": "/t/how-to-calculate-the-2nd-derivative-of-the-diagonal-of-the-hessian-matrix-from-a-function/15093/10",1197 "can_accept_answer": false,1198 "can_unaccept_answer": false,1199 "accepted_answer": false,1200 "topic_accepted_answer": null