Anurag1734/cuda-error-resolution-analysis
07
1[2 {3 "post_stream": {4 "posts": [5 {6 "id": 345029,7 "name": "razzle dazzle",8 "username": "4gatepylon",9 "avatar_template": "/user_avatar/discuss.pytorch.org/4gatepylon/{size}/48373_2.png",10 "created_at": "2022-05-06T19:30:34.610Z",11 "cooked": "<p>I have a sanity test for my model when I need to use it for evaluation. I do not understand why the accuracy changes every time I evaluate. It seems to change on some sort of period if I just keep evaluating, so I’m guessing the shuffling order will change periodically in some way (maybe the index the Dataloader is starting at changes or something like that).</p>\n<p>My models are ResNet10 on CIFAR-10.</p>\n<p>I’ve tried converting the tensors to doubles in the evaluate function (as shown below) to fix numerical issues but to no avail.</p>\n<p>I’m fixing all other sources of randomness I know. <strong>Why would this happen? Why would shuffling the data change the accuracy like that?</strong></p>\n<pre><code class=\"lang-auto\">def fix_seed(seed):\n random.seed(seed)\n np.random.seed(seed)\n torch.manual_seed(seed)\n torch.cuda.manual_seed(seed)\n torch.backends.cudnn.deterministic = True\n torch.backends.cudnn.benchmark = False\n</code></pre>\n<pre><code class=\"lang-auto\"> acc = evaluate(model, test_loader)\n print(\"Model 1 Original Accuracy: {}\".format(acc))\n assert acc == evaluate(model, test_loader)\n</code></pre>\n<p>and</p>\n<pre><code class=\"lang-auto\">def evaluate(model, test_loader):\n model.eval()\n\n # TODO, why would the total_correct / total_num change as we shuffled the data differently?\n for _, (images, labels) in enumerate(test_loader):\n total_correct, total_num = 0., 0.\n\n with torch.no_grad():\n labels = labels.cuda().double()\n img = images.cuda()\n h = model(img)\n preds = h.argmax(dim=1).double()\n total_correct = (preds == labels).sum().cpu().item()\n total_num += h.shape[0]\n\n return total_correct / total_num\n</code></pre>",12 "post_number": 1,13 "post_type": 1,14 "posts_count": 4,15 "updated_at": "2022-05-06T19:55:18.444Z",16 "reply_count": 1,17 "reply_to_post_number": null,18 "quote_count": 0,19 "incoming_link_count": 112,20 "reads": 6,21 "readers_count": 5,22 "score": 566.2,23 "yours": false,24 "topic_id": 150976,25 "topic_slug": "evaluate-twice-accuracy-changes-if-i-shuffle",26 "display_username": "razzle dazzle",27 "primary_group_name": null,28 "flair_name": null,29 "flair_url": null,30 "flair_bg_color": null,31 "flair_color": null,32 "flair_group_id": null,33 "badges_granted": [],34 "version": 3,35 "can_edit": false,36 "can_delete": false,37 "can_recover": false,38 "can_see_hidden_post": false,39 "can_wiki": false,40 "read": true,41 "user_title": null,42 "bookmarked": false,43 "actions_summary": [],44 "moderator": false,45 "admin": false,46 "staff": false,47 "user_id": 54777,48 "hidden": false,49 "trust_level": 1,50 "deleted_at": null,51 "user_deleted": false,52 "edit_reason": null,53 "can_view_edit_history": true,54 "wiki": false,55 "post_url": "/t/evaluate-twice-accuracy-changes-if-i-shuffle/150976/1",56 "can_accept_answer": false,57 "can_unaccept_answer": false,58 "accepted_answer": false,59 "topic_accepted_answer": null,60 "can_vote": false61 },62 {63 "id": 345030,64 "name": "Andrei Cristea",65 "username": "Andrei_Cristea",66 "avatar_template": "/user_avatar/discuss.pytorch.org/andrei_cristea/{size}/47843_2.png",67 "created_at": "2022-05-06T19:39:45.675Z",68 "cooked": "<p>Hello! Just a quick note / question. Below:</p>\n<aside class=\"quote no-group quote-modified\" data-username=\"4gatepylon\" data-post=\"1\" data-topic=\"150976\">\n<div class=\"title\">\n<div class=\"quote-controls\"></div>\n<img loading=\"lazy\" alt=\"\" width=\"24\" height=\"24\" src=\"https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/4gatepylon/48/48373_2.png\" class=\"avatar\"> 4gatepylon:</div>\n<blockquote>\n<pre><code class=\"lang-auto\"> acc2 = evaluate(model, test_loader)\n print(\"Model 1 Original Accuracy: {}\".format(acc))\n assert acc == evaluate(model, test_loader)\n</code></pre>\n</blockquote>\n</aside>\n<p>How do you define <code>acc</code>? This snippet shows where you define <code>acc2</code> but not <code>acc</code>. And you’re comparing <code>acc</code> rather than <code>acc2</code> with the re-evaluated model, so just wanted to confirm that this isn’t somehow causing the issue.</p>\n<p>Also, antother quick idea is: have you tried also adding this to your list of deterministic instructions:<br>\n<code>torch.use_deterministic_algorithms(True)</code></p>\n<p>Per the documentation, that does more than just <code>torch.backends.cudnn.deterministic = True</code> even though they sound similar.</p>",69 "post_number": 2,70 "post_type": 1,71 "posts_count": 4,72 "updated_at": "2022-05-06T19:39:45.675Z",73 "reply_count": 1,74 "reply_to_post_number": null,75 "quote_count": 1,76 "incoming_link_count": 2,77 "reads": 6,78 "readers_count": 5,79 "score": 16.2,80 "yours": false,81 "topic_id": 150976,82 "topic_slug": "evaluate-twice-accuracy-changes-if-i-shuffle",83 "display_username": "Andrei Cristea",84 "primary_group_name": null,85 "flair_name": null,86 "flair_url": null,87 "flair_bg_color": null,88 "flair_color": null,89 "flair_group_id": null,90 "badges_granted": [],91 "version": 1,92 "can_edit": false,93 "can_delete": false,94 "can_recover": false,95 "can_see_hidden_post": false,96 "can_wiki": false,97 "read": true,98 "user_title": null,99 "bookmarked": false,100 "actions_summary": [],101 "moderator": false,102 "admin": false,103 "staff": false,104 "user_id": 54308,105 "hidden": false,106 "trust_level": 2,107 "deleted_at": null,108 "user_deleted": false,109 "edit_reason": null,110 "can_view_edit_history": true,111 "wiki": false,112 "post_url": "/t/evaluate-twice-accuracy-changes-if-i-shuffle/150976/2",113 "can_accept_answer": false,114 "can_unaccept_answer": false,115 "accepted_answer": false,116 "topic_accepted_answer": null117 },118 {119 "id": 345031,120 "name": "razzle dazzle",121 "username": "4gatepylon",122 "avatar_template": "/user_avatar/discuss.pytorch.org/4gatepylon/{size}/48373_2.png",123 "created_at": "2022-05-06T19:56:01.782Z",124 "cooked": "<p>Ok thanks for the tip with the deterministic algorithms! Regarding acc, I actually have two models so I modified the snippet but made a mistake. They are all <code>acc</code>.</p>",125 "post_number": 3,126 "post_type": 1,127 "posts_count": 4,128 "updated_at": "2022-05-06T19:56:01.782Z",129 "reply_count": 0,130 "reply_to_post_number": 2,131 "quote_count": 0,132 "incoming_link_count": 0,133 "reads": 6,134 "readers_count": 5,135 "score": 1.2,136 "yours": false,137 "topic_id": 150976,138 "topic_slug": "evaluate-twice-accuracy-changes-if-i-shuffle",139 "display_username": "razzle dazzle",140 "primary_group_name": null,141 "flair_name": null,142 "flair_url": null,143 "flair_bg_color": null,144 "flair_color": null,145 "flair_group_id": null,146 "badges_granted": [],147 "version": 1,148 "can_edit": false,149 "can_delete": false,150 "can_recover": false,151 "can_see_hidden_post": false,152 "can_wiki": false,153 "read": true,154 "user_title": null,155 "reply_to_user": {156 "id": 54308,157 "username": "Andrei_Cristea",158 "name": "Andrei Cristea",159 "avatar_template": "/user_avatar/discuss.pytorch.org/andrei_cristea/{size}/47843_2.png"160 },161 "bookmarked": false,162 "actions_summary": [],163 "moderator": false,164 "admin": false,165 "staff": false,166 "user_id": 54777,167 "hidden": false,168 "trust_level": 1,169 "deleted_at": null,170 "user_deleted": false,171 "edit_reason": null,172 "can_view_edit_history": true,173 "wiki": false,174 "post_url": "/t/evaluate-twice-accuracy-changes-if-i-shuffle/150976/3",175 "can_accept_answer": false,176 "can_unaccept_answer": false,177 "accepted_answer": false,178 "topic_accepted_answer": null179 },180 {181 "id": 345032,182 "name": "razzle dazzle",183 "username": "4gatepylon",184 "avatar_template": "/user_avatar/discuss.pytorch.org/4gatepylon/{size}/48373_2.png",185 "created_at": "2022-05-06T20:09:36.364Z",186 "cooked": "<p>Resolved! If you look at the code inside evaluate you see that the total_number and total_correct are initialized in the wrong location. This means that every run it returns the value for a different batch (which is probably why it’s cyclic: it starts at different batches in some sort of cycle). The accuracies we were seeing were inside the batch.</p>",187 "post_number": 4,188 "post_type": 1,189 "posts_count": 4,190 "updated_at": "2022-05-06T20:09:36.364Z",191 "reply_count": 0,192 "reply_to_post_number": null,193 "quote_count": 0,194 "incoming_link_count": 1,195 "reads": 6,196 "readers_count": 5,197 "score": 6.2,198 "yours": false,199 "topic_id": 150976,200 "topic_slug": "evaluate-twice-accuracy-changes-if-i-shuffle",201 "display_username": "razzle dazzle",202 "primary_group_name": null,203 "flair_name": null,204 "flair_url": null,205 "flair_bg_color": null,206 "flair_color": null,207 "flair_group_id": null,208 "badges_granted": [],209 "version": 1,210 "can_edit": false,211 "can_delete": false,212 "can_recover": false,213 "can_see_hidden_post": false,214 "can_wiki": false,215 "read": true,216 "user_title": null,217 "bookmarked": false,218 "actions_summary": [],219 "moderator": false,220 "admin": false,221 "staff": false,222 "user_id": 54777,223 "hidden": false,224 "trust_level": 1,225 "deleted_at": null,226 "user_deleted": false,227 "edit_reason": null,228 "can_view_edit_history": true,229 "wiki": false,230 "post_url": "/t/evaluate-twice-accuracy-changes-if-i-shuffle/150976/4",231 "can_accept_answer": false,232 "can_unaccept_answer": false,233 "accepted_answer": false,234 "topic_accepted_answer": null235 }236 ],237 "stream": [238 345029,239 345030,240 345031,241 345032242 ]243 },244 "timeline_lookup": [245 [246 1,247 1268248 ]249 ],250 "suggested_topics": [251 {252 "fancy_title": "Cant’t find amdgpu.ids when running in venv",253 "id": 214980,254 "title": "Cant't find amdgpu.ids when running in venv",255 "slug": "cantt-find-amdgpu-ids-when-running-in-venv",256 "posts_count": 2,257 "reply_count": 0,258 "highest_post_number": 2,259 "image_url": null,260 "created_at": "2025-01-04T22:49:45.132Z",261 "last_posted_at": "2025-02-26T19:30:12.859Z",262 "bumped": true,263 "bumped_at": "2025-02-26T19:30:12.859Z",264 "archetype": "regular",265 "unseen": false,266 "pinned": false,267 "unpinned": null,268 "visible": true,269 "closed": false,270 "archived": false,271 "bookmarked": null,272 "liked": null,273 "tags_descriptions": {},274 "like_count": 1,275 "views": 514,276 "category_id": 1,277 "featured_link": null,278 "has_accepted_answer": false,279 "posters": [280 {281 "extras": null,282 "description": "Original Poster",283 "user": {284 "id": 81881,285 "username": "Outssiss",286 "name": "Outssiss",287 "avatar_template": "/user_avatar/discuss.pytorch.org/outssiss/{size}/74905_2.png",288 "trust_level": 0289 }290 },291 {292 "extras": "latest",293 "description": "Most Recent Poster",294 "user": {295 "id": 82743,296 "username": "fngarrett",297 "name": "Garrett",298 "avatar_template": "/letter_avatar_proxy/v4/letter/f/e79b87/{size}.png",299 "trust_level": 1300 }301 }302 ]303 },304 {305 "fancy_title": "FlexAttention customizability",306 "id": 216077,307 "title": "FlexAttention customizability",308 "slug": "flexattention-customizability",309 "posts_count": 1,310 "reply_count": 0,311 "highest_post_number": 1,312 "image_url": null,313 "created_at": "2025-01-31T03:29:15.775Z",314 "last_posted_at": "2025-01-31T03:29:15.819Z",315 "bumped": true,316 "bumped_at": "2025-01-31T03:29:15.819Z",317 "archetype": "regular",318 "unseen": false,319 "pinned": false,320 "unpinned": null,321 "visible": true,322 "closed": false,323 "archived": false,324 "bookmarked": null,325 "liked": null,326 "tags_descriptions": {},327 "like_count": 0,328 "views": 104,329 "category_id": 1,330 "featured_link": null,331 "has_accepted_answer": false,332 "posters": [333 {334 "extras": "latest single",335 "description": "Original Poster, Most Recent Poster",336 "user": {337 "id": 10472,338 "username": "veritas",339 "name": "",340 "avatar_template": "/user_avatar/discuss.pytorch.org/veritas/{size}/11018_2.png",341 "trust_level": 2342 }343 }344 ]345 },346 {347 "fancy_title": "[Pytorch1.12] RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation",348 "id": 216528,349 "title": "[Pytorch1.12] RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation",350 "slug": "pytorch1-12-runtimeerror-one-of-the-variables-needed-for-gradient-computation-has-been-modified-by-an-inplace-operation",351 "posts_count": 1,352 "reply_count": 0,353 "highest_post_number": 1,354 "image_url": null,355 "created_at": "2025-02-11T13:00:16.487Z",356 "last_posted_at": "2025-02-11T13:00:16.523Z",357 "bumped": true,358 "bumped_at": "2025-02-11T13:00:16.523Z",359 "archetype": "regular",360 "unseen": false,361 "pinned": false,362 "unpinned": null,363 "visible": true,364 "closed": false,365 "archived": false,366 "bookmarked": null,367 "liked": null,368 "tags_descriptions": {},369 "like_count": 0,370 "views": 21,371 "category_id": 1,372 "featured_link": null,373 "has_accepted_answer": false,374 "posters": [375 {376 "extras": "latest single",377 "description": "Original Poster, Most Recent Poster",378 "user": {379 "id": 82600,380 "username": "briwa",381 "name": "北千wa",382 "avatar_template": "/user_avatar/discuss.pytorch.org/briwa/{size}/75576_2.png",383 "trust_level": 1384 }385 }386 ]387 },388 {389 "fancy_title": "Torch.linalg.lstsq: (Batch element 0): Argument 6 has illegal value",390 "id": 216565,391 "title": "Torch.linalg.lstsq: (Batch element 0): Argument 6 has illegal value",392 "slug": "torch-linalg-lstsq-batch-element-0-argument-6-has-illegal-value",393 "posts_count": 6,394 "reply_count": 4,395 "highest_post_number": 6,396 "image_url": null,397 "created_at": "2025-02-12T07:27:24.277Z",398 "last_posted_at": "2025-02-19T02:48:46.645Z",399 "bumped": true,400 "bumped_at": "2025-02-19T02:48:46.645Z",401 "archetype": "regular",402 "unseen": false,403 "pinned": false,404 "unpinned": null,405 "visible": true,406 "closed": false,407 "archived": false,408 "bookmarked": null,409 "liked": null,410 "tags_descriptions": {},411 "like_count": 0,412 "views": 118,413 "category_id": 1,414 "featured_link": null,415 "has_accepted_answer": false,416 "posters": [417 {418 "extras": null,419 "description": "Original Poster",420 "user": {421 "id": 82561,422 "username": "CarlosD1119",423 "name": "Deng Yibin",424 "avatar_template": "/user_avatar/discuss.pytorch.org/carlosd1119/{size}/75541_2.png",425 "trust_level": 0426 }427 },428 {429 "extras": "latest",430 "description": "Most Recent Poster",431 "user": {432 "id": 3534,433 "username": "ptrblck",434 "name": "",435 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",436 "admin": true,437 "moderator": true,438 "trust_level": 2439 }440 }441 ]442 },443 {444 "fancy_title": "Errors when deploying PyTorch Lightning Model to AWS SageMaker TrainingJobs: SMDDP does not support ReduceOp",445 "id": 212720,446 "title": "Errors when deploying PyTorch Lightning Model to AWS SageMaker TrainingJobs: SMDDP does not support ReduceOp",447 "slug": "errors-when-deploying-pytorch-lightning-model-to-aws-sagemaker-trainingjobs-smddp-does-not-support-reduceop",448 "posts_count": 1,449 "reply_count": 0,450 "highest_post_number": 1,451 "image_url": null,452 "created_at": "2024-11-09T01:19:50.522Z",453 "last_posted_at": "2024-11-09T01:19:50.587Z",454 "bumped": true,455 "bumped_at": "2024-11-09T01:19:50.587Z",456 "archetype": "regular",457 "unseen": false,458 "pinned": false,459 "unpinned": null,460 "visible": true,461 "closed": false,462 "archived": false,463 "bookmarked": null,464 "liked": null,465 "tags_descriptions": {},466 "like_count": 0,467 "views": 68,468 "category_id": 1,469 "featured_link": null,470 "has_accepted_answer": false,471 "posters": [472 {473 "extras": "latest single",474 "description": "Original Poster, Most Recent Poster",475 "user": {476 "id": 32574,477 "username": "Nick_ishere",478 "name": "",479 "avatar_template": "/user_avatar/discuss.pytorch.org/nick_ishere/{size}/25121_2.png",480 "trust_level": 1481 }482 }483 ]484 }485 ],486 "tags_descriptions": {},487 "fancy_title": "Evaluate Twice, Accuracy Changes if I Shuffle",488 "id": 150976,489 "title": "Evaluate Twice, Accuracy Changes if I Shuffle",490 "posts_count": 4,491 "created_at": "2022-05-06T19:30:34.539Z",492 "views": 455,493 "reply_count": 1,494 "like_count": 0,495 "last_posted_at": "2022-05-06T20:09:36.364Z",496 "visible": true,497 "closed": false,498 "archived": false,499 "has_summary": false,500 "archetype": "regular",501 "slug": "evaluate-twice-accuracy-changes-if-i-shuffle",502 "category_id": 1,503 "word_count": 427,504 "deleted_at": null,505 "user_id": 54777,506 "featured_link": null,507 "pinned_globally": false,508 "pinned_at": null,509 "pinned_until": null,510 "image_url": null,511 "slow_mode_seconds": 0,512 "draft": null,513 "draft_key": "topic_150976",514 "draft_sequence": null,515 "unpinned": null,516 "pinned": false,517 "current_post_number": 1,518 "highest_post_number": 4,519 "deleted_by": null,520 "actions_summary": [521 {522 "id": 4,523 "count": 0,524 "hidden": false,525 "can_act": false526 },527 {528 "id": 8,529 "count": 0,530 "hidden": false,531 "can_act": false532 },533 {534 "id": 10,535 "count": 0,536 "hidden": false,537 "can_act": false538 },539 {540 "id": 7,541 "count": 0,542 "hidden": false,543 "can_act": false544 }545 ],546 "chunk_size": 20,547 "bookmarked": false,548 "topic_timer": null,549 "message_bus_last_id": 0,550 "participant_count": 2,551 "show_read_indicator": false,552 "thumbnails": null,553 "slow_mode_enabled_until": null,554 "can_vote": false,555 "vote_count": 0,556 "user_voted": false,557 "discourse_zendesk_plugin_zendesk_id": null,558 "discourse_zendesk_plugin_zendesk_url": "https://your-url.zendesk.com/agent/tickets/",559 "details": {560 "can_edit": false,561 "notification_level": 1,562 "participants": [563 {564 "id": 54777,565 "username": "4gatepylon",566 "name": "razzle dazzle",567 "avatar_template": "/user_avatar/discuss.pytorch.org/4gatepylon/{size}/48373_2.png",568 "post_count": 3,569 "primary_group_name": null,570 "flair_name": null,571 "flair_url": null,572 "flair_color": null,573 "flair_bg_color": null,574 "flair_group_id": null,575 "trust_level": 1576 },577 {578 "id": 54308,579 "username": "Andrei_Cristea",580 "name": "Andrei Cristea",581 "avatar_template": "/user_avatar/discuss.pytorch.org/andrei_cristea/{size}/47843_2.png",582 "post_count": 1,583 "primary_group_name": null,584 "flair_name": null,585 "flair_url": null,586 "flair_color": null,587 "flair_bg_color": null,588 "flair_group_id": null,589 "trust_level": 2590 }591 ],592 "created_by": {593 "id": 54777,594 "username": "4gatepylon",595 "name": "razzle dazzle",596 "avatar_template": "/user_avatar/discuss.pytorch.org/4gatepylon/{size}/48373_2.png"597 },598 "last_poster": {599 "id": 54777,600 "username": "4gatepylon",601 "name": "razzle dazzle",602 "avatar_template": "/user_avatar/discuss.pytorch.org/4gatepylon/{size}/48373_2.png"603 }604 },605 "bookmarks": []606 },607 {608 "post_stream": {609 "posts": [610 {611 "id": 345018,612 "name": "Scott Hoang",613 "username": "Scott_Hoang",614 "avatar_template": "/user_avatar/discuss.pytorch.org/scott_hoang/{size}/9907_2.png",615 "created_at": "2022-05-06T17:37:40.037Z",616 "cooked": "<p>Hi all,<br>\nWhile I am very familiar with PyTorch, I am a noob when it comes to torch.jit.<br>\nI have several questions:</p>\n<ol>\n<li>Is it possible to start writing the scripting module now and still train your model following the standard PyTorch training regime?</li>\n<li>does torch. Script support forward hooks?</li>\n<li>Is it possible to write a hybrid model with some sub-modules that are script/trace while others are just standard nn.Module and still enjoy some benefit?</li>\n</ol>\n<p>I’m doing this because my model is still a work-in-progress, and many parts are subjected to change, while some are entirely static. So I want to test what speed-up I can get starting the transition now.<br>\nThanks</p>",617 "post_number": 1,618 "post_type": 1,619 "posts_count": 1,620 "updated_at": "2022-05-06T17:37:40.037Z",621 "reply_count": 0,622 "reply_to_post_number": null,623 "quote_count": 0,624 "incoming_link_count": 235,625 "reads": 9,626 "readers_count": 8,627 "score": 1171.8,628 "yours": false,629 "topic_id": 150970,630 "topic_slug": "torch-script-and-eager-mode-hybrid-interface",631 "display_username": "Scott Hoang",632 "primary_group_name": null,633 "flair_name": null,634 "flair_url": null,635 "flair_bg_color": null,636 "flair_color": null,637 "flair_group_id": null,638 "badges_granted": [],639 "version": 1,640 "can_edit": false,641 "can_delete": false,642 "can_recover": false,643 "can_see_hidden_post": false,644 "can_wiki": false,645 "read": true,646 "user_title": null,647 "bookmarked": false,648 "actions_summary": [],649 "moderator": false,650 "admin": false,651 "staff": false,652 "user_id": 16242,653 "hidden": false,654 "trust_level": 2,655 "deleted_at": null,656 "user_deleted": false,657 "edit_reason": null,658 "can_view_edit_history": true,659 "wiki": false,660 "post_url": "/t/torch-script-and-eager-mode-hybrid-interface/150970/1",661 "can_accept_answer": false,662 "can_unaccept_answer": false,663 "accepted_answer": false,664 "topic_accepted_answer": null,665 "can_vote": false666 }667 ],668 "stream": [669 345018670 ]671 },672 "timeline_lookup": [673 [674 1,675 1268676 ]677 ],678 "suggested_topics": [679 {680 "fancy_title": "Unsupported value kind: Tensor",681 "id": 218252,682 "title": "Unsupported value kind: Tensor",683 "slug": "unsupported-value-kind-tensor",684 "posts_count": 1,685 "reply_count": 0,686 "highest_post_number": 1,687 "image_url": null,688 "created_at": "2025-03-25T15:01:18.367Z",689 "last_posted_at": "2025-03-25T15:01:18.412Z",690 "bumped": true,691 "bumped_at": "2025-03-25T15:01:18.412Z",692 "archetype": "regular",693 "unseen": false,694 "pinned": false,695 "unpinned": null,696 "visible": true,697 "closed": false,698 "archived": false,699 "bookmarked": null,700 "liked": null,701 "tags_descriptions": {},702 "like_count": 0,703 "views": 52,704 "category_id": 13,705 "featured_link": null,706 "has_accepted_answer": false,707 "posters": [708 {709 "extras": "latest single",710 "description": "Original Poster, Most Recent Poster",711 "user": {712 "id": 83460,713 "username": "FranciscoMessina00",714 "name": "Francisco Messina",715 "avatar_template": "/user_avatar/discuss.pytorch.org/franciscomessina00/{size}/76339_2.png",716 "trust_level": 1717 }718 }719 ]720 },721 {722 "fancy_title": "How to get around pad error when doing torch.jit.save?",723 "id": 214424,724 "title": "How to get around pad error when doing torch.jit.save?",725 "slug": "how-to-get-around-pad-error-when-doing-torch-jit-save",726 "posts_count": 2,727 "reply_count": 0,728 "highest_post_number": 2,729 "image_url": null,730 "created_at": "2024-12-20T00:31:12.244Z",731 "last_posted_at": "2024-12-20T01:35:29.934Z",732 "bumped": true,733 "bumped_at": "2024-12-20T01:35:29.934Z",734 "archetype": "regular",735 "unseen": false,736 "pinned": false,737 "unpinned": null,738 "visible": true,739 "closed": false,740 "archived": false,741 "bookmarked": null,742 "liked": null,743 "tags_descriptions": {},744 "like_count": 0,745 "views": 46,746 "category_id": 13,747 "featured_link": null,748 "has_accepted_answer": false,749 "posters": [750 {751 "extras": null,752 "description": "Original Poster",753 "user": {754 "id": 45116,755 "username": "JimW",756 "name": "",757 "avatar_template": "/user_avatar/discuss.pytorch.org/jimw/{size}/38000_2.png",758 "trust_level": 1759 }760 },761 {762 "extras": "latest",763 "description": "Most Recent Poster",764 "user": {765 "id": 81605,766 "username": "benjamin-perry-duke",767 "name": "Ben Perry",768 "avatar_template": "/user_avatar/discuss.pytorch.org/benjamin-perry-duke/{size}/74629_2.png",769 "trust_level": 1770 }771 }772 ]773 },774 {775 "fancy_title": "How to check grads in each step of model?",776 "id": 216615,777 "title": "How to check grads in each step of model?",778 "slug": "how-to-check-grads-in-each-step-of-model",779 "posts_count": 2,780 "reply_count": 0,781 "highest_post_number": 2,782 "image_url": null,783 "created_at": "2025-02-13T09:04:16.004Z",784 "last_posted_at": "2025-03-24T20:39:19.668Z",785 "bumped": true,786 "bumped_at": "2025-03-24T20:39:19.668Z",787 "archetype": "regular",788 "unseen": false,789 "pinned": false,790 "unpinned": null,791 "visible": true,792 "closed": false,793 "archived": false,794 "bookmarked": null,795 "liked": null,796 "tags_descriptions": {},797 "like_count": 0,798 "views": 53,799 "category_id": 13,800 "featured_link": null,801 "has_accepted_answer": false,802 "posters": [803 {804 "extras": null,805 "description": "Original Poster",806 "user": {807 "id": 73868,808 "username": "elinliu0823",809 "name": "轶霖 柳",810 "avatar_template": "/user_avatar/discuss.pytorch.org/elinliu0823/{size}/68215_2.png",811 "trust_level": 1812 }813 },814 {815 "extras": "latest",816 "description": "Most Recent Poster",817 "user": {818 "id": 60146,819 "username": "MyCenturaHealth",820 "name": "MyCenturaHealth",821 "avatar_template": "/user_avatar/discuss.pytorch.org/mycenturahealth/{size}/46345_2.png",822 "trust_level": 1823 }824 }825 ]826 },827 {828 "fancy_title": "Trying torch.jit.script with a super().forward",829 "id": 215784,830 "title": "Trying torch.jit.script with a super().forward",831 "slug": "trying-torch-jit-script-with-a-super-forward",832 "posts_count": 1,833 "reply_count": 0,834 "highest_post_number": 1,835 "image_url": null,836 "created_at": "2025-01-23T18:02:03.132Z",837 "last_posted_at": "2025-01-23T18:02:03.174Z",838 "bumped": true,839 "bumped_at": "2025-01-23T18:02:03.174Z",840 "archetype": "regular",841 "unseen": false,842 "pinned": false,843 "unpinned": null,844 "visible": true,845 "closed": false,846 "archived": false,847 "bookmarked": null,848 "liked": null,849 "tags_descriptions": {},850 "like_count": 0,851 "views": 50,852 "category_id": 13,853 "featured_link": null,854 "has_accepted_answer": false,855 "posters": [856 {857 "extras": "latest single",858 "description": "Original Poster, Most Recent Poster",859 "user": {860 "id": 1997,861 "username": "milongo",862 "name": "Milongo",863 "avatar_template": "/user_avatar/discuss.pytorch.org/milongo/{size}/2707_2.png",864 "trust_level": 1865 }866 }867 ]868 },869 {870 "fancy_title": "Torch.jit._get_trace_graph",871 "id": 219103,872 "title": "Torch.jit._get_trace_graph",873 "slug": "torch-jit-get-trace-graph",874 "posts_count": 1,875 "reply_count": 0,876 "highest_post_number": 1,877 "image_url": null,878 "created_at": "2025-04-15T11:21:23.571Z",879 "last_posted_at": "2025-04-15T11:21:23.612Z",880 "bumped": true,881 "bumped_at": "2025-04-16T05:35:42.219Z",882 "archetype": "regular",883 "unseen": false,884 "pinned": false,885 "unpinned": null,886 "visible": true,887 "closed": false,888 "archived": false,889 "bookmarked": null,890 "liked": null,891 "tags_descriptions": {},892 "like_count": 0,893 "views": 72,894 "category_id": 13,895 "featured_link": null,896 "has_accepted_answer": false,897 "posters": [898 {899 "extras": "latest single",900 "description": "Original Poster, Most Recent Poster",901 "user": {902 "id": 81709,903 "username": "QLYYLQ",904 "name": "qly",905 "avatar_template": "/user_avatar/discuss.pytorch.org/qlyylq/{size}/74718_2.png",906 "trust_level": 1907 }908 }909 ]910 }911 ],912 "tags_descriptions": {},913 "fancy_title": "Torch script and eager-mode hybrid interface",914 "id": 150970,915 "title": "Torch script and eager-mode hybrid interface",916 "posts_count": 1,917 "created_at": "2022-05-06T17:37:39.969Z",918 "views": 698,919 "reply_count": 0,920 "like_count": 0,921 "last_posted_at": "2022-05-06T17:37:40.037Z",922 "visible": true,923 "closed": false,924 "archived": false,925 "has_summary": false,926 "archetype": "regular",927 "slug": "torch-script-and-eager-mode-hybrid-interface",928 "category_id": 13,929 "word_count": 122,930 "deleted_at": null,931 "user_id": 16242,932 "featured_link": null,933 "pinned_globally": false,934 "pinned_at": null,935 "pinned_until": null,936 "image_url": null,937 "slow_mode_seconds": 0,938 "draft": null,939 "draft_key": "topic_150970",940 "draft_sequence": null,941 "unpinned": null,942 "pinned": false,943 "current_post_number": 1,944 "highest_post_number": 1,945 "deleted_by": null,946 "actions_summary": [947 {948 "id": 4,949 "count": 0,950 "hidden": false,951 "can_act": false952 },953 {954 "id": 8,955 "count": 0,956 "hidden": false,957 "can_act": false958 },959 {960 "id": 10,961 "count": 0,962 "hidden": false,963 "can_act": false964 },965 {966 "id": 7,967 "count": 0,968 "hidden": false,969 "can_act": false970 }971 ],972 "chunk_size": 20,973 "bookmarked": false,974 "topic_timer": null,975 "message_bus_last_id": 0,976 "participant_count": 1,977 "show_read_indicator": false,978 "thumbnails": null,979 "slow_mode_enabled_until": null,980 "can_vote": false,981 "vote_count": 0,982 "user_voted": false,983 "discourse_zendesk_plugin_zendesk_id": null,984 "discourse_zendesk_plugin_zendesk_url": "https://your-url.zendesk.com/agent/tickets/",985 "details": {986 "can_edit": false,987 "notification_level": 1,988 "participants": [989 {990 "id": 16242,991 "username": "Scott_Hoang",992 "name": "Scott Hoang",993 "avatar_template": "/user_avatar/discuss.pytorch.org/scott_hoang/{size}/9907_2.png",994 "post_count": 1,995 "primary_group_name": null,996 "flair_name": null,997 "flair_url": null,998 "flair_color": null,999 "flair_bg_color": null,1000 "flair_group_id": null,1001 "trust_level": 21002 }1003 ],1004 "created_by": {1005 "id": 16242,1006 "username": "Scott_Hoang",1007 "name": "Scott Hoang",1008 "avatar_template": "/user_avatar/discuss.pytorch.org/scott_hoang/{size}/9907_2.png"1009 },1010 "last_poster": {1011 "id": 16242,1012 "username": "Scott_Hoang",1013 "name": "Scott Hoang",1014 "avatar_template": "/user_avatar/discuss.pytorch.org/scott_hoang/{size}/9907_2.png"1015 }1016 },1017 "bookmarks": []1018 },1019 {1020 "post_stream": {1021 "posts": [1022 {1023 "id": 345012,1024 "name": "Gul Zain",1025 "username": "Gul_Zain",1026 "avatar_template": "/user_avatar/discuss.pytorch.org/gul_zain/{size}/13209_2.png",1027 "created_at": "2022-05-06T16:19:00.994Z",1028 "cooked": "<p>Hello,<br>\nI am trying to do a graphconv operation for a 3D data. Data ‘x’ is in shape [batch,seq_dim,feature_dim]. I have a sparse adjacency matrix ‘adj’ of shape [seq_dim,seq_dim]. I want to multiply adjacency matrix with input for all the samples. So i simply try to expand the adjacency matrix as adj.unsqueeze(0).expand([x.shape[0],-1,-1]). However, i get following error at this operation</p>\n<blockquote>\n<p>RuntimeError: sparse tensors do not have strides</p>\n</blockquote>\n<p>This error occurs at .expand operation. Unsqueeze() operation does not trigger this error. Please let me know if there is any other way around to do this operation with pytorch as well?</p>",1029 "post_number": 1,1030 "post_type": 1,1031 "posts_count": 3,1032 "updated_at": "2022-05-06T16:19:00.994Z",1033 "reply_count": 0,1034 "reply_to_post_number": null,1035 "quote_count": 0,1036 "incoming_link_count": 653,1037 "reads": 14,1038 "readers_count": 13,1039 "score": 3262.8,1040 "yours": false,1041 "topic_id": 150966,1042 "topic_slug": "expand-2d-sparse-tensor-to-do-multiplication-with-3d-tensor",1043 "display_username": "Gul Zain",1044 "primary_group_name": null,1045 "flair_name": null,1046 "flair_url": null,1047 "flair_bg_color": null,1048 "flair_color": null,1049 "flair_group_id": null,1050 "badges_granted": [],1051 "version": 1,1052 "can_edit": false,1053 "can_delete": false,1054 "can_recover": false,1055 "can_see_hidden_post": false,1056 "can_wiki": false,1057 "read": true,1058 "user_title": null,1059 "bookmarked": false,1060 "actions_summary": [],1061 "moderator": false,1062 "admin": false,1063 "staff": false,1064 "user_id": 15041,1065 "hidden": false,1066 "trust_level": 1,1067 "deleted_at": null,1068 "user_deleted": false,1069 "edit_reason": null,1070 "can_view_edit_history": true,1071 "wiki": false,1072 "post_url": "/t/expand-2d-sparse-tensor-to-do-multiplication-with-3d-tensor/150966/1",1073 "can_accept_answer": false,1074 "can_unaccept_answer": false,1075 "accepted_answer": false,1076 "topic_accepted_answer": null,1077 "can_vote": false1078 },1079 {1080 "id": 345014,1081 "name": "Gul Zain",1082 "username": "Gul_Zain",1083 "avatar_template": "/user_avatar/discuss.pytorch.org/gul_zain/{size}/13209_2.png",1084 "created_at": "2022-05-06T16:21:07.088Z",1085 "cooked": "<p><a class=\"mention\" href=\"/u/ptrblck\">@ptrblck</a> quick help will be much appreciated <img src=\"https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12\" title=\":slight_smile:\" class=\"emoji\" alt=\":slight_smile:\" loading=\"lazy\" width=\"20\" height=\"20\"></p>",1086 "post_number": 2,1087 "post_type": 1,1088 "posts_count": 3,1089 "updated_at": "2022-05-06T16:21:07.088Z",1090 "reply_count": 0,1091 "reply_to_post_number": null,1092 "quote_count": 0,1093 "incoming_link_count": 11,1094 "reads": 15,1095 "readers_count": 14,1096 "score": 58.0,1097 "yours": false,1098 "topic_id": 150966,1099 "topic_slug": "expand-2d-sparse-tensor-to-do-multiplication-with-3d-tensor",1100 "display_username": "Gul Zain",1101 "primary_group_name": null,1102 "flair_name": null,1103 "flair_url": null,1104 "flair_bg_color": null,1105 "flair_color": null,1106 "flair_group_id": null,1107 "badges_granted": [],1108 "version": 1,1109 "can_edit": false,1110 "can_delete": false,1111 "can_recover": false,1112 "can_see_hidden_post": false,1113 "can_wiki": false,1114 "read": true,1115 "user_title": null,1116 "bookmarked": false,1117 "actions_summary": [],1118 "moderator": false,1119 "admin": false,1120 "staff": false,1121 "user_id": 15041,1122 "hidden": false,1123 "trust_level": 1,1124 "deleted_at": null,1125 "user_deleted": false,1126 "edit_reason": null,1127 "can_view_edit_history": true,1128 "wiki": false,1129 "post_url": "/t/expand-2d-sparse-tensor-to-do-multiplication-with-3d-tensor/150966/2",1130 "can_accept_answer": false,1131 "can_unaccept_answer": false,1132 "accepted_answer": false,1133 "topic_accepted_answer": null1134 },1135 {1136 "id": 345016,1137 "name": "Matias Vasquez",1138 "username": "Matias_Vasquez",1139 "avatar_template": "/user_avatar/discuss.pytorch.org/matias_vasquez/{size}/47847_2.png",1140 "created_at": "2022-05-06T17:14:44.735Z",1141 "cooked": "<p>This is just a workaround, maybe someone has a better solution.</p>\n<p><a href=\"https://pytorch.org/docs/stable/sparse.html#supported-linear-algebra-operations\" rel=\"noopener nofollow ugc\">Here</a> is a list with the available linear algebra operations with sparse matrices.</p>\n<p>However some of them do not support batch multiplication yet. (<a href=\"https://github.com/pytorch/pytorch/issues/14489\" rel=\"noopener nofollow ugc\">github issue</a>)</p>\n<p>In this issue they mention that <code>bmm</code> can be used even with sparse matrices. However, it does not support broadcasting as shown in the <a href=\"https://pytorch.org/docs/stable/generated/torch.bmm.html\" rel=\"noopener nofollow ugc\">docs</a>. And sparse tensors cannot use the <code>repeat</code> method.</p>\n<p>So the solution posted <a href=\"https://discuss.pytorch.org/t/torch-repeat-for-sparse-matrix/127581\">here</a> is to stack your <code>sparse</code> tensors in the desired dimension.</p>\n<p>This should work.</p>\n<pre><code class=\"lang-python\">batch = 3\nseq_dim = 150\nfeature_dim = 300\n\ni = [[0, 1, seq_dim-1], [0, 2, seq_dim-1]]\nv = [3, 4, 5]\nadj = torch.sparse_coo_tensor(i, v, dtype=torch.float)\nnew_adj = torch.stack([adj for _ in range(3)], dim=0)\n\nx = torch.rand(batch, seq_dim, feature_dim)\n\nprint(torch.bmm(input=new_adj, mat2=x).shape)\n</code></pre>",1142 "post_number": 3,1143 "post_type": 1,1144 "posts_count": 3,1145 "updated_at": "2022-05-06T17:14:44.735Z",1146 "reply_count": 0,1147 "reply_to_post_number": null,1148 "quote_count": 0,1149 "incoming_link_count": 41,1150 "reads": 14,1151 "readers_count": 13,1152 "score": 222.8,1153 "yours": false,1154 "topic_id": 150966,1155 "topic_slug": "expand-2d-sparse-tensor-to-do-multiplication-with-3d-tensor",1156 "display_username": "Matias Vasquez",1157 "primary_group_name": null,1158 "flair_name": null,1159 "flair_url": null,1160 "flair_bg_color": null,1161 "flair_color": null,1162 "flair_group_id": null,1163 "badges_granted": [],1164 "version": 1,1165 "can_edit": false,1166 "can_delete": false,1167 "can_recover": false,1168 "can_see_hidden_post": false,1169 "can_wiki": false,1170 "link_counts": [1171 {1172 "url": "https://pytorch.org/docs/stable/sparse.html#supported-linear-algebra-operations",1173 "internal": false,1174 "reflection": false,1175 "title": "torch.sparse — PyTorch 1.11.0 documentation",1176 "clicks": 311177 },1178 {1179 "url": "https://discuss.pytorch.org/t/torch-repeat-for-sparse-matrix/127581",1180 "internal": true,1181 "reflection": false,1182 "title": "Torch.repeat for sparse matrix",1183 "clicks": 221184 },1185 {1186 "url": "https://github.com/pytorch/pytorch/issues/14489",1187 "internal": false,1188 "reflection": false,1189 "title": "Batch matmul with sparse matrix, dense vector · Issue #14489 · pytorch/pytorch · GitHub",1190 "clicks": 111191 },1192 {1193 "url": "https://pytorch.org/docs/stable/generated/torch.bmm.html",1194 "internal": false,1195 "reflection": false,1196 "title": "torch.bmm — PyTorch 1.11.0 documentation",1197 "clicks": 51198 }1199 ],1200 "read": true,