Anurag1734/cuda-error-resolution-analysis
07
1[2 {3 "post_stream": {4 "posts": [5 {6 "id": 397022,7 "name": "",8 "username": "Virat_Baahubali",9 "avatar_template": "/letter_avatar_proxy/v4/letter/v/848f3c/{size}.png",10 "created_at": "2023-04-12T01:51:22.364Z",11 "cooked": "<p>Hello,</p>\n<p>I am working on multiclass image classification where, I have a custom dataset with 13 classes such as Alien, Predator, Terminator, Robin, Batman, Superman, Spiderman, Valkyrie, Raven, BeastBoy, DeathStroke, Deadpool, PoisonIvy. I have around 5236 images for training and 1300 for validation. Each class has around approx 400 for training and 100 for validation. I went through <a href=\"https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html\" class=\"inline-onebox\" rel=\"noopener nofollow ugc\">Transfer Learning for Computer Vision Tutorial — PyTorch Tutorials 2.0.0+cu117 documentation</a> and Pytorch’s fine tuning tutorial. I am using pretrained <a href=\"https://pytorch.org/vision/main/models/generated/torchvision.models.convnext_small.html#torchvision.models.convnext_small\" rel=\"noopener nofollow ugc\">ConvNeXt</a> model and I have unfreeze layer 6,7 of feature extractor and classifier layer 2</p>\n<p>(</p>\n<pre><code class=\"lang-auto\">7): Sequential(\n (0): CNBlock(\n (block): Sequential(\n (0): Conv2d(768, 768, kernel_size=(7, 7), stride=(1, 1), padding=(3, 3), groups=768)\n (1): Permute()\n (2): LayerNorm((768,), eps=1e-06, elementwise_affine=True)\n (3): Linear(in_features=768, out_features=3072, bias=True)\n (4): GELU(approximate='none')\n (5): Linear(in_features=3072, out_features=768, bias=True)\n (6): Permute()\n )\n (stochastic_depth): StochasticDepth(p=0.37714285714285717, mode=row)\n )\n (1): CNBlock(\n (block): Sequential(\n (0): Conv2d(768, 768, kernel_size=(7, 7), stride=(1, 1), padding=(3, 3), groups=768)\n (1): Permute()\n (2): LayerNorm((768,), eps=1e-06, elementwise_affine=True)\n (3): Linear(in_features=768, out_features=3072, bias=True)\n (4): GELU(approximate='none')\n (5): Linear(in_features=3072, out_features=768, bias=True)\n (6): Permute()\n )\n (stochastic_depth): StochasticDepth(p=0.3885714285714286, mode=row)\n )\n (2): CNBlock(\n (block): Sequential(\n (0): Conv2d(768, 768, kernel_size=(7, 7), stride=(1, 1), padding=(3, 3), groups=768)\n (1): Permute()\n (2): LayerNorm((768,), eps=1e-06, elementwise_affine=True)\n (3): Linear(in_features=768, out_features=3072, bias=True)\n (4): GELU(approximate='none')\n (5): Linear(in_features=3072, out_features=768, bias=True)\n (6): Permute()\n )\n (stochastic_depth): StochasticDepth(p=0.4, mode=row)\n )\n )\n )\n (avgpool): AdaptiveAvgPool2d(output_size=1)\n (classifier): Sequential(\n (0): LayerNorm2d((768,), eps=1e-06, elementwise_affine=True)\n (1): Flatten(start_dim=1, end_dim=-1)\n (2): Linear(in_features=768, out_features=13, bias=True)\n )\n)\n</code></pre>\n<p>Here, is the model summary</p>\n<pre><code class=\"lang-auto\"> └─Sequential (6) [12, 384, 14, 14] [12, 768, 7, 7] -- True\n│ │ └─LayerNorm2d (0) [12, 384, 14, 14] [12, 384, 14, 14] 768 True\n│ │ └─Conv2d (1) [12, 384, 14, 14] [12, 768, 7, 7] 1,180,416 True\n│ └─Sequential (7) [12, 768, 7, 7] [12, 768, 7, 7] -- True\n│ │ └─CNBlock (0) [12, 768, 7, 7] [12, 768, 7, 7] 4,763,136 True\n│ │ └─CNBlock (1) [12, 768, 7, 7] [12, 768, 7, 7] 4,763,136 True\n│ │ └─CNBlock (2) [12, 768, 7, 7] [12, 768, 7, 7] 4,763,136 True\n├─AdaptiveAvgPool2d (avgpool) [12, 768, 7, 7] [12, 768, 1, 1] -- --\n├─Sequential (classifier) [12, 768, 1, 1] [12, 13] -- True\n│ └─LayerNorm2d (0) [12, 768, 1, 1] [12, 768, 1, 1] 1,536 True\n│ └─Flatten (1) [12, 768, 1, 1] [12, 768] -- --\n│ └─Linear (2) [12, 768] [12, 13] 9,997 True\n=======================================================================================================================================\nTotal params: 49,464,685\nTrainable params: 15,482,125\nNon-trainable params: 33,982,560\nTotal mult-adds (G): 4.93\n=======================================================================================================================================\nInput size (MB): 7.23\nForward/backward pass size (MB): 2485.59\nParams size (MB): 197.80\nEstimated Total Size (MB): 2690.62\n</code></pre>\n<p>Why unfreeze CNN feature extractor layers?<br>\nBecause, the custom dataset is completely new to the pretrained model(ConvNeXt). ConvNeXt has never seen such data. So, from my understanding its better to unfreeze last 2 layers of feature extractor to get some essential learnings specific to my custom data.</p>\n<p>I have couple of questions</p>\n<ol>\n<li>Which parameters should I pass into the optimizer? should it be whole model parameters or parameters of the layers which I have unfrozen(layer[6,7] from feature extractor and feature classifier layers)</li>\n</ol>\n<p><strong>Approach 1</strong> update weights of only unfrozen(layer[6,7] from feature extractor and feature classifier layers) while rest of the model weights are frozen</p>\n<pre><code class=\"lang-auto\">from torchvision import models\nmodel = models.convnext_small(pretrained=True)\nparams_to_update = []\n\n for param in model.classifier.parameters():\n param.requires_grad = True\n params_to_update.append(param)\n\n for name, block in model.features.named_modules():\n if(name in finetune_features_layers):\n for param in block.parameters():\n param.requires_grad = True\n params_to_update.append(param)\n\n# optimizer\noptimizer = optim.Adam(\n params_to_update,\n lr = 0.0001\n)\n</code></pre>\n<p><strong>Approach 2</strong> Update whole model weights. Observe below parameters of all layers will be optimized</p>\n<pre><code class=\"lang-auto\">from torchvision import models\nmodel = models.convnext_small(pretrained=True)\nparams_to_update = model.parameters()\n# optimizer\noptimizer = optim.Adam(\n params_to_update,\n lr = 0.0001\n)\n</code></pre>\n<ol start=\"2\">\n<li>Does the size and type of the dataset matters in generalization which could essentially vary the number of CNN layers to keep frozen or unfroze couple of them?</li>\n<li>I trained my model for</li>\n</ol>\n<pre><code class=\"lang-auto\">Epoch: 49 \nTrain Loss: 1.156555 Acc: 0.6353\nElapsed 12323.04s, 246.46 s/epoch, 3.01 s/batch, ets 0.00s\n\nTest set: Average loss: 0.8826, Accuracy: 930/1300 (72%)\n\nModel Improved. Saving the Model...\n</code></pre>\n<p>But when I am evaluating this newly trained model on test data(completely new/fresh unseen custom data with the same 13 classes as above). I get around 10% of accuracy. I am trying to understand what is going wrong here? I checked the dataset both training and validation they have right class labels and right images.</p>\n<p>Is my understanding of feature extraction and fine tuning correct? Am I heading in the right direction?</p>",12 "post_number": 1,13 "post_type": 1,14 "posts_count": 3,15 "updated_at": "2023-04-12T01:51:22.364Z",16 "reply_count": 0,17 "reply_to_post_number": null,18 "quote_count": 0,19 "incoming_link_count": 141,20 "reads": 7,21 "readers_count": 6,22 "score": 706.4,23 "yours": false,24 "topic_id": 177283,25 "topic_slug": "which-parameters-to-pass-in-optimizer-for-transfer-learning",26 "display_username": "",27 "primary_group_name": null,28 "flair_name": null,29 "flair_url": null,30 "flair_bg_color": null,31 "flair_color": null,32 "flair_group_id": null,33 "badges_granted": [],34 "version": 1,35 "can_edit": false,36 "can_delete": false,37 "can_recover": false,38 "can_see_hidden_post": false,39 "can_wiki": false,40 "link_counts": [41 {42 "url": "https://pytorch.org/vision/main/models/generated/torchvision.models.convnext_small.html#torchvision.models.convnext_small",43 "internal": false,44 "reflection": false,45 "title": "convnext_small — Torchvision main documentation",46 "clicks": 147 },48 {49 "url": "https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html",50 "internal": false,51 "reflection": false,52 "title": "Transfer Learning for Computer Vision Tutorial — PyTorch Tutorials 2.0.0+cu117 documentation",53 "clicks": 154 }55 ],56 "read": true,57 "user_title": null,58 "bookmarked": false,59 "actions_summary": [],60 "moderator": false,61 "admin": false,62 "staff": false,63 "user_id": 62891,64 "hidden": false,65 "trust_level": 0,66 "deleted_at": null,67 "user_deleted": false,68 "edit_reason": null,69 "can_view_edit_history": true,70 "wiki": false,71 "post_url": "/t/which-parameters-to-pass-in-optimizer-for-transfer-learning/177283/1",72 "can_accept_answer": false,73 "can_unaccept_answer": false,74 "accepted_answer": false,75 "topic_accepted_answer": null,76 "can_vote": false77 },78 {79 "id": 397064,80 "name": "",81 "username": "ptrblck",82 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",83 "created_at": "2023-04-12T06:13:17.713Z",84 "cooked": "<ol>\n<li>\n<p>I would prefer the explicit approach in passing only the trainable parameters to the <code>optimizer</code> assuming you don’t want to “unfreeze” other parameters later in the training.</p>\n</li>\n<li>\n<p>Yes, the dataset and “type” will certainly matter during finetuning in a similar way it would matter when trying to train a model from scratch.</p>\n</li>\n<li>\n<p>Could you reuse your training data during this testing step to make sure you are still able to achieve the previously reported accuracy and loss?</p>\n</li>\n</ol>",85 "post_number": 2,86 "post_type": 1,87 "posts_count": 3,88 "updated_at": "2023-04-12T06:13:17.713Z",89 "reply_count": 1,90 "reply_to_post_number": null,91 "quote_count": 0,92 "incoming_link_count": 1,93 "reads": 7,94 "readers_count": 6,95 "score": 11.4,96 "yours": false,97 "topic_id": 177283,98 "topic_slug": "which-parameters-to-pass-in-optimizer-for-transfer-learning",99 "display_username": "",100 "primary_group_name": null,101 "flair_name": null,102 "flair_url": null,103 "flair_bg_color": null,104 "flair_color": null,105 "flair_group_id": null,106 "badges_granted": [],107 "version": 1,108 "can_edit": false,109 "can_delete": false,110 "can_recover": false,111 "can_see_hidden_post": false,112 "can_wiki": false,113 "read": true,114 "user_title": "",115 "bookmarked": false,116 "actions_summary": [],117 "moderator": true,118 "admin": true,119 "staff": true,120 "user_id": 3534,121 "hidden": false,122 "trust_level": 2,123 "deleted_at": null,124 "user_deleted": false,125 "edit_reason": null,126 "can_view_edit_history": true,127 "wiki": false,128 "post_url": "/t/which-parameters-to-pass-in-optimizer-for-transfer-learning/177283/2",129 "can_accept_answer": false,130 "can_unaccept_answer": false,131 "accepted_answer": false,132 "topic_accepted_answer": null133 },134 {135 "id": 397154,136 "name": "",137 "username": "Virat_Baahubali",138 "avatar_template": "/letter_avatar_proxy/v4/letter/v/848f3c/{size}.png",139 "created_at": "2023-04-12T15:38:04.016Z",140 "cooked": "<p>Actually, on training my model for longer time around 70 epochs Train accuracy reaches 95% and Validation accuracy reaches upto 79% max. I got around 20% accuracy on training set when I did model evaluation.</p>",141 "post_number": 3,142 "post_type": 1,143 "posts_count": 3,144 "updated_at": "2023-04-12T15:38:04.016Z",145 "reply_count": 0,146 "reply_to_post_number": 2,147 "quote_count": 0,148 "incoming_link_count": 2,149 "reads": 5,150 "readers_count": 4,151 "score": 11.0,152 "yours": false,153 "topic_id": 177283,154 "topic_slug": "which-parameters-to-pass-in-optimizer-for-transfer-learning",155 "display_username": "",156 "primary_group_name": null,157 "flair_name": null,158 "flair_url": null,159 "flair_bg_color": null,160 "flair_color": null,161 "flair_group_id": null,162 "badges_granted": [],163 "version": 1,164 "can_edit": false,165 "can_delete": false,166 "can_recover": false,167 "can_see_hidden_post": false,168 "can_wiki": false,169 "read": true,170 "user_title": null,171 "reply_to_user": {172 "id": 3534,173 "username": "ptrblck",174 "name": "",175 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png"176 },177 "bookmarked": false,178 "actions_summary": [],179 "moderator": false,180 "admin": false,181 "staff": false,182 "user_id": 62891,183 "hidden": false,184 "trust_level": 0,185 "deleted_at": null,186 "user_deleted": false,187 "edit_reason": null,188 "can_view_edit_history": true,189 "wiki": false,190 "post_url": "/t/which-parameters-to-pass-in-optimizer-for-transfer-learning/177283/3",191 "can_accept_answer": false,192 "can_unaccept_answer": false,193 "accepted_answer": false,194 "topic_accepted_answer": null195 }196 ],197 "stream": [198 397022,199 397064,200 397154201 ]202 },203 "timeline_lookup": [204 [205 1,206 928207 ],208 [209 3,210 927211 ]212 ],213 "suggested_topics": [214 {215 "fancy_title": "Pytorch geometric temporal mamba installation",216 "id": 212651,217 "title": "Pytorch geometric temporal mamba installation",218 "slug": "pytorch-geometric-temporal-mamba-installation",219 "posts_count": 4,220 "reply_count": 2,221 "highest_post_number": 4,222 "image_url": null,223 "created_at": "2024-11-07T08:55:00.714Z",224 "last_posted_at": "2024-11-07T09:40:49.990Z",225 "bumped": true,226 "bumped_at": "2024-11-07T09:40:49.990Z",227 "archetype": "regular",228 "unseen": false,229 "pinned": false,230 "unpinned": null,231 "visible": true,232 "closed": false,233 "archived": false,234 "bookmarked": null,235 "liked": null,236 "tags_descriptions": {},237 "like_count": 0,238 "views": 615,239 "category_id": 1,240 "featured_link": null,241 "has_accepted_answer": false,242 "posters": [243 {244 "extras": null,245 "description": "Original Poster",246 "user": {247 "id": 80744,248 "username": "Charitini_S",249 "name": "Charitini S",250 "avatar_template": "/user_avatar/discuss.pytorch.org/charitini_s/{size}/73099_2.png",251 "trust_level": 1252 }253 },254 {255 "extras": "latest",256 "description": "Most Recent Poster",257 "user": {258 "id": 80724,259 "username": "paulge",260 "name": "",261 "avatar_template": "/letter_avatar_proxy/v4/letter/p/82dd89/{size}.png",262 "trust_level": 2263 }264 }265 ]266 },267 {268 "fancy_title": "Install PyTorch 1.7",269 "id": 213881,270 "title": "Install PyTorch 1.7",271 "slug": "install-pytorch-1-7",272 "posts_count": 5,273 "reply_count": 3,274 "highest_post_number": 5,275 "image_url": "https://discuss.pytorch.org/uploads/default/optimized/3X/9/d/9d6121f6c5d76c51efced35dfb3949c6f5d304a3_2_1023x144.png",276 "created_at": "2024-12-05T22:40:54.483Z",277 "last_posted_at": "2024-12-06T16:09:25.133Z",278 "bumped": true,279 "bumped_at": "2024-12-06T16:58:31.417Z",280 "archetype": "regular",281 "unseen": false,282 "pinned": false,283 "unpinned": null,284 "visible": true,285 "closed": false,286 "archived": false,287 "bookmarked": null,288 "liked": null,289 "tags_descriptions": {},290 "like_count": 2,291 "views": 1244,292 "category_id": 1,293 "featured_link": null,294 "has_accepted_answer": true,295 "posters": [296 {297 "extras": "latest",298 "description": "Original Poster, Most Recent Poster",299 "user": {300 "id": 81343,301 "username": "adb57",302 "name": "",303 "avatar_template": "/letter_avatar_proxy/v4/letter/a/e9bcb4/{size}.png",304 "trust_level": 0305 }306 },307 {308 "extras": null,309 "description": "Frequent Poster, Accepted Answer",310 "user": {311 "id": 3534,312 "username": "ptrblck",313 "name": "",314 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",315 "admin": true,316 "moderator": true,317 "trust_level": 2318 }319 }320 ]321 },322 {323 "fancy_title": "Pytorch binaries with cuda version 12.2/12.0",324 "id": 215078,325 "title": "Pytorch binaries with cuda version 12.2/12.0",326 "slug": "pytorch-binaries-with-cuda-version-12-2-12-0",327 "posts_count": 2,328 "reply_count": 0,329 "highest_post_number": 2,330 "image_url": null,331 "created_at": "2025-01-07T15:02:32.838Z",332 "last_posted_at": "2025-01-07T15:43:32.219Z",333 "bumped": true,334 "bumped_at": "2025-01-07T15:43:32.219Z",335 "archetype": "regular",336 "unseen": false,337 "pinned": false,338 "unpinned": null,339 "visible": true,340 "closed": false,341 "archived": false,342 "bookmarked": null,343 "liked": null,344 "tags_descriptions": {},345 "like_count": 0,346 "views": 127,347 "category_id": 1,348 "featured_link": null,349 "has_accepted_answer": false,350 "posters": [351 {352 "extras": null,353 "description": "Original Poster",354 "user": {355 "id": 81925,356 "username": "gal_kesten",357 "name": "gal kesten",358 "avatar_template": "/user_avatar/discuss.pytorch.org/gal_kesten/{size}/74948_2.png",359 "trust_level": 0360 }361 },362 {363 "extras": "latest",364 "description": "Most Recent Poster",365 "user": {366 "id": 3534,367 "username": "ptrblck",368 "name": "",369 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",370 "admin": true,371 "moderator": true,372 "trust_level": 2373 }374 }375 ]376 },377 {378 "fancy_title": "Convert ONNX to PyTorch: TypeError: Conv2d.__init__() missing 2 required positional arguments: ‘in_channels’ and ‘out_channels’",379 "id": 215169,380 "title": "Convert ONNX to PyTorch: TypeError: Conv2d.__init__() missing 2 required positional arguments: 'in_channels' and 'out_channels'",381 "slug": "convert-onnx-to-pytorch-typeerror-conv2d-init-missing-2-required-positional-arguments-in-channels-and-out-channels",382 "posts_count": 3,383 "reply_count": 1,384 "highest_post_number": 3,385 "image_url": null,386 "created_at": "2025-01-09T12:45:15.734Z",387 "last_posted_at": "2025-01-10T13:45:36.292Z",388 "bumped": true,389 "bumped_at": "2025-01-10T13:45:36.292Z",390 "archetype": "regular",391 "unseen": false,392 "pinned": false,393 "unpinned": null,394 "visible": true,395 "closed": false,396 "archived": false,397 "bookmarked": null,398 "liked": null,399 "tags_descriptions": {},400 "like_count": 0,401 "views": 92,402 "category_id": 1,403 "featured_link": null,404 "has_accepted_answer": false,405 "posters": [406 {407 "extras": "latest",408 "description": "Original Poster, Most Recent Poster",409 "user": {410 "id": 65147,411 "username": "natalia_meira",412 "name": "Natalia Meira",413 "avatar_template": "/user_avatar/discuss.pytorch.org/natalia_meira/{size}/59369_2.png",414 "trust_level": 1415 }416 },417 {418 "extras": null,419 "description": "Frequent Poster",420 "user": {421 "id": 3534,422 "username": "ptrblck",423 "name": "",424 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",425 "admin": true,426 "moderator": true,427 "trust_level": 2428 }429 }430 ]431 },432 {433 "fancy_title": "No rule registered for HOP triton_kernel_wrapper_mutation",434 "id": 221024,435 "title": "No rule registered for HOP triton_kernel_wrapper_mutation",436 "slug": "no-rule-registered-for-hop-triton-kernel-wrapper-mutation",437 "posts_count": 1,438 "reply_count": 0,439 "highest_post_number": 1,440 "image_url": null,441 "created_at": "2025-06-24T16:56:43.946Z",442 "last_posted_at": "2025-06-24T16:56:43.989Z",443 "bumped": true,444 "bumped_at": "2025-06-24T16:56:43.989Z",445 "archetype": "regular",446 "unseen": false,447 "pinned": false,448 "unpinned": null,449 "visible": true,450 "closed": false,451 "archived": false,452 "bookmarked": null,453 "liked": null,454 "tags_descriptions": {},455 "like_count": 0,456 "views": 40,457 "category_id": 1,458 "featured_link": null,459 "has_accepted_answer": false,460 "posters": [461 {462 "extras": "latest single",463 "description": "Original Poster, Most Recent Poster",464 "user": {465 "id": 75001,466 "username": "KC314",467 "name": "Casey",468 "avatar_template": "/letter_avatar_proxy/v4/letter/k/df788c/{size}.png",469 "trust_level": 1470 }471 }472 ]473 }474 ],475 "tags_descriptions": {},476 "fancy_title": "Which parameters to pass in optimizer for transfer learning?",477 "id": 177283,478 "title": "Which parameters to pass in optimizer for transfer learning?",479 "posts_count": 3,480 "created_at": "2023-04-12T01:51:22.241Z",481 "views": 480,482 "reply_count": 1,483 "like_count": 0,484 "last_posted_at": "2023-04-12T15:38:04.016Z",485 "visible": true,486 "closed": false,487 "archived": false,488 "has_summary": false,489 "archetype": "regular",490 "slug": "which-parameters-to-pass-in-optimizer-for-transfer-learning",491 "category_id": 1,492 "word_count": 960,493 "deleted_at": null,494 "user_id": 62891,495 "featured_link": null,496 "pinned_globally": false,497 "pinned_at": null,498 "pinned_until": null,499 "image_url": null,500 "slow_mode_seconds": 0,501 "draft": null,502 "draft_key": "topic_177283",503 "draft_sequence": null,504 "unpinned": null,505 "pinned": false,506 "current_post_number": 1,507 "highest_post_number": 3,508 "deleted_by": null,509 "actions_summary": [510 {511 "id": 4,512 "count": 0,513 "hidden": false,514 "can_act": false515 },516 {517 "id": 8,518 "count": 0,519 "hidden": false,520 "can_act": false521 },522 {523 "id": 10,524 "count": 0,525 "hidden": false,526 "can_act": false527 },528 {529 "id": 7,530 "count": 0,531 "hidden": false,532 "can_act": false533 }534 ],535 "chunk_size": 20,536 "bookmarked": false,537 "topic_timer": null,538 "message_bus_last_id": 0,539 "participant_count": 2,540 "show_read_indicator": false,541 "thumbnails": null,542 "slow_mode_enabled_until": null,543 "can_vote": false,544 "vote_count": 0,545 "user_voted": false,546 "discourse_zendesk_plugin_zendesk_id": null,547 "discourse_zendesk_plugin_zendesk_url": "https://your-url.zendesk.com/agent/tickets/",548 "details": {549 "can_edit": false,550 "notification_level": 1,551 "participants": [552 {553 "id": 62891,554 "username": "Virat_Baahubali",555 "name": "",556 "avatar_template": "/letter_avatar_proxy/v4/letter/v/848f3c/{size}.png",557 "post_count": 2,558 "primary_group_name": null,559 "flair_name": null,560 "flair_url": null,561 "flair_color": null,562 "flair_bg_color": null,563 "flair_group_id": null,564 "trust_level": 0565 },566 {567 "id": 3534,568 "username": "ptrblck",569 "name": "",570 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",571 "post_count": 1,572 "primary_group_name": null,573 "flair_name": null,574 "flair_url": null,575 "flair_color": null,576 "flair_bg_color": null,577 "flair_group_id": null,578 "admin": true,579 "moderator": true,580 "trust_level": 2581 }582 ],583 "created_by": {584 "id": 62891,585 "username": "Virat_Baahubali",586 "name": "",587 "avatar_template": "/letter_avatar_proxy/v4/letter/v/848f3c/{size}.png"588 },589 "last_poster": {590 "id": 62891,591 "username": "Virat_Baahubali",592 "name": "",593 "avatar_template": "/letter_avatar_proxy/v4/letter/v/848f3c/{size}.png"594 },595 "links": [596 {597 "url": "https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html",598 "title": "Transfer Learning for Computer Vision Tutorial — PyTorch Tutorials 2.0.0+cu117 documentation",599 "internal": false,600 "attachment": false,601 "reflection": false,602 "clicks": 1,603 "user_id": 62891,604 "domain": "pytorch.org",605 "root_domain": "pytorch.org"606 },607 {608 "url": "https://pytorch.org/vision/main/models/generated/torchvision.models.convnext_small.html#torchvision.models.convnext_small",609 "title": "convnext_small — Torchvision main documentation",610 "internal": false,611 "attachment": false,612 "reflection": false,613 "clicks": 1,614 "user_id": 62891,615 "domain": "pytorch.org",616 "root_domain": "pytorch.org"617 }618 ]619 },620 "bookmarks": []621 },622 {623 "post_stream": {624 "posts": [625 {626 "id": 397149,627 "name": "",628 "username": "Vendrick17",629 "avatar_template": "/letter_avatar_proxy/v4/letter/v/db5fbb/{size}.png",630 "created_at": "2023-04-12T15:15:41.093Z",631 "cooked": "<p>Hello, I am trying to implement the function below and compute its gradient using <code>backward()</code>, but I have an in-place operation that I cannot solve, below I show an option I tried and didn’t work.</p>\n<pre><code class=\"lang-auto\">def modified_gram_schmidt(A):\n m, n = A.shape\n\n Q = torch.zeros((m, n), requires_grad=True).clone()\n R = torch.zeros((n, n), requires_grad=True).clone()\n\n for j in range(n):\n v = A[:, j].clone()\n for i in range(j):\n R[i, j] = torch.dot(Q[:, i], v)\n v_prev = v.clone()\n v = v_prev - torch.dot(Q[:, i], v_prev) * Q[:, i]\n R[j, j] = torch.norm(v)\n v = v / torch.norm(v)\n Q[:, j] = v.clone()\n\n return Q, R\n</code></pre>\n<p>The operation which is causing the issue is <code>v = v_prev - torch.dot(Q[:, i], v_prev) * Q[:, i]</code>, when I try to compute the gradient by <code>Q.sum().backward()</code> I have the error: <code>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation</code>. I would appreciate any help.</p>",632 "post_number": 1,633 "post_type": 1,634 "posts_count": 3,635 "updated_at": "2023-04-12T15:15:41.093Z",636 "reply_count": 0,637 "reply_to_post_number": null,638 "quote_count": 0,639 "incoming_link_count": 11,640 "reads": 6,641 "readers_count": 5,642 "score": 56.2,643 "yours": false,644 "topic_id": 177356,645 "topic_slug": "dealing-with-in-place-operation-in-custo-function",646 "display_username": "",647 "primary_group_name": null,648 "flair_name": null,649 "flair_url": null,650 "flair_bg_color": null,651 "flair_color": null,652 "flair_group_id": null,653 "badges_granted": [],654 "version": 1,655 "can_edit": false,656 "can_delete": false,657 "can_recover": false,658 "can_see_hidden_post": false,659 "can_wiki": false,660 "read": true,661 "user_title": "",662 "bookmarked": false,663 "actions_summary": [],664 "moderator": false,665 "admin": false,666 "staff": false,667 "user_id": 29167,668 "hidden": false,669 "trust_level": 1,670 "deleted_at": null,671 "user_deleted": false,672 "edit_reason": null,673 "can_view_edit_history": true,674 "wiki": false,675 "post_url": "/t/dealing-with-in-place-operation-in-custo-function/177356/1",676 "can_accept_answer": false,677 "can_unaccept_answer": false,678 "accepted_answer": false,679 "topic_accepted_answer": true,680 "can_vote": false681 },682 {683 "id": 397151,684 "name": "Alban D",685 "username": "albanD",686 "avatar_template": "/user_avatar/discuss.pytorch.org/alband/{size}/215_2.png",687 "created_at": "2023-04-12T15:23:02.299Z",688 "cooked": "<p>Hi,</p>\n<p>A simple solution is just to remove the inplace change into Q and only create it at the end:</p>\n<pre><code class=\"lang-auto\">def modified_gram_schmidt(A):\n m, n = A.shape\n\n Q = []\n R = torch.zeros((n, n), requires_grad=True).clone()\n\n for j in range(n):\n v = A[:, j].clone()\n for i in range(j):\n R[i, j] = torch.dot(Q[i], v)\n v_prev = v.clone()\n v = v_prev - torch.dot(Q[i], v_prev) * Q[i]\n R[j, j] = torch.norm(v)\n v = v / torch.norm(v)\n Q.append(v)\n\n return torch.stack(Q), R\n</code></pre>",689 "post_number": 2,690 "post_type": 1,691 "posts_count": 3,692 "updated_at": "2023-04-12T15:23:02.299Z",693 "reply_count": 1,694 "reply_to_post_number": null,695 "quote_count": 0,696 "incoming_link_count": 0,697 "reads": 6,698 "readers_count": 5,699 "score": 6.2,700 "yours": false,701 "topic_id": 177356,702 "topic_slug": "dealing-with-in-place-operation-in-custo-function",703 "display_username": "Alban D",704 "primary_group_name": null,705 "flair_name": null,706 "flair_url": null,707 "flair_bg_color": null,708 "flair_color": null,709 "flair_group_id": null,710 "badges_granted": [],711 "version": 1,712 "can_edit": false,713 "can_delete": false,714 "can_recover": false,715 "can_see_hidden_post": false,716 "can_wiki": false,717 "read": true,718 "user_title": "",719 "bookmarked": false,720 "actions_summary": [],721 "moderator": true,722 "admin": true,723 "staff": true,724 "user_id": 211,725 "hidden": false,726 "trust_level": 4,727 "deleted_at": null,728 "user_deleted": false,729 "edit_reason": null,730 "can_view_edit_history": true,731 "wiki": false,732 "post_url": "/t/dealing-with-in-place-operation-in-custo-function/177356/2",733 "can_accept_answer": false,734 "can_unaccept_answer": false,735 "accepted_answer": true,736 "topic_accepted_answer": true737 },738 {739 "id": 397152,740 "name": "",741 "username": "Vendrick17",742 "avatar_template": "/letter_avatar_proxy/v4/letter/v/db5fbb/{size}.png",743 "created_at": "2023-04-12T15:26:18.001Z",744 "cooked": "<aside class=\"quote no-group\" data-username=\"albanD\" data-post=\"2\" data-topic=\"177356\">\n<div class=\"title\">\n<div class=\"quote-controls\"></div>\n<img loading=\"lazy\" alt=\"\" width=\"24\" height=\"24\" src=\"https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png\" class=\"avatar\"> albanD:</div>\n<blockquote>\n<pre><code class=\"lang-auto\">def modified_gram_schmidt(A):\n m, n = A.shape\n\n Q = []\n R = torch.zeros((n, n), requires_grad=True).clone()\n\n for j in range(n):\n v = A[:, j].clone()\n for i in range(j):\n R[i, j] = torch.dot(Q[i], v)\n v_prev = v.clone()\n v = v_prev - torch.dot(Q[i], v_prev) * Q[i]\n R[j, j] = torch.norm(v)\n v = v / torch.norm(v)\n Q.append(v)\n\n return torch.stack(Q), R\n</code></pre>\n</blockquote>\n</aside>\n<p>Thank you so much for the quick reply, that solved the problem!</p>",745 "post_number": 3,746 "post_type": 1,747 "posts_count": 3,748 "updated_at": "2023-04-12T15:26:18.001Z",749 "reply_count": 0,750 "reply_to_post_number": 2,751 "quote_count": 1,752 "incoming_link_count": 1,753 "reads": 6,754 "readers_count": 5,755 "score": 6.2,756 "yours": false,757 "topic_id": 177356,758 "topic_slug": "dealing-with-in-place-operation-in-custo-function",759 "display_username": "",760 "primary_group_name": null,761 "flair_name": null,762 "flair_url": null,763 "flair_bg_color": null,764 "flair_color": null,765 "flair_group_id": null,766 "badges_granted": [],767 "version": 1,768 "can_edit": false,769 "can_delete": false,770 "can_recover": false,771 "can_see_hidden_post": false,772 "can_wiki": false,773 "read": true,774 "user_title": "",775 "bookmarked": false,776 "actions_summary": [],777 "moderator": false,778 "admin": false,779 "staff": false,780 "user_id": 29167,781 "hidden": false,782 "trust_level": 1,783 "deleted_at": null,784 "user_deleted": false,785 "edit_reason": null,786 "can_view_edit_history": true,787 "wiki": false,788 "post_url": "/t/dealing-with-in-place-operation-in-custo-function/177356/3",789 "can_accept_answer": false,790 "can_unaccept_answer": false,791 "accepted_answer": false,792 "topic_accepted_answer": true793 }794 ],795 "stream": [796 397149,797 397151,798 397152799 ]800 },801 "timeline_lookup": [802 [803 1,804 927805 ]806 ],807 "suggested_topics": [808 {809 "fancy_title": "`compile` a function with `autograd`",810 "id": 217385,811 "title": "`compile` a function with `autograd`",812 "slug": "compile-a-function-with-autograd",813 "posts_count": 2,814 "reply_count": 0,815 "highest_post_number": 2,816 "image_url": null,817 "created_at": "2025-03-03T15:38:12.611Z",818 "last_posted_at": "2025-03-05T22:47:12.081Z",819 "bumped": true,820 "bumped_at": "2025-03-05T22:47:12.081Z",821 "archetype": "regular",822 "unseen": false,823 "pinned": false,824 "unpinned": null,825 "visible": true,826 "closed": false,827 "archived": false,828 "bookmarked": null,829 "liked": null,830 "tags_descriptions": {},831 "like_count": 0,832 "views": 68,833 "category_id": 7,834 "featured_link": null,835 "has_accepted_answer": false,836 "posters": [837 {838 "extras": null,839 "description": "Original Poster",840 "user": {841 "id": 83042,842 "username": "madirbah",843 "name": "madirbah",844 "avatar_template": "/user_avatar/discuss.pytorch.org/madirbah/{size}/74610_2.png",845 "trust_level": 1846 }847 },848 {849 "extras": "latest",850 "description": "Most Recent Poster",851 "user": {852 "id": 41396,853 "username": "soulitzer",854 "name": "",855 "avatar_template": "/letter_avatar_proxy/v4/letter/s/839c29/{size}.png",856 "trust_level": 2857 }858 }859 ]860 },861 {862 "fancy_title": "Autograd for duplicated index put operation",863 "id": 214251,864 "title": "Autograd for duplicated index put operation",865 "slug": "autograd-for-duplicated-index-put-operation",866 "posts_count": 2,867 "reply_count": 0,868 "highest_post_number": 2,869 "image_url": null,870 "created_at": "2024-12-16T02:42:21.311Z",871 "last_posted_at": "2024-12-16T16:13:04.741Z",872 "bumped": true,873 "bumped_at": "2024-12-16T16:13:04.741Z",874 "archetype": "regular",875 "unseen": false,876 "pinned": false,877 "unpinned": null,878 "visible": true,879 "closed": false,880 "archived": false,881 "bookmarked": null,882 "liked": null,883 "tags_descriptions": {},884 "like_count": 0,885 "views": 170,886 "category_id": 7,887 "featured_link": null,888 "has_accepted_answer": false,889 "posters": [890 {891 "extras": null,892 "description": "Original Poster",893 "user": {894 "id": 81516,895 "username": "skpighhh",896 "name": "",897 "avatar_template": "/user_avatar/discuss.pytorch.org/skpighhh/{size}/74528_2.png",898 "trust_level": 0899 }900 },901 {902 "extras": "latest",903 "description": "Most Recent Poster",904 "user": {905 "id": 3534,906 "username": "ptrblck",907 "name": "",908 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",909 "admin": true,910 "moderator": true,911 "trust_level": 2912 }913 }914 ]915 },916 {917 "fancy_title": "Autograd graph traversal",918 "id": 213658,919 "title": "Autograd graph traversal",920 "slug": "autograd-graph-traversal",921 "posts_count": 3,922 "reply_count": 1,923 "highest_post_number": 3,924 "image_url": null,925 "created_at": "2024-12-01T13:15:22.254Z",926 "last_posted_at": "2024-12-18T23:35:26.182Z",927 "bumped": true,928 "bumped_at": "2024-12-18T23:35:26.182Z",929 "archetype": "regular",930 "unseen": false,931 "pinned": false,932 "unpinned": null,933 "visible": true,934 "closed": false,935 "archived": false,936 "bookmarked": null,937 "liked": null,938 "tags_descriptions": {},939 "like_count": 1,940 "views": 259,941 "category_id": 7,942 "featured_link": null,943 "has_accepted_answer": true,944 "posters": [945 {946 "extras": "latest",947 "description": "Original Poster, Most Recent Poster",948 "user": {949 "id": 81227,950 "username": "valerian.rey",951 "name": "Valérian Rey",952 "avatar_template": "/user_avatar/discuss.pytorch.org/valerian.rey/{size}/74283_2.png",953 "trust_level": 2954 }955 },956 {957 "extras": null,958 "description": "Frequent Poster, Accepted Answer",959 "user": {960 "id": 41396,961 "username": "soulitzer",962 "name": "",963 "avatar_template": "/letter_avatar_proxy/v4/letter/s/839c29/{size}.png",964 "trust_level": 2965 }966 }967 ]968 },969 {970 "fancy_title": "Calling autograd.Function in autograd.Function",971 "id": 216485,972 "title": "Calling autograd.Function in autograd.Function",973 "slug": "calling-autograd-function-in-autograd-function",974 "posts_count": 5,975 "reply_count": 3,976 "highest_post_number": 5,977 "image_url": null,978 "created_at": "2025-02-10T18:51:08.185Z",979 "last_posted_at": "2025-02-11T17:50:21.707Z",980 "bumped": true,981 "bumped_at": "2025-02-11T17:50:21.707Z",982 "archetype": "regular",983 "unseen": false,984 "pinned": false,985 "unpinned": null,986 "visible": true,987 "closed": false,988 "archived": false,989 "bookmarked": null,990 "liked": null,991 "tags_descriptions": {},992 "like_count": 1,993 "views": 170,994 "category_id": 7,995 "featured_link": null,996 "has_accepted_answer": true,997 "posters": [998 {999 "extras": "latest",1000 "description": "Original Poster, Most Recent Poster",1001 "user": {1002 "id": 68520,1003 "username": "HanGuo97",1004 "name": "Han Guo",1005 "avatar_template": "/user_avatar/discuss.pytorch.org/hanguo97/{size}/63011_2.png",1006 "trust_level": 11007 }1008 },1009 {1010 "extras": null,1011 "description": "Frequent Poster, Accepted Answer",1012 "user": {1013 "id": 41396,1014 "username": "soulitzer",1015 "name": "",1016 "avatar_template": "/letter_avatar_proxy/v4/letter/s/839c29/{size}.png",1017 "trust_level": 21018 }1019 }1020 ]1021 },1022 {1023 "fancy_title": "Making autograd saved tensors hooks specific to certain arguments",1024 "id": 218433,1025 "title": "Making autograd saved tensors hooks specific to certain arguments",1026 "slug": "making-autograd-saved-tensors-hooks-specific-to-certain-arguments",1027 "posts_count": 9,1028 "reply_count": 6,1029 "highest_post_number": 9,1030 "image_url": null,1031 "created_at": "2025-03-31T10:12:52.310Z",1032 "last_posted_at": "2025-09-17T07:44:20.765Z",1033 "bumped": true,1034 "bumped_at": "2025-09-17T07:44:20.765Z",1035 "archetype": "regular",1036 "unseen": false,1037 "pinned": false,1038 "unpinned": null,1039 "visible": true,1040 "closed": false,1041 "archived": false,1042 "bookmarked": null,1043 "liked": null,1044 "tags_descriptions": {},1045 "like_count": 0,1046 "views": 192,1047 "category_id": 7,1048 "featured_link": null,1049 "has_accepted_answer": true,1050 "posters": [1051 {1052 "extras": "latest",1053 "description": "Original Poster, Most Recent Poster, Accepted Answer",1054 "user": {1055 "id": 81725,1056 "username": "mseeger",1057 "name": null,1058 "avatar_template": "/letter_avatar_proxy/v4/letter/m/6bbea6/{size}.png",1059 "trust_level": 11060 }1061 },1062 {1063 "extras": null,1064 "description": "Frequent Poster",1065 "user": {1066 "id": 41396,1067 "username": "soulitzer",1068 "name": "",1069 "avatar_template": "/letter_avatar_proxy/v4/letter/s/839c29/{size}.png",1070 "trust_level": 21071 }1072 }1073 ]1074 }1075 ],1076 "tags_descriptions": {},1077 "fancy_title": "Dealing with in-place operation in custo function",1078 "id": 177356,1079 "title": "Dealing with in-place operation in custo function",1080 "posts_count": 3,1081 "created_at": "2023-04-12T15:15:40.980Z",1082 "views": 372,1083 "reply_count": 1,1084 "like_count": 0,1085 "last_posted_at": "2023-04-12T15:26:18.001Z",1086 "visible": true,1087 "closed": false,1088 "archived": false,1089 "has_summary": false,1090 "archetype": "regular",1091 "slug": "dealing-with-in-place-operation-in-custo-function",1092 "category_id": 7,1093 "word_count": 342,1094 "deleted_at": null,1095 "user_id": 29167,1096 "featured_link": null,1097 "pinned_globally": false,1098 "pinned_at": null,1099 "pinned_until": null,1100 "image_url": null,1101 "slow_mode_seconds": 0,1102 "draft": null,1103 "draft_key": "topic_177356",1104 "draft_sequence": null,1105 "unpinned": null,1106 "pinned": false,1107 "current_post_number": 1,1108 "highest_post_number": 3,1109 "deleted_by": null,1110 "actions_summary": [1111 {1112 "id": 4,1113 "count": 0,1114 "hidden": false,1115 "can_act": false1116 },1117 {1118 "id": 8,1119 "count": 0,1120 "hidden": false,1121 "can_act": false1122 },1123 {1124 "id": 10,1125 "count": 0,1126 "hidden": false,1127 "can_act": false1128 },1129 {1130 "id": 7,1131 "count": 0,1132 "hidden": false,1133 "can_act": false1134 }1135 ],1136 "chunk_size": 20,1137 "bookmarked": false,1138 "topic_timer": null,1139 "message_bus_last_id": 0,1140 "participant_count": 2,1141 "show_read_indicator": false,1142 "thumbnails": null,1143 "slow_mode_enabled_until": null,1144 "accepted_answer": {1145 "post_number": 2,1146 "username": "albanD",1147 "name": "Alban D",1148 "excerpt": "Hi, \nA simple solution is just to remove the inplace change into Q and only create it at the end: \ndef modified_gram_schmidt(A):\n m, n = A.shape\n\n Q = []\n R = torch.zeros((n, n), requires_grad=True).clone()\n\n for j in range(n):\n v = A[:, j].clone()\n for i in range(j):\n …"1149 },1150 "can_vote": false,1151 "vote_count": 0,1152 "user_voted": false,1153 "discourse_zendesk_plugin_zendesk_id": null,1154 "discourse_zendesk_plugin_zendesk_url": "https://your-url.zendesk.com/agent/tickets/",1155 "details": {1156 "can_edit": false,1157 "notification_level": 1,1158 "participants": [1159 {1160 "id": 29167,1161 "username": "Vendrick17",1162 "name": "",1163 "avatar_template": "/letter_avatar_proxy/v4/letter/v/db5fbb/{size}.png",1164 "post_count": 2,1165 "primary_group_name": null,1166 "flair_name": null,1167 "flair_url": null,1168 "flair_color": null,1169 "flair_bg_color": null,1170 "flair_group_id": null,1171 "trust_level": 11172 },1173 {1174 "id": 211,1175 "username": "albanD",1176 "name": "Alban D",1177 "avatar_template": "/user_avatar/discuss.pytorch.org/alband/{size}/215_2.png",1178 "post_count": 1,1179 "primary_group_name": null,1180 "flair_name": null,1181 "flair_url": null,1182 "flair_color": null,1183 "flair_bg_color": null,1184 "flair_group_id": null,1185 "admin": true,1186 "moderator": true,1187 "trust_level": 41188 }1189 ],1190 "created_by": {1191 "id": 29167,1192 "username": "Vendrick17",1193 "name": "",1194 "avatar_template": "/letter_avatar_proxy/v4/letter/v/db5fbb/{size}.png"1195 },1196 "last_poster": {1197 "id": 29167,1198 "username": "Vendrick17",1199 "name": "",1200 "avatar_template": "/letter_avatar_proxy/v4/letter/v/db5fbb/{size}.png"