Anurag1734/cuda-error-resolution-analysis
07
1[2 {3 "post_stream": {4 "posts": [5 {6 "id": 421501,7 "name": "abdelkrim zitouni",8 "username": "abdelkrim_zitouni",9 "avatar_template": "/user_avatar/discuss.pytorch.org/abdelkrim_zitouni/{size}/51976_2.png",10 "created_at": "2023-10-23T15:45:59.205Z",11 "cooked": "<p>Hello, I’m currently trying to wrap my Model which contains some frozen params with nested FSDP. I read from the <code>pytorch.distributed.fsdp</code> this warning:</p>\n<blockquote>\n<p>FSDP has some constraints on freezing parameters (i.e. setting <code>param.requires_grad=False</code> ). For <code>use_orig_params=False</code> , each FSDP instance must manage parameters that are all frozen or all non-frozen. For <code>use_orig_params=True</code> , FSDP supports mixing frozen and non-frozen, but we recommend not doing so since then the gradient memory usage will be higher than expected (namely, equivalent to not freezing those parameters). This means that ideally, frozen parameters should be isolated into their own <code>nn.Module</code> s and wrapped separately with FSDP.</p>\n</blockquote>\n<p>I didn’t want to set <code>use_orig_params=True</code> and as they stated, I Isolated the frozen parameters into their own <code>nn.Module</code> s and then wrapped every <strong>submodule</strong> with FSDP, but I’m still getting this error:</p>\n<p>ValueError: <code>FlatParameter</code> requires uniform <code>requires_grad</code></p>\n<p>Could someone please clarify what they meant by ‘ideally’ or help me overcome this issue?</p>\n<p>Thanks!</p>",12 "post_number": 1,13 "post_type": 1,14 "posts_count": 3,15 "updated_at": "2023-10-23T16:30:45.287Z",16 "reply_count": 0,17 "reply_to_post_number": null,18 "quote_count": 0,19 "incoming_link_count": 2063,20 "reads": 28,21 "readers_count": 27,22 "score": 10250.6,23 "yours": false,24 "topic_id": 190511,25 "topic_slug": "fsdp-with-non-uniform-requires-grad",26 "display_username": "abdelkrim zitouni",27 "primary_group_name": null,28 "flair_name": null,29 "flair_url": null,30 "flair_bg_color": null,31 "flair_color": null,32 "flair_group_id": null,33 "badges_granted": [],34 "version": 2,35 "can_edit": false,36 "can_delete": false,37 "can_recover": false,38 "can_see_hidden_post": false,39 "can_wiki": false,40 "read": true,41 "user_title": null,42 "bookmarked": false,43 "actions_summary": [44 {45 "id": 2,46 "count": 147 }48 ],49 "moderator": false,50 "admin": false,51 "staff": false,52 "user_id": 58178,53 "hidden": false,54 "trust_level": 1,55 "deleted_at": null,56 "user_deleted": false,57 "edit_reason": null,58 "can_view_edit_history": true,59 "wiki": false,60 "post_url": "/t/fsdp-with-non-uniform-requires-grad/190511/1",61 "can_accept_answer": false,62 "can_unaccept_answer": false,63 "accepted_answer": false,64 "topic_accepted_answer": true,65 "can_vote": false66 },67 {68 "id": 437499,69 "name": "Cal Mitchell",70 "username": "cmitch29",71 "avatar_template": "/letter_avatar_proxy/v4/letter/c/8edcca/{size}.png",72 "created_at": "2024-03-28T11:03:33.683Z",73 "cooked": "<p>Hi, did you ever figure this out? I am trying to do something similar.</p>",74 "post_number": 2,75 "post_type": 1,76 "posts_count": 3,77 "updated_at": "2024-03-28T11:03:33.683Z",78 "reply_count": 1,79 "reply_to_post_number": null,80 "quote_count": 0,81 "incoming_link_count": 28,82 "reads": 22,83 "readers_count": 21,84 "score": 164.4,85 "yours": false,86 "topic_id": 190511,87 "topic_slug": "fsdp-with-non-uniform-requires-grad",88 "display_username": "Cal Mitchell",89 "primary_group_name": null,90 "flair_name": null,91 "flair_url": null,92 "flair_bg_color": null,93 "flair_color": null,94 "flair_group_id": null,95 "badges_granted": [],96 "version": 1,97 "can_edit": false,98 "can_delete": false,99 "can_recover": false,100 "can_see_hidden_post": false,101 "can_wiki": false,102 "read": true,103 "user_title": null,104 "bookmarked": false,105 "actions_summary": [106 {107 "id": 2,108 "count": 1109 }110 ],111 "moderator": false,112 "admin": false,113 "staff": false,114 "user_id": 73953,115 "hidden": false,116 "trust_level": 1,117 "deleted_at": null,118 "user_deleted": false,119 "edit_reason": null,120 "can_view_edit_history": true,121 "wiki": false,122 "post_url": "/t/fsdp-with-non-uniform-requires-grad/190511/2",123 "can_accept_answer": false,124 "can_unaccept_answer": false,125 "accepted_answer": false,126 "topic_accepted_answer": true127 },128 {129 "id": 443558,130 "name": "abdelkrim zitouni",131 "username": "abdelkrim_zitouni",132 "avatar_template": "/user_avatar/discuss.pytorch.org/abdelkrim_zitouni/{size}/51976_2.png",133 "created_at": "2024-05-23T13:32:42.786Z",134 "cooked": "<p>Hi, sorry for the late reply. I overcame the issue by using a type-based auto-wrap policy, which lets you exclude the modules with frozen parameters in FSDP <code>auto_wrapper_callable</code>:</p>\n<pre data-code-wrap=\"py\"><code class=\"lang-py\"># Automatic wrapping sub-modules with inner FSDP\n auto_wrap_policy = None\n auto_wrapper_callable = None\n if FLAGS.auto_wrap_policy != \"none\":\n if FLAGS.auto_wrap_policy == \"size_based\":\n # auto-wrap all sub-modules with a minimum number of parameters (default 1e6)\n auto_wrap_policy = partial(\n size_based_auto_wrap_policy,\n min_num_params=int(float(FLAGS.auto_wrap_min_num_params)))\n elif FLAGS.auto_wrap_policy == \"type_based\":\n # auto-wrap all sub-modules in MBConv or FFN\n auto_wrap_policy = partial(\n transformer_auto_wrap_policy,\n transformer_layer_cls={\n Backbone.MBConv, # you can specify the modules you want to include here\n Backbone.FeedForward,\n })\n else:\n raise Exception(\n f\"Invalid auto-wrap policy: {FLAGS.auto_wrap_policy}\")\n if FLAGS.use_gradient_checkpointing:\n # Apply gradient checkpointing to auto-wrapped sub-modules if specified\n auto_wrapper_callable = lambda m, *args, **kwargs: FSDP(\n checkpoint_module(m), *args, **kwargs)\n\n def fsdp_wrap(m): return FSDP(\n m,\n compute_dtype=getattr(torch, FLAGS.compute_dtype),\n fp32_reduce_scatter=FLAGS.fp32_reduce_scatter,\n flatten_parameters=FLAGS.flatten_parameters,\n pin_layout_in_collective_ops=FLAGS.pin_layout_in_collective_ops,\n auto_wrap_policy=auto_wrap_policy,\n auto_wrapper_callable=auto_wrapper_callable)\n</code></pre>\n<p>I don’t know if this helps in your case.</p>",135 "post_number": 3,136 "post_type": 1,137 "posts_count": 3,138 "updated_at": "2024-05-23T13:32:42.786Z",139 "reply_count": 0,140 "reply_to_post_number": 2,141 "quote_count": 0,142 "incoming_link_count": 41,143 "reads": 17,144 "readers_count": 16,145 "score": 208.4,146 "yours": false,147 "topic_id": 190511,148 "topic_slug": "fsdp-with-non-uniform-requires-grad",149 "display_username": "abdelkrim zitouni",150 "primary_group_name": null,151 "flair_name": null,152 "flair_url": null,153 "flair_bg_color": null,154 "flair_color": null,155 "flair_group_id": null,156 "badges_granted": [],157 "version": 1,158 "can_edit": false,159 "can_delete": false,160 "can_recover": false,161 "can_see_hidden_post": false,162 "can_wiki": false,163 "read": true,164 "user_title": null,165 "reply_to_user": {166 "id": 73953,167 "username": "cmitch29",168 "name": "Cal Mitchell",169 "avatar_template": "/letter_avatar_proxy/v4/letter/c/8edcca/{size}.png"170 },171 "bookmarked": false,172 "actions_summary": [],173 "moderator": false,174 "admin": false,175 "staff": false,176 "user_id": 58178,177 "hidden": false,178 "trust_level": 1,179 "deleted_at": null,180 "user_deleted": false,181 "edit_reason": null,182 "can_view_edit_history": true,183 "wiki": false,184 "post_url": "/t/fsdp-with-non-uniform-requires-grad/190511/3",185 "can_accept_answer": false,186 "can_unaccept_answer": false,187 "accepted_answer": true,188 "topic_accepted_answer": true189 }190 ],191 "stream": [192 421501,193 437499,194 443558195 ]196 },197 "timeline_lookup": [198 [199 1,200 733201 ],202 [203 2,204 576205 ],206 [207 3,208 520209 ]210 ],211 "suggested_topics": [212 {213 "fancy_title": "Supporting Autograd for Collectives",214 "id": 219430,215 "title": "Supporting Autograd for Collectives",216 "slug": "supporting-autograd-for-collectives",217 "posts_count": 7,218 "reply_count": 2,219 "highest_post_number": 7,220 "image_url": null,221 "created_at": "2025-04-24T18:27:01.695Z",222 "last_posted_at": "2025-05-06T13:02:37.140Z",223 "bumped": true,224 "bumped_at": "2025-05-06T13:14:10.582Z",225 "archetype": "regular",226 "unseen": false,227 "pinned": false,228 "unpinned": null,229 "visible": true,230 "closed": false,231 "archived": false,232 "bookmarked": null,233 "liked": null,234 "tags_descriptions": {},235 "like_count": 8,236 "views": 410,237 "category_id": 12,238 "featured_link": null,239 "has_accepted_answer": false,240 "posters": [241 {242 "extras": null,243 "description": "Original Poster",244 "user": {245 "id": 55883,246 "username": "wconstab",247 "name": "Will Constable",248 "avatar_template": "/user_avatar/discuss.pytorch.org/wconstab/{size}/49551_2.png",249 "trust_level": 2250 }251 },252 {253 "extras": null,254 "description": "Frequent Poster",255 "user": {256 "id": 42875,257 "username": "Chillee",258 "name": "Horace He",259 "avatar_template": "/user_avatar/discuss.pytorch.org/chillee/{size}/35574_2.png",260 "trust_level": 2261 }262 },263 {264 "extras": null,265 "description": "Frequent Poster",266 "user": {267 "id": 13180,268 "username": "wanchaol",269 "name": "Wanchao",270 "avatar_template": "/user_avatar/discuss.pytorch.org/wanchaol/{size}/50343_2.png",271 "trust_level": 2272 }273 },274 {275 "extras": "latest",276 "description": "Most Recent Poster",277 "user": {278 "id": 78505,279 "username": "tianyu",280 "name": "",281 "avatar_template": "/user_avatar/discuss.pytorch.org/tianyu/{size}/72368_2.png",282 "trust_level": 2283 }284 }285 ]286 },287 {288 "fancy_title": "Torch.distributed.barrier occupies additional CUDA memory",289 "id": 212502,290 "title": "Torch.distributed.barrier occupies additional CUDA memory",291 "slug": "torch-distributed-barrier-occupies-additional-cuda-memory",292 "posts_count": 1,293 "reply_count": 0,294 "highest_post_number": 1,295 "image_url": null,296 "created_at": "2024-11-04T12:49:06.441Z",297 "last_posted_at": "2024-11-04T12:49:06.619Z",298 "bumped": true,299 "bumped_at": "2024-11-04T12:49:06.619Z",300 "archetype": "regular",301 "unseen": false,302 "pinned": false,303 "unpinned": null,304 "visible": true,305 "closed": false,306 "archived": false,307 "bookmarked": null,308 "liked": null,309 "tags_descriptions": {},310 "like_count": 0,311 "views": 99,312 "category_id": 12,313 "featured_link": null,314 "has_accepted_answer": false,315 "posters": [316 {317 "extras": "latest single",318 "description": "Original Poster, Most Recent Poster",319 "user": {320 "id": 48526,321 "username": "yuan_wentao",322 "name": "yuan wentao",323 "avatar_template": "/user_avatar/discuss.pytorch.org/yuan_wentao/{size}/41715_2.png",324 "trust_level": 1325 }326 }327 ]328 },329 {330 "fancy_title": "Torchrun assigns same LOCAL_RANK to processes sharing node",331 "id": 212527,332 "title": "Torchrun assigns same LOCAL_RANK to processes sharing node",333 "slug": "torchrun-assigns-same-local-rank-to-processes-sharing-node",334 "posts_count": 2,335 "reply_count": 0,336 "highest_post_number": 2,337 "image_url": null,338 "created_at": "2024-11-05T00:34:50.635Z",339 "last_posted_at": "2024-11-13T10:56:57.379Z",340 "bumped": true,341 "bumped_at": "2024-11-13T10:56:57.379Z",342 "archetype": "regular",343 "unseen": false,344 "pinned": false,345 "unpinned": null,346 "visible": true,347 "closed": false,348 "archived": false,349 "bookmarked": null,350 "liked": null,351 "tags_descriptions": {},352 "like_count": 0,353 "views": 213,354 "category_id": 12,355 "featured_link": null,356 "has_accepted_answer": false,357 "posters": [358 {359 "extras": null,360 "description": "Original Poster",361 "user": {362 "id": 80689,363 "username": "chazm",364 "name": "Chaz Maschman",365 "avatar_template": "/user_avatar/discuss.pytorch.org/chazm/{size}/73772_2.png",366 "trust_level": 1367 }368 },369 {370 "extras": "latest",371 "description": "Most Recent Poster",372 "user": {373 "id": 80317,374 "username": "tj.solergibert",375 "name": "Antoni-Joan Solergibert",376 "avatar_template": "/letter_avatar_proxy/v4/letter/t/8491ac/{size}.png",377 "trust_level": 1378 }379 }380 ]381 },382 {383 "fancy_title": "What’s recommended way to intergrate FSDP with Customize Tensor Unit",384 "id": 215972,385 "title": "What's recommended way to intergrate FSDP with Customize Tensor Unit",386 "slug": "whats-recommended-way-to-intergrate-fsdp-with-customize-tensor-unit",387 "posts_count": 2,388 "reply_count": 0,389 "highest_post_number": 2,390 "image_url": "https://discuss.pytorch.org/uploads/default/optimized/3X/e/3/e3e984f7517f52631d72d097b7be74ee3481740f_2_1024x713.jpeg",391 "created_at": "2025-01-28T03:18:15.256Z",392 "last_posted_at": "2025-01-28T03:19:17.425Z",393 "bumped": true,394 "bumped_at": "2025-01-28T03:19:17.425Z",395 "archetype": "regular",396 "unseen": false,397 "pinned": false,398 "unpinned": null,399 "visible": true,400 "closed": false,401 "archived": false,402 "bookmarked": null,403 "liked": null,404 "tags_descriptions": {},405 "like_count": 0,406 "views": 134,407 "category_id": 12,408 "featured_link": null,409 "has_accepted_answer": false,410 "posters": [411 {412 "extras": "latest single",413 "description": "Original Poster, Most Recent Poster",414 "user": {415 "id": 82358,416 "username": "Irvingwangjr",417 "name": "wang",418 "avatar_template": "/user_avatar/discuss.pytorch.org/irvingwangjr/{size}/75341_2.png",419 "trust_level": 1420 }421 }422 ]423 },424 {425 "fancy_title": "Why would functional and non-functional broadcast use `src` with different semantics?",426 "id": 221721,427 "title": "Why would functional and non-functional broadcast use `src` with different semantics?",428 "slug": "why-would-functional-and-non-functional-broadcast-use-src-with-different-semantics",429 "posts_count": 2,430 "reply_count": 0,431 "highest_post_number": 2,432 "image_url": null,433 "created_at": "2025-07-22T05:58:18.477Z",434 "last_posted_at": "2025-07-26T00:44:19.892Z",435 "bumped": true,436 "bumped_at": "2025-07-26T00:44:19.892Z",437 "archetype": "regular",438 "unseen": false,439 "pinned": false,440 "unpinned": null,441 "visible": true,442 "closed": false,443 "archived": false,444 "bookmarked": null,445 "liked": null,446 "tags_descriptions": {},447 "like_count": 1,448 "views": 38,449 "category_id": 12,450 "featured_link": null,451 "has_accepted_answer": true,452 "posters": [453 {454 "extras": null,455 "description": "Original Poster",456 "user": {457 "id": 85181,458 "username": "hankzhwang",459 "name": "Zhanghan Wang",460 "avatar_template": "/user_avatar/discuss.pytorch.org/hankzhwang/{size}/75303_2.png",461 "trust_level": 0462 }463 },464 {465 "extras": "latest",466 "description": "Most Recent Poster, Accepted Answer",467 "user": {468 "id": 39542,469 "username": "H-Huang",470 "name": "Howard Huang",471 "avatar_template": "/user_avatar/discuss.pytorch.org/h-huang/{size}/35598_2.png",472 "trust_level": 2473 }474 }475 ]476 }477 ],478 "tags_descriptions": {},479 "fancy_title": "FSDP with non-uniform ‘requires_grad’",480 "id": 190511,481 "title": "FSDP with non-uniform 'requires_grad'",482 "posts_count": 3,483 "created_at": "2023-10-23T15:45:59.155Z",484 "views": 2149,485 "reply_count": 1,486 "like_count": 2,487 "last_posted_at": "2024-05-23T13:32:42.786Z",488 "visible": true,489 "closed": false,490 "archived": false,491 "has_summary": false,492 "archetype": "regular",493 "slug": "fsdp-with-non-uniform-requires-grad",494 "category_id": 12,495 "word_count": 351,496 "deleted_at": null,497 "user_id": 58178,498 "featured_link": null,499 "pinned_globally": false,500 "pinned_at": null,501 "pinned_until": null,502 "image_url": null,503 "slow_mode_seconds": 0,504 "draft": null,505 "draft_key": "topic_190511",506 "draft_sequence": null,507 "unpinned": null,508 "pinned": false,509 "current_post_number": 1,510 "highest_post_number": 3,511 "deleted_by": null,512 "actions_summary": [513 {514 "id": 4,515 "count": 0,516 "hidden": false,517 "can_act": false518 },519 {520 "id": 8,521 "count": 0,522 "hidden": false,523 "can_act": false524 },525 {526 "id": 10,527 "count": 0,528 "hidden": false,529 "can_act": false530 },531 {532 "id": 7,533 "count": 0,534 "hidden": false,535 "can_act": false536 }537 ],538 "chunk_size": 20,539 "bookmarked": false,540 "topic_timer": null,541 "message_bus_last_id": 0,542 "participant_count": 2,543 "show_read_indicator": false,544 "thumbnails": null,545 "slow_mode_enabled_until": null,546 "accepted_answer": {547 "post_number": 3,548 "username": "abdelkrim_zitouni",549 "name": "abdelkrim zitouni",550 "excerpt": "Hi, sorry for the late reply. I overcame the issue by using a type-based auto-wrap policy, which lets you exclude the modules with frozen parameters in FSDP auto_wrapper_callable: \n# Automatic wrapping sub-modules with inner FSDP\n auto_wrap_policy = None\n auto_wrapper_callable = None\n if FL…"551 },552 "can_vote": false,553 "vote_count": 0,554 "user_voted": false,555 "discourse_zendesk_plugin_zendesk_id": null,556 "discourse_zendesk_plugin_zendesk_url": "https://your-url.zendesk.com/agent/tickets/",557 "details": {558 "can_edit": false,559 "notification_level": 1,560 "participants": [561 {562 "id": 58178,563 "username": "abdelkrim_zitouni",564 "name": "abdelkrim zitouni",565 "avatar_template": "/user_avatar/discuss.pytorch.org/abdelkrim_zitouni/{size}/51976_2.png",566 "post_count": 2,567 "primary_group_name": null,568 "flair_name": null,569 "flair_url": null,570 "flair_color": null,571 "flair_bg_color": null,572 "flair_group_id": null,573 "trust_level": 1574 },575 {576 "id": 73953,577 "username": "cmitch29",578 "name": "Cal Mitchell",579 "avatar_template": "/letter_avatar_proxy/v4/letter/c/8edcca/{size}.png",580 "post_count": 1,581 "primary_group_name": null,582 "flair_name": null,583 "flair_url": null,584 "flair_color": null,585 "flair_bg_color": null,586 "flair_group_id": null,587 "trust_level": 1588 }589 ],590 "created_by": {591 "id": 58178,592 "username": "abdelkrim_zitouni",593 "name": "abdelkrim zitouni",594 "avatar_template": "/user_avatar/discuss.pytorch.org/abdelkrim_zitouni/{size}/51976_2.png"595 },596 "last_poster": {597 "id": 58178,598 "username": "abdelkrim_zitouni",599 "name": "abdelkrim zitouni",600 "avatar_template": "/user_avatar/discuss.pytorch.org/abdelkrim_zitouni/{size}/51976_2.png"601 }602 },603 "bookmarks": []604 },605 {606 "post_stream": {607 "posts": [608 {609 "id": 411766,610 "name": "",611 "username": "dirkcremers",612 "avatar_template": "/user_avatar/discuss.pytorch.org/dirkcremers/{size}/59197_2.png",613 "created_at": "2023-07-28T10:05:41.406Z",614 "cooked": "<p>Hi,</p>\n<p>I am currently working on implementing a Graph Neural Network (GNN) for a regression task aimed at predicting revenue. The dataset consists of 43 features describing the problem setting, and based on these features, the goal is to predict the revenue.</p>\n<p>Initially, I explored traditional regression models like Ordinary Least Squares (OLS) and Random Forest, both of which yielded respectable results with R-squared scores of 0.8 and 0.9, respectively. While these models provided valuable insights into the features, I decided to explore GNNs due to the unique characteristics of the problem.</p>\n<p>The GNN approach involves modeling the problem as a graph, which seems well-suited to capture the inherent relationships among the features. However, my initial attempts at using a GNN resulted in predictions far from the true values, and the loss remained relatively constant.</p>\n<p>Below, I provide an overview of the procedure, the GNN model, and some experiments I have conducted so far:</p>\n<ol>\n<li><strong>Data Preprocessing</strong>: I applied standard scaling to the data and split it into training and test sets.</li>\n<li><strong>Data Loader</strong>: I created a data loader using <code>torch_geometric.data</code>, which includes node features, edge_index, and edge_features.</li>\n<li><strong>GAT Model Definition</strong>: I defined the Graph Attention Network (GAT) model as follows:</li>\n</ol>\n<pre><code class=\"lang-auto\">class GAT(torch.nn.Module):\n \"\"\"Graph Attention Network\"\"\"\n def __init__(self, num_features, num_classes, hidden_channels, heads=1, edge_dimension=29):\n super().__init__()\n self.gat1 = GATv2Conv(num_features, hidden_channels, heads=heads, edge_dim=edge_dimension)\n\n self.linear1 = nn.Linear(heads*hidden_channels, hidden_channels)\n self.linear2 = nn.Linear(hidden_channels, hidden_channels)\n self.linear3 = nn.Linear(hidden_channels, hidden_channels)\n\n self.output = nn.Linear(hidden_channels, num_classes)\n\n def forward(self, x, edge_index, edge_attr):\n\n x = self.gat1(x, edge_index, edge_attr=edge_attr)\n x = x.relu()\n\n x= self.linear1(x)\n x = x.relu()\n\n x = self.linear2(x)\n x = x.relu()\n x = F.dropout(x, p=0.1, training=self.training)\n\n x = self.linear3(x)\n x = x.relu()\n\n x = self.output(x)\n return x\n</code></pre>\n<ol start=\"4\">\n<li><strong>Training Procedure</strong> : I implemented the training procedure using Mean Squared Error (MSE) loss and the Adam optimizer.</li>\n</ol>\n<pre><code class=\"lang-auto\">loss_func = nn.MSELoss()\noptimizer = torch.optim.Adam(model_gat.parameters(), lr=0.001) #, weight_decay=5e-4)\n\ndef train_gat():\n model_gat.train() # set the model to training 'mode' (i.e., apply dropout)\n total_loss = []\n for data in train_loader:\n data.to(device)\n optimizer.zero_grad()\n\n out = model_gat(data.node, data.edge_index, data.edge_features)\n\n loss = loss_func(out, data.y)\n loss.backward() # derive gradients\n optimizer.step() # update all parameters based on the gradients\n total_loss.append(loss)\n return sum(total_loss)/len(total_loss)\n\ndef test_gat():\n model_gat.eval()\n test_loss = []\n with torch.no_grad():\n for data in test_loader:\n data.to(device)\n out = model_gat(data.node, data.edge_index, data.edge_features) # propagate the data through the model\n test_loss.append(loss_func(out, data.y)) # create a tensor that evaluates whether predictions were correct\n return sum(test_loss)/len(test_loss), out, data.y\n\n\ntrain_losses = []\nepochs = 15\nfor epoch in range(1, epochs+1):\n loss = train_gat() # do one training step over the entire dataset\n print(f'Epoch: {epoch:03d}, Loss: {loss:.4f}')\n train_losses.append(loss.cpu().detach().numpy()) # save accuracies so we can plot them\n\ntest_loss, pred, true = test_gat()\nprint(test_loss)\n\nprint(pred, true)\n</code></pre>\n<p>Despite several attempts to improve the GNN’s performance, such as adjusting the learning rate, using different loss functions, and increasing the model’s size, the results have not improved significantly. Additionally, I experimented with a smaller dataset containing only three observations to check if the model could overfit, but it was not successful.</p>\n<p>Furthermore, as a comparison, I implemented a Multi-Layer Perceptron (MLP), but it also failed to produce satisfactory results. (Adjusting the data loader accordingly)</p>\n<pre><code class=\"lang-auto\">class NN(torch.nn.Module):\n \"\"\"Graph Attention Network with Batch Normalization\"\"\"\n def __init__(self, num_features, num_classes):\n super().__init__()\n\n self.linear1 = nn.Linear(num_features, 300)\n self.batch_norm1 = nn.BatchNorm1d(300)\n self.linear2 = nn.Linear(300, 50)\n self.batch_norm2 = nn.BatchNorm1d(50)\n self.linear3 = nn.Linear(50, 20)\n self.batch_norm3 = nn.BatchNorm1d(20)\n\n self.output = nn.Linear(20, num_classes)\n\n def forward(self, x):\n x = self.linear1(x)\n x = self.batch_norm1(x)\n x = F.relu(x)\n\n x = self.linear2(x)\n x = self.batch_norm2(x)\n x = F.relu(x)\n\n x = self.linear3(x)\n x = self.batch_norm3(x)\n x = F.relu(x)\n\n x = self.output(x)\n return x\n</code></pre>\n<p>At this point, I am seeking recommendations or insights into why the GNN approach may not be working as expected. Any help or suggestions would be greatly appreciated.</p>",615 "post_number": 1,616 "post_type": 1,617 "posts_count": 2,618 "updated_at": "2023-07-28T10:05:41.406Z",619 "reply_count": 0,620 "reply_to_post_number": null,621 "quote_count": 0,622 "incoming_link_count": 631,623 "reads": 10,624 "readers_count": 9,625 "score": 3152.0,626 "yours": false,627 "topic_id": 185246,628 "topic_slug": "graph-neural-network-for-regression-purpose-not-updating",629 "display_username": "",630 "primary_group_name": null,631 "flair_name": null,632 "flair_url": null,633 "flair_bg_color": null,634 "flair_color": null,635 "flair_group_id": null,636 "badges_granted": [],637 "version": 1,638 "can_edit": false,639 "can_delete": false,640 "can_recover": false,641 "can_see_hidden_post": false,642 "can_wiki": false,643 "read": true,644 "user_title": null,645 "bookmarked": false,646 "actions_summary": [],647 "moderator": false,648 "admin": false,649 "staff": false,650 "user_id": 65001,651 "hidden": false,652 "trust_level": 1,653 "deleted_at": null,654 "user_deleted": false,655 "edit_reason": null,656 "can_view_edit_history": true,657 "wiki": false,658 "post_url": "/t/graph-neural-network-for-regression-purpose-not-updating/185246/1",659 "can_accept_answer": false,660 "can_unaccept_answer": false,661 "accepted_answer": false,662 "topic_accepted_answer": null,663 "can_vote": false664 },665 {666 "id": 443555,667 "name": "elham gholamzade",668 "username": "elham_gholamzade",669 "avatar_template": "/user_avatar/discuss.pytorch.org/elham_gholamzade/{size}/70223_2.png",670 "created_at": "2024-05-23T13:09:52.456Z",671 "cooked": "<p>I have faced exactly the same problem as you. Could you please help me? How can I connect with you?</p>",672 "post_number": 2,673 "post_type": 1,674 "posts_count": 2,675 "updated_at": "2024-05-23T13:09:52.456Z",676 "reply_count": 0,677 "reply_to_post_number": null,678 "quote_count": 0,679 "incoming_link_count": 1,680 "reads": 4,681 "readers_count": 3,682 "score": 5.8,683 "yours": false,684 "topic_id": 185246,685 "topic_slug": "graph-neural-network-for-regression-purpose-not-updating",686 "display_username": "elham gholamzade",687 "primary_group_name": null,688 "flair_name": null,689 "flair_url": null,690 "flair_bg_color": null,691 "flair_color": null,692 "flair_group_id": null,693 "badges_granted": [],694 "version": 1,695 "can_edit": false,696 "can_delete": false,697 "can_recover": false,698 "can_see_hidden_post": false,699 "can_wiki": false,700 "read": true,701 "user_title": null,702 "bookmarked": false,703 "actions_summary": [],704 "moderator": false,705 "admin": false,706 "staff": false,707 "user_id": 76055,708 "hidden": false,709 "trust_level": 0,710 "deleted_at": null,711 "user_deleted": false,712 "edit_reason": null,713 "can_view_edit_history": true,714 "wiki": false,715 "post_url": "/t/graph-neural-network-for-regression-purpose-not-updating/185246/2",716 "can_accept_answer": false,717 "can_unaccept_answer": false,718 "accepted_answer": false,719 "topic_accepted_answer": null720 }721 ],722 "stream": [723 411766,724 443555725 ]726 },727 "timeline_lookup": [728 [729 1,730 820731 ],732 [733 2,734 520735 ]736 ],737 "suggested_topics": [738 {739 "fancy_title": "How to resolve with the issue of adjusting the gaps in my ‘time_idx’ feature in my stock market dataset?",740 "id": 213245,741 "title": "How to resolve with the issue of adjusting the gaps in my 'time_idx' feature in my stock market dataset?",742 "slug": "how-to-resolve-with-the-issue-of-adjusting-the-gaps-in-my-time-idx-feature-in-my-stock-market-dataset",743 "posts_count": 1,744 "reply_count": 0,745 "highest_post_number": 1,746 "image_url": null,747 "created_at": "2024-11-21T05:01:10.859Z",748 "last_posted_at": "2024-11-21T05:01:10.919Z",749 "bumped": true,750 "bumped_at": "2024-11-21T05:01:10.919Z",751 "archetype": "regular",752 "unseen": false,753 "pinned": false,754 "unpinned": null,755 "visible": true,756 "closed": false,757 "archived": false,758 "bookmarked": null,759 "liked": null,760 "tags_descriptions": {},761 "like_count": 0,762 "views": 26,763 "category_id": 1,764 "featured_link": null,765 "has_accepted_answer": false,766 "posters": [767 {768 "extras": "latest single",769 "description": "Original Poster, Most Recent Poster",770 "user": {771 "id": 28027,772 "username": "Satyam_sharma",773 "name": "Satyam sharma",774 "avatar_template": "/user_avatar/discuss.pytorch.org/satyam_sharma/{size}/20898_2.png",775 "trust_level": 0776 }777 }778 ]779 },780 {781 "fancy_title": "RuntimeError: CUDA error: invalid argument",782 "id": 214062,783 "title": "RuntimeError: CUDA error: invalid argument",784 "slug": "runtimeerror-cuda-error-invalid-argument",785 "posts_count": 9,786 "reply_count": 2,787 "highest_post_number": 9,788 "image_url": null,789 "created_at": "2024-12-10T15:39:59.445Z",790 "last_posted_at": "2025-08-02T10:44:13.170Z",791 "bumped": true,792 "bumped_at": "2025-08-02T10:44:13.170Z",793 "archetype": "regular",794 "unseen": false,795 "pinned": false,796 "unpinned": null,797 "visible": true,798 "closed": false,799 "archived": false,800 "bookmarked": null,801 "liked": null,802 "tags_descriptions": {},803 "like_count": 2,804 "views": 1206,805 "category_id": 1,806 "featured_link": null,807 "has_accepted_answer": false,808 "posters": [809 {810 "extras": null,811 "description": "Original Poster",812 "user": {813 "id": 81431,814 "username": "Andymulb",815 "name": "Andreas",816 "avatar_template": "/user_avatar/discuss.pytorch.org/andymulb/{size}/74455_2.png",817 "trust_level": 1818 }819 },820 {821 "extras": null,822 "description": "Frequent Poster",823 "user": {824 "id": 81449,825 "username": "NailikLN",826 "name": "kilian le nezet",827 "avatar_template": "/user_avatar/discuss.pytorch.org/nailikln/{size}/74471_2.png",828 "trust_level": 0829 }830 },831 {832 "extras": null,833 "description": "Frequent Poster",834 "user": {835 "id": 81492,836 "username": "Doruk_Sonmez",837 "name": "Doruk Sönmez",838 "avatar_template": "/user_avatar/discuss.pytorch.org/doruk_sonmez/{size}/74506_2.png",839 "trust_level": 1840 }841 },842 {843 "extras": null,844 "description": "Frequent Poster",845 "user": {846 "id": 80375,847 "username": "vixp24",848 "name": "",849 "avatar_template": "/user_avatar/discuss.pytorch.org/vixp24/{size}/73472_2.png",850 "trust_level": 1851 }852 },853 {854 "extras": "latest",855 "description": "Most Recent Poster",856 "user": {857 "id": 82530,858 "username": "J4Q8",859 "name": "Jakub Łucki",860 "avatar_template": "/user_avatar/discuss.pytorch.org/j4q8/{size}/75512_2.png",861 "trust_level": 1862 }863 }864 ]865 },866 {867 "fancy_title": "Best way to find threshold of multilabel action recognition after using MultilabelAveragePrecision",868 "id": 214501,869 "title": "Best way to find threshold of multilabel action recognition after using MultilabelAveragePrecision",870 "slug": "best-way-to-find-threshold-of-multilabel-action-recognition-after-using-multilabelaverageprecision",871 "posts_count": 1,872 "reply_count": 0,873 "highest_post_number": 1,874 "image_url": null,875 "created_at": "2024-12-21T14:51:22.097Z",876 "last_posted_at": "2024-12-21T14:51:22.130Z",877 "bumped": true,878 "bumped_at": "2024-12-21T14:51:22.130Z",879 "archetype": "regular",880 "unseen": false,881 "pinned": false,882 "unpinned": null,883 "visible": true,884 "closed": false,885 "archived": false,886 "bookmarked": null,887 "liked": null,888 "tags_descriptions": {},889 "like_count": 0,890 "views": 31,891 "category_id": 1,892 "featured_link": null,893 "has_accepted_answer": false,894 "posters": [895 {896 "extras": "latest single",897 "description": "Original Poster, Most Recent Poster",898 "user": {899 "id": 81641,900 "username": "Edoardo_Fazzari",901 "name": "Edoardo Fazzari",902 "avatar_template": "/user_avatar/discuss.pytorch.org/edoardo_fazzari/{size}/74661_2.png",903 "trust_level": 0904 }905 }906 ]907 },908 {909 "fancy_title": "Is Intel® Iris Xe Graphics compatible with torch==2.6.0?",910 "id": 216505,911 "title": "Is Intel® Iris Xe Graphics compatible with torch==2.6.0?",912 "slug": "is-intel-iris-xe-graphics-compatible-with-torch-2-6-0",913 "posts_count": 1,914 "reply_count": 0,915 "highest_post_number": 1,916 "image_url": null,917 "created_at": "2025-02-11T01:44:39.467Z",918 "last_posted_at": "2025-02-11T01:44:39.519Z",919 "bumped": true,920 "bumped_at": "2025-02-11T01:44:39.519Z",921 "archetype": "regular",922 "unseen": false,923 "pinned": false,924 "unpinned": null,925 "visible": true,926 "closed": false,927 "archived": false,928 "bookmarked": null,929 "liked": null,930 "tags_descriptions": {},931 "like_count": 0,932 "views": 166,933 "category_id": 1,934 "featured_link": null,935 "has_accepted_answer": false,936 "posters": [937 {938 "extras": "latest single",939 "description": "Original Poster, Most Recent Poster",940 "user": {941 "id": 82603,942 "username": "marceloprates",943 "name": "Marcelo de Oliveira Rosa Prates",944 "avatar_template": "/user_avatar/discuss.pytorch.org/marceloprates/{size}/75578_2.png",945 "trust_level": 0946 }947 }948 ]949 },950 {951 "fancy_title": "Why is it not necessary to set requires_grad=True in a class subclassing nn.Module",952 "id": 217003,953 "title": "Why is it not necessary to set requires_grad=True in a class subclassing nn.Module",954 "slug": "why-is-it-not-necessary-to-set-requires-grad-true-in-a-class-subclassing-nn-module",955 "posts_count": 2,956 "reply_count": 0,957 "highest_post_number": 2,958 "image_url": null,959 "created_at": "2025-02-21T18:42:45.833Z",960 "last_posted_at": "2025-02-22T00:05:46.040Z",961 "bumped": true,962 "bumped_at": "2025-02-22T00:05:46.040Z",963 "archetype": "regular",964 "unseen": false,965 "pinned": false,966 "unpinned": null,967 "visible": true,968 "closed": false,969 "archived": false,970 "bookmarked": null,971 "liked": null,972 "tags_descriptions": {},973 "like_count": 0,974 "views": 36,975 "category_id": 1,976 "featured_link": null,977 "has_accepted_answer": false,978 "posters": [979 {980 "extras": null,981 "description": "Original Poster",982 "user": {983 "id": 82705,984 "username": "make1234",985 "name": "",986 "avatar_template": "/user_avatar/discuss.pytorch.org/make1234/{size}/74826_2.png",987 "trust_level": 1988 }989 },990 {991 "extras": "latest",992 "description": "Most Recent Poster",993 "user": {994 "id": 3534,995 "username": "ptrblck",996 "name": "",997 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",998 "admin": true,999 "moderator": true,1000 "trust_level": 21001 }1002 }1003 ]1004 }1005 ],1006 "tags_descriptions": {},1007 "fancy_title": "Graph Neural Network for regression purpose not updating",1008 "id": 185246,1009 "title": "Graph Neural Network for regression purpose not updating",1010 "posts_count": 2,1011 "created_at": "2023-07-28T10:05:41.312Z",1012 "views": 884,1013 "reply_count": 0,1014 "like_count": 0,1015 "last_posted_at": "2024-05-23T13:09:52.456Z",1016 "visible": true,1017 "closed": false,1018 "archived": false,1019 "has_summary": false,1020 "archetype": "regular",1021 "slug": "graph-neural-network-for-regression-purpose-not-updating",1022 "category_id": 1,1023 "word_count": 737,1024 "deleted_at": null,1025 "user_id": 65001,1026 "featured_link": null,1027 "pinned_globally": false,1028 "pinned_at": null,1029 "pinned_until": null,1030 "image_url": null,1031 "slow_mode_seconds": 0,1032 "draft": null,1033 "draft_key": "topic_185246",1034 "draft_sequence": null,1035 "unpinned": null,1036 "pinned": false,1037 "current_post_number": 1,1038 "highest_post_number": 2,1039 "deleted_by": null,1040 "actions_summary": [1041 {1042 "id": 4,1043 "count": 0,1044 "hidden": false,1045 "can_act": false1046 },1047 {1048 "id": 8,1049 "count": 0,1050 "hidden": false,1051 "can_act": false1052 },1053 {1054 "id": 10,1055 "count": 0,1056 "hidden": false,1057 "can_act": false1058 },1059 {1060 "id": 7,1061 "count": 0,1062 "hidden": false,1063 "can_act": false1064 }1065 ],1066 "chunk_size": 20,1067 "bookmarked": false,1068 "topic_timer": null,1069 "message_bus_last_id": 0,1070 "participant_count": 2,1071 "show_read_indicator": false,1072 "thumbnails": null,1073 "slow_mode_enabled_until": null,1074 "can_vote": false,1075 "vote_count": 0,1076 "user_voted": false,1077 "discourse_zendesk_plugin_zendesk_id": null,1078 "discourse_zendesk_plugin_zendesk_url": "https://your-url.zendesk.com/agent/tickets/",1079 "details": {1080 "can_edit": false,1081 "notification_level": 1,1082 "participants": [1083 {1084 "id": 65001,1085 "username": "dirkcremers",1086 "name": "",1087 "avatar_template": "/user_avatar/discuss.pytorch.org/dirkcremers/{size}/59197_2.png",1088 "post_count": 1,1089 "primary_group_name": null,1090 "flair_name": null,1091 "flair_url": null,1092 "flair_color": null,1093 "flair_bg_color": null,1094 "flair_group_id": null,1095 "trust_level": 11096 },1097 {1098 "id": 76055,1099 "username": "elham_gholamzade",1100 "name": "elham gholamzade",1101 "avatar_template": "/user_avatar/discuss.pytorch.org/elham_gholamzade/{size}/70223_2.png",1102 "post_count": 1,1103 "primary_group_name": null,1104 "flair_name": null,1105 "flair_url": null,1106 "flair_color": null,1107 "flair_bg_color": null,1108 "flair_group_id": null,1109 "trust_level": 01110 }1111 ],1112 "created_by": {1113 "id": 65001,1114 "username": "dirkcremers",1115 "name": "",1116 "avatar_template": "/user_avatar/discuss.pytorch.org/dirkcremers/{size}/59197_2.png"1117 },1118 "last_poster": {1119 "id": 76055,1120 "username": "elham_gholamzade",1121 "name": "elham gholamzade",1122 "avatar_template": "/user_avatar/discuss.pytorch.org/elham_gholamzade/{size}/70223_2.png"1123 }1124 },1125 "bookmarks": []1126 },1127 {1128 "post_stream": {1129 "posts": [1130 {1131 "id": 443485,1132 "name": "",1133 "username": "ekrx",1134 "avatar_template": "/user_avatar/discuss.pytorch.org/ekrx/{size}/68501_2.png",1135 "created_at": "2024-05-22T20:44:47.643Z",1136 "cooked": "<p>I’m encountering an issue where even though I have pin_memory=True set in my DataLoader, the data remains on the CPU during training. My CUDA GPU is available (torch.cuda.is_available() returns True).</p>\n<pre><code class=\"lang-auto\">train = MyDataSet() # Instance of an IterableDataset\ntrain_loader = DataLoader(dataset=train, batch_size=batch_size, prefetch_factor=10, num_workers=4, pin_memory=True)\nfor batch_index, (x_batch, y_batch) in enumerate(train_loader):\n print(x_batch.device) # Shows \"cpu\"\n print(torch.cuda.is_available()) # Shows 'True\n output = model(x_batch)\n # ... training code ...\n</code></pre>\n<p>This results in a RuntimeError stating that input and parameter tensors are on different devices (CPU vs. GPU).</p>\n<p>I want to avoid having to explicitly move the data to the GPU because its causing a bottleneck.</p>",1137 "post_number": 1,1138 "post_type": 1,1139 "posts_count": 5,1140 "updated_at": "2024-05-22T20:44:47.643Z",1141 "reply_count": 0,1142 "reply_to_post_number": null,1143 "quote_count": 0,1144 "incoming_link_count": 84,1145 "reads": 7,1146 "readers_count": 6,1147 "score": 416.4,1148 "yours": false,1149 "topic_id": 203254,1150 "topic_slug": "pytorch-pin-memory-true-not-transferring-data-to-gpu",1151 "display_username": "",1152 "primary_group_name": null,1153 "flair_name": null,1154 "flair_url": null,1155 "flair_bg_color": null,1156 "flair_color": null,1157 "flair_group_id": null,1158 "badges_granted": [],1159 "version": 1,1160 "can_edit": false,1161 "can_delete": false,1162 "can_recover": false,1163 "can_see_hidden_post": false,1164 "can_wiki": false,1165 "read": true,1166 "user_title": null,1167 "bookmarked": false,1168 "actions_summary": [],1169 "moderator": false,1170 "admin": false,1171 "staff": false,1172 "user_id": 74167,1173 "hidden": false,1174 "trust_level": 1,1175 "deleted_at": null,1176 "user_deleted": false,1177 "edit_reason": null,1178 "can_view_edit_history": true,1179 "wiki": false,1180 "post_url": "/t/pytorch-pin-memory-true-not-transferring-data-to-gpu/203254/1",1181 "can_accept_answer": false,1182 "can_unaccept_answer": false,1183 "accepted_answer": false,1184 "topic_accepted_answer": true,1185 "can_vote": false1186 },1187 {1188 "id": 443486,1189 "name": "",1190 "username": "ptrblck",1191 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",1192 "created_at": "2024-05-22T20:46:05.552Z",1193 "cooked": "<p>Using <code>pin_memory=True</code> will use page-locked host memory and will not move the data behind your back. You still need to move the data to the device.</p>",1194 "post_number": 2,1195 "post_type": 1,1196 "posts_count": 5,1197 "updated_at": "2024-05-22T20:46:05.552Z",1198 "reply_count": 0,1199 "reply_to_post_number": null,1200 "quote_count": 0,