Anurag1734/cuda-error-resolution-analysis
07
1[2 {3 "post_stream": {4 "posts": [5 {6 "id": 456468,7 "name": "Yu I",8 "username": "Yu_I",9 "avatar_template": "/user_avatar/discuss.pytorch.org/yu_i/{size}/73135_2.png",10 "created_at": "2024-10-03T05:29:03.017Z",11 "cooked": "<p>I am new to FSDP.</p>\n<p>This is my understanding of the sub-module FSDP.<br>\nFSDP relies on <code>torch.nn.Module.forward()</code> because of all-gather parameters. So, when I create sub modules with <code>torch.nn.Sequential()</code> in <code>torch.nn.Module</code> class, it does not work.<br>\nAlso, FSDP class says</p>\n<blockquote>\n<p>FSDP does not support running the forward pass of a submodule<br>\nthat is contained in an FSDP instance. This is because the<br>\nsubmodule’s parameters will be sharded, but the submodule itself<br>\nis not an FSDP instance, so its forward pass will not all-gather<br>\nthe full parameters appropriately.</p>\n</blockquote>\n<p>For example, the FSDP tutorial (Pytorch) demonstrates Deep-ViT, Cross-ViT, and Cait on FSDP.<br>\nThis tutorial wrapped them with <code>transformer_auto_wrapper_policy</code>, but it does not work because of sub-modules.<br>\n(<a href=\"https://github.com/lessw2020/transformer_central/blob/main/transformer_wrapping_tutorial/transformer_wrapper_tutorial.ipynb\" class=\"inline-onebox\" rel=\"noopener nofollow ugc\">transformer_central/transformer_wrapping_tutorial/transformer_wrapper_tutorial.ipynb at main · lessw2020/transformer_central · GitHub</a>)</p>\n<pre><code class=\"lang-auto\">class DeepViT(nn.Module):\n def __init__(self, *, image_size, ...):\n super().__init__()\n self.to_patch_embedding = nn.Sequential(\n Rearrange('b c (h p1) (w p2) -> b (h w) (p1 p2 c)', p1 = patch_size, p2 = patch_size),\n nn.Linear(patch_dim, dim),\n ) <----- I think this does not work because of the sub-module.\n</code></pre>\n<p>In this case, we need to rewrite the sub-module (self.to_patch_embedding) with <code>torch.nn.module</code> like the following. (Assume add the following lines in Deep_ViT)</p>\n<pre><code class=\"lang-auto\">class PatchEmbedding(nn.Module):\n def __init__(self, in_channels: int = 3, patch_size: int = 16, emb_size: int = 768, img_size: int = 224):\n self.patch_size = patch_size\n super().__init__()\n self.projection = nn.Sequential(\n # using a conv layer instead of a linear one -> performance gains\n nn.Conv2d(in_channels, emb_size, kernel_size=patch_size, stride=patch_size),\n Rearrange('b e (h) (w) -> b (h w) e'),\n )\n self.cls_token = nn.Parameter(torch.randn(1, 1, emb_size))\n self.positions = nn.Parameter(torch.randn((img_size // patch_size) ** 2 + 1, emb_size))\n\n def forward(self, x: Tensor) -> Tensor:\n b, _, _, _ = x.shape\n x = self.projection(x)\n cls_tokens = repeat(self.cls_token, '() n e -> b n e', b=b)\n # prepend the cls token to the input\n x = torch.cat([cls_tokens, x], dim=1)\n # add position embedding\n x += self.positions\n return x\n\nclass DeepViT(nn.Module):\n def __init__(self, *, image_size, ...):\n super().__init__()\n self.to_patch_embedding = PatchEmbedding()\n ...\n\n def forward(self, img):\n x = self.to_patch_embedding(img)\n x = self.dropout(x)\n x = self.transformer(x)\n x = x.mean(dim = 1) if self.pool == 'mean' else x[:, 0]\n x = self.to_latent(x)\n return self.mlp_head(x)\n</code></pre>\n<p>Then, wrap with transformer_auto_wrap_policy.</p>\n<pre><code class=\"lang-auto\">from deep_vit import Residual, PatchEmbedding\nimport functools\n\ntransformer_auto_wrapper_policy = functools.partial(\n transformer_auto_wrap_policy,\n transformer_layer_cls={\n PatchEmbedding,\n Residual, # < ---- Your Transformer layer class\n },\n )\n\nsharded_model = FSDP(\n model,\n auto_wrap_policy=transformer_auto_wrapper_policy,\n mixed_precision=mp_policy,\n sharding_strategy=model_sharding_strategy,\n device_id=torch.cuda.current_device(), # streaming init\n )\n</code></pre>\n<p>Do the above steps collect?<br>\nIf so, FSDP shards PatchEmbedding across GPUs, right? (I’m not sure whether this is the right thing or not.)</p>",12 "post_number": 1,13 "post_type": 1,14 "posts_count": 6,15 "updated_at": "2024-10-03T05:29:03.017Z",16 "reply_count": 0,17 "reply_to_post_number": null,18 "quote_count": 0,19 "incoming_link_count": 241,20 "reads": 18,21 "readers_count": 17,22 "score": 1183.6,23 "yours": false,24 "topic_id": 211090,25 "topic_slug": "sub-modules-in-fsdp",26 "display_username": "Yu I",27 "primary_group_name": null,28 "flair_name": null,29 "flair_url": null,30 "flair_bg_color": null,31 "flair_color": null,32 "flair_group_id": null,33 "badges_granted": [],34 "version": 1,35 "can_edit": false,36 "can_delete": false,37 "can_recover": false,38 "can_see_hidden_post": false,39 "can_wiki": false,40 "link_counts": [41 {42 "url": "https://github.com/lessw2020/transformer_central/blob/main/transformer_wrapping_tutorial/transformer_wrapper_tutorial.ipynb",43 "internal": false,44 "reflection": false,45 "title": "transformer_central/transformer_wrapping_tutorial/transformer_wrapper_tutorial.ipynb at main · lessw2020/transformer_central · GitHub",46 "clicks": 847 }48 ],49 "read": true,50 "user_title": null,51 "bookmarked": false,52 "actions_summary": [],53 "moderator": false,54 "admin": false,55 "staff": false,56 "user_id": 80007,57 "hidden": false,58 "trust_level": 1,59 "deleted_at": null,60 "user_deleted": false,61 "edit_reason": null,62 "can_view_edit_history": true,63 "wiki": false,64 "post_url": "/t/sub-modules-in-fsdp/211090/1",65 "can_accept_answer": false,66 "can_unaccept_answer": false,67 "accepted_answer": false,68 "topic_accepted_answer": null,69 "can_vote": false70 },71 {72 "id": 456704,73 "name": "PyTorch Developer, Meta",74 "username": "yf225",75 "avatar_template": "/user_avatar/discuss.pytorch.org/yf225/{size}/3418_2.png",76 "created_at": "2024-10-07T19:23:30.031Z",77 "cooked": "<p>cc. FSDP1 experts <a class=\"mention\" href=\"/u/weifengpy\">@weifengpy</a> <a class=\"mention\" href=\"/u/agu\">@agu</a></p>",78 "post_number": 2,79 "post_type": 1,80 "posts_count": 6,81 "updated_at": "2024-10-07T19:23:30.031Z",82 "reply_count": 0,83 "reply_to_post_number": null,84 "quote_count": 0,85 "incoming_link_count": 1,86 "reads": 13,87 "readers_count": 12,88 "score": 7.6,89 "yours": false,90 "topic_id": 211090,91 "topic_slug": "sub-modules-in-fsdp",92 "display_username": "PyTorch Developer, Meta",93 "primary_group_name": null,94 "flair_name": null,95 "flair_url": null,96 "flair_bg_color": null,97 "flair_color": null,98 "flair_group_id": null,99 "badges_granted": [],100 "version": 1,101 "can_edit": false,102 "can_delete": false,103 "can_recover": false,104 "can_see_hidden_post": false,105 "can_wiki": false,106 "read": true,107 "user_title": "",108 "bookmarked": false,109 "actions_summary": [],110 "moderator": false,111 "admin": false,112 "staff": false,113 "user_id": 6225,114 "hidden": false,115 "trust_level": 2,116 "deleted_at": null,117 "user_deleted": false,118 "edit_reason": null,119 "can_view_edit_history": true,120 "wiki": false,121 "post_url": "/t/sub-modules-in-fsdp/211090/2",122 "can_accept_answer": false,123 "can_unaccept_answer": false,124 "accepted_answer": false,125 "topic_accepted_answer": null126 },127 {128 "id": 456706,129 "name": "Andrew Gu",130 "username": "agu",131 "avatar_template": "/user_avatar/discuss.pytorch.org/agu/{size}/49913_2.png",132 "created_at": "2024-10-07T20:08:12.605Z",133 "cooked": "<p>I am not sure if I followed completely. For <code>self.to_patch_embedding</code>, if you wrap it with FSDP and run <code>self.to_patch_embedding(...)</code> (its forward), then it should work.</p>",134 "post_number": 3,135 "post_type": 1,136 "posts_count": 6,137 "updated_at": "2024-10-07T20:08:12.605Z",138 "reply_count": 0,139 "reply_to_post_number": null,140 "quote_count": 0,141 "incoming_link_count": 5,142 "reads": 13,143 "readers_count": 12,144 "score": 12.6,145 "yours": false,146 "topic_id": 211090,147 "topic_slug": "sub-modules-in-fsdp",148 "display_username": "Andrew Gu",149 "primary_group_name": null,150 "flair_name": null,151 "flair_url": null,152 "flair_bg_color": null,153 "flair_color": null,154 "flair_group_id": null,155 "badges_granted": [],156 "version": 1,157 "can_edit": false,158 "can_delete": false,159 "can_recover": false,160 "can_see_hidden_post": false,161 "can_wiki": false,162 "read": true,163 "user_title": null,164 "bookmarked": false,165 "actions_summary": [],166 "moderator": false,167 "admin": false,168 "staff": false,169 "user_id": 49515,170 "hidden": false,171 "trust_level": 2,172 "deleted_at": null,173 "user_deleted": false,174 "edit_reason": null,175 "can_view_edit_history": true,176 "wiki": false,177 "post_url": "/t/sub-modules-in-fsdp/211090/3",178 "can_accept_answer": false,179 "can_unaccept_answer": false,180 "accepted_answer": false,181 "topic_accepted_answer": null182 },183 {184 "id": 456716,185 "name": "Yu_I2",186 "username": "Yu_I2",187 "avatar_template": "/user_avatar/discuss.pytorch.org/yu_i2/{size}/73215_2.png",188 "created_at": "2024-10-08T00:50:18.961Z",189 "cooked": "<p>I created a new account as I lost the previous password.</p>\n<p>When I wrap (transformer wrap) the <code>self.to_patch_embedding</code> following the tutorial, I get this error:</p>\n<blockquote>\n<p>RuntimeError: Output 0 of ViewBackward0 is a view and its base or another view of its base has been modified inplace. This view<br>\nis the output of a function that returns multiple views. Such functions do not allow the output views to be modified inplace.<br>\nYou should replace the inplace operation by an out-of-place one.</p>\n</blockquote>\n<p>The below codes are coming from the link (<a href=\"https://github.com/lessw2020/transformer_central/blob/main/transformer_wrapping_tutorial/transformer_wrapper_tutorial.ipynb\" class=\"inline-onebox\" rel=\"noopener nofollow ugc\">transformer_central/transformer_wrapping_tutorial/transformer_wrapper_tutorial.ipynb at main · lessw2020/transformer_central · GitHub</a>)</p>\n<pre><code class=\"lang-auto\">class DeepViT(nn.Module):\n def __init__(self, *, image_size, ...):\n super().__init__()\n self.to_patch_embedding = nn.Sequential(\n Rearrange('b c (h p1) (w p2) -> b (h w) (p1 p2 c)', p1 = patch_size, p2 = patch_size),\n nn.Linear(patch_dim, dim),\n )\n</code></pre>\n<p>Then,</p>\n<pre><code class=\"lang-auto\">from deep_vit import Residual\nimport functools\n\ntransformer_auto_wrapper_policy = functools.partial(\n transformer_auto_wrap_policy,\n transformer_layer_cls={\n Residual, # < ---- Your Transformer layer class\n },\n )\n\nsharded_model = FSDP(\n model,\n auto_wrap_policy=transformer_auto_wrapper_policy,\n mixed_precision=mp_policy,\n sharding_strategy=model_sharding_strategy,\n device_id=torch.cuda.current_device(), # streaming init\n )\n</code></pre>\n<p>So, I think somehow FSDP cannot wrap sub-modules <img src=\"https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=12\" title=\":thinking:\" class=\"emoji\" alt=\":thinking:\" loading=\"lazy\" width=\"20\" height=\"20\"></p>",190 "post_number": 4,191 "post_type": 1,192 "posts_count": 6,193 "updated_at": "2024-10-08T02:06:06.222Z",194 "reply_count": 1,195 "reply_to_post_number": null,196 "quote_count": 0,197 "incoming_link_count": 4,198 "reads": 12,199 "readers_count": 11,200 "score": 27.4,201 "yours": false,202 "topic_id": 211090,203 "topic_slug": "sub-modules-in-fsdp",204 "display_username": "Yu_I2",205 "primary_group_name": null,206 "flair_name": null,207 "flair_url": null,208 "flair_bg_color": null,209 "flair_color": null,210 "flair_group_id": null,211 "badges_granted": [],212 "version": 1,213 "can_edit": false,214 "can_delete": false,215 "can_recover": false,216 "can_see_hidden_post": false,217 "can_wiki": false,218 "link_counts": [219 {220 "url": "https://github.com/lessw2020/transformer_central/blob/main/transformer_wrapping_tutorial/transformer_wrapper_tutorial.ipynb",221 "internal": false,222 "reflection": false,223 "title": "transformer_central/transformer_wrapping_tutorial/transformer_wrapper_tutorial.ipynb at main · lessw2020/transformer_central · GitHub",224 "clicks": 2225 }226 ],227 "read": true,228 "user_title": null,229 "bookmarked": false,230 "actions_summary": [],231 "moderator": false,232 "admin": false,233 "staff": false,234 "user_id": 80087,235 "hidden": false,236 "trust_level": 1,237 "deleted_at": null,238 "user_deleted": false,239 "edit_reason": null,240 "can_view_edit_history": true,241 "wiki": false,242 "post_url": "/t/sub-modules-in-fsdp/211090/4",243 "can_accept_answer": false,244 "can_unaccept_answer": false,245 "accepted_answer": false,246 "topic_accepted_answer": null247 },248 {249 "id": 457059,250 "name": "Andrew Gu",251 "username": "agu",252 "avatar_template": "/user_avatar/discuss.pytorch.org/agu/{size}/49913_2.png",253 "created_at": "2024-10-12T15:37:54.135Z",254 "cooked": "<p>I am still not following exactly. How did you update the <code>auto_wrap_policy</code>? Did you include <code>nn.Sequential</code> as one of the classes?</p>",255 "post_number": 5,256 "post_type": 1,257 "posts_count": 6,258 "updated_at": "2024-10-12T15:37:54.135Z",259 "reply_count": 0,260 "reply_to_post_number": 4,261 "quote_count": 0,262 "incoming_link_count": 3,263 "reads": 7,264 "readers_count": 6,265 "score": 16.4,266 "yours": false,267 "topic_id": 211090,268 "topic_slug": "sub-modules-in-fsdp",269 "display_username": "Andrew Gu",270 "primary_group_name": null,271 "flair_name": null,272 "flair_url": null,273 "flair_bg_color": null,274 "flair_color": null,275 "flair_group_id": null,276 "badges_granted": [],277 "version": 1,278 "can_edit": false,279 "can_delete": false,280 "can_recover": false,281 "can_see_hidden_post": false,282 "can_wiki": false,283 "read": true,284 "user_title": null,285 "reply_to_user": {286 "id": 80087,287 "username": "Yu_I2",288 "name": "Yu_I2",289 "avatar_template": "/user_avatar/discuss.pytorch.org/yu_i2/{size}/73215_2.png"290 },291 "bookmarked": false,292 "actions_summary": [],293 "moderator": false,294 "admin": false,295 "staff": false,296 "user_id": 49515,297 "hidden": false,298 "trust_level": 2,299 "deleted_at": null,300 "user_deleted": false,301 "edit_reason": null,302 "can_view_edit_history": true,303 "wiki": false,304 "post_url": "/t/sub-modules-in-fsdp/211090/5",305 "can_accept_answer": false,306 "can_unaccept_answer": false,307 "accepted_answer": false,308 "topic_accepted_answer": null309 },310 {311 "id": 468926,312 "name": "Connelly Barnes",313 "username": "cobarnes_adobe",314 "avatar_template": "/user_avatar/discuss.pytorch.org/cobarnes_adobe/{size}/76553_2.png",315 "created_at": "2025-04-09T20:05:23.680Z",316 "cooked": "<p>I ran into the same error. I was able to resolve it by defining a new empty subclass of <code>nn.Sequential</code> e.g. <code>MySequential</code>, instantiating the submodule using that new subclass, and adding an <code>isinstance(module, MySequential)</code> check to my existing custom auto wrap policy such that the new class is wrapped (I did not use the <code>transformer_auto_wrapper_policy</code>).</p>\n<p>I feel the PyTorch <a href=\"https://pytorch.org/docs/stable/fsdp.html\" rel=\"noopener nofollow ugc\">FSDP documentation</a> is misleading when it says: “FSDP does not support running the forward pass of a submodule that is contained in an FSDP instance.” Apparently, this sentence meant to say: “FSDP does not support running the forward pass of a submodule that is contained in an FSDP instance, unless the submodule is itself wrapped by FSDP.”</p>",317 "post_number": 6,318 "post_type": 1,319 "posts_count": 6,320 "updated_at": "2025-04-09T20:08:36.874Z",321 "reply_count": 0,322 "reply_to_post_number": null,323 "quote_count": 0,324 "incoming_link_count": 4,325 "reads": 4,326 "readers_count": 3,327 "score": 20.8,328 "yours": false,329 "topic_id": 211090,330 "topic_slug": "sub-modules-in-fsdp",331 "display_username": "Connelly Barnes",332 "primary_group_name": null,333 "flair_name": null,334 "flair_url": null,335 "flair_bg_color": null,336 "flair_color": null,337 "flair_group_id": null,338 "badges_granted": [],339 "version": 2,340 "can_edit": false,341 "can_delete": false,342 "can_recover": false,343 "can_see_hidden_post": false,344 "can_wiki": false,345 "link_counts": [346 {347 "url": "https://pytorch.org/docs/stable/fsdp.html",348 "internal": false,349 "reflection": false,350 "title": "FullyShardedDataParallel — PyTorch 2.6 documentation",351 "clicks": 3352 }353 ],354 "read": true,355 "user_title": null,356 "bookmarked": false,357 "actions_summary": [],358 "moderator": false,359 "admin": false,360 "staff": false,361 "user_id": 83729,362 "hidden": false,363 "trust_level": 0,364 "deleted_at": null,365 "user_deleted": false,366 "edit_reason": null,367 "can_view_edit_history": true,368 "wiki": false,369 "post_url": "/t/sub-modules-in-fsdp/211090/6",370 "can_accept_answer": false,371 "can_unaccept_answer": false,372 "accepted_answer": false,373 "topic_accepted_answer": null374 }375 ],376 "stream": [377 456468,378 456704,379 456706,380 456716,381 457059,382 468926383 ]384 },385 "timeline_lookup": [386 [387 1,388 388389 ],390 [391 2,392 383393 ],394 [395 5,396 378397 ],398 [399 6,400 199401 ]402 ],403 "suggested_topics": [404 {405 "fancy_title": "Stalling on Simple Distributed Barrier",406 "id": 220404,407 "title": "Stalling on Simple Distributed Barrier",408 "slug": "stalling-on-simple-distributed-barrier",409 "posts_count": 13,410 "reply_count": 10,411 "highest_post_number": 13,412 "image_url": null,413 "created_at": "2025-05-28T12:45:47.833Z",414 "last_posted_at": "2025-05-30T00:44:27.746Z",415 "bumped": true,416 "bumped_at": "2025-05-30T00:44:27.746Z",417 "archetype": "regular",418 "unseen": false,419 "pinned": false,420 "unpinned": null,421 "visible": true,422 "closed": false,423 "archived": false,424 "bookmarked": null,425 "liked": null,426 "tags_descriptions": {},427 "like_count": 3,428 "views": 742,429 "category_id": 12,430 "featured_link": null,431 "has_accepted_answer": true,432 "posters": [433 {434 "extras": null,435 "description": "Original Poster",436 "user": {437 "id": 35595,438 "username": "5had3z",439 "name": "5had3z",440 "avatar_template": "/user_avatar/discuss.pytorch.org/5had3z/{size}/28066_2.png",441 "trust_level": 1442 }443 },444 {445 "extras": "latest",446 "description": "Most Recent Poster, Accepted Answer",447 "user": {448 "id": 3534,449 "username": "ptrblck",450 "name": "",451 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",452 "admin": true,453 "moderator": true,454 "trust_level": 2455 }456 }457 ]458 },459 {460 "fancy_title": "Cuda not available when running multi-gpu inference",461 "id": 212604,462 "title": "Cuda not available when running multi-gpu inference",463 "slug": "cuda-not-available-when-running-multi-gpu-inference",464 "posts_count": 5,465 "reply_count": 3,466 "highest_post_number": 5,467 "image_url": null,468 "created_at": "2024-11-06T09:40:57.712Z",469 "last_posted_at": "2024-11-19T09:13:35.999Z",470 "bumped": true,471 "bumped_at": "2024-11-19T09:13:35.999Z",472 "archetype": "regular",473 "unseen": false,474 "pinned": false,475 "unpinned": null,476 "visible": true,477 "closed": false,478 "archived": false,479 "bookmarked": null,480 "liked": null,481 "tags_descriptions": {},482 "like_count": 0,483 "views": 363,484 "category_id": 12,485 "featured_link": null,486 "has_accepted_answer": true,487 "posters": [488 {489 "extras": "latest",490 "description": "Original Poster, Most Recent Poster, Accepted Answer",491 "user": {492 "id": 80724,493 "username": "paulge",494 "name": "",495 "avatar_template": "/letter_avatar_proxy/v4/letter/p/82dd89/{size}.png",496 "trust_level": 2497 }498 },499 {500 "extras": null,501 "description": "Frequent Poster",502 "user": {503 "id": 3534,504 "username": "ptrblck",505 "name": "",506 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",507 "admin": true,508 "moderator": true,509 "trust_level": 2510 }511 }512 ]513 },514 {515 "fancy_title": "Training multiple independent models on one GPU/multiple GPUs using joblib",516 "id": 213775,517 "title": "Training multiple independent models on one GPU/multiple GPUs using joblib",518 "slug": "training-multiple-independent-models-on-one-gpu-multiple-gpus-using-joblib",519 "posts_count": 1,520 "reply_count": 0,521 "highest_post_number": 1,522 "image_url": null,523 "created_at": "2024-12-04T00:25:36.719Z",524 "last_posted_at": "2024-12-04T00:25:36.773Z",525 "bumped": true,526 "bumped_at": "2024-12-04T00:36:29.646Z",527 "archetype": "regular",528 "unseen": false,529 "pinned": false,530 "unpinned": null,531 "visible": true,532 "closed": false,533 "archived": false,534 "bookmarked": null,535 "liked": null,536 "tags_descriptions": {},537 "like_count": 0,538 "views": 150,539 "category_id": 12,540 "featured_link": null,541 "has_accepted_answer": false,542 "posters": [543 {544 "extras": "latest single",545 "description": "Original Poster, Most Recent Poster",546 "user": {547 "id": 68718,548 "username": "stonez",549 "name": "Sidong Zhang",550 "avatar_template": "/user_avatar/discuss.pytorch.org/stonez/{size}/72953_2.png",551 "trust_level": 1552 }553 }554 ]555 },556 {557 "fancy_title": "What is the best practice to send/recv multiple tensors across DDP ranks?",558 "id": 219051,559 "title": "What is the best practice to send/recv multiple tensors across DDP ranks?",560 "slug": "what-is-the-best-practice-to-send-recv-multiple-tensors-across-ddp-ranks",561 "posts_count": 1,562 "reply_count": 0,563 "highest_post_number": 1,564 "image_url": null,565 "created_at": "2025-04-14T08:35:11.997Z",566 "last_posted_at": "2025-04-14T08:35:12.043Z",567 "bumped": true,568 "bumped_at": "2025-04-14T08:35:12.043Z",569 "archetype": "regular",570 "unseen": false,571 "pinned": false,572 "unpinned": null,573 "visible": true,574 "closed": false,575 "archived": false,576 "bookmarked": null,577 "liked": null,578 "tags_descriptions": {},579 "like_count": 0,580 "views": 43,581 "category_id": 12,582 "featured_link": null,583 "has_accepted_answer": false,584 "posters": [585 {586 "extras": "latest single",587 "description": "Original Poster, Most Recent Poster",588 "user": {589 "id": 83802,590 "username": "lostkevin",591 "name": "Lostkevin",592 "avatar_template": "/user_avatar/discuss.pytorch.org/lostkevin/{size}/76607_2.png",593 "trust_level": 1594 }595 }596 ]597 },598 {599 "fancy_title": "Ddp training and eval question",600 "id": 221823,601 "title": "Ddp training and eval question",602 "slug": "ddp-training-and-eval-question",603 "posts_count": 3,604 "reply_count": 1,605 "highest_post_number": 3,606 "image_url": null,607 "created_at": "2025-07-26T07:56:48.862Z",608 "last_posted_at": "2025-07-26T21:43:09.321Z",609 "bumped": true,610 "bumped_at": "2025-07-26T21:43:09.321Z",611 "archetype": "regular",612 "unseen": false,613 "pinned": false,614 "unpinned": null,615 "visible": true,616 "closed": false,617 "archived": false,618 "bookmarked": null,619 "liked": null,620 "tags_descriptions": {},621 "like_count": 2,622 "views": 38,623 "category_id": 12,624 "featured_link": null,625 "has_accepted_answer": false,626 "posters": [627 {628 "extras": "latest",629 "description": "Original Poster, Most Recent Poster",630 "user": {631 "id": 85233,632 "username": "Julius_Lee",633 "name": "Julius Lee",634 "avatar_template": "/user_avatar/discuss.pytorch.org/julius_lee/{size}/77782_2.png",635 "trust_level": 0636 }637 },638 {639 "extras": null,640 "description": "Frequent Poster",641 "user": {642 "id": 39542,643 "username": "H-Huang",644 "name": "Howard Huang",645 "avatar_template": "/user_avatar/discuss.pytorch.org/h-huang/{size}/35598_2.png",646 "trust_level": 2647 }648 }649 ]650 }651 ],652 "tags_descriptions": {},653 "fancy_title": "Sub-modules in FSDP",654 "id": 211090,655 "title": "Sub-modules in FSDP",656 "posts_count": 6,657 "created_at": "2024-10-03T05:29:02.895Z",658 "views": 495,659 "reply_count": 1,660 "like_count": 0,661 "last_posted_at": "2025-04-09T20:05:23.680Z",662 "visible": true,663 "closed": false,664 "archived": false,665 "has_summary": false,666 "archetype": "regular",667 "slug": "sub-modules-in-fsdp",668 "category_id": 12,669 "word_count": 804,670 "deleted_at": null,671 "user_id": 80007,672 "featured_link": null,673 "pinned_globally": false,674 "pinned_at": null,675 "pinned_until": null,676 "image_url": null,677 "slow_mode_seconds": 0,678 "draft": null,679 "draft_key": "topic_211090",680 "draft_sequence": null,681 "unpinned": null,682 "pinned": false,683 "current_post_number": 1,684 "highest_post_number": 6,685 "deleted_by": null,686 "actions_summary": [687 {688 "id": 4,689 "count": 0,690 "hidden": false,691 "can_act": false692 },693 {694 "id": 8,695 "count": 0,696 "hidden": false,697 "can_act": false698 },699 {700 "id": 10,701 "count": 0,702 "hidden": false,703 "can_act": false704 },705 {706 "id": 7,707 "count": 0,708 "hidden": false,709 "can_act": false710 }711 ],712 "chunk_size": 20,713 "bookmarked": false,714 "topic_timer": null,715 "message_bus_last_id": 0,716 "participant_count": 5,717 "show_read_indicator": false,718 "thumbnails": null,719 "slow_mode_enabled_until": null,720 "can_vote": false,721 "vote_count": 0,722 "user_voted": false,723 "discourse_zendesk_plugin_zendesk_id": null,724 "discourse_zendesk_plugin_zendesk_url": "https://your-url.zendesk.com/agent/tickets/",725 "details": {726 "can_edit": false,727 "notification_level": 1,728 "participants": [729 {730 "id": 49515,731 "username": "agu",732 "name": "Andrew Gu",733 "avatar_template": "/user_avatar/discuss.pytorch.org/agu/{size}/49913_2.png",734 "post_count": 2,735 "primary_group_name": null,736 "flair_name": null,737 "flair_url": null,738 "flair_color": null,739 "flair_bg_color": null,740 "flair_group_id": null,741 "trust_level": 2742 },743 {744 "id": 6225,745 "username": "yf225",746 "name": "PyTorch Developer, Meta",747 "avatar_template": "/user_avatar/discuss.pytorch.org/yf225/{size}/3418_2.png",748 "post_count": 1,749 "primary_group_name": null,750 "flair_name": null,751 "flair_url": null,752 "flair_color": null,753 "flair_bg_color": null,754 "flair_group_id": null,755 "trust_level": 2756 },757 {758 "id": 80007,759 "username": "Yu_I",760 "name": "Yu I",761 "avatar_template": "/user_avatar/discuss.pytorch.org/yu_i/{size}/73135_2.png",762 "post_count": 1,763 "primary_group_name": null,764 "flair_name": null,765 "flair_url": null,766 "flair_color": null,767 "flair_bg_color": null,768 "flair_group_id": null,769 "trust_level": 1770 },771 {772 "id": 80087,773 "username": "Yu_I2",774 "name": "Yu_I2",775 "avatar_template": "/user_avatar/discuss.pytorch.org/yu_i2/{size}/73215_2.png",776 "post_count": 1,777 "primary_group_name": null,778 "flair_name": null,779 "flair_url": null,780 "flair_color": null,781 "flair_bg_color": null,782 "flair_group_id": null,783 "trust_level": 1784 },785 {786 "id": 83729,787 "username": "cobarnes_adobe",788 "name": "Connelly Barnes",789 "avatar_template": "/user_avatar/discuss.pytorch.org/cobarnes_adobe/{size}/76553_2.png",790 "post_count": 1,791 "primary_group_name": null,792 "flair_name": null,793 "flair_url": null,794 "flair_color": null,795 "flair_bg_color": null,796 "flair_group_id": null,797 "trust_level": 0798 }799 ],800 "created_by": {801 "id": 80007,802 "username": "Yu_I",803 "name": "Yu I",804 "avatar_template": "/user_avatar/discuss.pytorch.org/yu_i/{size}/73135_2.png"805 },806 "last_poster": {807 "id": 83729,808 "username": "cobarnes_adobe",809 "name": "Connelly Barnes",810 "avatar_template": "/user_avatar/discuss.pytorch.org/cobarnes_adobe/{size}/76553_2.png"811 },812 "links": [813 {814 "url": "https://github.com/lessw2020/transformer_central/blob/main/transformer_wrapping_tutorial/transformer_wrapper_tutorial.ipynb",815 "title": "transformer_central/transformer_wrapping_tutorial/transformer_wrapper_tutorial.ipynb at main · lessw2020/transformer_central · GitHub",816 "internal": false,817 "attachment": false,818 "reflection": false,819 "clicks": 10,820 "user_id": 80007,821 "domain": "github.com",822 "root_domain": "github.com"823 },824 {825 "url": "https://pytorch.org/docs/stable/fsdp.html",826 "title": "FullyShardedDataParallel — PyTorch 2.6 documentation",827 "internal": false,828 "attachment": false,829 "reflection": false,830 "clicks": 3,831 "user_id": 83729,832 "domain": "pytorch.org",833 "root_domain": "pytorch.org"834 }835 ]836 },837 "bookmarks": []838 },839 {840 "post_stream": {841 "posts": [842 {843 "id": 468923,844 "name": "gogineni kailashnath",845 "username": "gogineni_kailash",846 "avatar_template": "/user_avatar/discuss.pytorch.org/gogineni_kailash/{size}/76552_2.png",847 "created_at": "2025-04-09T18:38:58.021Z",848 "cooked": "<p>This is my code base and I want to add pytorch profiler to the fit() function. Can anyone let me know how to embed the pytorch profiler?</p>\n<aside class=\"onebox githubblob\" data-onebox-src=\"https://github.com/volcengine/verl/blob/main/verl/trainer/ppo/ray_trainer.py\">\n <header class=\"source\">\n\n <a href=\"https://github.com/volcengine/verl/blob/main/verl/trainer/ppo/ray_trainer.py\" target=\"_blank\" rel=\"noopener nofollow ugc\">github.com/volcengine/verl</a>\n </header>\n\n <article class=\"onebox-body\">\n <h4><a href=\"https://github.com/volcengine/verl/blob/main/verl/trainer/ppo/ray_trainer.py\" target=\"_blank\" rel=\"noopener nofollow ugc\">verl/trainer/ppo/ray_trainer.py</a></h4>\n\n<div class=\"git-blob-info\">\n <a href=\"https://github.com/volcengine/verl/blob/main/verl/trainer/ppo/ray_trainer.py\" rel=\"noopener nofollow ugc\"><code>main</code></a>\n</div>\n\n\n <pre><code class=\"lang-py\"># Copyright 2024 Bytedance Ltd. and/or its affiliates\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\"\"\"\nFSDP PPO Trainer with Ray-based single controller.\nThis trainer supports model-agonistic model initialization with huggingface\n\"\"\"\n\nimport os\nimport uuid\n</code></pre>\n\n\n\n This file has been truncated. <a href=\"https://github.com/volcengine/verl/blob/main/verl/trainer/ppo/ray_trainer.py\" target=\"_blank\" rel=\"noopener nofollow ugc\">show original</a>\n\n </article>\n\n <div class=\"onebox-metadata\">\n \n \n </div>\n\n <div style=\"clear: both\"></div>\n</aside>\n\n<p>I also raised the issue in the verl github, but was not resolved. Any help or insights will be appreciated.<br>\nIt only print the cpu events without CUDA events. But, the GPU utilization and VRAM is almost 80 to 100% over the course of training.</p>\n<p>Issue: <a href=\"https://github.com/volcengine/verl/issues/693\" class=\"inline-onebox\" rel=\"noopener nofollow ugc\">GitHub · Where software is built</a></p>",849 "post_number": 1,850 "post_type": 1,851 "posts_count": 1,852 "updated_at": "2025-04-09T18:38:58.021Z",853 "reply_count": 0,854 "reply_to_post_number": null,855 "quote_count": 0,856 "incoming_link_count": 94,857 "reads": 6,858 "readers_count": 5,859 "score": 456.2,860 "yours": false,861 "topic_id": 218916,862 "topic_slug": "how-to-embed-pytorch-profiler-to-verl",863 "display_username": "gogineni kailashnath",864 "primary_group_name": null,865 "flair_name": null,866 "flair_url": null,867 "flair_bg_color": null,868 "flair_color": null,869 "flair_group_id": null,870 "badges_granted": [],871 "version": 1,872 "can_edit": false,873 "can_delete": false,874 "can_recover": false,875 "can_see_hidden_post": false,876 "can_wiki": false,877 "link_counts": [878 {879 "url": "https://github.com/volcengine/verl/blob/main/verl/trainer/ppo/ray_trainer.py",880 "internal": false,881 "reflection": false,882 "title": "verl/verl/trainer/ppo/ray_trainer.py at main · volcengine/verl · GitHub",883 "clicks": 8884 },885 {886 "url": "https://github.com/volcengine/verl/issues/693",887 "internal": false,888 "reflection": false,889 "title": "How to profile the system level and kernel level analysis? · Issue #693 · volcengine/verl · GitHub",890 "clicks": 6891 }892 ],893 "read": true,894 "user_title": null,895 "bookmarked": false,896 "actions_summary": [],897 "moderator": false,898 "admin": false,899 "staff": false,900 "user_id": 83728,901 "hidden": false,902 "trust_level": 1,903 "deleted_at": null,904 "user_deleted": false,905 "edit_reason": null,906 "can_view_edit_history": true,907 "wiki": false,908 "post_url": "/t/how-to-embed-pytorch-profiler-to-verl/218916/1",909 "can_accept_answer": false,910 "can_unaccept_answer": false,911 "accepted_answer": false,912 "topic_accepted_answer": null,913 "can_vote": false914 }915 ],916 "stream": [917 468923918 ]919 },920 "timeline_lookup": [921 [922 1,923 199924 ]925 ],926 "suggested_topics": [927 {928 "fancy_title": "No download option for histogram visualizations in TensorBoard (when using torch-pruning)",929 "id": 222479,930 "title": "No download option for histogram visualizations in TensorBoard (when using torch-pruning)",931 "slug": "no-download-option-for-histogram-visualizations-in-tensorboard-when-using-torch-pruning",932 "posts_count": 1,933 "reply_count": 0,934 "highest_post_number": 1,935 "image_url": null,936 "created_at": "2025-08-19T12:08:54.212Z",937 "last_posted_at": "2025-08-19T12:08:54.266Z",938 "bumped": true,939 "bumped_at": "2025-08-19T12:08:54.266Z",940 "archetype": "regular",941 "unseen": false,942 "pinned": false,943 "unpinned": null,944 "visible": true,945 "closed": false,946 "archived": false,947 "bookmarked": null,948 "liked": null,949 "tags_descriptions": {},950 "like_count": 0,951 "views": 26,952 "category_id": 28,953 "featured_link": null,954 "has_accepted_answer": false,955 "posters": [956 {957 "extras": "latest single",958 "description": "Original Poster, Most Recent Poster",959 "user": {960 "id": 85553,961 "username": "SadMoon",962 "name": "",963 "avatar_template": "/letter_avatar_proxy/v4/letter/s/ac91a4/{size}.png",964 "trust_level": 0965 }966 }967 ]968 },969 {970 "fancy_title": "How am i supposed to code an axonal delay in my SNN?",971 "id": 223163,972 "title": "How am i supposed to code an axonal delay in my SNN?",973 "slug": "how-am-i-supposed-to-code-an-axonal-delay-in-my-snn",974 "posts_count": 1,975 "reply_count": 0,976 "highest_post_number": 1,977 "image_url": null,978 "created_at": "2025-09-18T07:15:54.444Z",979 "last_posted_at": "2025-09-18T07:15:54.504Z",980 "bumped": true,981 "bumped_at": "2025-09-18T09:55:48.325Z",982 "archetype": "regular",983 "unseen": false,984 "pinned": false,985 "unpinned": null,986 "visible": true,987 "closed": false,988 "archived": false,989 "bookmarked": null,990 "liked": null,991 "tags_descriptions": {},992 "like_count": 0,993 "views": 23,994 "category_id": 28,995 "featured_link": null,996 "has_accepted_answer": false,997 "posters": [998 {999 "extras": "latest single",1000 "description": "Original Poster, Most Recent Poster",1001 "user": {1002 "id": 85898,1003 "username": "Photon1",1004 "name": "Photon",1005 "avatar_template": "/user_avatar/discuss.pytorch.org/photon1/{size}/77068_2.png",1006 "trust_level": 01007 }1008 }1009 ]1010 },1011 {1012 "fancy_title": "Tensorboard add_graph (actually jit.trace) does not work with None inputs",1013 "id": 219863,1014 "title": "Tensorboard add_graph (actually jit.trace) does not work with None inputs",1015 "slug": "tensorboard-add-graph-actually-jit-trace-does-not-work-with-none-inputs",1016 "posts_count": 2,1017 "reply_count": 0,1018 "highest_post_number": 2,1019 "image_url": null,1020 "created_at": "2025-05-08T09:44:13.923Z",1021 "last_posted_at": "2025-05-08T09:52:55.880Z",1022 "bumped": true,1023 "bumped_at": "2025-05-08T10:04:01.686Z",1024 "archetype": "regular",1025 "unseen": false,1026 "pinned": false,1027 "unpinned": null,1028 "visible": true,1029 "closed": false,1030 "archived": false,1031 "bookmarked": null,1032 "liked": null,1033 "tags_descriptions": {},1034 "like_count": 0,1035 "views": 65,1036 "category_id": 28,1037 "featured_link": null,1038 "has_accepted_answer": false,1039 "posters": [1040 {1041 "extras": "latest single",1042 "description": "Original Poster, Most Recent Poster",1043 "user": {1044 "id": 75257,1045 "username": "turbotimon",1046 "name": "Timon Erhart",1047 "avatar_template": "/user_avatar/discuss.pytorch.org/turbotimon/{size}/69503_2.png",1048 "trust_level": 11049 }1050 }1051 ]1052 },1053 {1054 "fancy_title": "Segment Reduce memory problems while building?",1055 "id": 220488,1056 "title": "Segment Reduce memory problems while building?",1057 "slug": "segment-reduce-memory-problems-while-building",1058 "posts_count": 3,1059 "reply_count": 0,1060 "highest_post_number": 3,1061 "image_url": null,1062 "created_at": "2025-06-01T00:16:26.856Z",1063 "last_posted_at": "2025-06-17T13:49:29.491Z",1064 "bumped": true,1065 "bumped_at": "2025-06-17T13:49:29.491Z",1066 "archetype": "regular",1067 "unseen": false,1068 "pinned": false,1069 "unpinned": null,1070 "visible": true,1071 "closed": false,1072 "archived": false,1073 "bookmarked": null,1074 "liked": null,1075 "tags_descriptions": {},1076 "like_count": 0,1077 "views": 189,1078 "category_id": 1,1079 "featured_link": null,1080 "has_accepted_answer": false,1081 "posters": [1082 {1083 "extras": null,1084 "description": "Original Poster",1085 "user": {1086 "id": 84532,1087 "username": "NateTalley",1088 "name": "Nate Talley",1089 "avatar_template": "/user_avatar/discuss.pytorch.org/natetalley/{size}/77228_2.png",1090 "trust_level": 01091 }1092 },1093 {1094 "extras": null,1095 "description": "Frequent Poster",1096 "user": {1097 "id": 3534,1098 "username": "ptrblck",1099 "name": "",1100 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",1101 "admin": true,1102 "moderator": true,1103 "trust_level": 21104 }1105 },1106 {1107 "extras": "latest",1108 "description": "Most Recent Poster",1109 "user": {1110 "id": 67009,1111 "username": "atalman",1112 "name": "Andrey",1113 "avatar_template": "/user_avatar/discuss.pytorch.org/atalman/{size}/72308_2.png",1114 "trust_level": 11115 }1116 }1117 ]1118 },1119 {1120 "fancy_title": "[BUG] why does the C++ Libtorch performance slower than pytorch? (show the full code)",1121 "id": 222333,1122 "title": "[BUG] why does the C++ Libtorch performance slower than pytorch? (show the full code)",1123 "slug": "bug-why-does-the-c-libtorch-performance-slower-than-pytorch-show-the-full-code",1124 "posts_count": 1,1125 "reply_count": 0,1126 "highest_post_number": 1,1127 "image_url": null,1128 "created_at": "2025-08-13T14:58:27.870Z",1129 "last_posted_at": "2025-08-13T14:58:27.951Z",1130 "bumped": true,1131 "bumped_at": "2025-08-13T14:58:27.951Z",1132 "archetype": "regular",1133 "unseen": false,1134 "pinned": false,1135 "unpinned": null,1136 "visible": true,1137 "closed": false,1138 "archived": false,1139 "bookmarked": null,1140 "liked": null,1141 "tags_descriptions": {},1142 "like_count": 0,1143 "views": 68,1144 "category_id": 11,1145 "featured_link": null,1146 "has_accepted_answer": false,1147 "posters": [1148 {1149 "extras": "latest single",1150 "description": "Original Poster, Most Recent Poster",1151 "user": {1152 "id": 85492,1153 "username": "Sukidesyo",1154 "name": "Sukidesyo",1155 "avatar_template": "/user_avatar/discuss.pytorch.org/sukidesyo/{size}/77955_2.png",1156 "trust_level": 11157 }1158 }1159 ]1160 }1161 ],1162 "tags_descriptions": {},1163 "fancy_title": "How to embed pytorch profiler to verl?",1164 "id": 218916,1165 "title": "How to embed pytorch profiler to verl?",1166 "posts_count": 1,1167 "created_at": "2025-04-09T18:38:57.973Z",1168 "views": 106,1169 "reply_count": 0,1170 "like_count": 0,1171 "last_posted_at": "2025-04-09T18:38:58.021Z",1172 "visible": true,1173 "closed": false,1174 "archived": false,1175 "has_summary": false,1176 "archetype": "regular",1177 "slug": "how-to-embed-pytorch-profiler-to-verl",1178 "category_id": 28,1179 "word_count": 92,1180 "deleted_at": null,1181 "user_id": 83728,1182 "featured_link": null,1183 "pinned_globally": false,1184 "pinned_at": null,1185 "pinned_until": null,1186 "image_url": null,1187 "slow_mode_seconds": 0,1188 "draft": null,1189 "draft_key": "topic_218916",1190 "draft_sequence": null,1191 "unpinned": null,1192 "pinned": false,1193 "current_post_number": 1,1194 "highest_post_number": 1,1195 "deleted_by": null,1196 "actions_summary": [1197 {1198 "id": 4,1199 "count": 0,1200 "hidden": false,