Anurag1734/cuda-error-resolution-analysis
07
1[2 {3 "post_stream": {4 "posts": [5 {6 "id": 440239,7 "name": "Jamison Ballou",8 "username": "Jamison_Ballou",9 "avatar_template": "/user_avatar/discuss.pytorch.org/jamison_ballou/{size}/62486_2.png",10 "created_at": "2024-04-23T11:57:06.275Z",11 "cooked": "<p>Hello. I am new to using PyTorch, and am trying to follow the tutorial for creating a snake ML algorithm (<a href=\"https://www.youtube.com/watch?v=L8ypSXwyBds\" rel=\"noopener nofollow ugc\">https://www.youtube.com/watch?v=L8ypSXwyBds</a>). I got the program to work on my Mac, but it simply does not work with my Chromebook running Linux. I am using VS Code to run both programs. The source code can be found here: <a href=\"https://github.com/patrickloeber/snake-ai-pytorch\" class=\"inline-onebox\" rel=\"noopener nofollow ugc\">GitHub - patrickloeber/snake-ai-pytorch</a>. When I run the program, This is what appears in the terminal:<br>\n<code>Hello from the pygame community. https://www.pygame.org/contribute.html Traceback (most recent call last): File \"/home/jamballou/PyGames/Snake ML/snake-ai-pytorch-main/agent.py\", line 6, in <module> from model import Linear_QNet, QTrainer File \"/home/jamballou/PyGames/Snake ML/snake-ai-pytorch-main/model.py\", line 2, in <module> import torch.nn as nn File \"/home/jamballou/.local/lib/python3.9/site-packages/torch/nn/__init__.py\", line 1, in <module> from .modules import * # noqa: F403 File \"/home/jamballou/.local/lib/python3.9/site-packages/torch/nn/modules/__init__.py\", line 1, in <module> from .module import Module File \"/home/jamballou/.local/lib/python3.9/site-packages/torch/nn/modules/module.py\", line 8, in <module> from torch._prims_common import DeviceLikeType ModuleNotFoundError: No module named 'torch._prims_common'</code><br>\nI think the problem has to do with Importing the PyTorch modules. I have tried uninstalling and reinstalling all the used packages in this project, and even running the project from a virtual environment using virtualenv. As I said, this program works perfectly on my Mac, so I am baffled as to why it does not work on my Chromebook. If anyone could help, that would be much appreciated.</p>",12 "post_number": 1,13 "post_type": 1,14 "posts_count": 5,15 "updated_at": "2024-04-23T11:57:06.275Z",16 "reply_count": 0,17 "reply_to_post_number": null,18 "quote_count": 0,19 "incoming_link_count": 1695,20 "reads": 5,21 "readers_count": 4,22 "score": 8466.0,23 "yours": false,24 "topic_id": 201439,25 "topic_slug": "problems-with-importing-pytorch-for-snake-ml-algorithm-w-patrick-loeber",26 "display_username": "Jamison Ballou",27 "primary_group_name": null,28 "flair_name": null,29 "flair_url": null,30 "flair_bg_color": null,31 "flair_color": null,32 "flair_group_id": null,33 "badges_granted": [],34 "version": 1,35 "can_edit": false,36 "can_delete": false,37 "can_recover": false,38 "can_see_hidden_post": false,39 "can_wiki": false,40 "link_counts": [41 {42 "url": "https://www.youtube.com/watch?v=L8ypSXwyBds",43 "internal": false,44 "reflection": false,45 "clicks": 346 },47 {48 "url": "https://github.com/patrickloeber/snake-ai-pytorch",49 "internal": false,50 "reflection": false,51 "title": "GitHub - patrickloeber/snake-ai-pytorch",52 "clicks": 153 }54 ],55 "read": true,56 "user_title": null,57 "bookmarked": false,58 "actions_summary": [],59 "moderator": false,60 "admin": false,61 "staff": false,62 "user_id": 75248,63 "hidden": false,64 "trust_level": 0,65 "deleted_at": null,66 "user_deleted": false,67 "edit_reason": null,68 "can_view_edit_history": true,69 "wiki": false,70 "post_url": "/t/problems-with-importing-pytorch-for-snake-ml-algorithm-w-patrick-loeber/201439/1",71 "can_accept_answer": false,72 "can_unaccept_answer": false,73 "accepted_answer": false,74 "topic_accepted_answer": null,75 "can_vote": false76 },77 {78 "id": 440286,79 "name": "",80 "username": "ptrblck",81 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",82 "created_at": "2024-04-23T15:28:42.701Z",83 "cooked": "<p>Your PyTorch version might be too old. Could you check it via <code>torch.__version__</code> and update to the latest stable or nightly version, if possible?</p>",84 "post_number": 2,85 "post_type": 1,86 "posts_count": 5,87 "updated_at": "2024-04-23T15:28:42.701Z",88 "reply_count": 1,89 "reply_to_post_number": null,90 "quote_count": 0,91 "incoming_link_count": 21,92 "reads": 4,93 "readers_count": 3,94 "score": 110.8,95 "yours": false,96 "topic_id": 201439,97 "topic_slug": "problems-with-importing-pytorch-for-snake-ml-algorithm-w-patrick-loeber",98 "display_username": "",99 "primary_group_name": null,100 "flair_name": null,101 "flair_url": null,102 "flair_bg_color": null,103 "flair_color": null,104 "flair_group_id": null,105 "badges_granted": [],106 "version": 1,107 "can_edit": false,108 "can_delete": false,109 "can_recover": false,110 "can_see_hidden_post": false,111 "can_wiki": false,112 "read": true,113 "user_title": "",114 "bookmarked": false,115 "actions_summary": [],116 "moderator": true,117 "admin": true,118 "staff": true,119 "user_id": 3534,120 "hidden": false,121 "trust_level": 2,122 "deleted_at": null,123 "user_deleted": false,124 "edit_reason": null,125 "can_view_edit_history": true,126 "wiki": false,127 "post_url": "/t/problems-with-importing-pytorch-for-snake-ml-algorithm-w-patrick-loeber/201439/2",128 "can_accept_answer": false,129 "can_unaccept_answer": false,130 "accepted_answer": false,131 "topic_accepted_answer": null132 },133 {134 "id": 440387,135 "name": "Jamison Ballou",136 "username": "Jamison_Ballou",137 "avatar_template": "/user_avatar/discuss.pytorch.org/jamison_ballou/{size}/62486_2.png",138 "created_at": "2024-04-24T15:07:47.624Z",139 "cooked": "<p>Thank you for replying. PyTorch is the latest version, 2.2.</p>",140 "post_number": 3,141 "post_type": 1,142 "posts_count": 5,143 "updated_at": "2024-04-24T15:08:12.229Z",144 "reply_count": 1,145 "reply_to_post_number": 2,146 "quote_count": 0,147 "incoming_link_count": 9,148 "reads": 4,149 "readers_count": 3,150 "score": 50.8,151 "yours": false,152 "topic_id": 201439,153 "topic_slug": "problems-with-importing-pytorch-for-snake-ml-algorithm-w-patrick-loeber",154 "display_username": "Jamison Ballou",155 "primary_group_name": null,156 "flair_name": null,157 "flair_url": null,158 "flair_bg_color": null,159 "flair_color": null,160 "flair_group_id": null,161 "badges_granted": [],162 "version": 1,163 "can_edit": false,164 "can_delete": false,165 "can_recover": false,166 "can_see_hidden_post": false,167 "can_wiki": false,168 "read": true,169 "user_title": null,170 "reply_to_user": {171 "id": 3534,172 "username": "ptrblck",173 "name": "",174 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png"175 },176 "bookmarked": false,177 "actions_summary": [],178 "moderator": false,179 "admin": false,180 "staff": false,181 "user_id": 75248,182 "hidden": false,183 "trust_level": 0,184 "deleted_at": null,185 "user_deleted": false,186 "edit_reason": null,187 "can_view_edit_history": true,188 "wiki": false,189 "post_url": "/t/problems-with-importing-pytorch-for-snake-ml-algorithm-w-patrick-loeber/201439/3",190 "can_accept_answer": false,191 "can_unaccept_answer": false,192 "accepted_answer": false,193 "topic_accepted_answer": null194 },195 {196 "id": 440397,197 "name": "",198 "username": "ptrblck",199 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",200 "created_at": "2024-04-24T16:25:48.012Z",201 "cooked": "<p>I cannot reproduce the issue using the latest <code>2.3.0</code> release and <code>2.2.2</code>:</p>\n<pre data-code-wrap=\"python\"><code class=\"lang-python\">>>> import torch\n>>> torch.__version__\n'2.3.0+cu121'\n>>> from torch._prims_common import DeviceLikeType\n>>> DeviceLikeType\ntyping.Union[str, torch.device, int]\n</code></pre>\n<pre data-code-wrap=\"python\"><code class=\"lang-python\">>>> import torch\n>>> torch.__version__\n'2.2.2+cu121'\n>>> from torch._prims_common import DeviceLikeType\n>>> DeviceLikeType\ntyping.Union[str, torch.device, int]\n</code></pre>",202 "post_number": 4,203 "post_type": 1,204 "posts_count": 5,205 "updated_at": "2024-04-24T16:25:48.012Z",206 "reply_count": 1,207 "reply_to_post_number": 3,208 "quote_count": 0,209 "incoming_link_count": 16,210 "reads": 4,211 "readers_count": 3,212 "score": 85.8,213 "yours": false,214 "topic_id": 201439,215 "topic_slug": "problems-with-importing-pytorch-for-snake-ml-algorithm-w-patrick-loeber",216 "display_username": "",217 "primary_group_name": null,218 "flair_name": null,219 "flair_url": null,220 "flair_bg_color": null,221 "flair_color": null,222 "flair_group_id": null,223 "badges_granted": [],224 "version": 1,225 "can_edit": false,226 "can_delete": false,227 "can_recover": false,228 "can_see_hidden_post": false,229 "can_wiki": false,230 "read": true,231 "user_title": "",232 "reply_to_user": {233 "id": 75248,234 "username": "Jamison_Ballou",235 "name": "Jamison Ballou",236 "avatar_template": "/user_avatar/discuss.pytorch.org/jamison_ballou/{size}/62486_2.png"237 },238 "bookmarked": false,239 "actions_summary": [],240 "moderator": true,241 "admin": true,242 "staff": true,243 "user_id": 3534,244 "hidden": false,245 "trust_level": 2,246 "deleted_at": null,247 "user_deleted": false,248 "edit_reason": null,249 "can_view_edit_history": true,250 "wiki": false,251 "post_url": "/t/problems-with-importing-pytorch-for-snake-ml-algorithm-w-patrick-loeber/201439/4",252 "can_accept_answer": false,253 "can_unaccept_answer": false,254 "accepted_answer": false,255 "topic_accepted_answer": null256 },257 {258 "id": 440401,259 "name": "Jamison Ballou",260 "username": "Jamison_Ballou",261 "avatar_template": "/user_avatar/discuss.pytorch.org/jamison_ballou/{size}/62486_2.png",262 "created_at": "2024-04-24T16:42:16.364Z",263 "cooked": "<p>Ok, thanks for all your help. Since the program works on my Mac, I won’t worry about fixing it on the Chromebook. Thanks again for your help.</p>",264 "post_number": 5,265 "post_type": 1,266 "posts_count": 5,267 "updated_at": "2024-04-24T16:42:16.364Z",268 "reply_count": 0,269 "reply_to_post_number": 4,270 "quote_count": 0,271 "incoming_link_count": 7,272 "reads": 4,273 "readers_count": 3,274 "score": 35.8,275 "yours": false,276 "topic_id": 201439,277 "topic_slug": "problems-with-importing-pytorch-for-snake-ml-algorithm-w-patrick-loeber",278 "display_username": "Jamison Ballou",279 "primary_group_name": null,280 "flair_name": null,281 "flair_url": null,282 "flair_bg_color": null,283 "flair_color": null,284 "flair_group_id": null,285 "badges_granted": [],286 "version": 1,287 "can_edit": false,288 "can_delete": false,289 "can_recover": false,290 "can_see_hidden_post": false,291 "can_wiki": false,292 "read": true,293 "user_title": null,294 "reply_to_user": {295 "id": 3534,296 "username": "ptrblck",297 "name": "",298 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png"299 },300 "bookmarked": false,301 "actions_summary": [],302 "moderator": false,303 "admin": false,304 "staff": false,305 "user_id": 75248,306 "hidden": false,307 "trust_level": 0,308 "deleted_at": null,309 "user_deleted": false,310 "edit_reason": null,311 "can_view_edit_history": true,312 "wiki": false,313 "post_url": "/t/problems-with-importing-pytorch-for-snake-ml-algorithm-w-patrick-loeber/201439/5",314 "can_accept_answer": false,315 "can_unaccept_answer": false,316 "accepted_answer": false,317 "topic_accepted_answer": null318 }319 ],320 "stream": [321 440239,322 440286,323 440387,324 440397,325 440401326 ]327 },328 "timeline_lookup": [329 [330 1,331 550332 ],333 [334 3,335 549336 ]337 ],338 "suggested_topics": [339 {340 "fancy_title": "Predicting pregnancy codes with transformer",341 "id": 214296,342 "title": "Predicting pregnancy codes with transformer",343 "slug": "predicting-pregnancy-codes-with-transformer",344 "posts_count": 3,345 "reply_count": 2,346 "highest_post_number": 3,347 "image_url": null,348 "created_at": "2024-12-17T05:52:20.311Z",349 "last_posted_at": "2024-12-19T14:09:05.837Z",350 "bumped": true,351 "bumped_at": "2025-01-07T03:43:08.604Z",352 "archetype": "regular",353 "unseen": false,354 "pinned": false,355 "unpinned": null,356 "visible": true,357 "closed": false,358 "archived": false,359 "bookmarked": null,360 "liked": null,361 "tags_descriptions": {},362 "like_count": 0,363 "views": 189,364 "category_id": 1,365 "featured_link": null,366 "has_accepted_answer": false,367 "posters": [368 {369 "extras": "latest",370 "description": "Original Poster, Most Recent Poster",371 "user": {372 "id": 80875,373 "username": "random42",374 "name": "",375 "avatar_template": "/letter_avatar_proxy/v4/letter/r/e8c25b/{size}.png",376 "trust_level": 1377 }378 },379 {380 "extras": null,381 "description": "Frequent Poster",382 "user": {383 "id": 77632,384 "username": "deepsthecoder",385 "name": "Deepika",386 "avatar_template": "/user_avatar/discuss.pytorch.org/deepsthecoder/{size}/66316_2.png",387 "trust_level": 2388 }389 }390 ]391 },392 {393 "fancy_title": "[Bug] Memory leak in C++ libtorch",394 "id": 214376,395 "title": "[Bug] Memory leak in C++ libtorch",396 "slug": "bug-memory-leak-in-c-libtorch",397 "posts_count": 4,398 "reply_count": 1,399 "highest_post_number": 4,400 "image_url": null,401 "created_at": "2024-12-18T23:19:15.780Z",402 "last_posted_at": "2024-12-19T09:06:49.848Z",403 "bumped": true,404 "bumped_at": "2024-12-19T09:06:49.848Z",405 "archetype": "regular",406 "unseen": false,407 "pinned": false,408 "unpinned": null,409 "visible": true,410 "closed": false,411 "archived": false,412 "bookmarked": null,413 "liked": null,414 "tags_descriptions": {},415 "like_count": 1,416 "views": 68,417 "category_id": 1,418 "featured_link": null,419 "has_accepted_answer": false,420 "posters": [421 {422 "extras": "latest",423 "description": "Original Poster, Most Recent Poster",424 "user": {425 "id": 52896,426 "username": "Theophile_Champion",427 "name": "Theophile Champion",428 "avatar_template": "/user_avatar/discuss.pytorch.org/theophile_champion/{size}/30397_2.png",429 "trust_level": 1430 }431 },432 {433 "extras": null,434 "description": "Frequent Poster",435 "user": {436 "id": 41396,437 "username": "soulitzer",438 "name": "",439 "avatar_template": "/letter_avatar_proxy/v4/letter/s/839c29/{size}.png",440 "trust_level": 2441 }442 }443 ]444 },445 {446 "fancy_title": "Best way to find threshold of multilabel action recognition after using MultilabelAveragePrecision",447 "id": 214501,448 "title": "Best way to find threshold of multilabel action recognition after using MultilabelAveragePrecision",449 "slug": "best-way-to-find-threshold-of-multilabel-action-recognition-after-using-multilabelaverageprecision",450 "posts_count": 1,451 "reply_count": 0,452 "highest_post_number": 1,453 "image_url": null,454 "created_at": "2024-12-21T14:51:22.097Z",455 "last_posted_at": "2024-12-21T14:51:22.130Z",456 "bumped": true,457 "bumped_at": "2024-12-21T14:51:22.130Z",458 "archetype": "regular",459 "unseen": false,460 "pinned": false,461 "unpinned": null,462 "visible": true,463 "closed": false,464 "archived": false,465 "bookmarked": null,466 "liked": null,467 "tags_descriptions": {},468 "like_count": 0,469 "views": 31,470 "category_id": 1,471 "featured_link": null,472 "has_accepted_answer": false,473 "posters": [474 {475 "extras": "latest single",476 "description": "Original Poster, Most Recent Poster",477 "user": {478 "id": 81641,479 "username": "Edoardo_Fazzari",480 "name": "Edoardo Fazzari",481 "avatar_template": "/user_avatar/discuss.pytorch.org/edoardo_fazzari/{size}/74661_2.png",482 "trust_level": 0483 }484 }485 ]486 },487 {488 "fancy_title": "How to calculate the activation memory usage of a model",489 "id": 215402,490 "title": "How to calculate the activation memory usage of a model",491 "slug": "how-to-calculate-the-activation-memory-usage-of-a-model",492 "posts_count": 2,493 "reply_count": 0,494 "highest_post_number": 2,495 "image_url": null,496 "created_at": "2025-01-15T02:31:38.595Z",497 "last_posted_at": "2025-01-16T13:52:14.855Z",498 "bumped": true,499 "bumped_at": "2025-01-16T13:52:14.855Z",500 "archetype": "regular",501 "unseen": false,502 "pinned": false,503 "unpinned": null,504 "visible": true,505 "closed": false,506 "archived": false,507 "bookmarked": null,508 "liked": null,509 "tags_descriptions": {},510 "like_count": 0,511 "views": 185,512 "category_id": 1,513 "featured_link": null,514 "has_accepted_answer": false,515 "posters": [516 {517 "extras": null,518 "description": "Original Poster",519 "user": {520 "id": 72471,521 "username": "shadowshadow",522 "name": "",523 "avatar_template": "/user_avatar/discuss.pytorch.org/shadowshadow/{size}/62985_2.png",524 "trust_level": 2525 }526 },527 {528 "extras": "latest",529 "description": "Most Recent Poster",530 "user": {531 "id": 3534,532 "username": "ptrblck",533 "name": "",534 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",535 "admin": true,536 "moderator": true,537 "trust_level": 2538 }539 }540 ]541 },542 {543 "fancy_title": "How to resolve with the issue of adjusting the gaps in my ‘time_idx’ feature in my stock market dataset?",544 "id": 213245,545 "title": "How to resolve with the issue of adjusting the gaps in my 'time_idx' feature in my stock market dataset?",546 "slug": "how-to-resolve-with-the-issue-of-adjusting-the-gaps-in-my-time-idx-feature-in-my-stock-market-dataset",547 "posts_count": 1,548 "reply_count": 0,549 "highest_post_number": 1,550 "image_url": null,551 "created_at": "2024-11-21T05:01:10.859Z",552 "last_posted_at": "2024-11-21T05:01:10.919Z",553 "bumped": true,554 "bumped_at": "2024-11-21T05:01:10.919Z",555 "archetype": "regular",556 "unseen": false,557 "pinned": false,558 "unpinned": null,559 "visible": true,560 "closed": false,561 "archived": false,562 "bookmarked": null,563 "liked": null,564 "tags_descriptions": {},565 "like_count": 0,566 "views": 26,567 "category_id": 1,568 "featured_link": null,569 "has_accepted_answer": false,570 "posters": [571 {572 "extras": "latest single",573 "description": "Original Poster, Most Recent Poster",574 "user": {575 "id": 28027,576 "username": "Satyam_sharma",577 "name": "Satyam sharma",578 "avatar_template": "/user_avatar/discuss.pytorch.org/satyam_sharma/{size}/20898_2.png",579 "trust_level": 0580 }581 }582 ]583 }584 ],585 "tags_descriptions": {},586 "fancy_title": "Problems with Importing PyTorch for Snake ML Algorithm w/ Patrick Loeber",587 "id": 201439,588 "title": "Problems with Importing PyTorch for Snake ML Algorithm w/ Patrick Loeber",589 "posts_count": 5,590 "created_at": "2024-04-23T11:57:06.186Z",591 "views": 1623,592 "reply_count": 3,593 "like_count": 0,594 "last_posted_at": "2024-04-24T16:42:16.364Z",595 "visible": true,596 "closed": false,597 "archived": false,598 "has_summary": false,599 "archetype": "regular",600 "slug": "problems-with-importing-pytorch-for-snake-ml-algorithm-w-patrick-loeber",601 "category_id": 1,602 "word_count": 404,603 "deleted_at": null,604 "user_id": 75248,605 "featured_link": null,606 "pinned_globally": false,607 "pinned_at": null,608 "pinned_until": null,609 "image_url": null,610 "slow_mode_seconds": 0,611 "draft": null,612 "draft_key": "topic_201439",613 "draft_sequence": null,614 "unpinned": null,615 "pinned": false,616 "current_post_number": 1,617 "highest_post_number": 5,618 "deleted_by": null,619 "actions_summary": [620 {621 "id": 4,622 "count": 0,623 "hidden": false,624 "can_act": false625 },626 {627 "id": 8,628 "count": 0,629 "hidden": false,630 "can_act": false631 },632 {633 "id": 10,634 "count": 0,635 "hidden": false,636 "can_act": false637 },638 {639 "id": 7,640 "count": 0,641 "hidden": false,642 "can_act": false643 }644 ],645 "chunk_size": 20,646 "bookmarked": false,647 "topic_timer": null,648 "message_bus_last_id": 0,649 "participant_count": 2,650 "show_read_indicator": false,651 "thumbnails": null,652 "slow_mode_enabled_until": null,653 "can_vote": false,654 "vote_count": 0,655 "user_voted": false,656 "discourse_zendesk_plugin_zendesk_id": null,657 "discourse_zendesk_plugin_zendesk_url": "https://your-url.zendesk.com/agent/tickets/",658 "details": {659 "can_edit": false,660 "notification_level": 1,661 "participants": [662 {663 "id": 75248,664 "username": "Jamison_Ballou",665 "name": "Jamison Ballou",666 "avatar_template": "/user_avatar/discuss.pytorch.org/jamison_ballou/{size}/62486_2.png",667 "post_count": 3,668 "primary_group_name": null,669 "flair_name": null,670 "flair_url": null,671 "flair_color": null,672 "flair_bg_color": null,673 "flair_group_id": null,674 "trust_level": 0675 },676 {677 "id": 3534,678 "username": "ptrblck",679 "name": "",680 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",681 "post_count": 2,682 "primary_group_name": null,683 "flair_name": null,684 "flair_url": null,685 "flair_color": null,686 "flair_bg_color": null,687 "flair_group_id": null,688 "admin": true,689 "moderator": true,690 "trust_level": 2691 }692 ],693 "created_by": {694 "id": 75248,695 "username": "Jamison_Ballou",696 "name": "Jamison Ballou",697 "avatar_template": "/user_avatar/discuss.pytorch.org/jamison_ballou/{size}/62486_2.png"698 },699 "last_poster": {700 "id": 75248,701 "username": "Jamison_Ballou",702 "name": "Jamison Ballou",703 "avatar_template": "/user_avatar/discuss.pytorch.org/jamison_ballou/{size}/62486_2.png"704 },705 "links": [706 {707 "url": "https://www.youtube.com/watch?v=L8ypSXwyBds",708 "title": null,709 "internal": false,710 "attachment": false,711 "reflection": false,712 "clicks": 3,713 "user_id": 75248,714 "domain": "www.youtube.com",715 "root_domain": "youtube.com"716 },717 {718 "url": "https://github.com/patrickloeber/snake-ai-pytorch",719 "title": "GitHub - patrickloeber/snake-ai-pytorch",720 "internal": false,721 "attachment": false,722 "reflection": false,723 "clicks": 1,724 "user_id": 75248,725 "domain": "github.com",726 "root_domain": "github.com"727 }728 ]729 },730 "bookmarks": []731 },732 {733 "post_stream": {734 "posts": [735 {736 "id": 16098,737 "name": "Ben Usman",738 "username": "Ben_Usman",739 "avatar_template": "/user_avatar/discuss.pytorch.org/ben_usman/{size}/743_2.png",740 "created_at": "2017-08-26T02:50:33.353Z",741 "cooked": "<p>Hi,</p>\n<p>I have been trying to figure out why my code crashes after several batches because of cuda memory error. I understand that probably there is some variable(s) that is not freed because I keep it in the graph. The question is how to debug that kind of thing?</p>\n<p>I have wrote a simple line profiler that examines amount of GPU memory on each step, and it seems like on each loss.backwards() step there was massive memory allocation and not all of it frees, it that piles up for a couple of absolutely identical iterations leading to crash. It always crashes on backwards(). I understand that pytorch reuses memory and that is why it might seem like it is not freeing memory, but here is seems like something is indeed leaking.</p>\n<p>I tried explicitly del’ing variables (e.g. with data) to mark them for reallocation by torch, no help. I also used <code>CUDA_LAUNCH_BLOCKING=1</code> to force it to execute things “in place”.</p>\n<p><a href=\"https://pastebin.com/PuCqNyhP\" rel=\"nofollow noopener\">[GPU memory trace]</a></p>\n<p>Is there a way to get a memory footprint like “all tensors allocated on GPU”?</p>\n<p>Changes are that it is not backwards() fault, e.g. something like<br>\na +10Mb - 30 is free<br>\nb +10Mb - 20 is<br>\nc +20Mb - 0 is free<br>\nclean, b leaks and was not marked as free, 30 freed and not releazed<br>\na +10Mb - no allocation (20 is actually free)<br>\nb +10Mb - no allocation, (10 is actually free)<br>\nc +20Mb - new large allocation, breaks, even though b is leaking</p>\n<p>Anyway, how to track these things?</p>",742 "post_number": 1,743 "post_type": 1,744 "posts_count": 34,745 "updated_at": "2017-08-26T02:50:33.353Z",746 "reply_count": 0,747 "reply_to_post_number": null,748 "quote_count": 0,749 "incoming_link_count": 69520,750 "reads": 2414,751 "readers_count": 2413,752 "score": 348569.4,753 "yours": false,754 "topic_id": 6741,755 "topic_slug": "how-to-debug-causes-of-gpu-memory-leaks",756 "display_username": "Ben Usman",757 "primary_group_name": null,758 "flair_name": null,759 "flair_url": null,760 "flair_bg_color": null,761 "flair_color": null,762 "flair_group_id": null,763 "badges_granted": [],764 "version": 1,765 "can_edit": false,766 "can_delete": false,767 "can_recover": false,768 "can_see_hidden_post": false,769 "can_wiki": false,770 "link_counts": [771 {772 "url": "https://pastebin.com/PuCqNyhP",773 "internal": false,774 "reflection": false,775 "title": "model.fcn16 __init__:13 :3.9 Mb super().__i - Pastebin.com",776 "clicks": 1565777 },778 {779 "url": "https://discuss.pytorch.org/t/non-invasive-gpu-memory-profiling/17831",780 "internal": true,781 "reflection": true,782 "title": "Non-invasive GPU memory profiling",783 "clicks": 56784 },785 {786 "url": "https://discuss.pytorch.org/t/pytorch-1-6-0-seem-to-leak-memory-in-conv2d/96461/12",787 "internal": true,788 "reflection": true,789 "title": "Pytorch 1.6.0 seem to leak memory in conv2d",790 "clicks": 14791 },792 {793 "url": "https://discuss.pytorch.org/t/freeze-when-using-garbage-collector-for-debugging/110671",794 "internal": true,795 "reflection": true,796 "title": "Freeze when using garbage collector for debugging",797 "clicks": 3798 }799 ],800 "read": true,801 "user_title": null,802 "bookmarked": false,803 "actions_summary": [804 {805 "id": 2,806 "count": 35807 }808 ],809 "moderator": false,810 "admin": false,811 "staff": false,812 "user_id": 1727,813 "hidden": false,814 "trust_level": 2,815 "deleted_at": null,816 "user_deleted": false,817 "edit_reason": null,818 "can_view_edit_history": true,819 "wiki": false,820 "post_url": "/t/how-to-debug-causes-of-gpu-memory-leaks/6741/1",821 "can_accept_answer": false,822 "can_unaccept_answer": false,823 "accepted_answer": false,824 "topic_accepted_answer": true,825 "can_vote": false826 },827 {828 "id": 16100,829 "name": "Spandan Madan",830 "username": "SpandanMadan",831 "avatar_template": "/user_avatar/discuss.pytorch.org/spandanmadan/{size}/1371_2.png",832 "created_at": "2017-08-26T03:53:45.591Z",833 "cooked": "<p>Great question, hope someone answers. This will be useful to me too!</p>",834 "post_number": 2,835 "post_type": 1,836 "posts_count": 34,837 "updated_at": "2017-08-26T03:53:45.591Z",838 "reply_count": 1,839 "reply_to_post_number": null,840 "quote_count": 0,841 "incoming_link_count": 746,842 "reads": 2523,843 "readers_count": 2522,844 "score": 4284.2,845 "yours": false,846 "topic_id": 6741,847 "topic_slug": "how-to-debug-causes-of-gpu-memory-leaks",848 "display_username": "Spandan Madan",849 "primary_group_name": null,850 "flair_name": null,851 "flair_url": null,852 "flair_bg_color": null,853 "flair_color": null,854 "flair_group_id": null,855 "badges_granted": [],856 "version": 1,857 "can_edit": false,858 "can_delete": false,859 "can_recover": false,860 "can_see_hidden_post": false,861 "can_wiki": false,862 "link_counts": [863 {864 "url": "https://discuss.pytorch.org/t/how-to-automatically-free-intermediate-tensors-in-memory/127243",865 "internal": true,866 "reflection": true,867 "title": "How to automatically free intermediate tensors in memory?",868 "clicks": 19869 },870 {871 "url": "https://discuss.pytorch.org/t/memory-leak-with-wgan-gp-loss/112117",872 "internal": true,873 "reflection": true,874 "title": "Memory leak with WGAN-GP loss",875 "clicks": 16876 }877 ],878 "read": true,879 "user_title": null,880 "bookmarked": false,881 "actions_summary": [882 {883 "id": 2,884 "count": 3885 }886 ],887 "moderator": false,888 "admin": false,889 "staff": false,890 "user_id": 493,891 "hidden": false,892 "trust_level": 2,893 "deleted_at": null,894 "user_deleted": false,895 "edit_reason": null,896 "can_view_edit_history": true,897 "wiki": false,898 "post_url": "/t/how-to-debug-causes-of-gpu-memory-leaks/6741/2",899 "can_accept_answer": false,900 "can_unaccept_answer": false,901 "accepted_answer": false,902 "topic_accepted_answer": true903 },904 {905 "id": 16141,906 "name": "",907 "username": "smth",908 "avatar_template": "/user_avatar/discuss.pytorch.org/smth/{size}/13_2.png",909 "created_at": "2017-08-26T23:44:15.893Z",910 "cooked": "<p>In python, you can use the garbage collector’s book-keeping to print out the currently resident Tensors. Here’s a snippet that shows all the currently allocated Tensors:</p>\n<pre><code class=\"lang-python\"># prints currently alive Tensors and Variables\nimport torch\nimport gc\nfor obj in gc.get_objects():\n try:\n if torch.is_tensor(obj) or (hasattr(obj, 'data') and torch.is_tensor(obj.data)):\n print(type(obj), obj.size())\n except:\n pass\n</code></pre>\n<p>Edited on April 7th, 2019 to add a try/except</p>",911 "post_number": 3,912 "post_type": 1,913 "posts_count": 34,914 "updated_at": "2019-04-07T04:40:00.129Z",915 "reply_count": 3,916 "reply_to_post_number": 2,917 "quote_count": 0,918 "incoming_link_count": 5935,919 "reads": 2503,920 "readers_count": 2502,921 "score": 31069.2,922 "yours": false,923 "topic_id": 6741,924 "topic_slug": "how-to-debug-causes-of-gpu-memory-leaks",925 "display_username": "",926 "primary_group_name": null,927 "flair_name": null,928 "flair_url": null,929 "flair_bg_color": null,930 "flair_color": null,931 "flair_group_id": null,932 "badges_granted": [],933 "version": 2,934 "can_edit": false,935 "can_delete": false,936 "can_recover": false,937 "can_see_hidden_post": false,938 "can_wiki": false,939 "link_counts": [940 {941 "url": "https://discuss.pytorch.org/t/how-to-check-memory-leak-in-a-model/22903",942 "internal": true,943 "reflection": true,944 "title": "How to check memory leak in a model",945 "clicks": 331946 },947 {948 "url": "https://discuss.pytorch.org/t/scope-and-memory-consumption-of-tensors-created-using-self-new-api/19667/4",949 "internal": true,950 "reflection": true,951 "title": "Scope and memory consumption of tensors created using self.new_* API",952 "clicks": 61953 },954 {955 "url": "https://discuss.pytorch.org/t/unable-to-allocate-cuda-memory-when-there-is-enough-of-cached-memory/33296/6",956 "internal": true,957 "reflection": true,958 "title": "Unable to allocate cuda memory, when there is enough of cached memory",959 "clicks": 58960 },961 {962 "url": "https://discuss.pytorch.org/t/phantom-pytorch-data-on-gpu/77235",963 "internal": true,964 "reflection": true,965 "title": "Phantom PyTorch Data on GPU",966 "clicks": 54967 },968 {969 "url": "https://discuss.pytorch.org/t/cpu-memory-usage-leak-because-of-calling-backward/89375",970 "internal": true,971 "reflection": true,972 "title": "CPU memory usage leak because of calling backward",973 "clicks": 29974 },975 {976 "url": "https://discuss.pytorch.org/t/list-all-the-tensors-and-their-memory-allocation/144108/2",977 "internal": true,978 "reflection": true,979 "title": "List all the tensors and their memory allocation",980 "clicks": 4981 },982 {983 "url": "https://discuss.pytorch.org/t/memory-leak-when-using-rpc-for-pipeline-parallelism/127236/7",984 "internal": true,985 "reflection": true,986 "title": "Memory leak when using RPC for pipeline parallelism",987 "clicks": 3988 },989 {990 "url": "https://discuss.pytorch.org/t/memory-leak-when-using-rpc-for-pipeline-parallelism/127236/6",991 "internal": true,992 "reflection": true,993 "title": "Memory leak when using RPC for pipeline parallelism",994 "clicks": 0995 },996 {997 "url": "https://discuss.pytorch.org/t/invalid-gradient-shape-after-discarding-filters-during-training/126285/3",998 "internal": true,999 "reflection": true,1000 "title": "Invalid gradient shape after discarding filters during training",1001 "clicks": 01002 }1003 ],1004 "read": true,1005 "user_title": "PyTorch Dev, Facebook AI Research",1006 "title_is_group": false,1007 "reply_to_user": {1008 "id": 493,1009 "username": "SpandanMadan",1010 "name": "Spandan Madan",1011 "avatar_template": "/user_avatar/discuss.pytorch.org/spandanmadan/{size}/1371_2.png"1012 },1013 "bookmarked": false,1014 "actions_summary": [1015 {1016 "id": 2,1017 "count": 571018 }1019 ],1020 "moderator": true,1021 "admin": true,1022 "staff": true,1023 "user_id": 1,1024 "hidden": false,1025 "trust_level": 2,1026 "deleted_at": null,1027 "user_deleted": false,1028 "edit_reason": null,1029 "can_view_edit_history": true,1030 "wiki": false,1031 "post_url": "/t/how-to-debug-causes-of-gpu-memory-leaks/6741/3",1032 "can_accept_answer": false,1033 "can_unaccept_answer": false,1034 "accepted_answer": true,1035 "topic_accepted_answer": true1036 },1037 {1038 "id": 16430,1039 "name": "Ben Usman",1040 "username": "Ben_Usman",1041 "avatar_template": "/user_avatar/discuss.pytorch.org/ben_usman/{size}/743_2.png",1042 "created_at": "2017-08-29T23:49:32.257Z",1043 "cooked": "<p>Thanks! Seems to work with a <code>try: except</code> block around it (some objects like shared libraries throw exception when you try to do hasattr on them).</p>\n<p>I extended my code that tracked memory usage to also track where memory allocations appeared by comparing set of tensors before and after operation. And results are somewhat surprising. I stopped execution after first batch (it breaks on gpu memory allocation on second batch) and memory consumption was higher in the case where less tensors were allocated O_o</p>\n<p>Here is the diff between sorted two things below, i.e. the one that has more allocations ended up having less memory consumed. One possible explanation is that in model that requested more memory, single model was applied twice and gradient was computed, whereas in the “less consuming”, the trained model is applied once, and then a fixed model is applied ones.</p>\n<pre><code>< + start freeze_params:91 (128, 128, 3, 3) <class 'torch.cuda.FloatTensor'>\n< + start freeze_params:91 (128, 64, 3, 3) <class 'torch.cuda.FloatTensor'>\n< + start freeze_params:91 (128,) <class 'torch.cuda.FloatTensor'>\n< + start freeze_params:91 (21, 21, 32, 32) <class 'torch.cuda.FloatTensor'>\n< + start freeze_params:91 (21, 21, 4, 4) <class 'torch.cuda.FloatTensor'>\n< + start freeze_params:91 (21, 4096, 1, 1) <class 'torch.cuda.FloatTensor'>\n< + start freeze_params:91 (21, 512, 1, 1) <class 'torch.cuda.FloatTensor'>\n< + start freeze_params:91 (21,) <class 'torch.cuda.FloatTensor'>\n< + start freeze_params:91 (256, 128, 3, 3) <class 'torch.cuda.FloatTensor'>\n< + start freeze_params:91 (256, 256, 3, 3) <class 'torch.cuda.FloatTensor'>\n< + start freeze_params:91 (256,) <class 'torch.cuda.FloatTensor'>\n< + start freeze_params:91 (4096, 4096, 1, 1) <class 'torch.cuda.FloatTensor'>\n< + start freeze_params:91 (4096, 512, 7, 7) <class 'torch.cuda.FloatTensor'>\n< + start freeze_params:91 (4096,) <class 'torch.cuda.FloatTensor'>\n< + start freeze_params:91 (512, 256, 3, 3) <class 'torch.cuda.FloatTensor'>\n< + start freeze_params:91 (512, 512, 3, 3) <class 'torch.cuda.FloatTensor'>\n< + start freeze_params:91 (512,) <class 'torch.cuda.FloatTensor'>\n< + start freeze_params:91 (64, 3, 3, 3) <class 'torch.cuda.FloatTensor'>\n< + start freeze_params:91 (64, 64, 3, 3) <class 'torch.cuda.FloatTensor'>\n< + start freeze_params:91 (64,) <class 'torch.cuda.FloatTensor'>\n</code></pre>\n<p>Here are original logs</p>\n<pre><code>:7938.9 Mb\n\n+ __main__ match_source_target:174 (1, 3, 1052, 1914) <class 'torch.cuda.FloatTensor'>\n+ __main__ match_source_target:178 (1, 3, 1052, 1914) <class 'torch.cuda.FloatTensor'>\n+ __main__ match_source_target:190 (128, 128, 3, 3) <class 'torch.cuda.FloatTensor'>\n+ __main__ match_source_target:190 (128, 64, 3, 3) <class 'torch.cuda.FloatTensor'>\n+ __main__ match_source_target:190 (128,) <class 'torch.cuda.FloatTensor'>\n+ __main__ match_source_target:190 (21, 4096, 1, 1) <class 'torch.cuda.FloatTensor'>\n+ __main__ match_source_target:190 (21,) <class 'torch.cuda.FloatTensor'>\n+ __main__ match_source_target:190 (256, 128, 3, 3) <class 'torch.cuda.FloatTensor'>\n+ __main__ match_source_target:190 (256, 256, 3, 3) <class 'torch.cuda.FloatTensor'>\n+ __main__ match_source_target:190 (256,) <class 'torch.cuda.FloatTensor'>\n+ __main__ match_source_target:190 (4096, 4096, 1, 1) <class 'torch.cuda.FloatTensor'>\n+ __main__ match_source_target:190 (4096, 512, 7, 7) <class 'torch.cuda.FloatTensor'>\n+ __main__ match_source_target:190 (4096,) <class 'torch.cuda.FloatTensor'>\n+ __main__ match_source_target:190 (512, 256, 3, 3) <class 'torch.cuda.FloatTensor'>\n+ __main__ match_source_target:190 (512, 512, 3, 3) <class 'torch.cuda.FloatTensor'>\n+ __main__ match_source_target:190 (512,) <class 'torch.cuda.FloatTensor'>\n+ __main__ match_source_target:190 (64, 3, 3, 3) <class 'torch.cuda.FloatTensor'>\n+ __main__ match_source_target:190 (64, 64, 3, 3) <class 'torch.cuda.FloatTensor'>\n+ __main__ match_source_target:190 (64,) <class 'torch.cuda.FloatTensor'>\n+ __main__ run_adaptation:355 (128, 128, 3, 3) <class 'torch.cuda.FloatTensor'>\n+ __main__ run_adaptation:355 (128, 64, 3, 3) <class 'torch.cuda.FloatTensor'>\n+ __main__ run_adaptation:355 (128,) <class 'torch.cuda.FloatTensor'>\n+ __main__ run_adaptation:355 (21, 21, 32, 32) <class 'torch.cuda.FloatTensor'>\n+ __main__ run_adaptation:355 (21, 21, 4, 4) <class 'torch.cuda.FloatTensor'>\n+ __main__ run_adaptation:355 (21, 4096, 1, 1) <class 'torch.cuda.FloatTensor'>\n+ __main__ run_adaptation:355 (21, 512, 1, 1) <class 'torch.cuda.FloatTensor'>\n+ __main__ run_adaptation:355 (21,) <class 'torch.cuda.FloatTensor'>\n+ __main__ run_adaptation:355 (256, 128, 3, 3) <class 'torch.cuda.FloatTensor'>\n+ __main__ run_adaptation:355 (256, 256, 3, 3) <class 'torch.cuda.FloatTensor'>\n+ __main__ run_adaptation:355 (256,) <class 'torch.cuda.FloatTensor'>\n+ __main__ run_adaptation:355 (4096, 4096, 1, 1) <class 'torch.cuda.FloatTensor'>\n+ __main__ run_adaptation:355 (4096, 512, 7, 7) <class 'torch.cuda.FloatTensor'>\n+ __main__ run_adaptation:355 (4096,) <class 'torch.cuda.FloatTensor'>\n+ __main__ run_adaptation:355 (512, 256, 3, 3) <class 'torch.cuda.FloatTensor'>\n+ __main__ run_adaptation:355 (512, 512, 3, 3) <class 'torch.cuda.FloatTensor'>\n+ __main__ run_adaptation:355 (512,) <class 'torch.cuda.FloatTensor'>\n+ __main__ run_adaptation:355 (64, 3, 3, 3) <class 'torch.cuda.FloatTensor'>\n+ __main__ run_adaptation:355 (64, 64, 3, 3) <class 'torch.cuda.FloatTensor'>\n+ __main__ run_adaptation:355 (64,) <class 'torch.cuda.FloatTensor'>\n+ distances.mlp _init_net:55 (1, 500) <class 'torch.cuda.FloatTensor'>\n+ distances.mlp _init_net:55 (1,) <class 'torch.cuda.FloatTensor'>\n+ distances.mlp _init_net:55 (500, 21) <class 'torch.cuda.FloatTensor'>\n+ distances.mlp _init_net:55 (500, 500) <class 'torch.cuda.FloatTensor'>\n+ distances.mlp _init_net:55 (500,) <class 'torch.cuda.FloatTensor'>\n+ distances.mlp objective:26 (2,) <class 'torch.cuda.FloatTensor'>\n+ distances.mlp objective:33 (1,) <class 'torch.cuda.FloatTensor'>\n+ distances.mlp_base attempt_update_d:77 (1, 500) <class 'torch.cuda.FloatTensor'>\n+ distances.mlp_base attempt_update_d:77 (1,) <class 'torch.cuda.FloatTensor'>\n+ distances.mlp_base attempt_update_d:77 (500, 21) <class 'torch.cuda.FloatTensor'>\n+ distances.mlp_base attempt_update_d:77 (500, 500) <class 'torch.cuda.FloatTensor'>\n+ distances.mlp_base attempt_update_d:77 (500,) <class 'torch.cuda.FloatTensor'>\n+ model.fcn16 __init__:13 (21,) <class 'torch.cuda.FloatTensor'>\n+ model.fcn16 features_at:45 (1, 4096, 34, 60) <class 'torch.cuda.FloatTensor'>\n+ model.fcn16 features_at:48 (1, 4096, 34, 60) <class 'torch.cuda.FloatTensor'>\n+ start freeze_params:91 (128, 128, 3, 3) <class 'torch.cuda.FloatTensor'>\n+ start freeze_params:91 (128, 64, 3, 3) <class 'torch.cuda.FloatTensor'>\n+ start freeze_params:91 (128,) <class 'torch.cuda.FloatTensor'>\n+ start freeze_params:91 (21, 21, 32, 32) <class 'torch.cuda.FloatTensor'>\n+ start freeze_params:91 (21, 21, 4, 4) <class 'torch.cuda.FloatTensor'>\n+ start freeze_params:91 (21, 4096, 1, 1) <class 'torch.cuda.FloatTensor'>\n+ start freeze_params:91 (21, 512, 1, 1) <class 'torch.cuda.FloatTensor'>\n+ start freeze_params:91 (21,) <class 'torch.cuda.FloatTensor'>\n+ start freeze_params:91 (256, 128, 3, 3) <class 'torch.cuda.FloatTensor'>\n+ start freeze_params:91 (256, 256, 3, 3) <class 'torch.cuda.FloatTensor'>\n+ start freeze_params:91 (256,) <class 'torch.cuda.FloatTensor'>\n+ start freeze_params:91 (4096, 4096, 1, 1) <class 'torch.cuda.FloatTensor'>\n+ start freeze_params:91 (4096, 512, 7, 7) <class 'torch.cuda.FloatTensor'>\n+ start freeze_params:91 (4096,) <class 'torch.cuda.FloatTensor'>\n+ start freeze_params:91 (512, 256, 3, 3) <class 'torch.cuda.FloatTensor'>\n+ start freeze_params:91 (512, 512, 3, 3) <class 'torch.cuda.FloatTensor'>\n+ start freeze_params:91 (512,) <class 'torch.cuda.FloatTensor'>\n+ start freeze_params:91 (64, 3, 3, 3) <class 'torch.cuda.FloatTensor'>\n+ start freeze_params:91 (64, 64, 3, 3) <class 'torch.cuda.FloatTensor'>\n+ start freeze_params:91 (64,) <class 'torch.cuda.FloatTensor'>\n</code></pre>\n<p>and in the second case - it has less tensors (i.e. identical to above except ~10 tensors less), but higher memory consumption and breaks on second epoch. Any ides?</p>\n<pre><code>:11820.9Mb \n\n+ __main__ match_source_target:174 (1, 3, 1052, 1914) <class 'torch.cuda.FloatTensor'>\n+ __main__ match_source_target:178 (1, 3, 1052, 1914) <class 'torch.cuda.FloatTensor'>\n+ __main__ match_source_target:190 (128, 128, 3, 3) <class 'torch.cuda.FloatTensor'>\n+ __main__ match_source_target:190 (128, 64, 3, 3) <class 'torch.cuda.FloatTensor'>\n+ __main__ match_source_target:190 (128,) <class 'torch.cuda.FloatTensor'>\n+ __main__ match_source_target:190 (21, 4096, 1, 1) <class 'torch.cuda.FloatTensor'>\n+ __main__ match_source_target:190 (21,) <class 'torch.cuda.FloatTensor'>\n+ __main__ match_source_target:190 (256, 128, 3, 3) <class 'torch.cuda.FloatTensor'>\n+ __main__ match_source_target:190 (256, 256, 3, 3) <class 'torch.cuda.FloatTensor'>\n+ __main__ match_source_target:190 (256,) <class 'torch.cuda.FloatTensor'>\n+ __main__ match_source_target:190 (4096, 4096, 1, 1) <class 'torch.cuda.FloatTensor'>\n+ __main__ match_source_target:190 (4096, 512, 7, 7) <class 'torch.cuda.FloatTensor'>\n+ __main__ match_source_target:190 (4096,) <class 'torch.cuda.FloatTensor'>\n+ __main__ match_source_target:190 (512, 256, 3, 3) <class 'torch.cuda.FloatTensor'>\n+ __main__ match_source_target:190 (512, 512, 3, 3) <class 'torch.cuda.FloatTensor'>\n+ __main__ match_source_target:190 (512,) <class 'torch.cuda.FloatTensor'>\n+ __main__ match_source_target:190 (64, 3, 3, 3) <class 'torch.cuda.FloatTensor'>\n+ __main__ match_source_target:190 (64, 64, 3, 3) <class 'torch.cuda.FloatTensor'>\n+ __main__ match_source_target:190 (64,) <class 'torch.cuda.FloatTensor'>\n+ __main__ run_adaptation:355 (128, 128, 3, 3) <class 'torch.cuda.FloatTensor'>\n+ __main__ run_adaptation:355 (128, 64, 3, 3) <class 'torch.cuda.FloatTensor'>\n+ __main__ run_adaptation:355 (128,) <class 'torch.cuda.FloatTensor'>\n+ __main__ run_adaptation:355 (21, 21, 32, 32) <class 'torch.cuda.FloatTensor'>\n+ __main__ run_adaptation:355 (21, 21, 4, 4) <class 'torch.cuda.FloatTensor'>\n+ __main__ run_adaptation:355 (21, 4096, 1, 1) <class 'torch.cuda.FloatTensor'>\n+ __main__ run_adaptation:355 (21, 512, 1, 1) <class 'torch.cuda.FloatTensor'>\n+ __main__ run_adaptation:355 (21,) <class 'torch.cuda.FloatTensor'>\n+ __main__ run_adaptation:355 (256, 128, 3, 3) <class 'torch.cuda.FloatTensor'>\n+ __main__ run_adaptation:355 (256, 256, 3, 3) <class 'torch.cuda.FloatTensor'>\n+ __main__ run_adaptation:355 (256,) <class 'torch.cuda.FloatTensor'>\n+ __main__ run_adaptation:355 (4096, 4096, 1, 1) <class 'torch.cuda.FloatTensor'>\n+ __main__ run_adaptation:355 (4096, 512, 7, 7) <class 'torch.cuda.FloatTensor'>\n+ __main__ run_adaptation:355 (4096,) <class 'torch.cuda.FloatTensor'>\n+ __main__ run_adaptation:355 (512, 256, 3, 3) <class 'torch.cuda.FloatTensor'>\n+ __main__ run_adaptation:355 (512, 512, 3, 3) <class 'torch.cuda.FloatTensor'>\n+ __main__ run_adaptation:355 (512,) <class 'torch.cuda.FloatTensor'>\n+ __main__ run_adaptation:355 (64, 3, 3, 3) <class 'torch.cuda.FloatTensor'>\n+ __main__ run_adaptation:355 (64, 64, 3, 3) <class 'torch.cuda.FloatTensor'>\n+ __main__ run_adaptation:355 (64,) <class 'torch.cuda.FloatTensor'>\n+ distances.mlp _init_net:55 (1, 500) <class 'torch.cuda.FloatTensor'>\n+ distances.mlp _init_net:55 (1,) <class 'torch.cuda.FloatTensor'>\n+ distances.mlp _init_net:55 (500, 21) <class 'torch.cuda.FloatTensor'>\n+ distances.mlp _init_net:55 (500, 500) <class 'torch.cuda.FloatTensor'>\n+ distances.mlp _init_net:55 (500,) <class 'torch.cuda.FloatTensor'>\n+ distances.mlp objective:26 (2,) <class 'torch.cuda.FloatTensor'>\n+ distances.mlp objective:33 (1,) <class 'torch.cuda.FloatTensor'>\n+ distances.mlp_base attempt_update_d:77 (1, 500) <class 'torch.cuda.FloatTensor'>\n+ distances.mlp_base attempt_update_d:77 (1,) <class 'torch.cuda.FloatTensor'>\n+ distances.mlp_base attempt_update_d:77 (500, 21) <class 'torch.cuda.FloatTensor'>\n+ distances.mlp_base attempt_update_d:77 (500, 500) <class 'torch.cuda.FloatTensor'>\n+ distances.mlp_base attempt_update_d:77 (500,) <class 'torch.cuda.FloatTensor'>\n+ model.fcn16 __init__:13 (21,) <class 'torch.cuda.FloatTensor'>\n+ model.fcn16 features_at:45 (1, 4096, 34, 60) <class 'torch.cuda.FloatTensor'>\n+ model.fcn16 features_at:48 (1, 4096, 34, 60) <class 'torch.cuda.FloatTensor'></code></pre>",1044 "post_number": 4,1045 "post_type": 1,1046 "posts_count": 34,1047 "updated_at": "2017-08-30T01:34:20.418Z",1048 "reply_count": 1,1049 "reply_to_post_number": 3,1050 "quote_count": 0,1051 "incoming_link_count": 705,1052 "reads": 2392,1053 "readers_count": 2391,1054 "score": 4003.0,1055 "yours": false,1056 "topic_id": 6741,1057 "topic_slug": "how-to-debug-causes-of-gpu-memory-leaks",1058 "display_username": "Ben Usman",1059 "primary_group_name": null,1060 "flair_name": null,1061 "flair_url": null,1062 "flair_bg_color": null,1063 "flair_color": null,1064 "flair_group_id": null,1065 "badges_granted": [],1066 "version": 2,1067 "can_edit": false,1068 "can_delete": false,1069 "can_recover": false,1070 "can_see_hidden_post": false,1071 "can_wiki": false,1072 "link_counts": [1073 {1074 "url": "http://discuss.pytorch.org/t/understanding-gpu-memory-usage/7160",1075 "internal": true,1076 "reflection": true,1077 "title": "Understanding GPU memory usage",1078 "clicks": 1461079 }1080 ],1081 "read": true,1082 "user_title": null,1083 "reply_to_user": {1084 "id": 1,1085 "username": "smth",1086 "name": "",1087 "avatar_template": "/user_avatar/discuss.pytorch.org/smth/{size}/13_2.png"1088 },1089 "bookmarked": false,1090 "actions_summary": [],1091 "moderator": false,1092 "admin": false,1093 "staff": false,1094 "user_id": 1727,1095 "hidden": false,1096 "trust_level": 2,1097 "deleted_at": null,1098 "user_deleted": false,1099 "edit_reason": null,1100 "can_view_edit_history": true,1101 "wiki": false,1102 "post_url": "/t/how-to-debug-causes-of-gpu-memory-leaks/6741/4",1103 "can_accept_answer": false,1104 "can_unaccept_answer": false,1105 "accepted_answer": false,1106 "topic_accepted_answer": true1107 },1108 {1109 "id": 17154,1110 "name": "",1111 "username": "mys007",1112 "avatar_template": "/letter_avatar_proxy/v4/letter/m/898d66/{size}.png",1113 "created_at": "2017-09-08T10:19:29.582Z",1114 "cooked": "<p><a class=\"mention\" href=\"/u/ben_usman\">@Ben_Usman</a> May I ask what did you use to generate your GPU memory trace? Perhaps python’s trace with added py3nvml calls?</p>",1115 "post_number": 5,1116 "post_type": 1,1117 "posts_count": 34,1118 "updated_at": "2017-09-08T10:52:05.910Z",1119 "reply_count": 0,1120 "reply_to_post_number": null,1121 "quote_count": 0,1122 "incoming_link_count": 142,1123 "reads": 1815,1124 "readers_count": 1814,1125 "score": 1087.8,1126 "yours": false,1127 "topic_id": 6741,1128 "topic_slug": "how-to-debug-causes-of-gpu-memory-leaks",1129 "display_username": "",1130 "primary_group_name": null,1131 "flair_name": null,1132 "flair_url": null,1133 "flair_bg_color": null,1134 "flair_color": null,1135 "flair_group_id": null,1136 "badges_granted": [],1137 "version": 2,1138 "can_edit": false,1139 "can_delete": false,1140 "can_recover": false,1141 "can_see_hidden_post": false,1142 "can_wiki": false,1143 "read": true,1144 "user_title": null,1145 "bookmarked": false,1146 "actions_summary": [1147 {1148 "id": 2,1149 "count": 11150 }1151 ],1152 "moderator": false,1153 "admin": false,1154 "staff": false,1155 "user_id": 3218,1156 "hidden": false,1157 "trust_level": 1,1158 "deleted_at": null,1159 "user_deleted": false,1160 "edit_reason": null,1161 "can_view_edit_history": true,1162 "wiki": false,1163 "post_url": "/t/how-to-debug-causes-of-gpu-memory-leaks/6741/5",1164 "can_accept_answer": false,1165 "can_unaccept_answer": false,1166 "accepted_answer": false,1167 "topic_accepted_answer": true1168 },1169 {1170 "id": 17173,1171 "name": "",1172 "username": "mys007",1173 "avatar_template": "/letter_avatar_proxy/v4/letter/m/898d66/{size}.png",1174 "created_at": "2017-09-08T15:43:43.478Z",1175 "cooked": "<p>To comment on your question, do you use variable-sized batches as input? In that case, that might be caused by memory fragmentation (storages need to be re-allocated)…</p>",1176 "post_number": 6,1177 "post_type": 1,1178 "posts_count": 34,1179 "updated_at": "2017-09-08T15:43:43.478Z",1180 "reply_count": 1,1181 "reply_to_post_number": 4,1182 "quote_count": 0,1183 "incoming_link_count": 207,1184 "reads": 1833,1185 "readers_count": 1832,1186 "score": 1421.2,1187 "yours": false,1188 "topic_id": 6741,1189 "topic_slug": "how-to-debug-causes-of-gpu-memory-leaks",1190 "display_username": "",1191 "primary_group_name": null,1192 "flair_name": null,1193 "flair_url": null,1194 "flair_bg_color": null,1195 "flair_color": null,1196 "flair_group_id": null,1197 "badges_granted": [],1198 "version": 1,1199 "can_edit": false,1200 "can_delete": false,