Anurag1734/cuda-error-resolution-analysis
07
1[2 {3 "post_stream": {4 "posts": [5 {6 "id": 381629,7 "name": "Hendrik",8 "username": "hendrikl",9 "avatar_template": "/user_avatar/discuss.pytorch.org/hendrikl/{size}/50959_2.png",10 "created_at": "2023-01-04T17:31:28.598Z",11 "cooked": "<p>Hello,<br>\nWhen I try to implement ddp in my code (it’s very big, I can provide details), I get the following error on each node I call using SLURM.</p>\n<pre><code class=\"lang-auto\">terminate called after throwing an instance of 'c10::Error'\n what(): Socket Timeout\nException raised from recvBytes at /tmp/coulombc/pytorch_build_2021-11-09_14-57-01/avx2/python3.8/pytorch/torch/csrc/distributed/c10d/Utils.hpp:619 (most recent call first):\nframe #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x55 (0x2ba210653905 in /home/hlohse/.local/lib/python3.8/site-packages/torch/lib/libc10.so)\nframe #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, char const*) + 0xd6 (0x2ba2106352a9 in /home/hlohse/.local/lib/python3.8/site-packages/torch/lib/libc10.so)\nframe #2: c10d::TCPStore::doWait(c10::ArrayRef<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0xe8 (0x2ba1ee132058 in /home/hlohse/.local/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)\n</code></pre>\n<p>What could be the cause for this? I have read it might have to do with <code>init_process_group</code>. I use:<br>\n<code>dist.init_process_group(backend=HP.dist_backend, init_method=HP.init_method, world_size=HP.world_size, rank=rank)</code>, which worked fine earlier.</p>",12 "post_number": 1,13 "post_type": 1,14 "posts_count": 2,15 "updated_at": "2023-01-04T17:31:28.598Z",16 "reply_count": 0,17 "reply_to_post_number": null,18 "quote_count": 0,19 "incoming_link_count": 585,20 "reads": 15,21 "readers_count": 14,22 "score": 2908.0,23 "yours": false,24 "topic_id": 169600,25 "topic_slug": "c10-error-what-socket-timeout",26 "display_username": "Hendrik",27 "primary_group_name": null,28 "flair_name": null,29 "flair_url": null,30 "flair_bg_color": null,31 "flair_color": null,32 "flair_group_id": null,33 "badges_granted": [],34 "version": 1,35 "can_edit": false,36 "can_delete": false,37 "can_recover": false,38 "can_see_hidden_post": false,39 "can_wiki": false,40 "read": true,41 "user_title": null,42 "bookmarked": false,43 "actions_summary": [],44 "moderator": false,45 "admin": false,46 "staff": false,47 "user_id": 57189,48 "hidden": false,49 "trust_level": 1,50 "deleted_at": null,51 "user_deleted": false,52 "edit_reason": null,53 "can_view_edit_history": true,54 "wiki": false,55 "post_url": "/t/c10-error-what-socket-timeout/169600/1",56 "can_accept_answer": false,57 "can_unaccept_answer": false,58 "accepted_answer": false,59 "topic_accepted_answer": null,60 "can_vote": false61 },62 {63 "id": 381630,64 "name": "Hendrik",65 "username": "hendrikl",66 "avatar_template": "/user_avatar/discuss.pytorch.org/hendrikl/{size}/50959_2.png",67 "created_at": "2023-01-04T17:34:09.863Z",68 "cooked": "<p>When I copy <a href=\"https://docs.alliancecan.ca/wiki/PyTorch#PyTorch_with_Multiple_GPUs\" rel=\"noopener nofollow ugc\">the following</a> example, everything works as it should on the same server.</p>\n<pre><code class=\"lang-auto\">#!/bin/bash\n#SBATCH --nodes 1 \n#SBATCH --gres=gpu:2 # Request 2 GPU \"generic resources”.\n#SBATCH --tasks-per-node=2 # Request 1 process per GPU. You will get 1 CPU per process by default. Request more CPUs with the \"cpus-per-task\" parameter to enable multiple data-loader workers to load data in parallel.\n#SBATCH --mem=8G \n#SBATCH --time=0-03:00\n#SBATCH --output=%N-%j.out\n\nmodule load python # Using Default Python version - Make sure to choose a version that suits your application\nvirtualenv --no-download $SLURM_TMPDIR/env\nsource $SLURM_TMPDIR/env/bin/activate\npip install torchvision --no-index\n\nexport NCCL_BLOCKING_WAIT=1 #Set this environment variable if you wish to use the NCCL backend for inter-GPU communication.\nexport MASTER_ADDR=$(hostname) #Store the master node’s IP address in the MASTER_ADDR environment variable.\n\necho \"r$SLURM_NODEID master: $MASTER_ADDR\"\necho \"r$SLURM_NODEID Launching python script\"\n\n# The SLURM_NTASKS variable tells the script how many processes are available for this execution. “srun” executes the script <tasks-per-node * nodes> times\n\nsrun python pytorch-ddp-test.py --init_method tcp://$MASTER_ADDR:3456 --world_size $SLURM_NTASKS --batch_size 256\n</code></pre>\n<pre><code class=\"lang-auto\">import os\nimport time\nimport datetime\n\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport torch.optim as optim\nimport torch.backends.cudnn as cudnn\n\nimport torchvision\nimport torchvision.transforms as transforms\nfrom torchvision.datasets import CIFAR10\nfrom torch.utils.data import DataLoader\n\nimport torch.distributed as dist\nimport torch.utils.data.distributed\n\nimport argparse\n\nparser = argparse.ArgumentParser(description='cifar10 classification models, distributed data parallel test')\nparser.add_argument('--lr', default=0.1, help='')\nparser.add_argument('--batch_size', type=int, default=768, help='')\nparser.add_argument('--max_epochs', type=int, default=4, help='')\nparser.add_argument('--num_workers', type=int, default=0, help='')\n\nparser.add_argument('--init_method', default='tcp://127.0.0.1:3456', type=str, help='')\nparser.add_argument('--dist-backend', default='gloo', type=str, help='')\nparser.add_argument('--world_size', default=1, type=int, help='')\nparser.add_argument('--distributed', action='store_true', help='')\n\ndef main():\n print(\"Starting...\")\n\n args = parser.parse_args()\n\n ngpus_per_node = torch.cuda.device_count()\n\n \"\"\" This next line is the key to getting DistributedDataParallel working on SLURM:\n\t\tSLURM_NODEID is 0 or 1 in this example, SLURM_LOCALID is the id of the \n \t\tcurrent process inside a node and is also 0 or 1 in this example.\"\"\"\n\n local_rank = int(os.environ.get(\"SLURM_LOCALID\")) \n rank = int(os.environ.get(\"SLURM_NODEID\"))*ngpus_per_node + local_rank\n\n current_device = local_rank\n\n torch.cuda.set_device(current_device)\n\n \"\"\" this block initializes a process group and initiate communications\n\t\tbetween all processes running on all nodes \"\"\"\n\n print('From Rank: {}, ==> Initializing Process Group...'.format(rank))\n #init the process group\n dist.init_process_group(backend=args.dist_backend, init_method=args.init_method, world_size=args.world_size, rank=rank)\n print(\"process group ready!\")\n\n print('From Rank: {}, ==> Making model..'.format(rank))\n\n class Net(nn.Module):\n\n def __init__(self):\n super(Net, self).__init__()\n\n self.conv1 = nn.Conv2d(3, 6, 5)\n self.pool = nn.MaxPool2d(2, 2)\n self.conv2 = nn.Conv2d(6, 16, 5)\n self.fc1 = nn.Linear(16 * 5 * 5, 120)\n self.fc2 = nn.Linear(120, 84)\n self.fc3 = nn.Linear(84, 10)\n\n def forward(self, x):\n x = self.pool(F.relu(self.conv1(x)))\n x = self.pool(F.relu(self.conv2(x)))\n x = x.view(-1, 16 * 5 * 5)\n x = F.relu(self.fc1(x))\n x = F.relu(self.fc2(x))\n x = self.fc3(x)\n return x\n\n net = Net()\n\n net.cuda()\n net = torch.nn.parallel.DistributedDataParallel(net, device_ids=[current_device])\n\n print('From Rank: {}, ==> Preparing data..'.format(rank))\n\n transform_train = transforms.Compose([transforms.ToTensor(),transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])\n\n dataset_train = CIFAR10(root='./data', train=True, download=False, transform=transform_train)\n\n train_sampler = torch.utils.data.distributed.DistributedSampler(dataset_train)\n train_loader = DataLoader(dataset_train, batch_size=args.batch_size, shuffle=(train_sampler is None), num_workers=args.num_workers, sampler=train_sampler)\n\n criterion = nn.CrossEntropyLoss().cuda()\n optimizer = optim.SGD(net.parameters(), lr=args.lr, momentum=0.9, weight_decay=1e-4)\n\n for epoch in range(args.max_epochs):\n\n train_sampler.set_epoch(epoch)\n\n train(epoch, net, criterion, optimizer, train_loader, rank)\n\ndef train(epoch, net, criterion, optimizer, train_loader, train_rank):\n\n train_loss = 0\n correct = 0\n total = 0\n\n epoch_start = time.time()\n\n for batch_idx, (inputs, targets) in enumerate(train_loader):\n\n start = time.time()\n\n inputs = inputs.cuda()\n targets = targets.cuda()\n outputs = net(inputs)\n loss = criterion(outputs, targets)\n\n optimizer.zero_grad()\n loss.backward()\n optimizer.step()\n\n train_loss += loss.item()\n _, predicted = outputs.max(1)\n total += targets.size(0)\n correct += predicted.eq(targets).sum().item()\n acc = 100 * correct / total\n\n batch_time = time.time() - start\n\n elapse_time = time.time() - epoch_start\n elapse_time = datetime.timedelta(seconds=elapse_time)\n print(\"From Rank: {}, Training time {}\".format(train_rank, elapse_time))\n\nif __name__=='__main__':\n main()\n</code></pre>",69 "post_number": 2,70 "post_type": 1,71 "posts_count": 2,72 "updated_at": "2023-01-04T17:34:09.863Z",73 "reply_count": 0,74 "reply_to_post_number": null,75 "quote_count": 0,76 "incoming_link_count": 17,77 "reads": 15,78 "readers_count": 14,79 "score": 103.0,80 "yours": false,81 "topic_id": 169600,82 "topic_slug": "c10-error-what-socket-timeout",83 "display_username": "Hendrik",84 "primary_group_name": null,85 "flair_name": null,86 "flair_url": null,87 "flair_bg_color": null,88 "flair_color": null,89 "flair_group_id": null,90 "badges_granted": [],91 "version": 1,92 "can_edit": false,93 "can_delete": false,94 "can_recover": false,95 "can_see_hidden_post": false,96 "can_wiki": false,97 "link_counts": [98 {99 "url": "https://docs.alliancecan.ca/wiki/PyTorch#PyTorch_with_Multiple_GPUs",100 "internal": false,101 "reflection": false,102 "title": "PyTorch - CC Doc",103 "clicks": 10104 }105 ],106 "read": true,107 "user_title": null,108 "bookmarked": false,109 "actions_summary": [110 {111 "id": 2,112 "count": 1113 }114 ],115 "moderator": false,116 "admin": false,117 "staff": false,118 "user_id": 57189,119 "hidden": false,120 "trust_level": 1,121 "deleted_at": null,122 "user_deleted": false,123 "edit_reason": null,124 "can_view_edit_history": true,125 "wiki": false,126 "post_url": "/t/c10-error-what-socket-timeout/169600/2",127 "can_accept_answer": false,128 "can_unaccept_answer": false,129 "accepted_answer": false,130 "topic_accepted_answer": null131 }132 ],133 "stream": [134 381629,135 381630136 ]137 },138 "timeline_lookup": [139 [140 1,141 1025142 ]143 ],144 "suggested_topics": [145 {146 "fancy_title": "DDP with imbalanced loss values",147 "id": 218611,148 "title": "DDP with imbalanced loss values",149 "slug": "ddp-with-imbalanced-loss-values",150 "posts_count": 3,151 "reply_count": 0,152 "highest_post_number": 3,153 "image_url": null,154 "created_at": "2025-04-04T09:04:39.096Z",155 "last_posted_at": "2025-05-17T13:50:25.697Z",156 "bumped": true,157 "bumped_at": "2025-05-17T13:50:25.697Z",158 "archetype": "regular",159 "unseen": false,160 "pinned": false,161 "unpinned": null,162 "visible": true,163 "closed": false,164 "archived": false,165 "bookmarked": null,166 "liked": null,167 "tags_descriptions": {},168 "like_count": 0,169 "views": 114,170 "category_id": 12,171 "featured_link": null,172 "has_accepted_answer": false,173 "posters": [174 {175 "extras": null,176 "description": "Original Poster",177 "user": {178 "id": 8789,179 "username": "jian_zhang",180 "name": "",181 "avatar_template": "/letter_avatar_proxy/v4/letter/j/cc9497/{size}.png",182 "trust_level": 1183 }184 },185 {186 "extras": null,187 "description": "Frequent Poster",188 "user": {189 "id": 32812,190 "username": "Bjorn_Lindqvist",191 "name": "Björn Lindqvist",192 "avatar_template": "/user_avatar/discuss.pytorch.org/bjorn_lindqvist/{size}/25326_2.png",193 "trust_level": 2194 }195 },196 {197 "extras": "latest",198 "description": "Most Recent Poster",199 "user": {200 "id": 84077,201 "username": "python",202 "name": null,203 "avatar_template": "/letter_avatar_proxy/v4/letter/p/ee7513/{size}.png",204 "trust_level": 1205 }206 }207 ]208 },209 {210 "fancy_title": "Work vs. Future sync primitives for Distributed Torch backends",211 "id": 221056,212 "title": "Work vs. Future sync primitives for Distributed Torch backends",213 "slug": "work-vs-future-sync-primitives-for-distributed-torch-backends",214 "posts_count": 3,215 "reply_count": 1,216 "highest_post_number": 3,217 "image_url": null,218 "created_at": "2025-06-25T13:42:13.540Z",219 "last_posted_at": "2025-09-04T16:20:40.727Z",220 "bumped": true,221 "bumped_at": "2025-09-04T16:20:40.727Z",222 "archetype": "regular",223 "unseen": false,224 "pinned": false,225 "unpinned": null,226 "visible": true,227 "closed": false,228 "archived": false,229 "bookmarked": null,230 "liked": null,231 "tags_descriptions": {},232 "like_count": 0,233 "views": 62,234 "category_id": 12,235 "featured_link": null,236 "has_accepted_answer": false,237 "posters": [238 {239 "extras": "latest",240 "description": "Original Poster, Most Recent Poster",241 "user": {242 "id": 84176,243 "username": "mEm",244 "name": null,245 "avatar_template": "/letter_avatar_proxy/v4/letter/m/90ced4/{size}.png",246 "trust_level": 1247 }248 },249 {250 "extras": null,251 "description": "Frequent Poster",252 "user": {253 "id": 6225,254 "username": "yf225",255 "name": "PyTorch Developer, Meta",256 "avatar_template": "/user_avatar/discuss.pytorch.org/yf225/{size}/3418_2.png",257 "trust_level": 2258 }259 }260 ]261 },262 {263 "fancy_title": "What algorithm does NCCL use to perform distributed training?",264 "id": 214572,265 "title": "What algorithm does NCCL use to perform distributed training?",266 "slug": "what-algorithm-does-nccl-use-to-perform-distributed-training",267 "posts_count": 2,268 "reply_count": 0,269 "highest_post_number": 2,270 "image_url": null,271 "created_at": "2024-12-23T15:44:00.219Z",272 "last_posted_at": "2024-12-23T16:00:17.572Z",273 "bumped": true,274 "bumped_at": "2024-12-23T16:00:17.572Z",275 "archetype": "regular",276 "unseen": false,277 "pinned": false,278 "unpinned": null,279 "visible": true,280 "closed": false,281 "archived": false,282 "bookmarked": null,283 "liked": null,284 "tags_descriptions": {},285 "like_count": 0,286 "views": 99,287 "category_id": 12,288 "featured_link": null,289 "has_accepted_answer": false,290 "posters": [291 {292 "extras": null,293 "description": "Original Poster",294 "user": {295 "id": 81678,296 "username": "watalou",297 "name": "",298 "avatar_template": "/user_avatar/discuss.pytorch.org/watalou/{size}/74692_2.png",299 "trust_level": 0300 }301 },302 {303 "extras": "latest",304 "description": "Most Recent Poster",305 "user": {306 "id": 3534,307 "username": "ptrblck",308 "name": "",309 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",310 "admin": true,311 "moderator": true,312 "trust_level": 2313 }314 }315 ]316 },317 {318 "fancy_title": "PyTorch using both GPUs even when after setting explictly",319 "id": 218422,320 "title": "PyTorch using both GPUs even when after setting explictly",321 "slug": "pytorch-using-both-gpus-even-when-after-setting-explictly",322 "posts_count": 3,323 "reply_count": 1,324 "highest_post_number": 3,325 "image_url": null,326 "created_at": "2025-03-31T00:11:55.521Z",327 "last_posted_at": "2025-04-01T22:02:28.424Z",328 "bumped": true,329 "bumped_at": "2025-04-01T22:02:28.424Z",330 "archetype": "regular",331 "unseen": false,332 "pinned": false,333 "unpinned": null,334 "visible": true,335 "closed": false,336 "archived": false,337 "bookmarked": null,338 "liked": null,339 "tags_descriptions": {},340 "like_count": 0,341 "views": 68,342 "category_id": 12,343 "featured_link": null,344 "has_accepted_answer": false,345 "posters": [346 {347 "extras": "latest",348 "description": "Original Poster, Most Recent Poster",349 "user": {350 "id": 83395,351 "username": "Malitha96",352 "name": "",353 "avatar_template": "/letter_avatar_proxy/v4/letter/m/eb9ed0/{size}.png",354 "trust_level": 0355 }356 },357 {358 "extras": null,359 "description": "Frequent Poster",360 "user": {361 "id": 39542,362 "username": "H-Huang",363 "name": "Howard Huang",364 "avatar_template": "/user_avatar/discuss.pytorch.org/h-huang/{size}/35598_2.png",365 "trust_level": 2366 }367 }368 ]369 },370 {371 "fancy_title": "NCCL timeout when reducing batch size",372 "id": 221522,373 "title": "NCCL timeout when reducing batch size",374 "slug": "nccl-timeout-when-reducing-batch-size",375 "posts_count": 2,376 "reply_count": 0,377 "highest_post_number": 2,378 "image_url": null,379 "created_at": "2025-07-14T22:16:09.732Z",380 "last_posted_at": "2025-07-15T14:52:30.610Z",381 "bumped": true,382 "bumped_at": "2025-07-15T14:52:30.610Z",383 "archetype": "regular",384 "unseen": false,385 "pinned": false,386 "unpinned": null,387 "visible": true,388 "closed": false,389 "archived": false,390 "bookmarked": null,391 "liked": null,392 "tags_descriptions": {},393 "like_count": 0,394 "views": 90,395 "category_id": 12,396 "featured_link": null,397 "has_accepted_answer": false,398 "posters": [399 {400 "extras": null,401 "description": "Original Poster",402 "user": {403 "id": 26560,404 "username": "addisonklinke",405 "name": "Addison Klinke",406 "avatar_template": "/user_avatar/discuss.pytorch.org/addisonklinke/{size}/20034_2.png",407 "trust_level": 2408 }409 },410 {411 "extras": "latest",412 "description": "Most Recent Poster",413 "user": {414 "id": 3534,415 "username": "ptrblck",416 "name": "",417 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",418 "admin": true,419 "moderator": true,420 "trust_level": 2421 }422 }423 ]424 }425 ],426 "tags_descriptions": {},427 "fancy_title": "c10::Error what(): Socket Timeout",428 "id": 169600,429 "title": "c10::Error what(): Socket Timeout",430 "posts_count": 2,431 "created_at": "2023-01-04T17:31:28.519Z",432 "views": 1431,433 "reply_count": 0,434 "like_count": 1,435 "last_posted_at": "2023-01-04T17:34:09.863Z",436 "visible": true,437 "closed": false,438 "archived": false,439 "has_summary": false,440 "archetype": "regular",441 "slug": "c10-error-what-socket-timeout",442 "category_id": 12,443 "word_count": 961,444 "deleted_at": null,445 "user_id": 57189,446 "featured_link": null,447 "pinned_globally": false,448 "pinned_at": null,449 "pinned_until": null,450 "image_url": null,451 "slow_mode_seconds": 0,452 "draft": null,453 "draft_key": "topic_169600",454 "draft_sequence": null,455 "unpinned": null,456 "pinned": false,457 "current_post_number": 1,458 "highest_post_number": 2,459 "deleted_by": null,460 "actions_summary": [461 {462 "id": 4,463 "count": 0,464 "hidden": false,465 "can_act": false466 },467 {468 "id": 8,469 "count": 0,470 "hidden": false,471 "can_act": false472 },473 {474 "id": 10,475 "count": 0,476 "hidden": false,477 "can_act": false478 },479 {480 "id": 7,481 "count": 0,482 "hidden": false,483 "can_act": false484 }485 ],486 "chunk_size": 20,487 "bookmarked": false,488 "topic_timer": null,489 "message_bus_last_id": 0,490 "participant_count": 1,491 "show_read_indicator": false,492 "thumbnails": null,493 "slow_mode_enabled_until": null,494 "can_vote": false,495 "vote_count": 0,496 "user_voted": false,497 "discourse_zendesk_plugin_zendesk_id": null,498 "discourse_zendesk_plugin_zendesk_url": "https://your-url.zendesk.com/agent/tickets/",499 "details": {500 "can_edit": false,501 "notification_level": 1,502 "participants": [503 {504 "id": 57189,505 "username": "hendrikl",506 "name": "Hendrik",507 "avatar_template": "/user_avatar/discuss.pytorch.org/hendrikl/{size}/50959_2.png",508 "post_count": 2,509 "primary_group_name": null,510 "flair_name": null,511 "flair_url": null,512 "flair_color": null,513 "flair_bg_color": null,514 "flair_group_id": null,515 "trust_level": 1516 }517 ],518 "created_by": {519 "id": 57189,520 "username": "hendrikl",521 "name": "Hendrik",522 "avatar_template": "/user_avatar/discuss.pytorch.org/hendrikl/{size}/50959_2.png"523 },524 "last_poster": {525 "id": 57189,526 "username": "hendrikl",527 "name": "Hendrik",528 "avatar_template": "/user_avatar/discuss.pytorch.org/hendrikl/{size}/50959_2.png"529 },530 "links": [531 {532 "url": "https://docs.alliancecan.ca/wiki/PyTorch#PyTorch_with_Multiple_GPUs",533 "title": "PyTorch - CC Doc",534 "internal": false,535 "attachment": false,536 "reflection": false,537 "clicks": 10,538 "user_id": 57189,539 "domain": "docs.alliancecan.ca",540 "root_domain": "alliancecan.ca"541 }542 ]543 },544 "bookmarks": []545 },546 {547 "post_stream": {548 "posts": [549 {550 "id": 380817,551 "name": "Vadim Kantorov",552 "username": "vadimkantorov",553 "avatar_template": "/user_avatar/discuss.pytorch.org/vadimkantorov/{size}/365_2.png",554 "created_at": "2022-12-29T16:28:46.076Z",555 "cooked": "<p>Is there a way to get statistics of memory used by activations/intermediate tensors in the autograd graph (after the forward pass completed loss computation)?</p>\n<p>This is useful for debugging insufficient memory savings from autocast.</p>",556 "post_number": 1,557 "post_type": 1,558 "posts_count": 7,559 "updated_at": "2022-12-29T16:28:46.076Z",560 "reply_count": 0,561 "reply_to_post_number": null,562 "quote_count": 0,563 "incoming_link_count": 114,564 "reads": 13,565 "readers_count": 12,566 "score": 567.6,567 "yours": false,568 "topic_id": 169227,569 "topic_slug": "memory-size-of-all-tensors-referenced-by-autograd-graph",570 "display_username": "Vadim Kantorov",571 "primary_group_name": null,572 "flair_name": null,573 "flair_url": null,574 "flair_bg_color": null,575 "flair_color": null,576 "flair_group_id": null,577 "badges_granted": [],578 "version": 1,579 "can_edit": false,580 "can_delete": false,581 "can_recover": false,582 "can_see_hidden_post": false,583 "can_wiki": false,584 "read": true,585 "user_title": null,586 "bookmarked": false,587 "actions_summary": [],588 "moderator": false,589 "admin": false,590 "staff": false,591 "user_id": 775,592 "hidden": false,593 "trust_level": 2,594 "deleted_at": null,595 "user_deleted": false,596 "edit_reason": null,597 "can_view_edit_history": true,598 "wiki": false,599 "post_url": "/t/memory-size-of-all-tensors-referenced-by-autograd-graph/169227/1",600 "can_accept_answer": false,601 "can_unaccept_answer": false,602 "accepted_answer": false,603 "topic_accepted_answer": null,604 "can_vote": false605 },606 {607 "id": 380836,608 "name": "",609 "username": "ptrblck",610 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",611 "created_at": "2022-12-29T19:41:52.605Z",612 "cooked": "<p>You could use forward hooks to check the size of the intermediate forward activations.</p>",613 "post_number": 2,614 "post_type": 1,615 "posts_count": 7,616 "updated_at": "2022-12-29T19:41:52.605Z",617 "reply_count": 1,618 "reply_to_post_number": null,619 "quote_count": 0,620 "incoming_link_count": 1,621 "reads": 14,622 "readers_count": 13,623 "score": 12.8,624 "yours": false,625 "topic_id": 169227,626 "topic_slug": "memory-size-of-all-tensors-referenced-by-autograd-graph",627 "display_username": "",628 "primary_group_name": null,629 "flair_name": null,630 "flair_url": null,631 "flair_bg_color": null,632 "flair_color": null,633 "flair_group_id": null,634 "badges_granted": [],635 "version": 1,636 "can_edit": false,637 "can_delete": false,638 "can_recover": false,639 "can_see_hidden_post": false,640 "can_wiki": false,641 "read": true,642 "user_title": "",643 "bookmarked": false,644 "actions_summary": [],645 "moderator": true,646 "admin": true,647 "staff": true,648 "user_id": 3534,649 "hidden": false,650 "trust_level": 2,651 "deleted_at": null,652 "user_deleted": false,653 "edit_reason": null,654 "can_view_edit_history": true,655 "wiki": false,656 "post_url": "/t/memory-size-of-all-tensors-referenced-by-autograd-graph/169227/2",657 "can_accept_answer": false,658 "can_unaccept_answer": false,659 "accepted_answer": false,660 "topic_accepted_answer": null661 },662 {663 "id": 380841,664 "name": "Vadim Kantorov",665 "username": "vadimkantorov",666 "avatar_template": "/user_avatar/discuss.pytorch.org/vadimkantorov/{size}/365_2.png",667 "created_at": "2022-12-29T22:21:20.589Z",668 "cooked": "<p>But it won’t take into account certain tensors being garbage-collected, inplace ops and some intermediate tensor saved_for_backward within the C++ ops, right?</p>",669 "post_number": 3,670 "post_type": 1,671 "posts_count": 7,672 "updated_at": "2022-12-29T22:21:20.589Z",673 "reply_count": 1,674 "reply_to_post_number": 2,675 "quote_count": 0,676 "incoming_link_count": 0,677 "reads": 14,678 "readers_count": 13,679 "score": 7.8,680 "yours": false,681 "topic_id": 169227,682 "topic_slug": "memory-size-of-all-tensors-referenced-by-autograd-graph",683 "display_username": "Vadim Kantorov",684 "primary_group_name": null,685 "flair_name": null,686 "flair_url": null,687 "flair_bg_color": null,688 "flair_color": null,689 "flair_group_id": null,690 "badges_granted": [],691 "version": 1,692 "can_edit": false,693 "can_delete": false,694 "can_recover": false,695 "can_see_hidden_post": false,696 "can_wiki": false,697 "read": true,698 "user_title": null,699 "reply_to_user": {700 "id": 3534,701 "username": "ptrblck",702 "name": "",703 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png"704 },705 "bookmarked": false,706 "actions_summary": [],707 "moderator": false,708 "admin": false,709 "staff": false,710 "user_id": 775,711 "hidden": false,712 "trust_level": 2,713 "deleted_at": null,714 "user_deleted": false,715 "edit_reason": null,716 "can_view_edit_history": true,717 "wiki": false,718 "post_url": "/t/memory-size-of-all-tensors-referenced-by-autograd-graph/169227/3",719 "can_accept_answer": false,720 "can_unaccept_answer": false,721 "accepted_answer": false,722 "topic_accepted_answer": null723 },724 {725 "id": 380842,726 "name": "Vadim Kantorov",727 "username": "vadimkantorov",728 "avatar_template": "/user_avatar/discuss.pytorch.org/vadimkantorov/{size}/365_2.png",729 "created_at": "2022-12-29T22:26:25.995Z",730 "cooked": "<p>Basically, the question is still on not easily observable memory consumption in combination with unclearness what gets actually stored as fp16 as a single copy and how/where casts are done and whether they are done on the fly or not. Autocast is being too “auto” and too magical <img src=\"https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12\" title=\":slight_smile:\" class=\"emoji\" alt=\":slight_smile:\" loading=\"lazy\" width=\"20\" height=\"20\"></p>\n<p>It might be good to somehow have tracing mode to understand at what point are casts executed (while in eager mode).</p>",731 "post_number": 4,732 "post_type": 1,733 "posts_count": 7,734 "updated_at": "2022-12-29T22:29:07.616Z",735 "reply_count": 1,736 "reply_to_post_number": 3,737 "quote_count": 0,738 "incoming_link_count": 22,739 "reads": 14,740 "readers_count": 13,741 "score": 117.8,742 "yours": false,743 "topic_id": 169227,744 "topic_slug": "memory-size-of-all-tensors-referenced-by-autograd-graph",745 "display_username": "Vadim Kantorov",746 "primary_group_name": null,747 "flair_name": null,748 "flair_url": null,749 "flair_bg_color": null,750 "flair_color": null,751 "flair_group_id": null,752 "badges_granted": [],753 "version": 1,754 "can_edit": false,755 "can_delete": false,756 "can_recover": false,757 "can_see_hidden_post": false,758 "can_wiki": false,759 "read": true,760 "user_title": null,761 "reply_to_user": {762 "id": 775,763 "username": "vadimkantorov",764 "name": "Vadim Kantorov",765 "avatar_template": "/user_avatar/discuss.pytorch.org/vadimkantorov/{size}/365_2.png"766 },767 "bookmarked": false,768 "actions_summary": [],769 "moderator": false,770 "admin": false,771 "staff": false,772 "user_id": 775,773 "hidden": false,774 "trust_level": 2,775 "deleted_at": null,776 "user_deleted": false,777 "edit_reason": null,778 "can_view_edit_history": true,779 "wiki": false,780 "post_url": "/t/memory-size-of-all-tensors-referenced-by-autograd-graph/169227/4",781 "can_accept_answer": false,782 "can_unaccept_answer": false,783 "accepted_answer": false,784 "topic_accepted_answer": null785 },786 {787 "id": 380845,788 "name": "",789 "username": "ptrblck",790 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",791 "created_at": "2022-12-29T22:50:49.598Z",792 "cooked": "<p>To check for casts and track operations you could use something like this:</p>\n<pre><code class=\"lang-python\">import torch\nfrom torch.testing._internal.logging_tensor import LoggingTensorMode, capture_logs\n\nmodel = torch.nn.Linear(10, 10).cuda()\nx = torch.randn(1, 10).cuda()\n\nwith capture_logs(is_mode=True) as logs, LoggingTensorMode():\n with torch.autocast(device_type='cuda'):\n out = model(x)\n\nfor l in logs:\n print(l)\n</code></pre>\n<p>which should reduce the “magic”.<br>\nI’m not aware of another way to check for intermediates created in the backend.</p>",793 "post_number": 5,794 "post_type": 1,795 "posts_count": 7,796 "updated_at": "2022-12-29T22:50:49.598Z",797 "reply_count": 1,798 "reply_to_post_number": 4,799 "quote_count": 0,800 "incoming_link_count": 12,801 "reads": 13,802 "readers_count": 12,803 "score": 67.6,804 "yours": false,805 "topic_id": 169227,806 "topic_slug": "memory-size-of-all-tensors-referenced-by-autograd-graph",807 "display_username": "",808 "primary_group_name": null,809 "flair_name": null,810 "flair_url": null,811 "flair_bg_color": null,812 "flair_color": null,813 "flair_group_id": null,814 "badges_granted": [],815 "version": 1,816 "can_edit": false,817 "can_delete": false,818 "can_recover": false,819 "can_see_hidden_post": false,820 "can_wiki": false,821 "read": true,822 "user_title": "",823 "reply_to_user": {824 "id": 775,825 "username": "vadimkantorov",826 "name": "Vadim Kantorov",827 "avatar_template": "/user_avatar/discuss.pytorch.org/vadimkantorov/{size}/365_2.png"828 },829 "bookmarked": false,830 "actions_summary": [],831 "moderator": true,832 "admin": true,833 "staff": true,834 "user_id": 3534,835 "hidden": false,836 "trust_level": 2,837 "deleted_at": null,838 "user_deleted": false,839 "edit_reason": null,840 "can_view_edit_history": true,841 "wiki": false,842 "post_url": "/t/memory-size-of-all-tensors-referenced-by-autograd-graph/169227/5",843 "can_accept_answer": false,844 "can_unaccept_answer": false,845 "accepted_answer": false,846 "topic_accepted_answer": null847 },848 {849 "id": 381599,850 "name": "Vadim Kantorov",851 "username": "vadimkantorov",852 "avatar_template": "/user_avatar/discuss.pytorch.org/vadimkantorov/{size}/365_2.png",853 "created_at": "2023-01-04T11:05:48.409Z",854 "cooked": "<p>Curious, how will it show “cached” casts in autocast? What casts are “cached” in autocast? Casts of model parameters from fp32 to fp16, right? At what moments is “cache” invalidated or flushed?</p>\n<p>Is autocast implemented as some sort of wrapper for function calls? E.g. if I call several functions on activation of a previous module, will there be two redundant different casts fp32->fp16 for two branches of computation? magic <img src=\"https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12\" title=\":frowning:\" class=\"emoji\" alt=\":frowning:\" loading=\"lazy\" width=\"20\" height=\"20\"></p>",855 "post_number": 6,856 "post_type": 1,857 "posts_count": 7,858 "updated_at": "2023-01-04T11:07:35.040Z",859 "reply_count": 0,860 "reply_to_post_number": 5,861 "quote_count": 0,862 "incoming_link_count": 3,863 "reads": 11,864 "readers_count": 10,865 "score": 17.2,866 "yours": false,867 "topic_id": 169227,868 "topic_slug": "memory-size-of-all-tensors-referenced-by-autograd-graph",869 "display_username": "Vadim Kantorov",870 "primary_group_name": null,871 "flair_name": null,872 "flair_url": null,873 "flair_bg_color": null,874 "flair_color": null,875 "flair_group_id": null,876 "badges_granted": [],877 "version": 1,878 "can_edit": false,879 "can_delete": false,880 "can_recover": false,881 "can_see_hidden_post": false,882 "can_wiki": false,883 "read": true,884 "user_title": null,885 "reply_to_user": {886 "id": 3534,887 "username": "ptrblck",888 "name": "",889 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png"890 },891 "bookmarked": false,892 "actions_summary": [],893 "moderator": false,894 "admin": false,895 "staff": false,896 "user_id": 775,897 "hidden": false,898 "trust_level": 2,899 "deleted_at": null,900 "user_deleted": false,901 "edit_reason": null,902 "can_view_edit_history": true,903 "wiki": false,904 "post_url": "/t/memory-size-of-all-tensors-referenced-by-autograd-graph/169227/6",905 "can_accept_answer": false,906 "can_unaccept_answer": false,907 "accepted_answer": false,908 "topic_accepted_answer": null909 },910 {911 "id": 381626,912 "name": "Vadim Kantorov",913 "username": "vadimkantorov",914 "avatar_template": "/user_avatar/discuss.pytorch.org/vadimkantorov/{size}/365_2.png",915 "created_at": "2023-01-04T15:04:35.197Z",916 "cooked": "<p>This logging might also be much better if these extensions were implemented: <a href=\"https://github.com/albanD/subclass_zoo/blob/main/logging_mode.py\" class=\"inline-onebox\" rel=\"noopener nofollow ugc\">subclass_zoo/logging_mode.py at main · albanD/subclass_zoo · GitHub</a> (adding shape and dtype to the text trace)</p>\n<p>Related issue: <a href=\"https://github.com/pytorch/pytorch/issues/81750\" class=\"inline-onebox\" rel=\"noopener nofollow ugc\">Modernize logging tensor in torch.testing._internal · Issue #81750 · pytorch/pytorch · GitHub</a></p>",917 "post_number": 7,918 "post_type": 1,919 "posts_count": 7,920 "updated_at": "2023-01-04T15:04:35.197Z",921 "reply_count": 0,922 "reply_to_post_number": null,923 "quote_count": 0,924 "incoming_link_count": 4,925 "reads": 11,926 "readers_count": 10,927 "score": 22.2,928 "yours": false,929 "topic_id": 169227,930 "topic_slug": "memory-size-of-all-tensors-referenced-by-autograd-graph",931 "display_username": "Vadim Kantorov",932 "primary_group_name": null,933 "flair_name": null,934 "flair_url": null,935 "flair_bg_color": null,936 "flair_color": null,937 "flair_group_id": null,938 "badges_granted": [],939 "version": 1,940 "can_edit": false,941 "can_delete": false,942 "can_recover": false,943 "can_see_hidden_post": false,944 "can_wiki": false,945 "link_counts": [946 {947 "url": "https://github.com/albanD/subclass_zoo/blob/main/logging_mode.py",948 "internal": false,949 "reflection": false,950 "title": "subclass_zoo/logging_mode.py at main · albanD/subclass_zoo · GitHub",951 "clicks": 2952 },953 {954 "url": "https://github.com/pytorch/pytorch/issues/81750",955 "internal": false,956 "reflection": false,957 "title": "Modernize logging tensor in torch.testing._internal · Issue #81750 · pytorch/pytorch · GitHub",958 "clicks": 0959 }960 ],961 "read": true,962 "user_title": null,963 "bookmarked": false,964 "actions_summary": [],965 "moderator": false,966 "admin": false,967 "staff": false,968 "user_id": 775,969 "hidden": false,970 "trust_level": 2,971 "deleted_at": null,972 "user_deleted": false,973 "edit_reason": null,974 "can_view_edit_history": true,975 "wiki": false,976 "post_url": "/t/memory-size-of-all-tensors-referenced-by-autograd-graph/169227/7",977 "can_accept_answer": false,978 "can_unaccept_answer": false,979 "accepted_answer": false,980 "topic_accepted_answer": null981 }982 ],983 "stream": [984 380817,985 380836,986 380841,987 380842,988 380845,989 381599,990 381626991 ]992 },993 "timeline_lookup": [994 [995 1,996 1031997 ],998 [999 6,1000 10251001 ]1002 ],1003 "suggested_topics": [1004 {1005 "fancy_title": "Slow convolutions on CPU with autocast",1006 "id": 214226,1007 "title": "Slow convolutions on CPU with autocast",1008 "slug": "slow-convolutions-on-cpu-with-autocast",1009 "posts_count": 3,1010 "reply_count": 1,1011 "highest_post_number": 3,1012 "image_url": null,1013 "created_at": "2024-12-14T16:09:06.447Z",1014 "last_posted_at": "2024-12-14T17:25:11.751Z",1015 "bumped": true,1016 "bumped_at": "2024-12-14T17:33:08.481Z",1017 "archetype": "regular",1018 "unseen": false,1019 "pinned": false,1020 "unpinned": null,1021 "visible": true,1022 "closed": false,1023 "archived": false,1024 "bookmarked": null,1025 "liked": null,1026 "tags_descriptions": {},1027 "like_count": 0,1028 "views": 258,1029 "category_id": 27,1030 "featured_link": null,1031 "has_accepted_answer": false,1032 "posters": [1033 {1034 "extras": "latest single",1035 "description": "Original Poster, Most Recent Poster",1036 "user": {1037 "id": 81089,1038 "username": "Aknw_Fen",1039 "name": "Aknw Fen",1040 "avatar_template": "/user_avatar/discuss.pytorch.org/aknw_fen/{size}/74156_2.png",1041 "trust_level": 21042 }1043 }1044 ]1045 },1046 {1047 "fancy_title": "Any operator is supported on fp8 tensor?",1048 "id": 212371,1049 "title": "Any operator is supported on fp8 tensor?",1050 "slug": "any-operator-is-supported-on-fp8-tensor",1051 "posts_count": 8,1052 "reply_count": 6,1053 "highest_post_number": 8,1054 "image_url": null,1055 "created_at": "2024-10-31T16:06:44.298Z",1056 "last_posted_at": "2024-11-05T16:48:48.277Z",1057 "bumped": true,1058 "bumped_at": "2024-11-05T16:48:48.277Z",1059 "archetype": "regular",1060 "unseen": false,1061 "pinned": false,1062 "unpinned": null,1063 "visible": true,1064 "closed": false,1065 "archived": false,1066 "bookmarked": null,1067 "liked": null,1068 "tags_descriptions": {},1069 "like_count": 1,1070 "views": 3166,1071 "category_id": 27,1072 "featured_link": null,1073 "has_accepted_answer": false,1074 "posters": [1075 {1076 "extras": "latest",1077 "description": "Original Poster, Most Recent Poster",1078 "user": {1079 "id": 57464,1080 "username": "cokespace2",1081 "name": "Vince Mo",1082 "avatar_template": "/user_avatar/discuss.pytorch.org/cokespace2/{size}/51244_2.png",1083 "trust_level": 21084 }1085 },1086 {1087 "extras": null,1088 "description": "Frequent Poster",1089 "user": {1090 "id": 43941,1091 "username": "marksaroufim",1092 "name": "Mark Saroufim",1093 "avatar_template": "/user_avatar/discuss.pytorch.org/marksaroufim/{size}/36747_2.png",1094 "trust_level": 31095 }1096 },1097 {1098 "extras": null,1099 "description": "Frequent Poster",1100 "user": {1101 "id": 3534,1102 "username": "ptrblck",1103 "name": "",1104 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",1105 "admin": true,1106 "moderator": true,1107 "trust_level": 21108 }1109 }1110 ]1111 },1112 {1113 "fancy_title": "FCN ResNet18 low precision on SUNRGBD dataset",1114 "id": 213193,1115 "title": "FCN ResNet18 low precision on SUNRGBD dataset",1116 "slug": "fcn-resnet18-low-precision-on-sunrgbd-dataset",1117 "posts_count": 1,1118 "reply_count": 0,1119 "highest_post_number": 1,1120 "image_url": null,1121 "created_at": "2024-11-20T09:38:34.082Z",1122 "last_posted_at": "2024-11-20T09:38:34.146Z",1123 "bumped": true,1124 "bumped_at": "2024-11-20T09:38:34.146Z",1125 "archetype": "regular",1126 "unseen": false,1127 "pinned": false,1128 "unpinned": null,1129 "visible": true,1130 "closed": false,1131 "archived": false,1132 "bookmarked": null,1133 "liked": null,1134 "tags_descriptions": {},1135 "like_count": 0,1136 "views": 151,1137 "category_id": 27,1138 "featured_link": null,1139 "has_accepted_answer": false,1140 "posters": [1141 {1142 "extras": "latest single",1143 "description": "Original Poster, Most Recent Poster",1144 "user": {1145 "id": 80652,1146 "username": "Cupa_cups",1147 "name": "Luka Tankosić",1148 "avatar_template": "/letter_avatar_proxy/v4/letter/c/ed655f/{size}.png",1149 "trust_level": 01150 }1151 }1152 ]1153 },1154 {1155 "fancy_title": "The dtype of optimizer states in PyTorch AMP training",1156 "id": 214067,1157 "title": "The dtype of optimizer states in PyTorch AMP training",1158 "slug": "the-dtype-of-optimizer-states-in-pytorch-amp-training",1159 "posts_count": 2,1160 "reply_count": 0,1161 "highest_post_number": 2,1162 "image_url": null,1163 "created_at": "2024-12-10T18:02:16.235Z",1164 "last_posted_at": "2024-12-10T22:56:43.015Z",1165 "bumped": true,1166 "bumped_at": "2024-12-10T22:56:43.015Z",1167 "archetype": "regular",1168 "unseen": false,1169 "pinned": false,1170 "unpinned": null,1171 "visible": true,1172 "closed": false,1173 "archived": false,1174 "bookmarked": null,1175 "liked": null,1176 "tags_descriptions": {},1177 "like_count": 1,1178 "views": 254,1179 "category_id": 27,1180 "featured_link": null,1181 "has_accepted_answer": false,1182 "posters": [1183 {1184 "extras": null,1185 "description": "Original Poster",1186 "user": {1187 "id": 81434,1188 "username": "Chaohao_Yang",1189 "name": "Chaohao Yang",1190 "avatar_template": "/user_avatar/discuss.pytorch.org/chaohao_yang/{size}/74456_2.png",1191 "trust_level": 01192 }1193 },1194 {1195 "extras": "latest",1196 "description": "Most Recent Poster",1197 "user": {1198 "id": 3534,1199 "username": "ptrblck",1200 "name": "",