Anurag1734/cuda-error-resolution-analysis
07
1[2 {3 "post_stream": {4 "posts": [5 {6 "id": 435580,7 "name": "끼링 후",8 "username": "biology9763",9 "avatar_template": "/user_avatar/discuss.pytorch.org/biology9763/{size}/68362_2.png",10 "created_at": "2024-03-10T14:32:47.112Z",11 "cooked": "<p>I am currently using SlowFast on Facebook.<br>\nI am using two 4060ti(16GB) GPUs for training, and I have successfully parallelized the computation using NCCL on Ubuntu.</p>\n<p>However, when the memory consumption exceeds the capacity of one GPU (16GB), the training does not proceed. It seems that only parallel computation within 16GB is happening, instead of utilizing the full 32GB (16+16).</p>\n<p>For example) When training the same model with a single GPU on Windows, it consumes 15.6GB of VRAM and the training proceeds without any issues. However, when using two GPUs in a Ubuntu environment, an “out of memory” (OOM) error occurs:</p>\n<pre><code class=\"lang-auto\">RuntimeError: CUDA error: out of memory\nCUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.\nFor debugging consider passing CUDA_LAUNCH_BLOCKING=1.\n</code></pre>\n<p>Reducing the size of the training data allows the training to continue.</p>\n<p>It’s unclear whether this issue arises from exceeding the VRAM limit on GPU0, which might be over 16GB, or if it’s due to insufficient RAM allocated to Ubuntu causing the OOM problem.<br>\nI have heard about methods like Distribute and DDP for this purpose.</p>\n<p>In the SlowFast model, there is a setting gpu_id = None, which seems like it can be changed. However, when I set gpu_id=[0,1], it doesn’t work and throws an error.</p>\n<p>– Process 1 terminated with the following error:<br>\nTraceback (most recent call last):<br>\nFile “/home/hookkiring/.venv/lib/python3.10/site-packages/torch/multiprocessing/spawn.py”, line 69, in _wrap<br>\nfn(i, *args)<br>\nFile “/home/hookkiring/slowfast/slowfast/utils/multiprocessing.py”, line 60, in run<br>\nret = func(cfg)<br>\nFile “/home/hookkiring/slowfast/tools/train_net.py”, line 594, in train<br>\nmodel = build_model(cfg)<br>\nFile “/home/hookkiring/slowfast/slowfast/models/build.py”, line 67, in build_model<br>\nmodel = model.cuda(device=cur_device)<br>\nFile “/home/hookkiring/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 689, in cuda<br>\nreturn self._apply(lambda t: t.cuda(device))<br>\nFile “/home/hookkiring/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 579, in _apply<br>\nmodule._apply(fn)<br>\nFile “/home/hookkiring/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 579, in _apply<br>\nmodule._apply(fn)<br>\nFile “/home/hookkiring/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 579, in _apply<br>\nmodule._apply(fn)<br>\nFile “/home/hookkiring/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 602, in _apply<br>\nparam_applied = fn(param)<br>\nFile “/home/hookkiring/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 689, in <br>\nreturn self._apply(lambda t: t.cuda(device))<br>\nTypeError: cuda(): argument ‘device’ (position 1) must be torch.device, not list</p>\n<pre data-code-wrap=\"egistry\"><code class=\"lang-egistry\">\nThe registered object will be called with `obj(cfg)`.\nThe call should return a `torch.nn.Module` object.\n\"\"\"\n\n\ndef build_model(cfg, gpu_id=None):\n \"\"\"\n Builds the video model.\n Args:\n cfg (configs): configs that contains the hyper-parameters to build the\n backbone. Details can be seen in slowfast/config/defaults.py.\n gpu_id (Optional[int]): specify the gpu index to build model.\n \"\"\"\n if torch.cuda.is_available():\n assert (\n cfg.NUM_GPUS <= torch.cuda.device_count()\n ), \"Cannot use more GPU devices than available\"\n else:\n assert (\n cfg.NUM_GPUS == 0\n ), \"Cuda is not available. Please set `NUM_GPUS: 0 for running on CPUs.\"\n\n # Construct the model\n name = cfg.MODEL.MODEL_NAME\n model = MODEL_REGISTRY.get(name)(cfg)\n\n if cfg.BN.NORM_TYPE == \"sync_batchnorm_apex\":\n try:\n import apex\n except ImportError:\n raise ImportError(\"APEX is required for this model, pelase install\")\n\n logger.info(\"Converting BN layers to Apex SyncBN\")\n process_group = apex.parallel.create_syncbn_process_group(\n group_size=cfg.BN.NUM_SYNC_DEVICES\n )\n model = apex.parallel.convert_syncbn_model(\n model, process_group=process_group\n )\n\n if cfg.NUM_GPUS:\n if gpu_id is None:\n # Determine the GPU used by the current process\n cur_device = torch.cuda.current_device()\n else:\n cur_device = gpu_id\n # Transfer the model to the current GPU device\n model = model.cuda(device=cur_device)\n # Use multi-process data parallel model in the multi-gpu setting\n if cfg.NUM_GPUS > 1:\n # Make model replica operate on the current device\n model = torch.nn.parallel.DistributedDataParallel(\n module=model,\n device_ids=[cur_device],\n output_device=cur_device,\n find_unused_parameters=True\n if cfg.MODEL.DETACH_FINAL_FC\n or cfg.MODEL.MODEL_NAME == \"ContrastiveModel\"\n else False,\n )\n if cfg.MODEL.FP16_ALLREDUCE:\n model.register_comm_hook(\n state=None, hook=comm_hooks_default.fp16_compress_hook\n )\n return model\n\ntype or paste code here\n</code></pre>",12 "post_number": 1,13 "post_type": 1,14 "posts_count": 1,15 "updated_at": "2024-03-10T15:02:43.129Z",16 "reply_count": 0,17 "reply_to_post_number": null,18 "quote_count": 0,19 "incoming_link_count": 45,20 "reads": 6,21 "readers_count": 5,22 "score": 221.2,23 "yours": false,24 "topic_id": 198618,25 "topic_slug": "how-can-i-use-2-gpu-vram-100-slowfast-model",26 "display_username": "끼링 후",27 "primary_group_name": null,28 "flair_name": null,29 "flair_url": null,30 "flair_bg_color": null,31 "flair_color": null,32 "flair_group_id": null,33 "badges_granted": [],34 "version": 4,35 "can_edit": false,36 "can_delete": false,37 "can_recover": false,38 "can_see_hidden_post": false,39 "can_wiki": false,40 "read": true,41 "user_title": null,42 "bookmarked": false,43 "actions_summary": [],44 "moderator": false,45 "admin": false,46 "staff": false,47 "user_id": 74012,48 "hidden": false,49 "trust_level": 1,50 "deleted_at": null,51 "user_deleted": false,52 "edit_reason": null,53 "can_view_edit_history": true,54 "wiki": false,55 "post_url": "/t/how-can-i-use-2-gpu-vram-100-slowfast-model/198618/1",56 "can_accept_answer": false,57 "can_unaccept_answer": false,58 "accepted_answer": false,59 "topic_accepted_answer": null,60 "can_vote": false61 }62 ],63 "stream": [64 43558065 ]66 },67 "timeline_lookup": [68 [69 1,70 59471 ]72 ],73 "suggested_topics": [74 {75 "fancy_title": "PyTorch Multiprocessing: Train only some parameters at some epochs",76 "id": 220389,77 "title": "PyTorch Multiprocessing: Train only some parameters at some epochs",78 "slug": "pytorch-multiprocessing-train-only-some-parameters-at-some-epochs",79 "posts_count": 3,80 "reply_count": 1,81 "highest_post_number": 3,82 "image_url": null,83 "created_at": "2025-05-27T22:47:03.854Z",84 "last_posted_at": "2025-05-29T23:26:11.262Z",85 "bumped": true,86 "bumped_at": "2025-05-30T05:52:20.585Z",87 "archetype": "regular",88 "unseen": false,89 "pinned": false,90 "unpinned": null,91 "visible": true,92 "closed": false,93 "archived": false,94 "bookmarked": null,95 "liked": null,96 "tags_descriptions": {},97 "like_count": 0,98 "views": 61,99 "category_id": 12,100 "featured_link": null,101 "has_accepted_answer": false,102 "posters": [103 {104 "extras": "latest",105 "description": "Original Poster, Most Recent Poster",106 "user": {107 "id": 27555,108 "username": "ameliatqy",109 "name": "Amelia",110 "avatar_template": "/letter_avatar_proxy/v4/letter/a/edb3f5/{size}.png",111 "trust_level": 1112 }113 },114 {115 "extras": null,116 "description": "Frequent Poster",117 "user": {118 "id": 39542,119 "username": "H-Huang",120 "name": "Howard Huang",121 "avatar_template": "/user_avatar/discuss.pytorch.org/h-huang/{size}/35598_2.png",122 "trust_level": 2123 }124 }125 ]126 },127 {128 "fancy_title": "Pytorch meets deadlock when loading nn.Module with lightning.Fabric",129 "id": 215124,130 "title": "Pytorch meets deadlock when loading nn.Module with lightning.Fabric",131 "slug": "pytorch-meets-deadlock-when-loading-nn-module-with-lightning-fabric",132 "posts_count": 1,133 "reply_count": 0,134 "highest_post_number": 1,135 "image_url": null,136 "created_at": "2025-01-08T14:38:06.916Z",137 "last_posted_at": "2025-01-08T14:38:06.967Z",138 "bumped": true,139 "bumped_at": "2025-01-08T14:38:06.967Z",140 "archetype": "regular",141 "unseen": false,142 "pinned": false,143 "unpinned": null,144 "visible": true,145 "closed": false,146 "archived": false,147 "bookmarked": null,148 "liked": null,149 "tags_descriptions": {},150 "like_count": 0,151 "views": 79,152 "category_id": 12,153 "featured_link": null,154 "has_accepted_answer": false,155 "posters": [156 {157 "extras": "latest single",158 "description": "Original Poster, Most Recent Poster",159 "user": {160 "id": 81406,161 "username": "forestbat",162 "name": "forestbat",163 "avatar_template": "/user_avatar/discuss.pytorch.org/forestbat/{size}/74440_2.png",164 "trust_level": 1165 }166 }167 ]168 },169 {170 "fancy_title": "FP8 training with torchao but without torchtitan",171 "id": 215152,172 "title": "FP8 training with torchao but without torchtitan",173 "slug": "fp8-training-with-torchao-but-without-torchtitan",174 "posts_count": 3,175 "reply_count": 1,176 "highest_post_number": 4,177 "image_url": null,178 "created_at": "2025-01-09T05:37:00.335Z",179 "last_posted_at": "2025-02-07T08:08:30.443Z",180 "bumped": true,181 "bumped_at": "2025-02-07T08:08:30.443Z",182 "archetype": "regular",183 "unseen": false,184 "pinned": false,185 "unpinned": null,186 "visible": true,187 "closed": false,188 "archived": false,189 "bookmarked": null,190 "liked": null,191 "tags_descriptions": {},192 "like_count": 0,193 "views": 386,194 "category_id": 12,195 "featured_link": null,196 "has_accepted_answer": false,197 "posters": [198 {199 "extras": "latest",200 "description": "Original Poster, Most Recent Poster",201 "user": {202 "id": 57464,203 "username": "cokespace2",204 "name": "Vince Mo",205 "avatar_template": "/user_avatar/discuss.pytorch.org/cokespace2/{size}/51244_2.png",206 "trust_level": 2207 }208 },209 {210 "extras": null,211 "description": "Frequent Poster",212 "user": {213 "id": 24320,214 "username": "supriyar",215 "name": "",216 "avatar_template": "/letter_avatar_proxy/v4/letter/s/db5fbb/{size}.png",217 "trust_level": 2218 }219 }220 ]221 },222 {223 "fancy_title": "Training hangs on loss.backward() with DDP –nnodes=2 –nproc_per_node=3",224 "id": 220127,225 "title": "Training hangs on loss.backward() with DDP --nnodes=2 --nproc_per_node=3",226 "slug": "training-hangs-on-loss-backward-with-ddp-nnodes-2-nproc-per-node-3",227 "posts_count": 4,228 "reply_count": 2,229 "highest_post_number": 4,230 "image_url": null,231 "created_at": "2025-05-18T00:41:12.214Z",232 "last_posted_at": "2025-05-22T03:50:31.587Z",233 "bumped": true,234 "bumped_at": "2025-05-22T03:50:31.587Z",235 "archetype": "regular",236 "unseen": false,237 "pinned": false,238 "unpinned": null,239 "visible": true,240 "closed": false,241 "archived": false,242 "bookmarked": null,243 "liked": null,244 "tags_descriptions": {},245 "like_count": 0,246 "views": 170,247 "category_id": 12,248 "featured_link": null,249 "has_accepted_answer": false,250 "posters": [251 {252 "extras": "latest",253 "description": "Original Poster, Most Recent Poster",254 "user": {255 "id": 78676,256 "username": "Chard_Lee",257 "name": "Chard Lee",258 "avatar_template": "/user_avatar/discuss.pytorch.org/chard_lee/{size}/72521_2.png",259 "trust_level": 1260 }261 },262 {263 "extras": null,264 "description": "Frequent Poster",265 "user": {266 "id": 3534,267 "username": "ptrblck",268 "name": "",269 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",270 "admin": true,271 "moderator": true,272 "trust_level": 2273 }274 }275 ]276 },277 {278 "fancy_title": "How to Efficiently Gather Python Objects Across GPUs Without GPU-to-CPU-to-GPU-to-CPU Overhead in torch.distributed?",279 "id": 221307,280 "title": "How to Efficiently Gather Python Objects Across GPUs Without GPU-to-CPU-to-GPU-to-CPU Overhead in torch.distributed?",281 "slug": "how-to-efficiently-gather-python-objects-across-gpus-without-gpu-to-cpu-to-gpu-to-cpu-overhead-in-torch-distributed",282 "posts_count": 2,283 "reply_count": 0,284 "highest_post_number": 2,285 "image_url": null,286 "created_at": "2025-07-06T12:09:09.292Z",287 "last_posted_at": "2025-07-11T00:02:20.832Z",288 "bumped": true,289 "bumped_at": "2025-07-11T00:02:20.832Z",290 "archetype": "regular",291 "unseen": false,292 "pinned": false,293 "unpinned": null,294 "visible": true,295 "closed": false,296 "archived": false,297 "bookmarked": null,298 "liked": null,299 "tags_descriptions": {},300 "like_count": 0,301 "views": 43,302 "category_id": 12,303 "featured_link": null,304 "has_accepted_answer": false,305 "posters": [306 {307 "extras": "latest single",308 "description": "Original Poster, Most Recent Poster",309 "user": {310 "id": 81154,311 "username": "5o1",312 "name": "",313 "avatar_template": "/user_avatar/discuss.pytorch.org/5o1/{size}/74221_2.png",314 "trust_level": 1315 }316 }317 ]318 }319 ],320 "tags_descriptions": {},321 "fancy_title": "How can I use 2 gpu vram 100%? (SlowFast model)",322 "id": 198618,323 "title": "How can I use 2 gpu vram 100%? (SlowFast model)",324 "posts_count": 1,325 "created_at": "2024-03-10T14:32:47.023Z",326 "views": 232,327 "reply_count": 0,328 "like_count": 0,329 "last_posted_at": "2024-03-10T14:32:47.112Z",330 "visible": true,331 "closed": false,332 "archived": false,333 "has_summary": false,334 "archetype": "regular",335 "slug": "how-can-i-use-2-gpu-vram-100-slowfast-model",336 "category_id": 12,337 "word_count": 703,338 "deleted_at": null,339 "user_id": 74012,340 "featured_link": null,341 "pinned_globally": false,342 "pinned_at": null,343 "pinned_until": null,344 "image_url": null,345 "slow_mode_seconds": 0,346 "draft": null,347 "draft_key": "topic_198618",348 "draft_sequence": null,349 "unpinned": null,350 "pinned": false,351 "current_post_number": 1,352 "highest_post_number": 1,353 "deleted_by": null,354 "actions_summary": [355 {356 "id": 4,357 "count": 0,358 "hidden": false,359 "can_act": false360 },361 {362 "id": 8,363 "count": 0,364 "hidden": false,365 "can_act": false366 },367 {368 "id": 10,369 "count": 0,370 "hidden": false,371 "can_act": false372 },373 {374 "id": 7,375 "count": 0,376 "hidden": false,377 "can_act": false378 }379 ],380 "chunk_size": 20,381 "bookmarked": false,382 "topic_timer": null,383 "message_bus_last_id": 0,384 "participant_count": 1,385 "show_read_indicator": false,386 "thumbnails": null,387 "slow_mode_enabled_until": null,388 "can_vote": false,389 "vote_count": 0,390 "user_voted": false,391 "discourse_zendesk_plugin_zendesk_id": null,392 "discourse_zendesk_plugin_zendesk_url": "https://your-url.zendesk.com/agent/tickets/",393 "details": {394 "can_edit": false,395 "notification_level": 1,396 "participants": [397 {398 "id": 74012,399 "username": "biology9763",400 "name": "끼링 후",401 "avatar_template": "/user_avatar/discuss.pytorch.org/biology9763/{size}/68362_2.png",402 "post_count": 1,403 "primary_group_name": null,404 "flair_name": null,405 "flair_url": null,406 "flair_color": null,407 "flair_bg_color": null,408 "flair_group_id": null,409 "trust_level": 1410 }411 ],412 "created_by": {413 "id": 74012,414 "username": "biology9763",415 "name": "끼링 후",416 "avatar_template": "/user_avatar/discuss.pytorch.org/biology9763/{size}/68362_2.png"417 },418 "last_poster": {419 "id": 74012,420 "username": "biology9763",421 "name": "끼링 후",422 "avatar_template": "/user_avatar/discuss.pytorch.org/biology9763/{size}/68362_2.png"423 }424 },425 "bookmarks": []426 },427 {428 "post_stream": {429 "posts": [430 {431 "id": 44301,432 "name": "",433 "username": "DeepLearner17",434 "avatar_template": "/user_avatar/discuss.pytorch.org/deeplearner17/{size}/3185_2.png",435 "created_at": "2018-04-30T19:12:24.449Z",436 "cooked": "<p>Hello,</p>\n<p>l have stored my best model where the network is as follow</p>\n<p>net</p>\n<pre><code class=\"lang-auto\">My_Net(\n (cl1): Linear(in_features=25, out_features=6, bias=True)\n (cl2): Linear(in_features=60, out_features=16, bias=True)\n (fc1): Linear(in_features=16, out_features=120, bias=True)\n (fc2): Linear(in_features=120, out_features=84, bias=True)\n (fc3): Linear(in_features=84, out_features=10, bias=True)\n)\n</code></pre>\n<p>To load my best model, l did the following :</p>\n<pre><code class=\"lang-auto\">my_best_model = torch.load('path to best model')\nmy_model.keys()\ndict_keys(['epoch', 'state_dict', 'best_prec1', 'optimizer'])\n</code></pre>\n<p><strong>What l would like to do ?</strong></p>\n<p>How can load and use my_best_model as :</p>\n<ol>\n<li>Feature extractor ?</li>\n<li>Evaluator on new examples ?</li>\n</ol>\n<p>Thank you</p>",437 "post_number": 1,438 "post_type": 1,439 "posts_count": 89,440 "updated_at": "2018-04-30T19:12:24.449Z",441 "reply_count": 0,442 "reply_to_post_number": null,443 "quote_count": 0,444 "incoming_link_count": 2887,445 "reads": 967,446 "readers_count": 966,447 "score": 14692.4,448 "yours": false,449 "topic_id": 17254,450 "topic_slug": "how-can-l-load-my-best-model-as-a-feature-extractor-evaluator",451 "display_username": "",452 "primary_group_name": null,453 "flair_name": null,454 "flair_url": null,455 "flair_bg_color": null,456 "flair_color": null,457 "flair_group_id": null,458 "badges_granted": [],459 "version": 1,460 "can_edit": false,461 "can_delete": false,462 "can_recover": false,463 "can_see_hidden_post": false,464 "can_wiki": false,465 "link_counts": [466 {467 "url": "https://discuss.pytorch.org/t/extract-layer-output-from-cnn-and-use-it-as-input-for-lstm/150668",468 "internal": true,469 "reflection": true,470 "title": "Extract layer output from CNN and use it as input for LSTM",471 "clicks": 9472 }473 ],474 "read": true,475 "user_title": null,476 "bookmarked": false,477 "actions_summary": [478 {479 "id": 2,480 "count": 2481 }482 ],483 "moderator": false,484 "admin": false,485 "staff": false,486 "user_id": 4110,487 "hidden": false,488 "trust_level": 1,489 "deleted_at": null,490 "user_deleted": false,491 "edit_reason": null,492 "can_view_edit_history": true,493 "wiki": false,494 "post_url": "/t/how-can-l-load-my-best-model-as-a-feature-extractor-evaluator/17254/1",495 "can_accept_answer": false,496 "can_unaccept_answer": false,497 "accepted_answer": false,498 "topic_accepted_answer": true,499 "can_vote": false500 },501 {502 "id": 44415,503 "name": "",504 "username": "ptrblck",505 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",506 "created_at": "2018-05-01T13:27:56.759Z",507 "cooked": "<p>You would have to load the <code>state_dict</code> and set it to <code>eval()</code>:</p>\n<pre><code class=\"lang-auto\">model = MyModel(...)\nmodel.load_state_dict(my_model['state_dict'])\nmodel.eval()\n</code></pre>\n<p>Now you can use it to evaluate new samples:</p>\n<pre><code class=\"lang-auto\">new_sample = ...\noutput = model(new_sample)\n</code></pre>\n<p>What do you mean by feature extractor?<br>\nWould you like to get a certain intermediate layer output from your model?</p>",508 "post_number": 2,509 "post_type": 1,510 "posts_count": 89,511 "updated_at": "2018-05-01T13:27:56.759Z",512 "reply_count": 3,513 "reply_to_post_number": null,514 "quote_count": 0,515 "incoming_link_count": 92,516 "reads": 992,517 "readers_count": 991,518 "score": 703.4,519 "yours": false,520 "topic_id": 17254,521 "topic_slug": "how-can-l-load-my-best-model-as-a-feature-extractor-evaluator",522 "display_username": "",523 "primary_group_name": null,524 "flair_name": null,525 "flair_url": null,526 "flair_bg_color": null,527 "flair_color": null,528 "flair_group_id": null,529 "badges_granted": [],530 "version": 1,531 "can_edit": false,532 "can_delete": false,533 "can_recover": false,534 "can_see_hidden_post": false,535 "can_wiki": false,536 "read": true,537 "user_title": "",538 "bookmarked": false,539 "actions_summary": [540 {541 "id": 2,542 "count": 2543 }544 ],545 "moderator": true,546 "admin": true,547 "staff": true,548 "user_id": 3534,549 "hidden": false,550 "trust_level": 2,551 "deleted_at": null,552 "user_deleted": false,553 "edit_reason": null,554 "can_view_edit_history": true,555 "wiki": false,556 "post_url": "/t/how-can-l-load-my-best-model-as-a-feature-extractor-evaluator/17254/2",557 "can_accept_answer": false,558 "can_unaccept_answer": false,559 "accepted_answer": false,560 "topic_accepted_answer": true561 },562 {563 "id": 44422,564 "name": "",565 "username": "DeepLearner17",566 "avatar_template": "/user_avatar/discuss.pytorch.org/deeplearner17/{size}/3185_2.png",567 "created_at": "2018-05-01T14:17:57.821Z",568 "cooked": "<p>Thank you for your reply. By feature extractor, l mean get ouptut from the last fully connected layer before softmax</p>",569 "post_number": 3,570 "post_type": 1,571 "posts_count": 89,572 "updated_at": "2018-05-01T14:17:57.821Z",573 "reply_count": 1,574 "reply_to_post_number": 2,575 "quote_count": 0,576 "incoming_link_count": 80,577 "reads": 1087,578 "readers_count": 1086,579 "score": 622.4,580 "yours": false,581 "topic_id": 17254,582 "topic_slug": "how-can-l-load-my-best-model-as-a-feature-extractor-evaluator",583 "display_username": "",584 "primary_group_name": null,585 "flair_name": null,586 "flair_url": null,587 "flair_bg_color": null,588 "flair_color": null,589 "flair_group_id": null,590 "badges_granted": [],591 "version": 1,592 "can_edit": false,593 "can_delete": false,594 "can_recover": false,595 "can_see_hidden_post": false,596 "can_wiki": false,597 "read": true,598 "user_title": null,599 "reply_to_user": {600 "id": 3534,601 "username": "ptrblck",602 "name": "",603 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png"604 },605 "bookmarked": false,606 "actions_summary": [],607 "moderator": false,608 "admin": false,609 "staff": false,610 "user_id": 4110,611 "hidden": false,612 "trust_level": 1,613 "deleted_at": null,614 "user_deleted": false,615 "edit_reason": null,616 "can_view_edit_history": true,617 "wiki": false,618 "post_url": "/t/how-can-l-load-my-best-model-as-a-feature-extractor-evaluator/17254/3",619 "can_accept_answer": false,620 "can_unaccept_answer": false,621 "accepted_answer": false,622 "topic_accepted_answer": true623 },624 {625 "id": 44424,626 "name": "",627 "username": "ptrblck",628 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",629 "created_at": "2018-05-01T14:21:47.183Z",630 "cooked": "<p>You could return this activation together with the output in your <code>forward</code> method a use a <code>forward_hook</code> on the layer.</p>",631 "post_number": 4,632 "post_type": 1,633 "posts_count": 89,634 "updated_at": "2018-05-01T14:21:47.183Z",635 "reply_count": 2,636 "reply_to_post_number": 3,637 "quote_count": 0,638 "incoming_link_count": 227,639 "reads": 1298,640 "readers_count": 1297,641 "score": 1404.6,642 "yours": false,643 "topic_id": 17254,644 "topic_slug": "how-can-l-load-my-best-model-as-a-feature-extractor-evaluator",645 "display_username": "",646 "primary_group_name": null,647 "flair_name": null,648 "flair_url": null,649 "flair_bg_color": null,650 "flair_color": null,651 "flair_group_id": null,652 "badges_granted": [],653 "version": 1,654 "can_edit": false,655 "can_delete": false,656 "can_recover": false,657 "can_see_hidden_post": false,658 "can_wiki": false,659 "read": true,660 "user_title": "",661 "reply_to_user": {662 "id": 4110,663 "username": "DeepLearner17",664 "name": "",665 "avatar_template": "/user_avatar/discuss.pytorch.org/deeplearner17/{size}/3185_2.png"666 },667 "bookmarked": false,668 "actions_summary": [],669 "moderator": true,670 "admin": true,671 "staff": true,672 "user_id": 3534,673 "hidden": false,674 "trust_level": 2,675 "deleted_at": null,676 "user_deleted": false,677 "edit_reason": null,678 "can_view_edit_history": true,679 "wiki": false,680 "post_url": "/t/how-can-l-load-my-best-model-as-a-feature-extractor-evaluator/17254/4",681 "can_accept_answer": false,682 "can_unaccept_answer": false,683 "accepted_answer": false,684 "topic_accepted_answer": true685 },686 {687 "id": 44425,688 "name": "",689 "username": "DeepLearner17",690 "avatar_template": "/user_avatar/discuss.pytorch.org/deeplearner17/{size}/3185_2.png",691 "created_at": "2018-05-01T14:25:13.035Z",692 "cooked": "<p>What if l would like to do that for an intermediate layer lets say the forth layer ?</p>",693 "post_number": 5,694 "post_type": 1,695 "posts_count": 89,696 "updated_at": "2018-05-01T14:25:13.035Z",697 "reply_count": 1,698 "reply_to_post_number": 4,699 "quote_count": 0,700 "incoming_link_count": 867,701 "reads": 1927,702 "readers_count": 1926,703 "score": 4725.4,704 "yours": false,705 "topic_id": 17254,706 "topic_slug": "how-can-l-load-my-best-model-as-a-feature-extractor-evaluator",707 "display_username": "",708 "primary_group_name": null,709 "flair_name": null,710 "flair_url": null,711 "flair_bg_color": null,712 "flair_color": null,713 "flair_group_id": null,714 "badges_granted": [],715 "version": 1,716 "can_edit": false,717 "can_delete": false,718 "can_recover": false,719 "can_see_hidden_post": false,720 "can_wiki": false,721 "link_counts": [722 {723 "url": "https://discuss.pytorch.org/t/how-to-remove-some-layers-in-resnet18/148014/2",724 "internal": true,725 "reflection": true,726 "title": "How to remove some layers in Resnet18?",727 "clicks": 29728 }729 ],730 "read": true,731 "user_title": null,732 "reply_to_user": {733 "id": 3534,734 "username": "ptrblck",735 "name": "",736 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png"737 },738 "bookmarked": false,739 "actions_summary": [],740 "moderator": false,741 "admin": false,742 "staff": false,743 "user_id": 4110,744 "hidden": false,745 "trust_level": 1,746 "deleted_at": null,747 "user_deleted": false,748 "edit_reason": null,749 "can_view_edit_history": true,750 "wiki": false,751 "post_url": "/t/how-can-l-load-my-best-model-as-a-feature-extractor-evaluator/17254/5",752 "can_accept_answer": false,753 "can_unaccept_answer": false,754 "accepted_answer": false,755 "topic_accepted_answer": true756 },757 {758 "id": 44442,759 "name": "",760 "username": "ptrblck",761 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",762 "created_at": "2018-05-01T17:01:26.853Z",763 "cooked": "<p>I’ve created a small code snippet using a forward hook to store one activation from <code>fc2</code>:</p>\n<pre><code class=\"lang-auto\">class MyModel(nn.Module):\n def __init__(self):\n super(MyModel, self).__init__()\n self.cl1 = nn.Linear(25, 60)\n self.cl2 = nn.Linear(60, 16)\n self.fc1 = nn.Linear(16, 120)\n self.fc2 = nn.Linear(120, 84)\n self.fc3 = nn.Linear(84, 10)\n \n def forward(self, x):\n x = F.relu(self.cl1(x))\n x = F.relu(self.cl2(x))\n x = F.relu(self.fc1(x))\n x = F.relu(self.fc2(x))\n x = F.log_softmax(self.fc3(x), dim=1)\n return x\n\n\nactivation = {}\ndef get_activation(name):\n def hook(model, input, output):\n activation[name] = output.detach()\n return hook\n\n\nmodel = MyModel()\nmodel.fc2.register_forward_hook(get_activation('fc2'))\nx = torch.randn(1, 25)\noutput = model(x)\nprint(activation['fc2'])\n</code></pre>",764 "post_number": 6,765 "post_type": 1,766 "posts_count": 89,767 "updated_at": "2018-05-01T17:01:26.853Z",768 "reply_count": 17,769 "reply_to_post_number": 5,770 "quote_count": 0,771 "incoming_link_count": 5113,772 "reads": 1921,773 "readers_count": 1920,774 "score": 27456.2,775 "yours": false,776 "topic_id": 17254,777 "topic_slug": "how-can-l-load-my-best-model-as-a-feature-extractor-evaluator",778 "display_username": "",779 "primary_group_name": null,780 "flair_name": null,781 "flair_url": null,782 "flair_bg_color": null,783 "flair_color": null,784 "flair_group_id": null,785 "badges_granted": [],786 "version": 1,787 "can_edit": false,788 "can_delete": false,789 "can_recover": false,790 "can_see_hidden_post": false,791 "can_wiki": false,792 "link_counts": [793 {794 "url": "https://discuss.pytorch.org/t/get-intermediate-value-from-a-block-in-nn-sequential-in-a-model/32154/5",795 "internal": true,796 "reflection": true,797 "title": "Get intermediate value from a block in nn.Sequential in a model",798 "clicks": 577799 },800 {801 "url": "https://discuss.pytorch.org/t/undestanding-register-hook-functions/30376/2",802 "internal": true,803 "reflection": true,804 "title": "Undestanding register_*_hook() functions",805 "clicks": 289806 },807 {808 "url": "https://discuss.pytorch.org/t/getting-intermediate-output-of-self-created-sequential/21662/2",809 "internal": true,810 "reflection": true,811 "title": "Getting Intermediate Output of Self Created Sequential",812 "clicks": 122813 },814 {815 "url": "https://discuss.pytorch.org/t/extract-activation-maps/27229/2",816 "internal": true,817 "reflection": true,818 "title": "Extract activation maps",819 "clicks": 115820 },821 {822 "url": "https://discuss.pytorch.org/t/pre-trained-vgg-19-maxpool-layer-output/49720/6",823 "internal": true,824 "reflection": true,825 "title": "Pre trained VGG 19 maxpool layer output",826 "clicks": 67827 },828 {829 "url": "https://discuss.pytorch.org/t/extracting-features-from-specific-layers-on-a-trained-network/41588/2",830 "internal": true,831 "reflection": true,832 "title": "Extracting features from specific layers on a trained network",833 "clicks": 5834 },835 {836 "url": "https://discuss.pytorch.org/t/get-layers-output-from-nn-sequential/62804/2",837 "internal": true,838 "reflection": true,839 "title": "Get layer's output from nn.Sequential",840 "clicks": 3841 },842 {843 "url": "https://discuss.pytorch.org/t/using-feature-extraction-layers-from-pre-trained-frcnn/67621/4",844 "internal": true,845 "reflection": true,846 "title": "Using feature extraction layers from pre-trained FRCNN",847 "clicks": 3848 },849 {850 "url": "https://discuss.pytorch.org/t/how-to-get-output-of-layers/75457/2",851 "internal": true,852 "reflection": true,853 "title": "How to get output of layers?",854 "clicks": 2855 },856 {857 "url": "https://discuss.pytorch.org/t/how-to-check-or-view-the-intermediate-results-or-output-of-a-network/20906/2",858 "internal": true,859 "reflection": true,860 "title": "How to check or view the intermediate results or output of a network?",861 "clicks": 2862 },863 {864 "url": "https://discuss.pytorch.org/t/why-is-the-input-of-hook-function-a-tuple/54229",865 "internal": true,866 "reflection": true,867 "title": "Why is the input of hook function a tuple?",868 "clicks": 2869 },870 {871 "url": "https://discuss.pytorch.org/t/how-to-use-parameters-from-autoencoder-to-cnn-for-classification/46725/10",872 "internal": true,873 "reflection": true,874 "title": "How to use parameters from autoencoder to CNN for classification",875 "clicks": 2876 },877 {878 "url": "https://discuss.pytorch.org/t/resnet18-access-to-the-output-of-each-basicblock/46249/2",879 "internal": true,880 "reflection": true,881 "title": "ResNet18 - access to the output of each BasicBlock",882 "clicks": 2883 },884 {885 "url": "https://discuss.pytorch.org/t/how-to-add-activation-histogram-in-tensorboard/103465/2",886 "internal": true,887 "reflection": true,888 "title": "How to add activation histogram in tensorboard?",889 "clicks": 1890 },891 {892 "url": "https://discuss.pytorch.org/t/cyclegan-inference-pictures-differ-from-images-generated-in-training/68952/8",893 "internal": true,894 "reflection": true,895 "title": "cycleGAN inference pictures differ from images generated in training",896 "clicks": 1897 },898 {899 "url": "https://discuss.pytorch.org/t/retrieve-intermediate-node-values-without-explicitly-return-them/21640/2",900 "internal": true,901 "reflection": true,902 "title": "Retrieve intermediate node values without explicitly return them",903 "clicks": 1904 },905 {906 "url": "https://discuss.pytorch.org/t/how-to-access-intermediate-layer-in-a-pretrained-model-and-sequence-layers/99524/2",907 "internal": true,908 "reflection": true,909 "title": "How to access intermediate layer in a pretrained model and sequence layers?",910 "clicks": 1911 },912 {913 "url": "https://discuss.pytorch.org/t/extract-specific-outputs-through-pre-trained-classifier/87485/2",914 "internal": true,915 "reflection": true,916 "title": "Extract specific outputs through pre trained classifier",917 "clicks": 1918 },919 {920 "url": "https://discuss.pytorch.org/t/how-to-get-image-featrue-using-my-pre-trained-model/30762/2",921 "internal": true,922 "reflection": true,923 "title": "How to get image featrue using my pre-trained model?",924 "clicks": 1925 },926 {927 "url": "https://discuss.pytorch.org/t/how-to-get-the-output-at-the-2nd-block-of-a-resnet-50/80598/2",928 "internal": true,929 "reflection": true,930 "title": "How to get the output at the 2nd block of a resnet-50?",931 "clicks": 1932 },933 {934 "url": "https://discuss.pytorch.org/t/extraction-and-modification-of-intermediate-layer-outputs-from-pre-trained-models-and-layer-by-layer-inference/199050/4",935 "internal": true,936 "reflection": true,937 "title": "Extraction and modification of intermediate layer outputs from pre-trained models and layer by layer inference",938 "clicks": 0939 },940 {941 "url": "https://discuss.pytorch.org/t/lasagne-layers-get-output-equivalent-method-in-pytorch/85737/2",942 "internal": true,943 "reflection": true,944 "title": "Lasagne.layers.get_output equivalent method in pytorch",945 "clicks": 0946 },947 {948 "url": "https://discuss.pytorch.org/t/how-to-select-pixels-of-roi-from-feature-map/33699/14",949 "internal": true,950 "reflection": true,951 "title": "How to select pixels of ROI from feature map",952 "clicks": 0953 },954 {955 "url": "https://discuss.pytorch.org/t/how-to-get-feature-map-from-pre-trained-resnet50/86150/2",956 "internal": true,957 "reflection": true,958 "title": "How to get feature map from pre-trained Resnet50",959 "clicks": 0960 },961 {962 "url": "https://discuss.pytorch.org/t/intermediate-tensors-in-nemo/86714/2",963 "internal": true,964 "reflection": true,965 "title": "Intermediate tensors in NeMo",966 "clicks": 0967 },968 {969 "url": "https://discuss.pytorch.org/t/network-produces-different-output-for-the-same-image-from-different-dataloaders/88304/4",970 "internal": true,971 "reflection": true,972 "title": "Network produces different output for the same image from different Dataloaders",973 "clicks": 0974 },975 {976 "url": "https://discuss.pytorch.org/t/testing-models-performance-after-saving-and-without-saving-the-model/88743/4",977 "internal": true,978 "reflection": true,979 "title": "Testing model's performance after saving and without saving the model",980 "clicks": 0981 },982 {983 "url": "https://discuss.pytorch.org/t/to-get-a-middle-activation-map/90416/2",984 "internal": true,985 "reflection": true,986 "title": "To get a middle activation map",987 "clicks": 0988 },989 {990 "url": "https://discuss.pytorch.org/t/activation-values-in-resnet/128029/2",991 "internal": true,992 "reflection": true,993 "title": "Activation values in ResNet",994 "clicks": 0995 },996 {997 "url": "https://discuss.pytorch.org/t/heatmap-localization/121965/8",998 "internal": true,999 "reflection": true,1000 "title": "Heatmap localization",1001 "clicks": 01002 },1003 {1004 "url": "https://discuss.pytorch.org/t/make-values-consistent-across-versions/92175/7",1005 "internal": true,1006 "reflection": true,1007 "title": "Make values consistent across versions",1008 "clicks": 01009 },1010 {1011 "url": "https://discuss.pytorch.org/t/how-to-index-into-the-layers-within-the-efficentnet-backbone-block/181754/2",1012 "internal": true,1013 "reflection": true,1014 "title": "How to index into the layers within the EfficentNet backbone block?",1015 "clicks": 01016 },1017 {1018 "url": "https://discuss.pytorch.org/t/logging-and-changing-activation-values-outside-the-forward-method/97450/2",1019 "internal": true,1020 "reflection": true,1021 "title": "Logging and changing activation values outside the forward method",1022 "clicks": 01023 },1024 {1025 "url": "https://discuss.pytorch.org/t/loss-nan-when-resuming-from-a-pretrained-model/92234/8",1026 "internal": true,1027 "reflection": true,1028 "title": "Loss nan when resuming from a pretrained model",1029 "clicks": 01030 },1031 {1032 "url": "https://discuss.pytorch.org/t/fp16-gives-nan-loss-when-using-pre-trained-model/94133/10",1033 "internal": true,1034 "reflection": true,1035 "title": "FP16 gives NaN loss when using pre-trained model",1036 "clicks": 01037 },1038 {1039 "url": "https://discuss.pytorch.org/t/extract-feature-vector-latent-factors-from-embedding-layer-in-pytorch/94712/2",1040 "internal": true,1041 "reflection": true,1042 "title": "Extract feature vector/latent factors from Embedding layer in Pytorch",1043 "clicks": 01044 },1045 {1046 "url": "https://discuss.pytorch.org/t/get-2048-length-feature-vectors-of-inception-model-torchvision-0-7-0/94744/2",1047 "internal": true,1048 "reflection": true,1049 "title": "Get 2048 Length Feature Vectors of Inception Model TorchVision 0.7.0",1050 "clicks": 01051 },1052 {1053 "url": "https://discuss.pytorch.org/t/how-is-feature-extraction-done-from-by-using-pre-activations-last-cnn-layer-in-vgg-19/95025/2",1054 "internal": true,1055 "reflection": true,1056 "title": "How is feature extraction done from by using pre-activations last CNN layer in VGG-19",1057 "clicks": 01058 },1059 {1060 "url": "https://discuss.pytorch.org/t/how-do-i-print-output-of-each-layer-in-sequential/5773/11",1061 "internal": true,1062 "reflection": true,1063 "title": "How do I print output of each layer in sequential?",1064 "clicks": 01065 },1066 {1067 "url": "https://discuss.pytorch.org/t/converting-keras-model-to-pytorch/118585/4",1068 "internal": true,1069 "reflection": true,1070 "title": "Converting keras model to pytorch",1071 "clicks": 01072 },1073 {1074 "url": "https://discuss.pytorch.org/t/visualize-feature-map/29597/77",1075 "internal": true,1076 "reflection": true,1077 "title": "Visualize feature map",1078 "clicks": 01079 },1080 {1081 "url": "https://discuss.pytorch.org/t/how-to-save-inner-nn-sequential-layers-output/98579/2",1082 "internal": true,1083 "reflection": true,1084 "title": "How to save inner nn.Sequential layer's output",1085 "clicks": 01086 },1087 {1088 "url": "https://discuss.pytorch.org/t/online-knowledge-distillation/98769/2",1089 "internal": true,1090 "reflection": true,1091 "title": "Online Knowledge Distillation",1092 "clicks": 01093 },1094 {1095 "url": "https://discuss.pytorch.org/t/extracting-features-from-boxes-in-the-pretrained-faster-rcnn/100361/2",1096 "internal": true,1097 "reflection": true,1098 "title": "Extracting features from boxes in the pretrained Faster Rcnn",1099 "clicks": 01100 },1101 {1102 "url": "https://discuss.pytorch.org/t/about-normalization-using-pre-trained-vgg16-networks/23560/35",1103 "internal": true,1104 "reflection": true,1105 "title": "About Normalization using pre-trained vgg16 networks",1106 "clicks": 01107 },1108 {1109 "url": "https://discuss.pytorch.org/t/nan-loss-coming-after-some-time/11568/24",1110 "internal": true,1111 "reflection": true,1112 "title": "Nan Loss coming after some time",1113 "clicks": 01114 },1115 {1116 "url": "https://discuss.pytorch.org/t/basic-mnist-keras-model-to-pytorch-implementation/102664/2",1117 "internal": true,1118 "reflection": true,1119 "title": "Basic MNIST Keras model to Pytorch implementation",1120 "clicks": 01121 },1122 {1123 "url": "https://discuss.pytorch.org/t/get-output-of-intermediate-layer-of-pre-trained-model/102758/2",1124 "internal": true,1125 "reflection": true,1126 "title": "Get output of intermediate layer of pre-trained model",1127 "clicks": 01128 },1129 {1130 "url": "https://discuss.pytorch.org/t/extracting-features-using-my-qnn-hybrid-model/198785/2",1131 "internal": true,1132 "reflection": true,1133 "title": "Extracting features using my QNN Hybrid model",1134 "clicks": 01135 },1136 {1137 "url": "https://discuss.pytorch.org/t/how-can-i-get-the-hidden-embeddings-from-the-2nd-last-fully-connected-layer-for-t-sne-visualization/112053/2",1138 "internal": true,1139 "reflection": true,1140 "title": "How can I get the Hidden Embeddings from the 2nd last fully connected layer for t-SNE visualization?",1141 "clicks": 01142 },1143 {1144 "url": "https://discuss.pytorch.org/t/my-transformer-nmt-model-is-giving-nan-loss-value/122697/2",1145 "internal": true,1146 "reflection": true,1147 "title": "My transformer NMT model is giving \"nan\" loss value",1148 "clicks": 01149 },1150 {1151 "url": "https://discuss.pytorch.org/t/how-can-i-gets-output-of-one-layer/124553/9",1152 "internal": true,1153 "reflection": true,1154 "title": "How can I gets output of one layer?",1155 "clicks": 01156 },1157 {1158 "url": "https://discuss.pytorch.org/t/torchvision-mask-rcnn-with-resnext101-backbone-occur-nan-loss-during-the-training/112268/2",1159 "internal": true,1160 "reflection": true,1161 "title": "Torchvision Mask-rcnn with Resnext101 backbone occur Nan loss during the training",1162 "clicks": 01163 },1164 {1165 "url": "https://discuss.pytorch.org/t/how-to-modify-and-rewrite-the-activation-output-of-a-layer-before-applying-the-output-to-next-layer/104271/2",1166 "internal": true,1167 "reflection": true,1168 "title": "How to modify and rewrite the activation output of a layer before applying the output to next layer?",1169 "clicks": 01170 },1171 {1172 "url": "https://discuss.pytorch.org/t/return-item-in-forward-hook/124346/2",1173 "internal": true,1174 "reflection": true,1175 "title": "Return item in forward hook",1176 "clicks": 01177 },1178 {1179 "url": "https://discuss.pytorch.org/t/how-to-get-the-intermediate-features-from-alexnet/132903/2",1180 "internal": true,1181 "reflection": true,1182 "title": "How to get the intermediate features from Alexnet",1183 "clicks": 01184 },1185 {1186 "url": "https://discuss.pytorch.org/t/visualise-sequential-model-feature-maps/113908/2",1187 "internal": true,1188 "reflection": true,1189 "title": "Visualise sequential model feature maps",1190 "clicks": 01191 },1192 {1193 "url": "https://discuss.pytorch.org/t/nan-loss-with-torch-cuda-amp-and-crossentropyloss/108554/6",1194 "internal": true,1195 "reflection": true,1196 "title": "Nan Loss with torch.cuda.amp and CrossEntropyLoss",1197 "clicks": 01198 },1199 {1200 "url": "https://discuss.pytorch.org/t/how-can-i-get-the-mean-and-variance-from-a-saved-cnn-model/170734/5",