Anurag1734/cuda-error-resolution-analysis
07
1[2 {3 "post_stream": {4 "posts": [5 {6 "id": 335902,7 "name": "Davide Paglieri",8 "username": "Davide_Paglieri",9 "avatar_template": "/user_avatar/discuss.pytorch.org/davide_paglieri/{size}/47532_2.png",10 "created_at": "2022-03-14T14:34:05.516Z",11 "cooked": "<p>Hello, I am working with a model whose VRAM requirements with 1080p frames make it go out of memory due to caching after the first iteration. When it does so I get the following error about memory fragmentation:</p>\n<pre><code class=\"lang-auto\">RuntimeError: CUDA out of memory. \nTried to allocate 776.00 MiB (GPU 0; 14.76 GiB total capacity; \n11.41 GiB already allocated; 557.75 MiB free; 13.12 GiB reserved in total by PyTorch) \nIf reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. \nSee documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF\n</code></pre>\n<p>To circumvent this problem, I found out that I can simply use <code>torch.cuda.empty_cache()</code> at the end of the every iteration, like this:</p>\n<pre><code class=\"lang-auto\">with torch.no_grad():\n for i in range(100):\n start_time = time.time()\n output = pytorch_model(image_1, image_2)\n print(f'It took {time.time()-start_time}s')\n torch.cuda.empty_cache()\n</code></pre>\n<p>The strange thing that I found out, is that this not only allows me to run 1080p frames without going OOM, but by using this I get a speed up with lower resolution frames compared to if I did not empty the cache. I have read that this is not supposed to happen, and actually emptying the cache should slow down the process.</p>\n<p>My guess is that what is slowing everything down here even more is the memory fragmentation, and emptying the cache, although usually decreases speed, here actually makes it faster because it solves the fragmentation? I am not entirely sure what is going on… any ideas? I am trying to speed up the model as much as possible.</p>",12 "post_number": 1,13 "post_type": 1,14 "posts_count": 6,15 "updated_at": "2022-03-14T14:37:11.609Z",16 "reply_count": 1,17 "reply_to_post_number": null,18 "quote_count": 0,19 "incoming_link_count": 986,20 "reads": 24,21 "readers_count": 23,22 "score": 4904.8,23 "yours": false,24 "topic_id": 146385,25 "topic_slug": "torch-cuda-empty-cache-memory-segmentation-and-runtime",26 "display_username": "Davide Paglieri",27 "primary_group_name": null,28 "flair_name": null,29 "flair_url": null,30 "flair_bg_color": null,31 "flair_color": null,32 "flair_group_id": null,33 "badges_granted": [],34 "version": 1,35 "can_edit": false,36 "can_delete": false,37 "can_recover": false,38 "can_see_hidden_post": false,39 "can_wiki": false,40 "read": true,41 "user_title": null,42 "bookmarked": false,43 "actions_summary": [],44 "moderator": false,45 "admin": false,46 "staff": false,47 "user_id": 54035,48 "hidden": false,49 "trust_level": 1,50 "deleted_at": null,51 "user_deleted": false,52 "edit_reason": null,53 "can_view_edit_history": true,54 "wiki": false,55 "post_url": "/t/torch-cuda-empty-cache-memory-segmentation-and-runtime/146385/1",56 "can_accept_answer": false,57 "can_unaccept_answer": false,58 "accepted_answer": false,59 "topic_accepted_answer": null,60 "can_vote": false61 },62 {63 "id": 335946,64 "name": "",65 "username": "ptrblck",66 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",67 "created_at": "2022-03-14T18:58:02.630Z",68 "cooked": "<aside class=\"quote no-group\" data-username=\"Davide_Paglieri\" data-post=\"1\" data-topic=\"146385\">\n<div class=\"title\">\n<div class=\"quote-controls\"></div>\n<img loading=\"lazy\" alt=\"\" width=\"24\" height=\"24\" src=\"https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/davide_paglieri/48/47532_2.png\" class=\"avatar\"> Davide_Paglieri:</div>\n<blockquote>\n<p>The strange thing that I found out, is that this not only allows me to run 1080p frames without going OOM, but by using this I get a speed up with lower resolution frames compared to if I did not empty the cache. I have read that this is not supposed to happen, and actually emptying the cache should slow down the process.</p>\n</blockquote>\n</aside>\n<p>This is correct, since PyTorch calls <code>empty_cache()</code> internally once it hits an OOM and tries to reallocate the memory. If this fails, the error is raised so your code shouldn’t make a difference.</p>",69 "post_number": 2,70 "post_type": 1,71 "posts_count": 6,72 "updated_at": "2022-03-14T18:58:02.630Z",73 "reply_count": 1,74 "reply_to_post_number": null,75 "quote_count": 1,76 "incoming_link_count": 3,77 "reads": 21,78 "readers_count": 20,79 "score": 24.2,80 "yours": false,81 "topic_id": 146385,82 "topic_slug": "torch-cuda-empty-cache-memory-segmentation-and-runtime",83 "display_username": "",84 "primary_group_name": null,85 "flair_name": null,86 "flair_url": null,87 "flair_bg_color": null,88 "flair_color": null,89 "flair_group_id": null,90 "badges_granted": [],91 "version": 1,92 "can_edit": false,93 "can_delete": false,94 "can_recover": false,95 "can_see_hidden_post": false,96 "can_wiki": false,97 "read": true,98 "user_title": "",99 "bookmarked": false,100 "actions_summary": [],101 "moderator": true,102 "admin": true,103 "staff": true,104 "user_id": 3534,105 "hidden": false,106 "trust_level": 2,107 "deleted_at": null,108 "user_deleted": false,109 "edit_reason": null,110 "can_view_edit_history": true,111 "wiki": false,112 "post_url": "/t/torch-cuda-empty-cache-memory-segmentation-and-runtime/146385/2",113 "can_accept_answer": false,114 "can_unaccept_answer": false,115 "accepted_answer": false,116 "topic_accepted_answer": null117 },118 {119 "id": 335957,120 "name": "Davide Paglieri",121 "username": "Davide_Paglieri",122 "avatar_template": "/user_avatar/discuss.pytorch.org/davide_paglieri/{size}/47532_2.png",123 "created_at": "2022-03-14T20:19:56.618Z",124 "cooked": "<p>Thank you for your answer! It shouldn’t make a difference in the sense that manually calling empty_cache shouldn’t have any effects in this case?</p>\n<p>What I don’t undestand is why with lower resolution frames, where memory is not a problem, I get a speed up when calling empty_cache(). Why is that happening?</p>",125 "post_number": 3,126 "post_type": 1,127 "posts_count": 6,128 "updated_at": "2022-03-15T09:54:19.752Z",129 "reply_count": 1,130 "reply_to_post_number": 2,131 "quote_count": 0,132 "incoming_link_count": 1,133 "reads": 21,134 "readers_count": 20,135 "score": 14.2,136 "yours": false,137 "topic_id": 146385,138 "topic_slug": "torch-cuda-empty-cache-memory-segmentation-and-runtime",139 "display_username": "Davide Paglieri",140 "primary_group_name": null,141 "flair_name": null,142 "flair_url": null,143 "flair_bg_color": null,144 "flair_color": null,145 "flair_group_id": null,146 "badges_granted": [],147 "version": 3,148 "can_edit": false,149 "can_delete": false,150 "can_recover": false,151 "can_see_hidden_post": false,152 "can_wiki": false,153 "read": true,154 "user_title": null,155 "reply_to_user": {156 "id": 3534,157 "username": "ptrblck",158 "name": "",159 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png"160 },161 "bookmarked": false,162 "actions_summary": [],163 "moderator": false,164 "admin": false,165 "staff": false,166 "user_id": 54035,167 "hidden": false,168 "trust_level": 1,169 "deleted_at": null,170 "user_deleted": false,171 "edit_reason": null,172 "can_view_edit_history": true,173 "wiki": false,174 "post_url": "/t/torch-cuda-empty-cache-memory-segmentation-and-runtime/146385/3",175 "can_accept_answer": false,176 "can_unaccept_answer": false,177 "accepted_answer": false,178 "topic_accepted_answer": null179 },180 {181 "id": 335968,182 "name": "",183 "username": "ptrblck",184 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",185 "created_at": "2022-03-14T22:11:15.537Z",186 "cooked": "<p>I also don’t know why this should be the case, since you would be forcing PyTorch to call into synchronizing <code>cudaMalloc</code> calls.<br>\nCould you profile your workload and check the timeline?</p>",187 "post_number": 4,188 "post_type": 1,189 "posts_count": 6,190 "updated_at": "2022-03-14T22:11:15.537Z",191 "reply_count": 0,192 "reply_to_post_number": 3,193 "quote_count": 0,194 "incoming_link_count": 5,195 "reads": 20,196 "readers_count": 19,197 "score": 29.0,198 "yours": false,199 "topic_id": 146385,200 "topic_slug": "torch-cuda-empty-cache-memory-segmentation-and-runtime",201 "display_username": "",202 "primary_group_name": null,203 "flair_name": null,204 "flair_url": null,205 "flair_bg_color": null,206 "flair_color": null,207 "flair_group_id": null,208 "badges_granted": [],209 "version": 1,210 "can_edit": false,211 "can_delete": false,212 "can_recover": false,213 "can_see_hidden_post": false,214 "can_wiki": false,215 "read": true,216 "user_title": "",217 "reply_to_user": {218 "id": 54035,219 "username": "Davide_Paglieri",220 "name": "Davide Paglieri",221 "avatar_template": "/user_avatar/discuss.pytorch.org/davide_paglieri/{size}/47532_2.png"222 },223 "bookmarked": false,224 "actions_summary": [],225 "moderator": true,226 "admin": true,227 "staff": true,228 "user_id": 3534,229 "hidden": false,230 "trust_level": 2,231 "deleted_at": null,232 "user_deleted": false,233 "edit_reason": null,234 "can_view_edit_history": true,235 "wiki": false,236 "post_url": "/t/torch-cuda-empty-cache-memory-segmentation-and-runtime/146385/4",237 "can_accept_answer": false,238 "can_unaccept_answer": false,239 "accepted_answer": false,240 "topic_accepted_answer": null241 },242 {243 "id": 336086,244 "name": "Davide Paglieri",245 "username": "Davide_Paglieri",246 "avatar_template": "/user_avatar/discuss.pytorch.org/davide_paglieri/{size}/47532_2.png",247 "created_at": "2022-03-15T09:54:41.456Z",248 "cooked": "<p>This is the profiling when <strong>emptying the cache</strong></p>\n<pre><code class=\"lang-auto\">----------------------------- ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ \n Name Self CPU % Self CPU CPU total % CPU total CPU time avg CPU Mem Self CPU Mem CUDA Mem Self CUDA Mem # of Calls \n----------------------------- ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ \n aten::zeros 0.01% 59.000us 0.02% 144.000us 24.000us 24 b 0 b 0 b 0 b 6 \n aten::empty 0.26% 2.445ms 0.83% 7.814ms 26.578us 120 b 120 b 4.26 Gb 4.26 Gb 294 \n aten::zero_ 0.00% 5.000us 0.00% 5.000us 0.833us 0 b 0 b 0 b 0 b 6 \n Extraction layer 0.75% 7.012ms 3.48% 32.720ms 32.720ms -4 b -20 b 1.11 Gb -4.40 Gb 1 \n aten::avg_pool2d 0.22% 2.113ms 0.77% 7.207ms 171.595us 0 b 0 b 270.55 Mb 270.55 Mb 42 \n cudaLaunchKernel 0.78% 7.334ms 0.78% 7.334ms 7.656us 0 b 0 b 0 b 0 b 958 \n cudaMalloc 2.66% 24.999ms 2.66% 24.999ms 694.417us 0 b 0 b 0 b 0 b 36 \n aten::conv2d 0.05% 494.000us 2.40% 22.591ms 132.111us 0 b 0 b 4.95 Gb 0 b 171 \n aten::_convolution_mode 0.06% 589.000us 2.17% 20.403ms 130.788us 0 b 0 b 4.92 Gb 0 b 156 \n aten::_convolution 0.24% 2.231ms 2.28% 21.462ms 125.509us 0 b 0 b 4.95 Gb 0 b 171 \n aten::cudnn_convolution 0.96% 9.021ms 1.56% 14.724ms 86.105us 0 b 0 b 4.95 Gb 3.79 Gb 171 \n aten::resize_ 0.15% 1.428ms 1.37% 12.875ms 30.152us 0 b 0 b 6.85 Gb 6.85 Gb 427 \n cudaEventRecord 0.04% 360.000us 0.04% 360.000us 2.105us 0 b 0 b 0 b 0 b 171 \n aten::reshape 0.05% 513.000us 0.13% 1.252ms 7.322us 0 b 0 b 0 b 0 b 171 \n aten::_reshape_alias 0.08% 739.000us 0.08% 739.000us 4.322us 0 b 0 b 0 b 0 b 171 \n aten::add_ 0.23% 2.179ms 0.35% 3.255ms 19.035us 0 b 0 b 0 b 0 b 171 \n aten::leaky_relu 0.32% 3.034ms 0.68% 6.435ms 42.336us 0 b 0 b 4.50 Gb 4.50 Gb 152 \n aten::cat 0.04% 411.000us 1.66% 15.647ms 184.082us 0 b 0 b 6.85 Gb 0 b 85 \n aten::_cat 0.15% 1.373ms 1.62% 15.236ms 179.247us 0 b 0 b 6.85 Gb 0 b 85 \n Estimation layer 0.84% 7.926ms 45.34% 426.754ms 426.754ms -4 b -20 b 20.16 Mb -6.37 Gb 1 \n cudaBindTexture 0.01% 100.000us 0.01% 100.000us 6.250us 0 b 0 b 0 b 0 b 16 \n cudaUnbindTexture 0.00% 15.000us 0.00% 15.000us 0.938us 0 b 0 b 0 b 0 b 16 \n aten::convolution 0.00% 46.000us 0.18% 1.694ms 112.933us 0 b 0 b 30.71 Mb 0 b 15 \n aten::mul 0.08% 720.000us 0.10% 950.000us 27.941us 0 b 0 b 28.38 Mb 28.38 Mb 34 \n aten::upsample_bilinear2d 0.05% 517.000us 0.07% 670.000us 27.917us 0 b 0 b 37.61 Mb 37.61 Mb 24 \n aten::slice 0.06% 536.000us 0.07% 690.000us 3.750us 0 b 0 b 0 b 0 b 184 \n aten::as_strided 0.02% 233.000us 0.02% 233.000us 1.031us 0 b 0 b 0 b 0 b 226 \n aten::div 0.27% 2.576ms 0.34% 3.169ms 72.023us 0 b 0 b 38.36 Mb 38.36 Mb 44 \n aten::to 0.01% 106.000us 49.45% 465.446ms 21.157ms 0 b 0 b 38.89 Mb 0 b 22 \n aten::_to_copy 0.03% 251.000us 49.44% 465.340ms 21.152ms 0 b 0 b 38.89 Mb 0 b 22 \n aten::empty_strided 0.03% 325.000us 0.03% 325.000us 14.773us 0 b 0 b 38.89 Mb 38.89 Mb 22 \n aten::copy_ 0.05% 459.000us 49.39% 464.841ms 17.878ms 0 b 0 b 0 b 0 b 26 \n cudaMemcpyAsync 12.29% 115.691ms 12.29% 115.691ms 5.259ms 0 b 0 b 0 b 0 b 22 \n cudaStreamSynchronize 37.05% 348.666ms 37.05% 348.666ms 15.848ms 0 b 0 b 0 b 0 b 22 \n aten::add 0.13% 1.195ms 0.18% 1.660ms 36.087us 0 b 0 b 75.50 Mb 75.50 Mb 46 \n aten::permute 0.02% 220.000us 0.03% 299.000us 7.119us 0 b 0 b 0 b 0 b 42 \n aten::grid_sampler 0.02% 182.000us 0.47% 4.433ms 201.500us 0 b 0 b 2.22 Gb 0 b 22 \n aten::grid_sampler_2d 0.03% 320.000us 0.45% 4.251ms 193.227us 0 b 0 b 2.22 Gb 0 b 22 \n Concatenate 0.10% 955.000us 0.51% 4.808ms 2.404ms -8 b -40 b 1.17 Gb -23.56 Mb 2 \n Warp layer 0.21% 1.975ms 8.25% 77.666ms 77.666ms -4 b -20 b 2.24 Gb -74.93 Mb 1 \n Fusion layer 0.13% 1.186ms 0.93% 8.724ms 8.724ms -4 b -20 b 10.55 Mb -5.34 Gb 1 \n aten::upsample_nearest2d 0.02% 142.000us 0.06% 537.000us 134.250us 0 b 0 b 893.52 Mb 893.52 Mb 4 \n aten::constant_pad_nd 0.01% 65.000us 0.20% 1.876ms 469.000us 0 b 0 b 898.46 Mb 0 b 4 \n aten::fill_ 0.01% 53.000us 0.01% 98.000us 24.500us 0 b 0 b 0 b 0 b 4 \n aten::narrow 0.00% 17.000us 0.00% 45.000us 5.625us 0 b 0 b 0 b 0 b 8 \n [memory] 0.00% 0.000us 0.00% 0.000us 0.000us 0 b 0 b -4.55 Gb -4.55 Gb 86 \n cudaFree 41.47% 390.344ms 41.47% 390.344ms 10.843ms 0 b 0 b 0 b 0 b 36 \n----------------------------- ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ \nSelf CPU time total: 941.160ms\n</code></pre>\n<p>And this is the profiling when <strong>not emptying the cache</strong></p>\n<pre><code class=\"lang-auto\">----------------------------- ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ \n Name Self CPU % Self CPU CPU total % CPU total CPU time avg CPU Mem Self CPU Mem CUDA Mem Self CUDA Mem # of Calls \n----------------------------- ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ \n aten::zeros 0.01% 52.000us 0.03% 145.000us 24.167us 24 b 0 b 0 b 0 b 6 \n aten::empty 0.51% 2.718ms 0.51% 2.718ms 9.245us 120 b 120 b 4.26 Gb 4.26 Gb 294 \n aten::zero_ 0.00% 8.000us 0.00% 8.000us 1.333us 0 b 0 b 0 b 0 b 6 \n Extraction layer 1.17% 6.293ms 4.23% 22.747ms 22.747ms -4 b -20 b 1.11 Gb -4.40 Gb 1 \n aten::avg_pool2d 0.38% 2.027ms 0.44% 2.363ms 56.262us 0 b 0 b 269.72 Mb 269.72 Mb 42 \n cudaLaunchKernel 1.46% 7.829ms 1.46% 7.829ms 8.172us 0 b 0 b 0 b 0 b 958 \n aten::conv2d 0.08% 423.000us 4.33% 23.271ms 136.088us 0 b 0 b 4.94 Gb 0 b 171 \n aten::_convolution_mode 0.11% 575.000us 3.86% 20.716ms 132.795us 0 b 0 b 4.91 Gb 0 b 156 \n aten::_convolution 0.44% 2.351ms 4.14% 22.230ms 130.000us 0 b 0 b 4.94 Gb 0 b 171 \n aten::cudnn_convolution 1.81% 9.703ms 2.76% 14.803ms 86.567us 0 b 0 b 4.94 Gb 3.77 Gb 171 \n aten::resize_ 0.26% 1.383ms 0.26% 1.383ms 3.239us 0 b 0 b 6.85 Gb 6.85 Gb 427 \n cudaEventRecord 0.08% 413.000us 0.08% 413.000us 2.415us 0 b 0 b 0 b 0 b 171 \n aten::reshape 0.10% 556.000us 0.25% 1.348ms 7.883us 0 b 0 b 0 b 0 b 171 \n aten::_reshape_alias 0.15% 792.000us 0.15% 792.000us 4.632us 0 b 0 b 0 b 0 b 171 \n aten::add_ 0.46% 2.467ms 0.69% 3.728ms 21.801us 0 b 0 b 0 b 0 b 171 \n aten::leaky_relu 0.61% 3.277ms 0.82% 4.395ms 28.914us 0 b 0 b 4.50 Gb 4.50 Gb 152 \n aten::cat 0.09% 471.000us 0.78% 4.187ms 49.259us 0 b 0 b 6.85 Gb 0 b 85 \n aten::_cat 0.27% 1.438ms 0.69% 3.716ms 43.718us 0 b 0 b 6.85 Gb 0 b 85 \n Estimation layer 1.63% 8.730ms 80.54% 432.639ms 432.639ms -4 b -20 b 18.75 Mb -6.36 Gb 1 \n cudaBindTexture 0.02% 117.000us 0.02% 117.000us 7.312us 0 b 0 b 0 b 0 b 16 \n cudaUnbindTexture 0.00% 18.000us 0.00% 18.000us 1.125us 0 b 0 b 0 b 0 b 16 \n aten::convolution 0.01% 43.000us 0.40% 2.135ms 142.333us 0 b 0 b 29.30 Mb 0 b 15 \n aten::mul 0.17% 888.000us 0.22% 1.177ms 34.618us 0 b 0 b 28.49 Mb 28.49 Mb 34 \n aten::upsample_bilinear2d 0.12% 650.000us 0.16% 852.000us 35.500us 0 b 0 b 38.56 Mb 38.56 Mb 24 \n aten::slice 0.14% 772.000us 0.19% 1.016ms 5.522us 0 b 0 b 0 b 0 b 184 \n aten::as_strided 0.06% 339.000us 0.06% 339.000us 1.500us 0 b 0 b 0 b 0 b 226 \n aten::div 0.51% 2.759ms 0.59% 3.154ms 71.682us 0 b 0 b 37.48 Mb 37.48 Mb 44 \n aten::to 0.02% 85.000us 88.07% 473.090ms 21.504ms 0 b 0 b 37.48 Mb 0 b 22 \n aten::_to_copy 0.06% 300.000us 88.05% 473.005ms 21.500ms 0 b 0 b 37.48 Mb 0 b 22 \n aten::empty_strided 0.07% 399.000us 0.07% 399.000us 18.136us 0 b 0 b 37.48 Mb 37.48 Mb 22 \n aten::copy_ 0.09% 489.000us 87.95% 472.422ms 18.170ms 0 b 0 b 0 b 0 b 26 \n cudaMemcpyAsync 21.71% 116.611ms 21.71% 116.611ms 5.301ms 0 b 0 b 0 b 0 b 22 \n cudaStreamSynchronize 66.14% 355.270ms 66.14% 355.270ms 16.149ms 0 b 0 b 0 b 0 b 22 \n aten::add 0.27% 1.462ms 0.38% 2.030ms 44.130us 0 b 0 b 75.15 Mb 75.15 Mb 46 \n aten::permute 0.04% 234.000us 0.06% 329.000us 7.833us 0 b 0 b 0 b 0 b 42 \n aten::grid_sampler 0.03% 146.000us 0.21% 1.121ms 50.955us 0 b 0 b 2.22 Gb 0 b 22 \n aten::grid_sampler_2d 0.07% 358.000us 0.18% 975.000us 44.318us 0 b 0 b 2.22 Gb 0 b 22 \n Concatenate 0.21% 1.132ms 0.56% 2.985ms 1.492ms -8 b -40 b 1.17 Gb -24.04 Mb 2 \n Warp layer 0.39% 2.116ms 13.82% 74.230ms 74.230ms -4 b -20 b 2.24 Gb -75.82 Mb 1 \n Fusion layer 0.23% 1.217ms 0.82% 4.431ms 4.431ms -4 b -20 b 10.55 Mb -5.34 Gb 1 \n aten::upsample_nearest2d 0.02% 123.000us 0.03% 164.000us 41.000us 0 b 0 b 894.52 Mb 894.52 Mb 4 \n aten::constant_pad_nd 0.01% 73.000us 0.07% 396.000us 99.000us 0 b 0 b 898.30 Mb 0 b 4 \n aten::fill_ 0.01% 50.000us 0.02% 87.000us 21.750us 0 b 0 b 0 b 0 b 4 \n aten::narrow 0.00% 20.000us 0.01% 65.000us 8.125us 0 b 0 b 0 b 0 b 8 \n [memory] 0.00% 0.000us 0.00% 0.000us 0.000us 0 b 0 b -4.55 Gb -4.55 Gb 86 \n----------------------------- ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ \nSelf CPU time total: 537.177ms\n</code></pre>\n<p>Here it correctly says that <strong>when not emptying</strong> the cache the model is faster (both have been warmed up). The extra time taken by the first one is because it takes time to empty the cache.</p>\n<p>However, in reality when timing the models with this:</p>\n<pre><code class=\"lang-auto\">with torch.no_grad():\n for i in range(100):\n start_time = time.time()\n output = pytorch_model(image_1, image_2)\n # torch.cuda.empty_cache()\n print(f'It took {time.time()-start_time}s')\n</code></pre>\n<p>And I comment/uncomment the caching operation, they take about the same time (~0.95s) (slightly faster when <strong>not emptying</strong> the cache). However, the inference time of the model alone, calculated like this:</p>\n<pre><code class=\"lang-auto\">with torch.no_grad():\n for i in range(100):\n start_time = time.time()\n output = pytorch_model(image_1, image_2)\n print(f'It took {time.time()-start_time}s')\n # torch.cuda.empty_cache()\n</code></pre>\n<p><strong>when emptying</strong> the cache I get a much faster inference speed (0.55s) compare to when I <strong>don’t empty</strong> the cache (0.95s). Does this mean that my model would run faster if I didn’t have any caching enabled? It looks like when emptying the cache the inference speed is faster, however then emptying the cache takes time. I am a bit confused.</p>",249 "post_number": 5,250 "post_type": 1,251 "posts_count": 6,252 "updated_at": "2022-03-15T09:54:41.456Z",253 "reply_count": 1,254 "reply_to_post_number": null,255 "quote_count": 0,256 "incoming_link_count": 7,257 "reads": 19,258 "readers_count": 18,259 "score": 43.8,260 "yours": false,261 "topic_id": 146385,262 "topic_slug": "torch-cuda-empty-cache-memory-segmentation-and-runtime",263 "display_username": "Davide Paglieri",264 "primary_group_name": null,265 "flair_name": null,266 "flair_url": null,267 "flair_bg_color": null,268 "flair_color": null,269 "flair_group_id": null,270 "badges_granted": [],271 "version": 1,272 "can_edit": false,273 "can_delete": false,274 "can_recover": false,275 "can_see_hidden_post": false,276 "can_wiki": false,277 "read": true,278 "user_title": null,279 "bookmarked": false,280 "actions_summary": [],281 "moderator": false,282 "admin": false,283 "staff": false,284 "user_id": 54035,285 "hidden": false,286 "trust_level": 1,287 "deleted_at": null,288 "user_deleted": false,289 "edit_reason": null,290 "can_view_edit_history": true,291 "wiki": false,292 "post_url": "/t/torch-cuda-empty-cache-memory-segmentation-and-runtime/146385/5",293 "can_accept_answer": false,294 "can_unaccept_answer": false,295 "accepted_answer": false,296 "topic_accepted_answer": null297 },298 {299 "id": 336276,300 "name": "",301 "username": "ptrblck",302 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",303 "created_at": "2022-03-16T06:09:17.900Z",304 "cooked": "<aside class=\"quote no-group\" data-username=\"Davide_Paglieri\" data-post=\"5\" data-topic=\"146385\">\n<div class=\"title\">\n<div class=\"quote-controls\"></div>\n<img loading=\"lazy\" alt=\"\" width=\"24\" height=\"24\" src=\"https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/davide_paglieri/48/47532_2.png\" class=\"avatar\"> Davide_Paglieri:</div>\n<blockquote>\n<p>However, in reality when timing the models with this:</p>\n</blockquote>\n</aside>\n<p>This is not a proper way to profile the models, since CUDA operations are executed asynchronously.<br>\nYou are currently not timing the actual kernels, but the Python overhead, the kernel launches, etc. if you don’t synchronize the code.</p>\n<aside class=\"quote no-group\" data-username=\"Davide_Paglieri\" data-post=\"5\" data-topic=\"146385\">\n<div class=\"title\">\n<div class=\"quote-controls\"></div>\n<img loading=\"lazy\" alt=\"\" width=\"24\" height=\"24\" src=\"https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/davide_paglieri/48/47532_2.png\" class=\"avatar\"> Davide_Paglieri:</div>\n<blockquote>\n<p><strong>when emptying</strong> the cache I get a much faster inference speed (0.55s) compare to when I <strong>don’t empty</strong> the cache (0.95s).</p>\n</blockquote>\n</aside>\n<p>In the second code snippet you have moved the <code>empty_cache</code> call after stopping the (invalid) timer. Given that you are profiling at max the kernel launches, this might be expected.</p>",305 "post_number": 6,306 "post_type": 1,307 "posts_count": 6,308 "updated_at": "2022-03-16T06:09:17.900Z",309 "reply_count": 0,310 "reply_to_post_number": 5,311 "quote_count": 1,312 "incoming_link_count": 7,313 "reads": 19,314 "readers_count": 18,315 "score": 38.8,316 "yours": false,317 "topic_id": 146385,318 "topic_slug": "torch-cuda-empty-cache-memory-segmentation-and-runtime",319 "display_username": "",320 "primary_group_name": null,321 "flair_name": null,322 "flair_url": null,323 "flair_bg_color": null,324 "flair_color": null,325 "flair_group_id": null,326 "badges_granted": [],327 "version": 1,328 "can_edit": false,329 "can_delete": false,330 "can_recover": false,331 "can_see_hidden_post": false,332 "can_wiki": false,333 "read": true,334 "user_title": "",335 "bookmarked": false,336 "actions_summary": [],337 "moderator": true,338 "admin": true,339 "staff": true,340 "user_id": 3534,341 "hidden": false,342 "trust_level": 2,343 "deleted_at": null,344 "user_deleted": false,345 "edit_reason": null,346 "can_view_edit_history": true,347 "wiki": false,348 "post_url": "/t/torch-cuda-empty-cache-memory-segmentation-and-runtime/146385/6",349 "can_accept_answer": false,350 "can_unaccept_answer": false,351 "accepted_answer": false,352 "topic_accepted_answer": null353 }354 ],355 "stream": [356 335902,357 335946,358 335957,359 335968,360 336086,361 336276362 ]363 },364 "timeline_lookup": [365 [366 1,367 1321368 ],369 [370 5,371 1320372 ]373 ],374 "suggested_topics": [375 {376 "fancy_title": "How to save/load PyTorch model with custom Storage pickling",377 "id": 218741,378 "title": "How to save/load PyTorch model with custom Storage pickling",379 "slug": "how-to-save-load-pytorch-model-with-custom-storage-pickling",380 "posts_count": 1,381 "reply_count": 0,382 "highest_post_number": 1,383 "image_url": null,384 "created_at": "2025-04-07T01:58:17.592Z",385 "last_posted_at": "2025-04-07T01:58:17.629Z",386 "bumped": true,387 "bumped_at": "2025-04-07T02:32:17.618Z",388 "archetype": "regular",389 "unseen": false,390 "pinned": false,391 "unpinned": null,392 "visible": true,393 "closed": false,394 "archived": false,395 "bookmarked": null,396 "liked": null,397 "tags_descriptions": {},398 "like_count": 0,399 "views": 32,400 "category_id": 1,401 "featured_link": null,402 "has_accepted_answer": false,403 "posters": [404 {405 "extras": "latest single",406 "description": "Original Poster, Most Recent Poster",407 "user": {408 "id": 83670,409 "username": "brandon.pelfrey",410 "name": "Brandon Pelfrey",411 "avatar_template": "/user_avatar/discuss.pytorch.org/brandon.pelfrey/{size}/76512_2.png",412 "trust_level": 0413 }414 }415 ]416 },417 {418 "fancy_title": "Low GPU Utilization without obvious bottlnecks",419 "id": 215868,420 "title": "Low GPU Utilization without obvious bottlnecks",421 "slug": "low-gpu-utilization-without-obvious-bottlnecks",422 "posts_count": 1,423 "reply_count": 0,424 "highest_post_number": 1,425 "image_url": null,426 "created_at": "2025-01-25T18:26:13.003Z",427 "last_posted_at": "2025-01-25T18:26:13.040Z",428 "bumped": true,429 "bumped_at": "2025-01-25T18:26:13.040Z",430 "archetype": "regular",431 "unseen": false,432 "pinned": false,433 "unpinned": null,434 "visible": true,435 "closed": false,436 "archived": false,437 "bookmarked": null,438 "liked": null,439 "tags_descriptions": {},440 "like_count": 0,441 "views": 115,442 "category_id": 1,443 "featured_link": null,444 "has_accepted_answer": false,445 "posters": [446 {447 "extras": "latest single",448 "description": "Original Poster, Most Recent Poster",449 "user": {450 "id": 82312,451 "username": "Rezzy139",452 "name": "",453 "avatar_template": "/letter_avatar_proxy/v4/letter/r/7bcc69/{size}.png",454 "trust_level": 1455 }456 }457 ]458 },459 {460 "fancy_title": "Torch.onnx.export with FakeTensor",461 "id": 216481,462 "title": "Torch.onnx.export with FakeTensor",463 "slug": "torch-onnx-export-with-faketensor",464 "posts_count": 1,465 "reply_count": 0,466 "highest_post_number": 1,467 "image_url": null,468 "created_at": "2025-02-10T16:04:03.708Z",469 "last_posted_at": "2025-02-10T16:04:03.747Z",470 "bumped": true,471 "bumped_at": "2025-02-10T16:04:03.747Z",472 "archetype": "regular",473 "unseen": false,474 "pinned": false,475 "unpinned": null,476 "visible": true,477 "closed": false,478 "archived": false,479 "bookmarked": null,480 "liked": null,481 "tags_descriptions": {},482 "like_count": 0,483 "views": 56,484 "category_id": 1,485 "featured_link": null,486 "has_accepted_answer": false,487 "posters": [488 {489 "extras": "latest single",490 "description": "Original Poster, Most Recent Poster",491 "user": {492 "id": 82595,493 "username": "temikgo",494 "name": "Artem",495 "avatar_template": "/user_avatar/discuss.pytorch.org/temikgo/{size}/75571_2.png",496 "trust_level": 0497 }498 }499 ]500 },501 {502 "fancy_title": "Flex attention benchmarking",503 "id": 212440,504 "title": "Flex attention benchmarking",505 "slug": "flex-attention-benchmarking",506 "posts_count": 1,507 "reply_count": 0,508 "highest_post_number": 1,509 "image_url": null,510 "created_at": "2024-11-02T06:29:15.428Z",511 "last_posted_at": "2024-11-02T06:29:15.480Z",512 "bumped": true,513 "bumped_at": "2024-11-02T06:29:15.480Z",514 "archetype": "regular",515 "unseen": false,516 "pinned": false,517 "unpinned": null,518 "visible": true,519 "closed": false,520 "archived": false,521 "bookmarked": null,522 "liked": null,523 "tags_descriptions": {},524 "like_count": 0,525 "views": 193,526 "category_id": 1,527 "featured_link": null,528 "has_accepted_answer": false,529 "posters": [530 {531 "extras": "latest single",532 "description": "Original Poster, Most Recent Poster",533 "user": {534 "id": 76970,535 "username": "Lakshya_LNU",536 "name": "Lakshya LNU",537 "avatar_template": "/user_avatar/discuss.pytorch.org/lakshya_lnu/{size}/61585_2.png",538 "trust_level": 1539 }540 }541 ]542 },543 {544 "fancy_title": "How to install pytorch when cuda=12.0?",545 "id": 221574,546 "title": "How to install pytorch when cuda=12.0?",547 "slug": "how-to-install-pytorch-when-cuda-12-0",548 "posts_count": 4,549 "reply_count": 1,550 "highest_post_number": 4,551 "image_url": null,552 "created_at": "2025-07-16T09:07:28.724Z",553 "last_posted_at": "2025-07-17T11:56:28.018Z",554 "bumped": true,555 "bumped_at": "2025-07-17T11:56:28.018Z",556 "archetype": "regular",557 "unseen": false,558 "pinned": false,559 "unpinned": null,560 "visible": true,561 "closed": false,562 "archived": false,563 "bookmarked": null,564 "liked": null,565 "tags_descriptions": {},566 "like_count": 0,567 "views": 452,568 "category_id": 1,569 "featured_link": null,570 "has_accepted_answer": false,571 "posters": [572 {573 "extras": null,574 "description": "Original Poster",575 "user": {576 "id": 85111,577 "username": "2d28xoxk",578 "name": "",579 "avatar_template": "/letter_avatar_proxy/v4/letter/2/51bf81/{size}.png",580 "trust_level": 0581 }582 },583 {584 "extras": null,585 "description": "Frequent Poster",586 "user": {587 "id": 84865,588 "username": "Hamza_Javaid",589 "name": "Hamza Javaid",590 "avatar_template": "/user_avatar/discuss.pytorch.org/hamza_javaid/{size}/77517_2.png",591 "trust_level": 2592 }593 },594 {595 "extras": "latest",596 "description": "Most Recent Poster",597 "user": {598 "id": 3534,599 "username": "ptrblck",600 "name": "",601 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",602 "admin": true,603 "moderator": true,604 "trust_level": 2605 }606 }607 ]608 }609 ],610 "tags_descriptions": {},611 "fancy_title": "Torch.cuda.empty_cache(), memory segmentation and runtime",612 "id": 146385,613 "title": "Torch.cuda.empty_cache(), memory segmentation and runtime",614 "posts_count": 6,615 "created_at": "2022-03-14T14:34:05.456Z",616 "views": 1609,617 "reply_count": 3,618 "like_count": 0,619 "last_posted_at": "2022-03-16T06:09:17.900Z",620 "visible": true,621 "closed": false,622 "archived": false,623 "has_summary": false,624 "archetype": "regular",625 "slug": "torch-cuda-empty-cache-memory-segmentation-and-runtime",626 "category_id": 1,627 "word_count": 2846,628 "deleted_at": null,629 "user_id": 54035,630 "featured_link": null,631 "pinned_globally": false,632 "pinned_at": null,633 "pinned_until": null,634 "image_url": null,635 "slow_mode_seconds": 0,636 "draft": null,637 "draft_key": "topic_146385",638 "draft_sequence": null,639 "unpinned": null,640 "pinned": false,641 "current_post_number": 1,642 "highest_post_number": 6,643 "deleted_by": null,644 "actions_summary": [645 {646 "id": 4,647 "count": 0,648 "hidden": false,649 "can_act": false650 },651 {652 "id": 8,653 "count": 0,654 "hidden": false,655 "can_act": false656 },657 {658 "id": 10,659 "count": 0,660 "hidden": false,661 "can_act": false662 },663 {664 "id": 7,665 "count": 0,666 "hidden": false,667 "can_act": false668 }669 ],670 "chunk_size": 20,671 "bookmarked": false,672 "topic_timer": null,673 "message_bus_last_id": 0,674 "participant_count": 2,675 "show_read_indicator": false,676 "thumbnails": null,677 "slow_mode_enabled_until": null,678 "can_vote": false,679 "vote_count": 0,680 "user_voted": false,681 "discourse_zendesk_plugin_zendesk_id": null,682 "discourse_zendesk_plugin_zendesk_url": "https://your-url.zendesk.com/agent/tickets/",683 "details": {684 "can_edit": false,685 "notification_level": 1,686 "participants": [687 {688 "id": 3534,689 "username": "ptrblck",690 "name": "",691 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",692 "post_count": 3,693 "primary_group_name": null,694 "flair_name": null,695 "flair_url": null,696 "flair_color": null,697 "flair_bg_color": null,698 "flair_group_id": null,699 "admin": true,700 "moderator": true,701 "trust_level": 2702 },703 {704 "id": 54035,705 "username": "Davide_Paglieri",706 "name": "Davide Paglieri",707 "avatar_template": "/user_avatar/discuss.pytorch.org/davide_paglieri/{size}/47532_2.png",708 "post_count": 3,709 "primary_group_name": null,710 "flair_name": null,711 "flair_url": null,712 "flair_color": null,713 "flair_bg_color": null,714 "flair_group_id": null,715 "trust_level": 1716 }717 ],718 "created_by": {719 "id": 54035,720 "username": "Davide_Paglieri",721 "name": "Davide Paglieri",722 "avatar_template": "/user_avatar/discuss.pytorch.org/davide_paglieri/{size}/47532_2.png"723 },724 "last_poster": {725 "id": 3534,726 "username": "ptrblck",727 "name": "",728 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png"729 }730 },731 "bookmarks": []732 },733 {734 "post_stream": {735 "posts": [736 {737 "id": 335933,738 "name": "Santosh Gupta",739 "username": "Santosh_Gupta",740 "avatar_template": "/user_avatar/discuss.pytorch.org/santosh_gupta/{size}/41337_2.png",741 "created_at": "2022-03-14T17:50:21.152Z",742 "cooked": "<p>I’m looking how to do class weighting using BCEWithLogitsLoss.</p>\n<p><a href=\"https://pytorch.org/docs/stable/generated/torch.nn.BCEWithLogitsLoss.html\" class=\"onebox\" target=\"_blank\" rel=\"noopener nofollow ugc\">https://pytorch.org/docs/stable/generated/torch.nn.BCEWithLogitsLoss.html</a></p>\n<p>The example on how to use <code>pos_weight</code> seems clear to me. If there are 3x more negative samples than positive samples, then you can set pos_weight=3</p>\n<p>Does the <code>weight</code> parameter do the same thing?</p>\n<p>Say that I set it <code>weight=torch.tensor([1, 3])</code>. Is that the same thing as <code>pos_weight=3</code></p>\n<p>Also, is <code>weight</code> normalized? Is <code>weight=torch.tensor([1, 3])</code> the same as <code>weight=torch.tensor([3, 9])</code>, or are they different in how they affect the magnitude of the loss?</p>",743 "post_number": 1,744 "post_type": 1,745 "posts_count": 4,746 "updated_at": "2022-03-14T17:50:21.152Z",747 "reply_count": 1,748 "reply_to_post_number": null,749 "quote_count": 0,750 "incoming_link_count": 151,751 "reads": 13,752 "readers_count": 12,753 "score": 757.6,754 "yours": false,755 "topic_id": 146402,756 "topic_slug": "weights-in-bcewithlogitsloss-but-with-weight-instead-of-pos-weight",757 "display_username": "Santosh Gupta",758 "primary_group_name": null,759 "flair_name": null,760 "flair_url": null,761 "flair_bg_color": null,762 "flair_color": null,763 "flair_group_id": null,764 "badges_granted": [],765 "version": 1,766 "can_edit": false,767 "can_delete": false,768 "can_recover": false,769 "can_see_hidden_post": false,770 "can_wiki": false,771 "link_counts": [772 {773 "url": "https://pytorch.org/docs/stable/generated/torch.nn.BCEWithLogitsLoss.html",774 "internal": false,775 "reflection": false,776 "title": "BCEWithLogitsLoss — PyTorch 1.11.0 documentation",777 "clicks": 6778 }779 ],780 "read": true,781 "user_title": null,782 "bookmarked": false,783 "actions_summary": [],784 "moderator": false,785 "admin": false,786 "staff": false,787 "user_id": 48196,788 "hidden": false,789 "trust_level": 1,790 "deleted_at": null,791 "user_deleted": false,792 "edit_reason": null,793 "can_view_edit_history": true,794 "wiki": false,795 "post_url": "/t/weights-in-bcewithlogitsloss-but-with-weight-instead-of-pos-weight/146402/1",796 "can_accept_answer": false,797 "can_unaccept_answer": false,798 "accepted_answer": false,799 "topic_accepted_answer": true,800 "can_vote": false801 },802 {803 "id": 336017,804 "name": "",805 "username": "ptrblck",806 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",807 "created_at": "2022-03-15T02:58:42.484Z",808 "cooked": "<aside class=\"quote no-group\" data-username=\"Santosh_Gupta\" data-post=\"1\" data-topic=\"146402\">\n<div class=\"title\">\n<div class=\"quote-controls\"></div>\n<img loading=\"lazy\" alt=\"\" width=\"24\" height=\"24\" src=\"https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/santosh_gupta/48/41337_2.png\" class=\"avatar\"> Santosh_Gupta:</div>\n<blockquote>\n<p>Does the <code>weight</code> parameter do the same thing?</p>\n</blockquote>\n</aside>\n<p>No, the <code>weight</code> argument will apply the weight to each sample of the batch.</p>",809 "post_number": 2,810 "post_type": 1,811 "posts_count": 4,812 "updated_at": "2022-03-15T02:58:42.484Z",813 "reply_count": 0,814 "reply_to_post_number": null,815 "quote_count": 1,816 "incoming_link_count": 1,817 "reads": 11,818 "readers_count": 10,819 "score": 22.2,820 "yours": false,821 "topic_id": 146402,822 "topic_slug": "weights-in-bcewithlogitsloss-but-with-weight-instead-of-pos-weight",823 "display_username": "",824 "primary_group_name": null,825 "flair_name": null,826 "flair_url": null,827 "flair_bg_color": null,828 "flair_color": null,829 "flair_group_id": null,830 "badges_granted": [],831 "version": 1,832 "can_edit": false,833 "can_delete": false,834 "can_recover": false,835 "can_see_hidden_post": false,836 "can_wiki": false,837 "read": true,838 "user_title": "",839 "bookmarked": false,840 "actions_summary": [841 {842 "id": 2,843 "count": 1844 }845 ],846 "moderator": true,847 "admin": true,848 "staff": true,849 "user_id": 3534,850 "hidden": false,851 "trust_level": 2,852 "deleted_at": null,853 "user_deleted": false,854 "edit_reason": null,855 "can_view_edit_history": true,856 "wiki": false,857 "post_url": "/t/weights-in-bcewithlogitsloss-but-with-weight-instead-of-pos-weight/146402/2",858 "can_accept_answer": false,859 "can_unaccept_answer": false,860 "accepted_answer": true,861 "topic_accepted_answer": true862 },863 {864 "id": 336217,865 "name": "Santosh Gupta",866 "username": "Santosh_Gupta",867 "avatar_template": "/user_avatar/discuss.pytorch.org/santosh_gupta/{size}/41337_2.png",868 "created_at": "2022-03-15T19:26:23.118Z",869 "cooked": "<p>Thanks ptrblack!</p>\n<p>I have one more question, about the example</p>\n<blockquote>\n<blockquote>\n<blockquote>\n<p>target = torch.ones([10, 64], dtype=torch.float32) # 64 classes, batch size = 10<br>\noutput = torch.full([10, 64], 1.5) # A prediction (logit)<br>\npos_weight = torch.ones([64]) # All weights are equal to 1<br>\ncriterion = torch.nn.BCEWithLogitsLoss(pos_weight=pos_weight)<br>\ncriterion(output, target) # -log(sigmoid(1.5))<br>\ntensor(0.2014)</p>\n</blockquote>\n</blockquote>\n</blockquote>\n<p>This seems like it could be one of two scenarios. Either a single binary class, with 64 examples per batch, and a batch size of 10. Or, it could be 64 binary classes, with one example per batch, and a batch size of 10.</p>\n<p>How would <code>torch.nn.BCEWithLogitsLoss</code> know which scenario this is? If I put in a single value, pos_weight=torch.tensor(3), would it automatically assume it’s the former from the length of the tensor? And if I use pos_weight = torch.ones([64]*3, <code>BCEWithLogitsLoss</code> will assume it’s the former?</p>",870 "post_number": 3,871 "post_type": 1,872 "posts_count": 4,873 "updated_at": "2022-03-15T19:26:23.118Z",874 "reply_count": 1,875 "reply_to_post_number": null,876 "quote_count": 0,877 "incoming_link_count": 6,878 "reads": 8,879 "readers_count": 7,880 "score": 36.6,881 "yours": false,882 "topic_id": 146402,883 "topic_slug": "weights-in-bcewithlogitsloss-but-with-weight-instead-of-pos-weight",884 "display_username": "Santosh Gupta",885 "primary_group_name": null,886 "flair_name": null,887 "flair_url": null,888 "flair_bg_color": null,889 "flair_color": null,890 "flair_group_id": null,891 "badges_granted": [],892 "version": 1,893 "can_edit": false,894 "can_delete": false,895 "can_recover": false,896 "can_see_hidden_post": false,897 "can_wiki": false,898 "read": true,899 "user_title": null,900 "bookmarked": false,901 "actions_summary": [],902 "moderator": false,903 "admin": false,904 "staff": false,905 "user_id": 48196,906 "hidden": false,907 "trust_level": 1,908 "deleted_at": null,909 "user_deleted": false,910 "edit_reason": null,911 "can_view_edit_history": true,912 "wiki": false,913 "post_url": "/t/weights-in-bcewithlogitsloss-but-with-weight-instead-of-pos-weight/146402/3",914 "can_accept_answer": false,915 "can_unaccept_answer": false,916 "accepted_answer": false,917 "topic_accepted_answer": true918 },919 {920 "id": 336275,921 "name": "",922 "username": "ptrblck",923 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",924 "created_at": "2022-03-16T06:04:08.065Z",925 "cooked": "<p><code>pos_weight = torch.ones([64]*3)</code> won’t work in this example and will raise:</p>\n<pre data-code-wrap=\"python\"><code class=\"lang-python\">RuntimeError: The size of tensor a (64) must match the size of tensor b (10) at non-singleton dimension 1\n</code></pre>\n<pre data-code-wrap=\"python\"><code class=\"lang-python\">pos_weight=torch.tensor(3)\n</code></pre>\n<p>will broadcast the <code>pos_weight</code> and you will get the same loss value if you use <code>torch.tensor(1.)</code>.</p>\n<aside class=\"quote no-group\" data-username=\"Santosh_Gupta\" data-post=\"3\" data-topic=\"146402\">\n<div class=\"title\">\n<div class=\"quote-controls\"></div>\n<img loading=\"lazy\" alt=\"\" width=\"24\" height=\"24\" src=\"https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/santosh_gupta/48/41337_2.png\" class=\"avatar\"> Santosh_Gupta:</div>\n<blockquote>\n<p>This seems like it could be one of two scenarios. Either a single binary class, with 64 examples per batch, and a batch size of 10. Or, it could be 64 binary classes, with one example per batch, and a batch size of 10.</p>\n</blockquote>\n</aside>\n<p>The example uses 10 samples (<code>batch_size=10</code>) where each sample contains 64 classes which can be active or inactive and can thus be seen as a multi-label classification use case.</p>",926 "post_number": 4,927 "post_type": 1,928 "posts_count": 4,929 "updated_at": "2022-03-16T06:04:08.065Z",930 "reply_count": 0,931 "reply_to_post_number": 3,932 "quote_count": 1,933 "incoming_link_count": 2,934 "reads": 7,935 "readers_count": 6,936 "score": 26.4,937 "yours": false,938 "topic_id": 146402,939 "topic_slug": "weights-in-bcewithlogitsloss-but-with-weight-instead-of-pos-weight",940 "display_username": "",941 "primary_group_name": null,942 "flair_name": null,943 "flair_url": null,944 "flair_bg_color": null,945 "flair_color": null,946 "flair_group_id": null,947 "badges_granted": [],948 "version": 1,949 "can_edit": false,950 "can_delete": false,951 "can_recover": false,952 "can_see_hidden_post": false,953 "can_wiki": false,954 "read": true,955 "user_title": "",956 "bookmarked": false,957 "actions_summary": [958 {959 "id": 2,960 "count": 1961 }962 ],963 "moderator": true,964 "admin": true,965 "staff": true,966 "user_id": 3534,967 "hidden": false,968 "trust_level": 2,969 "deleted_at": null,970 "user_deleted": false,971 "edit_reason": null,972 "can_view_edit_history": true,973 "wiki": false,974 "post_url": "/t/weights-in-bcewithlogitsloss-but-with-weight-instead-of-pos-weight/146402/4",975 "can_accept_answer": false,976 "can_unaccept_answer": false,977 "accepted_answer": false,978 "topic_accepted_answer": true979 }980 ],981 "stream": [982 335933,983 336017,984 336217,985 336275986 ]987 },988 "timeline_lookup": [989 [990 1,991 1321992 ],993 [994 3,995 1320996 ]997 ],998 "suggested_topics": [999 {1000 "fancy_title": "RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Any Suggestions Appreciated",1001 "id": 212791,1002 "title": "RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Any Suggestions Appreciated",1003 "slug": "runtimeerror-trying-to-backward-through-the-graph-a-second-time-but-the-buffers-have-already-been-freed-any-suggestions-appreciated",1004 "posts_count": 1,1005 "reply_count": 0,1006 "highest_post_number": 1,1007 "image_url": null,1008 "created_at": "2024-11-11T06:12:01.423Z",1009 "last_posted_at": "2024-11-11T06:12:01.471Z",1010 "bumped": true,1011 "bumped_at": "2024-11-11T06:12:01.471Z",1012 "archetype": "regular",1013 "unseen": false,1014 "pinned": false,1015 "unpinned": null,1016 "visible": true,1017 "closed": false,1018 "archived": false,1019 "bookmarked": null,1020 "liked": null,1021 "tags_descriptions": {},1022 "like_count": 0,1023 "views": 38,1024 "category_id": 1,1025 "featured_link": null,1026 "has_accepted_answer": false,1027 "posters": [1028 {1029 "extras": "latest single",1030 "description": "Original Poster, Most Recent Poster",1031 "user": {1032 "id": 4746,1033 "username": "Mahir_Atmis",1034 "name": "Mahir Atmis",1035 "avatar_template": "/user_avatar/discuss.pytorch.org/mahir_atmis/{size}/2414_2.png",1036 "trust_level": 11037 }1038 }1039 ]1040 },1041 {1042 "fancy_title": "`num_features` parameter of `nn.InstanceNorm2d` does not change results",1043 "id": 214610,1044 "title": "`num_features` parameter of `nn.InstanceNorm2d` does not change results",1045 "slug": "num-features-parameter-of-nn-instancenorm2d-does-not-change-results",1046 "posts_count": 3,1047 "reply_count": 1,1048 "highest_post_number": 3,1049 "image_url": null,1050 "created_at": "2024-12-24T23:59:04.629Z",1051 "last_posted_at": "2024-12-31T04:48:24.867Z",1052 "bumped": true,1053 "bumped_at": "2024-12-31T04:48:24.867Z",1054 "archetype": "regular",1055 "unseen": false,1056 "pinned": false,1057 "unpinned": null,1058 "visible": true,1059 "closed": false,1060 "archived": false,1061 "bookmarked": null,1062 "liked": null,1063 "tags_descriptions": {},1064 "like_count": 0,1065 "views": 46,1066 "category_id": 1,1067 "featured_link": null,1068 "has_accepted_answer": true,1069 "posters": [1070 {1071 "extras": "latest",1072 "description": "Original Poster, Most Recent Poster",1073 "user": {1074 "id": 68684,1075 "username": "aoot",1076 "name": "",1077 "avatar_template": "/letter_avatar_proxy/v4/letter/a/df788c/{size}.png",1078 "trust_level": 21079 }1080 },1081 {1082 "extras": null,1083 "description": "Frequent Poster, Accepted Answer",1084 "user": {1085 "id": 3534,1086 "username": "ptrblck",1087 "name": "",1088 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",1089 "admin": true,1090 "moderator": true,1091 "trust_level": 21092 }1093 }1094 ]1095 },1096 {1097 "fancy_title": "Understanding the traceback of a compiler error with cache size limit",1098 "id": 215669,1099 "title": "Understanding the traceback of a compiler error with cache size limit",1100 "slug": "understanding-the-traceback-of-a-compiler-error-with-cache-size-limit",1101 "posts_count": 1,1102 "reply_count": 0,1103 "highest_post_number": 1,1104 "image_url": null,1105 "created_at": "2025-01-21T10:43:52.773Z",1106 "last_posted_at": "2025-01-21T10:43:52.812Z",1107 "bumped": true,1108 "bumped_at": "2025-01-21T10:43:52.812Z",1109 "archetype": "regular",1110 "unseen": false,1111 "pinned": false,1112 "unpinned": null,1113 "visible": true,1114 "closed": false,1115 "archived": false,1116 "bookmarked": null,1117 "liked": null,1118 "tags_descriptions": {},1119 "like_count": 0,1120 "views": 53,1121 "category_id": 1,1122 "featured_link": null,1123 "has_accepted_answer": false,1124 "posters": [1125 {1126 "extras": "latest single",1127 "description": "Original Poster, Most Recent Poster",1128 "user": {1129 "id": 82217,1130 "username": "Johannes_Vogt",1131 "name": "Johannes Vogt",1132 "avatar_template": "/user_avatar/discuss.pytorch.org/johannes_vogt/{size}/75220_2.png",1133 "trust_level": 11134 }1135 }1136 ]1137 },1138 {1139 "fancy_title": "Communication behavior of FSDP",1140 "id": 215787,1141 "title": "Communication behavior of FSDP",1142 "slug": "communication-behavior-of-fsdp",1143 "posts_count": 1,1144 "reply_count": 0,1145 "highest_post_number": 1,1146 "image_url": null,1147 "created_at": "2025-01-23T20:34:26.107Z",1148 "last_posted_at": "2025-01-23T20:34:26.158Z",1149 "bumped": true,1150 "bumped_at": "2025-01-24T07:34:35.438Z",1151 "archetype": "regular",1152 "unseen": false,1153 "pinned": false,1154 "unpinned": null,1155 "visible": true,1156 "closed": false,1157 "archived": false,1158 "bookmarked": null,1159 "liked": null,1160 "tags_descriptions": {},1161 "like_count": 0,1162 "views": 87,1163 "category_id": 1,1164 "featured_link": null,1165 "has_accepted_answer": false,1166 "posters": [1167 {1168 "extras": "latest single",1169 "description": "Original Poster, Most Recent Poster",1170 "user": {1171 "id": 80983,1172 "username": "HariSeldon11988",1173 "name": "",1174 "avatar_template": "/letter_avatar_proxy/v4/letter/h/9dc877/{size}.png",1175 "trust_level": 11176 }1177 }1178 ]1179 },1180 {1181 "fancy_title": "[Pytorch1.12] RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation",1182 "id": 216528,1183 "title": "[Pytorch1.12] RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation",1184 "slug": "pytorch1-12-runtimeerror-one-of-the-variables-needed-for-gradient-computation-has-been-modified-by-an-inplace-operation",1185 "posts_count": 1,1186 "reply_count": 0,1187 "highest_post_number": 1,1188 "image_url": null,1189 "created_at": "2025-02-11T13:00:16.487Z",1190 "last_posted_at": "2025-02-11T13:00:16.523Z",1191 "bumped": true,1192 "bumped_at": "2025-02-11T13:00:16.523Z",1193 "archetype": "regular",1194 "unseen": false,1195 "pinned": false,1196 "unpinned": null,1197 "visible": true,1198 "closed": false,1199 "archived": false,1200 "bookmarked": null,