Anurag1734/cuda-error-resolution-analysis
07
1[2 {3 "post_stream": {4 "posts": [5 {6 "id": 44653,7 "name": "",8 "username": "kbw",9 "avatar_template": "/letter_avatar_proxy/v4/letter/k/f14d63/{size}.png",10 "created_at": "2018-05-02T19:48:08.990Z",11 "cooked": "<p>I am training different models on different GPUs simultaneously in one of my remote machines, and I found that the processes running on GPUs whose id isn’t 0 are somehow duplicated on GPU whose id is 0, as shown below:<br>\n<div class=\"lightbox-wrapper\"><a class=\"lightbox\" href=\"https://discuss.pytorch.org/uploads/default/original/2X/7/73341f5e2c532438211538097b87723ae0c507d2.png\" data-download-href=\"https://discuss.pytorch.org/uploads/default/73341f5e2c532438211538097b87723ae0c507d2\" title=\"Screenshot%20from%202018-05-02%2012-13-39\"><img src=\"https://discuss.pytorch.org/uploads/default/optimized/2X/7/73341f5e2c532438211538097b87723ae0c507d2_2_411x500.png\" alt=\"Screenshot%20from%202018-05-02%2012-13-39\" data-base62-sha1=\"gr8A1Mt5I6Bulywpw5iK4gbo13A\" width=\"411\" height=\"500\" srcset=\"https://discuss.pytorch.org/uploads/default/optimized/2X/7/73341f5e2c532438211538097b87723ae0c507d2_2_411x500.png, https://discuss.pytorch.org/uploads/default/optimized/2X/7/73341f5e2c532438211538097b87723ae0c507d2_2_616x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/7/73341f5e2c532438211538097b87723ae0c507d2.png 2x\" data-dominant-color=\"3E1A33\"><div class=\"meta\"><svg class=\"fa d-icon d-icon-far-image svg-icon\" aria-hidden=\"true\"><use href=\"#far-image\"></use></svg><span class=\"filename\">Screenshot%20from%202018-05-02%2012-13-39</span><span class=\"informations\">714×867 105 KB</span><svg class=\"fa d-icon d-icon-discourse-expand svg-icon\" aria-hidden=\"true\"><use href=\"#discourse-expand\"></use></svg></div></a></div></p>\n<p>As you can see from the PIDs, the processes running on GPU_i with i nonzero are also duplicated on GPU_0. I am running the exact same code in each of these GPUs (with different hyperparameters). This is really puzzling because in another remote machine with very similar settings, I could not reproduce this behavior even though I ran exactly the same code in each GPU in exactly the same way. In that machine, there was no duplication of processes to GPU_0, only one process in each GPU.</p>\n<p>So, I concluded that this is likely not due to a bug in my code, but due to external settings like Pytorch, cuda, or driver version. The other machine I tried to reproduce this behavior has the exactly the same version of driver (387.26) and same number of the same GPU (Titan-XP), but they have somewhat different versions of pytorch and cuda. Shown below is the exact versions of the packages installed (all installed by conda) in each machine (the first one is from the machine where I had this problem, and the second from the other machine):<br>\n(The problematic machine)</p>\n<pre><code class=\"lang-auto\">certifi 2016.2.28 py36_0 \ncffi 1.10.0 py36_0 \ncudatoolkit 8.0 3 anaconda\ncudnn 6.0.21 cuda8.0_0 \ncycler 0.10.0 py36_0 \ncycler 0.10.0 <pip>\ndbus 1.10.20 0 \ndecorator 4.1.2 py36_0 \nexpat 2.1.0 0 \nfontconfig 2.12.1 3 \nfreetype 2.5.5 2 \nglib 2.50.2 1 \ngst-plugins-base 1.8.0 0 \ngstreamer 1.8.0 0 \nicu 54.1 0 \nimageio 2.3.0 py36_0 conda-forge\nintel-openmp 2018.0.0 hc7b2577_8 \njbig 2.1 0 \njpeg 9b 0 \nlibffi 3.2.1 1 \nlibgcc 5.2.0 0 \nlibgcc-ng 7.2.0 h7cc24e2_2 \nlibgfortran 3.0.0 1 \nlibgfortran-ng 7.2.0 h9f7466a_2 \nlibiconv 1.14 0 \nlibpng 1.6.30 1 \nlibstdcxx-ng 7.2.0 h7a57d05_2 \nlibtiff 4.0.6 3 \nlibxcb 1.12 1 \nlibxml2 2.9.4 0 \nmatplotlib 2.0.2 <pip>\nmatplotlib 2.0.2 np113py36_0 \nmkl 2018.0.1 h19d6760_4 \nnccl 1.3.4 cuda8.0_1 \nnetworkx 1.11 py36_0 \nnumpy 1.13.3 py36ha12f23b_0 \nolefile 0.44 py36_0 \nopenssl 1.0.2l 0 \npcre 8.39 1 \npillow 4.2.1 py36_0 \npip 9.0.1 py36_1 \npycparser 2.18 py36_0 \npyparsing 2.2.0 py36_0 \npyparsing 2.2.0 <pip>\npyqt 5.6.0 py36_2 \npython 3.6.2 0 \npython-dateutil 2.6.1 <pip>\npython-dateutil 2.6.1 py36_0 \npytorch 0.3.0 py36_cuda8.0.61_cudnn7.0.3h37a80b5_4 pytorch\npytz 2017.2 py36_0 \npytz 2017.2 <pip>\npywavelets 0.5.2 np113py36_0 \nqt 5.6.2 5 \nreadline 6.2 2 \nscikit-image 0.13.0 np113py36_0 \nscipy 1.0.0 py36hbf646e7_0 \nsetuptools 36.4.0 py36_0 \nsip 4.18 py36_0 \nsix 1.10.0 py36_0 \nsqlite 3.13.0 0 \ntk 8.5.18 0 \ntorchvision 0.2.0 py36h17b6947_1 pytorch\nwheel 0.29.0 py36_0 \nxz 5.2.3 0 \nzlib 1.2.11 0 \n\n</code></pre>\n<p>(The other machine)</p>\n<pre><code class=\"lang-auto\">ca-certificates 2018.03.07 0 \ncertifi 2018.4.16 py36_0 \ncffi 1.11.5 py36h9745a5d_0 \ncuda91 1.0 h4c16780_0 pytorch\ncycler 0.10.0 py36h93f1223_0 \ndbus 1.13.2 h714fa37_1 \nexpat 2.2.5 he0dffb1_0 \nfontconfig 2.12.6 h49f89f6_0 \nfreetype 2.8 hab7d2ae_1 \nglib 2.56.1 h000015b_0 \ngst-plugins-base 1.14.0 hbbd80ab_1 \ngstreamer 1.14.0 hb453b48_1 \nicu 58.2 h9c2bf20_1 \nimageio 2.3.0 py36_0 \nintel-openmp 2018.0.0 8 \njpeg 9b h024ee3a_2 \nkiwisolver 1.0.1 py36h764f252_0 \nlibedit 3.1 heed3624_0 \nlibffi 3.2.1 hd88cf55_4 \nlibgcc-ng 7.2.0 hdf63c60_3 \nlibgfortran-ng 7.2.0 hdf63c60_3 \nlibpng 1.6.34 hb9fc6fc_0 \nlibstdcxx-ng 7.2.0 hdf63c60_3 \nlibtiff 4.0.9 h28f6b97_0 \nlibxcb 1.13 h1bed415_1 \nlibxml2 2.9.8 hf84eae3_0 \nmatplotlib 2.2.2 py36h0e671d2_1 \nmkl 2018.0.2 1 \nmkl_fft 1.0.1 py36h3010b51_0 \nmkl_random 1.0.1 py36h629b387_0 \nncurses 6.0 h9df7e31_2 \nnumpy 1.14.2 py36hdbf6ddf_1 \nolefile 0.45.1 py36_0 \nopenssl 1.0.2o h20670df_0 \npcre 8.42 h439df22_0 \npillow 5.1.0 py36h3deb7b8_0 \npip 9.0.3 py36_0 \npycparser 2.18 py36hf9f622e_1 \npyparsing 2.2.0 py36hee85983_1 \npyqt 5.9.2 py36h751905a_0 \npython 3.6.4 hc3d631a_3 \npython-dateutil 2.7.2 py36_0 \npytorch 0.3.1 py36_cuda9.1.85_cudnn7.0.5_2 [cuda91] pytorch\npytz 2018.4 py36_0 \nqt 5.9.5 h7e424d6_0 \nreadline 7.0 ha6073c6_4 \nscipy 1.0.1 py36hfc37229_0 \nsetuptools 39.0.1 py36_0 \nsip 4.19.8 py36hf484d3e_0 \nsix 1.11.0 py36h372c433_1 \nsqlite 3.23.1 he433501_0 \ntk 8.6.7 hc745277_3 \ntorchvision 0.2.0 py36h17b6947_1 pytorch\ntornado 5.0.2 py36_0 \nwheel 0.31.0 py36_0 \nxz 5.2.3 h55aa19d_2 \nzlib 1.2.11 ha838bed_2 \n\n</code></pre>\n<p>As you can see, their Pytorch and cuda versions are different. Can this be the cause of this weird behavior?<br>\n(By the way, not sure if this is relevant information, but the cuda version I get from nvcc --version in both machines is V7.5.17, possibly due to remnants of the previous installations)</p>",12 "post_number": 1,13 "post_type": 1,14 "posts_count": 14,15 "updated_at": "2018-05-02T19:55:13.793Z",16 "reply_count": 0,17 "reply_to_post_number": null,18 "quote_count": 0,19 "incoming_link_count": 1046,20 "reads": 52,21 "readers_count": 51,22 "score": 5242.4,23 "yours": false,24 "topic_id": 17375,25 "topic_slug": "gpu-0-gets-duplicate-processes-from-processes-running-on-other-gpus",26 "display_username": "",27 "primary_group_name": null,28 "flair_name": null,29 "flair_url": null,30 "flair_bg_color": null,31 "flair_color": null,32 "flair_group_id": null,33 "badges_granted": [],34 "version": 2,35 "can_edit": false,36 "can_delete": false,37 "can_recover": false,38 "can_see_hidden_post": false,39 "can_wiki": false,40 "link_counts": [41 {42 "url": "https://discuss.pytorch.org/uploads/default/original/2X/7/73341f5e2c532438211538097b87723ae0c507d2.png",43 "internal": true,44 "reflection": false,45 "clicks": 046 }47 ],48 "read": true,49 "user_title": null,50 "bookmarked": false,51 "actions_summary": [],52 "moderator": false,53 "admin": false,54 "staff": false,55 "user_id": 8101,56 "hidden": false,57 "trust_level": 1,58 "deleted_at": null,59 "user_deleted": false,60 "edit_reason": null,61 "can_view_edit_history": true,62 "wiki": false,63 "post_url": "/t/gpu-0-gets-duplicate-processes-from-processes-running-on-other-gpus/17375/1",64 "can_accept_answer": false,65 "can_unaccept_answer": false,66 "accepted_answer": false,67 "topic_accepted_answer": null,68 "can_vote": false69 },70 {71 "id": 44677,72 "name": "",73 "username": "ptrblck",74 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",75 "created_at": "2018-05-02T22:18:30.982Z",76 "cooked": "<p>How did you run the different processes?<br>\nDid you use <code>CUDA_VISIBLE_DEVICES=ID</code> in your environment or did you push the <code>tensor</code>s to the appropriate GPU in the different codes?</p>\n<p>If you’ve used the second approach, PyTorch still sees all GPUs and uses GPU0 by default to initialize CUDA as far as I know.</p>",77 "post_number": 2,78 "post_type": 1,79 "posts_count": 14,80 "updated_at": "2018-05-02T22:18:30.982Z",81 "reply_count": 1,82 "reply_to_post_number": null,83 "quote_count": 0,84 "incoming_link_count": 2,85 "reads": 44,86 "readers_count": 43,87 "score": 23.8,88 "yours": false,89 "topic_id": 17375,90 "topic_slug": "gpu-0-gets-duplicate-processes-from-processes-running-on-other-gpus",91 "display_username": "",92 "primary_group_name": null,93 "flair_name": null,94 "flair_url": null,95 "flair_bg_color": null,96 "flair_color": null,97 "flair_group_id": null,98 "badges_granted": [],99 "version": 1,100 "can_edit": false,101 "can_delete": false,102 "can_recover": false,103 "can_see_hidden_post": false,104 "can_wiki": false,105 "read": true,106 "user_title": "",107 "bookmarked": false,108 "actions_summary": [],109 "moderator": true,110 "admin": true,111 "staff": true,112 "user_id": 3534,113 "hidden": false,114 "trust_level": 2,115 "deleted_at": null,116 "user_deleted": false,117 "edit_reason": null,118 "can_view_edit_history": true,119 "wiki": false,120 "post_url": "/t/gpu-0-gets-duplicate-processes-from-processes-running-on-other-gpus/17375/2",121 "can_accept_answer": false,122 "can_unaccept_answer": false,123 "accepted_answer": false,124 "topic_accepted_answer": null125 },126 {127 "id": 44684,128 "name": "",129 "username": "kbw",130 "avatar_template": "/letter_avatar_proxy/v4/letter/k/f14d63/{size}.png",131 "created_at": "2018-05-02T22:29:25.716Z",132 "cooked": "<p>Say my code is main.py. I create a tmux session and 8 panes in it. In i-th panel, I run 'python main.py --gpu_idx i\" where i starts from 0 to 7 and --gpu_idx indicates the device id of the gpu I want to use. At the very beginning of the code there is torch.cuda.set_device(args.gpu_idx) to set the gpu to use. Then, I apply .cuda() to the network and loss function, and in the training loop, I wrap the data and labels tensors as Variables, and do data = data.cuda() and labels = labels.cuda(), in a very standard way. I never used CUDA_VISIBLE_DEVICES=ID. So, I think I used the second approach you are mentioning. But, what is really strange is that in the other machine, the very same code works just as expected, meaning that running main.py --gpu_idx i only creates a process in the corresponding gpu, and not in GPU0.</p>",133 "post_number": 3,134 "post_type": 1,135 "posts_count": 14,136 "updated_at": "2018-05-02T22:29:25.716Z",137 "reply_count": 2,138 "reply_to_post_number": 2,139 "quote_count": 0,140 "incoming_link_count": 11,141 "reads": 45,142 "readers_count": 44,143 "score": 74.0,144 "yours": false,145 "topic_id": 17375,146 "topic_slug": "gpu-0-gets-duplicate-processes-from-processes-running-on-other-gpus",147 "display_username": "",148 "primary_group_name": null,149 "flair_name": null,150 "flair_url": null,151 "flair_bg_color": null,152 "flair_color": null,153 "flair_group_id": null,154 "badges_granted": [],155 "version": 1,156 "can_edit": false,157 "can_delete": false,158 "can_recover": false,159 "can_see_hidden_post": false,160 "can_wiki": false,161 "read": true,162 "user_title": null,163 "reply_to_user": {164 "id": 3534,165 "username": "ptrblck",166 "name": "",167 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png"168 },169 "bookmarked": false,170 "actions_summary": [],171 "moderator": false,172 "admin": false,173 "staff": false,174 "user_id": 8101,175 "hidden": false,176 "trust_level": 1,177 "deleted_at": null,178 "user_deleted": false,179 "edit_reason": null,180 "can_view_edit_history": true,181 "wiki": false,182 "post_url": "/t/gpu-0-gets-duplicate-processes-from-processes-running-on-other-gpus/17375/3",183 "can_accept_answer": false,184 "can_unaccept_answer": false,185 "accepted_answer": false,186 "topic_accepted_answer": null187 },188 {189 "id": 44687,190 "name": "",191 "username": "ptrblck",192 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",193 "created_at": "2018-05-02T22:37:12.313Z",194 "cooked": "<p>Ok, I see.<br>\nCould you try to use</p>\n<pre><code>CUDA_VISIBLE_DEVICES=gpu_idx python main.py\n</code></pre>\n<p>and see if this problem still occurs.</p>\n<p>Also, since you are using <code>0.3.x</code>, I would suggest to update to the latest stable release <code>0.4.0</code>.<br>\nIt has some nice features and bug fixes. Have a look at the <a href=\"https://pytorch.org/2018/04/22/0_4_0-migration-guide.html\">Migration Guide</a>.</p>\n<p>Unfortunately, I cannot test your script on my machine at the moment, since it’s busy.</p>",195 "post_number": 4,196 "post_type": 1,197 "posts_count": 14,198 "updated_at": "2018-05-02T22:37:12.313Z",199 "reply_count": 0,200 "reply_to_post_number": 3,201 "quote_count": 0,202 "incoming_link_count": 0,203 "reads": 42,204 "readers_count": 41,205 "score": 8.4,206 "yours": false,207 "topic_id": 17375,208 "topic_slug": "gpu-0-gets-duplicate-processes-from-processes-running-on-other-gpus",209 "display_username": "",210 "primary_group_name": null,211 "flair_name": null,212 "flair_url": null,213 "flair_bg_color": null,214 "flair_color": null,215 "flair_group_id": null,216 "badges_granted": [],217 "version": 1,218 "can_edit": false,219 "can_delete": false,220 "can_recover": false,221 "can_see_hidden_post": false,222 "can_wiki": false,223 "link_counts": [224 {225 "url": "https://pytorch.org/2018/04/22/0_4_0-migration-guide.html",226 "internal": false,227 "reflection": false,228 "title": "PyTorch 0.4.0 Migration Guide",229 "clicks": 4230 }231 ],232 "read": true,233 "user_title": "",234 "reply_to_user": {235 "id": 8101,236 "username": "kbw",237 "name": "",238 "avatar_template": "/letter_avatar_proxy/v4/letter/k/f14d63/{size}.png"239 },240 "bookmarked": false,241 "actions_summary": [],242 "moderator": true,243 "admin": true,244 "staff": true,245 "user_id": 3534,246 "hidden": false,247 "trust_level": 2,248 "deleted_at": null,249 "user_deleted": false,250 "edit_reason": null,251 "can_view_edit_history": true,252 "wiki": false,253 "post_url": "/t/gpu-0-gets-duplicate-processes-from-processes-running-on-other-gpus/17375/4",254 "can_accept_answer": false,255 "can_unaccept_answer": false,256 "accepted_answer": false,257 "topic_accepted_answer": null258 },259 {260 "id": 44693,261 "name": "ngimel",262 "username": "ngimel",263 "avatar_template": "/user_avatar/discuss.pytorch.org/ngimel/{size}/14766_2.png",264 "created_at": "2018-05-02T22:47:19.408Z",265 "cooked": "<p>I think there was a bug in pytorch 0.3 that was fixed in 0.3.1 where if the dataloader was using pinned memory it created context on a 0-th gpu even if 0-th gpu was not used. Probably that’s what you are seeing. As <a class=\"mention\" href=\"/u/ptrblck\">@ptrblck</a> is suggesting, try updating to pytorch 0.4, this is the latest stable version.</p>",266 "post_number": 5,267 "post_type": 1,268 "posts_count": 14,269 "updated_at": "2018-05-02T22:47:19.408Z",270 "reply_count": 1,271 "reply_to_post_number": 3,272 "quote_count": 0,273 "incoming_link_count": 0,274 "reads": 41,275 "readers_count": 40,276 "score": 58.2,277 "yours": false,278 "topic_id": 17375,279 "topic_slug": "gpu-0-gets-duplicate-processes-from-processes-running-on-other-gpus",280 "display_username": "ngimel",281 "primary_group_name": null,282 "flair_name": null,283 "flair_url": null,284 "flair_bg_color": null,285 "flair_color": null,286 "flair_group_id": null,287 "badges_granted": [],288 "version": 1,289 "can_edit": false,290 "can_delete": false,291 "can_recover": false,292 "can_see_hidden_post": false,293 "can_wiki": false,294 "read": true,295 "user_title": null,296 "reply_to_user": {297 "id": 8101,298 "username": "kbw",299 "name": "",300 "avatar_template": "/letter_avatar_proxy/v4/letter/k/f14d63/{size}.png"301 },302 "bookmarked": false,303 "actions_summary": [304 {305 "id": 2,306 "count": 1307 }308 ],309 "moderator": false,310 "admin": false,311 "staff": false,312 "user_id": 189,313 "hidden": false,314 "trust_level": 2,315 "deleted_at": null,316 "user_deleted": false,317 "edit_reason": null,318 "can_view_edit_history": true,319 "wiki": false,320 "post_url": "/t/gpu-0-gets-duplicate-processes-from-processes-running-on-other-gpus/17375/5",321 "can_accept_answer": false,322 "can_unaccept_answer": false,323 "accepted_answer": false,324 "topic_accepted_answer": null325 },326 {327 "id": 44716,328 "name": "",329 "username": "kbw",330 "avatar_template": "/letter_avatar_proxy/v4/letter/k/f14d63/{size}.png",331 "created_at": "2018-05-03T01:11:03.319Z",332 "cooked": "<p>Based on the fact that the exact same code doesn’t have such a problem in pytorch 0.3.1. and I indeed used pinned memory, I think ngimel’s answer is probably right, although I haven’t tried to install 0.3.1 and revisit the problem. I will probably install 0.4 later. Thank you so much guys!</p>",333 "post_number": 6,334 "post_type": 1,335 "posts_count": 14,336 "updated_at": "2018-05-03T01:11:03.319Z",337 "reply_count": 1,338 "reply_to_post_number": 5,339 "quote_count": 0,340 "incoming_link_count": 0,341 "reads": 40,342 "readers_count": 39,343 "score": 13.0,344 "yours": false,345 "topic_id": 17375,346 "topic_slug": "gpu-0-gets-duplicate-processes-from-processes-running-on-other-gpus",347 "display_username": "",348 "primary_group_name": null,349 "flair_name": null,350 "flair_url": null,351 "flair_bg_color": null,352 "flair_color": null,353 "flair_group_id": null,354 "badges_granted": [],355 "version": 1,356 "can_edit": false,357 "can_delete": false,358 "can_recover": false,359 "can_see_hidden_post": false,360 "can_wiki": false,361 "read": true,362 "user_title": null,363 "reply_to_user": {364 "id": 189,365 "username": "ngimel",366 "name": "ngimel",367 "avatar_template": "/user_avatar/discuss.pytorch.org/ngimel/{size}/14766_2.png"368 },369 "bookmarked": false,370 "actions_summary": [],371 "moderator": false,372 "admin": false,373 "staff": false,374 "user_id": 8101,375 "hidden": false,376 "trust_level": 1,377 "deleted_at": null,378 "user_deleted": false,379 "edit_reason": null,380 "can_view_edit_history": true,381 "wiki": false,382 "post_url": "/t/gpu-0-gets-duplicate-processes-from-processes-running-on-other-gpus/17375/6",383 "can_accept_answer": false,384 "can_unaccept_answer": false,385 "accepted_answer": false,386 "topic_accepted_answer": null387 },388 {389 "id": 45112,390 "name": "",391 "username": "kbw",392 "avatar_template": "/letter_avatar_proxy/v4/letter/k/f14d63/{size}.png",393 "created_at": "2018-05-04T21:16:55.201Z",394 "cooked": "<p>For future reference, I confirm that upgrading to 0.3.1 solved the problem.</p>",395 "post_number": 7,396 "post_type": 1,397 "posts_count": 14,398 "updated_at": "2018-05-04T21:16:55.201Z",399 "reply_count": 0,400 "reply_to_post_number": 6,401 "quote_count": 0,402 "incoming_link_count": 3,403 "reads": 35,404 "readers_count": 34,405 "score": 22.0,406 "yours": false,407 "topic_id": 17375,408 "topic_slug": "gpu-0-gets-duplicate-processes-from-processes-running-on-other-gpus",409 "display_username": "",410 "primary_group_name": null,411 "flair_name": null,412 "flair_url": null,413 "flair_bg_color": null,414 "flair_color": null,415 "flair_group_id": null,416 "badges_granted": [],417 "version": 1,418 "can_edit": false,419 "can_delete": false,420 "can_recover": false,421 "can_see_hidden_post": false,422 "can_wiki": false,423 "read": true,424 "user_title": null,425 "reply_to_user": {426 "id": 8101,427 "username": "kbw",428 "name": "",429 "avatar_template": "/letter_avatar_proxy/v4/letter/k/f14d63/{size}.png"430 },431 "bookmarked": false,432 "actions_summary": [],433 "moderator": false,434 "admin": false,435 "staff": false,436 "user_id": 8101,437 "hidden": false,438 "trust_level": 1,439 "deleted_at": null,440 "user_deleted": false,441 "edit_reason": null,442 "can_view_edit_history": true,443 "wiki": false,444 "post_url": "/t/gpu-0-gets-duplicate-processes-from-processes-running-on-other-gpus/17375/7",445 "can_accept_answer": false,446 "can_unaccept_answer": false,447 "accepted_answer": false,448 "topic_accepted_answer": null449 },450 {451 "id": 82818,452 "name": "Andre Nguyen",453 "username": "Andre_Nguyen",454 "avatar_template": "/letter_avatar_proxy/v4/letter/a/ea5d25/{size}.png",455 "created_at": "2018-12-18T16:43:09.163Z",456 "cooked": "<p>I am using PyTorch v1.0.0 and seem to be running into the same issue as described in the original post. Duplicate processes are started on GPU 0 from processes running on the other GPUs. See screenshot for an example.</p>\n<p><div class=\"lightbox-wrapper\"><a class=\"lightbox\" href=\"https://discuss.pytorch.org/uploads/default/original/2X/5/5fc221fcf033edf9fde16c9ec35d83e95dd1c265.jpeg\" data-download-href=\"https://discuss.pytorch.org/uploads/default/5fc221fcf033edf9fde16c9ec35d83e95dd1c265\" title=\"03%20PM\"><img src=\"https://discuss.pytorch.org/uploads/default/optimized/2X/5/5fc221fcf033edf9fde16c9ec35d83e95dd1c265_2_325x499.jpeg\" alt=\"03%20PM\" data-base62-sha1=\"dF7hO5MKeZ6SLVkc7jO24CFGUsZ\" width=\"325\" height=\"499\" srcset=\"https://discuss.pytorch.org/uploads/default/optimized/2X/5/5fc221fcf033edf9fde16c9ec35d83e95dd1c265_2_325x499.jpeg, https://discuss.pytorch.org/uploads/default/optimized/2X/5/5fc221fcf033edf9fde16c9ec35d83e95dd1c265_2_487x748.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/5/5fc221fcf033edf9fde16c9ec35d83e95dd1c265_2_650x998.jpeg 2x\" data-dominant-color=\"322A28\"><div class=\"meta\"><svg class=\"fa d-icon d-icon-far-image svg-icon\" aria-hidden=\"true\"><use href=\"#far-image\"></use></svg><span class=\"filename\">03%20PM</span><span class=\"informations\">966×1486 882 KB</span><svg class=\"fa d-icon d-icon-discourse-expand svg-icon\" aria-hidden=\"true\"><use href=\"#discourse-expand\"></use></svg></div></a></div></p>",457 "post_number": 8,458 "post_type": 1,459 "posts_count": 14,460 "updated_at": "2018-12-18T16:43:09.163Z",461 "reply_count": 1,462 "reply_to_post_number": null,463 "quote_count": 0,464 "incoming_link_count": 5,465 "reads": 29,466 "readers_count": 28,467 "score": 35.8,468 "yours": false,469 "topic_id": 17375,470 "topic_slug": "gpu-0-gets-duplicate-processes-from-processes-running-on-other-gpus",471 "display_username": "Andre Nguyen",472 "primary_group_name": null,473 "flair_name": null,474 "flair_url": null,475 "flair_bg_color": null,476 "flair_color": null,477 "flair_group_id": null,478 "badges_granted": [],479 "version": 1,480 "can_edit": false,481 "can_delete": false,482 "can_recover": false,483 "can_see_hidden_post": false,484 "can_wiki": false,485 "link_counts": [486 {487 "url": "https://discuss.pytorch.org/uploads/default/original/2X/5/5fc221fcf033edf9fde16c9ec35d83e95dd1c265.jpeg",488 "internal": true,489 "reflection": false,490 "clicks": 0491 }492 ],493 "read": true,494 "user_title": null,495 "bookmarked": false,496 "actions_summary": [],497 "moderator": false,498 "admin": false,499 "staff": false,500 "user_id": 13999,501 "hidden": false,502 "trust_level": 1,503 "deleted_at": null,504 "user_deleted": false,505 "edit_reason": null,506 "can_view_edit_history": true,507 "wiki": false,508 "post_url": "/t/gpu-0-gets-duplicate-processes-from-processes-running-on-other-gpus/17375/8",509 "can_accept_answer": false,510 "can_unaccept_answer": false,511 "accepted_answer": false,512 "topic_accepted_answer": null513 },514 {515 "id": 143423,516 "name": "",517 "username": "zqin",518 "avatar_template": "/user_avatar/discuss.pytorch.org/zqin/{size}/16343_2.png",519 "created_at": "2019-11-01T02:43:12.357Z",520 "cooked": "<p>Hi, I am facing the same issue. Did you solve this problem?</p>",521 "post_number": 9,522 "post_type": 1,523 "posts_count": 14,524 "updated_at": "2019-11-01T02:43:12.357Z",525 "reply_count": 0,526 "reply_to_post_number": 8,527 "quote_count": 0,528 "incoming_link_count": 6,529 "reads": 19,530 "readers_count": 18,531 "score": 33.8,532 "yours": false,533 "topic_id": 17375,534 "topic_slug": "gpu-0-gets-duplicate-processes-from-processes-running-on-other-gpus",535 "display_username": "",536 "primary_group_name": null,537 "flair_name": null,538 "flair_url": null,539 "flair_bg_color": null,540 "flair_color": null,541 "flair_group_id": null,542 "badges_granted": [],543 "version": 1,544 "can_edit": false,545 "can_delete": false,546 "can_recover": false,547 "can_see_hidden_post": false,548 "can_wiki": false,549 "read": true,550 "user_title": null,551 "reply_to_user": {552 "id": 13999,553 "username": "Andre_Nguyen",554 "name": "Andre Nguyen",555 "avatar_template": "/letter_avatar_proxy/v4/letter/a/ea5d25/{size}.png"556 },557 "bookmarked": false,558 "actions_summary": [],559 "moderator": false,560 "admin": false,561 "staff": false,562 "user_id": 23008,563 "hidden": false,564 "trust_level": 1,565 "deleted_at": null,566 "user_deleted": false,567 "edit_reason": null,568 "can_view_edit_history": true,569 "wiki": false,570 "post_url": "/t/gpu-0-gets-duplicate-processes-from-processes-running-on-other-gpus/17375/9",571 "can_accept_answer": false,572 "can_unaccept_answer": false,573 "accepted_answer": false,574 "topic_accepted_answer": null575 },576 {577 "id": 186831,578 "name": "Bruno Korbar",579 "username": "bjuncek",580 "avatar_template": "/user_avatar/discuss.pytorch.org/bjuncek/{size}/3351_2.png",581 "created_at": "2020-04-27T19:34:06.749Z",582 "cooked": "<p>Hi <a class=\"mention\" href=\"/u/ptrblck\">@ptrblck</a>, I’m still facing this issue now with DDP being set up by default. Any update on what might have caused this?</p>",583 "post_number": 10,584 "post_type": 1,585 "posts_count": 14,586 "updated_at": "2020-04-27T19:34:06.749Z",587 "reply_count": 1,588 "reply_to_post_number": null,589 "quote_count": 0,590 "incoming_link_count": 23,591 "reads": 16,592 "readers_count": 15,593 "score": 123.2,594 "yours": false,595 "topic_id": 17375,596 "topic_slug": "gpu-0-gets-duplicate-processes-from-processes-running-on-other-gpus",597 "display_username": "Bruno Korbar",598 "primary_group_name": null,599 "flair_name": null,600 "flair_url": null,601 "flair_bg_color": null,602 "flair_color": null,603 "flair_group_id": null,604 "badges_granted": [],605 "version": 1,606 "can_edit": false,607 "can_delete": false,608 "can_recover": false,609 "can_see_hidden_post": false,610 "can_wiki": false,611 "read": true,612 "user_title": null,613 "bookmarked": false,614 "actions_summary": [],615 "moderator": false,616 "admin": false,617 "staff": false,618 "user_id": 228,619 "hidden": false,620 "trust_level": 1,621 "deleted_at": null,622 "user_deleted": false,623 "edit_reason": null,624 "can_view_edit_history": true,625 "wiki": false,626 "post_url": "/t/gpu-0-gets-duplicate-processes-from-processes-running-on-other-gpus/17375/10",627 "can_accept_answer": false,628 "can_unaccept_answer": false,629 "accepted_answer": false,630 "topic_accepted_answer": null631 },632 {633 "id": 186911,634 "name": "",635 "username": "ptrblck",636 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",637 "created_at": "2020-04-28T02:25:13.653Z",638 "cooked": "<p>The original issue was created for PyTorch <code>0.3</code>, so let’s focus on your new problem instead. <img src=\"https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9\" title=\":wink:\" class=\"emoji\" alt=\":wink:\"></p>\n<p>Are you using any device-specific operations such as <code>empty_cache</code> without specifying the device id?<br>\nThis could create a new CUDA context on the default device from another process, if you haven’t masked them via <code>CUDA_VISIBLE_DEVICES</code>.<br>\nIf that’s not the case, could you post your code, so that we could have a look?</p>",639 "post_number": 11,640 "post_type": 1,641 "posts_count": 14,642 "updated_at": "2020-04-28T02:25:13.653Z",643 "reply_count": 2,644 "reply_to_post_number": 10,645 "quote_count": 0,646 "incoming_link_count": 8,647 "reads": 16,648 "readers_count": 15,649 "score": 53.2,650 "yours": false,651 "topic_id": 17375,652 "topic_slug": "gpu-0-gets-duplicate-processes-from-processes-running-on-other-gpus",653 "display_username": "",654 "primary_group_name": null,655 "flair_name": null,656 "flair_url": null,657 "flair_bg_color": null,658 "flair_color": null,659 "flair_group_id": null,660 "badges_granted": [],661 "version": 1,662 "can_edit": false,663 "can_delete": false,664 "can_recover": false,665 "can_see_hidden_post": false,666 "can_wiki": false,667 "read": true,668 "user_title": "",669 "reply_to_user": {670 "id": 228,671 "username": "bjuncek",672 "name": "Bruno Korbar",673 "avatar_template": "/user_avatar/discuss.pytorch.org/bjuncek/{size}/3351_2.png"674 },675 "bookmarked": false,676 "actions_summary": [],677 "moderator": true,678 "admin": true,679 "staff": true,680 "user_id": 3534,681 "hidden": false,682 "trust_level": 2,683 "deleted_at": null,684 "user_deleted": false,685 "edit_reason": null,686 "can_view_edit_history": true,687 "wiki": false,688 "post_url": "/t/gpu-0-gets-duplicate-processes-from-processes-running-on-other-gpus/17375/11",689 "can_accept_answer": false,690 "can_unaccept_answer": false,691 "accepted_answer": false,692 "topic_accepted_answer": null693 },694 {695 "id": 187800,696 "name": "Bruno Korbar",697 "username": "bjuncek",698 "avatar_template": "/user_avatar/discuss.pytorch.org/bjuncek/{size}/3351_2.png",699 "created_at": "2020-04-30T11:52:45.501Z",700 "cooked": "<aside class=\"quote no-group\" data-username=\"ptrblck\" data-post=\"11\" data-topic=\"17375\">\n<div class=\"title\">\n<div class=\"quote-controls\"></div>\n<img loading=\"lazy\" alt=\"\" width=\"24\" height=\"24\" src=\"https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png\" class=\"avatar\"> ptrblck:</div>\n<blockquote>\n<p>If that’s not the case, could you post your code, so that we could have a look?</p>\n</blockquote>\n</aside>\n<p>Ooh, there is a bunch of those, so it took me a few days to chase it out. It was indeed a device-specific op without specifying the ID (or rather, I parsed the args afterwards <img src=\"https://discuss.pytorch.org/images/emoji/apple/man_facepalming.png?v=12\" title=\":man_facepalming:\" class=\"emoji\" alt=\":man_facepalming:\" loading=\"lazy\" width=\"20\" height=\"20\">)!</p>",701 "post_number": 12,702 "post_type": 1,703 "posts_count": 14,704 "updated_at": "2020-04-30T11:52:45.501Z",705 "reply_count": 0,706 "reply_to_post_number": 11,707 "quote_count": 1,708 "incoming_link_count": 10,709 "reads": 16,710 "readers_count": 15,711 "score": 53.2,712 "yours": false,713 "topic_id": 17375,714 "topic_slug": "gpu-0-gets-duplicate-processes-from-processes-running-on-other-gpus",715 "display_username": "Bruno Korbar",716 "primary_group_name": null,717 "flair_name": null,718 "flair_url": null,719 "flair_bg_color": null,720 "flair_color": null,721 "flair_group_id": null,722 "badges_granted": [],723 "version": 1,724 "can_edit": false,725 "can_delete": false,726 "can_recover": false,727 "can_see_hidden_post": false,728 "can_wiki": false,729 "read": true,730 "user_title": null,731 "bookmarked": false,732 "actions_summary": [],733 "moderator": false,734 "admin": false,735 "staff": false,736 "user_id": 228,737 "hidden": false,738 "trust_level": 1,739 "deleted_at": null,740 "user_deleted": false,741 "edit_reason": null,742 "can_view_edit_history": true,743 "wiki": false,744 "post_url": "/t/gpu-0-gets-duplicate-processes-from-processes-running-on-other-gpus/17375/12",745 "can_accept_answer": false,746 "can_unaccept_answer": false,747 "accepted_answer": false,748 "topic_accepted_answer": null749 },750 {751 "id": 459688,752 "name": "Fabio Geraci",753 "username": "FGeraci",754 "avatar_template": "/letter_avatar_proxy/v4/letter/f/278dde/{size}.png",755 "created_at": "2024-11-19T13:51:05.893Z",756 "cooked": "<aside class=\"quote no-group\" data-username=\"ptrblck\" data-post=\"11\" data-topic=\"17375\">\n<div class=\"title\">\n<div class=\"quote-controls\"></div>\n<img loading=\"lazy\" alt=\"\" width=\"24\" height=\"24\" src=\"https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png\" class=\"avatar\"> ptrblck:</div>\n<blockquote>\n<p>empty_cache</p>\n</blockquote>\n</aside>\n<p>I do not have empty_cache, and i now changed <code>torch.cuda.synchronize(device)</code> but i still get the same issue.</p>",757 "post_number": 13,758 "post_type": 1,759 "posts_count": 14,760 "updated_at": "2024-11-19T13:51:05.893Z",761 "reply_count": 0,762 "reply_to_post_number": 11,763 "quote_count": 1,764 "incoming_link_count": 2,765 "reads": 2,766 "readers_count": 1,767 "score": 10.4,768 "yours": false,769 "topic_id": 17375,770 "topic_slug": "gpu-0-gets-duplicate-processes-from-processes-running-on-other-gpus",771 "display_username": "Fabio Geraci",772 "primary_group_name": null,773 "flair_name": null,774 "flair_url": null,775 "flair_bg_color": null,776 "flair_color": null,777 "flair_group_id": null,778 "badges_granted": [],779 "version": 1,780 "can_edit": false,781 "can_delete": false,782 "can_recover": false,783 "can_see_hidden_post": false,784 "can_wiki": false,785 "read": true,786 "user_title": null,787 "bookmarked": false,788 "actions_summary": [],789 "moderator": false,790 "admin": false,791 "staff": false,792 "user_id": 81002,793 "hidden": false,794 "trust_level": 1,795 "deleted_at": null,796 "user_deleted": false,797 "edit_reason": null,798 "can_view_edit_history": true,799 "wiki": false,800 "post_url": "/t/gpu-0-gets-duplicate-processes-from-processes-running-on-other-gpus/17375/13",801 "can_accept_answer": false,802 "can_unaccept_answer": false,803 "accepted_answer": false,804 "topic_accepted_answer": null805 },806 {807 "id": 459690,808 "name": "Fabio Geraci",809 "username": "FGeraci",810 "avatar_template": "/letter_avatar_proxy/v4/letter/f/278dde/{size}.png",811 "created_at": "2024-11-19T14:38:49.795Z",812 "cooked": "<p><a href=\"https://github.com/pytorch/examples/issues/969\" rel=\"noopener nofollow ugc\">Solution</a></p>",813 "post_number": 14,814 "post_type": 1,815 "posts_count": 14,816 "updated_at": "2024-11-19T14:38:49.795Z",817 "reply_count": 0,818 "reply_to_post_number": null,819 "quote_count": 0,820 "incoming_link_count": 0,821 "reads": 2,822 "readers_count": 1,823 "score": 0.4,824 "yours": false,825 "topic_id": 17375,826 "topic_slug": "gpu-0-gets-duplicate-processes-from-processes-running-on-other-gpus",827 "display_username": "Fabio Geraci",828 "primary_group_name": null,829 "flair_name": null,830 "flair_url": null,831 "flair_bg_color": null,832 "flair_color": null,833 "flair_group_id": null,834 "badges_granted": [],835 "version": 1,836 "can_edit": false,837 "can_delete": false,838 "can_recover": false,839 "can_see_hidden_post": false,840 "can_wiki": false,841 "link_counts": [842 {843 "url": "https://github.com/pytorch/examples/issues/969",844 "internal": false,845 "reflection": false,846 "title": "DDP: why does every process allocate memory of GPU 0 and how to avoid it? · Issue #969 · pytorch/examples · GitHub",847 "clicks": 10848 }849 ],850 "read": true,851 "user_title": null,852 "bookmarked": false,853 "actions_summary": [],854 "moderator": false,855 "admin": false,856 "staff": false,857 "user_id": 81002,858 "hidden": false,859 "trust_level": 1,860 "deleted_at": null,861 "user_deleted": false,862 "edit_reason": null,863 "can_view_edit_history": true,864 "wiki": false,865 "post_url": "/t/gpu-0-gets-duplicate-processes-from-processes-running-on-other-gpus/17375/14",866 "can_accept_answer": false,867 "can_unaccept_answer": false,868 "accepted_answer": false,869 "topic_accepted_answer": null870 }871 ],872 "stream": [873 44653,874 44677,875 44684,876 44687,877 44693,878 44716,879 45112,880 82818,881 143423,882 186831,883 186911,884 187800,885 459688,886 459690887 ]888 },889 "timeline_lookup": [890 [891 1,892 2733893 ],894 [895 7,896 2731897 ],898 [899 8,900 2503901 ],902 [903 9,904 2186905 ],906 [907 10,908 2007909 ],910 [911 12,912 2004913 ],914 [915 13,916 340917 ]918 ],919 "suggested_topics": [920 {921 "fancy_title": "Image segmentation advice",922 "id": 218616,923 "title": "Image segmentation advice",924 "slug": "image-segmentation-advice",925 "posts_count": 5,926 "reply_count": 2,927 "highest_post_number": 5,928 "image_url": null,929 "created_at": "2025-04-04T12:39:32.549Z",930 "last_posted_at": "2025-04-07T13:37:29.859Z",931 "bumped": true,932 "bumped_at": "2025-04-07T13:37:29.859Z",933 "archetype": "regular",934 "unseen": false,935 "pinned": false,936 "unpinned": null,937 "visible": true,938 "closed": false,939 "archived": false,940 "bookmarked": null,941 "liked": null,942 "tags_descriptions": {},943 "like_count": 0,944 "views": 109,945 "category_id": 1,946 "featured_link": null,947 "has_accepted_answer": false,948 "posters": [949 {950 "extras": null,951 "description": "Original Poster",952 "user": {953 "id": 83634,954 "username": "Rick1989",955 "name": "Rick",956 "avatar_template": "/user_avatar/discuss.pytorch.org/rick1989/{size}/76489_2.png",957 "trust_level": 0958 }959 },960 {961 "extras": null,962 "description": "Frequent Poster",963 "user": {964 "id": 46784,965 "username": "Ayush_Saini",966 "name": "Ayush Saini",967 "avatar_template": "/user_avatar/discuss.pytorch.org/ayush_saini/{size}/39854_2.png",968 "trust_level": 1969 }970 },971 {972 "extras": "latest",973 "description": "Most Recent Poster",974 "user": {975 "id": 18088,976 "username": "KFrank",977 "name": "K. Frank",978 "avatar_template": "/letter_avatar_proxy/v4/letter/k/ecb155/{size}.png",979 "trust_level": 2980 }981 }982 ]983 },984 {985 "fancy_title": "Why is custom RNN implementation so slow?",986 "id": 213168,987 "title": "Why is custom RNN implementation so slow?",988 "slug": "why-is-custom-rnn-implementation-so-slow",989 "posts_count": 6,990 "reply_count": 4,991 "highest_post_number": 6,992 "image_url": null,993 "created_at": "2024-11-19T19:27:11.458Z",994 "last_posted_at": "2024-11-23T01:41:42.810Z",995 "bumped": true,996 "bumped_at": "2024-11-23T01:41:42.810Z",997 "archetype": "regular",998 "unseen": false,999 "pinned": false,1000 "unpinned": null,1001 "visible": true,1002 "closed": false,1003 "archived": false,1004 "bookmarked": null,1005 "liked": null,1006 "tags_descriptions": {},1007 "like_count": 3,1008 "views": 219,1009 "category_id": 1,1010 "featured_link": null,1011 "has_accepted_answer": false,1012 "posters": [1013 {1014 "extras": null,1015 "description": "Original Poster",1016 "user": {1017 "id": 81008,1018 "username": "omnomc",1019 "name": "",1020 "avatar_template": "/user_avatar/discuss.pytorch.org/omnomc/{size}/74086_2.png",1021 "trust_level": 11022 }1023 },1024 {1025 "extras": null,1026 "description": "Frequent Poster",1027 "user": {1028 "id": 3534,1029 "username": "ptrblck",1030 "name": "",1031 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",1032 "admin": true,1033 "moderator": true,1034 "trust_level": 21035 }1036 },1037 {1038 "extras": "latest",1039 "description": "Most Recent Poster",1040 "user": {1041 "id": 41458,1042 "username": "J_Johnson",1043 "name": "J Johnson",1044 "avatar_template": "/user_avatar/discuss.pytorch.org/j_johnson/{size}/55494_2.png",1045 "trust_level": 21046 }1047 }1048 ]1049 },1050 {1051 "fancy_title": "How to checkpoint/pickle FlexAttention with sliding window mask",1052 "id": 215825,1053 "title": "How to checkpoint/pickle FlexAttention with sliding window mask",1054 "slug": "how-to-checkpoint-pickle-flexattention-with-sliding-window-mask",1055 "posts_count": 1,1056 "reply_count": 0,1057 "highest_post_number": 1,1058 "image_url": null,1059 "created_at": "2025-01-24T13:10:48.413Z",1060 "last_posted_at": "2025-01-24T13:10:48.453Z",1061 "bumped": true,1062 "bumped_at": "2025-01-24T13:10:48.453Z",1063 "archetype": "regular",1064 "unseen": false,1065 "pinned": false,1066 "unpinned": null,1067 "visible": true,1068 "closed": false,1069 "archived": false,1070 "bookmarked": null,1071 "liked": null,1072 "tags_descriptions": {},1073 "like_count": 0,1074 "views": 93,1075 "category_id": 1,1076 "featured_link": null,1077 "has_accepted_answer": false,1078 "posters": [1079 {1080 "extras": "latest single",1081 "description": "Original Poster, Most Recent Poster",1082 "user": {1083 "id": 82293,1084 "username": "cathal",1085 "name": "",1086 "avatar_template": "/letter_avatar_proxy/v4/letter/c/df788c/{size}.png",1087 "trust_level": 11088 }1089 }1090 ]1091 },1092 {1093 "fancy_title": "Bfloat16 dtype runtime error in amp auto_cast enable mode",1094 "id": 213333,1095 "title": "Bfloat16 dtype runtime error in amp auto_cast enable mode",1096 "slug": "bfloat16-dtype-runtime-error-in-amp-auto-cast-enable-mode",1097 "posts_count": 2,1098 "reply_count": 0,1099 "highest_post_number": 2,1100 "image_url": null,1101 "created_at": "2024-11-23T01:16:42.155Z",1102 "last_posted_at": "2024-11-23T02:20:35.809Z",1103 "bumped": true,1104 "bumped_at": "2024-11-23T02:20:35.809Z",1105 "archetype": "regular",1106 "unseen": false,1107 "pinned": false,1108 "unpinned": null,1109 "visible": true,1110 "closed": false,1111 "archived": false,1112 "bookmarked": null,1113 "liked": null,1114 "tags_descriptions": {},1115 "like_count": 0,1116 "views": 188,1117 "category_id": 1,1118 "featured_link": null,1119 "has_accepted_answer": true,1120 "posters": [1121 {1122 "extras": "latest single",1123 "description": "Original Poster, Most Recent Poster, Accepted Answer",1124 "user": {1125 "id": 48125,1126 "username": "dongdongtong",1127 "name": "Dongdongtong",1128 "avatar_template": "/user_avatar/discuss.pytorch.org/dongdongtong/{size}/41258_2.png",1129 "trust_level": 11130 }1131 }1132 ]1133 },1134 {1135 "fancy_title": "Unexpected CUDA IPC handle behaviour",1136 "id": 218867,1137 "title": "Unexpected CUDA IPC handle behaviour",1138 "slug": "unexpected-cuda-ipc-handle-behaviour",1139 "posts_count": 2,1140 "reply_count": 0,1141 "highest_post_number": 2,1142 "image_url": null,1143 "created_at": "2025-04-08T12:41:57.732Z",1144 "last_posted_at": "2025-04-09T14:31:15.553Z",1145 "bumped": true,1146 "bumped_at": "2025-04-09T14:31:15.553Z",1147 "archetype": "regular",1148 "unseen": false,1149 "pinned": false,1150 "unpinned": null,1151 "visible": true,1152 "closed": false,1153 "archived": false,1154 "bookmarked": null,1155 "liked": null,1156 "tags_descriptions": {},1157 "like_count": 0,1158 "views": 180,1159 "category_id": 1,1160 "featured_link": null,1161 "has_accepted_answer": true,1162 "posters": [1163 {1164 "extras": "latest single",1165 "description": "Original Poster, Most Recent Poster, Accepted Answer",1166 "user": {1167 "id": 83703,1168 "username": "Gino",1169 "name": "",1170 "avatar_template": "/user_avatar/discuss.pytorch.org/gino/{size}/74902_2.png",1171 "trust_level": 01172 }1173 }1174 ]1175 }1176 ],1177 "tags_descriptions": {},1178 "fancy_title": "GPU 0 gets duplicate processes from processes running on other GPUs",1179 "id": 17375,1180 "title": "GPU 0 gets duplicate processes from processes running on other GPUs",1181 "posts_count": 14,1182 "created_at": "2018-05-02T19:48:08.930Z",1183 "views": 3190,1184 "reply_count": 9,1185 "like_count": 1,1186 "last_posted_at": "2024-11-19T14:38:49.795Z",1187 "visible": true,1188 "closed": false,1189 "archived": false,1190 "has_summary": false,1191 "archetype": "regular",1192 "slug": "gpu-0-gets-duplicate-processes-from-processes-running-on-other-gpus",1193 "category_id": 1,1194 "word_count": 1607,1195 "deleted_at": null,1196 "user_id": 8101,1197 "featured_link": null,1198 "pinned_globally": false,1199 "pinned_at": null,1200 "pinned_until": null,