Anurag1734/cuda-error-resolution-analysis
07
1[2 {3 "post_stream": {4 "posts": [5 {6 "id": 126491,7 "name": "Dave Cole",8 "username": "ONTDave",9 "avatar_template": "/letter_avatar_proxy/v4/letter/o/6a8cbe/{size}.png",10 "created_at": "2019-07-31T12:40:46.017Z",11 "cooked": "<p>There is some chatter online that I can’t deepcopy a model… Is this right?<br>\nAdditionally, is there a way after loading a model to move it between cpu and gpu?</p>",12 "post_number": 1,13 "post_type": 1,14 "posts_count": 21,15 "updated_at": "2019-07-31T12:40:46.017Z",16 "reply_count": 0,17 "reply_to_post_number": null,18 "quote_count": 0,19 "incoming_link_count": 57652,20 "reads": 1183,21 "readers_count": 1182,22 "score": 288096.6,23 "yours": false,24 "topic_id": 52192,25 "topic_slug": "can-i-deepcopy-a-model",26 "display_username": "Dave Cole",27 "primary_group_name": null,28 "flair_name": null,29 "flair_url": null,30 "flair_bg_color": null,31 "flair_color": null,32 "flair_group_id": null,33 "badges_granted": [],34 "version": 1,35 "can_edit": false,36 "can_delete": false,37 "can_recover": false,38 "can_see_hidden_post": false,39 "can_wiki": false,40 "link_counts": [41 {42 "url": "https://discuss.pytorch.org/t/when-can-you-not-deepcopy-a-model/153226",43 "internal": true,44 "reflection": true,45 "title": "When can you NOT deepcopy a model?",46 "clicks": 11047 },48 {49 "url": "https://discuss.pytorch.org/t/deepcopy-model-errors/114566",50 "internal": true,51 "reflection": true,52 "title": "Deepcopy model errors",53 "clicks": 2954 }55 ],56 "read": true,57 "user_title": null,58 "bookmarked": false,59 "actions_summary": [60 {61 "id": 2,62 "count": 363 }64 ],65 "moderator": false,66 "admin": false,67 "staff": false,68 "user_id": 21124,69 "hidden": false,70 "trust_level": 1,71 "deleted_at": null,72 "user_deleted": false,73 "edit_reason": null,74 "can_view_edit_history": true,75 "wiki": false,76 "post_url": "/t/can-i-deepcopy-a-model/52192/1",77 "can_accept_answer": false,78 "can_unaccept_answer": false,79 "accepted_answer": false,80 "topic_accepted_answer": true,81 "can_vote": false82 },83 {84 "id": 126495,85 "name": "",86 "username": "ptrblck",87 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",88 "created_at": "2019-07-31T12:58:00.454Z",89 "cooked": "<p>You can deepcopy a model:</p>\n<pre><code class=\"lang-python\">model = nn.Linear(1, 1)\nmodel_copy = copy.deepcopy(model)\n\nwith torch.no_grad():\n model.weight.fill_(1.)\n\nprint(model.weight)\n> Parameter containing:\ntensor([[10.]], requires_grad=True)\n\nprint(model_copy.weight)\n> Parameter containing:\ntensor([[-0.5596]], requires_grad=True)\n</code></pre>\n<p>To move a model, just call:</p>\n<pre><code class=\"lang-python\">model.to('cuda:0') # moves model (its parameters) to GPU0\nmodel.to('cpu') # moved model to CPU\n</code></pre>",90 "post_number": 2,91 "post_type": 1,92 "posts_count": 21,93 "updated_at": "2020-08-19T20:16:36.366Z",94 "reply_count": 2,95 "reply_to_post_number": null,96 "quote_count": 0,97 "incoming_link_count": 621,98 "reads": 1176,99 "readers_count": 1175,100 "score": 3500.2,101 "yours": false,102 "topic_id": 52192,103 "topic_slug": "can-i-deepcopy-a-model",104 "display_username": "",105 "primary_group_name": null,106 "flair_name": null,107 "flair_url": null,108 "flair_bg_color": null,109 "flair_color": null,110 "flair_group_id": null,111 "badges_granted": [],112 "version": 1,113 "can_edit": false,114 "can_delete": false,115 "can_recover": false,116 "can_see_hidden_post": false,117 "can_wiki": false,118 "read": true,119 "user_title": "",120 "bookmarked": false,121 "actions_summary": [122 {123 "id": 2,124 "count": 11125 }126 ],127 "moderator": true,128 "admin": true,129 "staff": true,130 "user_id": 3534,131 "hidden": false,132 "trust_level": 2,133 "deleted_at": null,134 "user_deleted": false,135 "edit_reason": null,136 "can_view_edit_history": true,137 "wiki": false,138 "post_url": "/t/can-i-deepcopy-a-model/52192/2",139 "can_accept_answer": false,140 "can_unaccept_answer": false,141 "accepted_answer": true,142 "topic_accepted_answer": true143 },144 {145 "id": 126496,146 "name": "Dave Cole",147 "username": "ONTDave",148 "avatar_template": "/letter_avatar_proxy/v4/letter/o/6a8cbe/{size}.png",149 "created_at": "2019-07-31T13:05:20.945Z",150 "cooked": "<p>Hey! Many thanks for the quick reply <img src=\"https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9\" title=\":slight_smile:\" class=\"emoji\" alt=\":slight_smile:\"><br>\nPleased about the deepcopy.<br>\nSo the model.to syntax…<br>\nI was under the impression this didn’t move the model, just changed its formatting as the documentation here (<a href=\"https://pytorch.org/tutorials/beginner/saving_loading_models.html\" rel=\"nofollow noopener\">https://pytorch.org/tutorials/beginner/saving_loading_models.html</a>) suggests:<br>\n“converts the initialized <code>model</code> to a CUDA optimized model using <code>model.to(torch.device('cuda'))</code> .”<br>\nIt sounds like the map_location bit in torch.load(PATH, map_location=device) specifies where the model is?</p>",151 "post_number": 3,152 "post_type": 1,153 "posts_count": 21,154 "updated_at": "2019-07-31T13:08:26.036Z",155 "reply_count": 1,156 "reply_to_post_number": 2,157 "quote_count": 0,158 "incoming_link_count": 190,159 "reads": 1038,160 "readers_count": 1037,161 "score": 1157.6,162 "yours": false,163 "topic_id": 52192,164 "topic_slug": "can-i-deepcopy-a-model",165 "display_username": "Dave Cole",166 "primary_group_name": null,167 "flair_name": null,168 "flair_url": null,169 "flair_bg_color": null,170 "flair_color": null,171 "flair_group_id": null,172 "badges_granted": [],173 "version": 2,174 "can_edit": false,175 "can_delete": false,176 "can_recover": false,177 "can_see_hidden_post": false,178 "can_wiki": false,179 "link_counts": [180 {181 "url": "https://pytorch.org/tutorials/beginner/saving_loading_models.html",182 "internal": false,183 "reflection": false,184 "title": "Saving and Loading Models — PyTorch Tutorials 1.1.0 documentation",185 "clicks": 239186 }187 ],188 "read": true,189 "user_title": null,190 "reply_to_user": {191 "id": 3534,192 "username": "ptrblck",193 "name": "",194 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png"195 },196 "bookmarked": false,197 "actions_summary": [],198 "moderator": false,199 "admin": false,200 "staff": false,201 "user_id": 21124,202 "hidden": false,203 "trust_level": 1,204 "deleted_at": null,205 "user_deleted": false,206 "edit_reason": null,207 "can_view_edit_history": true,208 "wiki": false,209 "post_url": "/t/can-i-deepcopy-a-model/52192/3",210 "can_accept_answer": false,211 "can_unaccept_answer": false,212 "accepted_answer": false,213 "topic_accepted_answer": true214 },215 {216 "id": 126498,217 "name": "",218 "username": "ptrblck",219 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",220 "created_at": "2019-07-31T13:17:04.695Z",221 "cooked": "<p>The <code>map_location</code> argument specifies where to put the loaded parameters.<br>\nE.g. if you saved the <code>model.state_dict()</code> of a model, which was pushed to the GPU, and would like to load this <code>state_dict</code> on a CPU-only machine, you could specify <code>map_location='cpu'</code> to restore the parameters.</p>",222 "post_number": 4,223 "post_type": 1,224 "posts_count": 21,225 "updated_at": "2020-08-19T20:16:36.303Z",226 "reply_count": 1,227 "reply_to_post_number": 3,228 "quote_count": 0,229 "incoming_link_count": 100,230 "reads": 834,231 "readers_count": 833,232 "score": 671.8,233 "yours": false,234 "topic_id": 52192,235 "topic_slug": "can-i-deepcopy-a-model",236 "display_username": "",237 "primary_group_name": null,238 "flair_name": null,239 "flair_url": null,240 "flair_bg_color": null,241 "flair_color": null,242 "flair_group_id": null,243 "badges_granted": [],244 "version": 1,245 "can_edit": false,246 "can_delete": false,247 "can_recover": false,248 "can_see_hidden_post": false,249 "can_wiki": false,250 "read": true,251 "user_title": "",252 "reply_to_user": {253 "id": 21124,254 "username": "ONTDave",255 "name": "Dave Cole",256 "avatar_template": "/letter_avatar_proxy/v4/letter/o/6a8cbe/{size}.png"257 },258 "bookmarked": false,259 "actions_summary": [],260 "moderator": true,261 "admin": true,262 "staff": true,263 "user_id": 3534,264 "hidden": false,265 "trust_level": 2,266 "deleted_at": null,267 "user_deleted": false,268 "edit_reason": null,269 "can_view_edit_history": true,270 "wiki": false,271 "post_url": "/t/can-i-deepcopy-a-model/52192/4",272 "can_accept_answer": false,273 "can_unaccept_answer": false,274 "accepted_answer": false,275 "topic_accepted_answer": true276 },277 {278 "id": 126499,279 "name": "Dave Cole",280 "username": "ONTDave",281 "avatar_template": "/letter_avatar_proxy/v4/letter/o/6a8cbe/{size}.png",282 "created_at": "2019-07-31T13:21:52.463Z",283 "cooked": "<p>Does that mean that if I load a model from file (saved using torch.save(model) on either a cpu or gpu) and set map_location=‘cpu’ when loading, I don’t need to call .to - if I’m using it on the cpu. I only need to .to it if I’m moving it onto the gpu once again?</p>",284 "post_number": 5,285 "post_type": 1,286 "posts_count": 21,287 "updated_at": "2019-07-31T13:21:52.463Z",288 "reply_count": 1,289 "reply_to_post_number": 4,290 "quote_count": 0,291 "incoming_link_count": 105,292 "reads": 755,293 "readers_count": 754,294 "score": 681.0,295 "yours": false,296 "topic_id": 52192,297 "topic_slug": "can-i-deepcopy-a-model",298 "display_username": "Dave Cole",299 "primary_group_name": null,300 "flair_name": null,301 "flair_url": null,302 "flair_bg_color": null,303 "flair_color": null,304 "flair_group_id": null,305 "badges_granted": [],306 "version": 1,307 "can_edit": false,308 "can_delete": false,309 "can_recover": false,310 "can_see_hidden_post": false,311 "can_wiki": false,312 "read": true,313 "user_title": null,314 "reply_to_user": {315 "id": 3534,316 "username": "ptrblck",317 "name": "",318 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png"319 },320 "bookmarked": false,321 "actions_summary": [],322 "moderator": false,323 "admin": false,324 "staff": false,325 "user_id": 21124,326 "hidden": false,327 "trust_level": 1,328 "deleted_at": null,329 "user_deleted": false,330 "edit_reason": null,331 "can_view_edit_history": true,332 "wiki": false,333 "post_url": "/t/can-i-deepcopy-a-model/52192/5",334 "can_accept_answer": false,335 "can_unaccept_answer": false,336 "accepted_answer": false,337 "topic_accepted_answer": true338 },339 {340 "id": 126500,341 "name": "",342 "username": "ptrblck",343 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",344 "created_at": "2019-07-31T13:26:02.392Z",345 "cooked": "<p>I would recommend to save and load the <code>mode.state_dict()</code>, not the model directly.<br>\nThat being said, I prefer to push the model to CPU first before saving the <code>state_dict</code>.<br>\nThis approach makes sure that I’m able to restore the model on all systems, even when no GPU was found.</p>\n<p>After loading the model, I use <code>model.to('cuda')</code> to push it to the GPU again.</p>",346 "post_number": 6,347 "post_type": 1,348 "posts_count": 21,349 "updated_at": "2019-07-31T13:27:02.932Z",350 "reply_count": 0,351 "reply_to_post_number": 5,352 "quote_count": 0,353 "incoming_link_count": 173,354 "reads": 686,355 "readers_count": 685,356 "score": 1062.2,357 "yours": false,358 "topic_id": 52192,359 "topic_slug": "can-i-deepcopy-a-model",360 "display_username": "",361 "primary_group_name": null,362 "flair_name": null,363 "flair_url": null,364 "flair_bg_color": null,365 "flair_color": null,366 "flair_group_id": null,367 "badges_granted": [],368 "version": 1,369 "can_edit": false,370 "can_delete": false,371 "can_recover": false,372 "can_see_hidden_post": false,373 "can_wiki": false,374 "read": true,375 "user_title": "",376 "reply_to_user": {377 "id": 21124,378 "username": "ONTDave",379 "name": "Dave Cole",380 "avatar_template": "/letter_avatar_proxy/v4/letter/o/6a8cbe/{size}.png"381 },382 "bookmarked": false,383 "actions_summary": [384 {385 "id": 2,386 "count": 4387 }388 ],389 "moderator": true,390 "admin": true,391 "staff": true,392 "user_id": 3534,393 "hidden": false,394 "trust_level": 2,395 "deleted_at": null,396 "user_deleted": false,397 "edit_reason": null,398 "can_view_edit_history": true,399 "wiki": false,400 "post_url": "/t/can-i-deepcopy-a-model/52192/6",401 "can_accept_answer": false,402 "can_unaccept_answer": false,403 "accepted_answer": false,404 "topic_accepted_answer": true405 },406 {407 "id": 222394,408 "name": "Rene Sandoval",409 "username": "pinocchio",410 "avatar_template": "/user_avatar/discuss.pytorch.org/pinocchio/{size}/22093_2.png",411 "created_at": "2020-08-19T20:17:01.466Z",412 "cooked": "<p>this is sufficient:</p>\n<pre><code class=\"lang-auto\">model = nn.Linear(1, 1)\nmodel_copy = copy.deepcopy(model)\n</code></pre>",413 "post_number": 7,414 "post_type": 1,415 "posts_count": 21,416 "updated_at": "2020-08-19T20:17:09.585Z",417 "reply_count": 0,418 "reply_to_post_number": null,419 "quote_count": 0,420 "incoming_link_count": 107,421 "reads": 542,422 "readers_count": 541,423 "score": 643.4,424 "yours": false,425 "topic_id": 52192,426 "topic_slug": "can-i-deepcopy-a-model",427 "display_username": "Rene Sandoval",428 "primary_group_name": null,429 "flair_name": null,430 "flair_url": null,431 "flair_bg_color": null,432 "flair_color": null,433 "flair_group_id": null,434 "badges_granted": [],435 "version": 1,436 "can_edit": false,437 "can_delete": false,438 "can_recover": false,439 "can_see_hidden_post": false,440 "can_wiki": false,441 "read": true,442 "user_title": null,443 "bookmarked": false,444 "actions_summary": [],445 "moderator": false,446 "admin": false,447 "staff": false,448 "user_id": 2283,449 "hidden": false,450 "trust_level": 2,451 "deleted_at": null,452 "user_deleted": false,453 "edit_reason": null,454 "can_view_edit_history": true,455 "wiki": false,456 "post_url": "/t/can-i-deepcopy-a-model/52192/7",457 "can_accept_answer": false,458 "can_unaccept_answer": false,459 "accepted_answer": false,460 "topic_accepted_answer": true461 },462 {463 "id": 251392,464 "name": "Ofri Masad",465 "username": "Ofri_Masad",466 "avatar_template": "/user_avatar/discuss.pytorch.org/ofri_masad/{size}/32321_2.png",467 "created_at": "2020-12-14T15:05:22.417Z",468 "cooked": "<p>what about this code:</p>\n<p>`<br>\nmodel = nn.Linear(1, 1)</p>\n<p>model.to(‘cuda:0’)</p>\n<p>model_copy = copy.deepcopy(model)<br>\n`<br>\nwill this create a copy in the GPU? or will I now have two models pointing to the same location in the GPU? (hopefully not)</p>\n<p>thanks</p>",469 "post_number": 8,470 "post_type": 1,471 "posts_count": 21,472 "updated_at": "2020-12-14T15:05:43.014Z",473 "reply_count": 1,474 "reply_to_post_number": null,475 "quote_count": 0,476 "incoming_link_count": 83,477 "reads": 466,478 "readers_count": 465,479 "score": 558.2,480 "yours": false,481 "topic_id": 52192,482 "topic_slug": "can-i-deepcopy-a-model",483 "display_username": "Ofri Masad",484 "primary_group_name": null,485 "flair_name": null,486 "flair_url": null,487 "flair_bg_color": null,488 "flair_color": null,489 "flair_group_id": null,490 "badges_granted": [],491 "version": 1,492 "can_edit": false,493 "can_delete": false,494 "can_recover": false,495 "can_see_hidden_post": false,496 "can_wiki": false,497 "read": true,498 "user_title": null,499 "bookmarked": false,500 "actions_summary": [501 {502 "id": 2,503 "count": 3504 }505 ],506 "moderator": false,507 "admin": false,508 "staff": false,509 "user_id": 40050,510 "hidden": false,511 "trust_level": 1,512 "deleted_at": null,513 "user_deleted": false,514 "edit_reason": null,515 "can_view_edit_history": true,516 "wiki": false,517 "post_url": "/t/can-i-deepcopy-a-model/52192/8",518 "can_accept_answer": false,519 "can_unaccept_answer": false,520 "accepted_answer": false,521 "topic_accepted_answer": true522 },523 {524 "id": 271349,525 "name": "ynjiun",526 "username": "ynjiun_wang",527 "avatar_template": "/user_avatar/discuss.pytorch.org/ynjiun_wang/{size}/32014_2.png",528 "created_at": "2021-03-18T23:25:10.575Z",529 "cooked": "<p>Hi, I have a customized model and cannot be deepcopy. What would be the major causes of a model cannot be deepcopied? could you shed some light for me to debug my model? Thanks a lot for your help.</p>\n<p>The customized model is the squeezenet ssd lite model in this repo (<a href=\"https://github.com/qfgaohao/pytorch-ssd\" class=\"inline-onebox\" rel=\"noopener nofollow ugc\">GitHub - qfgaohao/pytorch-ssd: MobileNetV1, MobileNetV2, VGG based SSD/SSD-lite implementation in Pytorch 1.0 / Pytorch 0.4. Out-of-box support for retraining on Open Images dataset. ONNX and Caffe2 support. Experiment Ideas like CoordConv.</a>)</p>\n<p>The squeezenet model is defined under vision/nn<br>\nThe ssd model is defined under vision/ssd</p>\n<p>if I do:<br>\nbase_net = squeezenet1_1(False).features<br>\nI can deep copy base_net no problem:<br>\ndbcpy = copy.deepcopy(base_net)</p>\n<p>But if I do<br>\nnet = create_squeezenet_ssd_lite(no_classes, is_test=True)<br>\nthen I have trouble to deepcopy the net:<br>\ndnet = copy.deepcopy(net)</p>\n<p>it will report the following error:<br>\nTraceback (most recent call last):<br>\nFile “/home/paul/.eclipse/360744286_linux_gtk_x86_64/plugins/org.python.pydev.core_7.5.0.202001101138/pysrc/pydevd.py”, line 3129, in <br>\nmain()<br>\nFile “/home/paul/.eclipse/360744286_linux_gtk_x86_64/plugins/org.python.pydev.core_7.5.0.202001101138/pysrc/pydevd.py”, line 3122, in main<br>\nglobals = debugger.run(setup[‘file’], None, None, is_module)<br>\nFile “/home/paul/.eclipse/360744286_linux_gtk_x86_64/plugins/org.python.pydev.core_7.5.0.202001101138/pysrc/pydevd.py”, line 2195, in run<br>\nreturn self._exec(is_module, entry_point_fn, module_name, file, globals, locals)<br>\nFile “/home/paul/.eclipse/360744286_linux_gtk_x86_64/plugins/org.python.pydev.core_7.5.0.202001101138/pysrc/pydevd.py”, line 2202, in _exec<br>\npydev_imports.execfile(file, globals, locals) # execute the script<br>\nFile “/home/paul/.eclipse/360744286_linux_gtk_x86_64/plugins/org.python.pydev.core_7.5.0.202001101138/pysrc/_pydev_imps/_pydev_execfile.py”, line 25, in execfile<br>\nexec(compile(contents+\"\\n\", file, ‘exec’), glob, loc)<br>\nFile “/home/paul/pytorch/od-ssd/ssd-quantized/test_3d.py”, line 50, in <br>\ndnet = copy.deepcopy(net)<br>\nFile “/usr/lib/python3.6/copy.py”, line 180, in deepcopy<br>\ny = _reconstruct(x, memo, *rv)<br>\nFile “/usr/lib/python3.6/copy.py”, line 280, in _reconstruct<br>\nstate = deepcopy(state, memo)<br>\nFile “/usr/lib/python3.6/copy.py”, line 150, in deepcopy<br>\ny = copier(x, memo)<br>\nFile “/usr/lib/python3.6/copy.py”, line 240, in _deepcopy_dict<br>\ny[deepcopy(key, memo)] = deepcopy(value, memo)<br>\nFile “/usr/lib/python3.6/copy.py”, line 169, in deepcopy<br>\nrv = reductor(4)<br>\nTypeError: can’t pickle module objects</p>\n<p>The reason I am asking this question is when I do the QAT (Quantization Aware Training), and try to save the quantized model, using:<br>\nnet.eval()<br>\nnet_int8 = torch.quantization.convert(net)<br>\nnet_int8.save(model_path)<br>\nI will encounter the above deepcopy error. So what shall I do? fix the model to make it deepcopy-able? or QAT doesn’t work for ssd model? could you help to point me to the right direction? Thanks a lot for your help!</p>",530 "post_number": 9,531 "post_type": 1,532 "posts_count": 21,533 "updated_at": "2021-03-19T00:01:55.635Z",534 "reply_count": 1,535 "reply_to_post_number": 2,536 "quote_count": 0,537 "incoming_link_count": 110,538 "reads": 349,539 "readers_count": 348,540 "score": 624.8,541 "yours": false,542 "topic_id": 52192,543 "topic_slug": "can-i-deepcopy-a-model",544 "display_username": "ynjiun",545 "primary_group_name": null,546 "flair_name": null,547 "flair_url": null,548 "flair_bg_color": null,549 "flair_color": null,550 "flair_group_id": null,551 "badges_granted": [],552 "version": 3,553 "can_edit": false,554 "can_delete": false,555 "can_recover": false,556 "can_see_hidden_post": false,557 "can_wiki": false,558 "link_counts": [559 {560 "url": "https://github.com/qfgaohao/pytorch-ssd",561 "internal": false,562 "reflection": false,563 "title": "GitHub - qfgaohao/pytorch-ssd: MobileNetV1, MobileNetV2, VGG based SSD/SSD-lite implementation in Pytorch 1.0 / Pytorch 0.4. Out-of-box support for retraining on Open Images dataset. ONNX and Caffe2 support. Experiment Ideas like CoordConv.",564 "clicks": 6565 }566 ],567 "read": true,568 "user_title": null,569 "reply_to_user": {570 "id": 3534,571 "username": "ptrblck",572 "name": "",573 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png"574 },575 "bookmarked": false,576 "actions_summary": [],577 "moderator": false,578 "admin": false,579 "staff": false,580 "user_id": 43262,581 "hidden": false,582 "trust_level": 2,583 "deleted_at": null,584 "user_deleted": false,585 "edit_reason": null,586 "can_view_edit_history": true,587 "wiki": false,588 "post_url": "/t/can-i-deepcopy-a-model/52192/9",589 "can_accept_answer": false,590 "can_unaccept_answer": false,591 "accepted_answer": false,592 "topic_accepted_answer": true593 },594 {595 "id": 271381,596 "name": "",597 "username": "ptrblck",598 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",599 "created_at": "2021-03-19T04:26:55.924Z",600 "cooked": "<p>Based on the error message it seems that <code>pickle</code> is failing to copy the object, if it has a <a href=\"https://stackoverflow.com/a/2790884\">class attribute that references a module</a>. I don’t know, if this is the case for QAT or if this is caused by your custom model, but could you check for module references inside your model?</p>",601 "post_number": 10,602 "post_type": 1,603 "posts_count": 21,604 "updated_at": "2021-03-19T04:26:55.924Z",605 "reply_count": 1,606 "reply_to_post_number": 9,607 "quote_count": 0,608 "incoming_link_count": 18,609 "reads": 224,610 "readers_count": 223,611 "score": 139.8,612 "yours": false,613 "topic_id": 52192,614 "topic_slug": "can-i-deepcopy-a-model",615 "display_username": "",616 "primary_group_name": null,617 "flair_name": null,618 "flair_url": null,619 "flair_bg_color": null,620 "flair_color": null,621 "flair_group_id": null,622 "badges_granted": [],623 "version": 1,624 "can_edit": false,625 "can_delete": false,626 "can_recover": false,627 "can_see_hidden_post": false,628 "can_wiki": false,629 "link_counts": [630 {631 "url": "https://stackoverflow.com/a/2790884",632 "internal": false,633 "reflection": false,634 "clicks": 34635 }636 ],637 "read": true,638 "user_title": "",639 "reply_to_user": {640 "id": 43262,641 "username": "ynjiun_wang",642 "name": "ynjiun",643 "avatar_template": "/user_avatar/discuss.pytorch.org/ynjiun_wang/{size}/32014_2.png"644 },645 "bookmarked": false,646 "actions_summary": [],647 "moderator": true,648 "admin": true,649 "staff": true,650 "user_id": 3534,651 "hidden": false,652 "trust_level": 2,653 "deleted_at": null,654 "user_deleted": false,655 "edit_reason": null,656 "can_view_edit_history": true,657 "wiki": false,658 "post_url": "/t/can-i-deepcopy-a-model/52192/10",659 "can_accept_answer": false,660 "can_unaccept_answer": false,661 "accepted_answer": false,662 "topic_accepted_answer": true663 },664 {665 "id": 271517,666 "name": "ynjiun",667 "username": "ynjiun_wang",668 "avatar_template": "/user_avatar/discuss.pytorch.org/ynjiun_wang/{size}/32014_2.png",669 "created_at": "2021-03-19T15:39:11.350Z",670 "cooked": "<p>Thanks for your help. I looked into the create_sqeezenet_ssd_lite() code below, do you think that ModuleList() causing the problem? If yes, how do I fix this type of issue and enable the model created by this code deepcopyable? This type of model creation is typically used in a SSD network creation. Does this imply QAT doesn’t work for SSD type of network? Thanks again for your help!</p>\n<blockquote>\n<p>def create_squeezenet_ssd_lite(num_classes, is_test=False):<br>\nbase_net = squeezenet1_1(False).features # disable dropout layer</p>\n<pre><code>source_layer_indexes = [\n 12\n]\nextras = ModuleList([\n Sequential(\n Conv2d(in_channels=512, out_channels=256, kernel_size=1),\n ReLU(),\n SeperableConv2d(in_channels=256, out_channels=512, kernel_size=3, stride=2, padding=2),\n ),\n Sequential(\n Conv2d(in_channels=512, out_channels=256, kernel_size=1),\n ReLU(),\n SeperableConv2d(in_channels=256, out_channels=512, kernel_size=3, stride=2, padding=1),\n ),\n Sequential(\n Conv2d(in_channels=512, out_channels=128, kernel_size=1),\n ReLU(),\n SeperableConv2d(in_channels=128, out_channels=256, kernel_size=3, stride=2, padding=1),\n ),\n Sequential(\n Conv2d(in_channels=256, out_channels=128, kernel_size=1),\n ReLU(),\n SeperableConv2d(in_channels=128, out_channels=256, kernel_size=3, stride=2, padding=1),\n ),\n Sequential(\n Conv2d(in_channels=256, out_channels=128, kernel_size=1),\n ReLU(),\n SeperableConv2d(in_channels=128, out_channels=256, kernel_size=3, stride=2, padding=1)\n )\n])\n\nregression_headers = ModuleList([\n SeperableConv2d(in_channels=512, out_channels=6 * 4, kernel_size=3, padding=1),\n SeperableConv2d(in_channels=512, out_channels=6 * 4, kernel_size=3, padding=1),\n SeperableConv2d(in_channels=512, out_channels=6 * 4, kernel_size=3, padding=1),\n SeperableConv2d(in_channels=256, out_channels=6 * 4, kernel_size=3, padding=1),\n SeperableConv2d(in_channels=256, out_channels=6 * 4, kernel_size=3, padding=1),\n Conv2d(in_channels=256, out_channels=6 * 4, kernel_size=1),\n])\n\nclassification_headers = ModuleList([\n SeperableConv2d(in_channels=512, out_channels=6 * num_classes, kernel_size=3, padding=1),\n SeperableConv2d(in_channels=512, out_channels=6 * num_classes, kernel_size=3, padding=1),\n SeperableConv2d(in_channels=512, out_channels=6 * num_classes, kernel_size=3, padding=1),\n SeperableConv2d(in_channels=256, out_channels=6 * num_classes, kernel_size=3, padding=1),\n SeperableConv2d(in_channels=256, out_channels=6 * num_classes, kernel_size=3, padding=1),\n Conv2d(in_channels=256, out_channels=6 * num_classes, kernel_size=1),\n])\n\nreturn SSD(num_classes, base_net, source_layer_indexes,\n extras, classification_headers, regression_headers, is_test=is_test, config=config)\n</code></pre>\n</blockquote>",671 "post_number": 11,672 "post_type": 1,673 "posts_count": 21,674 "updated_at": "2021-03-19T15:56:05.134Z",675 "reply_count": 1,676 "reply_to_post_number": 10,677 "quote_count": 0,678 "incoming_link_count": 46,679 "reads": 229,680 "readers_count": 228,681 "score": 280.8,682 "yours": false,683 "topic_id": 52192,684 "topic_slug": "can-i-deepcopy-a-model",685 "display_username": "ynjiun",686 "primary_group_name": null,687 "flair_name": null,688 "flair_url": null,689 "flair_bg_color": null,690 "flair_color": null,691 "flair_group_id": null,692 "badges_granted": [],693 "version": 2,694 "can_edit": false,695 "can_delete": false,696 "can_recover": false,697 "can_see_hidden_post": false,698 "can_wiki": false,699 "read": true,700 "user_title": null,701 "reply_to_user": {702 "id": 3534,703 "username": "ptrblck",704 "name": "",705 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png"706 },707 "bookmarked": false,708 "actions_summary": [],709 "moderator": false,710 "admin": false,711 "staff": false,712 "user_id": 43262,713 "hidden": false,714 "trust_level": 2,715 "deleted_at": null,716 "user_deleted": false,717 "edit_reason": null,718 "can_view_edit_history": true,719 "wiki": false,720 "post_url": "/t/can-i-deepcopy-a-model/52192/11",721 "can_accept_answer": false,722 "can_unaccept_answer": false,723 "accepted_answer": false,724 "topic_accepted_answer": true725 },726 {727 "id": 271594,728 "name": "",729 "username": "ptrblck",730 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",731 "created_at": "2021-03-20T04:52:38.614Z",732 "cooked": "<p>The <code>nn.ModuleList</code> looks correct, but I also don’t know if QAT might be causing this issue. Are you seeing the same error without using QAT?</p>",733 "post_number": 12,734 "post_type": 1,735 "posts_count": 21,736 "updated_at": "2021-03-20T04:52:38.614Z",737 "reply_count": 1,738 "reply_to_post_number": 11,739 "quote_count": 0,740 "incoming_link_count": 120,741 "reads": 179,742 "readers_count": 178,743 "score": 640.8,744 "yours": false,745 "topic_id": 52192,746 "topic_slug": "can-i-deepcopy-a-model",747 "display_username": "",748 "primary_group_name": null,749 "flair_name": null,750 "flair_url": null,751 "flair_bg_color": null,752 "flair_color": null,753 "flair_group_id": null,754 "badges_granted": [],755 "version": 1,756 "can_edit": false,757 "can_delete": false,758 "can_recover": false,759 "can_see_hidden_post": false,760 "can_wiki": false,761 "read": true,762 "user_title": "",763 "reply_to_user": {764 "id": 43262,765 "username": "ynjiun_wang",766 "name": "ynjiun",767 "avatar_template": "/user_avatar/discuss.pytorch.org/ynjiun_wang/{size}/32014_2.png"768 },769 "bookmarked": false,770 "actions_summary": [],771 "moderator": true,772 "admin": true,773 "staff": true,774 "user_id": 3534,775 "hidden": false,776 "trust_level": 2,777 "deleted_at": null,778 "user_deleted": false,779 "edit_reason": null,780 "can_view_edit_history": true,781 "wiki": false,782 "post_url": "/t/can-i-deepcopy-a-model/52192/12",783 "can_accept_answer": false,784 "can_unaccept_answer": false,785 "accepted_answer": false,786 "topic_accepted_answer": true787 },788 {789 "id": 271665,790 "name": "ynjiun",791 "username": "ynjiun_wang",792 "avatar_template": "/user_avatar/discuss.pytorch.org/ynjiun_wang/{size}/32014_2.png",793 "created_at": "2021-03-20T16:08:41.551Z",794 "cooked": "<p>“Are you seeing the same error without using QAT?”</p>\n<p>Yes.</p>\n<p>I am seeing this link indicating that <a href=\"https://github.com/pytorch/pytorch/issues/7545\" rel=\"noopener nofollow ugc\">Can’t save a model with <code>torch.save</code> if model has a <code>torch.Device</code> attr #7545</a> and I tried to remove the self.device attribute in SSD class and still not working.</p>\n<p>I did try to run the code you suggested in the <a href=\"https://stackoverflow.com/questions/2790828/python-cant-pickle-module-objects-error/2790884#2790884\" rel=\"noopener nofollow ugc\">link</a> to exam where is the error by pickle_trick() it:</p>\n<blockquote>\n<p>net = create_squeezenet_ssd_lite(no_classes, is_test=True)<br>\nprint(pf(pickle_trick(net)))</p>\n</blockquote>\n<p>But the code crashes indicating there is exception caught within the exception even I increase the maximum depth to 10,000.</p>",795 "post_number": 13,796 "post_type": 1,797 "posts_count": 21,798 "updated_at": "2021-03-20T16:09:39.546Z",799 "reply_count": 1,800 "reply_to_post_number": 12,801 "quote_count": 0,802 "incoming_link_count": 30,803 "reads": 176,804 "readers_count": 175,805 "score": 190.2,806 "yours": false,807 "topic_id": 52192,808 "topic_slug": "can-i-deepcopy-a-model",809 "display_username": "ynjiun",810 "primary_group_name": null,811 "flair_name": null,812 "flair_url": null,813 "flair_bg_color": null,814 "flair_color": null,815 "flair_group_id": null,816 "badges_granted": [],817 "version": 1,818 "can_edit": false,819 "can_delete": false,820 "can_recover": false,821 "can_see_hidden_post": false,822 "can_wiki": false,823 "link_counts": [824 {825 "url": "https://github.com/pytorch/pytorch/issues/7545",826 "internal": false,827 "reflection": false,828 "title": "Can't save a model with `torch.save` if model has a `torch.Device` attr · Issue #7545 · pytorch/pytorch · GitHub",829 "clicks": 7830 },831 {832 "url": "https://stackoverflow.com/questions/2790828/python-cant-pickle-module-objects-error/2790884#2790884",833 "internal": false,834 "reflection": false,835 "title": "Python: can't pickle module objects error - Stack Overflow",836 "clicks": 0837 }838 ],839 "read": true,840 "user_title": null,841 "reply_to_user": {842 "id": 3534,843 "username": "ptrblck",844 "name": "",845 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png"846 },847 "bookmarked": false,848 "actions_summary": [],849 "moderator": false,850 "admin": false,851 "staff": false,852 "user_id": 43262,853 "hidden": false,854 "trust_level": 2,855 "deleted_at": null,856 "user_deleted": false,857 "edit_reason": null,858 "can_view_edit_history": true,859 "wiki": false,860 "post_url": "/t/can-i-deepcopy-a-model/52192/13",861 "can_accept_answer": false,862 "can_unaccept_answer": false,863 "accepted_answer": false,864 "topic_accepted_answer": true865 },866 {867 "id": 272259,868 "name": "ynjiun",869 "username": "ynjiun_wang",870 "avatar_template": "/user_avatar/discuss.pytorch.org/ynjiun_wang/{size}/32014_2.png",871 "created_at": "2021-03-24T03:56:08.730Z",872 "cooked": "<p><a class=\"mention\" href=\"/u/ptrblck\">@ptrblck</a></p>\n<p>after line by line elimination test, finally found out the root causes that make the SSD() not deepcopyable:</p>\n<blockquote>\n<p>class SSD(nn.Module):<br>\ndef <strong>init</strong>(self, num_classes: int, base_net: nn.ModuleList, source_layer_indexes: List[int],<br>\nextras: nn.ModuleList, classification_headers: nn.ModuleList,<br>\nregression_headers: nn.ModuleList, is_test=False, config=None, device=None):<br>\n“”“Compose a SSD model using the given components.<br>\n“””<br>\nsuper(SSD, self).<strong>init</strong>()</p>\n<pre><code> self.num_classes = num_classes\n self.base_net = base_net\n self.source_layer_indexes = source_layer_indexes\n self.extras = extras\n self.classification_headers = classification_headers\n self.regression_headers = regression_headers\n self.is_test = is_test\n self.config = config \n</code></pre>\n</blockquote>\n<p>If I comment out the “self.config = config” line, then SSD() is deepcopyable.</p>\n<p>Where the config is a module imported from:</p>\n<blockquote>\n<p>from .config import squeezenet_ssd_config as config</p>\n</blockquote>\n<p>And the squeezenet_ssd_config is as below:</p>\n<blockquote>\n<p>import numpy as np</p>\n<p>from vision.utils.box_utils import SSDSpec, SSDBoxSizes, generate_ssd_priors</p>\n<p>image_size = 300<br>\nimage_mean = np.array([127, 127, 127]) # RGB layout<br>\nimage_std = 128.0<br>\niou_threshold = 0.45<br>\ncenter_variance = 0.1<br>\nsize_variance = 0.2</p>\n<p>specs = [<br>\nSSDSpec(17, 16, SSDBoxSizes(60, 105), [2, 3]),<br>\nSSDSpec(10, 32, SSDBoxSizes(105, 150), [2, 3]),<br>\nSSDSpec(5, 64, SSDBoxSizes(150, 195), [2, 3]),<br>\nSSDSpec(3, 100, SSDBoxSizes(195, 240), [2, 3]),<br>\nSSDSpec(2, 150, SSDBoxSizes(240, 285), [2, 3]),<br>\nSSDSpec(1, 300, SSDBoxSizes(285, 330), [2, 3])<br>\n]</p>\n<p>priors = generate_ssd_priors(specs, image_size)</p>\n</blockquote>\n<p>How should I modify the code to keep the “config” and still make the SSD() deepcopyable? Any advice will be highly appreciated.</p>\n<p>Thanks a lot for your help in advanced.</p>",873 "post_number": 14,874 "post_type": 1,875 "posts_count": 21,876 "updated_at": "2021-03-24T03:56:08.730Z",877 "reply_count": 1,878 "reply_to_post_number": 13,879 "quote_count": 0,880 "incoming_link_count": 62,881 "reads": 173,882 "readers_count": 172,883 "score": 349.6,884 "yours": false,885 "topic_id": 52192,886 "topic_slug": "can-i-deepcopy-a-model",887 "display_username": "ynjiun",888 "primary_group_name": null,889 "flair_name": null,890 "flair_url": null,891 "flair_bg_color": null,892 "flair_color": null,893 "flair_group_id": null,894 "badges_granted": [],895 "version": 1,896 "can_edit": false,897 "can_delete": false,898 "can_recover": false,899 "can_see_hidden_post": false,900 "can_wiki": false,901 "read": true,902 "user_title": null,903 "reply_to_user": {904 "id": 43262,905 "username": "ynjiun_wang",906 "name": "ynjiun",907 "avatar_template": "/user_avatar/discuss.pytorch.org/ynjiun_wang/{size}/32014_2.png"908 },909 "bookmarked": false,910 "actions_summary": [],911 "moderator": false,912 "admin": false,913 "staff": false,914 "user_id": 43262,915 "hidden": false,916 "trust_level": 2,917 "deleted_at": null,918 "user_deleted": false,919 "edit_reason": null,920 "can_view_edit_history": true,921 "wiki": false,922 "post_url": "/t/can-i-deepcopy-a-model/52192/14",923 "can_accept_answer": false,924 "can_unaccept_answer": false,925 "accepted_answer": false,926 "topic_accepted_answer": true927 },928 {929 "id": 272272,930 "name": "",931 "username": "ptrblck",932 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",933 "created_at": "2021-03-24T05:13:52.283Z",934 "cooked": "<p>Is <code>squeezenet_ssd_config</code> only containing the posted code or any class definitions?<br>\nBased on the <code>import</code> statement I would assume it’s a class or any other object, but the posted code shows just executable Python code with more imports.<br>\nYou could try to <code>deepcopy</code> each imported class and see, if one of these classes might fail due to the previously mentioned reason.</p>",935 "post_number": 15,936 "post_type": 1,937 "posts_count": 21,938 "updated_at": "2021-03-24T05:13:52.283Z",939 "reply_count": 1,940 "reply_to_post_number": 14,941 "quote_count": 0,942 "incoming_link_count": 12,943 "reads": 123,944 "readers_count": 122,945 "score": 89.6,946 "yours": false,947 "topic_id": 52192,948 "topic_slug": "can-i-deepcopy-a-model",949 "display_username": "",950 "primary_group_name": null,951 "flair_name": null,952 "flair_url": null,953 "flair_bg_color": null,954 "flair_color": null,955 "flair_group_id": null,956 "badges_granted": [],957 "version": 1,958 "can_edit": false,959 "can_delete": false,960 "can_recover": false,961 "can_see_hidden_post": false,962 "can_wiki": false,963 "read": true,964 "user_title": "",965 "reply_to_user": {966 "id": 43262,967 "username": "ynjiun_wang",968 "name": "ynjiun",969 "avatar_template": "/user_avatar/discuss.pytorch.org/ynjiun_wang/{size}/32014_2.png"970 },971 "bookmarked": false,972 "actions_summary": [],973 "moderator": true,974 "admin": true,975 "staff": true,976 "user_id": 3534,977 "hidden": false,978 "trust_level": 2,979 "deleted_at": null,980 "user_deleted": false,981 "edit_reason": null,982 "can_view_edit_history": true,983 "wiki": false,984 "post_url": "/t/can-i-deepcopy-a-model/52192/15",985 "can_accept_answer": false,986 "can_unaccept_answer": false,987 "accepted_answer": false,988 "topic_accepted_answer": true989 },990 {991 "id": 272508,992 "name": "ynjiun",993 "username": "ynjiun_wang",994 "avatar_template": "/user_avatar/discuss.pytorch.org/ynjiun_wang/{size}/32014_2.png",995 "created_at": "2021-03-24T17:48:01.788Z",996 "cooked": "<p><a class=\"mention\" href=\"/u/ptrblck\">@ptrblck</a></p>\n<p>Thank you so much for your help.</p>\n<p>Eventually I “work around” the problem by doing the following:</p>\n<p>avoid “self.config=config” referencing, but directly instantiate it’s content:</p>\n<p>Was:</p>\n<blockquote>\n<p>in init<br>\nself.config = config<br>\n…<later in the body, self.config were referenced as><br>\nself.config.center_variance<br>\nself.config.size_variance</p>\n</blockquote>\n<p>Change to:</p>\n<blockquote>\n<p>in init<br>\n<span class=\"hashtag-raw\">#self</span>.config = config <span class=\"hashtag-raw\">#comment</span> out<br>\nself.config_center_variance = config.center_variance<br>\nself.config_size_variance = config.size_variance<br>\n…<later in the code, no more reference to self.config ><br>\nself.config_center_variance<br>\nself.config_size_variance</p>\n</blockquote>\n<p>Then the SSD() is deepcopyable, but QAT still having problems…</p>",997 "post_number": 16,998 "post_type": 1,999 "posts_count": 21,1000 "updated_at": "2021-03-24T23:21:20.614Z",1001 "reply_count": 1,1002 "reply_to_post_number": 15,1003 "quote_count": 0,1004 "incoming_link_count": 27,1005 "reads": 127,1006 "readers_count": 126,1007 "score": 165.4,1008 "yours": false,1009 "topic_id": 52192,1010 "topic_slug": "can-i-deepcopy-a-model",1011 "display_username": "ynjiun",1012 "primary_group_name": null,1013 "flair_name": null,1014 "flair_url": null,1015 "flair_bg_color": null,1016 "flair_color": null,1017 "flair_group_id": null,1018 "badges_granted": [],1019 "version": 4,1020 "can_edit": false,1021 "can_delete": false,1022 "can_recover": false,1023 "can_see_hidden_post": false,1024 "can_wiki": false,1025 "read": true,1026 "user_title": null,1027 "reply_to_user": {1028 "id": 3534,1029 "username": "ptrblck",1030 "name": "",1031 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png"1032 },1033 "bookmarked": false,1034 "actions_summary": [],1035 "moderator": false,1036 "admin": false,1037 "staff": false,1038 "user_id": 43262,1039 "hidden": false,1040 "trust_level": 2,1041 "deleted_at": null,1042 "user_deleted": false,1043 "edit_reason": null,1044 "can_view_edit_history": true,1045 "wiki": false,1046 "post_url": "/t/can-i-deepcopy-a-model/52192/16",1047 "can_accept_answer": false,1048 "can_unaccept_answer": false,1049 "accepted_answer": false,1050 "topic_accepted_answer": true1051 },1052 {1053 "id": 272565,1054 "name": "ynjiun",1055 "username": "ynjiun_wang",1056 "avatar_template": "/user_avatar/discuss.pytorch.org/ynjiun_wang/{size}/32014_2.png",1057 "created_at": "2021-03-24T23:22:12.856Z",1058 "cooked": "<p><a class=\"mention\" href=\"/u/ptrblck\">@ptrblck</a></p>\n<p>although SSD() is deepcopyable, but still having problem to run QAT…, the problem is: cannot torch.save(net_int8) and it complaints:</p>\n<blockquote>\n<pre><code>torch.save(net_int8, model_path) #to save the entire model\n</code></pre>\n<p>File “/home/paul/pytorch/lib/python3.6/site-packages/torch/serialization.py”, line 370, in save<br>\n_legacy_save(obj, opened_file, pickle_module, pickle_protocol)<br>\nFile “/home/paul/pytorch/lib/python3.6/site-packages/torch/serialization.py”, line 443, in _legacy_save<br>\npickler.dump(obj)<br>\nAttributeError: Can’t pickle local object ‘_with_args.._PartialWrapper’</p>\n</blockquote>\n<p>the code with problem area is:</p>\n<blockquote>\n<pre><code> net.eval()\n net_int8 = torch.quantization.convert(net)\n \n #torch.save(net_int8.state_dict(), model_path)\n torch.save(net_int8, model_path) #to save the entire model\n</code></pre>\n</blockquote>\n<p>If I only save state_dict(), there is no problem. But when save the entire model, the problem show.</p>\n<p>Further debug, finding the net was loaded and torch.save-able, but after the net.qconfig and it becomes not torch.save-able anymore:</p>\n<blockquote>\n<pre><code>torch.save(net,\"./net_before_qconfig\")\nnet.qconfig = torch.quantization.get_default_qat_qconfig('fbgemm')\ntorch.save(net,\"./net_after_qconfig\")\n</code></pre>\n</blockquote>\n<p>As shown above, the torch.save(net, “./net_before_qconfig”) works, but after net.qconfig, the net is not torch.save-able anymore… ;((</p>\n<p>And having the same complaints:</p>\n<blockquote>\n<pre><code>torch.save(net,\"./net_after_qconfig\")\n</code></pre>\n<p>File “/home/paul/pytorch/lib/python3.6/site-packages/torch/serialization.py”, line 370, in save<br>\n_legacy_save(obj, opened_file, pickle_module, pickle_protocol)<br>\nFile “/home/paul/pytorch/lib/python3.6/site-packages/torch/serialization.py”, line 443, in _legacy_save<br>\npickler.dump(obj)<br>\nAttributeError: Can’t pickle local object ‘_with_args.._PartialWrapper’</p>\n</blockquote>\n<p>Is this QAT problem? torch.save cannot save the entire model after QAT? I need the entire model saved to port to a micro processor for speed up testing. If this is QAT limitation, what would be the work around to still save the entire quantization model (not just state_dict)? Thank you again for your help!</p>",1059 "post_number": 17,1060 "post_type": 1,1061 "posts_count": 21,1062 "updated_at": "2021-03-24T23:22:12.856Z",1063 "reply_count": 1,1064 "reply_to_post_number": 16,1065 "quote_count": 0,1066 "incoming_link_count": 43,1067 "reads": 125,1068 "readers_count": 124,1069 "score": 240.0,1070 "yours": false,1071 "topic_id": 52192,1072 "topic_slug": "can-i-deepcopy-a-model",1073 "display_username": "ynjiun",1074 "primary_group_name": null,1075 "flair_name": null,1076 "flair_url": null,1077 "flair_bg_color": null,1078 "flair_color": null,1079 "flair_group_id": null,1080 "badges_granted": [],1081 "version": 1,1082 "can_edit": false,1083 "can_delete": false,1084 "can_recover": false,1085 "can_see_hidden_post": false,1086 "can_wiki": false,1087 "read": true,1088 "user_title": null,1089 "reply_to_user": {1090 "id": 43262,1091 "username": "ynjiun_wang",1092 "name": "ynjiun",1093 "avatar_template": "/user_avatar/discuss.pytorch.org/ynjiun_wang/{size}/32014_2.png"1094 },1095 "bookmarked": false,1096 "actions_summary": [],1097 "moderator": false,1098 "admin": false,1099 "staff": false,1100 "user_id": 43262,1101 "hidden": false,1102 "trust_level": 2,1103 "deleted_at": null,1104 "user_deleted": false,1105 "edit_reason": null,1106 "can_view_edit_history": true,1107 "wiki": false,1108 "post_url": "/t/can-i-deepcopy-a-model/52192/17",1109 "can_accept_answer": false,1110 "can_unaccept_answer": false,1111 "accepted_answer": false,1112 "topic_accepted_answer": true1113 },1114 {1115 "id": 272580,1116 "name": "",1117 "username": "ptrblck",1118 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",1119 "created_at": "2021-03-25T00:52:07.459Z",1120 "cooked": "<p>Based on your latest debugging, the issue might be related to QAT.<br>\nI’m not deeply familiar with QAT and the support to use <code>torch.save</code> on the model directly.<br>\nHowever, generally I would not recommend to save the model directly, as it can break in various ways.<br>\nThe better way would be to save the <code>state_dict</code> (and additional configs etc.). When reloading you would then recreate the model object and use <code>load_state_dict</code>.</p>",1121 "post_number": 18,1122 "post_type": 1,1123 "posts_count": 21,1124 "updated_at": "2021-03-25T00:52:07.459Z",1125 "reply_count": 0,1126 "reply_to_post_number": 17,1127 "quote_count": 0,1128 "incoming_link_count": 16,1129 "reads": 109,1130 "readers_count": 108,1131 "score": 101.8,1132 "yours": false,1133 "topic_id": 52192,1134 "topic_slug": "can-i-deepcopy-a-model",1135 "display_username": "",1136 "primary_group_name": null,1137 "flair_name": null,1138 "flair_url": null,1139 "flair_bg_color": null,1140 "flair_color": null,1141 "flair_group_id": null,1142 "badges_granted": [],1143 "version": 1,1144 "can_edit": false,1145 "can_delete": false,1146 "can_recover": false,1147 "can_see_hidden_post": false,1148 "can_wiki": false,1149 "read": true,1150 "user_title": "",1151 "reply_to_user": {1152 "id": 43262,1153 "username": "ynjiun_wang",1154 "name": "ynjiun",1155 "avatar_template": "/user_avatar/discuss.pytorch.org/ynjiun_wang/{size}/32014_2.png"1156 },1157 "bookmarked": false,1158 "actions_summary": [],1159 "moderator": true,1160 "admin": true,1161 "staff": true,1162 "user_id": 3534,1163 "hidden": false,1164 "trust_level": 2,1165 "deleted_at": null,1166 "user_deleted": false,1167 "edit_reason": null,1168 "can_view_edit_history": true,1169 "wiki": false,1170 "post_url": "/t/can-i-deepcopy-a-model/52192/18",1171 "can_accept_answer": false,1172 "can_unaccept_answer": false,1173 "accepted_answer": false,1174 "topic_accepted_answer": true1175 },1176 {1177 "id": 290866,1178 "name": "",1179 "username": "n40x1",1180 "avatar_template": "/letter_avatar_proxy/v4/letter/n/c37758/{size}.png",1181 "created_at": "2021-06-17T23:14:29.523Z",1182 "cooked": "<p>I using Pytorch’s <code>swa_utils</code> that internally calls <code>deepcopy</code>.<br>\nHowever, I get the following error:</p>\n<pre><code class=\"lang-auto\">> Only Tensors created explicitly by the user (graph leaves) \nsupport the deepcopy protocol at the moment\n</code></pre>\n<p>This seems to be caused by the weight dropout scheme I am using, where the original weight matrix is moved into _raw. The code for weight dropout is here:</p>\n<pre><code class=\"lang-auto\">import torch\nfrom torch.nn import Parameter\nimport torch.nn.functional as F \n\nclass WeightDrop(object):\n def __init__(self, name, dropout):\n self.name = name\n self.dropout = dropout\n\n def compute_weight(self, module):\n return F.dropout(\n getattr(module, self.name + \"_raw\"),\n p = self.dropout,\n training = module.training,\n inplace = False\n )\n\n @staticmethod\n def apply(module, name, dropout):\n for k, hook in module._forward_pre_hooks.items():\n if isinstance(hook, WeightDrop) and hook.name == name:\n raise RuntimeError(f\"Cannot register two weight_dropout hooks with name '{name}'\")\n fn = WeightDrop(name, dropout)\n weight = getattr(module, name)\n\n del module._parameters[name]\n\n ## creating _raw parameter\n module.register_parameter(name + \"_raw\", Parameter(weight.data))\n setattr(module, name, fn.compute_weight(module))\n\n module.register_forward_pre_hook(fn)\n return fn\n\n def remove(self, module):\n weight = module._parameters[self.name + \"_raw\"]\n delattr(module, self.name)\n del module._parameters[self.name + \"_raw\"]\n module.register_parameter(self.name, Parameter(weight.data))\n\n def __call__(self, module, inputs):\n if self.name in module._parameters:\n del module._parameters[self.name]\n setattr(module, self.name, self.compute_weight(module))\n\n\ndef weight_drop(module, name, dropout):\n WeightDrop.apply(module, name, dropout)\n return module\n\ndef apply_weight_drop(module, name, dropout):\n wdrop = weight_drop(module, name, dropout)\n fp = module.flatten_parameters\n\n def decorator(*args, **kwargs):\n device = getattr(module, name + \"_raw\").device\n setattr(module, name, getattr(module, name).to(device))\n return fp(*args, **kwargs)\n\n module.flatten_parameters = decorator\n return wdrop\n</code></pre>\n<p>Here is the code for applying deep copy on a GRU:</p>\n<pre><code class=\"lang-auto\">import copy\ngru = torch.nn.GRU(10, 10)\ngru_wd = apply_weight_drop(gru, \"weight_hh_l0\", 0.2)\ngru_wd_copy = copy.deepcopy(gru_wd)\n> RuntimeError: Only Tensors created explicitly by the user (graph leaves) support the deepcopy protocol at the moment\n</code></pre>\n<p>Any ideas how to fix this error?<br>\nThank you!</p>",1183 "post_number": 19,1184 "post_type": 1,1185 "posts_count": 21,1186 "updated_at": "2021-06-17T23:14:29.523Z",1187 "reply_count": 1,1188 "reply_to_post_number": null,1189 "quote_count": 0,1190 "incoming_link_count": 103,1191 "reads": 98,1192 "readers_count": 97,1193 "score": 554.6,1194 "yours": false,1195 "topic_id": 52192,1196 "topic_slug": "can-i-deepcopy-a-model",1197 "display_username": "",1198 "primary_group_name": null,1199 "flair_name": null,1200 "flair_url": null,