Anurag1734/cuda-error-resolution-analysis
07
1[2 {3 "post_stream": {4 "posts": [5 {6 "id": 440676,7 "name": "",8 "username": "nbusser",9 "avatar_template": "/letter_avatar_proxy/v4/letter/n/e8c25b/{size}.png",10 "created_at": "2024-04-27T03:55:46.812Z",11 "cooked": "<p>Hi everyone <img src=\"https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12\" title=\":slight_smile:\" class=\"emoji\" alt=\":slight_smile:\" loading=\"lazy\" width=\"20\" height=\"20\"></p>\n<p>I want to implement an early stopping mechanic to my DNN. Thus, I have to calculate the validation loss for each epoch.<br>\nI use <code>@torch.no_grad()</code> during the validation loss calculation to avoid any gradient computation that would alter the training.</p>\n<p>Here is my training loop:</p>\n<pre data-code-wrap=\"python\"><code class=\"lang-python\">def main():\n training_dataset = torch.load(\"dataset/training_dataset.pth\")\n\n net = dnn.Net()\n\n criterion = nn.CrossEntropyLoss()\n optimizer = optim.Adam(net.parameters(), lr=0.0005)\n\n early_stopping = EarlyStopping(tolerance=5, min_delta=10)\n validation_dataset = torch.load(\"dataset/validation_dataset.pth\")\n\n n_epochs = 150\n for epoch in tqdm.tqdm(range(n_epochs)):\n epoch_train_loss = train_epoch(training_dataset, net, criterion, optimizer)\n with torch.no_grad():\n epoch_validate_loss, _ = validation.validate(\n validation_dataset, net, criterion\n )\n</code></pre>\n<p>Here is the validation function:</p>\n<pre><code class=\"lang-auto\">@torch.no_grad()\ndef validate(\n validation_dataset: torch.utils.data.Dataset, net: nn.Module, criterion: nn.Module\n):\n net.eval()\n validation_loader = torch.utils.data.DataLoader(validation_dataset, batch_size=1)\n\n correct = 0.0\n loss = 0\n\n for inputs, target in validation_loader:\n output = net(inputs)\n loss = criterion(output, target)\n\n _, pred = output.max(1)\n correct += (pred == target).sum()\n\n accuracy = correct / len(validation_dataset) * 100.0\n return loss, accuracy\n</code></pre>\n<p><em>NB: I know that using both <code>@torch.no_grad</code> and <code>with torch.no_grad()</code> is overkill</em></p>\n<p>I run the training on 150 epochs.<br>\nAfter the training, I run calculate the accuracy on the validation set.</p>\n<p>Here is what I obtain when:</p>\n<ul>\n<li><strong>NOT</strong> calculating validation loss for each epoch: <strong>91.67%</strong></li>\n<li>Calculating validation loss for each epoch: <strong>97.22%</strong></li>\n</ul>\n<p>It then seems that calculating the validation loss for each epoch has definitely alters training by <strong>leaking validation data</strong>.</p>\n<p>Can you tell me what I am doing wrong here?</p>",12 "post_number": 1,13 "post_type": 1,14 "posts_count": 5,15 "updated_at": "2024-04-27T04:46:37.245Z",16 "reply_count": 1,17 "reply_to_post_number": null,18 "quote_count": 0,19 "incoming_link_count": 21,20 "reads": 5,21 "readers_count": 4,22 "score": 111.0,23 "yours": false,24 "topic_id": 201687,25 "topic_slug": "my-networks-weights-get-updated-despite-using-torch-no-grad",26 "display_username": "",27 "primary_group_name": null,28 "flair_name": null,29 "flair_url": null,30 "flair_bg_color": null,31 "flair_color": null,32 "flair_group_id": null,33 "badges_granted": [],34 "version": 2,35 "can_edit": false,36 "can_delete": false,37 "can_recover": false,38 "can_see_hidden_post": false,39 "can_wiki": false,40 "read": true,41 "user_title": null,42 "bookmarked": false,43 "actions_summary": [],44 "moderator": false,45 "admin": false,46 "staff": false,47 "user_id": 75360,48 "hidden": false,49 "trust_level": 1,50 "deleted_at": null,51 "user_deleted": false,52 "edit_reason": null,53 "can_view_edit_history": true,54 "wiki": false,55 "post_url": "/t/my-networks-weights-get-updated-despite-using-torch-no-grad/201687/1",56 "can_accept_answer": false,57 "can_unaccept_answer": false,58 "accepted_answer": false,59 "topic_accepted_answer": true,60 "can_vote": false61 },62 {63 "id": 440698,64 "name": "",65 "username": "ptrblck",66 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",67 "created_at": "2024-04-27T10:55:46.020Z",68 "cooked": "<aside class=\"quote no-group\" data-username=\"nbusser\" data-post=\"1\" data-topic=\"201687\">\n<div class=\"title\">\n<div class=\"quote-controls\"></div>\n<img loading=\"lazy\" alt=\"\" width=\"24\" height=\"24\" src=\"https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/n/e8c25b/48.png\" class=\"avatar\"> nbusser:</div>\n<blockquote>\n<p>by <strong>leaking validation data</strong></p>\n</blockquote>\n</aside>\n<p>Not necessarily, as you could also observe noise creating by additional calls to the pseudorandom number generator. You could check how sensitive your training is to the seed.</p>",69 "post_number": 2,70 "post_type": 1,71 "posts_count": 5,72 "updated_at": "2024-04-27T10:55:46.020Z",73 "reply_count": 1,74 "reply_to_post_number": null,75 "quote_count": 1,76 "incoming_link_count": 1,77 "reads": 4,78 "readers_count": 3,79 "score": 10.8,80 "yours": false,81 "topic_id": 201687,82 "topic_slug": "my-networks-weights-get-updated-despite-using-torch-no-grad",83 "display_username": "",84 "primary_group_name": null,85 "flair_name": null,86 "flair_url": null,87 "flair_bg_color": null,88 "flair_color": null,89 "flair_group_id": null,90 "badges_granted": [],91 "version": 1,92 "can_edit": false,93 "can_delete": false,94 "can_recover": false,95 "can_see_hidden_post": false,96 "can_wiki": false,97 "read": true,98 "user_title": "",99 "bookmarked": false,100 "actions_summary": [],101 "moderator": true,102 "admin": true,103 "staff": true,104 "user_id": 3534,105 "hidden": false,106 "trust_level": 2,107 "deleted_at": null,108 "user_deleted": false,109 "edit_reason": null,110 "can_view_edit_history": true,111 "wiki": false,112 "post_url": "/t/my-networks-weights-get-updated-despite-using-torch-no-grad/201687/2",113 "can_accept_answer": false,114 "can_unaccept_answer": false,115 "accepted_answer": true,116 "topic_accepted_answer": true117 },118 {119 "id": 440699,120 "name": "",121 "username": "nbusser",122 "avatar_template": "/letter_avatar_proxy/v4/letter/n/e8c25b/{size}.png",123 "created_at": "2024-04-27T11:30:05.706Z",124 "cooked": "<p>Oh sure I did not think of it! Socan I safely conclude that there is nothing wrong here?</p>",125 "post_number": 3,126 "post_type": 1,127 "posts_count": 5,128 "updated_at": "2024-04-27T11:30:05.706Z",129 "reply_count": 1,130 "reply_to_post_number": 2,131 "quote_count": 0,132 "incoming_link_count": 1,133 "reads": 4,134 "readers_count": 3,135 "score": 10.8,136 "yours": false,137 "topic_id": 201687,138 "topic_slug": "my-networks-weights-get-updated-despite-using-torch-no-grad",139 "display_username": "",140 "primary_group_name": null,141 "flair_name": null,142 "flair_url": null,143 "flair_bg_color": null,144 "flair_color": null,145 "flair_group_id": null,146 "badges_granted": [],147 "version": 1,148 "can_edit": false,149 "can_delete": false,150 "can_recover": false,151 "can_see_hidden_post": false,152 "can_wiki": false,153 "read": true,154 "user_title": null,155 "reply_to_user": {156 "id": 3534,157 "username": "ptrblck",158 "name": "",159 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png"160 },161 "bookmarked": false,162 "actions_summary": [],163 "moderator": false,164 "admin": false,165 "staff": false,166 "user_id": 75360,167 "hidden": false,168 "trust_level": 1,169 "deleted_at": null,170 "user_deleted": false,171 "edit_reason": null,172 "can_view_edit_history": true,173 "wiki": false,174 "post_url": "/t/my-networks-weights-get-updated-despite-using-torch-no-grad/201687/3",175 "can_accept_answer": false,176 "can_unaccept_answer": false,177 "accepted_answer": false,178 "topic_accepted_answer": true179 },180 {181 "id": 440721,182 "name": "",183 "username": "ptrblck",184 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",185 "created_at": "2024-04-27T16:37:56.735Z",186 "cooked": "<p>You could run additional tests using different seeds to estimate the <code>mean</code> and <code>stddev</code> of e.g. the accuracy or loss of your current training routine and compare it to the currently observed difference. You could also double check that weights are never updated explicitly by comparing (some) parameters before and after the validation run, just to make sure.</p>",187 "post_number": 4,188 "post_type": 1,189 "posts_count": 5,190 "updated_at": "2024-04-27T16:37:56.735Z",191 "reply_count": 1,192 "reply_to_post_number": 3,193 "quote_count": 0,194 "incoming_link_count": 0,195 "reads": 2,196 "readers_count": 1,197 "score": 5.4,198 "yours": false,199 "topic_id": 201687,200 "topic_slug": "my-networks-weights-get-updated-despite-using-torch-no-grad",201 "display_username": "",202 "primary_group_name": null,203 "flair_name": null,204 "flair_url": null,205 "flair_bg_color": null,206 "flair_color": null,207 "flair_group_id": null,208 "badges_granted": [],209 "version": 1,210 "can_edit": false,211 "can_delete": false,212 "can_recover": false,213 "can_see_hidden_post": false,214 "can_wiki": false,215 "read": true,216 "user_title": "",217 "reply_to_user": {218 "id": 75360,219 "username": "nbusser",220 "name": "",221 "avatar_template": "/letter_avatar_proxy/v4/letter/n/e8c25b/{size}.png"222 },223 "bookmarked": false,224 "actions_summary": [],225 "moderator": true,226 "admin": true,227 "staff": true,228 "user_id": 3534,229 "hidden": false,230 "trust_level": 2,231 "deleted_at": null,232 "user_deleted": false,233 "edit_reason": null,234 "can_view_edit_history": true,235 "wiki": false,236 "post_url": "/t/my-networks-weights-get-updated-despite-using-torch-no-grad/201687/4",237 "can_accept_answer": false,238 "can_unaccept_answer": false,239 "accepted_answer": false,240 "topic_accepted_answer": true241 },242 {243 "id": 440850,244 "name": "",245 "username": "nbusser",246 "avatar_template": "/letter_avatar_proxy/v4/letter/n/e8c25b/{size}.png",247 "created_at": "2024-04-29T02:38:29.566Z",248 "cooked": "<p>I’ve explicitely checked the model’s weights and they are not modified after the validation run. Thank you for your time and your answer!</p>",249 "post_number": 5,250 "post_type": 1,251 "posts_count": 5,252 "updated_at": "2024-04-29T02:38:29.566Z",253 "reply_count": 0,254 "reply_to_post_number": 4,255 "quote_count": 0,256 "incoming_link_count": 0,257 "reads": 2,258 "readers_count": 1,259 "score": 0.4,260 "yours": false,261 "topic_id": 201687,262 "topic_slug": "my-networks-weights-get-updated-despite-using-torch-no-grad",263 "display_username": "",264 "primary_group_name": null,265 "flair_name": null,266 "flair_url": null,267 "flair_bg_color": null,268 "flair_color": null,269 "flair_group_id": null,270 "badges_granted": [],271 "version": 1,272 "can_edit": false,273 "can_delete": false,274 "can_recover": false,275 "can_see_hidden_post": false,276 "can_wiki": false,277 "read": true,278 "user_title": null,279 "reply_to_user": {280 "id": 3534,281 "username": "ptrblck",282 "name": "",283 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png"284 },285 "bookmarked": false,286 "actions_summary": [],287 "moderator": false,288 "admin": false,289 "staff": false,290 "user_id": 75360,291 "hidden": false,292 "trust_level": 1,293 "deleted_at": null,294 "user_deleted": false,295 "edit_reason": null,296 "can_view_edit_history": true,297 "wiki": false,298 "post_url": "/t/my-networks-weights-get-updated-despite-using-torch-no-grad/201687/5",299 "can_accept_answer": false,300 "can_unaccept_answer": false,301 "accepted_answer": false,302 "topic_accepted_answer": true303 }304 ],305 "stream": [306 440676,307 440698,308 440699,309 440721,310 440850311 ]312 },313 "timeline_lookup": [314 [315 1,316 547317 ],318 [319 2,320 546321 ],322 [323 5,324 545325 ]326 ],327 "suggested_topics": [328 {329 "fancy_title": "activation memory",330 "id": 213468,331 "title": "activation memory",332 "slug": "activation-memory",333 "posts_count": 2,334 "reply_count": 0,335 "highest_post_number": 2,336 "image_url": null,337 "created_at": "2024-11-26T13:48:42.606Z",338 "last_posted_at": "2024-12-12T16:08:06.342Z",339 "bumped": true,340 "bumped_at": "2024-12-12T16:08:06.342Z",341 "archetype": "regular",342 "unseen": false,343 "pinned": false,344 "unpinned": null,345 "visible": true,346 "closed": false,347 "archived": false,348 "bookmarked": null,349 "liked": null,350 "tags_descriptions": {},351 "like_count": 0,352 "views": 171,353 "category_id": 7,354 "featured_link": null,355 "has_accepted_answer": false,356 "posters": [357 {358 "extras": null,359 "description": "Original Poster",360 "user": {361 "id": 81127,362 "username": "amber",363 "name": "",364 "avatar_template": "/letter_avatar_proxy/v4/letter/a/9fc348/{size}.png",365 "trust_level": 0366 }367 },368 {369 "extras": "latest",370 "description": "Most Recent Poster",371 "user": {372 "id": 41396,373 "username": "soulitzer",374 "name": "",375 "avatar_template": "/letter_avatar_proxy/v4/letter/s/839c29/{size}.png",376 "trust_level": 2377 }378 }379 ]380 },381 {382 "fancy_title": "Autograd for duplicated index put operation",383 "id": 214251,384 "title": "Autograd for duplicated index put operation",385 "slug": "autograd-for-duplicated-index-put-operation",386 "posts_count": 2,387 "reply_count": 0,388 "highest_post_number": 2,389 "image_url": null,390 "created_at": "2024-12-16T02:42:21.311Z",391 "last_posted_at": "2024-12-16T16:13:04.741Z",392 "bumped": true,393 "bumped_at": "2024-12-16T16:13:04.741Z",394 "archetype": "regular",395 "unseen": false,396 "pinned": false,397 "unpinned": null,398 "visible": true,399 "closed": false,400 "archived": false,401 "bookmarked": null,402 "liked": null,403 "tags_descriptions": {},404 "like_count": 0,405 "views": 170,406 "category_id": 7,407 "featured_link": null,408 "has_accepted_answer": false,409 "posters": [410 {411 "extras": null,412 "description": "Original Poster",413 "user": {414 "id": 81516,415 "username": "skpighhh",416 "name": "",417 "avatar_template": "/user_avatar/discuss.pytorch.org/skpighhh/{size}/74528_2.png",418 "trust_level": 0419 }420 },421 {422 "extras": "latest",423 "description": "Most Recent Poster",424 "user": {425 "id": 3534,426 "username": "ptrblck",427 "name": "",428 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",429 "admin": true,430 "moderator": true,431 "trust_level": 2432 }433 }434 ]435 },436 {437 "fancy_title": "Help with Projecting Gradients onto a Hypersphere’s Tangent Plane",438 "id": 214694,439 "title": "Help with Projecting Gradients onto a Hypersphere's Tangent Plane",440 "slug": "help-with-projecting-gradients-onto-a-hyperspheres-tangent-plane",441 "posts_count": 5,442 "reply_count": 2,443 "highest_post_number": 5,444 "image_url": null,445 "created_at": "2024-12-27T09:17:22.976Z",446 "last_posted_at": "2025-01-01T18:17:14.906Z",447 "bumped": true,448 "bumped_at": "2025-01-01T18:17:14.906Z",449 "archetype": "regular",450 "unseen": false,451 "pinned": false,452 "unpinned": null,453 "visible": true,454 "closed": false,455 "archived": false,456 "bookmarked": null,457 "liked": null,458 "tags_descriptions": {},459 "like_count": 2,460 "views": 74,461 "category_id": 7,462 "featured_link": null,463 "has_accepted_answer": true,464 "posters": [465 {466 "extras": "latest",467 "description": "Original Poster, Most Recent Poster",468 "user": {469 "id": 81742,470 "username": "jumdc",471 "name": "",472 "avatar_template": "/letter_avatar_proxy/v4/letter/j/82dd89/{size}.png",473 "trust_level": 1474 }475 },476 {477 "extras": null,478 "description": "Frequent Poster, Accepted Answer",479 "user": {480 "id": 18088,481 "username": "KFrank",482 "name": "K. Frank",483 "avatar_template": "/letter_avatar_proxy/v4/letter/k/ecb155/{size}.png",484 "trust_level": 2485 }486 },487 {488 "extras": null,489 "description": "Frequent Poster",490 "user": {491 "id": 41396,492 "username": "soulitzer",493 "name": "",494 "avatar_template": "/letter_avatar_proxy/v4/letter/s/839c29/{size}.png",495 "trust_level": 2496 }497 }498 ]499 },500 {501 "fancy_title": "Torch.autograd.grad and masking issue",502 "id": 215388,503 "title": "Torch.autograd.grad and masking issue",504 "slug": "torch-autograd-grad-and-masking-issue",505 "posts_count": 1,506 "reply_count": 0,507 "highest_post_number": 1,508 "image_url": null,509 "created_at": "2025-01-14T19:07:47.768Z",510 "last_posted_at": "2025-01-14T19:07:47.810Z",511 "bumped": true,512 "bumped_at": "2025-01-14T19:07:47.810Z",513 "archetype": "regular",514 "unseen": false,515 "pinned": false,516 "unpinned": null,517 "visible": true,518 "closed": false,519 "archived": false,520 "bookmarked": null,521 "liked": null,522 "tags_descriptions": {},523 "like_count": 0,524 "views": 61,525 "category_id": 7,526 "featured_link": null,527 "has_accepted_answer": false,528 "posters": [529 {530 "extras": "latest single",531 "description": "Original Poster, Most Recent Poster",532 "user": {533 "id": 78786,534 "username": "robofar123",535 "name": "Faris",536 "avatar_template": "/letter_avatar_proxy/v4/letter/r/22d042/{size}.png",537 "trust_level": 1538 }539 }540 ]541 },542 {543 "fancy_title": "Training does not progress using Pytorch LBFGS Optimizer",544 "id": 213093,545 "title": "Training does not progress using Pytorch LBFGS Optimizer",546 "slug": "training-does-not-progress-using-pytorch-lbfgs-optimizer",547 "posts_count": 1,548 "reply_count": 0,549 "highest_post_number": 1,550 "image_url": null,551 "created_at": "2024-11-18T02:42:58.802Z",552 "last_posted_at": "2024-11-18T02:42:58.854Z",553 "bumped": true,554 "bumped_at": "2024-11-18T12:00:38.537Z",555 "archetype": "regular",556 "unseen": false,557 "pinned": false,558 "unpinned": null,559 "visible": true,560 "closed": false,561 "archived": false,562 "bookmarked": null,563 "liked": null,564 "tags_descriptions": {},565 "like_count": 0,566 "views": 157,567 "category_id": 7,568 "featured_link": null,569 "has_accepted_answer": false,570 "posters": [571 {572 "extras": "latest single",573 "description": "Original Poster, Most Recent Poster",574 "user": {575 "id": 78824,576 "username": "tpwls0672",577 "name": "Se Jin An",578 "avatar_template": "/letter_avatar_proxy/v4/letter/t/ce73a5/{size}.png",579 "trust_level": 1580 }581 }582 ]583 }584 ],585 "tags_descriptions": {},586 "fancy_title": "My network’s weights get updated despite using torch.no_grad()",587 "id": 201687,588 "title": "My network's weights get updated despite using torch.no_grad()",589 "posts_count": 5,590 "created_at": "2024-04-27T03:55:46.603Z",591 "views": 197,592 "reply_count": 3,593 "like_count": 0,594 "last_posted_at": "2024-04-29T02:38:29.566Z",595 "visible": true,596 "closed": false,597 "archived": false,598 "has_summary": false,599 "archetype": "regular",600 "slug": "my-networks-weights-get-updated-despite-using-torch-no-grad",601 "category_id": 7,602 "word_count": 400,603 "deleted_at": null,604 "user_id": 75360,605 "featured_link": null,606 "pinned_globally": false,607 "pinned_at": null,608 "pinned_until": null,609 "image_url": null,610 "slow_mode_seconds": 0,611 "draft": null,612 "draft_key": "topic_201687",613 "draft_sequence": null,614 "unpinned": null,615 "pinned": false,616 "current_post_number": 1,617 "highest_post_number": 5,618 "deleted_by": null,619 "actions_summary": [620 {621 "id": 4,622 "count": 0,623 "hidden": false,624 "can_act": false625 },626 {627 "id": 8,628 "count": 0,629 "hidden": false,630 "can_act": false631 },632 {633 "id": 10,634 "count": 0,635 "hidden": false,636 "can_act": false637 },638 {639 "id": 7,640 "count": 0,641 "hidden": false,642 "can_act": false643 }644 ],645 "chunk_size": 20,646 "bookmarked": false,647 "topic_timer": null,648 "message_bus_last_id": 0,649 "participant_count": 2,650 "show_read_indicator": false,651 "thumbnails": null,652 "slow_mode_enabled_until": null,653 "accepted_answer": {654 "post_number": 2,655 "username": "ptrblck",656 "name": "",657 "excerpt": "Not necessarily, as you could also observe noise creating by additional calls to the pseudorandom number generator. You could check how sensitive your training is to the seed."658 },659 "can_vote": false,660 "vote_count": 0,661 "user_voted": false,662 "discourse_zendesk_plugin_zendesk_id": null,663 "discourse_zendesk_plugin_zendesk_url": "https://your-url.zendesk.com/agent/tickets/",664 "details": {665 "can_edit": false,666 "notification_level": 1,667 "participants": [668 {669 "id": 75360,670 "username": "nbusser",671 "name": "",672 "avatar_template": "/letter_avatar_proxy/v4/letter/n/e8c25b/{size}.png",673 "post_count": 3,674 "primary_group_name": null,675 "flair_name": null,676 "flair_url": null,677 "flair_color": null,678 "flair_bg_color": null,679 "flair_group_id": null,680 "trust_level": 1681 },682 {683 "id": 3534,684 "username": "ptrblck",685 "name": "",686 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",687 "post_count": 2,688 "primary_group_name": null,689 "flair_name": null,690 "flair_url": null,691 "flair_color": null,692 "flair_bg_color": null,693 "flair_group_id": null,694 "admin": true,695 "moderator": true,696 "trust_level": 2697 }698 ],699 "created_by": {700 "id": 75360,701 "username": "nbusser",702 "name": "",703 "avatar_template": "/letter_avatar_proxy/v4/letter/n/e8c25b/{size}.png"704 },705 "last_poster": {706 "id": 75360,707 "username": "nbusser",708 "name": "",709 "avatar_template": "/letter_avatar_proxy/v4/letter/n/e8c25b/{size}.png"710 }711 },712 "bookmarks": []713 },714 {715 "post_stream": {716 "posts": [717 {718 "id": 71573,719 "name": "Wei Chen",720 "username": "Wei_Chen",721 "avatar_template": "/user_avatar/discuss.pytorch.org/wei_chen/{size}/33855_2.png",722 "created_at": "2018-10-19T15:36:30.698Z",723 "cooked": "<p>Hi, I am using a set of 1D data for training and I noticed that GPU usage is quite low (<5%) and training takes very long time to finish. I profiled my code (following instruction here: <a href=\"https://www.sagivtech.com/2017/09/19/optimizing-pytorch-training-code/\" rel=\"nofollow noopener\">https://www.sagivtech.com/2017/09/19/optimizing-pytorch-training-code/</a>) and found that 73% of the computation time is spent on loading data. I wonder if it is possible to load all data into GPU memory to speed up training, and tried to include <code>pin_memory=True</code> in my code, but it told me “cannot pin ‘torch.cuda.FloatTensor’ only CPU memory can be pinned”. Does anyone have idea how I should do this?</p>\n<p>Thank you!</p>",724 "post_number": 1,725 "post_type": 1,726 "posts_count": 51,727 "updated_at": "2018-10-19T15:36:30.698Z",728 "reply_count": 0,729 "reply_to_post_number": null,730 "quote_count": 0,731 "incoming_link_count": 112532,732 "reads": 1850,733 "readers_count": 1849,734 "score": 562518.8,735 "yours": false,736 "topic_id": 27609,737 "topic_slug": "how-to-load-all-data-into-gpu-for-training",738 "display_username": "Wei Chen",739 "primary_group_name": null,740 "flair_name": null,741 "flair_url": null,742 "flair_bg_color": null,743 "flair_color": null,744 "flair_group_id": null,745 "badges_granted": [],746 "version": 1,747 "can_edit": false,748 "can_delete": false,749 "can_recover": false,750 "can_see_hidden_post": false,751 "can_wiki": false,752 "link_counts": [753 {754 "url": "https://www.sagivtech.com/2017/09/19/optimizing-pytorch-training-code/",755 "internal": false,756 "reflection": false,757 "title": "Optimizing PyTorch training code – Sagivtech",758 "clicks": 1532759 },760 {761 "url": "https://discuss.pytorch.org/t/cache-entire-train-test-data-on-gpu/114396/2",762 "internal": true,763 "reflection": true,764 "title": "Cache entire train/test data on GPU",765 "clicks": 38766 }767 ],768 "read": true,769 "user_title": null,770 "bookmarked": false,771 "actions_summary": [772 {773 "id": 2,774 "count": 2775 }776 ],777 "moderator": false,778 "admin": false,779 "staff": false,780 "user_id": 6998,781 "hidden": false,782 "trust_level": 1,783 "deleted_at": null,784 "user_deleted": false,785 "edit_reason": null,786 "can_view_edit_history": true,787 "wiki": false,788 "post_url": "/t/how-to-load-all-data-into-gpu-for-training/27609/1",789 "can_accept_answer": false,790 "can_unaccept_answer": false,791 "accepted_answer": false,792 "topic_accepted_answer": null,793 "can_vote": false794 },795 {796 "id": 71576,797 "name": "",798 "username": "ptrblck",799 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",800 "created_at": "2018-10-19T15:57:31.981Z",801 "cooked": "<p>To speed up the transfer of data from the host to your device you could use <code>pin_memory=True</code>. However, if your data already is on the GPU you don’t need this function and it will throw an error.<br>\nIf you have enough GPU memory to hold the data and model this is surely a valid approach to save the loading overhead.</p>",802 "post_number": 2,803 "post_type": 1,804 "posts_count": 51,805 "updated_at": "2018-10-19T15:57:31.981Z",806 "reply_count": 2,807 "reply_to_post_number": null,808 "quote_count": 0,809 "incoming_link_count": 542,810 "reads": 1849,811 "readers_count": 1848,812 "score": 3179.6,813 "yours": false,814 "topic_id": 27609,815 "topic_slug": "how-to-load-all-data-into-gpu-for-training",816 "display_username": "",817 "primary_group_name": null,818 "flair_name": null,819 "flair_url": null,820 "flair_bg_color": null,821 "flair_color": null,822 "flair_group_id": null,823 "badges_granted": [],824 "version": 1,825 "can_edit": false,826 "can_delete": false,827 "can_recover": false,828 "can_see_hidden_post": false,829 "can_wiki": false,830 "read": true,831 "user_title": "",832 "bookmarked": false,833 "actions_summary": [834 {835 "id": 2,836 "count": 6837 }838 ],839 "moderator": true,840 "admin": true,841 "staff": true,842 "user_id": 3534,843 "hidden": false,844 "trust_level": 2,845 "deleted_at": null,846 "user_deleted": false,847 "edit_reason": null,848 "can_view_edit_history": true,849 "wiki": false,850 "post_url": "/t/how-to-load-all-data-into-gpu-for-training/27609/2",851 "can_accept_answer": false,852 "can_unaccept_answer": false,853 "accepted_answer": false,854 "topic_accepted_answer": null855 },856 {857 "id": 71579,858 "name": "Wei Chen",859 "username": "Wei_Chen",860 "avatar_template": "/user_avatar/discuss.pytorch.org/wei_chen/{size}/33855_2.png",861 "created_at": "2018-10-19T16:08:19.574Z",862 "cooked": "<p>Thanks!</p>\n<p>How do I know if my data are in GPU memory?</p>",863 "post_number": 3,864 "post_type": 1,865 "posts_count": 51,866 "updated_at": "2018-10-19T16:08:28.315Z",867 "reply_count": 1,868 "reply_to_post_number": 2,869 "quote_count": 0,870 "incoming_link_count": 464,871 "reads": 1812,872 "readers_count": 1811,873 "score": 2687.2,874 "yours": false,875 "topic_id": 27609,876 "topic_slug": "how-to-load-all-data-into-gpu-for-training",877 "display_username": "Wei Chen",878 "primary_group_name": null,879 "flair_name": null,880 "flair_url": null,881 "flair_bg_color": null,882 "flair_color": null,883 "flair_group_id": null,884 "badges_granted": [],885 "version": 1,886 "can_edit": false,887 "can_delete": false,888 "can_recover": false,889 "can_see_hidden_post": false,890 "can_wiki": false,891 "read": true,892 "user_title": null,893 "reply_to_user": {894 "id": 3534,895 "username": "ptrblck",896 "name": "",897 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png"898 },899 "bookmarked": false,900 "actions_summary": [],901 "moderator": false,902 "admin": false,903 "staff": false,904 "user_id": 6998,905 "hidden": false,906 "trust_level": 1,907 "deleted_at": null,908 "user_deleted": false,909 "edit_reason": null,910 "can_view_edit_history": true,911 "wiki": false,912 "post_url": "/t/how-to-load-all-data-into-gpu-for-training/27609/3",913 "can_accept_answer": false,914 "can_unaccept_answer": false,915 "accepted_answer": false,916 "topic_accepted_answer": null917 },918 {919 "id": 71586,920 "name": "",921 "username": "ptrblck",922 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",923 "created_at": "2018-10-19T17:05:20.797Z",924 "cooked": "<p>You would have to push it onto the GPU with <code>data = data.to('cuda')</code>. Then you can check the device using <code>print(data.device)</code>.</p>",925 "post_number": 4,926 "post_type": 1,927 "posts_count": 51,928 "updated_at": "2018-10-19T17:05:20.797Z",929 "reply_count": 2,930 "reply_to_post_number": 3,931 "quote_count": 0,932 "incoming_link_count": 180,933 "reads": 1716,934 "readers_count": 1715,935 "score": 1345.0,936 "yours": false,937 "topic_id": 27609,938 "topic_slug": "how-to-load-all-data-into-gpu-for-training",939 "display_username": "",940 "primary_group_name": null,941 "flair_name": null,942 "flair_url": null,943 "flair_bg_color": null,944 "flair_color": null,945 "flair_group_id": null,946 "badges_granted": [],947 "version": 1,948 "can_edit": false,949 "can_delete": false,950 "can_recover": false,951 "can_see_hidden_post": false,952 "can_wiki": false,953 "read": true,954 "user_title": "",955 "reply_to_user": {956 "id": 6998,957 "username": "Wei_Chen",958 "name": "Wei Chen",959 "avatar_template": "/user_avatar/discuss.pytorch.org/wei_chen/{size}/33855_2.png"960 },961 "bookmarked": false,962 "actions_summary": [963 {964 "id": 2,965 "count": 5966 }967 ],968 "moderator": true,969 "admin": true,970 "staff": true,971 "user_id": 3534,972 "hidden": false,973 "trust_level": 2,974 "deleted_at": null,975 "user_deleted": false,976 "edit_reason": null,977 "can_view_edit_history": true,978 "wiki": false,979 "post_url": "/t/how-to-load-all-data-into-gpu-for-training/27609/4",980 "can_accept_answer": false,981 "can_unaccept_answer": false,982 "accepted_answer": false,983 "topic_accepted_answer": null984 },985 {986 "id": 71592,987 "name": "Wei Chen",988 "username": "Wei_Chen",989 "avatar_template": "/user_avatar/discuss.pytorch.org/wei_chen/{size}/33855_2.png",990 "created_at": "2018-10-19T17:45:51.254Z",991 "cooked": "<p>I got <code>cuda:0</code> as output of <code>print(data.device)</code>, does it mean all data are already in GPU memory? If so, what might be the reason that dataloader takes 70% of the computation time?</p>",992 "post_number": 5,993 "post_type": 1,994 "posts_count": 51,995 "updated_at": "2018-10-19T17:45:51.254Z",996 "reply_count": 1,997 "reply_to_post_number": 4,998 "quote_count": 0,999 "incoming_link_count": 206,1000 "reads": 1657,1001 "readers_count": 1656,1002 "score": 1366.2,1003 "yours": false,1004 "topic_id": 27609,1005 "topic_slug": "how-to-load-all-data-into-gpu-for-training",1006 "display_username": "Wei Chen",1007 "primary_group_name": null,1008 "flair_name": null,1009 "flair_url": null,1010 "flair_bg_color": null,1011 "flair_color": null,1012 "flair_group_id": null,1013 "badges_granted": [],1014 "version": 1,1015 "can_edit": false,1016 "can_delete": false,1017 "can_recover": false,1018 "can_see_hidden_post": false,1019 "can_wiki": false,1020 "read": true,1021 "user_title": null,1022 "reply_to_user": {1023 "id": 3534,1024 "username": "ptrblck",1025 "name": "",1026 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png"1027 },1028 "bookmarked": false,1029 "actions_summary": [],1030 "moderator": false,1031 "admin": false,1032 "staff": false,1033 "user_id": 6998,1034 "hidden": false,1035 "trust_level": 1,1036 "deleted_at": null,1037 "user_deleted": false,1038 "edit_reason": null,1039 "can_view_edit_history": true,1040 "wiki": false,1041 "post_url": "/t/how-to-load-all-data-into-gpu-for-training/27609/5",1042 "can_accept_answer": false,1043 "can_unaccept_answer": false,1044 "accepted_answer": false,1045 "topic_accepted_answer": null1046 },1047 {1048 "id": 71593,1049 "name": "",1050 "username": "ptrblck",1051 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",1052 "created_at": "2018-10-19T17:48:40.311Z",1053 "cooked": "<p>Yes, it is on the first GPU device.<br>\nHow do you time your code? Could you share a small example?<br>\nAre you processing the data somehow in your <code>Dataset</code>?</p>",1054 "post_number": 6,1055 "post_type": 1,1056 "posts_count": 51,1057 "updated_at": "2018-10-19T17:48:40.311Z",1058 "reply_count": 1,1059 "reply_to_post_number": 5,1060 "quote_count": 0,1061 "incoming_link_count": 310,1062 "reads": 1559,1063 "readers_count": 1558,1064 "score": 1866.6,1065 "yours": false,1066 "topic_id": 27609,1067 "topic_slug": "how-to-load-all-data-into-gpu-for-training",1068 "display_username": "",1069 "primary_group_name": null,1070 "flair_name": null,1071 "flair_url": null,1072 "flair_bg_color": null,1073 "flair_color": null,1074 "flair_group_id": null,1075 "badges_granted": [],1076 "version": 1,1077 "can_edit": false,1078 "can_delete": false,1079 "can_recover": false,1080 "can_see_hidden_post": false,1081 "can_wiki": false,1082 "read": true,1083 "user_title": "",1084 "reply_to_user": {1085 "id": 6998,1086 "username": "Wei_Chen",1087 "name": "Wei Chen",1088 "avatar_template": "/user_avatar/discuss.pytorch.org/wei_chen/{size}/33855_2.png"1089 },1090 "bookmarked": false,1091 "actions_summary": [],1092 "moderator": true,1093 "admin": true,1094 "staff": true,1095 "user_id": 3534,1096 "hidden": false,1097 "trust_level": 2,1098 "deleted_at": null,1099 "user_deleted": false,1100 "edit_reason": null,1101 "can_view_edit_history": true,1102 "wiki": false,1103 "post_url": "/t/how-to-load-all-data-into-gpu-for-training/27609/6",1104 "can_accept_answer": false,1105 "can_unaccept_answer": false,1106 "accepted_answer": false,1107 "topic_accepted_answer": null1108 },1109 {1110 "id": 71597,1111 "name": "Wei Chen",1112 "username": "Wei_Chen",1113 "avatar_template": "/user_avatar/discuss.pytorch.org/wei_chen/{size}/33855_2.png",1114 "created_at": "2018-10-19T17:59:31.950Z",1115 "cooked": "<p>Following is the main part of my code (for simplicity I remove implementation of model and loss function):</p>\n<pre><code class=\"lang-auto\">dataset = TensorDataset(data_1, data_2)\ntrain_loader = DataLoader(dataset, batch_size=5000, shuffle=True, drop_last=False)\n\noptimizer = torch.optim.Adam(model.parameters(), lr=0.001, weight_decay=0)\nepochs = 100\nfor index_epoch in range(epochs):\n print index_epoch\n for x_1, x_2 in train_loader:\n optimizer.zero_grad()\n z_1 = model(x_1) # model is the neural network model\n z_2 = model(x_2)\n loss = get_loss(z_1, z_2)\n loss.backward()\n # print loss_list\n optimizer.step()\n</code></pre>\n<p>I time it using python cProfile, and it turns out 70% of time is on <code>dataloader.py</code>, while only <10% on actual backward propagation. There is no additional processing of my data, I simply load it with <code>TensorDataset</code> and <code>DataLoader</code>. And the shape of my data is (5000000, 1).</p>",1116 "post_number": 7,1117 "post_type": 1,1118 "posts_count": 51,1119 "updated_at": "2018-10-19T18:00:20.497Z",1120 "reply_count": 2,1121 "reply_to_post_number": 6,1122 "quote_count": 0,1123 "incoming_link_count": 842,1124 "reads": 1446,1125 "readers_count": 1445,1126 "score": 4524.0,1127 "yours": false,1128 "topic_id": 27609,1129 "topic_slug": "how-to-load-all-data-into-gpu-for-training",1130 "display_username": "Wei Chen",1131 "primary_group_name": null,1132 "flair_name": null,1133 "flair_url": null,1134 "flair_bg_color": null,1135 "flair_color": null,1136 "flair_group_id": null,1137 "badges_granted": [],1138 "version": 1,1139 "can_edit": false,1140 "can_delete": false,1141 "can_recover": false,1142 "can_see_hidden_post": false,1143 "can_wiki": false,1144 "read": true,1145 "user_title": null,1146 "reply_to_user": {1147 "id": 3534,1148 "username": "ptrblck",1149 "name": "",1150 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png"1151 },1152 "bookmarked": false,1153 "actions_summary": [1154 {1155 "id": 2,1156 "count": 11157 }1158 ],1159 "moderator": false,1160 "admin": false,1161 "staff": false,1162 "user_id": 6998,1163 "hidden": false,1164 "trust_level": 1,1165 "deleted_at": null,1166 "user_deleted": false,1167 "edit_reason": null,1168 "can_view_edit_history": true,1169 "wiki": false,1170 "post_url": "/t/how-to-load-all-data-into-gpu-for-training/27609/7",1171 "can_accept_answer": false,1172 "can_unaccept_answer": false,1173 "accepted_answer": false,1174 "topic_accepted_answer": null1175 },1176 {1177 "id": 71599,1178 "name": "",1179 "username": "ptrblck",1180 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",1181 "created_at": "2018-10-19T18:11:58.665Z",1182 "cooked": "<p>Thanks for the information!<br>\nHow big is your model? Could you profile it again with <a href=\"https://pytorch.org/docs/stable/bottleneck.html\">torch.utils.bottleneck</a>?</p>\n<p>Since CUDA operations are run asynchronously, your <code>DataLoader</code> might have to wait for the CUDA op to finish, thus reporting a false number.</p>\n<p>An alternative would be to add <code>torch.cuda.synchronize()</code> calls before and after the forward and backward calls, and time it manually.</p>",1183 "post_number": 8,1184 "post_type": 1,1185 "posts_count": 51,1186 "updated_at": "2018-10-19T18:11:58.665Z",1187 "reply_count": 1,1188 "reply_to_post_number": 7,1189 "quote_count": 0,1190 "incoming_link_count": 217,1191 "reads": 1288,1192 "readers_count": 1287,1193 "score": 1437.4,1194 "yours": false,1195 "topic_id": 27609,1196 "topic_slug": "how-to-load-all-data-into-gpu-for-training",1197 "display_username": "",1198 "primary_group_name": null,1199 "flair_name": null,1200 "flair_url": null,