Anurag1734/cuda-error-resolution-analysis
07
1[2 {3 "post_stream": {4 "posts": [5 {6 "id": 210085,7 "name": "Stefano Savian",8 "username": "Stefano_Savian",9 "avatar_template": "/user_avatar/discuss.pytorch.org/stefano_savian/{size}/14377_2.png",10 "created_at": "2020-07-07T14:21:41.168Z",11 "cooked": "<p>Dear all,</p>\n<p>I have a doubt on data augmentation using dataloader. I want to apply a transformation with a certain probability <em>p</em> which can decreases either as the number of epochs progress, or when the loss function is higher than a certain threshold. However, due to the concurrent nature of the dataloader workers, I cannot figure out how to apply this to the transorm class, as I don’t know how I could get the epoch or loss as inputs to my function. So my question is</p>\n<ol>\n<li>Is it possible to apply augmentation with the probability changing on the epoch number or loss function?</li>\n<li>How could it be implemented?</li>\n</ol>\n<p>below is the pseudo code of my transform function.</p>\n<p>class custom_transform(object):</p>\n<pre><code>def __init__(self, size,p):\n if isinstance(size, numbers.Number):\n self.size = (int(size), int(size))\n else:\n self.size = size\n self.p = p\ndef __call__(self, inputs, target):\n if random.random() < self.p:\n GENERATE TRANSFORMED inputs, target\n\n inputs, target\n else:\n return inputs, target\n</code></pre>\n<p><a class=\"mention\" href=\"/u/alband\">@albanD</a> <a class=\"mention\" href=\"/u/ptrblck\">@ptrblck</a> your ideas are very welcome!</p>\n<p>Many Thanks,<br>\nStefano</p>",12 "post_number": 1,13 "post_type": 1,14 "posts_count": 3,15 "updated_at": "2020-07-09T20:26:45.493Z",16 "reply_count": 0,17 "reply_to_post_number": null,18 "quote_count": 0,19 "incoming_link_count": 771,20 "reads": 24,21 "readers_count": 23,22 "score": 3839.8,23 "yours": false,24 "topic_id": 88247,25 "topic_slug": "making-augmentation-transform-dependent-on-epoch-number",26 "display_username": "Stefano Savian",27 "primary_group_name": null,28 "flair_name": null,29 "flair_url": null,30 "flair_bg_color": null,31 "flair_color": null,32 "flair_group_id": null,33 "badges_granted": [],34 "version": 2,35 "can_edit": false,36 "can_delete": false,37 "can_recover": false,38 "can_see_hidden_post": false,39 "can_wiki": false,40 "read": true,41 "user_title": null,42 "bookmarked": false,43 "actions_summary": [],44 "moderator": false,45 "admin": false,46 "staff": false,47 "user_id": 20931,48 "hidden": false,49 "trust_level": 2,50 "deleted_at": null,51 "user_deleted": false,52 "edit_reason": null,53 "can_view_edit_history": true,54 "wiki": false,55 "post_url": "/t/making-augmentation-transform-dependent-on-epoch-number/88247/1",56 "can_accept_answer": false,57 "can_unaccept_answer": false,58 "accepted_answer": false,59 "topic_accepted_answer": null,60 "can_vote": false61 },62 {63 "id": 210840,64 "name": "Alban D",65 "username": "albanD",66 "avatar_template": "/user_avatar/discuss.pytorch.org/alband/{size}/215_2.png",67 "created_at": "2020-07-09T21:48:59.626Z",68 "cooked": "<p>Hey,</p>\n<p>I can’t think of a builtin way to do this. But you should be able to have a global variable shared by all the workers that contain this p that they will read and will be set by the main training process?<br>\nMaybe a 1 element Tensor will work as it will be shared across the processes.</p>",69 "post_number": 2,70 "post_type": 1,71 "posts_count": 3,72 "updated_at": "2020-07-09T21:48:59.626Z",73 "reply_count": 0,74 "reply_to_post_number": null,75 "quote_count": 0,76 "incoming_link_count": 10,77 "reads": 19,78 "readers_count": 18,79 "score": 68.8,80 "yours": false,81 "topic_id": 88247,82 "topic_slug": "making-augmentation-transform-dependent-on-epoch-number",83 "display_username": "Alban D",84 "primary_group_name": null,85 "flair_name": null,86 "flair_url": null,87 "flair_bg_color": null,88 "flair_color": null,89 "flair_group_id": null,90 "badges_granted": [],91 "version": 1,92 "can_edit": false,93 "can_delete": false,94 "can_recover": false,95 "can_see_hidden_post": false,96 "can_wiki": false,97 "read": true,98 "user_title": "",99 "bookmarked": false,100 "actions_summary": [101 {102 "id": 2,103 "count": 1104 }105 ],106 "moderator": true,107 "admin": true,108 "staff": true,109 "user_id": 211,110 "hidden": false,111 "trust_level": 4,112 "deleted_at": null,113 "user_deleted": false,114 "edit_reason": null,115 "can_view_edit_history": true,116 "wiki": false,117 "post_url": "/t/making-augmentation-transform-dependent-on-epoch-number/88247/2",118 "can_accept_answer": false,119 "can_unaccept_answer": false,120 "accepted_answer": false,121 "topic_accepted_answer": null122 },123 {124 "id": 211343,125 "name": "",126 "username": "ptrblck",127 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",128 "created_at": "2020-07-11T23:29:27.395Z",129 "cooked": "<p><a class=\"mention\" href=\"/u/alband\">@albanD</a>’s idea sounds good.</p>\n<p>If your workflow would allow you to change the probability after each epoch finished, you could also directly manipulate the <code>loader.dataset</code>. At the beginning of each epoch, the workers of the <code>DataLoader</code> will be respawned and will thus use the changed dataset.</p>",130 "post_number": 3,131 "post_type": 1,132 "posts_count": 3,133 "updated_at": "2020-07-11T23:29:27.395Z",134 "reply_count": 0,135 "reply_to_post_number": null,136 "quote_count": 0,137 "incoming_link_count": 8,138 "reads": 19,139 "readers_count": 18,140 "score": 58.8,141 "yours": false,142 "topic_id": 88247,143 "topic_slug": "making-augmentation-transform-dependent-on-epoch-number",144 "display_username": "",145 "primary_group_name": null,146 "flair_name": null,147 "flair_url": null,148 "flair_bg_color": null,149 "flair_color": null,150 "flair_group_id": null,151 "badges_granted": [],152 "version": 1,153 "can_edit": false,154 "can_delete": false,155 "can_recover": false,156 "can_see_hidden_post": false,157 "can_wiki": false,158 "read": true,159 "user_title": "",160 "bookmarked": false,161 "actions_summary": [162 {163 "id": 2,164 "count": 1165 }166 ],167 "moderator": true,168 "admin": true,169 "staff": true,170 "user_id": 3534,171 "hidden": false,172 "trust_level": 2,173 "deleted_at": null,174 "user_deleted": false,175 "edit_reason": null,176 "can_view_edit_history": true,177 "wiki": false,178 "post_url": "/t/making-augmentation-transform-dependent-on-epoch-number/88247/3",179 "can_accept_answer": false,180 "can_unaccept_answer": false,181 "accepted_answer": false,182 "topic_accepted_answer": null183 }184 ],185 "stream": [186 210085,187 210840,188 211343189 ]190 },191 "timeline_lookup": [192 [193 1,194 1936195 ],196 [197 2,198 1934199 ],200 [201 3,202 1932203 ]204 ],205 "suggested_topics": [206 {207 "fancy_title": "DDU for segmentation",208 "id": 217843,209 "title": "DDU for segmentation",210 "slug": "ddu-for-segmentation",211 "posts_count": 1,212 "reply_count": 0,213 "highest_post_number": 1,214 "image_url": null,215 "created_at": "2025-03-14T08:24:38.392Z",216 "last_posted_at": "2025-03-14T08:24:38.430Z",217 "bumped": true,218 "bumped_at": "2025-03-14T08:24:38.430Z",219 "archetype": "regular",220 "unseen": false,221 "pinned": false,222 "unpinned": null,223 "visible": true,224 "closed": false,225 "archived": false,226 "bookmarked": null,227 "liked": null,228 "tags_descriptions": {},229 "like_count": 0,230 "views": 36,231 "category_id": 5,232 "featured_link": null,233 "has_accepted_answer": false,234 "posters": [235 {236 "extras": "latest single",237 "description": "Original Poster, Most Recent Poster",238 "user": {239 "id": 42293,240 "username": "Mohamed_Farag",241 "name": "Mohamed Farag",242 "avatar_template": "/user_avatar/discuss.pytorch.org/mohamed_farag/{size}/34888_2.png",243 "trust_level": 1244 }245 }246 ]247 },248 {249 "fancy_title": "CNN produces the same output for different inputs",250 "id": 219543,251 "title": "CNN produces the same output for different inputs",252 "slug": "cnn-produces-the-same-output-for-different-inputs",253 "posts_count": 2,254 "reply_count": 0,255 "highest_post_number": 2,256 "image_url": null,257 "created_at": "2025-04-28T10:10:19.221Z",258 "last_posted_at": "2025-05-06T16:32:11.777Z",259 "bumped": true,260 "bumped_at": "2025-05-06T16:32:11.777Z",261 "archetype": "regular",262 "unseen": false,263 "pinned": false,264 "unpinned": null,265 "visible": true,266 "closed": false,267 "archived": false,268 "bookmarked": null,269 "liked": null,270 "tags_descriptions": {},271 "like_count": 0,272 "views": 54,273 "category_id": 5,274 "featured_link": null,275 "has_accepted_answer": false,276 "posters": [277 {278 "extras": null,279 "description": "Original Poster",280 "user": {281 "id": 83291,282 "username": "Nyx",283 "name": "Nyx",284 "avatar_template": "/user_avatar/discuss.pytorch.org/nyx/{size}/76178_2.png",285 "trust_level": 1286 }287 },288 {289 "extras": "latest",290 "description": "Most Recent Poster",291 "user": {292 "id": 32812,293 "username": "Bjorn_Lindqvist",294 "name": "Björn Lindqvist",295 "avatar_template": "/user_avatar/discuss.pytorch.org/bjorn_lindqvist/{size}/25326_2.png",296 "trust_level": 2297 }298 }299 ]300 },301 {302 "fancy_title": "How to implement pytorchvideo from input as images",303 "id": 215938,304 "title": "How to implement pytorchvideo from input as images",305 "slug": "how-to-implement-pytorchvideo-from-input-as-images",306 "posts_count": 1,307 "reply_count": 0,308 "highest_post_number": 1,309 "image_url": null,310 "created_at": "2025-01-27T14:58:17.998Z",311 "last_posted_at": "2025-01-27T14:58:18.042Z",312 "bumped": true,313 "bumped_at": "2025-01-27T15:02:57.679Z",314 "archetype": "regular",315 "unseen": false,316 "pinned": false,317 "unpinned": null,318 "visible": true,319 "closed": false,320 "archived": false,321 "bookmarked": null,322 "liked": null,323 "tags_descriptions": {},324 "like_count": 0,325 "views": 56,326 "category_id": 5,327 "featured_link": null,328 "has_accepted_answer": false,329 "posters": [330 {331 "extras": "latest single",332 "description": "Original Poster, Most Recent Poster",333 "user": {334 "id": 82344,335 "username": "trungnb34",336 "name": "",337 "avatar_template": "/letter_avatar_proxy/v4/letter/t/ecb155/{size}.png",338 "trust_level": 0339 }340 }341 ]342 },343 {344 "fancy_title": "CNN Model is not learning after some epochs",345 "id": 215485,346 "title": "CNN Model is not learning after some epochs",347 "slug": "cnn-model-is-not-learning-after-some-epochs",348 "posts_count": 5,349 "reply_count": 3,350 "highest_post_number": 5,351 "image_url": "https://discuss.pytorch.org/uploads/default/optimized/3X/6/c/6cf1deea54130767dd867db4827e1505ce26e24b_2_1024x817.jpeg",352 "created_at": "2025-01-16T17:42:58.865Z",353 "last_posted_at": "2025-01-17T15:02:23.101Z",354 "bumped": true,355 "bumped_at": "2025-01-17T15:02:23.101Z",356 "archetype": "regular",357 "unseen": false,358 "pinned": false,359 "unpinned": null,360 "visible": true,361 "closed": false,362 "archived": false,363 "bookmarked": null,364 "liked": null,365 "tags_descriptions": {},366 "like_count": 1,367 "views": 170,368 "category_id": 5,369 "featured_link": null,370 "has_accepted_answer": false,371 "posters": [372 {373 "extras": "latest",374 "description": "Original Poster, Most Recent Poster",375 "user": {376 "id": 81840,377 "username": "iran_boy",378 "name": "iran boy",379 "avatar_template": "/user_avatar/discuss.pytorch.org/iran_boy/{size}/74864_2.png",380 "trust_level": 0381 }382 },383 {384 "extras": null,385 "description": "Frequent Poster",386 "user": {387 "id": 77908,388 "username": "mycul",389 "name": "",390 "avatar_template": "/user_avatar/discuss.pytorch.org/mycul/{size}/72394_2.png",391 "trust_level": 2392 }393 }394 ]395 },396 {397 "fancy_title": "My predicted output ends up being a list of outputs",398 "id": 219642,399 "title": "My predicted output ends up being a list of outputs",400 "slug": "my-predicted-output-ends-up-being-a-list-of-outputs",401 "posts_count": 1,402 "reply_count": 0,403 "highest_post_number": 1,404 "image_url": null,405 "created_at": "2025-04-30T22:42:29.505Z",406 "last_posted_at": "2025-04-30T22:42:29.546Z",407 "bumped": true,408 "bumped_at": "2025-05-01T05:14:11.761Z",409 "archetype": "regular",410 "unseen": false,411 "pinned": false,412 "unpinned": null,413 "visible": true,414 "closed": false,415 "archived": false,416 "bookmarked": null,417 "liked": null,418 "tags_descriptions": {},419 "like_count": 0,420 "views": 32,421 "category_id": 5,422 "featured_link": null,423 "has_accepted_answer": false,424 "posters": [425 {426 "extras": "latest single",427 "description": "Original Poster, Most Recent Poster",428 "user": {429 "id": 84040,430 "username": "akulkarni3472",431 "name": "A Kulkarni",432 "avatar_template": "/user_avatar/discuss.pytorch.org/akulkarni3472/{size}/76809_2.png",433 "trust_level": 1434 }435 }436 ]437 }438 ],439 "tags_descriptions": {},440 "fancy_title": "Making augmentation transform dependent on epoch number",441 "id": 88247,442 "title": "Making augmentation transform dependent on epoch number",443 "posts_count": 3,444 "created_at": "2020-07-07T14:21:41.114Z",445 "views": 1522,446 "reply_count": 0,447 "like_count": 2,448 "last_posted_at": "2020-07-11T23:29:27.395Z",449 "visible": true,450 "closed": false,451 "archived": false,452 "has_summary": false,453 "archetype": "regular",454 "slug": "making-augmentation-transform-dependent-on-epoch-number",455 "category_id": 5,456 "word_count": 283,457 "deleted_at": null,458 "user_id": 20931,459 "featured_link": null,460 "pinned_globally": false,461 "pinned_at": null,462 "pinned_until": null,463 "image_url": null,464 "slow_mode_seconds": 0,465 "draft": null,466 "draft_key": "topic_88247",467 "draft_sequence": null,468 "unpinned": null,469 "pinned": false,470 "current_post_number": 1,471 "highest_post_number": 3,472 "deleted_by": null,473 "actions_summary": [474 {475 "id": 4,476 "count": 0,477 "hidden": false,478 "can_act": false479 },480 {481 "id": 8,482 "count": 0,483 "hidden": false,484 "can_act": false485 },486 {487 "id": 10,488 "count": 0,489 "hidden": false,490 "can_act": false491 },492 {493 "id": 7,494 "count": 0,495 "hidden": false,496 "can_act": false497 }498 ],499 "chunk_size": 20,500 "bookmarked": false,501 "topic_timer": null,502 "message_bus_last_id": 0,503 "participant_count": 3,504 "show_read_indicator": false,505 "thumbnails": null,506 "slow_mode_enabled_until": null,507 "can_vote": false,508 "vote_count": 0,509 "user_voted": false,510 "discourse_zendesk_plugin_zendesk_id": null,511 "discourse_zendesk_plugin_zendesk_url": "https://your-url.zendesk.com/agent/tickets/",512 "details": {513 "can_edit": false,514 "notification_level": 1,515 "participants": [516 {517 "id": 211,518 "username": "albanD",519 "name": "Alban D",520 "avatar_template": "/user_avatar/discuss.pytorch.org/alband/{size}/215_2.png",521 "post_count": 1,522 "primary_group_name": null,523 "flair_name": null,524 "flair_url": null,525 "flair_color": null,526 "flair_bg_color": null,527 "flair_group_id": null,528 "admin": true,529 "moderator": true,530 "trust_level": 4531 },532 {533 "id": 3534,534 "username": "ptrblck",535 "name": "",536 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",537 "post_count": 1,538 "primary_group_name": null,539 "flair_name": null,540 "flair_url": null,541 "flair_color": null,542 "flair_bg_color": null,543 "flair_group_id": null,544 "admin": true,545 "moderator": true,546 "trust_level": 2547 },548 {549 "id": 20931,550 "username": "Stefano_Savian",551 "name": "Stefano Savian",552 "avatar_template": "/user_avatar/discuss.pytorch.org/stefano_savian/{size}/14377_2.png",553 "post_count": 1,554 "primary_group_name": null,555 "flair_name": null,556 "flair_url": null,557 "flair_color": null,558 "flair_bg_color": null,559 "flair_group_id": null,560 "trust_level": 2561 }562 ],563 "created_by": {564 "id": 20931,565 "username": "Stefano_Savian",566 "name": "Stefano Savian",567 "avatar_template": "/user_avatar/discuss.pytorch.org/stefano_savian/{size}/14377_2.png"568 },569 "last_poster": {570 "id": 3534,571 "username": "ptrblck",572 "name": "",573 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png"574 }575 },576 "bookmarks": []577 },578 {579 "post_stream": {580 "posts": [581 {582 "id": 210835,583 "name": "Ruben Rodriguez Torrado",584 "username": "rubentorrado",585 "avatar_template": "/user_avatar/discuss.pytorch.org/rubentorrado/{size}/12519_2.png",586 "created_at": "2020-07-09T20:41:44.444Z",587 "cooked": "<p>We are running a self-attention mechanism with a conv1d network. We have gotten oom error when we multiply the quey tensor with the key tensor. The size of the query and key tensors are [16, 4, 22778]<br>\nt_1 = torch.bmm(proj_query, proj_key)</p>\n<p>We have tried converting our tensors to half-precision; however, we still get the same error OOM:</p>\n<p>RuntimeError: CUDA out of memory. Tried to allocate 15.46 GiB (GPU 0; 14.76 GiB total capacity; 53.88 MiB already allocated; 13.97 GiB free; 80.00 MiB reserved in total by PyTorch)</p>\n<p>Do you have any recommendations to fix this issue?</p>",588 "post_number": 1,589 "post_type": 1,590 "posts_count": 2,591 "updated_at": "2020-07-09T20:41:44.444Z",592 "reply_count": 0,593 "reply_to_post_number": null,594 "quote_count": 0,595 "incoming_link_count": 130,596 "reads": 7,597 "readers_count": 6,598 "score": 651.4,599 "yours": false,600 "topic_id": 88608,601 "topic_slug": "oom-for-self-attention-mechanism",602 "display_username": "Ruben Rodriguez Torrado",603 "primary_group_name": null,604 "flair_name": null,605 "flair_url": null,606 "flair_bg_color": null,607 "flair_color": null,608 "flair_group_id": null,609 "badges_granted": [],610 "version": 1,611 "can_edit": false,612 "can_delete": false,613 "can_recover": false,614 "can_see_hidden_post": false,615 "can_wiki": false,616 "read": true,617 "user_title": null,618 "bookmarked": false,619 "actions_summary": [],620 "moderator": false,621 "admin": false,622 "staff": false,623 "user_id": 34115,624 "hidden": false,625 "trust_level": 1,626 "deleted_at": null,627 "user_deleted": false,628 "edit_reason": null,629 "can_view_edit_history": true,630 "wiki": false,631 "post_url": "/t/oom-for-self-attention-mechanism/88608/1",632 "can_accept_answer": false,633 "can_unaccept_answer": false,634 "accepted_answer": false,635 "topic_accepted_answer": null,636 "can_vote": false637 },638 {639 "id": 211342,640 "name": "",641 "username": "ptrblck",642 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",643 "created_at": "2020-07-11T23:25:24.152Z",644 "cooked": "<p>The shape of both tensors shouldn’t be equal, as this would return a shape mismatch error for the matrix multiplication.<br>\nCould you recheck the shapes and post it here, please?</p>",645 "post_number": 2,646 "post_type": 1,647 "posts_count": 2,648 "updated_at": "2020-07-11T23:25:24.152Z",649 "reply_count": 0,650 "reply_to_post_number": null,651 "quote_count": 0,652 "incoming_link_count": 0,653 "reads": 6,654 "readers_count": 5,655 "score": 1.2,656 "yours": false,657 "topic_id": 88608,658 "topic_slug": "oom-for-self-attention-mechanism",659 "display_username": "",660 "primary_group_name": null,661 "flair_name": null,662 "flair_url": null,663 "flair_bg_color": null,664 "flair_color": null,665 "flair_group_id": null,666 "badges_granted": [],667 "version": 1,668 "can_edit": false,669 "can_delete": false,670 "can_recover": false,671 "can_see_hidden_post": false,672 "can_wiki": false,673 "read": true,674 "user_title": "",675 "bookmarked": false,676 "actions_summary": [],677 "moderator": true,678 "admin": true,679 "staff": true,680 "user_id": 3534,681 "hidden": false,682 "trust_level": 2,683 "deleted_at": null,684 "user_deleted": false,685 "edit_reason": null,686 "can_view_edit_history": true,687 "wiki": false,688 "post_url": "/t/oom-for-self-attention-mechanism/88608/2",689 "can_accept_answer": false,690 "can_unaccept_answer": false,691 "accepted_answer": false,692 "topic_accepted_answer": null693 }694 ],695 "stream": [696 210835,697 211342698 ]699 },700 "timeline_lookup": [701 [702 1,703 1934704 ],705 [706 2,707 1932708 ]709 ],710 "suggested_topics": [711 {712 "fancy_title": "Accuracy on test set is higher training with DP than training DDP, even using 1 GPU",713 "id": 213345,714 "title": "Accuracy on test set is higher training with DP than training DDP, even using 1 GPU",715 "slug": "accuracy-on-test-set-is-higher-training-with-dp-than-training-ddp-even-using-1-gpu",716 "posts_count": 1,717 "reply_count": 0,718 "highest_post_number": 1,719 "image_url": null,720 "created_at": "2024-11-23T12:42:30.198Z",721 "last_posted_at": "2024-11-23T12:42:30.272Z",722 "bumped": true,723 "bumped_at": "2024-11-23T12:42:30.272Z",724 "archetype": "regular",725 "unseen": false,726 "pinned": false,727 "unpinned": null,728 "visible": true,729 "closed": false,730 "archived": false,731 "bookmarked": null,732 "liked": null,733 "tags_descriptions": {},734 "like_count": 0,735 "views": 16,736 "category_id": 1,737 "featured_link": null,738 "has_accepted_answer": false,739 "posters": [740 {741 "extras": "latest single",742 "description": "Original Poster, Most Recent Poster",743 "user": {744 "id": 80463,745 "username": "iviivi_01",746 "name": "Iván",747 "avatar_template": "/letter_avatar_proxy/v4/letter/i/ebca7d/{size}.png",748 "trust_level": 1749 }750 }751 ]752 },753 {754 "fancy_title": "Does PyTorch verify the consistency of training functionality between previous and current versions through loss alignment?",755 "id": 212621,756 "title": "Does PyTorch verify the consistency of training functionality between previous and current versions through loss alignment?",757 "slug": "does-pytorch-verify-the-consistency-of-training-functionality-between-previous-and-current-versions-through-loss-alignment",758 "posts_count": 1,759 "reply_count": 0,760 "highest_post_number": 1,761 "image_url": null,762 "created_at": "2024-11-06T16:30:07.028Z",763 "last_posted_at": "2024-11-06T16:30:07.073Z",764 "bumped": true,765 "bumped_at": "2024-11-06T16:30:07.073Z",766 "archetype": "regular",767 "unseen": false,768 "pinned": false,769 "unpinned": null,770 "visible": true,771 "closed": false,772 "archived": false,773 "bookmarked": null,774 "liked": null,775 "tags_descriptions": {},776 "like_count": 0,777 "views": 33,778 "category_id": 1,779 "featured_link": null,780 "has_accepted_answer": false,781 "posters": [782 {783 "extras": "latest single",784 "description": "Original Poster, Most Recent Poster",785 "user": {786 "id": 80146,787 "username": "hlh",788 "name": "heng",789 "avatar_template": "/letter_avatar_proxy/v4/letter/h/f475e1/{size}.png",790 "trust_level": 0791 }792 }793 ]794 },795 {796 "fancy_title": "When I use the optimizer, there is no gradient due to the use of unit8, but I have to use unit8",797 "id": 214893,798 "title": "When I use the optimizer, there is no gradient due to the use of unit8, but I have to use unit8",799 "slug": "when-i-use-the-optimizer-there-is-no-gradient-due-to-the-use-of-unit8-but-i-have-to-use-unit8",800 "posts_count": 9,801 "reply_count": 5,802 "highest_post_number": 9,803 "image_url": null,804 "created_at": "2025-01-02T14:06:04.839Z",805 "last_posted_at": "2025-01-13T12:39:45.943Z",806 "bumped": true,807 "bumped_at": "2025-01-13T12:39:45.943Z",808 "archetype": "regular",809 "unseen": false,810 "pinned": false,811 "unpinned": null,812 "visible": true,813 "closed": false,814 "archived": false,815 "bookmarked": null,816 "liked": null,817 "tags_descriptions": {},818 "like_count": 2,819 "views": 67,820 "category_id": 1,821 "featured_link": null,822 "has_accepted_answer": false,823 "posters": [824 {825 "extras": "latest",826 "description": "Original Poster, Most Recent Poster",827 "user": {828 "id": 81832,829 "username": "wang1528186571",830 "name": "Wang1528186571",831 "avatar_template": "/user_avatar/discuss.pytorch.org/wang1528186571/{size}/74860_2.png",832 "trust_level": 0833 }834 },835 {836 "extras": null,837 "description": "Frequent Poster",838 "user": {839 "id": 41396,840 "username": "soulitzer",841 "name": "",842 "avatar_template": "/letter_avatar_proxy/v4/letter/s/839c29/{size}.png",843 "trust_level": 2844 }845 },846 {847 "extras": null,848 "description": "Frequent Poster",849 "user": {850 "id": 27175,851 "username": "Sayed_Nadim",852 "name": "",853 "avatar_template": "/user_avatar/discuss.pytorch.org/sayed_nadim/{size}/34051_2.png",854 "trust_level": 2855 }856 }857 ]858 },859 {860 "fancy_title": "Do I need to use nn.Parameter() again when changing a tensor wrapped in nn.Parameter()?",861 "id": 215213,862 "title": "Do I need to use nn.Parameter() again when changing a tensor wrapped in nn.Parameter()?",863 "slug": "do-i-need-to-use-nn-parameter-again-when-changing-a-tensor-wrapped-in-nn-parameter",864 "posts_count": 3,865 "reply_count": 1,866 "highest_post_number": 3,867 "image_url": null,868 "created_at": "2025-01-10T14:50:53.511Z",869 "last_posted_at": "2025-01-11T07:21:50.293Z",870 "bumped": true,871 "bumped_at": "2025-01-11T07:21:50.293Z",872 "archetype": "regular",873 "unseen": false,874 "pinned": false,875 "unpinned": null,876 "visible": true,877 "closed": false,878 "archived": false,879 "bookmarked": null,880 "liked": null,881 "tags_descriptions": {},882 "like_count": 1,883 "views": 43,884 "category_id": 1,885 "featured_link": null,886 "has_accepted_answer": true,887 "posters": [888 {889 "extras": "latest",890 "description": "Original Poster, Most Recent Poster",891 "user": {892 "id": 68895,893 "username": "DawidL",894 "name": "",895 "avatar_template": "/user_avatar/discuss.pytorch.org/dawidl/{size}/62949_2.png",896 "trust_level": 1897 }898 },899 {900 "extras": null,901 "description": "Frequent Poster, Accepted Answer",902 "user": {903 "id": 3534,904 "username": "ptrblck",905 "name": "",906 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",907 "admin": true,908 "moderator": true,909 "trust_level": 2910 }911 }912 ]913 },914 {915 "fancy_title": "GradCAM while training break gradients flow",916 "id": 215992,917 "title": "GradCAM while training break gradients flow",918 "slug": "gradcam-while-training-break-gradients-flow",919 "posts_count": 1,920 "reply_count": 0,921 "highest_post_number": 1,922 "image_url": null,923 "created_at": "2025-01-28T14:52:46.641Z",924 "last_posted_at": "2025-01-28T14:52:46.685Z",925 "bumped": true,926 "bumped_at": "2025-01-28T14:52:46.685Z",927 "archetype": "regular",928 "unseen": false,929 "pinned": false,930 "unpinned": null,931 "visible": true,932 "closed": false,933 "archived": false,934 "bookmarked": null,935 "liked": null,936 "tags_descriptions": {},937 "like_count": 0,938 "views": 113,939 "category_id": 1,940 "featured_link": null,941 "has_accepted_answer": false,942 "posters": [943 {944 "extras": "latest single",945 "description": "Original Poster, Most Recent Poster",946 "user": {947 "id": 34558,948 "username": "Giovanni_Cena",949 "name": "Giovanni Cena",950 "avatar_template": "/user_avatar/discuss.pytorch.org/giovanni_cena/{size}/16672_2.png",951 "trust_level": 1952 }953 }954 ]955 }956 ],957 "tags_descriptions": {},958 "fancy_title": "OOM for self.attention mechanism",959 "id": 88608,960 "title": "OOM for self.attention mechanism",961 "posts_count": 2,962 "created_at": "2020-07-09T20:41:44.393Z",963 "views": 602,964 "reply_count": 0,965 "like_count": 0,966 "last_posted_at": "2020-07-11T23:25:24.152Z",967 "visible": true,968 "closed": false,969 "archived": false,970 "has_summary": false,971 "archetype": "regular",972 "slug": "oom-for-self-attention-mechanism",973 "category_id": 1,974 "word_count": 135,975 "deleted_at": null,976 "user_id": 34115,977 "featured_link": null,978 "pinned_globally": false,979 "pinned_at": null,980 "pinned_until": null,981 "image_url": null,982 "slow_mode_seconds": 0,983 "draft": null,984 "draft_key": "topic_88608",985 "draft_sequence": null,986 "unpinned": null,987 "pinned": false,988 "current_post_number": 1,989 "highest_post_number": 2,990 "deleted_by": null,991 "actions_summary": [992 {993 "id": 4,994 "count": 0,995 "hidden": false,996 "can_act": false997 },998 {999 "id": 8,1000 "count": 0,1001 "hidden": false,1002 "can_act": false1003 },1004 {1005 "id": 10,1006 "count": 0,1007 "hidden": false,1008 "can_act": false1009 },1010 {1011 "id": 7,1012 "count": 0,1013 "hidden": false,1014 "can_act": false1015 }1016 ],1017 "chunk_size": 20,1018 "bookmarked": false,1019 "topic_timer": null,1020 "message_bus_last_id": 0,1021 "participant_count": 2,1022 "show_read_indicator": false,1023 "thumbnails": null,1024 "slow_mode_enabled_until": null,1025 "can_vote": false,1026 "vote_count": 0,1027 "user_voted": false,1028 "discourse_zendesk_plugin_zendesk_id": null,1029 "discourse_zendesk_plugin_zendesk_url": "https://your-url.zendesk.com/agent/tickets/",1030 "details": {1031 "can_edit": false,1032 "notification_level": 1,1033 "participants": [1034 {1035 "id": 3534,1036 "username": "ptrblck",1037 "name": "",1038 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",1039 "post_count": 1,1040 "primary_group_name": null,1041 "flair_name": null,1042 "flair_url": null,1043 "flair_color": null,1044 "flair_bg_color": null,1045 "flair_group_id": null,1046 "admin": true,1047 "moderator": true,1048 "trust_level": 21049 },1050 {1051 "id": 34115,1052 "username": "rubentorrado",1053 "name": "Ruben Rodriguez Torrado",1054 "avatar_template": "/user_avatar/discuss.pytorch.org/rubentorrado/{size}/12519_2.png",1055 "post_count": 1,1056 "primary_group_name": null,1057 "flair_name": null,1058 "flair_url": null,1059 "flair_color": null,1060 "flair_bg_color": null,1061 "flair_group_id": null,1062 "trust_level": 11063 }1064 ],1065 "created_by": {1066 "id": 34115,1067 "username": "rubentorrado",1068 "name": "Ruben Rodriguez Torrado",1069 "avatar_template": "/user_avatar/discuss.pytorch.org/rubentorrado/{size}/12519_2.png"1070 },1071 "last_poster": {1072 "id": 3534,1073 "username": "ptrblck",1074 "name": "",1075 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png"1076 }1077 },1078 "bookmarks": []1079 },1080 {1081 "post_stream": {1082 "posts": [1083 {1084 "id": 90698,1085 "name": "Wafaa Wardah",1086 "username": "Wafaa_Wardah",1087 "avatar_template": "/user_avatar/discuss.pytorch.org/wafaa_wardah/{size}/53794_2.png",1088 "created_at": "2019-02-01T12:40:26.725Z",1089 "cooked": "<p>I am using CNN for a binary classification problem and need to use Bayesian optimization to tune parameters like learning rate, number of hidden layers, optimizers, etc</p>\n<p>I have built my model with fixed learning rate and number of epochs and Adam optimizer. Can you please give me some guidance on where to go from here in order to have Bayesian optimization implemented for tuning these parameters? And how would I deal with my 3 data sets (training, validation and test)?</p>\n<p>I will really appreciate your help.<br>\nThanks.</p>",1090 "post_number": 1,1091 "post_type": 1,1092 "posts_count": 8,1093 "updated_at": "2019-02-01T12:40:26.725Z",1094 "reply_count": 0,1095 "reply_to_post_number": null,1096 "quote_count": 0,1097 "incoming_link_count": 13372,1098 "reads": 416,1099 "readers_count": 415,1100 "score": 66870.2,1101 "yours": false,1102 "topic_id": 36145,1103 "topic_slug": "hyperparameter-tuning-using-bayesian-optimization",1104 "display_username": "Wafaa Wardah",1105 "primary_group_name": null,1106 "flair_name": null,1107 "flair_url": null,1108 "flair_bg_color": null,1109 "flair_color": null,1110 "flair_group_id": null,1111 "badges_granted": [],1112 "version": 1,1113 "can_edit": false,1114 "can_delete": false,1115 "can_recover": false,1116 "can_see_hidden_post": false,1117 "can_wiki": false,1118 "read": true,1119 "user_title": "",1120 "bookmarked": false,1121 "actions_summary": [1122 {1123 "id": 2,1124 "count": 11125 }1126 ],1127 "moderator": false,1128 "admin": false,1129 "staff": false,1130 "user_id": 15213,1131 "hidden": false,1132 "trust_level": 1,1133 "deleted_at": null,1134 "user_deleted": false,1135 "edit_reason": null,1136 "can_view_edit_history": true,1137 "wiki": false,1138 "post_url": "/t/hyperparameter-tuning-using-bayesian-optimization/36145/1",1139 "can_accept_answer": false,1140 "can_unaccept_answer": false,1141 "accepted_answer": false,1142 "topic_accepted_answer": true,1143 "can_vote": false1144 },1145 {1146 "id": 90711,1147 "name": "Sebastian Raschka",1148 "username": "rasbt",1149 "avatar_template": "/user_avatar/discuss.pytorch.org/rasbt/{size}/60_2.png",1150 "created_at": "2019-02-01T14:47:11.134Z",1151 "cooked": "<p>You don’t need to do anything special to perform bayesian optimization for your hyperparameter tuning when using pytorch. You could just setup a script with command line arguments like <code>--learning_rate</code>, <code>--num_layers</code> for the hyperparameters you want to tune and maybe have a second script that calls this script with the diff. hyperparameter values in your bayesian parameter optimization loop.</p>\n<p>Conceptually, you can do sth like this</p>\n<pre><code class=\"lang-auto\">for round i in max_rounds:\n output = subprocess.call('model.py', '--learning_rate', 'some_val', '--num_layers', 'some_number')\n score = # parse score from output string (e.g., your loss output)\n #optimize p(score | hyperparam values)\n # get some_val, some_number for next round\n</code></pre>\n<p>But it is probably more efficient if you use a library like hyperopt (<a href=\"https://github.com/hyperopt/hyperopt\" rel=\"nofollow noopener\">https://github.com/hyperopt/hyperopt</a>)</p>\n<p>Maybe sth like</p>\n<pre><code class=\"lang-auto\">\ndef run_model(learning_rate):\n # your model init here\n # your training here\n return loss\n\n\nimport numpy as np\nfrom hyperopt import hp, tpe, fmin\n\n# Single line bayesian optimization of polynomial function\nbest = fmin(fn=lambda x: run_model(x),\n space=hp.normal('x', 0.00001, 0.9),\n algo=tpe.suggest, \n max_evals=2000)\n</code></pre>\n<p>But given how expensive typical model training for DL is, it may take a long time …</p>",1152 "post_number": 2,1153 "post_type": 1,1154 "posts_count": 8,1155 "updated_at": "2020-01-13T17:55:36.695Z",1156 "reply_count": 0,1157 "reply_to_post_number": null,1158 "quote_count": 0,1159 "incoming_link_count": 714,1160 "reads": 408,1161 "readers_count": 407,1162 "score": 3711.6,1163 "yours": false,1164 "topic_id": 36145,1165 "topic_slug": "hyperparameter-tuning-using-bayesian-optimization",1166 "display_username": "Sebastian Raschka",1167 "primary_group_name": null,1168 "flair_name": null,1169 "flair_url": null,1170 "flair_bg_color": null,1171 "flair_color": null,1172 "flair_group_id": null,1173 "badges_granted": [],1174 "version": 1,1175 "can_edit": false,1176 "can_delete": false,1177 "can_recover": false,1178 "can_see_hidden_post": false,1179 "can_wiki": false,1180 "link_counts": [1181 {1182 "url": "https://github.com/hyperopt/hyperopt",1183 "internal": false,1184 "reflection": false,1185 "title": "GitHub - hyperopt/hyperopt: Distributed Asynchronous Hyperparameter Optimization in Python",1186 "clicks": 9481187 }1188 ],1189 "read": true,1190 "user_title": null,1191 "bookmarked": false,1192 "actions_summary": [1193 {1194 "id": 2,1195 "count": 41196 }1197 ],1198 "moderator": false,1199 "admin": false,1200 "staff": false,