Anurag1734/cuda-error-resolution-analysis
07
1[2 {3 "post_stream": {4 "posts": [5 {6 "id": 277052,7 "name": "Tue",8 "username": "tueboesen",9 "avatar_template": "/user_avatar/discuss.pytorch.org/tueboesen/{size}/17097_2.png",10 "created_at": "2021-04-13T18:11:16.171Z",11 "cooked": "<p>I have a code that looks like this:</p>\n<pre><code class=\"lang-auto\"> for epoch in range(epochs):\n aloss_t,aloss_E_t,aloss_F_t,MAE_t,Fps_t,Fts_t, t_dataload_t, t_prepare_t, t_model_t, t_backprop_t = use_model(model, dataloader_train, train=True, max_samples=1e6, optimizer=optimizer, device=device, batch_size=batch_size)\n aloss_v,aloss_E_v,aloss_F_v,MAE_v,Fps_v,Fts_v,t_dataload_v, t_prepare_v, t_model_v, t_backprop_v = use_model(model, dataloader_val, train=False, max_samples=100, optimizer=optimizer, device=device, batch_size=batch_size)\n</code></pre>\n<p>where use_model is a function that can either be used for training or validation over a dataset. The model runs fine during training and uses about 70% of my gpu memory with batchsize=40, however when it exists the training loop and switches to validation the memory does not get freed, and hence it runs out of memory when running the validation.</p>\n<p>I thought the memory would be automatically freed when I left the training function, but apparently not, how do I ensure that this happens?</p>",12 "post_number": 1,13 "post_type": 1,14 "posts_count": 3,15 "updated_at": "2021-04-13T18:12:02.545Z",16 "reply_count": 0,17 "reply_to_post_number": null,18 "quote_count": 0,19 "incoming_link_count": 776,20 "reads": 22,21 "readers_count": 21,22 "score": 3879.4,23 "yours": false,24 "topic_id": 118065,25 "topic_slug": "exiting-training-function-does-not-free-memory-for-validation-which-runs-out-of-memory",26 "display_username": "Tue",27 "primary_group_name": null,28 "flair_name": null,29 "flair_url": null,30 "flair_bg_color": null,31 "flair_color": null,32 "flair_group_id": null,33 "badges_granted": [],34 "version": 1,35 "can_edit": false,36 "can_delete": false,37 "can_recover": false,38 "can_see_hidden_post": false,39 "can_wiki": false,40 "read": true,41 "user_title": null,42 "bookmarked": false,43 "actions_summary": [],44 "moderator": false,45 "admin": false,46 "staff": false,47 "user_id": 20011,48 "hidden": false,49 "trust_level": 1,50 "deleted_at": null,51 "user_deleted": false,52 "edit_reason": null,53 "can_view_edit_history": true,54 "wiki": false,55 "post_url": "/t/exiting-training-function-does-not-free-memory-for-validation-which-runs-out-of-memory/118065/1",56 "can_accept_answer": false,57 "can_unaccept_answer": false,58 "accepted_answer": false,59 "topic_accepted_answer": null,60 "can_vote": false61 },62 {63 "id": 277071,64 "name": "shoh",65 "username": "sio277",66 "avatar_template": "/user_avatar/discuss.pytorch.org/sio277/{size}/37684_2.png",67 "created_at": "2021-04-13T20:34:04.646Z",68 "cooked": "<p>Python object defined in local scope is freed outing the scope (no pointer to it). Check if your GPU tensors are saved to some objects in a global scope while/after training. Note that even if you remove Torch GPU tensors, the memory is not released to OS but kept in a pool, for a faster realloc of the future tensors (<a href=\"https://discuss.pytorch.org/t/about-torch-cuda-empty-cache/34232/2\" class=\"inline-onebox\">About torch.cuda.empty_cache() - #2 by albanD</a>). So even if your nvidia-smi memory usage looks full, but is still available.</p>",69 "post_number": 2,70 "post_type": 1,71 "posts_count": 3,72 "updated_at": "2021-05-05T04:30:47.258Z",73 "reply_count": 1,74 "reply_to_post_number": null,75 "quote_count": 0,76 "incoming_link_count": 8,77 "reads": 21,78 "readers_count": 20,79 "score": 49.2,80 "yours": false,81 "topic_id": 118065,82 "topic_slug": "exiting-training-function-does-not-free-memory-for-validation-which-runs-out-of-memory",83 "display_username": "shoh",84 "primary_group_name": null,85 "flair_name": null,86 "flair_url": null,87 "flair_bg_color": null,88 "flair_color": null,89 "flair_group_id": null,90 "badges_granted": [],91 "version": 3,92 "can_edit": false,93 "can_delete": false,94 "can_recover": false,95 "can_see_hidden_post": false,96 "can_wiki": false,97 "link_counts": [98 {99 "url": "https://discuss.pytorch.org/t/about-torch-cuda-empty-cache/34232/2",100 "internal": true,101 "reflection": false,102 "title": "About torch.cuda.empty_cache()",103 "clicks": 41104 }105 ],106 "read": true,107 "user_title": "",108 "bookmarked": false,109 "actions_summary": [],110 "moderator": false,111 "admin": false,112 "staff": false,113 "user_id": 43016,114 "hidden": false,115 "trust_level": 2,116 "deleted_at": null,117 "user_deleted": false,118 "edit_reason": null,119 "can_view_edit_history": true,120 "wiki": false,121 "post_url": "/t/exiting-training-function-does-not-free-memory-for-validation-which-runs-out-of-memory/118065/2",122 "can_accept_answer": false,123 "can_unaccept_answer": false,124 "accepted_answer": false,125 "topic_accepted_answer": null126 },127 {128 "id": 277283,129 "name": "Tue",130 "username": "tueboesen",131 "avatar_template": "/user_avatar/discuss.pytorch.org/tueboesen/{size}/17097_2.png",132 "created_at": "2021-04-14T16:34:09.463Z",133 "cooked": "<p>Based on your post I went back and took another look at it, and I narrowed down my issue. I was wrong in the memory not being freed it is instead the eval mode of my code that is requiring more memory than my training mode.</p>\n<p>The problem is that in eval mode I’m still running the code with autograd, which is needed since I explicitly need to get gradient information even in eval mode. However I guess this gradient information is accumulating, so I need a way to zero it after each iteration.<br>\nUsing zero_grad on the optimizer doesn’t solve the problem, guess I need to use it on each variable individually</p>",134 "post_number": 3,135 "post_type": 1,136 "posts_count": 3,137 "updated_at": "2021-04-14T16:37:49.037Z",138 "reply_count": 0,139 "reply_to_post_number": 2,140 "quote_count": 0,141 "incoming_link_count": 7,142 "reads": 20,143 "readers_count": 19,144 "score": 39.0,145 "yours": false,146 "topic_id": 118065,147 "topic_slug": "exiting-training-function-does-not-free-memory-for-validation-which-runs-out-of-memory",148 "display_username": "Tue",149 "primary_group_name": null,150 "flair_name": null,151 "flair_url": null,152 "flair_bg_color": null,153 "flair_color": null,154 "flair_group_id": null,155 "badges_granted": [],156 "version": 2,157 "can_edit": false,158 "can_delete": false,159 "can_recover": false,160 "can_see_hidden_post": false,161 "can_wiki": false,162 "read": true,163 "user_title": null,164 "reply_to_user": {165 "id": 43016,166 "username": "sio277",167 "name": "shoh",168 "avatar_template": "/user_avatar/discuss.pytorch.org/sio277/{size}/37684_2.png"169 },170 "bookmarked": false,171 "actions_summary": [],172 "moderator": false,173 "admin": false,174 "staff": false,175 "user_id": 20011,176 "hidden": false,177 "trust_level": 1,178 "deleted_at": null,179 "user_deleted": false,180 "edit_reason": null,181 "can_view_edit_history": true,182 "wiki": false,183 "post_url": "/t/exiting-training-function-does-not-free-memory-for-validation-which-runs-out-of-memory/118065/3",184 "can_accept_answer": false,185 "can_unaccept_answer": false,186 "accepted_answer": false,187 "topic_accepted_answer": null188 }189 ],190 "stream": [191 277052,192 277071,193 277283194 ]195 },196 "timeline_lookup": [197 [198 1,199 1656200 ],201 [202 3,203 1655204 ]205 ],206 "suggested_topics": [207 {208 "fancy_title": "How does PyTorch’s cross-entropy loss transform logits with a soft probability target vector?",209 "id": 218037,210 "title": "How does PyTorch's cross-entropy loss transform logits with a soft probability target vector?",211 "slug": "how-does-pytorchs-cross-entropy-loss-transform-logits-with-a-soft-probability-target-vector",212 "posts_count": 2,213 "reply_count": 0,214 "highest_post_number": 2,215 "image_url": null,216 "created_at": "2025-03-19T20:18:01.440Z",217 "last_posted_at": "2025-03-20T00:12:00.990Z",218 "bumped": true,219 "bumped_at": "2025-03-20T00:12:00.990Z",220 "archetype": "regular",221 "unseen": false,222 "pinned": false,223 "unpinned": null,224 "visible": true,225 "closed": false,226 "archived": false,227 "bookmarked": null,228 "liked": null,229 "tags_descriptions": {},230 "like_count": 1,231 "views": 79,232 "category_id": 1,233 "featured_link": null,234 "has_accepted_answer": true,235 "posters": [236 {237 "extras": null,238 "description": "Original Poster",239 "user": {240 "id": 83365,241 "username": "N_Beladel",242 "name": "Nassim Beladel",243 "avatar_template": "/user_avatar/discuss.pytorch.org/n_beladel/{size}/76250_2.png",244 "trust_level": 0245 }246 },247 {248 "extras": "latest",249 "description": "Most Recent Poster, Accepted Answer",250 "user": {251 "id": 18088,252 "username": "KFrank",253 "name": "K. Frank",254 "avatar_template": "/letter_avatar_proxy/v4/letter/k/ecb155/{size}.png",255 "trust_level": 2256 }257 }258 ]259 },260 {261 "fancy_title": "Understanding Encoder-Decoder Transformer Architecture in Image Captioning",262 "id": 215322,263 "title": "Understanding Encoder-Decoder Transformer Architecture in Image Captioning",264 "slug": "understanding-encoder-decoder-transformer-architecture-in-image-captioning",265 "posts_count": 1,266 "reply_count": 0,267 "highest_post_number": 1,268 "image_url": null,269 "created_at": "2025-01-13T11:28:56.543Z",270 "last_posted_at": "2025-01-13T11:28:56.633Z",271 "bumped": true,272 "bumped_at": "2025-01-13T11:28:56.633Z",273 "archetype": "regular",274 "unseen": false,275 "pinned": false,276 "unpinned": null,277 "visible": true,278 "closed": false,279 "archived": false,280 "bookmarked": null,281 "liked": null,282 "tags_descriptions": {},283 "like_count": 0,284 "views": 82,285 "category_id": 1,286 "featured_link": null,287 "has_accepted_answer": false,288 "posters": [289 {290 "extras": "latest single",291 "description": "Original Poster, Most Recent Poster",292 "user": {293 "id": 80254,294 "username": "sriramgs",295 "name": "Sriram",296 "avatar_template": "/letter_avatar_proxy/v4/letter/s/c0e974/{size}.png",297 "trust_level": 1298 }299 }300 ]301 },302 {303 "fancy_title": "Limiting PyTorch Worker Thread Memory Usage",304 "id": 216585,305 "title": "Limiting PyTorch Worker Thread Memory Usage",306 "slug": "limiting-pytorch-worker-thread-memory-usage",307 "posts_count": 1,308 "reply_count": 0,309 "highest_post_number": 1,310 "image_url": null,311 "created_at": "2025-02-12T16:21:25.234Z",312 "last_posted_at": "2025-02-12T16:21:25.284Z",313 "bumped": true,314 "bumped_at": "2025-02-12T16:21:25.284Z",315 "archetype": "regular",316 "unseen": false,317 "pinned": false,318 "unpinned": null,319 "visible": true,320 "closed": false,321 "archived": false,322 "bookmarked": null,323 "liked": null,324 "tags_descriptions": {},325 "like_count": 0,326 "views": 58,327 "category_id": 1,328 "featured_link": null,329 "has_accepted_answer": false,330 "posters": [331 {332 "extras": "latest single",333 "description": "Original Poster, Most Recent Poster",334 "user": {335 "id": 82544,336 "username": "saff",337 "name": "",338 "avatar_template": "/user_avatar/discuss.pytorch.org/saff/{size}/75527_2.png",339 "trust_level": 1340 }341 }342 ]343 },344 {345 "fancy_title": "Using BatchNorm1d for standardization",346 "id": 214473,347 "title": "Using BatchNorm1d for standardization",348 "slug": "using-batchnorm1d-for-standardization",349 "posts_count": 2,350 "reply_count": 0,351 "highest_post_number": 2,352 "image_url": null,353 "created_at": "2024-12-20T23:17:37.293Z",354 "last_posted_at": "2024-12-20T23:21:22.922Z",355 "bumped": true,356 "bumped_at": "2024-12-20T23:21:22.922Z",357 "archetype": "regular",358 "unseen": false,359 "pinned": false,360 "unpinned": null,361 "visible": true,362 "closed": false,363 "archived": false,364 "bookmarked": null,365 "liked": null,366 "tags_descriptions": {},367 "like_count": 0,368 "views": 35,369 "category_id": 1,370 "featured_link": null,371 "has_accepted_answer": false,372 "posters": [373 {374 "extras": null,375 "description": "Original Poster",376 "user": {377 "id": 65042,378 "username": "ado_sar",379 "name": "ado sar",380 "avatar_template": "/user_avatar/discuss.pytorch.org/ado_sar/{size}/59241_2.png",381 "trust_level": 2382 }383 },384 {385 "extras": "latest",386 "description": "Most Recent Poster",387 "user": {388 "id": 41396,389 "username": "soulitzer",390 "name": "",391 "avatar_template": "/letter_avatar_proxy/v4/letter/s/839c29/{size}.png",392 "trust_level": 2393 }394 }395 ]396 },397 {398 "fancy_title": "Load the (model.pt) built in c++",399 "id": 218510,400 "title": "Load the (model.pt) built in c++",401 "slug": "load-the-model-pt-built-in-c",402 "posts_count": 2,403 "reply_count": 1,404 "highest_post_number": 3,405 "image_url": null,406 "created_at": "2025-04-02T02:59:11.240Z",407 "last_posted_at": "2025-04-02T09:12:36.426Z",408 "bumped": true,409 "bumped_at": "2025-04-02T09:12:36.426Z",410 "archetype": "regular",411 "unseen": false,412 "pinned": false,413 "unpinned": null,414 "visible": true,415 "closed": false,416 "archived": false,417 "bookmarked": null,418 "liked": null,419 "tags_descriptions": {},420 "like_count": 0,421 "views": 38,422 "category_id": 1,423 "featured_link": null,424 "has_accepted_answer": true,425 "posters": [426 {427 "extras": "latest single",428 "description": "Original Poster, Most Recent Poster, Accepted Answer",429 "user": {430 "id": 83566,431 "username": "Hits",432 "name": "",433 "avatar_template": "/letter_avatar_proxy/v4/letter/h/779978/{size}.png",434 "trust_level": 1435 }436 }437 ]438 }439 ],440 "tags_descriptions": {},441 "fancy_title": "Exiting Training function, does not free memory for validation, which runs out of memory",442 "id": 118065,443 "title": "Exiting Training function, does not free memory for validation, which runs out of memory",444 "posts_count": 3,445 "created_at": "2021-04-13T18:11:16.113Z",446 "views": 1745,447 "reply_count": 1,448 "like_count": 0,449 "last_posted_at": "2021-04-14T16:34:09.463Z",450 "visible": true,451 "closed": false,452 "archived": false,453 "has_summary": false,454 "archetype": "regular",455 "slug": "exiting-training-function-does-not-free-memory-for-validation-which-runs-out-of-memory",456 "category_id": 1,457 "word_count": 349,458 "deleted_at": null,459 "user_id": 20011,460 "featured_link": null,461 "pinned_globally": false,462 "pinned_at": null,463 "pinned_until": null,464 "image_url": null,465 "slow_mode_seconds": 0,466 "draft": null,467 "draft_key": "topic_118065",468 "draft_sequence": null,469 "unpinned": null,470 "pinned": false,471 "current_post_number": 1,472 "highest_post_number": 3,473 "deleted_by": null,474 "actions_summary": [475 {476 "id": 4,477 "count": 0,478 "hidden": false,479 "can_act": false480 },481 {482 "id": 8,483 "count": 0,484 "hidden": false,485 "can_act": false486 },487 {488 "id": 10,489 "count": 0,490 "hidden": false,491 "can_act": false492 },493 {494 "id": 7,495 "count": 0,496 "hidden": false,497 "can_act": false498 }499 ],500 "chunk_size": 20,501 "bookmarked": false,502 "topic_timer": null,503 "message_bus_last_id": 0,504 "participant_count": 2,505 "show_read_indicator": false,506 "thumbnails": null,507 "slow_mode_enabled_until": null,508 "can_vote": false,509 "vote_count": 0,510 "user_voted": false,511 "discourse_zendesk_plugin_zendesk_id": null,512 "discourse_zendesk_plugin_zendesk_url": "https://your-url.zendesk.com/agent/tickets/",513 "details": {514 "can_edit": false,515 "notification_level": 1,516 "participants": [517 {518 "id": 20011,519 "username": "tueboesen",520 "name": "Tue",521 "avatar_template": "/user_avatar/discuss.pytorch.org/tueboesen/{size}/17097_2.png",522 "post_count": 2,523 "primary_group_name": null,524 "flair_name": null,525 "flair_url": null,526 "flair_color": null,527 "flair_bg_color": null,528 "flair_group_id": null,529 "trust_level": 1530 },531 {532 "id": 43016,533 "username": "sio277",534 "name": "shoh",535 "avatar_template": "/user_avatar/discuss.pytorch.org/sio277/{size}/37684_2.png",536 "post_count": 1,537 "primary_group_name": null,538 "flair_name": null,539 "flair_url": null,540 "flair_color": null,541 "flair_bg_color": null,542 "flair_group_id": null,543 "trust_level": 2544 }545 ],546 "created_by": {547 "id": 20011,548 "username": "tueboesen",549 "name": "Tue",550 "avatar_template": "/user_avatar/discuss.pytorch.org/tueboesen/{size}/17097_2.png"551 },552 "last_poster": {553 "id": 20011,554 "username": "tueboesen",555 "name": "Tue",556 "avatar_template": "/user_avatar/discuss.pytorch.org/tueboesen/{size}/17097_2.png"557 },558 "links": [559 {560 "url": "https://discuss.pytorch.org/t/about-torch-cuda-empty-cache/34232/2",561 "title": "About torch.cuda.empty_cache()",562 "internal": true,563 "attachment": false,564 "reflection": false,565 "clicks": 41,566 "user_id": 43016,567 "domain": "discuss.pytorch.org",568 "root_domain": "pytorch.org"569 }570 ]571 },572 "bookmarks": []573 },574 {575 "post_stream": {576 "posts": [577 {578 "id": 277282,579 "name": "",580 "username": "spadel",581 "avatar_template": "/letter_avatar_proxy/v4/letter/s/8797f3/{size}.png",582 "created_at": "2021-04-14T16:32:59.242Z",583 "cooked": "<p>I usually like the PyTorch Documentation but I think for DistributedDataParallel it is terrible. I already cancelled several approaches to make it work and at the moment I am really depending on it. So some help would be amazing!</p>\n<p>In all the tutorials I have seen so far, they never showed a way to specify the GPU’s I want to use. I am working on a server with 10 GPU’s, but they are not mine and I usually can only use 4 when I am lucky. So I somehow need to specify which exact GPU’s I would to use - this is covered nowhere in the tutorials.</p>\n<p>In the official dokumentation [<a href=\"https://pytorch.org/docs/master/generated/torch.nn.parallel.DistributedDataParallel.html\" class=\"inline-onebox\" rel=\"noopener nofollow ugc\">DistributedDataParallel — PyTorch master documentation</a>] it says as follows:</p>\n<pre><code class=\"lang-auto\">torch.cuda.set_device(i)\ntorch.distributed.init_process_group(\nbackend='nccl', world_size=N, init_method='...')\nmodel = DistributedDataParallel(model, device_ids=[i], output_device=i)\n</code></pre>\n<p>It says further:</p>\n<blockquote>\n<p>In order to spawn up multiple processes per node, you can use either <code>torch.distributed.launch</code> or <code>torch.multiprocessing.spawn</code> .</p>\n</blockquote>\n<p>But the documentation over there is even more complicated… Is there no easier way to do it? Can someone please help me out here or refer to a useful example? Thanks a lot!!</p>",584 "post_number": 1,585 "post_type": 1,586 "posts_count": 1,587 "updated_at": "2021-04-14T16:34:21.664Z",588 "reply_count": 0,589 "reply_to_post_number": null,590 "quote_count": 0,591 "incoming_link_count": 12,592 "reads": 5,593 "readers_count": 4,594 "score": 61.0,595 "yours": false,596 "topic_id": 118203,597 "topic_slug": "example-for-distributeddataparallel",598 "display_username": "",599 "primary_group_name": null,600 "flair_name": null,601 "flair_url": null,602 "flair_bg_color": null,603 "flair_color": null,604 "flair_group_id": null,605 "badges_granted": [],606 "version": 1,607 "can_edit": false,608 "can_delete": false,609 "can_recover": false,610 "can_see_hidden_post": false,611 "can_wiki": false,612 "link_counts": [613 {614 "url": "https://pytorch.org/docs/master/generated/torch.nn.parallel.DistributedDataParallel.html",615 "internal": false,616 "reflection": false,617 "title": "DistributedDataParallel — PyTorch master documentation",618 "clicks": 0619 }620 ],621 "read": true,622 "user_title": null,623 "bookmarked": false,624 "actions_summary": [],625 "moderator": false,626 "admin": false,627 "staff": false,628 "user_id": 29989,629 "hidden": false,630 "trust_level": 1,631 "deleted_at": null,632 "user_deleted": false,633 "edit_reason": null,634 "can_view_edit_history": true,635 "wiki": false,636 "post_url": "/t/example-for-distributeddataparallel/118203/1",637 "can_accept_answer": false,638 "can_unaccept_answer": false,639 "accepted_answer": false,640 "topic_accepted_answer": null,641 "can_vote": false642 }643 ],644 "stream": [645 277282646 ]647 },648 "timeline_lookup": [649 [650 1,651 1655652 ]653 ],654 "suggested_topics": [655 {656 "fancy_title": "StyleGAN2 Training Illegal Instruction on cache clear instruction",657 "id": 214804,658 "title": "StyleGAN2 Training Illegal Instruction on cache clear instruction",659 "slug": "stylegan2-training-illegal-instruction-on-cache-clear-instruction",660 "posts_count": 1,661 "reply_count": 0,662 "highest_post_number": 1,663 "image_url": null,664 "created_at": "2024-12-30T22:41:09.300Z",665 "last_posted_at": "2024-12-30T22:41:09.340Z",666 "bumped": true,667 "bumped_at": "2024-12-30T22:41:09.340Z",668 "archetype": "regular",669 "unseen": false,670 "pinned": false,671 "unpinned": null,672 "visible": true,673 "closed": false,674 "archived": false,675 "bookmarked": null,676 "liked": null,677 "tags_descriptions": {},678 "like_count": 0,679 "views": 31,680 "category_id": 1,681 "featured_link": null,682 "has_accepted_answer": false,683 "posters": [684 {685 "extras": "latest single",686 "description": "Original Poster, Most Recent Poster",687 "user": {688 "id": 76733,689 "username": "YM2132",690 "name": "",691 "avatar_template": "/user_avatar/discuss.pytorch.org/ym2132/{size}/70814_2.png",692 "trust_level": 1693 }694 }695 ]696 },697 {698 "fancy_title": "Return type of the closure in optimizer.step method",699 "id": 219112,700 "title": "Return type of the closure in optimizer.step method",701 "slug": "return-type-of-the-closure-in-optimizer-step-method",702 "posts_count": 1,703 "reply_count": 0,704 "highest_post_number": 1,705 "image_url": null,706 "created_at": "2025-04-15T15:55:29.901Z",707 "last_posted_at": "2025-04-15T15:55:29.940Z",708 "bumped": true,709 "bumped_at": "2025-04-15T15:55:29.940Z",710 "archetype": "regular",711 "unseen": false,712 "pinned": false,713 "unpinned": null,714 "visible": true,715 "closed": false,716 "archived": false,717 "bookmarked": null,718 "liked": null,719 "tags_descriptions": {},720 "like_count": 0,721 "views": 9,722 "category_id": 1,723 "featured_link": null,724 "has_accepted_answer": false,725 "posters": [726 {727 "extras": "latest single",728 "description": "Original Poster, Most Recent Poster",729 "user": {730 "id": 83830,731 "username": "arampatzis",732 "name": "Georgios",733 "avatar_template": "/user_avatar/discuss.pytorch.org/arampatzis/{size}/76639_2.png",734 "trust_level": 0735 }736 }737 ]738 },739 {740 "fancy_title": "Bf16 and tesla V100 32GB",741 "id": 216167,742 "title": "Bf16 and tesla V100 32GB",743 "slug": "bf16-and-tesla-v100-32gb",744 "posts_count": 3,745 "reply_count": 1,746 "highest_post_number": 3,747 "image_url": null,748 "created_at": "2025-02-03T08:20:28.823Z",749 "last_posted_at": "2025-02-04T11:43:13.185Z",750 "bumped": true,751 "bumped_at": "2025-02-04T11:43:13.185Z",752 "archetype": "regular",753 "unseen": false,754 "pinned": false,755 "unpinned": null,756 "visible": true,757 "closed": false,758 "archived": false,759 "bookmarked": null,760 "liked": null,761 "tags_descriptions": {},762 "like_count": 0,763 "views": 237,764 "category_id": 1,765 "featured_link": null,766 "has_accepted_answer": false,767 "posters": [768 {769 "extras": "latest",770 "description": "Original Poster, Most Recent Poster",771 "user": {772 "id": 82364,773 "username": "Sourabh_Yadav",774 "name": "Sourabh Yadav",775 "avatar_template": "/user_avatar/discuss.pytorch.org/sourabh_yadav/{size}/75350_2.png",776 "trust_level": 1777 }778 },779 {780 "extras": null,781 "description": "Frequent Poster",782 "user": {783 "id": 3534,784 "username": "ptrblck",785 "name": "",786 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",787 "admin": true,788 "moderator": true,789 "trust_level": 2790 }791 }792 ]793 },794 {795 "fancy_title": "5070Ti+Ubuntu 20.04.6+cuda?",796 "id": 217901,797 "title": "5070Ti+Ubuntu 20.04.6+cuda?",798 "slug": "5070ti-ubuntu-20-04-6-cuda",799 "posts_count": 3,800 "reply_count": 1,801 "highest_post_number": 3,802 "image_url": null,803 "created_at": "2025-03-16T03:22:30.520Z",804 "last_posted_at": "2025-03-19T10:15:03.040Z",805 "bumped": true,806 "bumped_at": "2025-03-19T10:15:03.040Z",807 "archetype": "regular",808 "unseen": false,809 "pinned": false,810 "unpinned": null,811 "visible": true,812 "closed": false,813 "archived": false,814 "bookmarked": null,815 "liked": null,816 "tags_descriptions": {},817 "like_count": 0,818 "views": 273,819 "category_id": 1,820 "featured_link": null,821 "has_accepted_answer": false,822 "posters": [823 {824 "extras": "latest",825 "description": "Original Poster, Most Recent Poster",826 "user": {827 "id": 83301,828 "username": "riva_lei",829 "name": "riva lei",830 "avatar_template": "/user_avatar/discuss.pytorch.org/riva_lei/{size}/76187_2.png",831 "trust_level": 1832 }833 },834 {835 "extras": null,836 "description": "Frequent Poster",837 "user": {838 "id": 3534,839 "username": "ptrblck",840 "name": "",841 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",842 "admin": true,843 "moderator": true,844 "trust_level": 2845 }846 }847 ]848 },849 {850 "fancy_title": "Error ‘object not callable’",851 "id": 219729,852 "title": "Error 'object not callable'",853 "slug": "error-object-not-callable",854 "posts_count": 3,855 "reply_count": 0,856 "highest_post_number": 3,857 "image_url": null,858 "created_at": "2025-05-04T07:18:19.078Z",859 "last_posted_at": "2025-05-06T05:41:24.404Z",860 "bumped": true,861 "bumped_at": "2025-05-06T05:41:24.404Z",862 "archetype": "regular",863 "unseen": false,864 "pinned": false,865 "unpinned": null,866 "visible": true,867 "closed": false,868 "archived": false,869 "bookmarked": null,870 "liked": null,871 "tags_descriptions": {},872 "like_count": 0,873 "views": 92,874 "category_id": 1,875 "featured_link": null,876 "has_accepted_answer": false,877 "posters": [878 {879 "extras": "latest",880 "description": "Original Poster, Most Recent Poster",881 "user": {882 "id": 84139,883 "username": "Abhishek_Sood",884 "name": "Abhishek Sood",885 "avatar_template": "/user_avatar/discuss.pytorch.org/abhishek_sood/{size}/76903_2.png",886 "trust_level": 0887 }888 },889 {890 "extras": null,891 "description": "Frequent Poster",892 "user": {893 "id": 18088,894 "username": "KFrank",895 "name": "K. Frank",896 "avatar_template": "/letter_avatar_proxy/v4/letter/k/ecb155/{size}.png",897 "trust_level": 2898 }899 }900 ]901 }902 ],903 "tags_descriptions": {},904 "fancy_title": "Example for DistributedDataParallel?",905 "id": 118203,906 "title": "Example for DistributedDataParallel?",907 "posts_count": 1,908 "created_at": "2021-04-14T16:32:59.186Z",909 "views": 216,910 "reply_count": 0,911 "like_count": 0,912 "last_posted_at": "2021-04-14T16:32:59.242Z",913 "visible": true,914 "closed": false,915 "archived": false,916 "has_summary": false,917 "archetype": "regular",918 "slug": "example-for-distributeddataparallel",919 "category_id": 1,920 "word_count": 203,921 "deleted_at": null,922 "user_id": 29989,923 "featured_link": null,924 "pinned_globally": false,925 "pinned_at": null,926 "pinned_until": null,927 "image_url": null,928 "slow_mode_seconds": 0,929 "draft": null,930 "draft_key": "topic_118203",931 "draft_sequence": null,932 "unpinned": null,933 "pinned": false,934 "current_post_number": 1,935 "highest_post_number": 1,936 "deleted_by": null,937 "actions_summary": [938 {939 "id": 4,940 "count": 0,941 "hidden": false,942 "can_act": false943 },944 {945 "id": 8,946 "count": 0,947 "hidden": false,948 "can_act": false949 },950 {951 "id": 10,952 "count": 0,953 "hidden": false,954 "can_act": false955 },956 {957 "id": 7,958 "count": 0,959 "hidden": false,960 "can_act": false961 }962 ],963 "chunk_size": 20,964 "bookmarked": false,965 "topic_timer": null,966 "message_bus_last_id": 0,967 "participant_count": 1,968 "show_read_indicator": false,969 "thumbnails": null,970 "slow_mode_enabled_until": null,971 "can_vote": false,972 "vote_count": 0,973 "user_voted": false,974 "discourse_zendesk_plugin_zendesk_id": null,975 "discourse_zendesk_plugin_zendesk_url": "https://your-url.zendesk.com/agent/tickets/",976 "details": {977 "can_edit": false,978 "notification_level": 1,979 "participants": [980 {981 "id": 29989,982 "username": "spadel",983 "name": "",984 "avatar_template": "/letter_avatar_proxy/v4/letter/s/8797f3/{size}.png",985 "post_count": 1,986 "primary_group_name": null,987 "flair_name": null,988 "flair_url": null,989 "flair_color": null,990 "flair_bg_color": null,991 "flair_group_id": null,992 "trust_level": 1993 }994 ],995 "created_by": {996 "id": 29989,997 "username": "spadel",998 "name": "",999 "avatar_template": "/letter_avatar_proxy/v4/letter/s/8797f3/{size}.png"1000 },1001 "last_poster": {1002 "id": 29989,1003 "username": "spadel",1004 "name": "",1005 "avatar_template": "/letter_avatar_proxy/v4/letter/s/8797f3/{size}.png"1006 }1007 },1008 "bookmarks": []1009 },1010 {1011 "post_stream": {1012 "posts": [1013 {1014 "id": 277280,1015 "name": "Rjt2nd",1016 "username": "RJT2nd",1017 "avatar_template": "/user_avatar/discuss.pytorch.org/rjt2nd/{size}/37116_2.png",1018 "created_at": "2021-04-14T16:11:48.140Z",1019 "cooked": "<p>I am getting an issue with my model using an unexpectedly large amount of RAM (around 15GB per sample when using 1 transformer block and 1 attention head). I have developed a UNet + Transformer architecture where the bottom of the UNet contains the transformer blocks. I want to use 6 transformer blocks with a 6-headed multihead attention. The attention and encoding size are both 16 and there are 1152 words. At the bottom of the UNet, the dimensions of the image has been downsampled to 64x18 with 128 channels and a batch size of 2. Before feeding into the transformer, I join the batch and channels dimensions and I also flatten the image before embedding into a 16-size vector for each word. This means that the dimensions for the inputs to the embedding layer are 256 (batches) x 1 (used to be channels) x 1152 (flattened image) x 1 and the outputs are 256 x 1 x 1152 x 16 (embedding size). This is then passed to the transformer. So far, none of these numbers stand out to me as being problematically large, though I feel like I am missing something. Is there anything that stands out as wasting or just generally consuming a lot of memory? Also, is there any way to conviently monitor peak GPU VRAM usage by module?</p>\n<p>For the convolutional portions, I am using 16, 32, 64, and 128 for the number of channels and I am starting with relatively large images as inputs (1024 x 302). The image is complex so I am using complex versions of many of the layers found here: <a href=\"https://github.com/wavefrontshaping/complexPyTorch\" class=\"inline-onebox\" rel=\"noopener nofollow ugc\">GitHub - wavefrontshaping/complexPyTorch: A high-level toolbox for using complex valued neural networks in PyTorch</a>. These are my convolutional encoder and decoder blocks for downsampling and upsampling on the UNet:</p>\n<pre><code class=\"lang-auto\">class ConvEncoderBlock(nn.Module):\n\n def __init__(self, in_channels, out_channels):\n super(ConvEncoderBlock, self).__init__()\n\n self.conv1 = nn.Sequential(ComplexConv2d(\n in_channels, out_channels, kernel_size=config.encoder_kernel_size, padding=config.encoder_padding),\n ComplexReLU())\n self.conv2 = nn.Sequential(ComplexConv2d(\n out_channels, out_channels, kernel_size=config.encoder_kernel_size, padding=config.encoder_padding),\n ComplexReLU())\n self.downsample = ComplexMaxPool2d(2)\n\n def forward(self, x):\n\n x = self.conv1(x)\n skip = self.conv2(x)\n x = self.downsample(skip)\n return x, skip\n\nclass DecoderBlock(nn.Module):\n\n def __init__(self, in_channels, out_channels):\n super(DecoderBlock, self).__init__()\n\n upsample_channels = in_channels * 2 // 3\n skip_channels = in_channels * 2 // 3\n\n self.upsample = ComplexConvTranspose2d(\n upsample_channels, upsample_channels // 2, kernel_size=2, stride=2)\n\n upsample_channels = upsample_channels // 2\n\n self.conv1 = nn.Sequential(ComplexConv2d(upsample_channels + skip_channels, out_channels, kernel_size=config.decoder_kernel_size, stride=1, padding=config.decoder_padding),\n ComplexReLU())\n self.conv2 = nn.Sequential(ComplexConv2d(out_channels, out_channels, kernel_size=config.decoder_kernel_size, stride=1, padding=config.decoder_padding),\n ComplexReLU())\n\n def forward(self, x, skip):\n\n # print('in decoder block', x.shape, skip.shape)\n\n # input: [batch, channel, freq, time]\n x = self.upsample(x) # [batch, channel//2, freq*2, time*2]\n\n # print('catting', skip.shape, x.shape)\n x = cat(skip, x, dimension=1)\n\n # print('feeding into conv1', x.shape)\n x = self.conv1(x) # [batch, channel, freq*2, time*2]\n x = self.conv2(x) # [batch, channel, freq*2, time*2]\n\n # print('out of convolutionals', x.shape)\n\n return x\n</code></pre>\n<p>There are my transformer components:</p>\n<pre><code class=\"lang-auto\">class Embedding(nn.Module):\n\n def __init__(self, params, in_channels=1):\n super(Embedding, self).__init__()\n\n self.patch_embeddings = ComplexConv2d(\n in_channels, config.encoding_size, kernel_size=config.patch_size, stride=config.patch_size)\n self.positional_embeddings = nn.Parameter(torch.zeros(\n 1, in_channels, config.num_patches, config.encoding_size, 2))\n\n # shape of x: [batch, channel, f, w] :=> type(torch.complex64)\n def forward(self, x):\n\n # print('embedding input shape', x.shape)\n\n # [batch * channels, 1, f, w] :=> type(torch.complex64)\n x = self.patch_embeddings(x)\n x = x.permute(0, 3, 2, 1)\n # [batch * channels, encoding_size, patches_per_column * w, 1] :=> type(torch.complex64)\n\n # [batch * channels, 1, patches_per_column * w, encoding_size] :=> type(torch.complex64)\n # x = x.permute(0, 2, 3, 1)\n # print('after permute', x.shape)\n\n x = torch.view_as_real(x)\n\n x = x + self.positional_embeddings\n\n x = torch.view_as_complex(x)\n\n # [batch * channels, 1, patches_per_column * w, encoding_size] :=> type(torch.complex64)\n return x\n\n\nclass AttentionHead(nn.Module):\n\n def __init__(self, in_channels=2, out_channels=1):\n super(AttentionHead, self).__init__()\n\n #self.num_heads = config.num_heads\n\n self.keys = ComplexLinear(\n config.encoding_size, config.attention_size, bias=config.attention_bias)\n self.queries = ComplexLinear(\n config.encoding_size, config.attention_size, bias=config.attention_bias)\n self.values = ComplexLinear(\n config.encoding_size, config.attention_size, bias=config.attention_bias)\n\n self.complex_map = nn.Conv2d(in_channels, out_channels, 3, padding=1)\n\n self.dropout = nn.Dropout(config.dropout_rate)\n\n def forward(self, x):\n\n keys = self.keys(x)\n queries = self.queries(x)\n values = self.values(x)\n\n scores = complex_matmul(queries, keys.transpose(-1, -2))\n scores /= config.attention_size ** 0.5\n\n scores = torch.view_as_real(scores)\n scores = scores[:, 0, :, :]\n scores = scores.permute(0, 3, 1, 2)\n scores = self.complex_map(scores)\n scores = nn.Softmax(dim=-1)(scores)\n\n scores = self.dropout(scores)\n\n scores = torch.complex(scores, torch.zeros_like(scores))\n\n #print('after attention', complex_matmul(scores, values))\n\n return complex_matmul(scores, values)\n\n\nclass MSA(nn.Module):\n def __init__(self, params):\n super(MSA, self).__init__()\n\n self.device = params[\"device\"]\n\n self.heads = nn.ModuleList([AttentionHead(2, 1)\n for _ in range(config.num_heads)])\n\n self.w = ComplexLinear(config.attention_size * config.num_heads,\n config.encoding_size, bias=config.attention_bias)\n\n self.dropout = ComplexDropout(config.dropout_rate)\n\n def forward(self, x):\n\n all_heads = self.heads[0](x)\n for i, head in enumerate(self.heads[1:]):\n all_heads = torch.cat((all_heads, head(x)), dim=-1)\n\n x = self.w(all_heads)\n x = self.dropout(x)\n\n #print('after MSA', x)\n\n return x\n\n\nclass MLP(nn.Module):\n def __init__(self):\n super(MLP, self).__init__()\n\n self.fc1 = ComplexLinear(config.encoding_size, config.mlp_size)\n self.fc2 = ComplexLinear(config.mlp_size, config.encoding_size)\n\n self.activation = ComplexReLU()\n\n self.dropout = ComplexDropout(config.dropout_rate)\n\n def forward(self, x):\n x = self.fc1(x)\n x = self.activation(x)\n x = self.dropout(x)\n\n x = self.fc2(x)\n x = self.dropout(x)\n\n #print('after mlp', x)\n\n return x\n\n\nclass TransformerBlock(nn.Module):\n\n def __init__(self, params):\n super(TransformerBlock, self).__init__()\n\n self.attn_norm = NaiveComplexLayerNorm(\n (params[\"num_patches\"], config.encoding_size), eps=config.norm_eps)\n self.attn = MSA(params)\n\n self.ffn_norm = NaiveComplexLayerNorm(\n (params[\"num_patches\"], config.encoding_size), eps=config.norm_eps)\n self.ffn = MLP()\n\n def forward(self, x):\n\n # print('transformer input', x.shape)\n\n h = x\n x = self.attn_norm(x)\n x = self.attn(x)\n x = x + h\n\n h = x\n x = self.ffn_norm(x)\n x = self.ffn(x)\n x = x + h\n\n # print('after transformer', x)\n\n return x\n</code></pre>\n<p>This is how I define my convolutional layers and transformer layers together in an overarching encoder class.</p>\n<pre><code class=\"lang-auto\">class Encoder(nn.Module):\n\n def __init__(self, params):\n super(Encoder, self).__init__()\n\n self.convEncoderBlock1 = ConvEncoderBlock(1, 16)\n self.convEncoderBlock2 = ConvEncoderBlock(16, 32)\n self.convEncoderBlock3 = ConvEncoderBlock(32, 64)\n self.convEncoderBlock4 = ConvEncoderBlock(64, 128)\n\n self.embedding = Embedding(params=params)\n\n self.transformers = nn.Sequential(OrderedDict(\n [(\"Block \" + str(i), TransformerBlock(params)) for i in range(config.num_transformers)]))\n\n self.unembedding = nn.Sequential(\n ComplexConv2d(config.encoding_size, 1, kernel_size=1)\n )\n\n def forward(self, x):\n\n # print('encoder input shape', x.shape)\n\n # Convolutional Layers\n x, skip1 = self.convEncoderBlock1(x)\n x, skip2 = self.convEncoderBlock2(x)\n x, skip3 = self.convEncoderBlock3(x)\n x, skip4 = self.convEncoderBlock4(x)\n\n batch_size = x.shape[0]\n freq_size = x.shape[2]\n time_size = x.shape[3]\n\n # print(\"transformer input: \", x.shape)\n\n x = x.reshape((batch_size*x.shape[1], 1, x.shape[2], x.shape[3]))\n # print(\"channels to batches: \", x.shape)\n\n batch_channels = x.shape[0]\n\n\n x = x.permute(0, 1, 3, 2).reshape(batch_channels, 1, freq_size*time_size, 1)\n # print(\"transformer words as rows: \", x.shape)\n\n\n x = self.embedding(x)\n # print(\"embedding output: \", x.shape)\n\n x = self.transformers(x)\n # print(\"transformer output: \", x.shape)\n\n\n # Unembedding\n x = x.permute(0, 3, 2, 1)\n # print(\"unembed input: \", x.shape)\n\n x = self.unembedding(x)\n # print(\"unembed output: \", x.shape)\n\n x = x.reshape(batch_channels, 1, time_size,\n freq_size).permute(0, 1, 3, 2)\n # print(\"reshape as spec: \", x.shape)\n\n x = x.reshape(\n (batch_size, x.shape[0]//batch_size, x.shape[2], x.shape[3]))\n # print(\"separate batch and channels: \", x.shape)\n\n return x, [skip1, skip2, skip3, skip4]\n</code></pre>",1020 "post_number": 1,1021 "post_type": 1,1022 "posts_count": 1,1023 "updated_at": "2021-04-14T16:11:48.140Z",1024 "reply_count": 0,1025 "reply_to_post_number": null,1026 "quote_count": 0,1027 "incoming_link_count": 177,1028 "reads": 5,1029 "readers_count": 4,1030 "score": 876.0,1031 "yours": false,1032 "topic_id": 118201,1033 "topic_slug": "unexpectantly-large-memory-usage",1034 "display_username": "Rjt2nd",1035 "primary_group_name": null,1036 "flair_name": null,1037 "flair_url": null,1038 "flair_bg_color": null,1039 "flair_color": null,1040 "flair_group_id": null,1041 "badges_granted": [],1042 "version": 1,1043 "can_edit": false,1044 "can_delete": false,1045 "can_recover": false,1046 "can_see_hidden_post": false,1047 "can_wiki": false,1048 "link_counts": [1049 {1050 "url": "https://github.com/wavefrontshaping/complexPyTorch",1051 "internal": false,1052 "reflection": false,1053 "title": "GitHub - wavefrontshaping/complexPyTorch: A high-level toolbox for using complex valued neural networks in PyTorch",1054 "clicks": 01055 }1056 ],1057 "read": true,1058 "user_title": null,1059 "bookmarked": false,1060 "actions_summary": [],1061 "moderator": false,1062 "admin": false,1063 "staff": false,1064 "user_id": 44297,1065 "hidden": false,1066 "trust_level": 1,1067 "deleted_at": null,1068 "user_deleted": false,1069 "edit_reason": null,1070 "can_view_edit_history": true,1071 "wiki": false,1072 "post_url": "/t/unexpectantly-large-memory-usage/118201/1",1073 "can_accept_answer": false,1074 "can_unaccept_answer": false,1075 "accepted_answer": false,1076 "topic_accepted_answer": null,1077 "can_vote": false1078 }1079 ],1080 "stream": [1081 2772801082 ]1083 },1084 "timeline_lookup": [1085 [1086 1,1087 16551088 ]1089 ],1090 "suggested_topics": [1091 {1092 "fancy_title": "Can’t run forward pass of WaveRNN model due to unsuccessful GPU RAM allocation",1093 "id": 219358,1094 "title": "Can't run forward pass of WaveRNN model due to unsuccessful GPU RAM allocation",1095 "slug": "cant-run-forward-pass-of-wavernn-model-due-to-unsuccessful-gpu-ram-allocation",1096 "posts_count": 4,1097 "reply_count": 2,1098 "highest_post_number": 4,1099 "image_url": null,1100 "created_at": "2025-04-23T03:28:01.943Z",1101 "last_posted_at": "2025-05-24T22:16:30.465Z",1102 "bumped": true,1103 "bumped_at": "2025-05-24T22:16:30.465Z",1104 "archetype": "regular",1105 "unseen": false,1106 "pinned": false,1107 "unpinned": null,1108 "visible": true,1109 "closed": false,1110 "archived": false,1111 "bookmarked": null,1112 "liked": null,1113 "tags_descriptions": {},1114 "like_count": 0,1115 "views": 112,1116 "category_id": 9,1117 "featured_link": null,1118 "has_accepted_answer": false,1119 "posters": [1120 {1121 "extras": "latest",1122 "description": "Original Poster, Most Recent Poster",1123 "user": {1124 "id": 83956,1125 "username": "j-silv",1126 "name": "Justin",1127 "avatar_template": "/letter_avatar_proxy/v4/letter/j/e36b37/{size}.png",1128 "trust_level": 11129 }1130 },1131 {1132 "extras": null,1133 "description": "Frequent Poster",1134 "user": {1135 "id": 9081,1136 "username": "JuanFMontesinos",1137 "name": "Juan Montesinos",1138 "avatar_template": "/user_avatar/discuss.pytorch.org/juanfmontesinos/{size}/76115_2.png",1139 "trust_level": 21140 }1141 }1142 ]1143 },1144 {1145 "fancy_title": "Stft behavior with center=False and n_fft > win_length",1146 "id": 213169,1147 "title": "Stft behavior with center=False and n_fft > win_length",1148 "slug": "stft-behavior-with-center-false-and-n-fft-win-length",1149 "posts_count": 1,1150 "reply_count": 0,1151 "highest_post_number": 1,1152 "image_url": null,1153 "created_at": "2024-11-19T20:21:19.461Z",1154 "last_posted_at": "2024-11-19T20:21:19.529Z",1155 "bumped": true,1156 "bumped_at": "2024-11-19T20:23:38.893Z",1157 "archetype": "regular",1158 "unseen": false,1159 "pinned": false,1160 "unpinned": null,1161 "visible": true,1162 "closed": false,1163 "archived": false,1164 "bookmarked": null,1165 "liked": null,1166 "tags_descriptions": {},1167 "like_count": 0,1168 "views": 58,1169 "category_id": 9,1170 "featured_link": null,1171 "has_accepted_answer": false,1172 "posters": [1173 {1174 "extras": "latest single",1175 "description": "Original Poster, Most Recent Poster",1176 "user": {1177 "id": 76225,1178 "username": "nedlrichards",1179 "name": "Ned Richards",1180 "avatar_template": "/user_avatar/discuss.pytorch.org/nedlrichards/{size}/70382_2.png",1181 "trust_level": 11182 }1183 }1184 ]1185 },1186 {1187 "fancy_title": "Torchaudio.functional.speed very slow when called repeatedly using randomly generated factors",1188 "id": 212781,1189 "title": "Torchaudio.functional.speed very slow when called repeatedly using randomly generated factors",1190 "slug": "torchaudio-functional-speed-very-slow-when-called-repeatedly-using-randomly-generated-factors",1191 "posts_count": 1,1192 "reply_count": 0,1193 "highest_post_number": 1,1194 "image_url": null,1195 "created_at": "2024-11-11T01:35:41.065Z",1196 "last_posted_at": "2024-11-11T01:35:41.142Z",1197 "bumped": true,1198 "bumped_at": "2024-11-11T01:35:41.142Z",1199 "archetype": "regular",1200 "unseen": false,