Anurag1734/cuda-error-resolution-analysis
07
1[2 {3 "post_stream": {4 "posts": [5 {6 "id": 461207,7 "name": "Janina ",8 "username": "Janina",9 "avatar_template": "/user_avatar/discuss.pytorch.org/janina/{size}/74468_2.png",10 "created_at": "2024-12-11T08:43:09.185Z",11 "cooked": "<p>I am implementing a Vision Transformer (ViT) Encoder-Decoder architecture trained within a Beta VAE framework on noisy latent codes. My encoder returns the learned mean (<code>mu</code> ) and log variance (<code>logvar</code> ) of the latent space. Despite various attempts, I haven’t achieved the expected results for my decoded samples, yet. I suspect that the issue might also be in the definition of my objective, particularly the reconstruction loss. I have come across various ways of implementing the reconstruction loss in different works and am unsure if my current approach is correct.</p>\n<p><strong>Current Implementation</strong> :<br>\nI am using <code>F.mse_loss</code> with <code>reduction='none'</code> to get the pixel-wise error, then summing the errors over the height, width, and channel dimensions to get the loss per sample. I noticed an improvement when summing over the height, width, and channel dimensions in the variance of my decoded samples. Additionally, I have been experimenting with implementing the Negative Log-Likelihood (NLL) Loss, as I am working with noisy (normally distributed) latent codes of shape <code>(B, 4, 32, 32)</code>.</p>\n<p><strong>Questions</strong>:</p>\n<ol>\n<li>Is the reconstruction loss correctly implemented for a ViT decoder?</li>\n<li>Are there any potential issues or improvements that I could consider?</li>\n</ol>\n<p><strong>Code Sample</strong>:</p>\n<pre><code class=\"lang-auto\"> def reparametrize(self, mu, logvar):\n \"\"\"Reparametrization trick to sample latent variable z ~ N(mu, sigma^2).\"\"\"\n if self.training:\n std = torch.exp(0.5 * logvar)\n eps = torch.randn_like(std) # Sample from standard normal distribution\n return eps * std + mu\n else:\n return mu\n\n def reconstruction_loss(self, x, recon_x):\n \"\"\"Reconstruction loss.\"\"\"\n if self.recon_loss_type == 'L2':\n # Standard MSE Loss (L2, pixel-wise no uncertainty)\n mse_loss = F.mse_loss(recon_x, x, reduction='none') # Pixel-wise error\n recon_loss = mse_loss.sum(dim=(1, 2, 3)) # Sum over all samples when (B, C, H, W)\n elif self.recon_loss_type == 'NLL':\n # Negative Log-Likelihood with learnable scale (models aleatoric uncertainty)\n var = torch.exp(self.scale) # Convert to variance (log ensures positive values)\n dist_normal = torch.distributions.Normal(recon_x, torch.sqrt(var)) # Learned variance\n log_pxz = dist_normal.log_prob(x)\n recon_loss = -log_pxz.sum(dim=(1, 2, 3)) # Sum over all samples when (B, C, H, W)\n else:\n raise ValueError('Undefined reconstruction loss type.')\n\n return torch.mean(recon_loss, dim=0) # Mean over the batch dimension\n\n def kl_divergence(self, mu, logvar):\n \"\"\"KL divergence between approximate posterior and prior.\"\"\"\n kld_loss = torch.mean(-0.5 * torch.sum(1 + logvar - mu ** 2 - logvar.exp(), dim=1), dim=0)\n return kld_loss\n</code></pre>",12 "post_number": 1,13 "post_type": 1,14 "posts_count": 1,15 "updated_at": "2024-12-11T08:54:44.602Z",16 "reply_count": 0,17 "reply_to_post_number": null,18 "quote_count": 0,19 "incoming_link_count": 212,20 "reads": 4,21 "readers_count": 3,22 "score": 1050.8,23 "yours": false,24 "topic_id": 214087,25 "topic_slug": "correct-implementation-of-beta-vae-reconstruction-loss-with-vit-encoder-decoder-architecture",26 "display_username": "Janina ",27 "primary_group_name": null,28 "flair_name": null,29 "flair_url": null,30 "flair_bg_color": null,31 "flair_color": null,32 "flair_group_id": null,33 "badges_granted": [],34 "version": 3,35 "can_edit": false,36 "can_delete": false,37 "can_recover": false,38 "can_see_hidden_post": false,39 "can_wiki": false,40 "read": true,41 "user_title": "",42 "bookmarked": false,43 "actions_summary": [],44 "moderator": false,45 "admin": false,46 "staff": false,47 "user_id": 81444,48 "hidden": false,49 "trust_level": 1,50 "deleted_at": null,51 "user_deleted": false,52 "edit_reason": null,53 "can_view_edit_history": true,54 "wiki": false,55 "post_url": "/t/correct-implementation-of-beta-vae-reconstruction-loss-with-vit-encoder-decoder-architecture/214087/1",56 "can_accept_answer": false,57 "can_unaccept_answer": false,58 "accepted_answer": false,59 "topic_accepted_answer": null,60 "can_vote": false61 }62 ],63 "stream": [64 46120765 ]66 },67 "timeline_lookup": [68 [69 1,70 31871 ]72 ],73 "suggested_topics": [74 {75 "fancy_title": "Reasons behind using convolutional layers of 1x1 kernel sizes",76 "id": 217058,77 "title": "Reasons behind using convolutional layers of 1x1 kernel sizes",78 "slug": "reasons-behind-using-convolutional-layers-of-1x1-kernel-sizes",79 "posts_count": 2,80 "reply_count": 0,81 "highest_post_number": 2,82 "image_url": null,83 "created_at": "2025-02-23T16:30:45.966Z",84 "last_posted_at": "2025-02-24T04:54:29.251Z",85 "bumped": true,86 "bumped_at": "2025-02-24T04:54:29.251Z",87 "archetype": "regular",88 "unseen": false,89 "pinned": false,90 "unpinned": null,91 "visible": true,92 "closed": false,93 "archived": false,94 "bookmarked": null,95 "liked": null,96 "tags_descriptions": {},97 "like_count": 0,98 "views": 64,99 "category_id": 1,100 "featured_link": null,101 "has_accepted_answer": false,102 "posters": [103 {104 "extras": null,105 "description": "Original Poster",106 "user": {107 "id": 55555,108 "username": "amirtmgr",109 "name": "amir T mgr",110 "avatar_template": "/user_avatar/discuss.pytorch.org/amirtmgr/{size}/49195_2.png",111 "trust_level": 1112 }113 },114 {115 "extras": "latest",116 "description": "Most Recent Poster",117 "user": {118 "id": 19553,119 "username": "anantguptadbl",120 "name": "Anant Gupta",121 "avatar_template": "/user_avatar/discuss.pytorch.org/anantguptadbl/{size}/17784_2.png",122 "trust_level": 2123 }124 }125 ]126 },127 {128 "fancy_title": "How to save/load PyTorch model with custom Storage pickling",129 "id": 218741,130 "title": "How to save/load PyTorch model with custom Storage pickling",131 "slug": "how-to-save-load-pytorch-model-with-custom-storage-pickling",132 "posts_count": 1,133 "reply_count": 0,134 "highest_post_number": 1,135 "image_url": null,136 "created_at": "2025-04-07T01:58:17.592Z",137 "last_posted_at": "2025-04-07T01:58:17.629Z",138 "bumped": true,139 "bumped_at": "2025-04-07T02:32:17.618Z",140 "archetype": "regular",141 "unseen": false,142 "pinned": false,143 "unpinned": null,144 "visible": true,145 "closed": false,146 "archived": false,147 "bookmarked": null,148 "liked": null,149 "tags_descriptions": {},150 "like_count": 0,151 "views": 32,152 "category_id": 1,153 "featured_link": null,154 "has_accepted_answer": false,155 "posters": [156 {157 "extras": "latest single",158 "description": "Original Poster, Most Recent Poster",159 "user": {160 "id": 83670,161 "username": "brandon.pelfrey",162 "name": "Brandon Pelfrey",163 "avatar_template": "/user_avatar/discuss.pytorch.org/brandon.pelfrey/{size}/76512_2.png",164 "trust_level": 0165 }166 }167 ]168 },169 {170 "fancy_title": "Pytorch model not learning at all",171 "id": 212471,172 "title": "Pytorch model not learning at all",173 "slug": "pytorch-model-not-learning-at-all",174 "posts_count": 1,175 "reply_count": 0,176 "highest_post_number": 1,177 "image_url": null,178 "created_at": "2024-11-03T11:14:57.075Z",179 "last_posted_at": "2024-11-03T11:14:57.125Z",180 "bumped": true,181 "bumped_at": "2024-11-03T11:16:16.811Z",182 "archetype": "regular",183 "unseen": false,184 "pinned": false,185 "unpinned": null,186 "visible": true,187 "closed": false,188 "archived": false,189 "bookmarked": null,190 "liked": null,191 "tags_descriptions": {},192 "like_count": 0,193 "views": 112,194 "category_id": 1,195 "featured_link": null,196 "has_accepted_answer": false,197 "posters": [198 {199 "extras": "latest single",200 "description": "Original Poster, Most Recent Poster",201 "user": {202 "id": 80664,203 "username": "S_Ha",204 "name": "Seoyeong Ha",205 "avatar_template": "/user_avatar/discuss.pytorch.org/s_ha/{size}/72764_2.png",206 "trust_level": 0207 }208 }209 ]210 },211 {212 "fancy_title": "Inference_mode() doesn’t remove grad",213 "id": 216739,214 "title": "Inference_mode() doesn't remove grad",215 "slug": "inference-mode-doesnt-remove-grad",216 "posts_count": 2,217 "reply_count": 0,218 "highest_post_number": 2,219 "image_url": "https://discuss.pytorch.org/uploads/default/original/3X/1/d/1d5da9ab3a51d653e4e1cc79efce5660959de40d.png",220 "created_at": "2025-02-16T12:10:09.684Z",221 "last_posted_at": "2025-02-16T15:15:50.947Z",222 "bumped": true,223 "bumped_at": "2025-02-16T15:15:50.947Z",224 "archetype": "regular",225 "unseen": false,226 "pinned": false,227 "unpinned": null,228 "visible": true,229 "closed": false,230 "archived": false,231 "bookmarked": null,232 "liked": null,233 "tags_descriptions": {},234 "like_count": 0,235 "views": 76,236 "category_id": 1,237 "featured_link": null,238 "has_accepted_answer": false,239 "posters": [240 {241 "extras": null,242 "description": "Original Poster",243 "user": {244 "id": 82719,245 "username": "blmppes",246 "name": "Trung Hiếu Đỗ",247 "avatar_template": "/user_avatar/discuss.pytorch.org/blmppes/{size}/75683_2.png",248 "trust_level": 0249 }250 },251 {252 "extras": "latest",253 "description": "Most Recent Poster",254 "user": {255 "id": 3534,256 "username": "ptrblck",257 "name": "",258 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",259 "admin": true,260 "moderator": true,261 "trust_level": 2262 }263 }264 ]265 },266 {267 "fancy_title": "Torch.split with multiple start and end indices",268 "id": 220337,269 "title": "Torch.split with multiple start and end indices",270 "slug": "torch-split-with-multiple-start-and-end-indices",271 "posts_count": 2,272 "reply_count": 0,273 "highest_post_number": 2,274 "image_url": null,275 "created_at": "2025-05-26T07:18:30.878Z",276 "last_posted_at": "2025-05-26T16:35:05.174Z",277 "bumped": true,278 "bumped_at": "2025-05-26T16:35:05.174Z",279 "archetype": "regular",280 "unseen": false,281 "pinned": false,282 "unpinned": null,283 "visible": true,284 "closed": false,285 "archived": false,286 "bookmarked": null,287 "liked": null,288 "tags_descriptions": {},289 "like_count": 2,290 "views": 49,291 "category_id": 1,292 "featured_link": null,293 "has_accepted_answer": false,294 "posters": [295 {296 "extras": null,297 "description": "Original Poster",298 "user": {299 "id": 84453,300 "username": "Nielsen",301 "name": "Nielsen",302 "avatar_template": "/user_avatar/discuss.pytorch.org/nielsen/{size}/77172_2.png",303 "trust_level": 1304 }305 },306 {307 "extras": "latest",308 "description": "Most Recent Poster",309 "user": {310 "id": 18088,311 "username": "KFrank",312 "name": "K. Frank",313 "avatar_template": "/letter_avatar_proxy/v4/letter/k/ecb155/{size}.png",314 "trust_level": 2315 }316 }317 ]318 }319 ],320 "tags_descriptions": {},321 "fancy_title": "Correct Implementation of Beta-VAE Reconstruction Loss with ViT Encoder-Decoder Architecture",322 "id": 214087,323 "title": "Correct Implementation of Beta-VAE Reconstruction Loss with ViT Encoder-Decoder Architecture",324 "posts_count": 1,325 "created_at": "2024-12-11T08:43:09.073Z",326 "views": 245,327 "reply_count": 0,328 "like_count": 0,329 "last_posted_at": "2024-12-11T08:43:09.185Z",330 "visible": true,331 "closed": false,332 "archived": false,333 "has_summary": false,334 "archetype": "regular",335 "slug": "correct-implementation-of-beta-vae-reconstruction-loss-with-vit-encoder-decoder-architecture",336 "category_id": 1,337 "word_count": 389,338 "deleted_at": null,339 "user_id": 81444,340 "featured_link": null,341 "pinned_globally": false,342 "pinned_at": null,343 "pinned_until": null,344 "image_url": null,345 "slow_mode_seconds": 0,346 "draft": null,347 "draft_key": "topic_214087",348 "draft_sequence": null,349 "unpinned": null,350 "pinned": false,351 "current_post_number": 1,352 "highest_post_number": 1,353 "deleted_by": null,354 "actions_summary": [355 {356 "id": 4,357 "count": 0,358 "hidden": false,359 "can_act": false360 },361 {362 "id": 8,363 "count": 0,364 "hidden": false,365 "can_act": false366 },367 {368 "id": 10,369 "count": 0,370 "hidden": false,371 "can_act": false372 },373 {374 "id": 7,375 "count": 0,376 "hidden": false,377 "can_act": false378 }379 ],380 "chunk_size": 20,381 "bookmarked": false,382 "topic_timer": null,383 "message_bus_last_id": 0,384 "participant_count": 1,385 "show_read_indicator": false,386 "thumbnails": null,387 "slow_mode_enabled_until": null,388 "can_vote": false,389 "vote_count": 0,390 "user_voted": false,391 "discourse_zendesk_plugin_zendesk_id": null,392 "discourse_zendesk_plugin_zendesk_url": "https://your-url.zendesk.com/agent/tickets/",393 "details": {394 "can_edit": false,395 "notification_level": 1,396 "participants": [397 {398 "id": 81444,399 "username": "Janina",400 "name": "Janina ",401 "avatar_template": "/user_avatar/discuss.pytorch.org/janina/{size}/74468_2.png",402 "post_count": 1,403 "primary_group_name": null,404 "flair_name": null,405 "flair_url": null,406 "flair_color": null,407 "flair_bg_color": null,408 "flair_group_id": null,409 "trust_level": 1410 }411 ],412 "created_by": {413 "id": 81444,414 "username": "Janina",415 "name": "Janina ",416 "avatar_template": "/user_avatar/discuss.pytorch.org/janina/{size}/74468_2.png"417 },418 "last_poster": {419 "id": 81444,420 "username": "Janina",421 "name": "Janina ",422 "avatar_template": "/user_avatar/discuss.pytorch.org/janina/{size}/74468_2.png"423 }424 },425 "bookmarks": []426 },427 {428 "post_stream": {429 "posts": [430 {431 "id": 461120,432 "name": "Idrees Bhat",433 "username": "Idrees11",434 "avatar_template": "/user_avatar/discuss.pytorch.org/idrees11/{size}/74448_2.png",435 "created_at": "2024-12-10T07:27:04.912Z",436 "cooked": "<p>Hi,</p>\n<p>I am designing encoder decoder network. In which number of convolutional layers are four and number of Transposed convolutional layers are four. My image size is : 8x 1x 256x256. 8 is batch size, 1 is number of channels ( gray image). Size is H = 256 W is 256. Kernel size is 3 x 3. Stride is 2 x2. Padding 1. However, I have to fix number of filters as 96 each layer…I don’t understand how to do that .</p>",437 "post_number": 1,438 "post_type": 1,439 "posts_count": 4,440 "updated_at": "2024-12-10T07:27:04.912Z",441 "reply_count": 1,442 "reply_to_post_number": null,443 "quote_count": 0,444 "incoming_link_count": 3,445 "reads": 10,446 "readers_count": 9,447 "score": 21.8,448 "yours": false,449 "topic_id": 214033,450 "topic_slug": "fixing-number-of-filters-in-conv2ds",451 "display_username": "Idrees Bhat",452 "primary_group_name": null,453 "flair_name": null,454 "flair_url": null,455 "flair_bg_color": null,456 "flair_color": null,457 "flair_group_id": null,458 "badges_granted": [],459 "version": 1,460 "can_edit": false,461 "can_delete": false,462 "can_recover": false,463 "can_see_hidden_post": false,464 "can_wiki": false,465 "read": true,466 "user_title": null,467 "bookmarked": false,468 "actions_summary": [],469 "moderator": false,470 "admin": false,471 "staff": false,472 "user_id": 81422,473 "hidden": false,474 "trust_level": 1,475 "deleted_at": null,476 "user_deleted": false,477 "edit_reason": null,478 "can_view_edit_history": true,479 "wiki": false,480 "post_url": "/t/fixing-number-of-filters-in-conv2ds/214033/1",481 "can_accept_answer": false,482 "can_unaccept_answer": false,483 "accepted_answer": false,484 "topic_accepted_answer": null,485 "can_vote": false486 },487 {488 "id": 461127,489 "name": "Aknw Fen",490 "username": "Aknw_Fen",491 "avatar_template": "/user_avatar/discuss.pytorch.org/aknw_fen/{size}/74156_2.png",492 "created_at": "2024-12-10T09:55:48.868Z",493 "cooked": "<p>96 filters will always make 96 channels.</p>\n<p>The first time though, the input channels will be 3, output 96. Then, it will be input 96, output 96.</p>\n<p>You should try, <a href=\"https://github.com/TylerYep/torchinfo\" rel=\"noopener nofollow ugc\">and use <code>torchinfo</code></a> to understand the dimensionality changes.</p>",494 "post_number": 2,495 "post_type": 1,496 "posts_count": 4,497 "updated_at": "2024-12-10T09:55:48.868Z",498 "reply_count": 0,499 "reply_to_post_number": null,500 "quote_count": 0,501 "incoming_link_count": 1,502 "reads": 9,503 "readers_count": 8,504 "score": 6.6,505 "yours": false,506 "topic_id": 214033,507 "topic_slug": "fixing-number-of-filters-in-conv2ds",508 "display_username": "Aknw Fen",509 "primary_group_name": null,510 "flair_name": null,511 "flair_url": null,512 "flair_bg_color": null,513 "flair_color": null,514 "flair_group_id": null,515 "badges_granted": [],516 "version": 1,517 "can_edit": false,518 "can_delete": false,519 "can_recover": false,520 "can_see_hidden_post": false,521 "can_wiki": false,522 "link_counts": [523 {524 "url": "https://github.com/TylerYep/torchinfo",525 "internal": false,526 "reflection": false,527 "title": "GitHub - TylerYep/torchinfo: View model summaries in PyTorch!",528 "clicks": 2529 }530 ],531 "read": true,532 "user_title": null,533 "bookmarked": false,534 "actions_summary": [],535 "moderator": false,536 "admin": false,537 "staff": false,538 "user_id": 81089,539 "hidden": false,540 "trust_level": 2,541 "deleted_at": null,542 "user_deleted": false,543 "edit_reason": null,544 "can_view_edit_history": true,545 "wiki": false,546 "post_url": "/t/fixing-number-of-filters-in-conv2ds/214033/2",547 "can_accept_answer": false,548 "can_unaccept_answer": false,549 "accepted_answer": false,550 "topic_accepted_answer": null551 },552 {553 "id": 461132,554 "name": "Idrees Bhat",555 "username": "Idrees11",556 "avatar_template": "/user_avatar/discuss.pytorch.org/idrees11/{size}/74448_2.png",557 "created_at": "2024-12-10T10:42:59.627Z",558 "cooked": "<p>first convltional layer will be having in_channels =1, out_channels = 96 . why since image is gray<br>\nsecond convolutional layer will be having in_channles = 96, out_channels= ?<br>\n<span class=\"hashtag-raw\">#-----------------------------------------------------------------------------------------------------</span>-#<br>\nI am having four convolutional layers… And each layer is having 96 filters…<br>\n<span class=\"hashtag-raw\">#-----------------------------------------------------------------------------------------------------</span>---#<br>\nI am not understanding, how It can be done ?</p>",559 "post_number": 3,560 "post_type": 1,561 "posts_count": 4,562 "updated_at": "2024-12-10T10:42:59.627Z",563 "reply_count": 0,564 "reply_to_post_number": null,565 "quote_count": 0,566 "incoming_link_count": 1,567 "reads": 9,568 "readers_count": 8,569 "score": 6.6,570 "yours": false,571 "topic_id": 214033,572 "topic_slug": "fixing-number-of-filters-in-conv2ds",573 "display_username": "Idrees Bhat",574 "primary_group_name": null,575 "flair_name": null,576 "flair_url": null,577 "flair_bg_color": null,578 "flair_color": null,579 "flair_group_id": null,580 "badges_granted": [],581 "version": 1,582 "can_edit": false,583 "can_delete": false,584 "can_recover": false,585 "can_see_hidden_post": false,586 "can_wiki": false,587 "read": true,588 "user_title": null,589 "bookmarked": false,590 "actions_summary": [],591 "moderator": false,592 "admin": false,593 "staff": false,594 "user_id": 81422,595 "hidden": false,596 "trust_level": 1,597 "deleted_at": null,598 "user_deleted": false,599 "edit_reason": null,600 "can_view_edit_history": true,601 "wiki": false,602 "post_url": "/t/fixing-number-of-filters-in-conv2ds/214033/3",603 "can_accept_answer": false,604 "can_unaccept_answer": false,605 "accepted_answer": false,606 "topic_accepted_answer": null607 },608 {609 "id": 461203,610 "name": "Idrees Bhat",611 "username": "Idrees11",612 "avatar_template": "/user_avatar/discuss.pytorch.org/idrees11/{size}/74448_2.png",613 "created_at": "2024-12-11T04:37:44.049Z",614 "cooked": "<aside class=\"quote no-group\" data-username=\"Idrees11\" data-post=\"1\" data-topic=\"214033\" data-full=\"true\">\n<div class=\"title\">\n<div class=\"quote-controls\"></div>\n<img loading=\"lazy\" alt=\"\" width=\"24\" height=\"24\" src=\"https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/idrees11/48/74448_2.png\" class=\"avatar\"> Idrees11:</div>\n<blockquote>\n<p>Hi,</p>\n<p>I am designing encoder decoder network. In which number of convolutional layers are four and number of Transposed convolutional layers are four. My image size is : 8x 1x 256x256. 8 is batch size, 1 is number of channels ( gray image). Size is H = 256 W is 256. Kernel size is 3 x 3. Stride is 2 x2. Padding 1. However, I have to fix number of filters as 96 each layer…I don’t understand how to do that .</p>\n</blockquote>\n</aside>\n<p>import cv2<br>\nimport torch<br>\nfrom torchvision import transforms<br>\n<span class=\"hashtag-raw\">#-----------------------------------------------------------------------------</span>#<br>\n<span class=\"hashtag-raw\">#-----------------------------------------------------------------------------</span>#<br>\nclass conv_block(nn.Module):</p>\n<pre><code>def __init__(self, in_c, out_c):\n super().__init__()\n self.conv1 = nn.Conv2d(in_c, out_c, kernel_size=3, padding=1)\n self.bn1 = nn.InstanceNorm2d(out_c)\n self.relu = nn.ReLU(inplace=True)\n \ndef forward(self, inputs):\n x = self.conv1(inputs)\n x = self.bn1(x)\n x = self.relu(x)\n return x\n</code></pre>\n<p>class encoder_block(nn.Module):</p>\n<pre><code>def __init__(self, in_c, out_c):\n super().__init__()\n self.conv = conv_block(in_c, out_c)\n self.pool = nn.MaxPool2d((2, 2))\n \ndef forward(self, inputs):\n x = self.conv(inputs)\n p = self.pool(x)\n return x, p\n</code></pre>\n<p>class decoder_block(nn.Module):</p>\n<pre><code>def __init__(self, in_c, out_c):\n super().__init__()\n self.up = nn.ConvTranspose2d(in_c, out_c, kernel_size=3, stride=2, padding=1)\n self.conv = conv_block(out_c+ out_c, out_c)\n \ndef forward(self, inputs, skip):\n x = self.up(inputs)\n x = torch.cat([x, skip], axis=1)\n x = self.conv(x)\n return x\n</code></pre>\n<p>class build_unet(nn.Module):</p>\n<pre><code>def __init__(self):\n super().__init__()\n</code></pre>\n<p><span class=\"hashtag-raw\">#-----------------------------------------------------------------------------</span>#</p>\n<h1><a name=\"p-461203-encoder-1\" class=\"anchor\" href=\"#p-461203-encoder-1\"></a>ENCODER</h1>\n<p><span class=\"hashtag-raw\">#-----------------------------------------------------------------------------</span>#<br>\nself.e1 = encoder_block(1,96)<br>\nself.e2 = encoder_block(96,96)<br>\nself.e3 = encoder_block(96,96)<br>\nself.e4 = encoder_block(96,96)<br>\n<span class=\"hashtag-raw\">#-----------------------------------------------------------------------------</span>#</p>\n<h1><a name=\"p-461203-bottleneck-2\" class=\"anchor\" href=\"#p-461203-bottleneck-2\"></a>BOTTLENECK</h1>\n<p><span class=\"hashtag-raw\">#-----------------------------------------------------------------------------</span>#<br>\nself.b = conv_block(96, 96)<br>\n<span class=\"hashtag-raw\">#-----------------------------------------------------------------------------</span>#</p>\n<h1><a name=\"p-461203-decoder-3\" class=\"anchor\" href=\"#p-461203-decoder-3\"></a>DECODER</h1>\n<p><span class=\"hashtag-raw\">#-----------------------------------------------------------------------------</span>#<br>\nself.d1 = decoder_block(96,96)<br>\nself.d2 = decoder_block(96,96)<br>\nself.d3 = decoder_block(96,96)<br>\nself.d4 = decoder_block(96, 3)<br>\n<span class=\"hashtag-raw\">#----------------------------------------------------</span>#</p>\n<h1><a name=\"p-461203-classifier-4\" class=\"anchor\" href=\"#p-461203-classifier-4\"></a>“”\" Classifier “”\"</h1>\n<p><span class=\"hashtag-raw\">#----------------------------------------------------</span>#<br>\nself.outputs = nn.Conv2d(96, 1, kernel_size=1, padding=0)<br>\n<span class=\"hashtag-raw\">#------------------------------------------------------------------------------</span>#</p>\n<h1><a name=\"p-461203-encoder-5\" class=\"anchor\" href=\"#p-461203-encoder-5\"></a>“”\" Encoder “”\"</h1>\n<p><span class=\"hashtag-raw\">#-----------------------------------------------------------------------------</span>#<br>\ndef forward(self, inputs):<br>\ns1, p1 = self.e1(inputs)<br>\ns2, p2 = self.e2(p1)<br>\ns3, p3 = self.e3(p2)<br>\ns4, p4 = self.e4(p3)<br>\n<span class=\"hashtag-raw\">#------------------------------------------------------------------------------</span>#</p>\n<h1><a name=\"p-461203-bottleneck-6\" class=\"anchor\" href=\"#p-461203-bottleneck-6\"></a>“”\" Bottleneck “”\"</h1>\n<p><span class=\"hashtag-raw\">#-----------------------------------------------------------------------------</span>#<br>\nb = self.b(p4)<br>\n<span class=\"hashtag-raw\">#------------------------------------------------------------------------------</span>#</p>\n<h1><a name=\"p-461203-decoder-7\" class=\"anchor\" href=\"#p-461203-decoder-7\"></a>“”\" Decoder “”\"</h1>\n<p><span class=\"hashtag-raw\">#------------------------------------------------------------------------------</span>#<br>\nd1 = self.d1(b, s4)<br>\nd2 = self.d2(d1, s3)<br>\nd3 = self.d3(d2, s2)<br>\nd4 = self.d4(d3, s1)<br>\n<span class=\"hashtag-raw\">#-------------------------------------------</span>#</p>\n<h1><a name=\"p-461203-classifier-8\" class=\"anchor\" href=\"#p-461203-classifier-8\"></a>“”\" Classifier “”\"</h1>\n<p><span class=\"hashtag-raw\">#-------------------------------------------</span>#<br>\noutputs = self.outputs(d4)<br>\nreturn outputs<br>\n<span class=\"hashtag-raw\">#-------------------------------------------------------------------------------</span>#</p>\n<p>if <strong>name</strong> == “<strong>main</strong>”:<br>\n# inputs = torch.randn([1,1,256,256])<br>\nimage = cv2.imread (r’C:\\Users\\Idrees Bhat\\Desktop\\Research\\Insha\\Dataset\\o_gray\\1.png’,0)<br>\nconvert_tensor = transforms.ToTensor()<br>\ninputs=convert_tensor(image)<br>\n# print (inputs.shape)<br>\n# print (type(inputs))<br>\nmodel = build_unet()<br>\ny= model (inputs)<br>\nprint(y.shape)<br>\n``<br>\nError : RuntimeError: Sizes of tensors must match except in dimension 1. Expected size 31 but got size 32 for tensor number 1 in the list.</p>\n<p>Note: Image size gray and size is [256 256]… I want fix filters per layer i.e. 96 filters per layer</p>",615 "post_number": 4,616 "post_type": 1,617 "posts_count": 4,618 "updated_at": "2024-12-11T04:37:44.049Z",619 "reply_count": 0,620 "reply_to_post_number": null,621 "quote_count": 1,622 "incoming_link_count": 0,623 "reads": 7,624 "readers_count": 6,625 "score": 1.2,626 "yours": false,627 "topic_id": 214033,628 "topic_slug": "fixing-number-of-filters-in-conv2ds",629 "display_username": "Idrees Bhat",630 "primary_group_name": null,631 "flair_name": null,632 "flair_url": null,633 "flair_bg_color": null,634 "flair_color": null,635 "flair_group_id": null,636 "badges_granted": [],637 "version": 1,638 "can_edit": false,639 "can_delete": false,640 "can_recover": false,641 "can_see_hidden_post": false,642 "can_wiki": false,643 "read": true,644 "user_title": null,645 "bookmarked": false,646 "actions_summary": [],647 "moderator": false,648 "admin": false,649 "staff": false,650 "user_id": 81422,651 "hidden": false,652 "trust_level": 1,653 "deleted_at": null,654 "user_deleted": false,655 "edit_reason": null,656 "can_view_edit_history": true,657 "wiki": false,658 "post_url": "/t/fixing-number-of-filters-in-conv2ds/214033/4",659 "can_accept_answer": false,660 "can_unaccept_answer": false,661 "accepted_answer": false,662 "topic_accepted_answer": null663 }664 ],665 "stream": [666 461120,667 461127,668 461132,669 461203670 ]671 },672 "timeline_lookup": [673 [674 1,675 319676 ]677 ],678 "suggested_topics": [679 {680 "fancy_title": "Finnetuning problems on googlenet architecture",681 "id": 218083,682 "title": "Finnetuning problems on googlenet architecture",683 "slug": "finnetuning-problems-on-googlenet-architecture",684 "posts_count": 2,685 "reply_count": 0,686 "highest_post_number": 2,687 "image_url": null,688 "created_at": "2025-03-20T17:21:53.117Z",689 "last_posted_at": "2025-03-21T19:04:52.661Z",690 "bumped": true,691 "bumped_at": "2025-03-21T19:04:52.661Z",692 "archetype": "regular",693 "unseen": false,694 "pinned": false,695 "unpinned": null,696 "visible": true,697 "closed": false,698 "archived": false,699 "bookmarked": null,700 "liked": null,701 "tags_descriptions": {},702 "like_count": 0,703 "views": 40,704 "category_id": 5,705 "featured_link": null,706 "has_accepted_answer": false,707 "posters": [708 {709 "extras": null,710 "description": "Original Poster",711 "user": {712 "id": 83388,713 "username": "Gabriel_Duran",714 "name": "Gabriel Duran",715 "avatar_template": "/user_avatar/discuss.pytorch.org/gabriel_duran/{size}/76266_2.png",716 "trust_level": 1717 }718 },719 {720 "extras": "latest",721 "description": "Most Recent Poster",722 "user": {723 "id": 3534,724 "username": "ptrblck",725 "name": "",726 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",727 "admin": true,728 "moderator": true,729 "trust_level": 2730 }731 }732 ]733 },734 {735 "fancy_title": "In GPU-memory compression",736 "id": 213987,737 "title": "In GPU-memory compression",738 "slug": "in-gpu-memory-compression",739 "posts_count": 1,740 "reply_count": 0,741 "highest_post_number": 1,742 "image_url": null,743 "created_at": "2024-12-09T11:15:32.161Z",744 "last_posted_at": "2024-12-09T11:15:32.219Z",745 "bumped": true,746 "bumped_at": "2024-12-09T11:15:32.219Z",747 "archetype": "regular",748 "unseen": false,749 "pinned": false,750 "unpinned": null,751 "visible": true,752 "closed": false,753 "archived": false,754 "bookmarked": null,755 "liked": null,756 "tags_descriptions": {},757 "like_count": 0,758 "views": 125,759 "category_id": 5,760 "featured_link": null,761 "has_accepted_answer": false,762 "posters": [763 {764 "extras": "latest single",765 "description": "Original Poster, Most Recent Poster",766 "user": {767 "id": 52896,768 "username": "Theophile_Champion",769 "name": "Theophile Champion",770 "avatar_template": "/user_avatar/discuss.pytorch.org/theophile_champion/{size}/30397_2.png",771 "trust_level": 1772 }773 }774 ]775 },776 {777 "fancy_title": "Is there a KL divergence operation supported by coreML conversion?",778 "id": 213818,779 "title": "Is there a KL divergence operation supported by coreML conversion?",780 "slug": "is-there-a-kl-divergence-operation-supported-by-coreml-conversion",781 "posts_count": 2,782 "reply_count": 0,783 "highest_post_number": 2,784 "image_url": null,785 "created_at": "2024-12-04T20:59:15.797Z",786 "last_posted_at": "2024-12-05T14:18:09.896Z",787 "bumped": true,788 "bumped_at": "2024-12-06T04:34:55.097Z",789 "archetype": "regular",790 "unseen": false,791 "pinned": false,792 "unpinned": null,793 "visible": true,794 "closed": false,795 "archived": false,796 "bookmarked": null,797 "liked": null,798 "tags_descriptions": {},799 "like_count": 0,800 "views": 35,801 "category_id": 5,802 "featured_link": null,803 "has_accepted_answer": false,804 "posters": [805 {806 "extras": null,807 "description": "Original Poster",808 "user": {809 "id": 45116,810 "username": "JimW",811 "name": "",812 "avatar_template": "/user_avatar/discuss.pytorch.org/jimw/{size}/38000_2.png",813 "trust_level": 1814 }815 },816 {817 "extras": "latest",818 "description": "Most Recent Poster",819 "user": {820 "id": 81329,821 "username": "esteban78edbrf",822 "name": "esteban lynn",823 "avatar_template": "/letter_avatar_proxy/v4/letter/e/ac8455/{size}.png",824 "trust_level": 0825 }826 }827 ]828 },829 {830 "fancy_title": "F.scaled_dot_product_attention get query @ key",831 "id": 215697,832 "title": "F.scaled_dot_product_attention get query @ key",833 "slug": "f-scaled-dot-product-attention-get-query-key",834 "posts_count": 1,835 "reply_count": 0,836 "highest_post_number": 1,837 "image_url": null,838 "created_at": "2025-01-22T05:03:27.120Z",839 "last_posted_at": "2025-01-22T05:03:27.155Z",840 "bumped": true,841 "bumped_at": "2025-01-22T05:03:27.155Z",842 "archetype": "regular",843 "unseen": false,844 "pinned": false,845 "unpinned": null,846 "visible": true,847 "closed": false,848 "archived": false,849 "bookmarked": null,850 "liked": null,851 "tags_descriptions": {},852 "like_count": 0,853 "views": 109,854 "category_id": 5,855 "featured_link": null,856 "has_accepted_answer": false,857 "posters": [858 {859 "extras": "latest single",860 "description": "Original Poster, Most Recent Poster",861 "user": {862 "id": 82230,863 "username": "b10901187",864 "name": "閎凱 鍾",865 "avatar_template": "/user_avatar/discuss.pytorch.org/b10901187/{size}/75232_2.png",866 "trust_level": 1867 }868 }869 ]870 },871 {872 "fancy_title": "Loss becomes constant afterwarmup?",873 "id": 219517,874 "title": "Loss becomes constant afterwarmup?",875 "slug": "loss-becomes-constant-afterwarmup",876 "posts_count": 2,877 "reply_count": 0,878 "highest_post_number": 2,879 "image_url": "https://discuss.pytorch.org/uploads/default/optimized/3X/d/6/d6075778b201d499c33037d473674dbaf8d2eec7_2_1024x320.png",880 "created_at": "2025-04-27T17:21:06.202Z",881 "last_posted_at": "2025-04-27T19:51:08.959Z",882 "bumped": true,883 "bumped_at": "2025-04-27T19:51:08.959Z",884 "archetype": "regular",885 "unseen": false,886 "pinned": false,887 "unpinned": null,888 "visible": true,889 "closed": false,890 "archived": false,891 "bookmarked": null,892 "liked": null,893 "tags_descriptions": {},894 "like_count": 0,895 "views": 67,896 "category_id": 5,897 "featured_link": null,898 "has_accepted_answer": false,899 "posters": [900 {901 "extras": null,902 "description": "Original Poster",903 "user": {904 "id": 81430,905 "username": "ecoArcGaming",906 "name": "",907 "avatar_template": "/user_avatar/discuss.pytorch.org/ecoarcgaming/{size}/74453_2.png",908 "trust_level": 1909 }910 },911 {912 "extras": "latest",913 "description": "Most Recent Poster",914 "user": {915 "id": 18088,916 "username": "KFrank",917 "name": "K. Frank",918 "avatar_template": "/letter_avatar_proxy/v4/letter/k/ecb155/{size}.png",919 "trust_level": 2920 }921 }922 ]923 }924 ],925 "tags_descriptions": {},926 "fancy_title": "Fixing number of filters in Conv2ds",927 "id": 214033,928 "title": "Fixing number of filters in Conv2ds",929 "posts_count": 4,930 "created_at": "2024-12-10T07:27:04.845Z",931 "views": 187,932 "reply_count": 0,933 "like_count": 0,934 "last_posted_at": "2024-12-11T04:37:44.049Z",935 "visible": true,936 "closed": false,937 "archived": false,938 "has_summary": false,939 "archetype": "regular",940 "slug": "fixing-number-of-filters-in-conv2ds",941 "category_id": 5,942 "word_count": 622,943 "deleted_at": null,944 "user_id": 81422,945 "featured_link": null,946 "pinned_globally": false,947 "pinned_at": null,948 "pinned_until": null,949 "image_url": null,950 "slow_mode_seconds": 0,951 "draft": null,952 "draft_key": "topic_214033",953 "draft_sequence": null,954 "unpinned": null,955 "pinned": false,956 "current_post_number": 1,957 "highest_post_number": 4,958 "deleted_by": null,959 "actions_summary": [960 {961 "id": 4,962 "count": 0,963 "hidden": false,964 "can_act": false965 },966 {967 "id": 8,968 "count": 0,969 "hidden": false,970 "can_act": false971 },972 {973 "id": 10,974 "count": 0,975 "hidden": false,976 "can_act": false977 },978 {979 "id": 7,980 "count": 0,981 "hidden": false,982 "can_act": false983 }984 ],985 "chunk_size": 20,986 "bookmarked": false,987 "topic_timer": null,988 "message_bus_last_id": 0,989 "participant_count": 2,990 "show_read_indicator": false,991 "thumbnails": null,992 "slow_mode_enabled_until": null,993 "can_vote": false,994 "vote_count": 0,995 "user_voted": false,996 "discourse_zendesk_plugin_zendesk_id": null,997 "discourse_zendesk_plugin_zendesk_url": "https://your-url.zendesk.com/agent/tickets/",998 "details": {999 "can_edit": false,1000 "notification_level": 1,1001 "participants": [1002 {1003 "id": 81422,1004 "username": "Idrees11",1005 "name": "Idrees Bhat",1006 "avatar_template": "/user_avatar/discuss.pytorch.org/idrees11/{size}/74448_2.png",1007 "post_count": 3,1008 "primary_group_name": null,1009 "flair_name": null,1010 "flair_url": null,1011 "flair_color": null,1012 "flair_bg_color": null,1013 "flair_group_id": null,1014 "trust_level": 11015 },1016 {1017 "id": 81089,1018 "username": "Aknw_Fen",1019 "name": "Aknw Fen",1020 "avatar_template": "/user_avatar/discuss.pytorch.org/aknw_fen/{size}/74156_2.png",1021 "post_count": 1,1022 "primary_group_name": null,1023 "flair_name": null,1024 "flair_url": null,1025 "flair_color": null,1026 "flair_bg_color": null,1027 "flair_group_id": null,1028 "trust_level": 21029 }1030 ],1031 "created_by": {1032 "id": 81422,1033 "username": "Idrees11",1034 "name": "Idrees Bhat",1035 "avatar_template": "/user_avatar/discuss.pytorch.org/idrees11/{size}/74448_2.png"1036 },1037 "last_poster": {1038 "id": 81422,1039 "username": "Idrees11",1040 "name": "Idrees Bhat",1041 "avatar_template": "/user_avatar/discuss.pytorch.org/idrees11/{size}/74448_2.png"1042 },1043 "links": [1044 {1045 "url": "https://github.com/TylerYep/torchinfo",1046 "title": "GitHub - TylerYep/torchinfo: View model summaries in PyTorch!",1047 "internal": false,1048 "attachment": false,1049 "reflection": false,1050 "clicks": 2,1051 "user_id": 81089,1052 "domain": "github.com",1053 "root_domain": "github.com"1054 }1055 ]1056 },1057 "bookmarks": []1058 },1059 {1060 "post_stream": {1061 "posts": [1062 {1063 "id": 461182,1064 "name": "Chaohao Yang",1065 "username": "Chaohao_Yang",1066 "avatar_template": "/user_avatar/discuss.pytorch.org/chaohao_yang/{size}/74456_2.png",1067 "created_at": "2024-12-10T18:02:16.299Z",1068 "cooked": "<p>Hello. I know that when using PyTorch AMP for training, the model weights are of type float32, while the gradients are 16 bits. I’m now wondering what the type of optimizer states is. For example, if I use Adam optimizer for training in the PyTorch AMP context, will the dtype of the states of Adam be 32bit or 16bit? Thanks.</p>",1069 "post_number": 1,1070 "post_type": 1,1071 "posts_count": 2,1072 "updated_at": "2024-12-10T18:02:16.299Z",1073 "reply_count": 0,1074 "reply_to_post_number": null,1075 "quote_count": 0,1076 "incoming_link_count": 144,1077 "reads": 3,1078 "readers_count": 2,1079 "score": 705.6,1080 "yours": false,1081 "topic_id": 214067,1082 "topic_slug": "the-dtype-of-optimizer-states-in-pytorch-amp-training",1083 "display_username": "Chaohao Yang",1084 "primary_group_name": null,1085 "flair_name": null,1086 "flair_url": null,1087 "flair_bg_color": null,1088 "flair_color": null,1089 "flair_group_id": null,1090 "badges_granted": [],1091 "version": 1,1092 "can_edit": false,1093 "can_delete": false,1094 "can_recover": false,1095 "can_see_hidden_post": false,1096 "can_wiki": false,1097 "read": true,1098 "user_title": null,1099 "bookmarked": false,1100 "actions_summary": [],1101 "moderator": false,1102 "admin": false,1103 "staff": false,1104 "user_id": 81434,1105 "hidden": false,1106 "trust_level": 0,1107 "deleted_at": null,1108 "user_deleted": false,1109 "edit_reason": null,1110 "can_view_edit_history": true,1111 "wiki": false,1112 "post_url": "/t/the-dtype-of-optimizer-states-in-pytorch-amp-training/214067/1",1113 "can_accept_answer": false,1114 "can_unaccept_answer": false,1115 "accepted_answer": false,1116 "topic_accepted_answer": null,1117 "can_vote": false1118 },1119 {1120 "id": 461196,1121 "name": "",1122 "username": "ptrblck",1123 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",1124 "created_at": "2024-12-10T22:56:43.015Z",1125 "cooked": "<p>The optimizer states should use the same dtype as the parameters but you can quickly check it by printing the <code>optimizer.state_dict()</code>.</p>",1126 "post_number": 2,1127 "post_type": 1,1128 "posts_count": 2,1129 "updated_at": "2024-12-10T22:56:43.015Z",1130 "reply_count": 0,1131 "reply_to_post_number": null,1132 "quote_count": 0,1133 "incoming_link_count": 1,1134 "reads": 3,1135 "readers_count": 2,1136 "score": 20.6,1137 "yours": false,1138 "topic_id": 214067,1139 "topic_slug": "the-dtype-of-optimizer-states-in-pytorch-amp-training",1140 "display_username": "",1141 "primary_group_name": null,1142 "flair_name": null,1143 "flair_url": null,1144 "flair_bg_color": null,1145 "flair_color": null,1146 "flair_group_id": null,1147 "badges_granted": [],1148 "version": 1,1149 "can_edit": false,1150 "can_delete": false,1151 "can_recover": false,1152 "can_see_hidden_post": false,1153 "can_wiki": false,1154 "read": true,1155 "user_title": "",1156 "bookmarked": false,1157 "actions_summary": [1158 {1159 "id": 2,1160 "count": 11161 }1162 ],1163 "moderator": true,1164 "admin": true,1165 "staff": true,1166 "user_id": 3534,1167 "hidden": false,1168 "trust_level": 2,1169 "deleted_at": null,1170 "user_deleted": false,1171 "edit_reason": null,1172 "can_view_edit_history": true,1173 "wiki": false,1174 "post_url": "/t/the-dtype-of-optimizer-states-in-pytorch-amp-training/214067/2",1175 "can_accept_answer": false,1176 "can_unaccept_answer": false,1177 "accepted_answer": false,1178 "topic_accepted_answer": null1179 }1180 ],1181 "stream": [1182 461182,1183 4611961184 ]1185 },1186 "timeline_lookup": [1187 [1188 1,1189 3191190 ]1191 ],1192 "suggested_topics": [1193 {1194 "fancy_title": "Autocast behaviour in different GPUs?",1195 "id": 218209,1196 "title": "Autocast behaviour in different GPUs?",1197 "slug": "autocast-behaviour-in-different-gpus",1198 "posts_count": 2,1199 "reply_count": 0,1200 "highest_post_number": 2,