Anurag1734/cuda-error-resolution-analysis
07
1[2 {3 "post_stream": {4 "posts": [5 {6 "id": 366874,7 "name": "",8 "username": "yannbane",9 "avatar_template": "/user_avatar/discuss.pytorch.org/yannbane/{size}/53347_2.png",10 "created_at": "2022-09-20T09:14:04.225Z",11 "cooked": "<p>I’m quantizing the Swin transformer (static PTQ) using the following function:</p>\n<pre><code class=\"lang-python\">def static_quantize(m, data_loader):\n backend = 'qnnpack'\n torch.backends.quantized.engine = backend\n m.eval()\n\n m.qconfig = torch.quantization.get_default_qconfig(backend)\n torch.quantization.prepare(m, inplace=True)\n\n with torch.no_grad():\n for i, data in enumerate(data_loader):\n result = m(return_loss=False, **data)\n if i > 10:\n break\n \n torch.quantization.convert(m, inplace=True)\n\n return m\n</code></pre>\n<p>Most modules, including linear layers, do get quantized. However some linear layers of a <code>SwinBlock</code> are skipped, as you can see here:</p>\n<pre><code class=\"lang-auto\">(3): SwinBlockSequence(\n (blocks): ModuleList(\n (0): SwinBlock(\n (quant): Quantize(scale=tensor([0.3938]), zero_point=tensor([122]), dtype=torch.quint8)\n (dequant): DeQuantize()\n (norm1): QuantizedLayerNorm((768,), eps=1e-05, elementwise_affine=True)\n (attn): ShiftWindowMSA(\n (w_msa): WindowMSA(\n (quant): Quantize(scale=tensor([0.0294]), zero_point=tensor([155]), dtype=torch.quint8)\n (dequant): DeQuantize()\n (qkv): QuantizedLinear(in_features=768, out_features=2304, scale=0.039033032953739166, zero_point=133, qscheme=torch.per_tensor_affine)\n (attn_drop): Dropout(p=0, inplace=False)\n (proj): QuantizedLinear(in_features=768, out_features=768, scale=0.0369536317884922, zero_point=110, qscheme=torch.per_tensor_affine)\n (proj_drop): Dropout(p=0, inplace=False)\n (softmax): Softmax(dim=-1)\n )\n (drop): DropPath()\n )\n (norm2): QuantizedLayerNorm((768,), eps=1e-05, elementwise_affine=True)\n (ffn): FFN( // <------- HERE (children not quantized)\n (activate): GELU()\n (layers): Sequential(\n (0): Sequential(\n (0): Linear(in_features=768, out_features=3072, bias=True)\n (1): GELU()\n (2): Dropout(p=0, inplace=False)\n )\n (1): Linear(in_features=3072, out_features=768, bias=True)\n (2): Dropout(p=0, inplace=False)\n )\n (dropout_layer): DropPath()\n )\n )\n</code></pre>\n<p>I am referring to the <code>FFN</code> submodule, where nothing is quantized. However, it contains linear layers, which ought to pose no problems for quantization.</p>\n<p>Here’s how FFN is added to the module:</p>\n<pre><code class=\"lang-python\"> _ffn_cfgs = {\n 'embed_dims': embed_dims,\n 'feedforward_channels': int(embed_dims * ffn_ratio),\n 'num_fcs': 2,\n 'ffn_drop': 0,\n 'dropout_layer': dict(type='DropPath', drop_prob=drop_path),\n 'act_cfg': dict(type='GELU'),\n **ffn_cfgs\n }\n self.norm2 = build_norm_layer(norm_cfg, embed_dims)[1]\n self.ffn = FFN(**_ffn_cfgs)\n</code></pre>\n<p>Here’s the source code for FFN:</p>\n<pre><code class=\"lang-python\">\n@FEEDFORWARD_NETWORK.register_module()\nclass FFN(BaseModule):\n \"\"\"Implements feed-forward networks (FFNs) with identity connection.\n\n Args:\n embed_dims (int): The feature dimension. Same as\n `MultiheadAttention`. Defaults: 256.\n feedforward_channels (int): The hidden dimension of FFNs.\n Defaults: 1024.\n num_fcs (int, optional): The number of fully-connected layers in\n FFNs. Default: 2.\n act_cfg (dict, optional): The activation config for FFNs.\n Default: dict(type='ReLU')\n ffn_drop (float, optional): Probability of an element to be\n zeroed in FFN. Default 0.0.\n add_identity (bool, optional): Whether to add the\n identity connection. Default: `True`.\n dropout_layer (obj:`ConfigDict`): The dropout_layer used\n when adding the shortcut.\n init_cfg (obj:`mmcv.ConfigDict`): The Config for initialization.\n Default: None.\n \"\"\"\n\n @deprecated_api_warning(\n {\n 'dropout': 'ffn_drop',\n 'add_residual': 'add_identity'\n },\n cls_name='FFN')\n def __init__(self,\n embed_dims=256,\n feedforward_channels=1024,\n num_fcs=2,\n act_cfg=dict(type='ReLU', inplace=True),\n ffn_drop=0.,\n dropout_layer=None,\n add_identity=True,\n init_cfg=None,\n **kwargs):\n super().__init__(init_cfg)\n assert num_fcs >= 2, 'num_fcs should be no less ' \\\n f'than 2. got {num_fcs}.'\n self.embed_dims = embed_dims\n self.feedforward_channels = feedforward_channels\n self.num_fcs = num_fcs\n self.act_cfg = act_cfg\n self.activate = build_activation_layer(act_cfg)\n\n layers = []\n in_channels = embed_dims\n for _ in range(num_fcs - 1):\n layers.append(\n Sequential(\n Linear(in_channels, feedforward_channels), self.activate,\n nn.Dropout(ffn_drop)))\n in_channels = feedforward_channels\n layers.append(Linear(feedforward_channels, embed_dims))\n layers.append(nn.Dropout(ffn_drop))\n self.layers = Sequential(*layers)\n self.dropout_layer = build_dropout(\n dropout_layer) if dropout_layer else torch.nn.Identity()\n self.add_identity = add_identity\n\n @deprecated_api_warning({'residual': 'identity'}, cls_name='FFN')\n def forward(self, x, identity=None):\n \"\"\"Forward function for `FFN`.\n\n The function would add x to the output tensor if residue is None.\n \"\"\"\n out = self.layers(x)\n if not self.add_identity:\n return self.dropout_layer(out)\n if identity is None:\n identity = x\n return identity + self.dropout_layer(out)\n</code></pre>",12 "post_number": 1,13 "post_type": 1,14 "posts_count": 2,15 "updated_at": "2022-09-20T09:15:26.621Z",16 "reply_count": 0,17 "reply_to_post_number": null,18 "quote_count": 0,19 "incoming_link_count": 285,20 "reads": 9,21 "readers_count": 8,22 "score": 1421.8,23 "yours": false,24 "topic_id": 161766,25 "topic_slug": "why-are-some-linear-layers-not-being-quantized",26 "display_username": "",27 "primary_group_name": null,28 "flair_name": null,29 "flair_url": null,30 "flair_bg_color": null,31 "flair_color": null,32 "flair_group_id": null,33 "badges_granted": [],34 "version": 1,35 "can_edit": false,36 "can_delete": false,37 "can_recover": false,38 "can_see_hidden_post": false,39 "can_wiki": false,40 "read": true,41 "user_title": null,42 "bookmarked": false,43 "actions_summary": [],44 "moderator": false,45 "admin": false,46 "staff": false,47 "user_id": 59464,48 "hidden": false,49 "trust_level": 1,50 "deleted_at": null,51 "user_deleted": false,52 "edit_reason": null,53 "can_view_edit_history": true,54 "wiki": false,55 "post_url": "/t/why-are-some-linear-layers-not-being-quantized/161766/1",56 "can_accept_answer": false,57 "can_unaccept_answer": false,58 "accepted_answer": false,59 "topic_accepted_answer": true,60 "can_vote": false61 },62 {63 "id": 366891,64 "name": "",65 "username": "yannbane",66 "avatar_template": "/user_avatar/discuss.pytorch.org/yannbane/{size}/53347_2.png",67 "created_at": "2022-09-20T10:47:11.924Z",68 "cooked": "<p>The answer is here: <a href=\"https://stackoverflow.com/a/73785433/924313\" class=\"inline-onebox\" rel=\"noopener nofollow ugc\">quantization - Why are some nn.Linear layers not quantized by Pytorch? - Stack Overflow</a></p>\n<p>In short <code>Linear</code> refers to a wrapper class from mmcv, not <code>nn.Linear</code>. Changing the FFN class to explicitly use <code>nn.Linear</code> seems to be the solution.</p>",69 "post_number": 2,70 "post_type": 1,71 "posts_count": 2,72 "updated_at": "2022-09-20T10:47:11.924Z",73 "reply_count": 0,74 "reply_to_post_number": null,75 "quote_count": 0,76 "incoming_link_count": 4,77 "reads": 7,78 "readers_count": 6,79 "score": 21.4,80 "yours": false,81 "topic_id": 161766,82 "topic_slug": "why-are-some-linear-layers-not-being-quantized",83 "display_username": "",84 "primary_group_name": null,85 "flair_name": null,86 "flair_url": null,87 "flair_bg_color": null,88 "flair_color": null,89 "flair_group_id": null,90 "badges_granted": [],91 "version": 1,92 "can_edit": false,93 "can_delete": false,94 "can_recover": false,95 "can_see_hidden_post": false,96 "can_wiki": false,97 "link_counts": [98 {99 "url": "https://stackoverflow.com/a/73785433/924313",100 "internal": false,101 "reflection": false,102 "title": "quantization - Why are some nn.Linear layers not quantized by Pytorch? - Stack Overflow",103 "clicks": 46104 }105 ],106 "read": true,107 "user_title": null,108 "bookmarked": false,109 "actions_summary": [],110 "moderator": false,111 "admin": false,112 "staff": false,113 "user_id": 59464,114 "hidden": false,115 "trust_level": 1,116 "deleted_at": null,117 "user_deleted": false,118 "edit_reason": null,119 "can_view_edit_history": true,120 "wiki": false,121 "post_url": "/t/why-are-some-linear-layers-not-being-quantized/161766/2",122 "can_accept_answer": false,123 "can_unaccept_answer": false,124 "accepted_answer": true,125 "topic_accepted_answer": true126 }127 ],128 "stream": [129 366874,130 366891131 ]132 },133 "timeline_lookup": [134 [135 1,136 1131137 ]138 ],139 "suggested_topics": [140 {141 "fancy_title": "Loss stuck at quantization aware training for 16bits",142 "id": 219149,143 "title": "Loss stuck at quantization aware training for 16bits",144 "slug": "loss-stuck-at-quantization-aware-training-for-16bits",145 "posts_count": 2,146 "reply_count": 0,147 "highest_post_number": 2,148 "image_url": null,149 "created_at": "2025-04-16T09:16:48.811Z",150 "last_posted_at": "2025-04-18T03:21:06.885Z",151 "bumped": true,152 "bumped_at": "2025-04-18T03:21:06.885Z",153 "archetype": "regular",154 "unseen": false,155 "pinned": false,156 "unpinned": null,157 "visible": true,158 "closed": false,159 "archived": false,160 "bookmarked": null,161 "liked": null,162 "tags_descriptions": {},163 "like_count": 0,164 "views": 45,165 "category_id": 17,166 "featured_link": null,167 "has_accepted_answer": false,168 "posters": [169 {170 "extras": null,171 "description": "Original Poster",172 "user": {173 "id": 83848,174 "username": "hoyuet_wu",175 "name": "hoyuet wu",176 "avatar_template": "/user_avatar/discuss.pytorch.org/hoyuet_wu/{size}/76658_2.png",177 "trust_level": 0178 }179 },180 {181 "extras": "latest",182 "description": "Most Recent Poster",183 "user": {184 "id": 21770,185 "username": "jerryzh168",186 "name": "Jerry Zhang",187 "avatar_template": "/user_avatar/discuss.pytorch.org/jerryzh168/{size}/15217_2.png",188 "trust_level": 2189 }190 }191 ]192 },193 {194 "fancy_title": "Qint8 Activations in PyTorch",195 "id": 219267,196 "title": "Qint8 Activations in PyTorch",197 "slug": "qint8-activations-in-pytorch",198 "posts_count": 2,199 "reply_count": 0,200 "highest_post_number": 3,201 "image_url": null,202 "created_at": "2025-04-20T13:21:52.095Z",203 "last_posted_at": "2025-04-25T22:54:35.772Z",204 "bumped": true,205 "bumped_at": "2025-04-25T22:54:35.772Z",206 "archetype": "regular",207 "unseen": false,208 "pinned": false,209 "unpinned": null,210 "visible": true,211 "closed": false,212 "archived": false,213 "bookmarked": null,214 "liked": null,215 "tags_descriptions": {},216 "like_count": 1,217 "views": 159,218 "category_id": 17,219 "featured_link": null,220 "has_accepted_answer": true,221 "posters": [222 {223 "extras": null,224 "description": "Original Poster",225 "user": {226 "id": 82558,227 "username": "xiechengyan",228 "name": "承諺 謝",229 "avatar_template": "/letter_avatar_proxy/v4/letter/x/a8b319/{size}.png",230 "trust_level": 1231 }232 },233 {234 "extras": "latest",235 "description": "Most Recent Poster, Accepted Answer",236 "user": {237 "id": 21770,238 "username": "jerryzh168",239 "name": "Jerry Zhang",240 "avatar_template": "/user_avatar/discuss.pytorch.org/jerryzh168/{size}/15217_2.png",241 "trust_level": 2242 }243 }244 ]245 },246 {247 "fancy_title": "[pt2e][quant] Quantization of operators with multiple outputs (RNN, LSTM)",248 "id": 218207,249 "title": "[pt2e][quant] Quantization of operators with multiple outputs (RNN, LSTM)",250 "slug": "pt2e-quant-quantization-of-operators-with-multiple-outputs-rnn-lstm",251 "posts_count": 5,252 "reply_count": 1,253 "highest_post_number": 5,254 "image_url": null,255 "created_at": "2025-03-24T18:12:14.530Z",256 "last_posted_at": "2025-09-15T11:12:40.113Z",257 "bumped": true,258 "bumped_at": "2025-09-15T11:12:40.113Z",259 "archetype": "regular",260 "unseen": false,261 "pinned": false,262 "unpinned": null,263 "visible": true,264 "closed": false,265 "archived": false,266 "bookmarked": null,267 "liked": null,268 "tags_descriptions": {},269 "like_count": 1,270 "views": 245,271 "category_id": 17,272 "featured_link": null,273 "has_accepted_answer": true,274 "posters": [275 {276 "extras": "latest",277 "description": "Original Poster, Most Recent Poster, Accepted Answer",278 "user": {279 "id": 83439,280 "username": "roman-janik-nxp",281 "name": "Roman Janik Nxp",282 "avatar_template": "/user_avatar/discuss.pytorch.org/roman-janik-nxp/{size}/76316_2.png",283 "trust_level": 0284 }285 },286 {287 "extras": null,288 "description": "Frequent Poster",289 "user": {290 "id": 21770,291 "username": "jerryzh168",292 "name": "Jerry Zhang",293 "avatar_template": "/user_avatar/discuss.pytorch.org/jerryzh168/{size}/15217_2.png",294 "trust_level": 2295 }296 }297 ]298 },299 {300 "fancy_title": "USing Quantization tutorial,but the result different",301 "id": 216178,302 "title": "USing Quantization tutorial,but the result different",303 "slug": "using-quantization-tutorial-but-the-result-different",304 "posts_count": 3,305 "reply_count": 1,306 "highest_post_number": 3,307 "image_url": null,308 "created_at": "2025-02-03T16:16:13.451Z",309 "last_posted_at": "2025-02-04T12:05:53.320Z",310 "bumped": true,311 "bumped_at": "2025-02-04T12:05:53.320Z",312 "archetype": "regular",313 "unseen": false,314 "pinned": false,315 "unpinned": null,316 "visible": true,317 "closed": false,318 "archived": false,319 "bookmarked": null,320 "liked": null,321 "tags_descriptions": {},322 "like_count": 0,323 "views": 66,324 "category_id": 17,325 "featured_link": null,326 "has_accepted_answer": false,327 "posters": [328 {329 "extras": "latest",330 "description": "Original Poster, Most Recent Poster",331 "user": {332 "id": 82456,333 "username": "BambooKui",334 "name": "Bamboo Kui",335 "avatar_template": "/user_avatar/discuss.pytorch.org/bambookui/{size}/75441_2.png",336 "trust_level": 1337 }338 },339 {340 "extras": null,341 "description": "Frequent Poster",342 "user": {343 "id": 19553,344 "username": "anantguptadbl",345 "name": "Anant Gupta",346 "avatar_template": "/user_avatar/discuss.pytorch.org/anantguptadbl/{size}/17784_2.png",347 "trust_level": 2348 }349 }350 ]351 },352 {353 "fancy_title": "[MPS] When device=‘mps’, aten.linear.default op is not decomposed",354 "id": 220573,355 "title": "[MPS] When device='mps', aten.linear.default op is not decomposed",356 "slug": "mps-when-device-mps-aten-linear-default-op-is-not-decomposed",357 "posts_count": 2,358 "reply_count": 0,359 "highest_post_number": 2,360 "image_url": null,361 "created_at": "2025-06-04T14:40:40.468Z",362 "last_posted_at": "2025-06-05T01:55:02.908Z",363 "bumped": true,364 "bumped_at": "2025-06-05T01:55:02.908Z",365 "archetype": "regular",366 "unseen": false,367 "pinned": false,368 "unpinned": null,369 "visible": true,370 "closed": false,371 "archived": false,372 "bookmarked": null,373 "liked": null,374 "tags_descriptions": {},375 "like_count": 0,376 "views": 46,377 "category_id": 17,378 "featured_link": null,379 "has_accepted_answer": false,380 "posters": [381 {382 "extras": "latest single",383 "description": "Original Poster, Most Recent Poster",384 "user": {385 "id": 84580,386 "username": "saeonnuri",387 "name": "Saeonnuri",388 "avatar_template": "/user_avatar/discuss.pytorch.org/saeonnuri/{size}/77264_2.png",389 "trust_level": 1390 }391 }392 ]393 }394 ],395 "tags_descriptions": {},396 "fancy_title": "Why are some linear layers not being quantized?",397 "id": 161766,398 "title": "Why are some linear layers not being quantized?",399 "posts_count": 2,400 "created_at": "2022-09-20T09:14:04.123Z",401 "views": 737,402 "reply_count": 0,403 "like_count": 0,404 "last_posted_at": "2022-09-20T10:47:11.924Z",405 "visible": true,406 "closed": false,407 "archived": false,408 "has_summary": false,409 "archetype": "regular",410 "slug": "why-are-some-linear-layers-not-being-quantized",411 "category_id": 17,412 "word_count": 608,413 "deleted_at": null,414 "user_id": 59464,415 "featured_link": null,416 "pinned_globally": false,417 "pinned_at": null,418 "pinned_until": null,419 "image_url": null,420 "slow_mode_seconds": 0,421 "draft": null,422 "draft_key": "topic_161766",423 "draft_sequence": null,424 "unpinned": null,425 "pinned": false,426 "current_post_number": 1,427 "highest_post_number": 2,428 "deleted_by": null,429 "actions_summary": [430 {431 "id": 4,432 "count": 0,433 "hidden": false,434 "can_act": false435 },436 {437 "id": 8,438 "count": 0,439 "hidden": false,440 "can_act": false441 },442 {443 "id": 10,444 "count": 0,445 "hidden": false,446 "can_act": false447 },448 {449 "id": 7,450 "count": 0,451 "hidden": false,452 "can_act": false453 }454 ],455 "chunk_size": 20,456 "bookmarked": false,457 "topic_timer": null,458 "message_bus_last_id": 0,459 "participant_count": 1,460 "show_read_indicator": false,461 "thumbnails": null,462 "slow_mode_enabled_until": null,463 "accepted_answer": {464 "post_number": 2,465 "username": "yannbane",466 "name": "",467 "excerpt": "The answer is here: <a href=\"https://stackoverflow.com/a/73785433/924313\" class=\"inline-onebox\" rel=\"noopener nofollow ugc\">quantization - Why are some nn.Linear layers not quantized by Pytorch? - Stack Overflow</a> \nIn short Linear refers to a wrapper class from mmcv, not nn.Linear. Changing the FFN class to explicitly use nn.Linear seems to be the solution."468 },469 "can_vote": false,470 "vote_count": 0,471 "user_voted": false,472 "discourse_zendesk_plugin_zendesk_id": null,473 "discourse_zendesk_plugin_zendesk_url": "https://your-url.zendesk.com/agent/tickets/",474 "details": {475 "can_edit": false,476 "notification_level": 1,477 "participants": [478 {479 "id": 59464,480 "username": "yannbane",481 "name": "",482 "avatar_template": "/user_avatar/discuss.pytorch.org/yannbane/{size}/53347_2.png",483 "post_count": 2,484 "primary_group_name": null,485 "flair_name": null,486 "flair_url": null,487 "flair_color": null,488 "flair_bg_color": null,489 "flair_group_id": null,490 "trust_level": 1491 }492 ],493 "created_by": {494 "id": 59464,495 "username": "yannbane",496 "name": "",497 "avatar_template": "/user_avatar/discuss.pytorch.org/yannbane/{size}/53347_2.png"498 },499 "last_poster": {500 "id": 59464,501 "username": "yannbane",502 "name": "",503 "avatar_template": "/user_avatar/discuss.pytorch.org/yannbane/{size}/53347_2.png"504 },505 "links": [506 {507 "url": "https://stackoverflow.com/a/73785433/924313",508 "title": "quantization - Why are some nn.Linear layers not quantized by Pytorch? - Stack Overflow",509 "internal": false,510 "attachment": false,511 "reflection": false,512 "clicks": 46,513 "user_id": 59464,514 "domain": "stackoverflow.com",515 "root_domain": "stackoverflow.com"516 }517 ]518 },519 "bookmarks": []520 },521 {522 "post_stream": {523 "posts": [524 {525 "id": 365727,526 "name": "saad khan",527 "username": "saad_khan",528 "avatar_template": "/user_avatar/discuss.pytorch.org/saad_khan/{size}/44721_2.png",529 "created_at": "2022-09-12T02:23:13.473Z",530 "cooked": "<p>Hi,<br>\nI am trying to implement the self-attention mechanism in MSG-GAN for Grayscale images. I have implemented this <a href=\"https://github.com/mdraw/BMSG-GAN/tree/img_channels\" class=\"inline-onebox\" rel=\"noopener nofollow ugc\">GitHub - mdraw/BMSG-GAN at img_channels</a> code for generating X-ray images. I am integrating the self-attention layer in the generator and discriminator as in <a href=\"https://github.com/akanimax/some-randon-gan-1/blob/master/sourcecode/SMSG_GAN/CustomLayers.py\" class=\"inline-onebox\" rel=\"noopener nofollow ugc\">some-randon-gan-1/CustomLayers.py at master · akanimax/some-randon-gan-1 · GitHub</a>.</p>\n<p>I got following Runtime memory error. I tried reducing batch size to 1 but didn’t work. I tried it only for 10 images as well but didn’t workout.</p>\n<p>The error log:</p>\n<pre><code>Traceback (most recent call last):\n File \"train.py\", line 281, in <module>\n main(parse_arguments())\n File \"train.py\", line 275, in main\n start=args.start\n File \"/home/r00206978/AICS/MSG_X/SA/MSG_GAN/GAN.py\", line 556, in train\n images, loss_fn)\n File \"/home/r00206978/AICS/MSG_X/SA/MSG_GAN/GAN.py\", line 413, in optimize_discriminator\n loss = loss_fn.dis_loss(real_batch, fake_samples)\n File \"/home/r00206978/AICS/MSG_X/SA/MSG_GAN/Losses.py\", line 202, in dis_loss\n f_preds = self.dis(fake_samps)\n File \"/home/r00206978/.local/lib/python3.7/site-packages/torch/nn/modules/module.py\", line 1110, in _call_impl\n return forward_call(*input, **kwargs)\n File \"/home/r00206978/AICS/MSG_X/SA/MSG_GAN/GAN.py\", line 304, in forward\n y = self.layers[self.depth - 2](y)\n File \"/home/r00206978/.local/lib/python3.7/site-packages/torch/nn/modules/module.py\", line 1110, in _call_impl\n return forward_call(*input, **kwargs)\n File \"/home/r00206978/.local/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py\", line 166, in forward\n return self.module(*inputs[0], **kwargs[0])\n File \"/home/r00206978/.local/lib/python3.7/site-packages/torch/nn/modules/module.py\", line 1110, in _call_impl\n return forward_call(*input, **kwargs)\n File \"/home/r00206978/AICS/MSG_X/SA/MSG_GAN/CustomLayers.py\", line 566, in forward\n y, _ = self.self_attention(x)\n File \"/home/r00206978/.local/lib/python3.7/site-packages/torch/nn/modules/module.py\", line 1110, in _call_impl\n return forward_call(*input, **kwargs)\n File \"/home/r00206978/AICS/MSG_X/SA/MSG_GAN/CustomLayers.py\", line 87, in forward\n energy = th.bmm(proj_query, proj_key) # energy\nRuntimeError: CUDA out of memory. Tried to allocate 4.00 GiB (GPU 0; 15.78 GiB total capacity; 10.18 GiB already allocated; 846.00 MiB free; 13.73 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting m$\n</code></pre>\n<p>Self Attention Layer:</p>\n<pre><code>class SelfAttention(th.nn.Module):\n \"\"\"\n Layer implements the self-attention module\n which is the main logic behind this architecture.\n Mechanism described in the paper ->\n Self Attention GAN: refer /literature/Zhang_et_al_2018_SAGAN.pdf\n args:\n channels: number of channels in the image tensor\n activation: activation function to be applied (default: lrelu(0.2))\n squeeze_factor: squeeze factor for query and keys (default: 8)\n bias: whether to apply bias or not (default: True)\n \"\"\"\n from torch.nn import LeakyReLU\n\n def __init__(self, channels, activation=LeakyReLU(0.2), squeeze_factor=8, bias=True):\n \"\"\" constructor for the layer \"\"\"\n\n from torch.nn import Conv2d, Parameter, Softmax\n\n # base constructor call\n super().__init__()\n\n # state of the layer\n self.activation = activation\n self.gamma = Parameter(th.zeros(1))\n\n # Modules required for computations\n self.query_conv = Conv2d( # query convolution\n in_channels=channels,\n out_channels=channels // squeeze_factor,\n kernel_size=(1, 1),\n stride=1,\n padding=0,\n bias=bias\n )\n\n self.key_conv = Conv2d( # key convolution\n in_channels=channels,\n out_channels=channels // squeeze_factor,\n kernel_size=(1, 1),\n stride=1,\n padding=0,\n bias=bias\n )\n\n self.value_conv = Conv2d( # value convolution\n in_channels=channels,\n out_channels=channels,\n kernel_size=(1, 1),\n stride=1,\n padding=0,\n bias=bias\n )\n\n # softmax module for applying attention\n self.softmax = Softmax(dim=-1)\n\ndef forward(self, x):\n \"\"\"\n forward computations of the layer\n :param x: input feature maps (B x C x H x W)\n :return:\n out: self attention value + input feature (B x O x H x W)\n attention: attention map (B x H x W x H x W)\n \"\"\"\n\n # extract the shape of the input tensor\n m_batchsize, c, height, width = x.size()\n\n # create the query projection\n proj_query = self.query_conv(x).view(\n m_batchsize, -1, width * height).permute(0, 2, 1) # B x (N) x C\n\n # create the key projection\n proj_key = self.key_conv(x).view(\n m_batchsize, -1, width * height) # B x C x (N)\n\n # calculate the attention maps\n energy = th.bmm(proj_query, proj_key) # energy\n attention = self.softmax(energy) # attention B x (N) x (N)\n\n # create the value projection\n proj_value = self.value_conv(x).view(\n m_batchsize, -1, width * height) # B X C X (N)\n\n # calculate the output\n out = th.bmm(proj_value, attention.permute(0, 2, 1))\n out = out.view(m_batchsize, c, height, width)\n\n attention = attention.view(m_batchsize, height, width, height, width)\n\n if self.activation is not None:\n out = self.activation(out)\n\n # apply the residual connection\n out = (self.gamma * out) + x\n return out, attention\n</code></pre>\n<p>Could you please help me out to solve this problem?<br>\nThanks in advance…</p>",531 "post_number": 1,532 "post_type": 1,533 "posts_count": 4,534 "updated_at": "2022-09-12T02:28:30.923Z",535 "reply_count": 0,536 "reply_to_post_number": null,537 "quote_count": 0,538 "incoming_link_count": 438,539 "reads": 13,540 "readers_count": 12,541 "score": 2192.6,542 "yours": false,543 "topic_id": 161178,544 "topic_slug": "runtimeerror-cuda-out-of-memory-with-self-attention-in-gans",545 "display_username": "saad khan",546 "primary_group_name": null,547 "flair_name": null,548 "flair_url": null,549 "flair_bg_color": null,550 "flair_color": null,551 "flair_group_id": null,552 "badges_granted": [],553 "version": 2,554 "can_edit": false,555 "can_delete": false,556 "can_recover": false,557 "can_see_hidden_post": false,558 "can_wiki": false,559 "link_counts": [560 {561 "url": "https://github.com/mdraw/BMSG-GAN/tree/img_channels",562 "internal": false,563 "reflection": false,564 "title": "GitHub - mdraw/BMSG-GAN at img_channels",565 "clicks": 1566 },567 {568 "url": "https://github.com/akanimax/some-randon-gan-1/blob/master/sourcecode/SMSG_GAN/CustomLayers.py",569 "internal": false,570 "reflection": false,571 "title": "some-randon-gan-1/CustomLayers.py at master · akanimax/some-randon-gan-1 · GitHub",572 "clicks": 1573 }574 ],575 "read": true,576 "user_title": null,577 "bookmarked": false,578 "actions_summary": [],579 "moderator": false,580 "admin": false,581 "staff": false,582 "user_id": 57099,583 "hidden": false,584 "trust_level": 1,585 "deleted_at": null,586 "user_deleted": false,587 "edit_reason": null,588 "can_view_edit_history": true,589 "wiki": false,590 "post_url": "/t/runtimeerror-cuda-out-of-memory-with-self-attention-in-gans/161178/1",591 "can_accept_answer": false,592 "can_unaccept_answer": false,593 "accepted_answer": false,594 "topic_accepted_answer": null,595 "can_vote": false596 },597 {598 "id": 365736,599 "name": "",600 "username": "ptrblck",601 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",602 "created_at": "2022-09-12T04:16:49.808Z",603 "cooked": "<p>Since reducing the batch size didn’t work, try to reduce the spatial size of your images and check which max. size would allow the model to train. Alternatively, you could also try to use <code>torch.utils.checkpoint</code> to trade compute for memory.</p>",604 "post_number": 2,605 "post_type": 1,606 "posts_count": 4,607 "updated_at": "2022-09-12T04:16:49.808Z",608 "reply_count": 1,609 "reply_to_post_number": null,610 "quote_count": 0,611 "incoming_link_count": 2,612 "reads": 9,613 "readers_count": 8,614 "score": 16.8,615 "yours": false,616 "topic_id": 161178,617 "topic_slug": "runtimeerror-cuda-out-of-memory-with-self-attention-in-gans",618 "display_username": "",619 "primary_group_name": null,620 "flair_name": null,621 "flair_url": null,622 "flair_bg_color": null,623 "flair_color": null,624 "flair_group_id": null,625 "badges_granted": [],626 "version": 1,627 "can_edit": false,628 "can_delete": false,629 "can_recover": false,630 "can_see_hidden_post": false,631 "can_wiki": false,632 "read": true,633 "user_title": "",634 "bookmarked": false,635 "actions_summary": [],636 "moderator": true,637 "admin": true,638 "staff": true,639 "user_id": 3534,640 "hidden": false,641 "trust_level": 2,642 "deleted_at": null,643 "user_deleted": false,644 "edit_reason": null,645 "can_view_edit_history": true,646 "wiki": false,647 "post_url": "/t/runtimeerror-cuda-out-of-memory-with-self-attention-in-gans/161178/2",648 "can_accept_answer": false,649 "can_unaccept_answer": false,650 "accepted_answer": false,651 "topic_accepted_answer": null652 },653 {654 "id": 366041,655 "name": "saad khan",656 "username": "saad_khan",657 "avatar_template": "/user_avatar/discuss.pytorch.org/saad_khan/{size}/44721_2.png",658 "created_at": "2022-09-13T21:57:44.964Z",659 "cooked": "<p>Hi <a class=\"mention\" href=\"/u/ptrblck\">@ptrblck</a> thanks for your attention. I did both of these but error is still there. I did reduce the image size upto 6 KB. I also checked it for only 10 images as well but not worked. I added spectral normalization as well but not worked. Why does the model not starts training with self attention? The model works perfect without self-attention mechanism and trained well.</p>",660 "post_number": 3,661 "post_type": 1,662 "posts_count": 4,663 "updated_at": "2022-09-13T21:57:44.964Z",664 "reply_count": 1,665 "reply_to_post_number": 2,666 "quote_count": 0,667 "incoming_link_count": 1,668 "reads": 7,669 "readers_count": 6,670 "score": 11.4,671 "yours": false,672 "topic_id": 161178,673 "topic_slug": "runtimeerror-cuda-out-of-memory-with-self-attention-in-gans",674 "display_username": "saad khan",675 "primary_group_name": null,676 "flair_name": null,677 "flair_url": null,678 "flair_bg_color": null,679 "flair_color": null,680 "flair_group_id": null,681 "badges_granted": [],682 "version": 1,683 "can_edit": false,684 "can_delete": false,685 "can_recover": false,686 "can_see_hidden_post": false,687 "can_wiki": false,688 "read": true,689 "user_title": null,690 "reply_to_user": {691 "id": 3534,692 "username": "ptrblck",693 "name": "",694 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png"695 },696 "bookmarked": false,697 "actions_summary": [],698 "moderator": false,699 "admin": false,700 "staff": false,701 "user_id": 57099,702 "hidden": false,703 "trust_level": 1,704 "deleted_at": null,705 "user_deleted": false,706 "edit_reason": null,707 "can_view_edit_history": true,708 "wiki": false,709 "post_url": "/t/runtimeerror-cuda-out-of-memory-with-self-attention-in-gans/161178/3",710 "can_accept_answer": false,711 "can_unaccept_answer": false,712 "accepted_answer": false,713 "topic_accepted_answer": null714 },715 {716 "id": 366050,717 "name": "",718 "username": "ptrblck",719 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",720 "created_at": "2022-09-14T00:32:36.631Z",721 "cooked": "<p>Your self-attention layer might use too much memory for your GPU so check your implementation in isolation and profile its memory usage.<br>\nThe memory usage could also give you more information if the implementation might be wrong.</p>",722 "post_number": 4,723 "post_type": 1,724 "posts_count": 4,725 "updated_at": "2022-09-14T00:32:36.631Z",726 "reply_count": 0,727 "reply_to_post_number": 3,728 "quote_count": 0,729 "incoming_link_count": 1,730 "reads": 6,731 "readers_count": 5,732 "score": 6.2,733 "yours": false,734 "topic_id": 161178,735 "topic_slug": "runtimeerror-cuda-out-of-memory-with-self-attention-in-gans",736 "display_username": "",737 "primary_group_name": null,738 "flair_name": null,739 "flair_url": null,740 "flair_bg_color": null,741 "flair_color": null,742 "flair_group_id": null,743 "badges_granted": [],744 "version": 1,745 "can_edit": false,746 "can_delete": false,747 "can_recover": false,748 "can_see_hidden_post": false,749 "can_wiki": false,750 "read": true,751 "user_title": "",752 "reply_to_user": {753 "id": 57099,754 "username": "saad_khan",755 "name": "saad khan",756 "avatar_template": "/user_avatar/discuss.pytorch.org/saad_khan/{size}/44721_2.png"757 },758 "bookmarked": false,759 "actions_summary": [],760 "moderator": true,761 "admin": true,762 "staff": true,763 "user_id": 3534,764 "hidden": false,765 "trust_level": 2,766 "deleted_at": null,767 "user_deleted": false,768 "edit_reason": null,769 "can_view_edit_history": true,770 "wiki": false,771 "post_url": "/t/runtimeerror-cuda-out-of-memory-with-self-attention-in-gans/161178/4",772 "can_accept_answer": false,773 "can_unaccept_answer": false,774 "accepted_answer": false,775 "topic_accepted_answer": null776 }777 ],778 "stream": [779 365727,780 365736,781 366041,782 366050783 ]784 },785 "timeline_lookup": [786 [787 1,788 1140789 ],790 [791 3,792 1138793 ]794 ],795 "suggested_topics": [796 {797 "fancy_title": "Cannot find unused Parameters (DDP Training)",798 "id": 219344,799 "title": "Cannot find unused Parameters (DDP Training)",800 "slug": "cannot-find-unused-parameters-ddp-training",801 "posts_count": 1,802 "reply_count": 0,803 "highest_post_number": 1,804 "image_url": null,805 "created_at": "2025-04-22T16:49:57.607Z",806 "last_posted_at": "2025-04-22T16:49:57.652Z",807 "bumped": true,808 "bumped_at": "2025-04-22T16:53:49.141Z",809 "archetype": "regular",810 "unseen": false,811 "pinned": false,812 "unpinned": null,813 "visible": true,814 "closed": false,815 "archived": false,816 "bookmarked": null,817 "liked": null,818 "tags_descriptions": {},819 "like_count": 0,820 "views": 200,821 "category_id": 5,822 "featured_link": null,823 "has_accepted_answer": false,824 "posters": [825 {826 "extras": "latest single",827 "description": "Original Poster, Most Recent Poster",828 "user": {829 "id": 83950,830 "username": "rdslater",831 "name": "rds",832 "avatar_template": "/user_avatar/discuss.pytorch.org/rdslater/{size}/75820_2.png",833 "trust_level": 1834 }835 }836 ]837 },838 {839 "fancy_title": "How to Build a More Efficient DataLoader to Load Large Image Datasets?",840 "id": 213229,841 "title": "How to Build a More Efficient DataLoader to Load Large Image Datasets?",842 "slug": "how-to-build-a-more-efficient-dataloader-to-load-large-image-datasets",843 "posts_count": 3,844 "reply_count": 1,845 "highest_post_number": 3,846 "image_url": null,847 "created_at": "2024-11-20T20:09:54.513Z",848 "last_posted_at": "2024-11-22T16:09:19.099Z",849 "bumped": true,850 "bumped_at": "2024-11-22T16:09:19.099Z",851 "archetype": "regular",852 "unseen": false,853 "pinned": false,854 "unpinned": null,855 "visible": true,856 "closed": false,857 "archived": false,858 "bookmarked": null,859 "liked": null,860 "tags_descriptions": {},861 "like_count": 0,862 "views": 172,863 "category_id": 5,864 "featured_link": null,865 "has_accepted_answer": false,866 "posters": [867 {868 "extras": null,869 "description": "Original Poster",870 "user": {871 "id": 32574,872 "username": "Nick_ishere",873 "name": "",874 "avatar_template": "/user_avatar/discuss.pytorch.org/nick_ishere/{size}/25121_2.png",875 "trust_level": 1876 }877 },878 {879 "extras": null,880 "description": "Frequent Poster",881 "user": {882 "id": 72430,883 "username": "Eduardo_Lawson",884 "name": "Eduardo Lawson da Silva",885 "avatar_template": "/user_avatar/discuss.pytorch.org/eduardo_lawson/{size}/66899_2.png",886 "trust_level": 2887 }888 },889 {890 "extras": "latest",891 "description": "Most Recent Poster",892 "user": {893 "id": 64498,894 "username": "nickums",895 "name": "Nickums",896 "avatar_template": "/user_avatar/discuss.pytorch.org/nickums/{size}/58652_2.png",897 "trust_level": 1898 }899 }900 ]901 },902 {903 "fancy_title": "How can I prevent infinity value after switching from float32 to float16?",904 "id": 214380,905 "title": "How can I prevent infinity value after switching from float32 to float16?",906 "slug": "how-can-i-prevent-infinity-value-after-switching-from-float32-to-float16",907 "posts_count": 2,908 "reply_count": 0,909 "highest_post_number": 2,910 "image_url": null,911 "created_at": "2024-12-19T00:47:57.130Z",912 "last_posted_at": "2024-12-19T02:02:53.245Z",913 "bumped": true,914 "bumped_at": "2024-12-19T02:02:53.245Z",915 "archetype": "regular",916 "unseen": false,917 "pinned": false,918 "unpinned": null,919 "visible": true,920 "closed": false,921 "archived": false,922 "bookmarked": null,923 "liked": null,924 "tags_descriptions": {},925 "like_count": 1,926 "views": 48,927 "category_id": 5,928 "featured_link": null,929 "has_accepted_answer": true,930 "posters": [931 {932 "extras": null,933 "description": "Original Poster",934 "user": {935 "id": 45116,936 "username": "JimW",937 "name": "",938 "avatar_template": "/user_avatar/discuss.pytorch.org/jimw/{size}/38000_2.png",939 "trust_level": 1940 }941 },942 {943 "extras": "latest",944 "description": "Most Recent Poster, Accepted Answer",945 "user": {946 "id": 41396,947 "username": "soulitzer",948 "name": "",949 "avatar_template": "/letter_avatar_proxy/v4/letter/s/839c29/{size}.png",950 "trust_level": 2951 }952 }953 ]954 },955 {956 "fancy_title": "Training Time is Increasing per epoch, Can somebody help me?",957 "id": 214900,958 "title": "Training Time is Increasing per epoch, Can somebody help me?",959 "slug": "training-time-is-increasing-per-epoch-can-somebody-help-me",960 "posts_count": 6,961 "reply_count": 4,962 "highest_post_number": 6,963 "image_url": null,964 "created_at": "2025-01-02T17:21:10.454Z",965 "last_posted_at": "2025-01-17T18:10:59.673Z",966 "bumped": true,967 "bumped_at": "2025-01-17T18:10:59.673Z",968 "archetype": "regular",969 "unseen": false,970 "pinned": false,971 "unpinned": null,972 "visible": true,973 "closed": false,974 "archived": false,975 "bookmarked": null,976 "liked": null,977 "tags_descriptions": {},978 "like_count": 3,979 "views": 183,980 "category_id": 5,981 "featured_link": null,982 "has_accepted_answer": true,983 "posters": [984 {985 "extras": "latest",986 "description": "Original Poster, Most Recent Poster",987 "user": {988 "id": 81840,989 "username": "iran_boy",990 "name": "iran boy",991 "avatar_template": "/user_avatar/discuss.pytorch.org/iran_boy/{size}/74864_2.png",992 "trust_level": 0993 }994 },995 {996 "extras": null,997 "description": "Frequent Poster, Accepted Answer",998 "user": {999 "id": 3534,1000 "username": "ptrblck",1001 "name": "",1002 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",1003 "admin": true,1004 "moderator": true,1005 "trust_level": 21006 }1007 }1008 ]1009 },1010 {1011 "fancy_title": "I install torchvision from source, but I see RuntimeError: operator torchvision::nms does not exist",1012 "id": 219674,1013 "title": "I install torchvision from source, but I see RuntimeError: operator torchvision::nms does not exist",1014 "slug": "i-install-torchvision-from-source-but-i-see-runtimeerror-operator-torchvision-nms-does-not-exist",1015 "posts_count": 1,1016 "reply_count": 0,1017 "highest_post_number": 1,1018 "image_url": null,1019 "created_at": "2025-05-02T00:55:03.470Z",1020 "last_posted_at": "2025-05-02T00:55:03.513Z",1021 "bumped": true,1022 "bumped_at": "2025-05-02T01:41:08.936Z",1023 "archetype": "regular",1024 "unseen": false,1025 "pinned": false,1026 "unpinned": null,1027 "visible": true,1028 "closed": false,1029 "archived": false,1030 "bookmarked": null,1031 "liked": null,1032 "tags_descriptions": {},1033 "like_count": 0,1034 "views": 55,1035 "category_id": 5,1036 "featured_link": null,1037 "has_accepted_answer": false,1038 "posters": [1039 {1040 "extras": "latest single",1041 "description": "Original Poster, Most Recent Poster",1042 "user": {1043 "id": 83166,1044 "username": "George_Polya",1045 "name": "George Polya",1046 "avatar_template": "/user_avatar/discuss.pytorch.org/george_polya/{size}/76064_2.png",1047 "trust_level": 11048 }1049 }1050 ]1051 }1052 ],1053 "tags_descriptions": {},1054 "fancy_title": "RuntimeError: CUDA out of memory with Self-Attention in GANs",1055 "id": 161178,1056 "title": "RuntimeError: CUDA out of memory with Self-Attention in GANs",1057 "posts_count": 4,1058 "created_at": "2022-09-12T02:23:13.362Z",1059 "views": 1124,1060 "reply_count": 3,1061 "like_count": 0,1062 "last_posted_at": "2022-09-14T00:32:36.631Z",1063 "visible": true,1064 "closed": false,1065 "archived": false,1066 "has_summary": false,1067 "archetype": "regular",1068 "slug": "runtimeerror-cuda-out-of-memory-with-self-attention-in-gans",1069 "category_id": 5,1070 "word_count": 923,1071 "deleted_at": null,1072 "user_id": 57099,1073 "featured_link": null,1074 "pinned_globally": false,1075 "pinned_at": null,1076 "pinned_until": null,1077 "image_url": null,1078 "slow_mode_seconds": 0,1079 "draft": null,1080 "draft_key": "topic_161178",1081 "draft_sequence": null,1082 "unpinned": null,1083 "pinned": false,1084 "current_post_number": 1,1085 "highest_post_number": 4,1086 "deleted_by": null,1087 "actions_summary": [1088 {1089 "id": 4,1090 "count": 0,1091 "hidden": false,1092 "can_act": false1093 },1094 {1095 "id": 8,1096 "count": 0,1097 "hidden": false,1098 "can_act": false1099 },1100 {1101 "id": 10,1102 "count": 0,1103 "hidden": false,1104 "can_act": false1105 },1106 {1107 "id": 7,1108 "count": 0,1109 "hidden": false,1110 "can_act": false1111 }1112 ],1113 "chunk_size": 20,1114 "bookmarked": false,1115 "topic_timer": null,1116 "message_bus_last_id": 0,1117 "participant_count": 2,1118 "show_read_indicator": false,1119 "thumbnails": null,1120 "slow_mode_enabled_until": null,1121 "can_vote": false,1122 "vote_count": 0,1123 "user_voted": false,1124 "discourse_zendesk_plugin_zendesk_id": null,1125 "discourse_zendesk_plugin_zendesk_url": "https://your-url.zendesk.com/agent/tickets/",1126 "details": {1127 "can_edit": false,1128 "notification_level": 1,1129 "participants": [1130 {1131 "id": 3534,1132 "username": "ptrblck",1133 "name": "",1134 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",1135 "post_count": 2,1136 "primary_group_name": null,1137 "flair_name": null,1138 "flair_url": null,1139 "flair_color": null,1140 "flair_bg_color": null,1141 "flair_group_id": null,1142 "admin": true,1143 "moderator": true,1144 "trust_level": 21145 },1146 {1147 "id": 57099,1148 "username": "saad_khan",1149 "name": "saad khan",1150 "avatar_template": "/user_avatar/discuss.pytorch.org/saad_khan/{size}/44721_2.png",1151 "post_count": 2,1152 "primary_group_name": null,1153 "flair_name": null,1154 "flair_url": null,1155 "flair_color": null,1156 "flair_bg_color": null,1157 "flair_group_id": null,1158 "trust_level": 11159 }1160 ],1161 "created_by": {1162 "id": 57099,1163 "username": "saad_khan",1164 "name": "saad khan",1165 "avatar_template": "/user_avatar/discuss.pytorch.org/saad_khan/{size}/44721_2.png"1166 },1167 "last_poster": {1168 "id": 3534,1169 "username": "ptrblck",1170 "name": "",1171 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png"1172 },1173 "links": [1174 {1175 "url": "https://github.com/akanimax/some-randon-gan-1/blob/master/sourcecode/SMSG_GAN/CustomLayers.py",1176 "title": "some-randon-gan-1/CustomLayers.py at master · akanimax/some-randon-gan-1 · GitHub",1177 "internal": false,1178 "attachment": false,1179 "reflection": false,1180 "clicks": 1,1181 "user_id": 57099,1182 "domain": "github.com",1183 "root_domain": "github.com"1184 },1185 {1186 "url": "https://github.com/mdraw/BMSG-GAN/tree/img_channels",1187 "title": "GitHub - mdraw/BMSG-GAN at img_channels",1188 "internal": false,1189 "attachment": false,1190 "reflection": false,1191 "clicks": 1,1192 "user_id": 57099,1193 "domain": "github.com",1194 "root_domain": "github.com"1195 }1196 ]1197 },1198 "bookmarks": []1199 },1200 {