Anurag1734/cuda-error-resolution-analysis
07
1[2 {3 "post_stream": {4 "posts": [5 {6 "id": 406317,7 "name": "",8 "username": "weiskohlmoe",9 "avatar_template": "/letter_avatar_proxy/v4/letter/w/eada6e/{size}.png",10 "created_at": "2023-06-16T16:15:16.532Z",11 "cooked": "<p>I want to implement a Machine Learning model that, among other things, should learn coefficient for (Quadratic) spline functions. (<a href=\"https://en.wikipedia.org/wiki/B-spline\" class=\"inline-onebox\" rel=\"noopener nofollow ugc\">B-spline - Wikipedia</a> for reference). As a part of the forward operation the spline function is applied on the input.</p>\n<p>For the computation value of this function I implemented the algorithm as described here:</p><aside class=\"onebox wikipedia\" data-onebox-src=\"https://en.wikipedia.org/wiki/De_Boor%27s_algorithm\">\n <header class=\"source\">\n\n <a href=\"https://en.wikipedia.org/wiki/De_Boor%27s_algorithm\" target=\"_blank\" rel=\"noopener nofollow ugc\">en.wikipedia.org</a>\n </header>\n\n <article class=\"onebox-body\">\n \n\n<h3><a href=\"https://en.wikipedia.org/wiki/De_Boor%27s_algorithm\" target=\"_blank\" rel=\"noopener nofollow ugc\">De Boor's algorithm</a></h3>\n\n<p>In the mathematical subfield of numerical analysis de Boor's algorithm is a polynomial-time and numerically stable algorithm for evaluating spline curves in B-spline form. It is a generalization of de Casteljau's algorithm for Bézier curves. The algorithm was devised by Carl R. de Boor. Simplified, potentially faster variants of the de Boor algorithm have been created but they suffer from comparatively lower stability.\n A general introduction to B-splines is given in the main article. Here we di...</p>\n\n </article>\n\n <div class=\"onebox-metadata\">\n \n \n </div>\n\n <div style=\"clear: both\"></div>\n</aside>\n<p>\nIn the initialization step I need to get values out of a tensor of coefficients according to a tensor of indices.<br>\nSince I need a range of Indices I used <code>torch.narrow</code> for this problem.<br>\nHowever my runtime is dominated by this computation, so I am searching for ways to speed it up</p>\n<p>My current code to measure the runtime looks like this:</p>\n<pre><code class=\"lang-auto\">import torch\nimport time\nimport torch.nn.functional as F\n\nnum_weights = 31\nbatch_size = 24\nvalues_size = 45 * 45\ncoeffs = torch.rand(num_weights)\nvalues = torch.rand(batch_size, values_size)\ninterval = torch.arange(0, 1.1, 0.1)\n\n#This is just so the snippet below works without error\nindex_tensor = torch.searchsorted(interval, values, right=True, side=\"right\") + 2*torch.ones(values.size(), dtype=torch.int64)\npd = (2,2)\ninterval = interval.unsqueeze(0)\ninterval = F.pad(interval, pd, \"replicate\")\ninterval = interval.squeeze()\n\n\nstart = time.time()\ncoeff_matrix = torch.stack([torch.stack([torch.narrow(coeffs, 0, index_tensor[j][i] - 2, 3) for i in range(values_size)]) for j in range(batch_size)])\nprint(\"Time to build coeff_matrix\", time.time() - start)\n</code></pre>\n<p>On my computer I get a runtime of ~0.5 seconds. Just for reference, the other operations in the evaluation amount to ~0.01 seconds<br>\nI tried to calculate the matrix first without <code>torch.stack()</code> in the middle, but it was slower.<br>\nSo my question is, if there is a faster method to compute my <code>coeff_matrix</code> I thought I could use<code>torch.gather(),</code>but I am unsure about how to do it.<br>\nAdditionally I thought I could abuse the fact that the coefficients I want to take depend on i and j linearly, but I don’t know how.<br>\nThe coefficients are also parameters in the real application, so it is important to not break the gradient.</p>",12 "post_number": 1,13 "post_type": 1,14 "posts_count": 2,15 "updated_at": "2023-06-16T16:15:16.532Z",16 "reply_count": 0,17 "reply_to_post_number": null,18 "quote_count": 0,19 "incoming_link_count": 203,20 "reads": 6,21 "readers_count": 5,22 "score": 1011.2,23 "yours": false,24 "topic_id": 182257,25 "topic_slug": "speed-up-torch-narrow-over-multiple-dimensions",26 "display_username": "",27 "primary_group_name": null,28 "flair_name": null,29 "flair_url": null,30 "flair_bg_color": null,31 "flair_color": null,32 "flair_group_id": null,33 "badges_granted": [],34 "version": 1,35 "can_edit": false,36 "can_delete": false,37 "can_recover": false,38 "can_see_hidden_post": false,39 "can_wiki": false,40 "link_counts": [41 {42 "url": "https://en.wikipedia.org/wiki/De_Boor%27s_algorithm",43 "internal": false,44 "reflection": false,45 "title": "De Boor's algorithm - Wikipedia",46 "clicks": 447 },48 {49 "url": "https://en.wikipedia.org/wiki/B-spline",50 "internal": false,51 "reflection": false,52 "title": "B-spline - Wikipedia",53 "clicks": 054 }55 ],56 "read": true,57 "user_title": null,58 "bookmarked": false,59 "actions_summary": [],60 "moderator": false,61 "admin": false,62 "staff": false,63 "user_id": 67068,64 "hidden": false,65 "trust_level": 1,66 "deleted_at": null,67 "user_deleted": false,68 "edit_reason": null,69 "can_view_edit_history": true,70 "wiki": false,71 "post_url": "/t/speed-up-torch-narrow-over-multiple-dimensions/182257/1",72 "can_accept_answer": false,73 "can_unaccept_answer": false,74 "accepted_answer": false,75 "topic_accepted_answer": null,76 "can_vote": false77 },78 {79 "id": 406499,80 "name": "",81 "username": "weiskohlmoe",82 "avatar_template": "/letter_avatar_proxy/v4/letter/w/eada6e/{size}.png",83 "created_at": "2023-06-18T22:13:29.342Z",84 "cooked": "<p>My Idea about using <code>torch.gather()</code> worked out fine for me. Here is the solution I used:</p>\n<pre><code class=\"lang-auto\">bigger_coeffs = coeffs.unsqueeze(0).unsqueeze(0)\n\nbigger_coeffs = coeffs.expand(batch_size, values_size, -1)\n\nbigger_index_tensor = index_tensor.unsqueeze(-1).expand(index_tensor.shape[0], index_tensor.shape[1], 3)\n\nsub = torch.tensor([2, 1, 0])\n\nindex_tensor_res = bigger_index_tensor - sub[None, None, :]\n\ncoeff_matrix = torch.gather(bigger_coeffs, 2, index_tensor_res)\n</code></pre>",85 "post_number": 2,86 "post_type": 1,87 "posts_count": 2,88 "updated_at": "2023-06-18T22:13:52.410Z",89 "reply_count": 0,90 "reply_to_post_number": null,91 "quote_count": 0,92 "incoming_link_count": 1,93 "reads": 4,94 "readers_count": 3,95 "score": 5.8,96 "yours": false,97 "topic_id": 182257,98 "topic_slug": "speed-up-torch-narrow-over-multiple-dimensions",99 "display_username": "",100 "primary_group_name": null,101 "flair_name": null,102 "flair_url": null,103 "flair_bg_color": null,104 "flair_color": null,105 "flair_group_id": null,106 "badges_granted": [],107 "version": 1,108 "can_edit": false,109 "can_delete": false,110 "can_recover": false,111 "can_see_hidden_post": false,112 "can_wiki": false,113 "read": true,114 "user_title": null,115 "bookmarked": false,116 "actions_summary": [],117 "moderator": false,118 "admin": false,119 "staff": false,120 "user_id": 67068,121 "hidden": false,122 "trust_level": 1,123 "deleted_at": null,124 "user_deleted": false,125 "edit_reason": null,126 "can_view_edit_history": true,127 "wiki": false,128 "post_url": "/t/speed-up-torch-narrow-over-multiple-dimensions/182257/2",129 "can_accept_answer": false,130 "can_unaccept_answer": false,131 "accepted_answer": false,132 "topic_accepted_answer": null133 }134 ],135 "stream": [136 406317,137 406499138 ]139 },140 "timeline_lookup": [141 [142 1,143 862144 ],145 [146 2,147 860148 ]149 ],150 "suggested_topics": [151 {152 "fancy_title": "Problem with fork-like multiprocess Dataloader on Ubuntu",153 "id": 213451,154 "title": "Problem with fork-like multiprocess Dataloader on Ubuntu",155 "slug": "problem-with-fork-like-multiprocess-dataloader-on-ubuntu",156 "posts_count": 1,157 "reply_count": 0,158 "highest_post_number": 1,159 "image_url": null,160 "created_at": "2024-11-26T09:11:42.083Z",161 "last_posted_at": "2024-11-26T09:11:42.224Z",162 "bumped": true,163 "bumped_at": "2024-11-26T09:22:55.705Z",164 "archetype": "regular",165 "unseen": false,166 "pinned": false,167 "unpinned": null,168 "visible": true,169 "closed": false,170 "archived": false,171 "bookmarked": null,172 "liked": null,173 "tags_descriptions": {},174 "like_count": 0,175 "views": 82,176 "category_id": 1,177 "featured_link": null,178 "has_accepted_answer": false,179 "posters": [180 {181 "extras": "latest single",182 "description": "Original Poster, Most Recent Poster",183 "user": {184 "id": 81132,185 "username": "neyronon",186 "name": "Victor Lg",187 "avatar_template": "/user_avatar/discuss.pytorch.org/neyronon/{size}/74205_2.png",188 "trust_level": 1189 }190 }191 ]192 },193 {194 "fancy_title": "Using BatchNorm1d for standardization",195 "id": 214473,196 "title": "Using BatchNorm1d for standardization",197 "slug": "using-batchnorm1d-for-standardization",198 "posts_count": 2,199 "reply_count": 0,200 "highest_post_number": 2,201 "image_url": null,202 "created_at": "2024-12-20T23:17:37.293Z",203 "last_posted_at": "2024-12-20T23:21:22.922Z",204 "bumped": true,205 "bumped_at": "2024-12-20T23:21:22.922Z",206 "archetype": "regular",207 "unseen": false,208 "pinned": false,209 "unpinned": null,210 "visible": true,211 "closed": false,212 "archived": false,213 "bookmarked": null,214 "liked": null,215 "tags_descriptions": {},216 "like_count": 0,217 "views": 35,218 "category_id": 1,219 "featured_link": null,220 "has_accepted_answer": false,221 "posters": [222 {223 "extras": null,224 "description": "Original Poster",225 "user": {226 "id": 65042,227 "username": "ado_sar",228 "name": "ado sar",229 "avatar_template": "/user_avatar/discuss.pytorch.org/ado_sar/{size}/59241_2.png",230 "trust_level": 2231 }232 },233 {234 "extras": "latest",235 "description": "Most Recent Poster",236 "user": {237 "id": 41396,238 "username": "soulitzer",239 "name": "",240 "avatar_template": "/letter_avatar_proxy/v4/letter/s/839c29/{size}.png",241 "trust_level": 2242 }243 }244 ]245 },246 {247 "fancy_title": "How to disable these two types of log output?",248 "id": 214492,249 "title": "How to disable these two types of log output?",250 "slug": "how-to-disable-these-two-types-of-log-output",251 "posts_count": 5,252 "reply_count": 3,253 "highest_post_number": 5,254 "image_url": null,255 "created_at": "2024-12-21T13:12:19.618Z",256 "last_posted_at": "2024-12-23T01:54:08.647Z",257 "bumped": true,258 "bumped_at": "2024-12-23T01:54:08.647Z",259 "archetype": "regular",260 "unseen": false,261 "pinned": false,262 "unpinned": null,263 "visible": true,264 "closed": false,265 "archived": false,266 "bookmarked": null,267 "liked": null,268 "tags_descriptions": {},269 "like_count": 1,270 "views": 152,271 "category_id": 1,272 "featured_link": null,273 "has_accepted_answer": false,274 "posters": [275 {276 "extras": "latest",277 "description": "Original Poster, Most Recent Poster",278 "user": {279 "id": 17807,280 "username": "AlexLuya",281 "name": "Alex Luya",282 "avatar_template": "/user_avatar/discuss.pytorch.org/alexluya/{size}/15408_2.png",283 "trust_level": 1284 }285 },286 {287 "extras": null,288 "description": "Frequent Poster",289 "user": {290 "id": 3534,291 "username": "ptrblck",292 "name": "",293 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",294 "admin": true,295 "moderator": true,296 "trust_level": 2297 }298 }299 ]300 },301 {302 "fancy_title": "How to categorize tensor float values into long values?",303 "id": 214711,304 "title": "How to categorize tensor float values into long values?",305 "slug": "how-to-categorize-tensor-float-values-into-long-values",306 "posts_count": 2,307 "reply_count": 0,308 "highest_post_number": 2,309 "image_url": null,310 "created_at": "2024-12-27T20:32:46.532Z",311 "last_posted_at": "2024-12-27T23:52:07.865Z",312 "bumped": true,313 "bumped_at": "2024-12-27T23:52:07.865Z",314 "archetype": "regular",315 "unseen": false,316 "pinned": false,317 "unpinned": null,318 "visible": true,319 "closed": false,320 "archived": false,321 "bookmarked": null,322 "liked": null,323 "tags_descriptions": {},324 "like_count": 0,325 "views": 33,326 "category_id": 1,327 "featured_link": null,328 "has_accepted_answer": true,329 "posters": [330 {331 "extras": null,332 "description": "Original Poster",333 "user": {334 "id": 78321,335 "username": "Omaralmaqtari",336 "name": "Omar al-maqtari",337 "avatar_template": "/user_avatar/discuss.pytorch.org/omaralmaqtari/{size}/72219_2.png",338 "trust_level": 1339 }340 },341 {342 "extras": "latest",343 "description": "Most Recent Poster, Accepted Answer",344 "user": {345 "id": 41396,346 "username": "soulitzer",347 "name": "",348 "avatar_template": "/letter_avatar_proxy/v4/letter/s/839c29/{size}.png",349 "trust_level": 2350 }351 }352 ]353 },354 {355 "fancy_title": "2 gpu model loading cuda error",356 "id": 216669,357 "title": "2 gpu model loading cuda error",358 "slug": "2-gpu-model-loading-cuda-error",359 "posts_count": 1,360 "reply_count": 0,361 "highest_post_number": 1,362 "image_url": null,363 "created_at": "2025-02-14T10:56:48.064Z",364 "last_posted_at": "2025-02-14T10:56:48.102Z",365 "bumped": true,366 "bumped_at": "2025-02-14T11:01:40.215Z",367 "archetype": "regular",368 "unseen": false,369 "pinned": false,370 "unpinned": null,371 "visible": true,372 "closed": false,373 "archived": false,374 "bookmarked": null,375 "liked": null,376 "tags_descriptions": {},377 "like_count": 0,378 "views": 87,379 "category_id": 1,380 "featured_link": null,381 "has_accepted_answer": false,382 "posters": [383 {384 "extras": "latest single",385 "description": "Original Poster, Most Recent Poster",386 "user": {387 "id": 82364,388 "username": "Sourabh_Yadav",389 "name": "Sourabh Yadav",390 "avatar_template": "/user_avatar/discuss.pytorch.org/sourabh_yadav/{size}/75350_2.png",391 "trust_level": 1392 }393 }394 ]395 }396 ],397 "tags_descriptions": {},398 "fancy_title": "Speed up torch.narrow over multiple dimensions",399 "id": 182257,400 "title": "Speed up torch.narrow over multiple dimensions",401 "posts_count": 2,402 "created_at": "2023-06-16T16:15:16.454Z",403 "views": 465,404 "reply_count": 0,405 "like_count": 0,406 "last_posted_at": "2023-06-18T22:13:29.342Z",407 "visible": true,408 "closed": false,409 "archived": false,410 "has_summary": false,411 "archetype": "regular",412 "slug": "speed-up-torch-narrow-over-multiple-dimensions",413 "category_id": 1,414 "word_count": 414,415 "deleted_at": null,416 "user_id": 67068,417 "featured_link": null,418 "pinned_globally": false,419 "pinned_at": null,420 "pinned_until": null,421 "image_url": null,422 "slow_mode_seconds": 0,423 "draft": null,424 "draft_key": "topic_182257",425 "draft_sequence": null,426 "unpinned": null,427 "pinned": false,428 "current_post_number": 1,429 "highest_post_number": 2,430 "deleted_by": null,431 "actions_summary": [432 {433 "id": 4,434 "count": 0,435 "hidden": false,436 "can_act": false437 },438 {439 "id": 8,440 "count": 0,441 "hidden": false,442 "can_act": false443 },444 {445 "id": 10,446 "count": 0,447 "hidden": false,448 "can_act": false449 },450 {451 "id": 7,452 "count": 0,453 "hidden": false,454 "can_act": false455 }456 ],457 "chunk_size": 20,458 "bookmarked": false,459 "topic_timer": null,460 "message_bus_last_id": 0,461 "participant_count": 1,462 "show_read_indicator": false,463 "thumbnails": null,464 "slow_mode_enabled_until": null,465 "can_vote": false,466 "vote_count": 0,467 "user_voted": false,468 "discourse_zendesk_plugin_zendesk_id": null,469 "discourse_zendesk_plugin_zendesk_url": "https://your-url.zendesk.com/agent/tickets/",470 "details": {471 "can_edit": false,472 "notification_level": 1,473 "participants": [474 {475 "id": 67068,476 "username": "weiskohlmoe",477 "name": "",478 "avatar_template": "/letter_avatar_proxy/v4/letter/w/eada6e/{size}.png",479 "post_count": 2,480 "primary_group_name": null,481 "flair_name": null,482 "flair_url": null,483 "flair_color": null,484 "flair_bg_color": null,485 "flair_group_id": null,486 "trust_level": 1487 }488 ],489 "created_by": {490 "id": 67068,491 "username": "weiskohlmoe",492 "name": "",493 "avatar_template": "/letter_avatar_proxy/v4/letter/w/eada6e/{size}.png"494 },495 "last_poster": {496 "id": 67068,497 "username": "weiskohlmoe",498 "name": "",499 "avatar_template": "/letter_avatar_proxy/v4/letter/w/eada6e/{size}.png"500 },501 "links": [502 {503 "url": "https://en.wikipedia.org/wiki/De_Boor%27s_algorithm",504 "title": "De Boor's algorithm - Wikipedia",505 "internal": false,506 "attachment": false,507 "reflection": false,508 "clicks": 4,509 "user_id": 67068,510 "domain": "en.wikipedia.org",511 "root_domain": "wikipedia.org"512 }513 ]514 },515 "bookmarks": []516 },517 {518 "post_stream": {519 "posts": [520 {521 "id": 406483,522 "name": "",523 "username": "Picus",524 "avatar_template": "/user_avatar/discuss.pytorch.org/picus/{size}/61399_2.png",525 "created_at": "2023-06-18T16:43:20.913Z",526 "cooked": "<p>Hi!</p>\n<p>For the first time, I’m trying to use a LLM from Jupyterlab and not from an UI. I already met a lot of technical issues but I don’t have any idea about how to fix this one.</p>\n<p>I already installed all the necessary libraries including GPTQ_for_LlaMa (for cuda).</p>\n<p>Here is my code to load this model : <a href=\"https://huggingface.co/TheBloke/Wizard-Vicuna-13B-Uncensored-GPTQ\" class=\"inline-onebox\" rel=\"noopener nofollow ugc\">TheBloke/Wizard-Vicuna-13B-Uncensored-GPTQ · Hugging Face</a></p>\n<pre><code class=\"lang-auto\">import sys\nimport os\nimport guidance\nimport transformers\n\nsys.path.append(os.path.realpath(\"./libs/gptq\")+\"/\")\nimport llama_inference\n\nllama_inference.transformers = transformers\n\ntokenizer = transformers.LlamaTokenizer.from_pretrained(\"TheBloke/Wizard-Vicuna-13B-Uncensored-GPTQ\")\n\nmodel = llama_inference.load_quant(\"TheBloke/Wizard-Vicuna-13B-Uncensored-GPTQ\",\"Wizard-Vicuna-13B-Uncensored-GPTQ-4bit-128g.compat.no-act-order.safetensors\",4,128,0)\n</code></pre>\n<p>But after that, during the generation using Guidance, I get :</p>\n<pre><code class=\"lang-auto\">llm = guidance.llms.transformers.Vicuna(model=model, tokenizer=tokenizer)\n\n# we can pre-define valid option sets\nvalid_weapons = [\"sword\", \"axe\", \"mace\", \"spear\", \"bow\", \"crossbow\"]\n\n# define the prompt\ncharacter_maker = guidance(\"\"\"The following is a character profile for an RPG game in JSON format.\n```json\n{\n \"id\": \"{{id}}\",\n \"description\": \"{{description}}\",\n \"name\": \"{{gen 'name'}}\",\n \"age\": {{gen 'age' pattern='[0-9]+' stop=','}},\n \"armor\": \"{{#select 'armor'}}leather{{or}}chainmail{{or}}plate{{/select}}\",\n \"weapon\": \"{{select 'weapon' options=valid_weapons}}\",\n \"class\": \"{{gen 'class'}}\",\n \"mantra\": \"{{gen 'mantra' temperature=0.7}}\",\n \"strength\": {{gen 'strength' pattern='[0-9]+' stop=','}},\n \"items\": [{{#geneach 'items' num_iterations=5 join=', '}}\"{{gen 'this' temperature=0.7}}\"{{/geneach}}]\n}```\"\"\")\n\n# generate a character\ncharacter_maker(\n id=\"e1f491f7-7ab8-4dac-8c20-c92b5e7d883d\",\n description=\"A quick and nimble fighter.\",\n valid_weapons=valid_weapons, llm=llm\n)\n\n\n#===================================\n\n\nException in thread Thread-6 (generate):\nTraceback (most recent call last):\n\n File \"/home/x00/anaconda3/envs/GPT/lib/python3.10/threading.py\", line 1016, in _bootstrap_inner\n self.run()\n\n File \"/home/x00/anaconda3/envs/GPT/lib/python3.10/threading.py\", line 953, in run\n self._target(*self._args, **self._kwargs)\n\n File \"/home/x00/anaconda3/envs/GPT/lib/python3.10/site-packages/torch/utils/_contextlib.py\", line 115, in decorate_context\n return func(*args, **kwargs)\n\n File \"/home/x00/anaconda3/envs/GPT/lib/python3.10/site-packages/transformers/generation/utils.py\", line 1522, in generate\n return self.greedy_search(\n\n File \"/home/x00/anaconda3/envs/GPT/lib/python3.10/site-packages/transformers/generation/utils.py\", line 2339, in greedy_search\n outputs = self(\n\n File \"/home/x00/anaconda3/envs/GPT/lib/python3.10/site-packages/torch/nn/modules/module.py\", line 1501, in _call_impl\n return forward_call(*args, **kwargs)\n\n File \"/home/x00/anaconda3/envs/GPT/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py\", line 691, in forward\n outputs = self.model(\n\n File \"/home/x00/anaconda3/envs/GPT/lib/python3.10/site-packages/torch/nn/modules/module.py\", line 1501, in _call_impl\n return forward_call(*args, **kwargs)\n\n File \"/home/x00/anaconda3/envs/GPT/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py\", line 579, in forward\n layer_outputs = decoder_layer(\n\n File \"/home/x00/anaconda3/envs/GPT/lib/python3.10/site-packages/torch/nn/modules/module.py\", line 1501, in _call_impl\n return forward_call(*args, **kwargs)\n\n File \"/home/x00/anaconda3/envs/GPT/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py\", line 293, in forward\n hidden_states, self_attn_weights, present_key_value = self.self_attn(\n\n File \"/home/x00/anaconda3/envs/GPT/lib/python3.10/site-packages/torch/nn/modules/module.py\", line 1501, in _call_impl\n return forward_call(*args, **kwargs)\n\n File \"/home/x00/anaconda3/envs/GPT/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py\", line 195, in forward\n query_states = self.q_proj(hidden_states).view(bsz, q_len, self.num_heads, self.head_dim).transpose(1, 2)\n\n File \"/home/x00/anaconda3/envs/GPT/lib/python3.10/site-packages/torch/nn/modules/module.py\", line 1501, in _call_impl\n return forward_call(*args, **kwargs)\n\n File \"/home/x00/Bureau/libs/gptq/quant.py\", line 279, in forward\n quant_cuda.vecquant4matmul(x.float(), self.qweight, out, self.scales.float(), self.qzeros, self.g_idx)\n\nRuntimeError: t == DeviceType::CUDA INTERNAL ASSERT FAILED at \"/home/x00/anaconda3/envs/GPT/lib/python3.10/site-packages/torch/include/ATen/hip/impl/HIPGuardImplMasqueradingAsCUDA.h\":60, please report a bug to PyTorch.\n</code></pre>\n<p>My PC :</p>\n<ul>\n<li>\n<p>Linux mint cinnamon</p>\n</li>\n<li>\n<p>AMD Radeon 6800XT</p>\n</li>\n</ul>\n<p>I’m really lost now, I don’t even know where to start. What should I do to fix this? I tried to add everything I could but do not hesitate to ask me the questions you could have.</p>",527 "post_number": 1,528 "post_type": 1,529 "posts_count": 2,530 "updated_at": "2023-06-18T16:43:20.913Z",531 "reply_count": 0,532 "reply_to_post_number": null,533 "quote_count": 0,534 "incoming_link_count": 149,535 "reads": 8,536 "readers_count": 7,537 "score": 746.6,538 "yours": false,539 "topic_id": 182361,540 "topic_slug": "do-i-need-to-renaming-keys-in-state-dict-warning-beginner",541 "display_username": "",542 "primary_group_name": null,543 "flair_name": null,544 "flair_url": null,545 "flair_bg_color": null,546 "flair_color": null,547 "flair_group_id": null,548 "badges_granted": [],549 "version": 1,550 "can_edit": false,551 "can_delete": false,552 "can_recover": false,553 "can_see_hidden_post": false,554 "can_wiki": false,555 "link_counts": [556 {557 "url": "https://huggingface.co/TheBloke/Wizard-Vicuna-13B-Uncensored-GPTQ",558 "internal": false,559 "reflection": false,560 "title": "TheBloke/Wizard-Vicuna-13B-Uncensored-GPTQ · Hugging Face",561 "clicks": 2562 }563 ],564 "read": true,565 "user_title": null,566 "bookmarked": false,567 "actions_summary": [],568 "moderator": false,569 "admin": false,570 "staff": false,571 "user_id": 67107,572 "hidden": false,573 "trust_level": 1,574 "deleted_at": null,575 "user_deleted": false,576 "edit_reason": null,577 "can_view_edit_history": true,578 "wiki": false,579 "post_url": "/t/do-i-need-to-renaming-keys-in-state-dict-warning-beginner/182361/1",580 "can_accept_answer": false,581 "can_unaccept_answer": false,582 "accepted_answer": false,583 "topic_accepted_answer": null,584 "can_vote": false585 },586 {587 "id": 406493,588 "name": "",589 "username": "ptrblck",590 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",591 "created_at": "2023-06-18T19:54:51.895Z",592 "cooked": "<p>It seems you are running into an internal assert in the rocm stack. I would recommend creating an issue on GitHub with a minimal and executable code snippet (if possible) so that the code owners could take a look at it.</p>",593 "post_number": 2,594 "post_type": 1,595 "posts_count": 2,596 "updated_at": "2023-06-18T19:54:51.895Z",597 "reply_count": 0,598 "reply_to_post_number": null,599 "quote_count": 0,600 "incoming_link_count": 1,601 "reads": 7,602 "readers_count": 6,603 "score": 6.4,604 "yours": false,605 "topic_id": 182361,606 "topic_slug": "do-i-need-to-renaming-keys-in-state-dict-warning-beginner",607 "display_username": "",608 "primary_group_name": null,609 "flair_name": null,610 "flair_url": null,611 "flair_bg_color": null,612 "flair_color": null,613 "flair_group_id": null,614 "badges_granted": [],615 "version": 1,616 "can_edit": false,617 "can_delete": false,618 "can_recover": false,619 "can_see_hidden_post": false,620 "can_wiki": false,621 "read": true,622 "user_title": "",623 "bookmarked": false,624 "actions_summary": [],625 "moderator": true,626 "admin": true,627 "staff": true,628 "user_id": 3534,629 "hidden": false,630 "trust_level": 2,631 "deleted_at": null,632 "user_deleted": false,633 "edit_reason": null,634 "can_view_edit_history": true,635 "wiki": false,636 "post_url": "/t/do-i-need-to-renaming-keys-in-state-dict-warning-beginner/182361/2",637 "can_accept_answer": false,638 "can_unaccept_answer": false,639 "accepted_answer": false,640 "topic_accepted_answer": null641 }642 ],643 "stream": [644 406483,645 406493646 ]647 },648 "timeline_lookup": [649 [650 1,651 860652 ]653 ],654 "suggested_topics": [655 {656 "fancy_title": "LSTM model does not change predictions",657 "id": 215006,658 "title": "LSTM model does not change predictions",659 "slug": "lstm-model-does-not-change-predictions",660 "posts_count": 3,661 "reply_count": 0,662 "highest_post_number": 3,663 "image_url": null,664 "created_at": "2025-01-05T19:24:25.918Z",665 "last_posted_at": "2025-01-06T07:54:46.550Z",666 "bumped": true,667 "bumped_at": "2025-01-06T07:54:46.550Z",668 "archetype": "regular",669 "unseen": false,670 "pinned": false,671 "unpinned": null,672 "visible": true,673 "closed": false,674 "archived": false,675 "bookmarked": null,676 "liked": null,677 "tags_descriptions": {},678 "like_count": 0,679 "views": 107,680 "category_id": 1,681 "featured_link": null,682 "has_accepted_answer": false,683 "posters": [684 {685 "extras": "latest",686 "description": "Original Poster, Most Recent Poster",687 "user": {688 "id": 81891,689 "username": "Ale_DZ",690 "name": "Ale DZ",691 "avatar_template": "/user_avatar/discuss.pytorch.org/ale_dz/{size}/74527_2.png",692 "trust_level": 1693 }694 },695 {696 "extras": null,697 "description": "Frequent Poster",698 "user": {699 "id": 41396,700 "username": "soulitzer",701 "name": "",702 "avatar_template": "/letter_avatar_proxy/v4/letter/s/839c29/{size}.png",703 "trust_level": 2704 }705 }706 ]707 },708 {709 "fancy_title": "`sm_89` not listed in the `torch.cuda.get_arch_list()`",710 "id": 215827,711 "title": "`sm_89` not listed in the `torch.cuda.get_arch_list()`",712 "slug": "sm-89-not-listed-in-the-torch-cuda-get-arch-list",713 "posts_count": 6,714 "reply_count": 4,715 "highest_post_number": 6,716 "image_url": null,717 "created_at": "2025-01-24T14:41:00.900Z",718 "last_posted_at": "2025-01-24T20:32:06.683Z",719 "bumped": true,720 "bumped_at": "2025-01-24T20:32:06.683Z",721 "archetype": "regular",722 "unseen": false,723 "pinned": false,724 "unpinned": null,725 "visible": true,726 "closed": false,727 "archived": false,728 "bookmarked": null,729 "liked": null,730 "tags_descriptions": {},731 "like_count": 1,732 "views": 997,733 "category_id": 1,734 "featured_link": null,735 "has_accepted_answer": false,736 "posters": [737 {738 "extras": null,739 "description": "Original Poster",740 "user": {741 "id": 3938,742 "username": "vgoklani",743 "name": "Vishal Goklani",744 "avatar_template": "/user_avatar/discuss.pytorch.org/vgoklani/{size}/1971_2.png",745 "trust_level": 1746 }747 },748 {749 "extras": "latest",750 "description": "Most Recent Poster",751 "user": {752 "id": 3534,753 "username": "ptrblck",754 "name": "",755 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",756 "admin": true,757 "moderator": true,758 "trust_level": 2759 }760 }761 ]762 },763 {764 "fancy_title": "Error trying to import function from torchtext",765 "id": 216555,766 "title": "Error trying to import function from torchtext",767 "slug": "error-trying-to-import-function-from-torchtext",768 "posts_count": 5,769 "reply_count": 2,770 "highest_post_number": 5,771 "image_url": null,772 "created_at": "2025-02-12T03:51:04.961Z",773 "last_posted_at": "2025-02-18T17:46:12.987Z",774 "bumped": true,775 "bumped_at": "2025-02-18T17:46:12.987Z",776 "archetype": "regular",777 "unseen": false,778 "pinned": false,779 "unpinned": null,780 "visible": true,781 "closed": false,782 "archived": false,783 "bookmarked": null,784 "liked": null,785 "tags_descriptions": {},786 "like_count": 0,787 "views": 247,788 "category_id": 1,789 "featured_link": null,790 "has_accepted_answer": false,791 "posters": [792 {793 "extras": null,794 "description": "Original Poster",795 "user": {796 "id": 784,797 "username": "EvanZ",798 "name": "Evan Zamir",799 "avatar_template": "/user_avatar/discuss.pytorch.org/evanz/{size}/372_2.png",800 "trust_level": 2801 }802 },803 {804 "extras": "latest",805 "description": "Most Recent Poster",806 "user": {807 "id": 3534,808 "username": "ptrblck",809 "name": "",810 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",811 "admin": true,812 "moderator": true,813 "trust_level": 2814 }815 }816 ]817 },818 {819 "fancy_title": "How to install Torch version that supports RTX 5090 on Windows? - CUDA kernel errors might be asynchronously reported at some other API call",820 "id": 216644,821 "title": "How to install Torch version that supports RTX 5090 on Windows? - CUDA kernel errors might be asynchronously reported at some other API call",822 "slug": "how-to-install-torch-version-that-supports-rtx-5090-on-windows-cuda-kernel-errors-might-be-asynchronously-reported-at-some-other-api-call",823 "posts_count": 4,824 "reply_count": 2,825 "highest_post_number": 4,826 "image_url": null,827 "created_at": "2025-02-13T20:16:06.488Z",828 "last_posted_at": "2025-02-20T13:51:59.917Z",829 "bumped": true,830 "bumped_at": "2025-02-20T13:51:59.917Z",831 "archetype": "regular",832 "unseen": false,833 "pinned": false,834 "unpinned": null,835 "visible": true,836 "closed": false,837 "archived": false,838 "bookmarked": null,839 "liked": null,840 "tags_descriptions": {},841 "like_count": 3,842 "views": 5624,843 "category_id": 1,844 "featured_link": null,845 "has_accepted_answer": false,846 "posters": [847 {848 "extras": null,849 "description": "Original Poster",850 "user": {851 "id": 59578,852 "username": "FurkanGozukara",853 "name": "Furkan Gözükara",854 "avatar_template": "/user_avatar/discuss.pytorch.org/furkangozukara/{size}/53457_2.png",855 "trust_level": 1856 }857 },858 {859 "extras": null,860 "description": "Frequent Poster",861 "user": {862 "id": 3534,863 "username": "ptrblck",864 "name": "",865 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",866 "admin": true,867 "moderator": true,868 "trust_level": 2869 }870 },871 {872 "extras": "latest",873 "description": "Most Recent Poster",874 "user": {875 "id": 82812,876 "username": "pzlong",877 "name": "pzlong",878 "avatar_template": "/letter_avatar_proxy/v4/letter/p/5f8ce5/{size}.png",879 "trust_level": 0880 }881 }882 ]883 },884 {885 "fancy_title": "Will profiler.record_function be affected by the asynchronous execution?",886 "id": 214993,887 "title": "Will profiler.record_function be affected by the asynchronous execution?",888 "slug": "will-profiler-record-function-be-affected-by-the-asynchronous-execution",889 "posts_count": 2,890 "reply_count": 0,891 "highest_post_number": 2,892 "image_url": null,893 "created_at": "2025-01-05T09:31:28.370Z",894 "last_posted_at": "2025-01-05T19:46:14.748Z",895 "bumped": true,896 "bumped_at": "2025-01-05T19:46:14.748Z",897 "archetype": "regular",898 "unseen": false,899 "pinned": false,900 "unpinned": null,901 "visible": true,902 "closed": false,903 "archived": false,904 "bookmarked": null,905 "liked": null,906 "tags_descriptions": {},907 "like_count": 0,908 "views": 41,909 "category_id": 1,910 "featured_link": null,911 "has_accepted_answer": false,912 "posters": [913 {914 "extras": null,915 "description": "Original Poster",916 "user": {917 "id": 81887,918 "username": "ideaV",919 "name": "",920 "avatar_template": "/user_avatar/discuss.pytorch.org/ideav/{size}/74914_2.png",921 "trust_level": 1922 }923 },924 {925 "extras": "latest",926 "description": "Most Recent Poster",927 "user": {928 "id": 41396,929 "username": "soulitzer",930 "name": "",931 "avatar_template": "/letter_avatar_proxy/v4/letter/s/839c29/{size}.png",932 "trust_level": 2933 }934 }935 ]936 }937 ],938 "tags_descriptions": {},939 "fancy_title": "Do I need to renaming keys in state_dict? (warning : beginner)",940 "id": 182361,941 "title": "Do I need to renaming keys in state_dict? (warning : beginner)",942 "posts_count": 2,943 "created_at": "2023-06-18T16:43:20.832Z",944 "views": 482,945 "reply_count": 0,946 "like_count": 0,947 "last_posted_at": "2023-06-18T19:54:51.895Z",948 "visible": true,949 "closed": false,950 "archived": false,951 "has_summary": false,952 "archetype": "regular",953 "slug": "do-i-need-to-renaming-keys-in-state-dict-warning-beginner",954 "category_id": 1,955 "word_count": 736,956 "deleted_at": null,957 "user_id": 67107,958 "featured_link": null,959 "pinned_globally": false,960 "pinned_at": null,961 "pinned_until": null,962 "image_url": null,963 "slow_mode_seconds": 0,964 "draft": null,965 "draft_key": "topic_182361",966 "draft_sequence": null,967 "unpinned": null,968 "pinned": false,969 "current_post_number": 1,970 "highest_post_number": 2,971 "deleted_by": null,972 "actions_summary": [973 {974 "id": 4,975 "count": 0,976 "hidden": false,977 "can_act": false978 },979 {980 "id": 8,981 "count": 0,982 "hidden": false,983 "can_act": false984 },985 {986 "id": 10,987 "count": 0,988 "hidden": false,989 "can_act": false990 },991 {992 "id": 7,993 "count": 0,994 "hidden": false,995 "can_act": false996 }997 ],998 "chunk_size": 20,999 "bookmarked": false,1000 "topic_timer": null,1001 "message_bus_last_id": 0,1002 "participant_count": 2,1003 "show_read_indicator": false,1004 "thumbnails": null,1005 "slow_mode_enabled_until": null,1006 "can_vote": false,1007 "vote_count": 0,1008 "user_voted": false,1009 "discourse_zendesk_plugin_zendesk_id": null,1010 "discourse_zendesk_plugin_zendesk_url": "https://your-url.zendesk.com/agent/tickets/",1011 "details": {1012 "can_edit": false,1013 "notification_level": 1,1014 "participants": [1015 {1016 "id": 3534,1017 "username": "ptrblck",1018 "name": "",1019 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",1020 "post_count": 1,1021 "primary_group_name": null,1022 "flair_name": null,1023 "flair_url": null,1024 "flair_color": null,1025 "flair_bg_color": null,1026 "flair_group_id": null,1027 "admin": true,1028 "moderator": true,1029 "trust_level": 21030 },1031 {1032 "id": 67107,1033 "username": "Picus",1034 "name": "",1035 "avatar_template": "/user_avatar/discuss.pytorch.org/picus/{size}/61399_2.png",1036 "post_count": 1,1037 "primary_group_name": null,1038 "flair_name": null,1039 "flair_url": null,1040 "flair_color": null,1041 "flair_bg_color": null,1042 "flair_group_id": null,1043 "trust_level": 11044 }1045 ],1046 "created_by": {1047 "id": 67107,1048 "username": "Picus",1049 "name": "",1050 "avatar_template": "/user_avatar/discuss.pytorch.org/picus/{size}/61399_2.png"1051 },1052 "last_poster": {1053 "id": 3534,1054 "username": "ptrblck",1055 "name": "",1056 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png"1057 },1058 "links": [1059 {1060 "url": "https://huggingface.co/TheBloke/Wizard-Vicuna-13B-Uncensored-GPTQ",1061 "title": "TheBloke/Wizard-Vicuna-13B-Uncensored-GPTQ · Hugging Face",1062 "internal": false,1063 "attachment": false,1064 "reflection": false,1065 "clicks": 2,1066 "user_id": 67107,1067 "domain": "huggingface.co",1068 "root_domain": "huggingface.co"1069 }1070 ]1071 },1072 "bookmarks": []1073 },1074 {1075 "post_stream": {1076 "posts": [1077 {1078 "id": 293239,1079 "name": "Muhammad Zaid",1080 "username": "Zaid",1081 "avatar_template": "/user_avatar/discuss.pytorch.org/zaid/{size}/32466_2.png",1082 "created_at": "2021-06-30T10:44:12.656Z",1083 "cooked": "<p>Hi,</p>\n<p>So I have a dataset class responsible to read from two directories having different sizes. I have the following code for my dataset class:</p>\n<pre><code class=\"lang-auto\"> def __getitem__(self, idx):\n\n idx_ood = random.randint(0,5374)\n image = Image.open(\n os.path.join(\n self.path_to_images,\n self.df.index[idx]))\n image = image.convert('RGB')\n\n image_ood = Image.open(self.ood_names[idx_ood])\n image_ood = image_ood.convert('RGB')\n\n label = np.zeros(len(self.PRED_LABEL), dtype=int)\n for i in range(0, len(self.PRED_LABEL)):\n # can leave zero if zero, else make one\n if(self.df[self.PRED_LABEL[i].strip()].iloc[idx].astype('int') > 0):\n label[i] = self.df[self.PRED_LABEL[i].strip()\n ].iloc[idx].astype('int')\n\n if self.transform:\n image = self.transform(image)\n image_ood_tr = self.transform(image_ood)\n\n if torch.any(torch.isnan(image_ood_tr)):\n print(\"NAN in ood input image!\")\n\n return (image, label,self.df.index[idx]),(image_ood_tr,idx_ood,self.totensor(image_ood))\n\n</code></pre>\n<p>When I am fetching data via data loader, one of the images has nan at an element inside the tensor. While debugging, I checked the original image before transform and it does not have any nan. When I try to run the code again, there was no nan in the same image tensor.</p>\n<p>Could this be a hardware issue? Or can you find any error?</p>\n<p>I would be grateful for your help.</p>",1084 "post_number": 1,1085 "post_type": 1,1086 "posts_count": 6,1087 "updated_at": "2021-06-30T11:40:49.964Z",1088 "reply_count": 0,1089 "reply_to_post_number": null,1090 "quote_count": 0,1091 "incoming_link_count": 1010,1092 "reads": 33,1093 "readers_count": 32,1094 "score": 5036.6,1095 "yours": false,1096 "topic_id": 125455,1097 "topic_slug": "input-is-nan-after-transformation",1098 "display_username": "Muhammad Zaid",1099 "primary_group_name": null,1100 "flair_name": null,1101 "flair_url": null,1102 "flair_bg_color": null,1103 "flair_color": null,1104 "flair_group_id": null,1105 "badges_granted": [],1106 "version": 2,1107 "can_edit": false,1108 "can_delete": false,1109 "can_recover": false,1110 "can_see_hidden_post": false,1111 "can_wiki": false,1112 "read": true,1113 "user_title": null,1114 "bookmarked": false,1115 "actions_summary": [],1116 "moderator": false,1117 "admin": false,1118 "staff": false,1119 "user_id": 46760,1120 "hidden": false,1121 "trust_level": 1,1122 "deleted_at": null,1123 "user_deleted": false,1124 "edit_reason": null,1125 "can_view_edit_history": true,1126 "wiki": false,1127 "post_url": "/t/input-is-nan-after-transformation/125455/1",1128 "can_accept_answer": false,1129 "can_unaccept_answer": false,1130 "accepted_answer": false,1131 "topic_accepted_answer": null,1132 "can_vote": false1133 },1134 {1135 "id": 293316,1136 "name": "",1137 "username": "ptrblck",1138 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",1139 "created_at": "2021-06-30T21:49:01.205Z",1140 "cooked": "<p>Could you check the tensors for invalid values before and after applying the transform and run a few epochs? Based on your description it seems that this issue is not reproducible, but is visible randomly?</p>",1141 "post_number": 2,1142 "post_type": 1,1143 "posts_count": 6,1144 "updated_at": "2021-06-30T21:49:01.205Z",1145 "reply_count": 1,1146 "reply_to_post_number": null,1147 "quote_count": 0,1148 "incoming_link_count": 4,1149 "reads": 28,1150 "readers_count": 27,1151 "score": 30.6,1152 "yours": false,1153 "topic_id": 125455,1154 "topic_slug": "input-is-nan-after-transformation",1155 "display_username": "",1156 "primary_group_name": null,1157 "flair_name": null,1158 "flair_url": null,1159 "flair_bg_color": null,1160 "flair_color": null,1161 "flair_group_id": null,1162 "badges_granted": [],1163 "version": 1,1164 "can_edit": false,1165 "can_delete": false,1166 "can_recover": false,1167 "can_see_hidden_post": false,1168 "can_wiki": false,1169 "read": true,1170 "user_title": "",1171 "bookmarked": false,1172 "actions_summary": [],1173 "moderator": true,1174 "admin": true,1175 "staff": true,1176 "user_id": 3534,1177 "hidden": false,1178 "trust_level": 2,1179 "deleted_at": null,1180 "user_deleted": false,1181 "edit_reason": null,1182 "can_view_edit_history": true,1183 "wiki": false,1184 "post_url": "/t/input-is-nan-after-transformation/125455/2",1185 "can_accept_answer": false,1186 "can_unaccept_answer": false,1187 "accepted_answer": false,1188 "topic_accepted_answer": null1189 },1190 {1191 "id": 293613,1192 "name": "Muhammad Zaid",1193 "username": "Zaid",1194 "avatar_template": "/user_avatar/discuss.pytorch.org/zaid/{size}/32466_2.png",1195 "created_at": "2021-07-02T11:03:35.104Z",1196 "cooked": "<p>Hi <a class=\"mention\" href=\"/u/ptrblck\">@ptrblck</a>, yes you are absolutely correct. The issue is visible and random.</p>\n<p>I did check the tensors before and after applying the transformation. The tensor before the transformation has no NaNs. Interestingly, I opened the debug console and applied transformation and there was no NaN. But the same transformation produce NaN randomly when running.</p>\n<p>Other than that, the same tensor does not give any NaN in another run of the same code. I am unable to comprehend it.</p>\n<p>Do you think the issue could be because of reading two datasets of different sizes (though it should not be the case)? Or could you recommend anything else?</p>",1197 "post_number": 3,1198 "post_type": 1,1199 "posts_count": 6,1200 "updated_at": "2021-07-02T11:03:35.104Z",