Anurag1734/cuda-error-resolution-analysis
07
1[2 {3 "post_stream": {4 "posts": [5 {6 "id": 444725,7 "name": "",8 "username": "rXp",9 "avatar_template": "/letter_avatar_proxy/v4/letter/r/3ab097/{size}.png",10 "created_at": "2024-06-03T09:39:30.554Z",11 "cooked": "<p>Environment :</p>\n<ul>\n<li>3050 Ti</li>\n<li>Cuda 12.5 (latest drivers)</li>\n<li>Windows 11</li>\n<li>Miniconda 3</li>\n</ul>\n<p>I’ve been trying to make PyTorch Cuda work for two days but I have been unable to.</p>\n<blockquote>\n<p>torch.cuda.is_available()</p>\n</blockquote>\n<p>Will always return false.</p>\n<p>Nvidia-smi returns<br>\n<div class=\"lightbox-wrapper\"><a class=\"lightbox\" href=\"https://discuss.pytorch.org/uploads/default/original/3X/1/9/19327070558ddbbdec436b1e8dca3aa624312b79.png\" data-download-href=\"https://discuss.pytorch.org/uploads/default/19327070558ddbbdec436b1e8dca3aa624312b79\" title=\"image\"><img src=\"https://discuss.pytorch.org/uploads/default/original/3X/1/9/19327070558ddbbdec436b1e8dca3aa624312b79.png\" alt=\"image\" data-base62-sha1=\"3AU08L9o5CV4ETwRQZOEFCBGQWt\" width=\"690\" height=\"301\" data-dominant-color=\"1B1B1B\"><div class=\"meta\"><svg class=\"fa d-icon d-icon-far-image svg-icon\" aria-hidden=\"true\"><use href=\"#far-image\"></use></svg><span class=\"filename\">image</span><span class=\"informations\">1296×566 18.8 KB</span><svg class=\"fa d-icon d-icon-discourse-expand svg-icon\" aria-hidden=\"true\"><use href=\"#discourse-expand\"></use></svg></div></a></div></p>\n<p>I have simply ran this command :</p>\n<blockquote>\n<p>conda install pytorch torchvision torchaudio pytorch-cuda=12.4 -c pytorch -c nvidia</p>\n</blockquote>\n<p>I have tried different versions (12.1, 12.4) and also tried to reinstall my NVidia drivers or the miniconda environment.<br>\nI am trying to run this on my laptop which has 2 GPU, one from the NVIDIA card with Cuda and one from the AMD CPU/GPU for low energy consumption.</p>\n<p>When I try to get the environment information it is missing Cuda for Pytorch :</p>\n<blockquote>\n<p>Collecting environment information…<br>\nPyTorch version: 2.3.0<br>\nIs debug build: False<br>\nCUDA used to build PyTorch: Could not collect<br>\nROCM used to build PyTorch: N/A</p>\n<p>OS: Microsoft Windows 11 Pro<br>\nGCC version: Could not collect<br>\nClang version: Could not collect<br>\nCMake version: Could not collect<br>\nLibc version: N/A</p>\n<p>Python version: 3.12.3 | packaged by Anaconda, Inc. | (main, May 6 2024, 19:42:21) [MSC v.1916 64 bit (AMD64)] (64-bit runtime)<br>\nPython platform: Windows-11-10.0.22631-SP0<br>\nIs CUDA available: False<br>\nCUDA runtime version: 12.5.40<br>\nCUDA_MODULE_LOADING set to: N/A<br>\nGPU models and configuration: GPU 0: NVIDIA GeForce RTX 3050 Ti Laptop GPU<br>\nNvidia driver version: 555.85<br>\ncuDNN version: Could not collect<br>\nHIP runtime version: N/A<br>\nMIOpen runtime version: N/A<br>\nIs XNNPACK available: True</p>\n</blockquote>\n<p>I can run software that uses Cuda without issue (cuda-z or a few graphics softwares) so I know the drivers doesn’t have an issue.</p>\n<p>The only time I managed somehow to have the environment setup properly, any Cuda calls would hang forever.</p>\n<p>Is there a problem of version ? What can I do to make this work ?</p>",12 "post_number": 1,13 "post_type": 1,14 "posts_count": 1,15 "updated_at": "2024-06-03T09:42:22.741Z",16 "reply_count": 0,17 "reply_to_post_number": null,18 "quote_count": 0,19 "incoming_link_count": 156,20 "reads": 10,21 "readers_count": 9,22 "score": 782.0,23 "yours": false,24 "topic_id": 203957,25 "topic_slug": "cuda-gpu-never-found",26 "display_username": "",27 "primary_group_name": null,28 "flair_name": null,29 "flair_url": null,30 "flair_bg_color": null,31 "flair_color": null,32 "flair_group_id": null,33 "badges_granted": [],34 "version": 1,35 "can_edit": false,36 "can_delete": false,37 "can_recover": false,38 "can_see_hidden_post": false,39 "can_wiki": false,40 "link_counts": [41 {42 "url": "https://discuss.pytorch.org/uploads/default/original/3X/1/9/19327070558ddbbdec436b1e8dca3aa624312b79.png",43 "internal": true,44 "reflection": false,45 "clicks": 046 }47 ],48 "read": true,49 "user_title": null,50 "bookmarked": false,51 "actions_summary": [],52 "moderator": false,53 "admin": false,54 "staff": false,55 "user_id": 76354,56 "hidden": false,57 "trust_level": 0,58 "deleted_at": null,59 "user_deleted": false,60 "edit_reason": null,61 "can_view_edit_history": true,62 "wiki": false,63 "post_url": "/t/cuda-gpu-never-found/203957/1",64 "can_accept_answer": false,65 "can_unaccept_answer": false,66 "accepted_answer": false,67 "topic_accepted_answer": null,68 "can_vote": false69 }70 ],71 "stream": [72 44472573 ]74 },75 "timeline_lookup": [76 [77 1,78 50979 ]80 ],81 "suggested_topics": [82 {83 "fancy_title": "Understanding the `torch.distributed.elastic.multiprocessing.errors.ChildFailedError` error",84 "id": 212774,85 "title": "Understanding the `torch.distributed.elastic.multiprocessing.errors.ChildFailedError` error",86 "slug": "understanding-the-torch-distributed-elastic-multiprocessing-errors-childfailederror-error",87 "posts_count": 2,88 "reply_count": 0,89 "highest_post_number": 2,90 "image_url": null,91 "created_at": "2024-11-10T22:22:37.482Z",92 "last_posted_at": "2024-11-15T21:50:33.244Z",93 "bumped": true,94 "bumped_at": "2024-11-15T21:50:33.244Z",95 "archetype": "regular",96 "unseen": false,97 "pinned": false,98 "unpinned": null,99 "visible": true,100 "closed": false,101 "archived": false,102 "bookmarked": null,103 "liked": null,104 "tags_descriptions": {},105 "like_count": 0,106 "views": 2314,107 "category_id": 1,108 "featured_link": null,109 "has_accepted_answer": true,110 "posters": [111 {112 "extras": "latest single",113 "description": "Original Poster, Most Recent Poster, Accepted Answer",114 "user": {115 "id": 64540,116 "username": "kzaitse",117 "name": "Kirill Zaitsev",118 "avatar_template": "/user_avatar/discuss.pytorch.org/kzaitse/{size}/58706_2.png",119 "trust_level": 1120 }121 }122 ]123 },124 {125 "fancy_title": "Jumping loss after a few epochs",126 "id": 216303,127 "title": "Jumping loss after a few epochs",128 "slug": "jumping-loss-after-a-few-epochs",129 "posts_count": 6,130 "reply_count": 3,131 "highest_post_number": 6,132 "image_url": "https://discuss.pytorch.org/uploads/default/optimized/3X/9/d/9d5ae47419115b980ddc4b516b3447432231544f_2_1024x289.png",133 "created_at": "2025-02-06T10:17:11.787Z",134 "last_posted_at": "2025-02-06T17:33:10.843Z",135 "bumped": true,136 "bumped_at": "2025-02-06T17:33:10.843Z",137 "archetype": "regular",138 "unseen": false,139 "pinned": false,140 "unpinned": null,141 "visible": true,142 "closed": false,143 "archived": false,144 "bookmarked": null,145 "liked": null,146 "tags_descriptions": {},147 "like_count": 1,148 "views": 65,149 "category_id": 1,150 "featured_link": null,151 "has_accepted_answer": false,152 "posters": [153 {154 "extras": null,155 "description": "Original Poster",156 "user": {157 "id": 82518,158 "username": "FMM",159 "name": "",160 "avatar_template": "/user_avatar/discuss.pytorch.org/fmm/{size}/75502_2.png",161 "trust_level": 1162 }163 },164 {165 "extras": null,166 "description": "Frequent Poster",167 "user": {168 "id": 9081,169 "username": "JuanFMontesinos",170 "name": "Juan Montesinos",171 "avatar_template": "/user_avatar/discuss.pytorch.org/juanfmontesinos/{size}/76115_2.png",172 "trust_level": 2173 }174 },175 {176 "extras": "latest",177 "description": "Most Recent Poster",178 "user": {179 "id": 40671,180 "username": "yassineAlouini",181 "name": "Yassine Alouini",182 "avatar_template": "/user_avatar/discuss.pytorch.org/yassinealouini/{size}/32990_2.png",183 "trust_level": 2184 }185 }186 ]187 },188 {189 "fancy_title": "Loading and saving a checkpoint changes it?",190 "id": 217082,191 "title": "Loading and saving a checkpoint changes it?",192 "slug": "loading-and-saving-a-checkpoint-changes-it",193 "posts_count": 2,194 "reply_count": 0,195 "highest_post_number": 2,196 "image_url": null,197 "created_at": "2025-02-24T07:20:16.283Z",198 "last_posted_at": "2025-03-02T02:12:37.190Z",199 "bumped": true,200 "bumped_at": "2025-03-02T02:12:37.190Z",201 "archetype": "regular",202 "unseen": false,203 "pinned": false,204 "unpinned": null,205 "visible": true,206 "closed": false,207 "archived": false,208 "bookmarked": null,209 "liked": null,210 "tags_descriptions": {},211 "like_count": 0,212 "views": 37,213 "category_id": 1,214 "featured_link": null,215 "has_accepted_answer": true,216 "posters": [217 {218 "extras": "latest single",219 "description": "Original Poster, Most Recent Poster, Accepted Answer",220 "user": {221 "id": 82889,222 "username": "amagibaba",223 "name": "Ryan T. J.",224 "avatar_template": "/user_avatar/discuss.pytorch.org/amagibaba/{size}/73890_2.png",225 "trust_level": 0226 }227 }228 ]229 },230 {231 "fancy_title": "Hardware Asus rog xg 2025- compatiblity pytorch",232 "id": 215259,233 "title": "Hardware Asus rog xg 2025- compatiblity pytorch",234 "slug": "hardware-asus-rog-xg-2025-compatiblity-pytorch",235 "posts_count": 1,236 "reply_count": 0,237 "highest_post_number": 1,238 "image_url": null,239 "created_at": "2025-01-11T14:19:13.780Z",240 "last_posted_at": "2025-01-11T14:19:13.867Z",241 "bumped": true,242 "bumped_at": "2025-01-11T14:19:13.867Z",243 "archetype": "regular",244 "unseen": false,245 "pinned": false,246 "unpinned": null,247 "visible": true,248 "closed": false,249 "archived": false,250 "bookmarked": null,251 "liked": null,252 "tags_descriptions": {},253 "like_count": 0,254 "views": 107,255 "category_id": 1,256 "featured_link": null,257 "has_accepted_answer": false,258 "posters": [259 {260 "extras": "latest single",261 "description": "Original Poster, Most Recent Poster",262 "user": {263 "id": 61938,264 "username": "bruno_menon",265 "name": "bruno menon",266 "avatar_template": "/user_avatar/discuss.pytorch.org/bruno_menon/{size}/55846_2.png",267 "trust_level": 1268 }269 }270 ]271 },272 {273 "fancy_title": "Model inference consumes memory, every time model(input) is called",274 "id": 218528,275 "title": "Model inference consumes memory, every time model(input) is called",276 "slug": "model-inference-consumes-memory-every-time-model-input-is-called",277 "posts_count": 1,278 "reply_count": 0,279 "highest_post_number": 1,280 "image_url": null,281 "created_at": "2025-04-02T09:47:44.477Z",282 "last_posted_at": "2025-04-02T09:47:44.531Z",283 "bumped": true,284 "bumped_at": "2025-04-02T09:47:44.531Z",285 "archetype": "regular",286 "unseen": false,287 "pinned": false,288 "unpinned": null,289 "visible": true,290 "closed": false,291 "archived": false,292 "bookmarked": null,293 "liked": null,294 "tags_descriptions": {},295 "like_count": 0,296 "views": 13,297 "category_id": 1,298 "featured_link": null,299 "has_accepted_answer": false,300 "posters": [301 {302 "extras": "latest single",303 "description": "Original Poster, Most Recent Poster",304 "user": {305 "id": 83593,306 "username": "shakeel604",307 "name": "Shakeel",308 "avatar_template": "/letter_avatar_proxy/v4/letter/s/f17d59/{size}.png",309 "trust_level": 1310 }311 }312 ]313 }314 ],315 "tags_descriptions": {},316 "fancy_title": "Cuda GPU never found",317 "id": 203957,318 "title": "Cuda GPU never found",319 "posts_count": 1,320 "created_at": "2024-06-03T09:39:30.377Z",321 "views": 428,322 "reply_count": 0,323 "like_count": 0,324 "last_posted_at": "2024-06-03T09:39:30.554Z",325 "visible": true,326 "closed": false,327 "archived": false,328 "has_summary": false,329 "archetype": "regular",330 "slug": "cuda-gpu-never-found",331 "category_id": 1,332 "word_count": 325,333 "deleted_at": null,334 "user_id": 76354,335 "featured_link": null,336 "pinned_globally": false,337 "pinned_at": null,338 "pinned_until": null,339 "image_url": "https://discuss.pytorch.org/uploads/default/optimized/3X/1/9/19327070558ddbbdec436b1e8dca3aa624312b79_2_1024x447.png",340 "slow_mode_seconds": 0,341 "draft": null,342 "draft_key": "topic_203957",343 "draft_sequence": null,344 "unpinned": null,345 "pinned": false,346 "current_post_number": 1,347 "highest_post_number": 1,348 "deleted_by": null,349 "actions_summary": [350 {351 "id": 4,352 "count": 0,353 "hidden": false,354 "can_act": false355 },356 {357 "id": 8,358 "count": 0,359 "hidden": false,360 "can_act": false361 },362 {363 "id": 10,364 "count": 0,365 "hidden": false,366 "can_act": false367 },368 {369 "id": 7,370 "count": 0,371 "hidden": false,372 "can_act": false373 }374 ],375 "chunk_size": 20,376 "bookmarked": false,377 "topic_timer": null,378 "message_bus_last_id": 0,379 "participant_count": 1,380 "show_read_indicator": false,381 "thumbnails": [382 {383 "max_width": null,384 "max_height": null,385 "width": 1296,386 "height": 566,387 "url": "https://discuss.pytorch.org/uploads/default/original/3X/1/9/19327070558ddbbdec436b1e8dca3aa624312b79.png"388 },389 {390 "max_width": 1024,391 "max_height": 1024,392 "width": 1024,393 "height": 447,394 "url": "https://discuss.pytorch.org/uploads/default/optimized/3X/1/9/19327070558ddbbdec436b1e8dca3aa624312b79_2_1024x447.png"395 }396 ],397 "slow_mode_enabled_until": null,398 "can_vote": false,399 "vote_count": 0,400 "user_voted": false,401 "discourse_zendesk_plugin_zendesk_id": null,402 "discourse_zendesk_plugin_zendesk_url": "https://your-url.zendesk.com/agent/tickets/",403 "details": {404 "can_edit": false,405 "notification_level": 1,406 "participants": [407 {408 "id": 76354,409 "username": "rXp",410 "name": "",411 "avatar_template": "/letter_avatar_proxy/v4/letter/r/3ab097/{size}.png",412 "post_count": 1,413 "primary_group_name": null,414 "flair_name": null,415 "flair_url": null,416 "flair_color": null,417 "flair_bg_color": null,418 "flair_group_id": null,419 "trust_level": 0420 }421 ],422 "created_by": {423 "id": 76354,424 "username": "rXp",425 "name": "",426 "avatar_template": "/letter_avatar_proxy/v4/letter/r/3ab097/{size}.png"427 },428 "last_poster": {429 "id": 76354,430 "username": "rXp",431 "name": "",432 "avatar_template": "/letter_avatar_proxy/v4/letter/r/3ab097/{size}.png"433 }434 },435 "bookmarks": []436 },437 {438 "post_stream": {439 "posts": [440 {441 "id": 444382,442 "name": "71~073~",443 "username": "71M073J",444 "avatar_template": "/user_avatar/discuss.pytorch.org/71m073j/{size}/66398_2.png",445 "created_at": "2024-05-30T11:31:32.864Z",446 "cooked": "<p>Hi, when building a pytorch extension for a project, the building succeeds, but when loading it, the import fails as shown below.</p>\n<pre data-code-wrap=\"import\"><code class=\"lang-import\">import torch.nn as nn\nimport torch.functional as F\nfrom torch.utils.cpp_extension import load\n\nperf = load(name=\"test1\", sources=[\"ConvolutionMM2d_.cpp\"], verbose=True)\n/.../\n</code></pre>\n<p>I get the error:</p>\n<pre><code class=\"lang-auto\">ImportError: /home/username/.cache/torch_extensions/py310_cu121/test1/test1.so: undefined symbol: _ZN2at6native7cpublas4gemmENS0_13TransposeTypeES2_lllfPKN3c104HalfElS6_lfPS4_l\n</code></pre>\n<p>here is the cpp file of <a href=\"https://github.com/pytorch/pytorch/blob/v2.3.0/aten/src/ATen/native/ConvolutionMM2d.cpp\" rel=\"noopener nofollow ugc\">ConvolutionMM2d_.cpp</a>, from v2.3.0 tag of pytorch source, edited such that it can be registered as an extension. (my project involves small changes in convolution algorithm - so this would be a great starting point)</p>\n<pre><code class=\"lang-auto\">#include <torch/extension.h>\n\n#include <vector>\n#include <iostream>\n\n#define TORCH_ASSERT_ONLY_METHOD_OPERATORS\n\n#include <ATen/core/Tensor.h>\n#include <ATen/Dispatch.h>\n#include <ATen/Parallel.h>\n#include <ATen/TensorUtils.h>\n#include <ATen/div_rtn.h>\n#include <ATen/native/ConvUtils.h>\n#include <ATen/native/CPUBlas.h>\n#include <ATen/native/Unfold2d.h>\n#include <c10/util/irange.h>\n\n#ifndef AT_PER_OPERATOR_HEADERS\n#include <ATen/Functions.h>\n#include <ATen/NativeFunctions.h>\n#else\n#include <ATen/ops/_slow_conv2d_backward_native.h>\n#include <ATen/ops/_slow_conv2d_forward.h>\n#include <ATen/ops/_slow_conv2d_forward_native.h>\n#include <ATen/ops/empty.h>\n#include <ATen/ops/sum.h>\n#include <ATen/ops/thnn_conv2d_native.h>\n#endif\n\nnamespace at::native {\n\nnamespace {\n\nstatic Tensor compute_columns2d(\n const Tensor& input,\n IntArrayRef padding,\n IntArrayRef stride,\n IntArrayRef kernel_size,\n bool is_channels_last) {\n const int64_t kernel_height = kernel_size[0];\n const int64_t kernel_width = kernel_size[1];\n const int64_t pad_height = padding[0];\n const int64_t pad_width = padding[1];\n const int64_t stride_height = stride[0];\n const int64_t stride_width = stride[1];\n const int64_t batch_size = input.size(0);\n const int64_t n_input_plane = input.size(1);\n const int64_t input_height = input.size(2);\n const int64_t input_width = input.size(3);\n const int64_t output_height = (input_height + 2 * pad_height - kernel_height) / stride_height + 1;\n const int64_t output_width = (input_width + 2 * pad_width - kernel_width) / stride_width + 1;\n\n Tensor columns;\n if ((kernel_height == 1) && (stride_height == 1) && (pad_height == 0) &&\n (kernel_width == 1) && (stride_width == 1) && (pad_width == 0)) {\n // Columns are just a view on the input for the 1x1 kernel special case.\n if (is_channels_last) {\n columns = input.as_strided({batch_size, output_height * output_width, n_input_plane},\n {output_height * output_width * n_input_plane, n_input_plane, 1}).detach();\n } else {\n columns = input.view({batch_size, n_input_plane, output_height * output_width}).detach();\n }\n } else {\n int64_t row = is_channels_last ?\n output_height * output_width : n_input_plane * kernel_height * kernel_width;\n int64_t col = is_channels_last ?\n kernel_height * kernel_width * n_input_plane : output_height * output_width;\n columns = at::empty({batch_size, row, col}, input.options());\n AT_DISPATCH_ALL_TYPES_AND2(kBFloat16, kHalf, input.scalar_type(), \"slow_conv2d_cpu\", [&]{\n auto input_a = input.accessor<scalar_t, 4>();\n auto columns_a = columns.accessor<scalar_t, 3>();\n\n at::parallel_for(0, batch_size, 0, [&](int64_t start, int64_t end) {\n for (const auto t : c10::irange(start, end)) {\n auto input_t = input_a[t];\n auto columns_t = columns_a[t];\n unfolded2d_copy_stub(\n kCPU,\n c10::CppTypeToScalarType<scalar_t>::value,\n columns_t.data(),\n input_t.data(),\n kernel_height,\n kernel_width,\n stride_height,\n stride_width,\n pad_height,\n pad_width,\n n_input_plane,\n input_height,\n input_width,\n output_height,\n output_width,\n is_channels_last);\n }\n });\n });\n }\n\n return columns.contiguous();\n}\n\nstatic inline void slow_conv2d_shape_check(\n const Tensor& input,\n const Tensor& grad_output,\n const Tensor& weight,\n const Tensor& bias,\n int64_t kernel_height,\n int64_t kernel_width,\n int64_t stride_height,\n int64_t stride_width,\n int64_t pad_height,\n int64_t pad_width,\n bool weight_optional) {\n TORCH_CHECK(\n kernel_width > 0 && kernel_height > 0,\n \"kernel size should be greater than zero, but got kernel_height: \",\n kernel_height,\n \" kernel_width: \",\n kernel_width);\n TORCH_CHECK(\n stride_width > 0 && stride_height > 0,\n \"stride should be greater than zero, but got stride_height: \",\n stride_height,\n \" stride_width: \",\n stride_width);\n\n if (weight.defined()) {\n TORCH_CHECK(\n weight.numel() > 0 && (weight.dim() == 2 || weight.dim() == 4),\n \"non-empty 2D or 4D weight tensor expected, but got: \",\n weight.sizes());\n if (bias.defined()) {\n check_dim_size(bias, 1, 0, weight.size(0));\n }\n } else {\n TORCH_CHECK(weight_optional, \"weight tensor is undefined\");\n }\n\n const int64_t ndim = input.dim();\n const int64_t dim_planes = 1;\n const int64_t dim_height = 2;\n const int64_t dim_width = 3;\n\n // Allow for empty batch size and channel size but not other dimensions\n TORCH_CHECK(ndim == 4, \"Expected 4D input tensor, but got: \", input.sizes());\n for (const auto dim : c10::irange(2, ndim)) {\n TORCH_CHECK(input.size(dim) != 0,\n \"Expected non-zero size for input dimension \", dim,\n \", but got input shape: \", input.sizes(), \". Only the batch and channel dimensions support size 0.\");\n }\n\n const int64_t input_height = input.size(dim_height);\n const int64_t input_width = input.size(dim_width);\n\n const int64_t exact_input_height = input_height + 2 * pad_height;\n const int64_t exact_input_width = input_width + 2 * pad_width;\n\n TORCH_CHECK(\n exact_input_height >= kernel_height && exact_input_width >= kernel_width,\n \"Calculated padded input size per channel: (\",\n exact_input_height,\n \" x \",\n exact_input_width,\n \"). \",\n \"Kernel size: (\",\n kernel_height,\n \" x \",\n kernel_width,\n \"). Kernel size can't be greater than actual input size\");\n\n const int64_t output_height =\n div_rtn<int64_t>(exact_input_height - kernel_height, stride_height) + 1;\n const int64_t output_width =\n div_rtn<int64_t>(exact_input_width - kernel_width, stride_width) + 1;\n\n TORCH_CHECK(\n output_width >= 1 && output_height >= 1,\n \"Given input size per channel: (\",\n input_height,\n \" x \",\n input_width,\n \"). \"\n \"Calculated output size per channel: (\",\n output_height,\n \" x \",\n output_width,\n \"). Output size is too small\");\n\n if (weight.defined()) {\n int64_t n_input_plane = weight.size(1);\n if (weight.dim() == 2) {\n n_input_plane /= (kernel_height * kernel_width);\n }\n if (input.size(1) != 0) {\n check_dim_size(input, ndim, dim_planes, n_input_plane);\n }\n }\n\n if (grad_output.defined()) {\n if (weight.defined()) {\n int64_t n_output_plane = weight.size(0);\n check_dim_size(grad_output, ndim, dim_planes, n_output_plane);\n } else if (bias.defined()) {\n TORCH_CHECK(bias.numel() > 0, \"non-empty bias tensor expected\");\n const int64_t n_output_plane = bias.dim() == 0 ? 1 : bias.size(0);\n check_dim_size(grad_output, ndim, dim_planes, n_output_plane);\n }\n check_dim_size(grad_output, ndim, dim_height, output_height);\n check_dim_size(grad_output, ndim, dim_width, output_width);\n }\n}\n\nstatic inline Tensor view_weight_2d(const Tensor& weight_,\n at::MemoryFormat memory_format = at::MemoryFormat::Contiguous) {\n Tensor weight = weight_.contiguous(memory_format);\n if (weight.dim() == 4) {\n const int64_t s1 = weight.size(0);\n const int64_t s2 = weight.size(1) * weight.size(2) * weight.size(3);\n return memory_format == at::MemoryFormat::ChannelsLast\n ? weight.as_strided({s1, s2}, {s2, 1}) // CL: view as {oc, kh*kw*ic}\n : weight.view({s1, s2}); // CF: view as {oc, ic*kh*kw}\n } else {\n return weight;\n }\n}\n\ntemplate <typename scalar_t>\nstatic void slow_conv2d_update_output_frame(\n TensorAccessor<scalar_t, 3> input,\n TensorAccessor<scalar_t, 3> output,\n TensorAccessor<scalar_t, 2> weight,\n bool has_bias,\n TensorAccessor<scalar_t, 2> finput,\n int64_t kernel_height,\n int64_t kernel_width,\n int64_t stride_height,\n int64_t stride_width,\n int64_t pad_height,\n int64_t pad_width,\n int64_t n_input_plane,\n int64_t input_height,\n int64_t input_width,\n int64_t n_output_plane,\n int64_t output_height,\n int64_t output_width,\n bool is_channels_last) {\n const int beta = has_bias ? 1 : 0;\n\n // Compute out = weight * input\n // Note gemm expects fortran order, so all 3 matrices are transposed.\n // Swapping argument order cancels this, since C == AB <=> T(C) == T(B)T(A)\n if (is_channels_last) {\n const int64_t m = n_output_plane;\n const int64_t n = output_height * output_width;\n const int64_t k = n_input_plane * kernel_height * kernel_width;\n\n const int64_t lda = k;\n const int64_t ldb = k;\n const int64_t ldc = m;\n\n at::native::cpublas::gemm(\n TransposeType::Transpose,\n TransposeType::NoTranspose,\n m, n, k,\n static_cast<scalar_t>(1),\n weight.data(), lda,\n finput.data(), ldb,\n static_cast<scalar_t>(beta),\n output.data(), ldc);\n } else {\n const int64_t m = output_height * output_width;\n const int64_t n = n_output_plane;\n const int64_t k = n_input_plane * kernel_height * kernel_width;\n\n const int64_t lda = m;\n const int64_t ldb = k;\n const int64_t ldc = m;\n\n at::native::cpublas::gemm(\n TransposeType::NoTranspose,\n TransposeType::NoTranspose,\n m, n, k,\n static_cast<scalar_t>(1),\n finput.data(), lda,\n weight.data(), ldb,\n static_cast<scalar_t>(beta),\n output.data(), ldc);\n }\n}\n\ntemplate <typename scalar_t>\nvoid slow_conv2d_backward_update_grad_input_frame(\n TensorAccessor<scalar_t, 3> grad_input,\n TensorAccessor<scalar_t, 3> grad_output,\n TensorAccessor<scalar_t, 2> weight,\n scalar_t *fgrad_input,\n int64_t kernel_height,\n int64_t kernel_width,\n int64_t stride_height,\n int64_t stride_width,\n int64_t pad_height,\n int64_t pad_width,\n bool is_channels_last) {\n // Compute fgrad_input = weight.T * grad_output.reshape({grad_output.shape(0), -1})\n // Note gemm expects fortran order, so all 3 matrices are transposed.\n // Swapping argument order cancels this, since C == AB <=> T(C) == T(B)T(A)\n if (is_channels_last) {\n const int64_t m = weight.size(1);\n const int64_t n = grad_output.size(1) * grad_output.size(2);\n const int64_t k = weight.size(0);\n\n const int64_t lda = m;\n const int64_t ldb = k;\n const int64_t ldc = m;\n\n at::native::cpublas::gemm(\n TransposeType::NoTranspose,\n TransposeType::NoTranspose,\n m, n, k,\n static_cast<scalar_t>(1),\n weight.data(), lda,\n grad_output.data(), ldb,\n static_cast<scalar_t>(0),\n fgrad_input, ldc);\n } else {\n const int64_t m = grad_output.size(1) * grad_output.size(2);\n const int64_t n = weight.size(1);\n const int64_t k = weight.size(0);\n\n const int64_t lda = m;\n const int64_t ldb = n;\n const int64_t ldc = m;\n\n at::native::cpublas::gemm(\n TransposeType::NoTranspose,\n TransposeType::Transpose,\n m, n, k,\n static_cast<scalar_t>(1),\n grad_output.data(), lda,\n weight.data(), ldb,\n static_cast<scalar_t>(0),\n fgrad_input, ldc);\n }\n\n unfolded2d_acc_stub(\n kCPU,\n c10::CppTypeToScalarType<scalar_t>::value,\n fgrad_input,\n grad_input.data(),\n kernel_height,\n kernel_width,\n stride_height,\n stride_width,\n pad_height,\n pad_width,\n grad_input.size(0),\n grad_input.size(1),\n grad_input.size(2),\n grad_output.size(1),\n grad_output.size(2),\n is_channels_last);\n}\n\nvoid slow_conv2d_backward_out_cpu_template(\n Tensor& grad_input,\n const Tensor& grad_output_,\n const Tensor& input_,\n const Tensor& weight_,\n IntArrayRef kernel_size,\n IntArrayRef stride,\n IntArrayRef padding) {\n const int64_t kernel_height = kernel_size[0];\n const int64_t kernel_width = kernel_size[1];\n const int64_t pad_height = padding[0];\n const int64_t pad_width = padding[1];\n const int64_t stride_height = stride[0];\n const int64_t stride_width = stride[1];\n\n bool use_channels_last = thnn_conv_use_channels_last(input_, weight_);\n auto memory_format = use_channels_last ? at::MemoryFormat::ChannelsLast : at::MemoryFormat::Contiguous;\n\n const Tensor weight = view_weight_2d(weight_, memory_format);\n slow_conv2d_shape_check(\n input_,\n grad_output_,\n weight,\n Tensor(),\n kernel_height,\n kernel_width,\n stride_height,\n stride_width,\n pad_height,\n pad_width,\n false);\n\n const Tensor input = input_.contiguous(memory_format);\n\n // Compute shape of columnized data excluding batch dim.\n const int64_t batch_size = input.size(0);\n const int64_t n_input_plane = input.size(1);\n const int64_t input_height = input.size(2);\n const int64_t input_width = input.size(3);\n const int64_t output_height = (input_height + 2 * pad_height - kernel_height) / stride_height + 1;\n const int64_t output_width = (input_width + 2 * pad_width - kernel_width) / stride_width + 1;\n const int64_t fgrad_input_size = n_input_plane * kernel_height * kernel_width * output_height * output_width;\n\n const Tensor grad_output = grad_output_.contiguous(memory_format);\n grad_input.resize_as_(input, memory_format);\n grad_input.zero_();\n TORCH_CHECK(grad_input.is_contiguous(memory_format), \"slow_conv2d: grad_input must be contiguous\");\n\n AT_DISPATCH_FLOATING_TYPES_AND2(\n kBFloat16, kHalf, input.scalar_type(), \"slow_conv2d_cpu_grad_input\", [&] {\n auto grad_output_a = grad_output.accessor<scalar_t, 4>();\n auto grad_input_a = grad_input.accessor<scalar_t, 4>();\n auto weight_a = weight.accessor<scalar_t, 2>();\n\n at::parallel_for(0, batch_size, 0, [&](int64_t start, int64_t end) {\n auto fgrad_input = std::make_unique<scalar_t[]>(fgrad_input_size);\n for (const auto t : c10::irange(start, end)) {\n auto grad_input_t = grad_input_a[t];\n auto grad_output_t = grad_output_a[t];\n slow_conv2d_backward_update_grad_input_frame(\n grad_input_t,\n grad_output_t,\n weight_a,\n fgrad_input.get(),\n kernel_height,\n kernel_width,\n stride_height,\n stride_width,\n pad_height,\n pad_width,\n use_channels_last);\n }\n });\n });\n}\n\ntemplate <typename scalar_t>\nvoid slow_conv2d_backward_weight_frame(\n TensorAccessor<scalar_t, 2> grad_weight,\n TensorAccessor<scalar_t, 3> grad_output,\n TensorAccessor<scalar_t, 2> finput,\n bool is_channels_last) {\n // Compute grad_weight += grad_output.reshape({grad_output.shape(0), -1}) * finput.T\n // Note gemm expects fortran order, so all 3 matrices are transposed.\n // Swapping argument order cancels this, since C == AB <=> T(C) == T(B)T(A)\n if (is_channels_last) {\n const int64_t m = finput.size(1);\n const int64_t n = grad_output.size(0);\n const int64_t k = grad_output.size(1) * grad_output.size(2);\n\n const int64_t lda = m;\n const int64_t ldb = n;\n const int64_t ldc = m;\n\n at::native::cpublas::gemm(\n TransposeType::NoTranspose,\n TransposeType::Transpose,\n m, n, k,\n static_cast<scalar_t>(1),\n finput.data(), lda,\n grad_output.data(), ldb,\n static_cast<scalar_t>(1),\n grad_weight.data(), ldc);\n } else {\n const int64_t m = finput.size(0);\n const int64_t n = grad_output.size(0);\n const int64_t k = grad_output.size(1) * grad_output.size(2);\n\n const int64_t lda = k;\n const int64_t ldb = k;\n const int64_t ldc = m;\n\n at::native::cpublas::gemm(\n TransposeType::Transpose,\n TransposeType::NoTranspose,\n m, n, k,\n static_cast<scalar_t>(1),\n finput.data(), lda,\n grad_output.data(), ldb,\n static_cast<scalar_t>(1),\n grad_weight.data(), ldc);\n }\n}\n\nstatic void slow_conv2d_backward_weight_out_cpu_template(\n Tensor& grad_weight,\n const Tensor& input,\n const Tensor& grad_output_,\n IntArrayRef kernel_size,\n IntArrayRef stride,\n IntArrayRef padding) {\n const int64_t kernel_height = kernel_size[0];\n const int64_t kernel_width = kernel_size[1];\n const int64_t pad_height = padding[0];\n const int64_t pad_width = padding[1];\n const int64_t stride_height = stride[0];\n const int64_t stride_width = stride[1];\n\n bool use_channels_last = thnn_conv_use_channels_last(input, grad_weight);\n auto memory_format = use_channels_last ? at::MemoryFormat::ChannelsLast : at::MemoryFormat::Contiguous;\n\n TORCH_CHECK(grad_weight.is_contiguous(memory_format), \"slow_conv2d: grad_weight must be contiguous\");\n Tensor grad_weight_2d = view_weight_2d(grad_weight, memory_format);\n\n slow_conv2d_shape_check(\n input,\n grad_output_,\n grad_weight_2d,\n {},\n kernel_height,\n kernel_width,\n stride_height,\n stride_width,\n pad_height,\n pad_width,\n true);\n\n auto grad_output = grad_output_.contiguous(memory_format);\n Tensor finput = compute_columns2d(input, padding, stride, kernel_size, use_channels_last);\n\n const int64_t batch_size = input.size(0);\n\n AT_DISPATCH_FLOATING_TYPES_AND2(\n kBFloat16, kHalf, input.scalar_type(), \"slow_conv2d_cpu_grad_weight\", [&] {\n auto grad_output_a = grad_output.accessor<scalar_t, 4>();\n auto grad_weight_2d_a = grad_weight_2d.accessor<scalar_t, 2>();\n auto finput_a = finput.accessor<scalar_t, 3>();\n\n for (const auto t : c10::irange(batch_size)) {\n auto grad_output_t = grad_output_a[t];\n auto finput_t = finput_a[t];\n\n slow_conv2d_backward_weight_frame(\n grad_weight_2d_a, grad_output_t, finput_t, use_channels_last);\n }\n });\n}\n\n} // namespace\n\nTensor& slow_conv2d_forward_out_cpu(\n const Tensor& self,\n const Tensor& weight_,\n IntArrayRef kernel_size, const c10::optional<Tensor>& bias_opt,\n IntArrayRef stride,\n IntArrayRef padding,\n Tensor& output) {\n // See [Note: hacky wrapper removal for optional tensor]\n c10::MaybeOwned<Tensor> bias_maybe_owned = at::borrow_from_optional_tensor(bias_opt);\n const Tensor& bias = *bias_maybe_owned;\n\n const int64_t kernel_height = kernel_size[0];\n const int64_t kernel_width = kernel_size[1];\n const int64_t pad_height = padding[0];\n const int64_t pad_width = padding[1];\n const int64_t stride_height = stride[0];\n const int64_t stride_width = stride[1];\n\n bool use_channels_last = thnn_conv_use_channels_last(self, weight_);\n auto memory_format = use_channels_last ? at::MemoryFormat::ChannelsLast : at::MemoryFormat::Contiguous;\n\n const Tensor weight_2d = view_weight_2d(weight_, memory_format);\n\n slow_conv2d_shape_check(\n self,\n Tensor(),\n weight_2d,\n bias,\n kernel_height,\n kernel_width,\n stride_height,\n stride_width,\n pad_height,\n pad_width,\n false);\n\n const Tensor input = self.contiguous(memory_format);\n const int64_t batch_size = input.size(0);\n const int64_t n_input_plane = input.size(1);\n const int64_t input_height = input.size(2);\n const int64_t input_width = input.size(3);\n const int64_t n_output_plane = weight_2d.size(0);\n const int64_t output_height = (input_height + 2 * pad_height - kernel_height) / stride_height + 1;\n const int64_t output_width = (input_width + 2 * pad_width - kernel_width) / stride_width + 1;\n\n Tensor finput = compute_columns2d(input, padding, stride, kernel_size, use_channels_last);\n output.resize_({batch_size, n_output_plane, output_height, output_width}, memory_format);\n if (bias.defined()) {\n output.copy_(bias.reshape({-1, 1, 1}));\n }\n TORCH_CHECK(output.is_contiguous(memory_format), \"slow_conv2d output tensor must be contiguous\");\n\n AT_DISPATCH_ALL_TYPES_AND2(kBFloat16, kHalf, input.scalar_type(), \"slow_conv2d_cpu\", [&]{\n auto input_a = input.accessor<scalar_t, 4>();\n auto output_a = output.accessor<scalar_t, 4>();\n auto finput_a = finput.accessor<scalar_t, 3>();\n auto weight_2d_a = weight_2d.accessor<scalar_t, 2>();\n\n at::parallel_for(0, batch_size, 0, [&](int64_t start, int64_t end) {\n for (const auto t : c10::irange(start, end)) {\n auto input_t = input_a[t];\n auto output_t = output_a[t];\n auto finput_t = finput_a[t];\n slow_conv2d_update_output_frame(\n input_t,\n output_t,\n weight_2d_a,\n bias.defined(),\n finput_t,\n kernel_height,\n kernel_width,\n stride_height,\n stride_width,\n pad_height,\n pad_width,\n n_input_plane,\n input_height,\n input_width,\n n_output_plane,\n output_height,\n output_width,\n use_channels_last);\n }\n });\n });\n\n return output;\n}\n\nTensor slow_conv2d_forward_cpu_(\n const Tensor& self,\n const Tensor& weight,\n IntArrayRef kernel_size, const c10::optional<Tensor>& bias_opt,\n IntArrayRef stride,\n IntArrayRef padding) {\n // See [Note: hacky wrapper removal for optional tensor]\n c10::MaybeOwned<Tensor> bias_maybe_owned = at::borrow_from_optional_tensor(bias_opt);\n const Tensor& bias = *bias_maybe_owned;\n\n auto output = at::empty({0}, self.options());\n at::native::slow_conv2d_forward_out_cpu(\n self,\n weight,\n kernel_size,\n bias,\n stride,\n padding,\n output);\n\n return output;\n}\n\nstd::tuple<Tensor&, Tensor&, Tensor&> slow_conv2d_backward_out_cpu(\n const Tensor& grad_output,\n const Tensor& self,\n const Tensor& weight,\n IntArrayRef kernel_size,\n IntArrayRef stride,\n IntArrayRef padding,\n Tensor& grad_input,\n Tensor& grad_weight,\n Tensor& grad_bias) {\n if (grad_input.defined()) {\n slow_conv2d_backward_out_cpu_template(\n grad_input,\n grad_output,\n self,\n weight,\n kernel_size,\n stride,\n padding);\n }\n\n if (grad_bias.defined()) {\n at::sum_out(grad_bias, grad_output, IntArrayRef{0, 2, 3});\n }\n\n if (grad_weight.defined()) {\n grad_weight.resize_(weight.sizes(), weight.suggest_memory_format());\n grad_weight.zero_();\n slow_conv2d_backward_weight_out_cpu_template(\n grad_weight,\n self,\n grad_output,\n kernel_size,\n stride,\n padding);\n }\n\n return std::tuple<Tensor&, Tensor&, Tensor&>(\n grad_input, grad_weight, grad_bias);\n}\n\nstd::tuple<Tensor, Tensor, Tensor> slow_conv2d_backward_cpu_(\n const Tensor& grad_output,\n const Tensor& self,\n const Tensor& weight,\n IntArrayRef kernel_size,\n IntArrayRef stride,\n IntArrayRef padding,\n std::array<bool, 3> output_mask) {\n Tensor grad_input;\n Tensor grad_weight;\n Tensor grad_bias;\n\n if (output_mask[0]) {\n grad_input = at::empty({0}, grad_output.options());\n }\n\n if (output_mask[1]) {\n grad_weight = at::empty({0}, grad_output.options());\n }\n\n if (output_mask[2]) {\n grad_bias = at::empty({0}, grad_output.options());\n }\n\n at::native::slow_conv2d_backward_out_cpu(\n grad_output,\n self,\n weight,\n kernel_size,\n stride,\n padding,\n grad_input,\n grad_weight,\n grad_bias);\n\n return std::make_tuple(grad_input, grad_weight, grad_bias);\n}\n\nTensor & thnn_conv2d_out(const Tensor & self, const Tensor & weight, IntArrayRef kernel_size, const c10::optional<Tensor>& bias_opt, IntArrayRef stride, IntArrayRef padding, Tensor & output) {\n // See [Note: hacky wrapper removal for optional tensor]\n c10::MaybeOwned<Tensor> bias_maybe_owned = at::borrow_from_optional_tensor(bias_opt);\n const Tensor& bias = *bias_maybe_owned;\n\n return at::_slow_conv2d_forward_out(output, self, weight, kernel_size, bias, stride, padding);\n}\n\nTensor thnn_conv2d(const Tensor & self, const Tensor & weight, IntArrayRef kernel_size, const c10::optional<Tensor>& bias_opt, IntArrayRef stride, IntArrayRef padding) {\n // See [Note: hacky wrapper removal for optional tensor]\n c10::MaybeOwned<Tensor> bias_maybe_owned = at::borrow_from_optional_tensor(bias_opt);\n const Tensor& bias = *bias_maybe_owned;\n\n return at::_slow_conv2d_forward(self, weight, kernel_size, bias, stride, padding);\n}\nPYBIND11_MODULE(TORCH_EXTENSION_NAME, m) {\n m.def(\"forward\", &slow_conv2d_forward_cpu_, \"conv2dPerf forward\");\n m.def(\"backward\", &slow_conv2d_backward_cpu_, \"conv2dPerf backward\");\n}\n} // namespace at::native\n</code></pre>\n<p>Any ideas how to solve this? Thanks in advance.</p>",447 "post_number": 1,448 "post_type": 1,449 "posts_count": 2,450 "updated_at": "2024-05-30T11:32:54.650Z",451 "reply_count": 0,452 "reply_to_post_number": null,453 "quote_count": 0,454 "incoming_link_count": 25,455 "reads": 3,456 "readers_count": 2,457 "score": 120.6,458 "yours": false,459 "topic_id": 203763,460 "topic_slug": "undefined-symbol-cpublas4gemm-when-loading-extension",461 "display_username": "71~073~",462 "primary_group_name": null,463 "flair_name": null,464 "flair_url": null,465 "flair_bg_color": null,466 "flair_color": null,467 "flair_group_id": null,468 "badges_granted": [],469 "version": 1,470 "can_edit": false,471 "can_delete": false,472 "can_recover": false,473 "can_see_hidden_post": false,474 "can_wiki": false,475 "link_counts": [476 {477 "url": "https://github.com/pytorch/pytorch/blob/v2.3.0/aten/src/ATen/native/ConvolutionMM2d.cpp",478 "internal": false,479 "reflection": false,480 "title": "pytorch/aten/src/ATen/native/ConvolutionMM2d.cpp at v2.3.0 · pytorch/pytorch · GitHub",481 "clicks": 1482 },483 {484 "url": "https://discuss.pytorch.org/t/extension-cannot-find-libc10-so-libtorch-cpu-so-libtorch-python-so-even-after-importing-torch/203929",485 "internal": true,486 "reflection": true,487 "title": "Extension cannot find libc10.so/libtorch_cpu.so/libtorch_python.so even after importing torch",488 "clicks": 0489 }490 ],491 "read": true,492 "user_title": null,493 "bookmarked": false,494 "actions_summary": [],495 "moderator": false,496 "admin": false,497 "staff": false,498 "user_id": 71900,499 "hidden": false,500 "trust_level": 1,501 "deleted_at": null,502 "user_deleted": false,503 "edit_reason": null,504 "can_view_edit_history": true,505 "wiki": false,506 "post_url": "/t/undefined-symbol-cpublas4gemm-when-loading-extension/203763/1",507 "can_accept_answer": false,508 "can_unaccept_answer": false,509 "accepted_answer": false,510 "topic_accepted_answer": null,511 "can_vote": false512 },513 {514 "id": 444722,515 "name": "71~073~",516 "username": "71M073J",517 "avatar_template": "/user_avatar/discuss.pytorch.org/71m073j/{size}/66398_2.png",518 "created_at": "2024-06-03T09:12:23.549Z",519 "cooked": "<p>This seems to be the same issue as mentioned <a href=\"https://discuss.pytorch.org/t/how-can-i-use-the-function-at-gemm-float/95341\">here</a>, with advice to not use such low-level functions at all. There isn’t, however, any workaround for the cases when usage of those is necessary!</p>",520 "post_number": 2,521 "post_type": 1,522 "posts_count": 2,523 "updated_at": "2024-06-03T09:12:23.549Z",524 "reply_count": 0,525 "reply_to_post_number": null,526 "quote_count": 0,527 "incoming_link_count": 1,528 "reads": 3,529 "readers_count": 2,530 "score": 5.6,531 "yours": false,532 "topic_id": 203763,533 "topic_slug": "undefined-symbol-cpublas4gemm-when-loading-extension",534 "display_username": "71~073~",535 "primary_group_name": null,536 "flair_name": null,537 "flair_url": null,538 "flair_bg_color": null,539 "flair_color": null,540 "flair_group_id": null,541 "badges_granted": [],542 "version": 1,543 "can_edit": false,544 "can_delete": false,545 "can_recover": false,546 "can_see_hidden_post": false,547 "can_wiki": false,548 "link_counts": [549 {550 "url": "https://discuss.pytorch.org/t/how-can-i-use-the-function-at-gemm-float/95341",551 "internal": true,552 "reflection": false,553 "title": "How can I use the function at::cuda::blas::gemm<float>()?",554 "clicks": 17555 }556 ],557 "read": true,558 "user_title": null,559 "bookmarked": false,560 "actions_summary": [],561 "moderator": false,562 "admin": false,563 "staff": false,564 "user_id": 71900,565 "hidden": false,566 "trust_level": 1,567 "deleted_at": null,568 "user_deleted": false,569 "edit_reason": null,570 "can_view_edit_history": true,571 "wiki": false,572 "post_url": "/t/undefined-symbol-cpublas4gemm-when-loading-extension/203763/2",573 "can_accept_answer": false,574 "can_unaccept_answer": false,575 "accepted_answer": false,576 "topic_accepted_answer": null577 }578 ],579 "stream": [580 444382,581 444722582 ]583 },584 "timeline_lookup": [585 [586 1,587 513588 ],589 [590 2,591 509592 ]593 ],594 "suggested_topics": [595 {596 "fancy_title": "ANN implementation using c++ for a text classification model",597 "id": 219040,598 "title": "ANN implementation using c++ for a text classification model",599 "slug": "ann-implementation-using-c-for-a-text-classification-model",600 "posts_count": 1,601 "reply_count": 0,602 "highest_post_number": 1,603 "image_url": null,604 "created_at": "2025-04-14T02:27:58.483Z",605 "last_posted_at": "2025-04-14T02:27:58.520Z",606 "bumped": true,607 "bumped_at": "2025-04-14T02:27:58.520Z",608 "archetype": "regular",609 "unseen": false,610 "pinned": false,611 "unpinned": null,612 "visible": true,613 "closed": false,614 "archived": false,615 "bookmarked": null,616 "liked": null,617 "tags_descriptions": {},618 "like_count": 0,619 "views": 29,620 "category_id": 11,621 "featured_link": null,622 "has_accepted_answer": false,623 "posters": [624 {625 "extras": "latest single",626 "description": "Original Poster, Most Recent Poster",627 "user": {628 "id": 83566,629 "username": "Hits",630 "name": "",631 "avatar_template": "/letter_avatar_proxy/v4/letter/h/779978/{size}.png",632 "trust_level": 1633 }634 }635 ]636 },637 {638 "fancy_title": "Abort() has been called During DataLoader Iteration in Libtorch on Windows (MSVC + CUDA 11.8)",639 "id": 213027,640 "title": "Abort() has been called During DataLoader Iteration in Libtorch on Windows (MSVC + CUDA 11.8)",641 "slug": "abort-has-been-called-during-dataloader-iteration-in-libtorch-on-windows-msvc-cuda-11-8",642 "posts_count": 2,643 "reply_count": 0,644 "highest_post_number": 2,645 "image_url": "https://discuss.pytorch.org/uploads/default/original/3X/e/a/eac3861e1abbaf2ef43c6a9f8e35b41e9e862848.png",646 "created_at": "2024-11-15T21:10:45.318Z",647 "last_posted_at": "2024-11-16T09:09:27.020Z",648 "bumped": true,649 "bumped_at": "2024-11-16T09:09:27.020Z",650 "archetype": "regular",651 "unseen": false,652 "pinned": false,653 "unpinned": null,654 "visible": true,655 "closed": false,656 "archived": false,657 "bookmarked": null,658 "liked": null,659 "tags_descriptions": {},660 "like_count": 0,661 "views": 44,662 "category_id": 11,663 "featured_link": null,664 "has_accepted_answer": true,665 "posters": [666 {667 "extras": "latest single",668 "description": "Original Poster, Most Recent Poster, Accepted Answer",669 "user": {670 "id": 80942,671 "username": "Saul_Leyva",672 "name": "Saúl",673 "avatar_template": "/user_avatar/discuss.pytorch.org/saul_leyva/{size}/74023_2.png",674 "trust_level": 1675 }676 }677 ]678 },679 {680 "fancy_title": "Frontend C++ model loading error",681 "id": 212639,682 "title": "Frontend C++ model loading error",683 "slug": "frontend-c-model-loading-error",684 "posts_count": 1,685 "reply_count": 0,686 "highest_post_number": 1,687 "image_url": null,688 "created_at": "2024-11-07T03:00:25.420Z",689 "last_posted_at": "2024-11-07T03:00:25.478Z",690 "bumped": true,691 "bumped_at": "2024-11-07T03:00:25.478Z",692 "archetype": "regular",693 "unseen": false,694 "pinned": false,695 "unpinned": null,696 "visible": true,697 "closed": false,698 "archived": false,699 "bookmarked": null,700 "liked": null,701 "tags_descriptions": {},702 "like_count": 0,703 "views": 47,704 "category_id": 11,705 "featured_link": null,706 "has_accepted_answer": false,707 "posters": [708 {709 "extras": "latest single",710 "description": "Original Poster, Most Recent Poster",711 "user": {712 "id": 38591,713 "username": "complexhhs",714 "name": "HsHwang",715 "avatar_template": "/user_avatar/discuss.pytorch.org/complexhhs/{size}/40116_2.png",716 "trust_level": 1717 }718 }719 ]720 },721 {722 "fancy_title": "Cpp torch.sparse usage?",723 "id": 214645,724 "title": "Cpp torch.sparse usage?",725 "slug": "cpp-torch-sparse-usage",726 "posts_count": 2,727 "reply_count": 0,728 "highest_post_number": 2,729 "image_url": null,730 "created_at": "2024-12-26T06:15:05.096Z",731 "last_posted_at": "2024-12-26T19:36:04.276Z",732 "bumped": true,733 "bumped_at": "2024-12-26T19:36:04.276Z",734 "archetype": "regular",735 "unseen": false,736 "pinned": false,737 "unpinned": null,738 "visible": true,739 "closed": false,740 "archived": false,741 "bookmarked": null,742 "liked": null,743 "tags_descriptions": {},744 "like_count": 1,745 "views": 208,746 "category_id": 11,747 "featured_link": null,748 "has_accepted_answer": false,749 "posters": [750 {751 "extras": null,752 "description": "Original Poster",753 "user": {754 "id": 78912,755 "username": "Tanay_Arora",756 "name": "Tanay Arora",757 "avatar_template": "/user_avatar/discuss.pytorch.org/tanay_arora/{size}/72747_2.png",758 "trust_level": 1759 }760 },761 {762 "extras": "latest",763 "description": "Most Recent Poster",764 "user": {765 "id": 3534,766 "username": "ptrblck",767 "name": "",768 "avatar_template": "/user_avatar/discuss.pytorch.org/ptrblck/{size}/1823_2.png",769 "admin": true,770 "moderator": true,771 "trust_level": 2772 }773 }774 ]775 },776 {777 "fancy_title": "Questions about onnx with aten::copy",778 "id": 220532,779 "title": "Questions about onnx with aten::copy",780 "slug": "questions-about-onnx-with-aten-copy",781 "posts_count": 3,782 "reply_count": 0,783 "highest_post_number": 3,784 "image_url": null,785 "created_at": "2025-06-03T08:38:16.945Z",786 "last_posted_at": "2025-09-23T20:56:19.099Z",787 "bumped": true,788 "bumped_at": "2025-09-23T20:56:19.099Z",789 "archetype": "regular",790 "unseen": false,791 "pinned": false,792 "unpinned": null,793 "visible": true,794 "closed": false,795 "archived": false,796 "bookmarked": null,797 "liked": null,798 "tags_descriptions": {},799 "like_count": 0,800 "views": 94,801 "category_id": 11,802 "featured_link": null,803 "has_accepted_answer": false,804 "posters": [805 {806 "extras": null,807 "description": "Original Poster",808 "user": {809 "id": 84559,810 "username": "ZongYingLi",811 "name": "Zoey",812 "avatar_template": "/user_avatar/discuss.pytorch.org/zongyingli/{size}/77250_2.png",813 "trust_level": 0814 }815 },816 {817 "extras": null,818 "description": "Frequent Poster",819 "user": {820 "id": 78425,821 "username": "Vitali-Matteo",822 "name": "Matteo",823 "avatar_template": "/user_avatar/discuss.pytorch.org/vitali-matteo/{size}/72295_2.png",824 "trust_level": 1825 }826 },827 {828 "extras": "latest",829 "description": "Most Recent Poster",830 "user": {831 "id": 56628,832 "username": "justinchuby",833 "name": "Justin",834 "avatar_template": "/user_avatar/discuss.pytorch.org/justinchuby/{size}/50395_2.png",835 "trust_level": 1836 }837 }838 ]839 }840 ],841 "tags_descriptions": {},842 "fancy_title": "Undefined symbol: cpublas4gemm… when loading extension",843 "id": 203763,844 "title": "Undefined symbol: cpublas4gemm... when loading extension",845 "posts_count": 2,846 "created_at": "2024-05-30T11:31:32.708Z",847 "views": 334,848 "reply_count": 0,849 "like_count": 0,850 "last_posted_at": "2024-06-03T09:12:23.549Z",851 "visible": true,852 "closed": false,853 "archived": false,854 "has_summary": false,855 "archetype": "regular",856 "slug": "undefined-symbol-cpublas4gemm-when-loading-extension",857 "category_id": 11,858 "word_count": 2510,859 "deleted_at": null,860 "user_id": 71900,861 "featured_link": null,862 "pinned_globally": false,863 "pinned_at": null,864 "pinned_until": null,865 "image_url": null,866 "slow_mode_seconds": 0,867 "draft": null,868 "draft_key": "topic_203763",869 "draft_sequence": null,870 "unpinned": null,871 "pinned": false,872 "current_post_number": 1,873 "highest_post_number": 2,874 "deleted_by": null,875 "actions_summary": [876 {877 "id": 4,878 "count": 0,879 "hidden": false,880 "can_act": false881 },882 {883 "id": 8,884 "count": 0,885 "hidden": false,886 "can_act": false887 },888 {889 "id": 10,890 "count": 0,891 "hidden": false,892 "can_act": false893 },894 {895 "id": 7,896 "count": 0,897 "hidden": false,898 "can_act": false899 }900 ],901 "chunk_size": 20,902 "bookmarked": false,903 "topic_timer": null,904 "message_bus_last_id": 0,905 "participant_count": 1,906 "show_read_indicator": false,907 "thumbnails": null,908 "slow_mode_enabled_until": null,909 "can_vote": false,910 "vote_count": 0,911 "user_voted": false,912 "discourse_zendesk_plugin_zendesk_id": null,913 "discourse_zendesk_plugin_zendesk_url": "https://your-url.zendesk.com/agent/tickets/",914 "details": {915 "can_edit": false,916 "notification_level": 1,917 "participants": [918 {919 "id": 71900,920 "username": "71M073J",921 "name": "71~073~",922 "avatar_template": "/user_avatar/discuss.pytorch.org/71m073j/{size}/66398_2.png",923 "post_count": 2,924 "primary_group_name": null,925 "flair_name": null,926 "flair_url": null,927 "flair_color": null,928 "flair_bg_color": null,929 "flair_group_id": null,930 "trust_level": 1931 }932 ],933 "created_by": {934 "id": 71900,935 "username": "71M073J",936 "name": "71~073~",937 "avatar_template": "/user_avatar/discuss.pytorch.org/71m073j/{size}/66398_2.png"938 },939 "last_poster": {940 "id": 71900,941 "username": "71M073J",942 "name": "71~073~",943 "avatar_template": "/user_avatar/discuss.pytorch.org/71m073j/{size}/66398_2.png"944 },945 "links": [946 {947 "url": "https://discuss.pytorch.org/t/how-can-i-use-the-function-at-gemm-float/95341",948 "title": "How can I use the function at::cuda::blas::gemm<float>()?",949 "internal": true,950 "attachment": false,951 "reflection": false,952 "clicks": 17,953 "user_id": 71900,954 "domain": "discuss.pytorch.org",955 "root_domain": "pytorch.org"956 },957 {958 "url": "https://github.com/pytorch/pytorch/blob/v2.3.0/aten/src/ATen/native/ConvolutionMM2d.cpp",959 "title": "pytorch/aten/src/ATen/native/ConvolutionMM2d.cpp at v2.3.0 · pytorch/pytorch · GitHub",960 "internal": false,961 "attachment": false,962 "reflection": false,963 "clicks": 1,964 "user_id": 71900,965 "domain": "github.com",966 "root_domain": "github.com"967 }968 ]969 },970 "bookmarks": []971 },972 {973 "post_stream": {974 "posts": [975 {976 "id": 413821,977 "name": "Jing Jin",978 "username": "jinshubai",979 "avatar_template": "/user_avatar/discuss.pytorch.org/jinshubai/{size}/63157_2.png",980 "created_at": "2023-08-14T12:34:29.774Z",981 "cooked": "<p>print(‘args.workers = {}’.format(args.workers))<br>\nval_loader = torch.utils.data.DataLoader(<br>\nval_dataset, batch_size=1000, shuffle=False,<br>\nnum_workers=args.workers, pin_memory=True, sampler=val_sampler)</p>\n<pre><code>val_loader_onnx = torch.utils.data.DataLoader(\n val_dataset, batch_size=1000, shuffle=False,\n num_workers=16, pin_memory=True, sampler=val_sampler)\n\n\ncriterion = nn.CrossEntropyLoss().to(device)\n\n\n**print('QAT Model In Pytorch (SimQuant):')**\nvalidate(val_loader, model, criterion, args)\n\nquantized_eval_model = copy.deepcopy(model.module) \nquantized_eval_model.eval()\nquantized_eval_model.to(torch.device(\"cpu\"))\ntorch.ao.quantization.convert(quantized_eval_model, inplace=True)\n\n\n**print('Converted Model in Pytorch (Quantized):')**\nacc1_quantized = validate_cpu(val_loader, quantized_eval_model, criterion, args)\n\n############################################### Here we convert quantized_model to onnx format ########\ninput_size = torch.randn(1000, 3, 224, 224)\nimport onnx\nimport onnxsim\nquantized_eval_model(input_size)\ntorch.onnx.export(quantized_eval_model, # model being run\n input_size, # model input (or a tuple for multiple inputs)\n './quantized_mobilenetv3_qat.onnx', # where to save the model (can be a file or file-like object)\n export_params=True, # store the trained parameter weights inside the model file\n opset_version=18, # the ONNX version to export the model to\n # do_constant_folding=True, # whether to execute constant folding for optimization\n input_names=['input'], # the model's input names\n output_names=['output'], # the model's output names\n # dynamic_axes={\"input\":{0: \"batch_size\"}, \"output\":{0: \"batch_size\"},}\n # example_outputs=traced(input_fp32)\n )\nmodel_onnx = onnx.load('./quantized_mobilenetv3_qat.onnx')\nonnx.checker.check_model(model_onnx)\nmodel_onnx, check = onnxsim.simplify(\n model_onnx,\n dynamic_input_shape=False,\n overwrite_input_shapes={'input': list(input_size.shape)}\n )\nassert check, 'assert check failed'\nonnx.save(model_onnx, 'quantized_mobilenetv3_sim_qat.onnx')\nprint(\" \")\nprint('Model has been converted to onnx successfully.')\n#######################################################################################################\n</code></pre>\n<p>################################## do onnx-runtime inference #########################################<br>\nimport onnx<br>\nimport onnxruntime as ort<br>\nimport numpy as np<br>\nprint(‘onnx runtime gpu with CUDA’)<br>\nsessionOption = ort.SessionOptions()<br>\nsessionOption.graph_optimization_level = ort.GraphOptimizationLevel.ORT_DISABLE_ALL<br>\nsess = ort.InferenceSession(“./quantized_mobilenetv3_sim_qat.onnx”, sess_options=sessionOption,<br>\nproviders=[‘CUDAExecutionProvider’, ‘CPUExecutionProvider’])<br>\ninput_name = sess.get_inputs()[0].name<br>\noutput_name= sess.get_outputs()[0].name<br>\noutput_shape = sess.get_outputs()[0].shape</p>\n<pre><code>**print('Converted Model in Onnx (Quantized):')**\nvalidate_onnx(val_loader_onnx, sess, criterion, args)\n</code></pre>\n<p>#######################################################################################################</p>\n<p>def validate_onnx(val_loader, ort_session, criterion, args):</p>\n<pre><code>def run_validate(loader, base_progress=0):\n with torch.no_grad():\n end = time.time()\n for i, (images, target) in enumerate(loader):\n i = base_progress + i\n images = images.to('cpu')\n target = target.to('cpu')\n\n # compute output\n input_data = images.detach().cpu().numpy()\n input_name = ort_session.get_inputs()[0].name\n output_name= ort_session.get_outputs()[0].name\n res = ort_session.run([output_name], {input_name: input_data})\n # out = res\n \n import numpy as np\n out = np.array(res)\n\n output_tensor = torch.from_numpy(out[0])\n # print(output_tensor.shape)\n # exit()\n\n loss = criterion(output_tensor, target)\n\n # measure accuracy and record loss\n acc1, acc5 = accuracy(output_tensor, target, topk=(1, 5))\n losses.update(loss.item(), images.size(0))\n top1.update(acc1[0], images.size(0))\n top5.update(acc5[0], images.size(0))\n\n # measure elapsed time\n batch_time.update(time.time() - end)\n end = time.time()\n\n if i % args.print_freq == 0:\n progress.display(i + 1)\n # break\n\nbatch_time = AverageMeter('Time', ':6.3f', Summary.NONE)\nlosses = AverageMeter('Loss', ':.4e', Summary.NONE)\ntop1 = AverageMeter('Acc@1', ':6.2f', Summary.AVERAGE)\ntop5 = AverageMeter('Acc@5', ':6.2f', Summary.AVERAGE)\nprogress = ProgressMeter(\n len(val_loader),\n [batch_time, losses, top1, top5],\n prefix='Test_ONNX: ')\n\n\nrun_validate(val_loader)\nprogress.display_summary()\n\nreturn top1.avg\n</code></pre>\n<p>def validate(val_loader, model, criterion, args):</p>\n<pre><code>def run_validate(loader, base_progress=0):\n with torch.no_grad():\n end = time.time()\n for i, (images, target) in enumerate(loader):\n i = base_progress + i\n if torch.cuda.is_available():\n images = images.cuda()\n\n if torch.cuda.is_available():\n target = target.cuda()\n\n # compute output\n output = model(images)\n loss = criterion(output, target)\n\n # measure accuracy and record loss\n acc1, acc5 = accuracy(output, target, topk=(1, 5))\n losses.update(loss.item(), images.size(0))\n top1.update(acc1[0], images.size(0))\n top5.update(acc5[0], images.size(0))\n\n # measure elapsed time\n batch_time.update(time.time() - end)\n end = time.time()\n\n if i % args.print_freq == 0:\n progress.display(i + 1)\n\nbatch_time = AverageMeter('Time', ':6.3f', Summary.NONE)\nlosses = AverageMeter('Loss', ':.4e', Summary.NONE)\ntop1 = AverageMeter('Acc@1', ':6.2f', Summary.AVERAGE)\ntop5 = AverageMeter('Acc@5', ':6.2f', Summary.AVERAGE)\nprogress = ProgressMeter(\n len(val_loader),\n [batch_time, losses, top1, top5],\n prefix='Test: ')\n\n# switch to evaluate mode\nmodel.eval()\n\nrun_validate(val_loader)\n\nprogress.display_summary()\n\nreturn top1.avg\n</code></pre>\n<p>def validate_cpu(val_loader, model, criterion, args):</p>\n<pre><code>def run_validate(loader, base_progress=0):\n with torch.no_grad():\n end = time.time()\n for i, (images, target) in enumerate(loader):\n i = base_progress + i\n images = images.to('cpu')\n target = target.to('cpu')\n # compute output\n output = model(images)\n loss = criterion(output, target)\n\n # measure accuracy and record loss\n acc1, acc5 = accuracy(output, target, topk=(1, 5))\n losses.update(loss.item(), images.size(0))\n top1.update(acc1[0], images.size(0))\n top5.update(acc5[0], images.size(0))\n\n # measure elapsed time\n batch_time.update(time.time() - end)\n end = time.time()\n\n if i % args.print_freq == 0:\n progress.display(i + 1)\n\nbatch_time = AverageMeter('Time', ':6.3f', Summary.NONE)\nlosses = AverageMeter('Loss', ':.4e', Summary.NONE)\ntop1 = AverageMeter('Acc@1', ':6.2f', Summary.AVERAGE)\ntop5 = AverageMeter('Acc@5', ':6.2f', Summary.AVERAGE)\nprogress = ProgressMeter(\n len(val_loader),\n [batch_time, losses, top1, top5],\n prefix='Test_quantized: ')\n\n# switch to evaluate mode\nmodel.to('cpu')\nmodel.eval()\n\nrun_validate(val_loader)\nprogress.display_summary()\n\nreturn top1.avg\n</code></pre>\n<h1><a name=\"the-log-as-follows-1\" class=\"anchor\" href=\"#the-log-as-follows-1\"></a>the log as follows:</h1>\n<p>QAT Model In Pytorch (SimQuant):<br>\nTest: [ 1/50] Time 19.430 (19.430) Loss 4.4070e-01 (4.4070e-01) Acc@1 87.70 ( 87.70) Acc@5 97.40 ( 97.40)<br>\nTest: [11/50] Time 0.282 ( 2.129) Loss 7.0866e-01 (7.4780e-01) Acc@1 78.10 ( 80.42) Acc@5 97.00 ( 94.98)<br>\nTest: [21/50] Time 0.278 ( 1.248) Loss 1.1263e+00 (7.6287e-01) Acc@1 71.70 ( 79.76) Acc@5 91.20 ( 95.11)<br>\nTest: [31/50] Time 0.282 ( 0.936) Loss 1.0681e+00 (8.8677e-01) Acc@1 74.10 ( 77.07) Acc@5 90.90 ( 93.68)<br>\nTest: [41/50] Time 0.279 ( 0.776) Loss 1.1727e+00 (9.6547e-01) Acc@1 71.50 ( 75.39) Acc@5 90.10 ( 92.52)</p>\n<ul>\n<li>Acc@1 74.776 Acc@5 92.252</li>\n</ul>\n<p>Converted Model in Pytorch (Quantized):<br>\nTest_quantized: [ 1/50] Time 17.260 (17.260) Loss 4.7875e-01 (4.7875e-01) Acc@1 88.20 ( 88.20) Acc@5 97.00 ( 97.00)<br>\nTest_quantized: [11/50] Time 2.513 ( 4.091) Loss 7.3136e-01 (7.6658e-01) Acc@1 79.30 ( 80.25) Acc@5 96.70 ( 94.70)<br>\nTest_quantized: [21/50] Time 2.554 ( 3.346) Loss 1.2043e+00 (7.8450e-01) Acc@1 68.50 ( 79.26) Acc@5 90.70 ( 94.75)<br>\nTest_quantized: [31/50] Time 2.561 ( 3.079) Loss 1.0992e+00 (9.1164e-01) Acc@1 74.10 ( 76.61) Acc@5 90.00 ( 93.15)<br>\nTest_quantized: [41/50] Time 2.560 ( 2.950) Loss 1.2187e+00 (9.9316e-01) Acc@1 69.40 ( 74.74) Acc@5 89.70 ( 92.14)</p>\n<ul>\n<li>Acc@1 74.184 Acc@5 91.934<br>\n============= Diagnostic Run torch.onnx.export version 2.0.1+cu117 =============<br>\nverbose: False, log level: Level.ERROR<br>\n======================= 0 NONE 0 NOTE 0 WARNING 0 ERROR ========================</li>\n</ul>\n<p>Model has been converted to onnx successfully.<br>\nonnx runtime gpu with CUDA<br>\nConverted Model in Onnx (Quantized):<br>\nTest_ONNX: [ 1/50] Time 24.736 (24.736) Loss 4.9879e-01 (4.9879e-01) Acc@1 86.10 ( 86.10) Acc@5 97.10 ( 97.10)<br>\nTest_ONNX: [11/50] Time 7.599 (10.646) Loss 8.6524e-01 (8.7809e-01) Acc@1 75.10 ( 77.11) Acc@5 95.30 ( 93.49)<br>\nTest_ONNX: [21/50] Time 7.718 ( 9.352) Loss 1.2917e+00 (8.8729e-01) Acc@1 64.70 ( 76.34) Acc@5 90.10 ( 93.80)<br>\nTest_ONNX: [31/50] Time 8.425 ( 8.953) Loss 1.1920e+00 (1.0216e+00) Acc@1 71.10 ( 73.64) Acc@5 89.40 ( 92.00)<br>\nTest_ONNX: [41/50] Time 7.646 ( 8.619) Loss 1.2383e+00 (1.0992e+00) Acc@1 68.30 ( 72.02) Acc@5 89.50 ( 90.94)</p>\n<ul>\n<li>Acc@1 71.432 Acc@5 90.598</li>\n</ul>\n<p>How to export a correct quantized model for onnx? Acc1@1 71.423 for onnxruntime drops about 3% accuracy compared to Acc@74.148 for quantied model in pytorch. WHY? Anyone can help me to figure out?? Thanks in advance.</p>",982 "post_number": 1,983 "post_type": 1,984 "posts_count": 7,985 "updated_at": "2023-08-15T06:16:18.600Z",986 "reply_count": 1,987 "reply_to_post_number": null,988 "quote_count": 0,989 "incoming_link_count": 2116,990 "reads": 56,991 "readers_count": 55,992 "score": 10561.2,993 "yours": false,994 "topic_id": 186322,995 "topic_slug": "how-to-export-a-correct-quantized-model-to-onnx-format",996 "display_username": "Jing Jin",997 "primary_group_name": null,998 "flair_name": null,999 "flair_url": null,1000 "flair_bg_color": null,1001 "flair_color": null,1002 "flair_group_id": null,1003 "badges_granted": [],1004 "version": 3,1005 "can_edit": false,1006 "can_delete": false,1007 "can_recover": false,1008 "can_see_hidden_post": false,1009 "can_wiki": false,1010 "read": true,1011 "user_title": null,1012 "bookmarked": false,1013 "actions_summary": [],1014 "moderator": false,1015 "admin": false,1016 "staff": false,1017 "user_id": 68670,1018 "hidden": false,1019 "trust_level": 1,1020 "deleted_at": null,1021 "user_deleted": false,1022 "edit_reason": null,1023 "can_view_edit_history": true,1024 "wiki": false,1025 "post_url": "/t/how-to-export-a-correct-quantized-model-to-onnx-format/186322/1",1026 "can_accept_answer": false,1027 "can_unaccept_answer": false,1028 "accepted_answer": false,1029 "topic_accepted_answer": null,1030 "can_vote": false1031 },1032 {1033 "id": 415062,1034 "name": "Jing Jin",1035 "username": "jinshubai",1036 "avatar_template": "/user_avatar/discuss.pytorch.org/jinshubai/{size}/63157_2.png",1037 "created_at": "2023-08-24T06:14:02.646Z",1038 "cooked": "<p>I figure out that the bias quantization should take care. Here bias is the final bias, i.e. the bias from the fused CNN+BN.</p>",1039 "post_number": 2,1040 "post_type": 1,1041 "posts_count": 7,1042 "updated_at": "2023-08-24T06:14:02.646Z",1043 "reply_count": 1,1044 "reply_to_post_number": null,1045 "quote_count": 0,1046 "incoming_link_count": 8,1047 "reads": 46,1048 "readers_count": 45,1049 "score": 54.2,1050 "yours": false,1051 "topic_id": 186322,1052 "topic_slug": "how-to-export-a-correct-quantized-model-to-onnx-format",1053 "display_username": "Jing Jin",1054 "primary_group_name": null,1055 "flair_name": null,1056 "flair_url": null,1057 "flair_bg_color": null,1058 "flair_color": null,1059 "flair_group_id": null,1060 "badges_granted": [],1061 "version": 1,1062 "can_edit": false,1063 "can_delete": false,1064 "can_recover": false,1065 "can_see_hidden_post": false,1066 "can_wiki": false,1067 "read": true,1068 "user_title": null,1069 "bookmarked": false,1070 "actions_summary": [],1071 "moderator": false,1072 "admin": false,1073 "staff": false,1074 "user_id": 68670,1075 "hidden": false,1076 "trust_level": 1,1077 "deleted_at": null,1078 "user_deleted": false,1079 "edit_reason": null,1080 "can_view_edit_history": true,1081 "wiki": false,1082 "post_url": "/t/how-to-export-a-correct-quantized-model-to-onnx-format/186322/2",1083 "can_accept_answer": false,1084 "can_unaccept_answer": false,1085 "accepted_answer": false,1086 "topic_accepted_answer": null1087 },1088 {1089 "id": 415274,1090 "name": "Hd Charles",1091 "username": "HDCharles",1092 "avatar_template": "/user_avatar/discuss.pytorch.org/hdcharles/{size}/38408_2.png",1093 "created_at": "2023-08-25T17:42:51.943Z",1094 "cooked": "<aside class=\"quote no-group\" data-username=\"jinshubai\" data-post=\"1\" data-topic=\"186322\">\n<div class=\"title\">\n<div class=\"quote-controls\"></div>\n<img loading=\"lazy\" alt=\"\" width=\"24\" height=\"24\" src=\"https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jinshubai/48/63157_2.png\" class=\"avatar\"> jinshubai:</div>\n<blockquote>\n<p>orch.utils.data.DataLoader(<br>\nval_dataset, batch_size=1000, shuffle=False,<br>\nnum_workers=args.workers, pin_memory=True, sampler=val_sampler)</p>\n</blockquote>\n</aside>\n<p>it looks like the quantization part is working but the onnx export is whats causing an issue, you may have better luck asking some of the onnx folks or make a github issue and tag the onnx: oncall since i don’t think anyone in the quantization team has too much onnx experience. If you want to debug this further, i’d probably export each module individually, convert to onnx, and compare the output.</p>",1095 "post_number": 3,1096 "post_type": 1,1097 "posts_count": 7,1098 "updated_at": "2023-08-25T17:42:51.943Z",1099 "reply_count": 1,1100 "reply_to_post_number": 2,1101 "quote_count": 1,1102 "incoming_link_count": 11,1103 "reads": 45,1104 "readers_count": 44,1105 "score": 84.0,1106 "yours": false,1107 "topic_id": 186322,1108 "topic_slug": "how-to-export-a-correct-quantized-model-to-onnx-format",1109 "display_username": "Hd Charles",1110 "primary_group_name": null,1111 "flair_name": null,1112 "flair_url": null,1113 "flair_bg_color": null,1114 "flair_color": null,1115 "flair_group_id": null,1116 "badges_granted": [],1117 "version": 1,1118 "can_edit": false,1119 "can_delete": false,1120 "can_recover": false,1121 "can_see_hidden_post": false,1122 "can_wiki": false,1123 "read": true,1124 "user_title": null,1125 "reply_to_user": {1126 "id": 68670,1127 "username": "jinshubai",1128 "name": "Jing Jin",1129 "avatar_template": "/user_avatar/discuss.pytorch.org/jinshubai/{size}/63157_2.png"1130 },1131 "bookmarked": false,1132 "actions_summary": [1133 {1134 "id": 2,1135 "count": 11136 }1137 ],1138 "moderator": false,1139 "admin": false,1140 "staff": false,1141 "user_id": 45475,1142 "hidden": false,1143 "trust_level": 2,1144 "deleted_at": null,1145 "user_deleted": false,1146 "edit_reason": null,1147 "can_view_edit_history": true,1148 "wiki": false,1149 "post_url": "/t/how-to-export-a-correct-quantized-model-to-onnx-format/186322/3",1150 "can_accept_answer": false,1151 "can_unaccept_answer": false,1152 "accepted_answer": false,1153 "topic_accepted_answer": null1154 },1155 {1156 "id": 415439,1157 "name": "Jing Jin",1158 "username": "jinshubai",1159 "avatar_template": "/user_avatar/discuss.pytorch.org/jinshubai/{size}/63157_2.png",1160 "created_at": "2023-08-28T03:29:52.663Z",1161 "cooked": "<p>Thank, I figured out that what the problem was. It is the issue from the bias quantization. If the scale is extreme small, the bias quantization will beyond the range of int32.</p>",1162 "post_number": 4,1163 "post_type": 1,1164 "posts_count": 7,1165 "updated_at": "2023-08-28T03:29:52.663Z",1166 "reply_count": 1,1167 "reply_to_post_number": 3,1168 "quote_count": 0,1169 "incoming_link_count": 4,1170 "reads": 43,1171 "readers_count": 42,1172 "score": 33.6,1173 "yours": false,1174 "topic_id": 186322,1175 "topic_slug": "how-to-export-a-correct-quantized-model-to-onnx-format",1176 "display_username": "Jing Jin",1177 "primary_group_name": null,1178 "flair_name": null,1179 "flair_url": null,1180 "flair_bg_color": null,1181 "flair_color": null,1182 "flair_group_id": null,1183 "badges_granted": [],1184 "version": 1,1185 "can_edit": false,1186 "can_delete": false,1187 "can_recover": false,1188 "can_see_hidden_post": false,1189 "can_wiki": false,1190 "read": true,1191 "user_title": null,1192 "reply_to_user": {1193 "id": 45475,1194 "username": "HDCharles",1195 "name": "Hd Charles",1196 "avatar_template": "/user_avatar/discuss.pytorch.org/hdcharles/{size}/38408_2.png"1197 },1198 "bookmarked": false,1199 "actions_summary": [],1200 "moderator": false,