RedHatAI/DeepSeek-R1-Distill-Llama-70B-quantized.w8a8
2294
1---2license: mit3tags:4- deepseek5- int86- vllm7- llmcompressor8base_model: deepseek-ai/DeepSeek-R1-Distill-Llama-70B9library_name: transformers10---11 12# DeepSeek-R1-Distill-Llama-70B-quantized.w8a813 14## Model Overview15- **Model Architecture:** LlamaForCausalLM16 - **Input:** Text17 - **Output:** Text18- **Model Optimizations:**19 - **Weight quantization:** INT820 - **Activation quantization:** INT821- **Release Date:** 2/3/202522- **Version:** 1.023- **Model Developers:** Neural Magic24 25Quantized version of [DeepSeek-R1-Distill-Llama-70B](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Llama-70B).26 27 28### Model Optimizations29 30This model was obtained by quantizing the weights and activations of [DeepSeek-R1-Distill-Llama-70B](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Llama-70B) to INT8 data type.31This optimization reduces the number of bits used to represent weights and activations from 16 to 8, reducing GPU memory requirements (by approximately 50%) and increasing matrix-multiply compute throughput (by approximately 2x).32Weight quantization also reduces disk size requirements by approximately 50%.33 34Only the weights and activations of the linear operators within transformers blocks are quantized.35Weights are quantized using a symmetric per-channel scheme, whereas quantizations are quantized using a symmetric per-token scheme.36The [GPTQ](https://arxiv.org/abs/2210.17323) algorithm is applied for quantization, as implemented in the [llm-compressor](https://github.com/vllm-project/llm-compressor) library.37 38 39## Use with vLLM40 41This model can be deployed efficiently using the [vLLM](https://docs.vllm.ai/en/latest/) backend, as shown in the example below.42 43```python44from transformers import AutoTokenizer45from vllm import LLM, SamplingParams46 47number_gpus = 248model_name = "neuralmagic/DeepSeek-R1-Distill-Llama-70B-quantized.w8a8"49 50tokenizer = AutoTokenizer.from_pretrained(model_name)51sampling_params = SamplingParams(temperature=0.6, max_tokens=256, stop_token_ids=[tokenizer.eos_token_id])52llm = LLM(model=model_name, tensor_parallel_size=number_gpus, trust_remote_code=True)53 54messages_list = [55 [{"role": "user", "content": "Who are you? Please respond in pirate speak!"}],56]57 58prompt_token_ids = [tokenizer.apply_chat_template(messages, add_generation_prompt=True) for messages in messages_list]59 60outputs = llm.generate(prompt_token_ids=prompt_token_ids, sampling_params=sampling_params)61 62generated_text = [output.outputs[0].text for output in outputs]63print(generated_text)64```65 66vLLM also supports OpenAI-compatible serving. See the [documentation](https://docs.vllm.ai/en/latest/) for more details.67 68## Creation69 70This model was created with [llm-compressor](https://github.com/vllm-project/llm-compressor) by running the code snippet below. 71 72 73```python74from transformers import AutoModelForCausalLM, AutoTokenizer75from llmcompressor.modifiers.quantization import QuantizationModifier76from llmcompressor.modifiers.smoothquant import SmoothQuantModifier77from llmcompressor.transformers import oneshot78from llmcompressor.transformers.compression.helpers import calculate_offload_device_map79 80# Load model81model_stub = "deepseek-ai/DeepSeek-R1-Distill-Llama-70B"82model_name = model_stub.split("/")[-1]83 84num_samples = 102485max_seq_len = 819286 87tokenizer = AutoTokenizer.from_pretrained(model_stub)88 89device_map = calculate_offload_device_map(90 model_stub,91 reserve_for_hessians=True,92 num_gpus=2,93 torch_dtype="auto",94)95 96model = AutoModelForCausalLM.from_pretrained(97 model_stub,98 device_map=device_map,99 torch_dtype="auto",100)101 102def preprocess_fn(example):103 return {"text": tokenizer.apply_chat_template(example["messages"], add_generation_prompt=False, tokenize=False)}104 105ds = load_dataset("neuralmagic/LLM_compression_calibration", split="train")106ds = ds.map(preprocess_fn)107 108# Configure the quantization algorithm and scheme109recipe = [110 SmoothQuantModifier(smoothing_strength=0.7),111 QuantizationModifier(112 targets="Linear",113 scheme="W8A8",114 ignore=["lm_head"],115 dampening_frac=0.1,116 ),117]118 119# Apply quantization120oneshot(121 model=model,122 dataset=ds, 123 recipe=recipe,124 max_seq_length=max_seq_len,125 num_calibration_samples=num_samples,126)127 128# Save to disk in compressed-tensors format129save_path = model_name + "-quantized.w8a8130model.save_pretrained(save_path)131tokenizer.save_pretrained(save_path)132print(f"Model and tokenizer saved to: {save_path}")133```134 135## Evaluation136 137The model was evaluated on OpenLLM Leaderboard [V1](https://huggingface.co/spaces/open-llm-leaderboard-old/open_llm_leaderboard) and [V2](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard#/), using the following commands:138 139OpenLLM Leaderboard V1:140```141lm_eval \142 --model vllm \143 --model_args pretrained="neuralmagic/DeepSeek-R1-Distill-Llama-70B-quantized.w8a8",dtype=auto,max_model_len=4096,tensor_parallel_size=2,enable_chunked_prefill=True \144 --tasks openllm \145 --write_out \146 --batch_size auto \147 --output_path output_dir \148 --show_config149```150 151OpenLLM Leaderboard V2:152```153lm_eval \154 --model vllm \155 --model_args pretrained="neuralmagic/DeepSeek-R1-Distill-Llama-70B-quantized.w8a8",dtype=auto,max_model_len=4096,tensor_parallel_size=2,enable_chunked_prefill=True \156 --apply_chat_template \157 --fewshot_as_multiturn \158 --tasks leaderboard \159 --write_out \160 --batch_size auto \161 --output_path output_dir \162 --show_config163```164 165### Accuracy166 167<table>168 <thead>169 <tr>170 <th>Category</th>171 <th>Metric</th>172 <th>deepseek-ai/DeepSeek-R1-Distill-Llama-70B</th>173 <th>neuralmagic/DeepSeek-R1-Distill-Llama-70B-quantized.w8a8</th>174 <th>Recovery</th>175 </tr>176 </thead>177 <tbody>178 <tr>179<td rowspan="4"><b>Reasoning</b></td>180<td>AIME 2024 (pass@1)</td>181<td>67.83</td>182<td>67.78</td>183<td>99.93%</td>184</tr>185<tr>186<td>MATH-500 (pass@1)</td>187<td>95.29</td>188<td>95.27</td>189<td>99.98%</td>190</tr>191<tr>192<td>GPQA Diamond (pass@1)</td>193<td>65.57</td>194<td>65.01</td>195<td>99.15%</td>196</tr>197<tr>198<td><b>Average Score</b></td>199<td><b>76.23</b></td>200<td><b>76.02</b></td>201<td><b>99.72%</b></td>202</tr>203 <tr>204 <td rowspan="7"><b>OpenLLM V1</b></td>205 <td>ARC-Challenge (Acc-Norm, 25-shot)</td>206 <td>63.65</td>207 <td>63.57</td>208 <td>99.9%</td>209 </tr>210 <tr>211 <td>GSM8K (Strict-Match, 5-shot)</td>212 <td>93.03</td>213 <td>93.56</td>214 <td>100.6%</td>215 </tr>216 <tr>217 <td>HellaSwag (Acc-Norm, 10-shot)</td>218 <td>84.85</td>219 <td>85.15</td>220 <td>100.4%</td>221 </tr>222 <tr>223 <td>MMLU (Acc, 5-shot)</td>224 <td>78.04</td>225 <td>78.01</td>226 <td>100.0%</td>227 </tr>228 <tr>229 <td>TruthfulQA (MC2, 0-shot)</td>230 <td>56.67</td>231 <td>57.47</td>232 <td>101.4%</td>233 </tr>234 <tr>235 <td>Winogrande (Acc, 5-shot)</td>236 <td>78.22</td>237 <td>78.37</td>238 <td>100.2%</td>239 </tr>240 <tr>241 <td><b>Average Score</b></td>242 <td><b>75.74</b></td>243 <td><b>76.02</b></td>244 <td><b>100.4%</b></td>245 </tr>246 <tr>247 <td rowspan="7"><b>OpenLLM V2</b></td>248 <td>IFEval (Inst Level Strict Acc, 0-shot)</td>249 <td>42.45</td>250 <td>42.51</td>251 <td>100.1%</td>252 </tr>253 <tr>254 <td>BBH (Acc-Norm, 3-shot)</td>255 <td>21.26</td>256 <td>20.78</td>257 <td>97.8%</td>258 </tr>259 <tr>260 <td>Math-Hard (Exact-Match, 4-shot)</td>261 <td>0.00</td>262 <td>0.00</td>263 <td>---</td>264 </tr>265 <tr>266 <td>GPQA (Acc-Norm, 0-shot)</td>267 <td>9.51</td>268 <td>7.25</td>269 <td>---</td>270 </tr>271 <tr>272 <td>MUSR (Acc-Norm, 0-shot)</td>273 <td>14.87</td>274 <td>15.24</td>275 <td>---</td>276 </tr>277 <tr>278 <td>MMLU-Pro (Acc, 5-shot)</td>279 <td>4.27</td>280 <td>5.62</td>281 <td>---</td>282 </tr>283 <tr>284 <td><b>Average Score</b></td>285 <td><b>15.39</b></td>286 <td><b>15.23</b></td>287 <td><b>99.0%</b></td>288 </tr>289 <tr>290 <td rowspan="4"><b>Coding</b></td>291 <td>HumanEval (pass@1)</td>292 <td>81.10</td>293 <td>81.00</td>294 <td><b>99.9%</b></td>295 </tr>296 <tr>297 <td>HumanEval (pass@10)</td>298 <td>87.60</td>299 <td>86.80</td>300 <td>99.1%</td>301 </tr>302 <tr>303 <td>HumanEval+ (pass@10)</td>304 <td>75.20</td>305 <td>75.80</td>306 <td>100.8%</td>307 </tr>308 <tr>309 <td>HumanEval+ (pass@10)</td>310 <td>83.10</td>311 <td>83.40</td>312 <td>100.4%</td>313 </tr>314 </tbody>315</table>316 317## Inference Performance318 319 320This model achieves up to 2.0x speedup in single-stream deployment and up to 2.2x speedup in multi-stream asynchronous deployment, depending on hardware and use-case scenario.321The following performance benchmarks were conducted with [vLLM](https://docs.vllm.ai/en/latest/) version 0.7.2, and [GuideLLM](https://github.com/neuralmagic/guidellm).322 323<details>324<summary>Benchmarking Command</summary>325 326```327guidellm --model neuralmagic/DeepSeek-R1-Distill-Llama-70B-quantized.w8a8 --target "http://localhost:8000/v1" --data-type emulated --data "prompt_tokens=<prompt_tokens>,generated_tokens=<generated_tokens>" --max seconds 360 --backend aiohttp_server328```329</details>330 331### Single-stream performance (measured with vLLM version 0.7.2)332<table>333 <thead>334 <tr>335 <th></th>336 <th></th>337 <th></th>338 <th></th>339 <th style="text-align: center;" colspan="2" >Instruction Following<br>256 / 128</th>340 <th style="text-align: center;" colspan="2" >Multi-turn Chat<br>512 / 256</th>341 <th style="text-align: center;" colspan="2" >Docstring Generation<br>768 / 128</th>342 <th style="text-align: center;" colspan="2" >RAG<br>1024 / 128</th>343 <th style="text-align: center;" colspan="2" >Code Completion<br>256 / 1024</th>344 <th style="text-align: center;" colspan="2" >Code Fixing<br>1024 / 1024</th>345 <th style="text-align: center;" colspan="2" >Large Summarization<br>4096 / 512</th>346 <th style="text-align: center;" colspan="2" >Large RAG<br>10240 / 1536</th>347 </tr>348 <tr>349 <th>GPU class</th>350 <th>Number of GPUs</th>351 <th>Model</th>352 <th>Average cost reduction</th>353 <th>Latency (s)</th>354 <th>QPD</th>355 <th>Latency (s)</th>356 <th>QPD</th>357 <th>Latency (s)</th>358 <th>QPD</th>359 <th>Latency (s)</th>360 <th>QPD</th>361 <th>Latency (s)</th>362 <th>QPD</th>363 <th>Latency (s)</th>364 <th>QPD</th>365 <th>Latency (s)</th>366 <th>QPD</th>367 <th>Latency (s)</th>368 <th>QPD</th>369 </tr>370 </thead>371 <tbody style="text-align: center" >372 <tr>373 <th rowspan="3" valign="top">A6000</th>374 <td>4</td>375 <th>deepseek-ai/DeepSeek-R1-Distill-Llama-70B</th>376 <td>---</td>377 <td>7.4</td>378 <td>152</td>379 <td>14.9</td>380 <td>76</td>381 <td>7.5</td>382 <td>149</td>383 <td>7.7</td>384 <td>146</td>385 <td>57.2</td>386 <td>20</td>387 <td>58.9</td>388 <td>19</td>389 <td>31.9</td>390 <td>35</td>391 <td>98.4</td>392 <td>11</td>393 </tr>394 <tr>395 <td>2</td>396 <th>neuralmagic/DeepSeek-R1-Distill-Llama-70B-quantized.w8a8</th>397 <td>1.93</td>398 <td>7.7</td>399 <td>292</td>400 <td>15.2</td>401 <td>148</td>402 <td>7.8</td>403 <td>287</td>404 <td>8.0</td>405 <td>282</td>406 <td>60.7</td>407 <td>37</td>408 <td>60.2</td>409 <td>37</td>410 <td>32.3</td>411 <td>70</td>412 <td>104.0</td>413 <td>22</td>414 </tr>415 <tr>416 <td>2</td>417 <th>neuralmagic/DeepSeek-R1-Distill-Llama-70B-quantized.w4a16</th>418 <td>2.83</td>419 <td>4.9</td>420 <td>457</td>421 <td>10.0</td>422 <td>225</td>423 <td>5.5</td>424 <td>411</td>425 <td>5.8</td>426 <td>389</td>427 <td>38.9</td>428 <td>58</td>429 <td>39.2</td>430 <td>57</td>431 <td>23.7</td>432 <td>95</td>433 <td>76.6</td>434 <td>29</td>435 </tr>436 <tr>437 <th rowspan="3" valign="top">A100</th>438 <td>2</td>439 <th>deepseek-ai/DeepSeek-R1-Distill-Llama-70B</th>440 <td>---</td>441 <td>6.4</td>442 <td>157</td>443 <td>12.8</td>444 <td>79</td>445 <td>6.6</td>446 <td>153</td>447 <td>6.7</td>448 <td>151</td>449 <td>50.4</td>450 <td>20</td>451 <td>50.8</td>452 <td>20</td>453 <td>27.0</td>454 <td>37</td>455 <td>85.4</td>456 <td>12</td>457 </tr>458 <tr>459 <td>2</td>460 <th>neuralmagic/DeepSeek-R1-Distill-Llama-70B-quantized.w8a8</th>461 <td>1.48</td>462 <td>4.1</td>463 <td>245</td>464 <td>8.2</td>465 <td>123</td>466 <td>4.2</td>467 <td>238</td>468 <td>4.3</td>469 <td>235</td>470 <td>32.4</td>471 <td>31</td>472 <td>32.8</td>473 <td>31</td>474 <td>17.6</td>475 <td>57</td>476 <td>90.8</td>477 <td>11</td>478 </tr>479 <tr>480 <td>1</td>481 <th>neuralmagic/DeepSeek-R1-Distill-Llama-70B-quantized.w4a16</th>482 <td>2.69</td>483 <td>4.6</td>484 <td>440</td>485 <td>9.2</td>486 <td>220</td>487 <td>4.9</td>488 <td>407</td>489 <td>5.2</td>490 <td>389</td>491 <td>35.3</td>492 <td>57</td>493 <td>36.3</td>494 <td>55</td>495 <td>21.2</td>496 <td>95</td>497 <td>68.1</td>498 <td>30</td>499 </tr>500 <tr>501 <th rowspan="3" valign="top">H100</th>502 <td>2</td>503 <th>deepseek-ai/DeepSeek-R1-Distill-Llama-70B</th>504 <td>---</td>505 <td>3.8</td>506 <td>149</td>507 <td>7.6</td>508 <td>74</td>509 <td>3.9</td>510 <td>146</td>511 <td>3.9</td>512 <td>144</td>513 <td>30.0</td>514 <td>19</td>515 <td>30.4</td>516 <td>19</td>517 <td>16.1</td>518 <td>35</td>519 <td>56.5</td>520 <td>10</td>521 </tr>522 <tr>523 <td>2</td>524 <th>neuralmagic/DeepSeek-R1-Distill-Llama-70B-FP8-dynamic</th>525 <td>1.39</td>526 <td>2.7</td>527 <td>210</td>528 <td>5.3</td>529 <td>106</td>530 <td>2.7</td>531 <td>207</td>532 <td>2.8</td>533 <td>203</td>534 <td>21.1</td>535 <td>27</td>536 <td>21.4</td>537 <td>26</td>538 <td>11.5</td>539 <td>49</td>540 <td>47.2</td>541 <td>12</td>542 </tr>543 <tr>544 <td>1</td>545 <th>neuralmagic/DeepSeek-R1-Distill-Llama-70B-quantized.w4a16</th>546 <td>1.83</td>547 <td>4.0</td>548 <td>277</td>549 <td>7.9</td>550 <td>138</td>551 <td>4.1</td>552 <td>266</td>553 <td>4.2</td>554 <td>262</td>555 <td>31.2</td>556 <td>35</td>557 <td>31.8</td>558 <td>34</td>559 <td>17.8</td>560 <td>61</td>561 <td>61.4</td>562 <td>18</td>563 </tr>564 </tbody>565</table>566 567**Use case profiles: prompt tokens / generation tokens568 569**QPD: Queries per dollar, based on on-demand cost at [Lambda Labs](https://lambdalabs.com/service/gpu-cloud) (observed on 2/18/2025).570 571 572### Multi-stream asynchronous performance (measured with vLLM version 0.7.2)573<table>574 <thead>575 <tr>576 <th></th>577 <th></th>578 <th></th>579 <th style="text-align: center;" colspan="2" >Instruction Following<br>256 / 128</th>580 <th style="text-align: center;" colspan="2" >Multi-turn Chat<br>512 / 256</th>581 <th style="text-align: center;" colspan="2" >Docstring Generation<br>768 / 128</th>582 <th style="text-align: center;" colspan="2" >RAG<br>1024 / 128</th>583 <th style="text-align: center;" colspan="2" >Code Completion<br>256 / 1024</th>584 <th style="text-align: center;" colspan="2" >Code Fixing<br>1024 / 1024</th>585 <th style="text-align: center;" colspan="2" >Large Summarization<br>4096 / 512</th>586 <th style="text-align: center;" colspan="2" >Large RAG<br>10240 / 1536</th>587 </tr>588 <tr>589 <th>Hardware</th>590 <th>Model</th>591 <th>Average cost reduction</th>592 <th>Maximum throughput (QPS)</th>593 <th>QPD</th>594 <th>Maximum throughput (QPS)</th>595 <th>QPD</th>596 <th>Maximum throughput (QPS)</th>597 <th>QPD</th>598 <th>Maximum throughput (QPS)</th>599 <th>QPD</th>600 <th>Maximum throughput (QPS)</th>601 <th>QPD</th>602 <th>Maximum throughput (QPS)</th>603 <th>QPD</th>604 <th>Maximum throughput (QPS)</th>605 <th>QPD</th>606 <th>Maximum throughput (QPS)</th>607 <th>QPD</th>608 </tr>609 </thead>610 <tbody style="text-align: center" >611 <tr>612 <th rowspan="3" valign="top">A6000x4</th>613 <th>deepseek-ai/DeepSeek-R1-Distill-Llama-70B</th>614 <td>---</td>615 <td>3.65</td>616 <td>4102</td>617 <td>1.56</td>618 <td>1757</td>619 <td>1.90</td>620 <td>2143</td>621 <td>1.48</td>622 <td>1665</td>623 <td>0.44</td>624 <td>493</td>625 <td>0.34</td>626 <td>380</td>627 <td>0.22</td>628 <td>245</td>629 <td>0.05</td>630 <td>55</td>631 </tr>632 <tr>633 <th>neuralmagic/DeepSeek-R1-Distill-Llama-70B-quantized.w8a8</th>634 <td>1.76</td>635 <td>5.89</td>636 <td>6625</td>637 <td>2.94</td>638 <td>3307</td>639 <td>3.36</td>640 <td>3775</td>641 <td>2.59</td>642 <td>2916</td>643 <td>0.74</td>644 <td>828</td>645 <td>0.53</td>646 <td>601</td>647 <td>0.35</td>648 <td>398</td>649 <td>0.11</td>650 <td>120</td>651 </tr>652 <tr>653 <th>neuralmagic/DeepSeek-R1-Distill-Llama-70B-quantized.w4a16</th>654 <td>1.48</td>655 <td>4.91</td>656 <td>5528</td>657 <td>2.01</td>658 <td>2259</td>659 <td>2.03</td>660 <td>2280</td>661 <td>1.12</td>662 <td>1255</td>663 <td>1.11</td>664 <td>1251</td>665 <td>0.76</td>666 <td>852</td>667 <td>0.24</td>668 <td>267</td>669 <td>0.07</td>670 <td>81</td>671 </tr>672 <tr>673 <th rowspan="3" valign="top">A100x4</th>674 <th>deepseek-ai/DeepSeek-R1-Distill-Llama-70B</th>675 <td>---</td>676 <td>10.41</td>677 <td>5235</td>678 <td>5.10</td>679 <td>2565</td>680 <td>5.50</td>681 <td>2766</td>682 <td>4.36</td>683 <td>2193</td>684 <td>1.49</td>685 <td>751</td>686 <td>1.21</td>687 <td>607</td>688 <td>0.89</td>689 <td>447</td>690 <td>0.19</td>691 <td>98</td>692 </tr>693 <tr>694 <th>neuralmagic/DeepSeek-R1-Distill-Llama-70B-quantized.w8a8</th>695 <td>1.63</td>696 <td>18.11</td>697 <td>9103</td>698 <td>8.90</td>699 <td>4477</td>700 <td>9.41</td>701 <td>4730</td>702 <td>7.42</td>703 <td>3731</td>704 <td>2.44</td>705 <td>1229</td>706 <td>1.89</td>707 <td>948</td>708 <td>1.26</td>709 <td>631</td>710 <td>0.30</td>711 <td>149</td>712 </tr>713 <tr>714 <th>neuralmagic/DeepSeek-R1-Distill-Llama-70B-quantized.w4a16</th>715 <td>1.12</td>716 <td>12.63</td>717 <td>6353</td>718 <td>5.32</td>719 <td>2673</td>720 <td>5.58</td>721 <td>2804</td>722 <td>4.27</td>723 <td>2144</td>724 <td>2.30</td>725 <td>1158</td>726 <td>1.45</td>727 <td>729</td>728 <td>0.76</td>729 <td>381</td>730 <td>0.22</td>731 <td>110</td>732 </tr>733 <tr>734 <th rowspan="3" valign="top">H100x4</th>735 <th>deepseek-ai/DeepSeek-R1-Distill-Llama-70B</th>736 <td>---</td>737 <td>14.04</td>738 <td>2113</td>739 <td>10.85</td>740 <td>1634</td>741 <td>12.25</td>742 <td>1844</td>743 <td>9.93</td>744 <td>1494</td>745 <td>3.68</td>746 <td>554</td>747 <td>2.82</td>748 <td>425</td>749 <td>1.81</td>750 <td>273</td>751 <td>0.35</td>752 <td>52</td>753 </tr>754 <tr>755 <th>neuralmagic/DeepSeek-R1-Distill-Llama-70B-FP8-dynamic</th>756 <td>1.78</td>757 <td>41.44</td>758 <td>6236</td>759 <td>19.64</td>760 <td>2956</td>761 <td>21.03</td>762 <td>3166</td>763 <td>16.72</td>764 <td>2516</td>765 <td>6.01</td>766 <td>904</td>767 <td>4.46</td>768 <td>672</td>769 <td>2.55</td>770 <td>383</td>771 <td>0.49</td>772 <td>74</td>773 </tr>774 <tr>775 <th>neuralmagic/DeepSeek-R1-Distill-Llama-70B-quantized.w4a16</th>776 <td>1.45</td>777 <td>36.61</td>778 <td>5509</td>779 <td>15.12</td>780 <td>2275</td>781 <td>16.24</td>782 <td>2443</td>783 <td>13.22</td>784 <td>1990</td>785 <td>5.48</td>786 <td>825</td>787 <td>3.01</td>788 <td>453</td>789 <td>2.07</td>790 <td>312</td>791 <td>0.43</td>792 <td>64</td>793 </tr>794 </tbody>795</table>796 797**Use case profiles: prompt tokens / generation tokens798 799**QPS: Queries per second.800 801**QPD: Queries per dollar, based on on-demand cost at [Lambda Labs](https://lambdalabs.com/service/gpu-cloud) (observed on 2/18/2025).802 803 804