itsanan/codebert-finetuned-crewai-base

1---2language:3- en4license: apache-2.05tags:6- sentence-transformers7- sentence-similarity8- feature-extraction9- dense10- generated_from_trainer11- dataset_size:90012- loss:MatryoshkaLoss13- loss:MultipleNegativesRankingLoss14base_model: microsoft/codebert-base15widget:16- source_sentence: How to implement __del__?17  sentences:18  - "class SampleMultiCrewFlow(Flow[SimpleState]):\n        @start()\n        def\19    \ first_crew(self):\n            \"\"\"Run first crew.\"\"\"\n            agent\20    \ = Agent(\n                role=\"first agent\",\n                goal=\"first\21    \ task\",\n                backstory=\"first agent\",\n                llm=mock_llm_1,\n\22    \            )\n            task = Task(\n                description=\"First\23    \ task\",\n                expected_output=\"first result\",\n               \24    \ agent=agent,\n            )\n            crew = Crew(\n                agents=[agent],\n\25    \                tasks=[task],\n                share_crew=True,\n           \26    \ )\n\n            result = crew.kickoff()\n\n            assert crew._execution_span\27    \ is not None\n            return str(result.raw)\n\n        @listen(first_crew)\n\28    \        def second_crew(self, first_result: str):\n            \"\"\"Run second\29    \ crew.\"\"\"\n            agent = Agent(\n                role=\"second agent\"\30    ,\n                goal=\"second task\",\n                backstory=\"second agent\"\31    ,\n                llm=mock_llm_2,\n            )\n            task = Task(\n\32    \                description=\"Second task\",\n                expected_output=\"\33    second result\",\n                agent=agent,\n            )\n            crew\34    \ = Crew(\n                agents=[agent],\n                tasks=[task],\n  \35    \              share_crew=True,\n            )\n\n            result = crew.kickoff()\n\36    \n            assert crew._execution_span is not None\n\n            self.state.result\37    \ = f\"{first_result} + {result.raw}\"\n            return self.state.result"38  - "async def test_anthropic_async_with_tools():\n    \"\"\"Test async call with\39    \ tools.\"\"\"\n    llm = AnthropicCompletion(model=\"claude-sonnet-4-0\")\n\n\40    \    tools = [\n        {\n            \"type\": \"function\",\n            \"\41    function\": {\n                \"name\": \"get_weather\",\n                \"\42    description\": \"Get the current weather for a location\",\n                \"\43    parameters\": {\n                    \"type\": \"object\",\n                 \44    \   \"properties\": {\n                        \"location\": {\n             \45    \               \"type\": \"string\",\n                            \"description\"\46    : \"The city and state, e.g. San Francisco, CA\"\n                        }\n\47    \                    },\n                    \"required\": [\"location\"]\n  \48    \              }\n            }\n        }\n    ]\n\n    result = await llm.acall(\n\49    \        \"What's the weather in San Francisco?\",\n        tools=tools\n    )\n\50    \    logging.debug(\"result: %s\", result)\n\n    assert result is not None\n\51    \    assert isinstance(result, str)"52  - "def __del__(self):\n        \"\"\"Cleanup connections on deletion.\"\"\"\n  \53    \      try:\n            if self._connection_pool:\n                for conn in\54    \ self._connection_pool:\n                    try:\n                        conn.close()\n\55    \                    except Exception:  # noqa: PERF203, S110\n              \56    \          pass\n            if self._thread_pool:\n                self._thread_pool.shutdown()\n\57    \        except Exception:  # noqa: S110\n            pass"58- source_sentence: How does route_to_cycle work in Python?59  sentences:60  - "def route_to_cycle(self):\n            execution_log.append(\"router_initial\"\61    )\n            return \"loop\""62  - "def _register_system_event_handlers(self, event_bus: CrewAIEventsBus) -> None:\n\63    \        \"\"\"Register handlers for system signal events (SIGTERM, SIGINT, etc.).\"\64    \"\"\n\n        @on_signal\n        def handle_signal(source: Any, event: SignalEvent)\65    \ -> None:\n            \"\"\"Flush trace batch on system signals to prevent data\66    \ loss.\"\"\"\n            if self.batch_manager.is_batch_initialized():\n   \67    \             self.batch_manager.finalize_batch()"68  - "async def aadd(self) -> None:\n        \"\"\"Add JSON file content asynchronously.\"\69    \"\"\n        content_str = (\n            str(self.content) if isinstance(self.content,\70    \ dict) else self.content\n        )\n        new_chunks = self._chunk_text(content_str)\n\71    \        self.chunks.extend(new_chunks)\n        await self._asave_documents()"72- source_sentence: Explain the test_evaluate logic73  sentences:74  - "def test_flow_copy_state_with_unpickleable_objects():\n    \"\"\"Test that _copy_state\75    \ handles unpickleable objects like RLock.\n\n    Regression test for issue #3828:\76    \ Flow should not crash when state contains\n    objects that cannot be deep copied\77    \ (like threading.RLock).\n    \"\"\"\n\n    class StateWithRLock(BaseModel):\n\78    \        counter: int = 0\n        lock: Optional[threading.RLock] = None\n\n\79    \    class FlowWithRLock(Flow[StateWithRLock]):\n        @start()\n        def\80    \ step_1(self):\n            self.state.counter += 1\n\n        @listen(step_1)\n\81    \        def step_2(self):\n            self.state.counter += 1\n\n    flow =\82    \ FlowWithRLock(initial_state=StateWithRLock())\n    flow._state.lock = threading.RLock()\n\83    \n    copied_state = flow._copy_state()\n    assert copied_state.counter == 0\n\84    \    assert copied_state.lock is not None"85  - "def test_evaluate(self, crew_planner):\n        task_output = TaskOutput(\n \86    \           description=\"Task 1\", agent=str(crew_planner.crew.agents[0])\n \87    \       )\n\n        with mock.patch.object(Task, \"execute_sync\") as execute:\n\88    \            execute().pydantic = TaskEvaluationPydanticOutput(quality=9.5)\n\89    \            crew_planner.evaluate(task_output)\n            assert crew_planner.tasks_scores[0]\90    \ == [9.5]"91  - "class SlowAsyncTool(BaseTool):\n            name: str = \"slow_async\"\n    \92    \        description: str = \"Simulates slow I/O\"\n\n            def _run(self,\93    \ task_id: int, delay: float) -> str:\n                return f\"Task {task_id}\94    \ done\"\n\n            async def _arun(self, task_id: int, delay: float) -> str:\n\95    \                await asyncio.sleep(delay)\n                return f\"Task {task_id}\96    \ done\""97- source_sentence: Explain the test_clean_action_no_formatting logic98  sentences:99  - "def test_task_interpolation_with_hyphens():\n    agent = Agent(\n        role=\"\100    Researcher\",\n        goal=\"be an assistant that responds with {interpolation-with-hyphens}\"\101    ,\n        backstory=\"You're an expert researcher, specialized in technology,\102    \ software engineering, AI and startups. You work as a freelancer and is now working\103    \ on doing research and analysis for a new customer.\",\n        allow_delegation=False,\n\104    \    )\n    task = Task(\n        description=\"be an assistant that responds\105    \ with {interpolation-with-hyphens}\",\n        expected_output=\"The response\106    \ should be addressing: {interpolation-with-hyphens}\",\n        agent=agent,\n\107    \    )\n    crew = Crew(\n        agents=[agent],\n        tasks=[task],\n   \108    \     verbose=True,\n    )\n    result = crew.kickoff(inputs={\"interpolation-with-hyphens\"\109    : \"say hello world\"})\n    assert \"say hello world\" in task.prompt()\n\n \110    \   assert result.raw == \"Hello, World!\""111  - "class LLMCallCompletedEvent(LLMEventBase):\n    \"\"\"Event emitted when a LLM\112    \ call completes\"\"\"\n\n    type: str = \"llm_call_completed\"\n    messages:\113    \ str | list[dict[str, Any]] | None = None\n    response: Any\n    call_type:\114    \ LLMCallType\n    model: str | None = None"115  - "def test_clean_action_no_formatting():\n    action = \"Ask question to senior\116    \ researcher\"\n    cleaned_action = parser._clean_action(action)\n    assert\117    \ cleaned_action == \"Ask question to senior researcher\""118- source_sentence: Example usage of test_status_code_and_content_type119  sentences:120  - "class NavigateBackToolInput(BaseModel):\n    \"\"\"Input for NavigateBackTool.\"\121    \"\"\n\n    thread_id: str = Field(\n        default=\"default\", description=\"\122    Thread ID for the browser session\"\n    )"123  - "def test_status_code_and_content_type(self, mock_bs, mock_get):\n        for\124    \ status in [200, 201, 301]:\n            mock_get.return_value = self.setup_mock_response(\n\125    \                f\"<html><body>Status {status}</body></html>\", status_code=status\n\126    \            )\n            mock_bs.return_value = self.setup_mock_soup(f\"Status\127    \ {status}\")\n            result = WebPageLoader().load(\n                SourceContent(f\"\128    https://example.com/{status}\")\n            )\n            assert result.metadata[\"\129    status_code\"] == status\n\n        for ctype in [\"text/html\", \"text/plain\"\130    , \"application/xhtml+xml\"]:\n            mock_get.return_value = self.setup_mock_response(\n\131    \                \"<html><body>Content</body></html>\", content_type=ctype\n \132    \           )\n            mock_bs.return_value = self.setup_mock_soup(\"Content\"\133    )\n            result = WebPageLoader().load(SourceContent(\"https://example.com\"\134    ))\n            assert result.metadata[\"content_type\"] == ctype"135  - "def set_crew(self, crew: Any) -> Memory:\n        \"\"\"Set the crew for this\136    \ memory instance.\"\"\"\n        self.crew = crew\n        return self"137pipeline_tag: sentence-similarity138library_name: sentence-transformers139metrics:140- cosine_accuracy@1141- cosine_accuracy@3142- cosine_accuracy@5143- cosine_accuracy@10144- cosine_precision@1145- cosine_precision@3146- cosine_precision@5147- cosine_precision@10148- cosine_recall@1149- cosine_recall@3150- cosine_recall@5151- cosine_recall@10152- cosine_ndcg@10153- cosine_mrr@10154- cosine_map@100155model-index:156- name: CodeBERT Fine-tuned on CrewAI (LR=2e-05)157  results:158  - task:159      type: information-retrieval160      name: Information Retrieval161    dataset:162      name: dim 768163      type: dim_768164    metrics:165    - type: cosine_accuracy@1166      value: 0.04167      name: Cosine Accuracy@1168    - type: cosine_accuracy@3169      value: 0.04170      name: Cosine Accuracy@3171    - type: cosine_accuracy@5172      value: 0.04173      name: Cosine Accuracy@5174    - type: cosine_accuracy@10175      value: 0.06176      name: Cosine Accuracy@10177    - type: cosine_precision@1178      value: 0.04179      name: Cosine Precision@1180    - type: cosine_precision@3181      value: 0.04182      name: Cosine Precision@3183    - type: cosine_precision@5184      value: 0.04185      name: Cosine Precision@5186    - type: cosine_precision@10187      value: 0.03188      name: Cosine Precision@10189    - type: cosine_recall@1190      value: 0.008191      name: Cosine Recall@1192    - type: cosine_recall@3193      value: 0.024194      name: Cosine Recall@3195    - type: cosine_recall@5196      value: 0.04197      name: Cosine Recall@5198    - type: cosine_recall@10199      value: 0.06200      name: Cosine Recall@10201    - type: cosine_ndcg@10202      value: 0.050819890355577976203      name: Cosine Ndcg@10204    - type: cosine_mrr@10205      value: 0.04333333333333334206      name: Cosine Mrr@10207    - type: cosine_map@100208      value: 0.06130275691848844209      name: Cosine Map@100210  - task:211      type: information-retrieval212      name: Information Retrieval213    dataset:214      name: dim 512215      type: dim_512216    metrics:217    - type: cosine_accuracy@1218      value: 0.01219      name: Cosine Accuracy@1220    - type: cosine_accuracy@3221      value: 0.01222      name: Cosine Accuracy@3223    - type: cosine_accuracy@5224      value: 0.01225      name: Cosine Accuracy@5226    - type: cosine_accuracy@10227      value: 0.01228      name: Cosine Accuracy@10229    - type: cosine_precision@1230      value: 0.01231      name: Cosine Precision@1232    - type: cosine_precision@3233      value: 0.01234      name: Cosine Precision@3235    - type: cosine_precision@5236      value: 0.01237      name: Cosine Precision@5238    - type: cosine_precision@10239      value: 0.005240      name: Cosine Precision@10241    - type: cosine_recall@1242      value: 0.002243      name: Cosine Recall@1244    - type: cosine_recall@3245      value: 0.006246      name: Cosine Recall@3247    - type: cosine_recall@5248      value: 0.01249      name: Cosine Recall@5250    - type: cosine_recall@10251      value: 0.01252      name: Cosine Recall@10253    - type: cosine_ndcg@10254      value: 0.01255      name: Cosine Ndcg@10256    - type: cosine_mrr@10257      value: 0.01258      name: Cosine Mrr@10259    - type: cosine_map@100260      value: 0.019316331411936505261      name: Cosine Map@100262  - task:263      type: information-retrieval264      name: Information Retrieval265    dataset:266      name: dim 256267      type: dim_256268    metrics:269    - type: cosine_accuracy@1270      value: 0.01271      name: Cosine Accuracy@1272    - type: cosine_accuracy@3273      value: 0.01274      name: Cosine Accuracy@3275    - type: cosine_accuracy@5276      value: 0.01277      name: Cosine Accuracy@5278    - type: cosine_accuracy@10279      value: 0.03280      name: Cosine Accuracy@10281    - type: cosine_precision@1282      value: 0.01283      name: Cosine Precision@1284    - type: cosine_precision@3285      value: 0.01286      name: Cosine Precision@3287    - type: cosine_precision@5288      value: 0.01289      name: Cosine Precision@5290    - type: cosine_precision@10291      value: 0.015292      name: Cosine Precision@10293    - type: cosine_recall@1294      value: 0.002295      name: Cosine Recall@1296    - type: cosine_recall@3297      value: 0.006298      name: Cosine Recall@3299    - type: cosine_recall@5300      value: 0.01301      name: Cosine Recall@5302    - type: cosine_recall@10303      value: 0.03304      name: Cosine Recall@10305    - type: cosine_ndcg@10306      value: 0.020819890355577977307      name: Cosine Ndcg@10308    - type: cosine_mrr@10309      value: 0.013333333333333334310      name: Cosine Mrr@10311    - type: cosine_map@100312      value: 0.028978936077832484313      name: Cosine Map@100314  - task:315      type: information-retrieval316      name: Information Retrieval317    dataset:318      name: dim 128319      type: dim_128320    metrics:321    - type: cosine_accuracy@1322      value: 0.01323      name: Cosine Accuracy@1324    - type: cosine_accuracy@3325      value: 0.01326      name: Cosine Accuracy@3327    - type: cosine_accuracy@5328      value: 0.01329      name: Cosine Accuracy@5330    - type: cosine_accuracy@10331      value: 0.01332      name: Cosine Accuracy@10333    - type: cosine_precision@1334      value: 0.01335      name: Cosine Precision@1336    - type: cosine_precision@3337      value: 0.01338      name: Cosine Precision@3339    - type: cosine_precision@5340      value: 0.01341      name: Cosine Precision@5342    - type: cosine_precision@10343      value: 0.005344      name: Cosine Precision@10345    - type: cosine_recall@1346      value: 0.002347      name: Cosine Recall@1348    - type: cosine_recall@3349      value: 0.006350      name: Cosine Recall@3351    - type: cosine_recall@5352      value: 0.01353      name: Cosine Recall@5354    - type: cosine_recall@10355      value: 0.01356      name: Cosine Recall@10357    - type: cosine_ndcg@10358      value: 0.01359      name: Cosine Ndcg@10360    - type: cosine_mrr@10361      value: 0.01362      name: Cosine Mrr@10363    - type: cosine_map@100364      value: 0.027544667112101906365      name: Cosine Map@100366  - task:367      type: information-retrieval368      name: Information Retrieval369    dataset:370      name: dim 64371      type: dim_64372    metrics:373    - type: cosine_accuracy@1374      value: 0.05375      name: Cosine Accuracy@1376    - type: cosine_accuracy@3377      value: 0.05378      name: Cosine Accuracy@3379    - type: cosine_accuracy@5380      value: 0.05381      name: Cosine Accuracy@5382    - type: cosine_accuracy@10383      value: 0.07384      name: Cosine Accuracy@10385    - type: cosine_precision@1386      value: 0.05387      name: Cosine Precision@1388    - type: cosine_precision@3389      value: 0.05390      name: Cosine Precision@3391    - type: cosine_precision@5392      value: 0.05393      name: Cosine Precision@5394    - type: cosine_precision@10395      value: 0.035396      name: Cosine Precision@10397    - type: cosine_recall@1398      value: 0.01399      name: Cosine Recall@1400    - type: cosine_recall@3401      value: 0.03402      name: Cosine Recall@3403    - type: cosine_recall@5404      value: 0.05405      name: Cosine Recall@5406    - type: cosine_recall@10407      value: 0.07408      name: Cosine Recall@10409    - type: cosine_ndcg@10410      value: 0.06081989035557797411      name: Cosine Ndcg@10412    - type: cosine_mrr@10413      value: 0.05333333333333334414      name: Cosine Mrr@10415    - type: cosine_map@100416      value: 0.0838507480466874417      name: Cosine Map@100418---419 420# CodeBERT Fine-tuned on CrewAI (LR=2e-05)421 422This is a [sentence-transformers](https://www.SBERT.net) model finetuned from [microsoft/codebert-base](https://huggingface.co/microsoft/codebert-base). It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.423 424## Model Details425 426### Model Description427- **Model Type:** Sentence Transformer428- **Base model:** [microsoft/codebert-base](https://huggingface.co/microsoft/codebert-base) <!-- at revision 3b0952feddeffad0063f274080e3c23d75e7eb39 -->429- **Maximum Sequence Length:** 512 tokens430- **Output Dimensionality:** 768 dimensions431- **Similarity Function:** Cosine Similarity432<!-- - **Training Dataset:** Unknown -->433- **Language:** en434- **License:** apache-2.0435 436### Model Sources437 438- **Documentation:** [Sentence Transformers Documentation](https://sbert.net)439- **Repository:** [Sentence Transformers on GitHub](https://github.com/huggingface/sentence-transformers)440- **Hugging Face:** [Sentence Transformers on Hugging Face](https://huggingface.co/models?library=sentence-transformers)441 442### Full Model Architecture443 444```445SentenceTransformer(446  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False, 'architecture': 'RobertaModel'})447  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})448)449```450 451## Usage452 453### Direct Usage (Sentence Transformers)454 455First install the Sentence Transformers library:456 457```bash458pip install -U sentence-transformers459```460 461Then you can load this model and run inference.462```python463from sentence_transformers import SentenceTransformer464 465# Download from the 🤗 Hub466model = SentenceTransformer("itsanan/codebert-finetuned-crewai-base")467# Run inference468sentences = [469    'Example usage of test_status_code_and_content_type',470    'def test_status_code_and_content_type(self, mock_bs, mock_get):\n        for status in [200, 201, 301]:\n            mock_get.return_value = self.setup_mock_response(\n                f"<html><body>Status {status}</body></html>", status_code=status\n            )\n            mock_bs.return_value = self.setup_mock_soup(f"Status {status}")\n            result = WebPageLoader().load(\n                SourceContent(f"https://example.com/{status}")\n            )\n            assert result.metadata["status_code"] == status\n\n        for ctype in ["text/html", "text/plain", "application/xhtml+xml"]:\n            mock_get.return_value = self.setup_mock_response(\n                "<html><body>Content</body></html>", content_type=ctype\n            )\n            mock_bs.return_value = self.setup_mock_soup("Content")\n            result = WebPageLoader().load(SourceContent("https://example.com"))\n            assert result.metadata["content_type"] == ctype',471    'def set_crew(self, crew: Any) -> Memory:\n        """Set the crew for this memory instance."""\n        self.crew = crew\n        return self',472]473embeddings = model.encode(sentences)474print(embeddings.shape)475# [3, 768]476 477# Get the similarity scores for the embeddings478similarities = model.similarity(embeddings, embeddings)479print(similarities)480# tensor([[1.0000, 0.9009, 0.9087],481#         [0.9009, 1.0000, 0.9053],482#         [0.9087, 0.9053, 1.0000]])483```484 485<!--486### Direct Usage (Transformers)487 488<details><summary>Click to see the direct usage in Transformers</summary>489 490</details>491-->492 493<!--494### Downstream Usage (Sentence Transformers)495 496You can finetune this model on your own dataset.497 498<details><summary>Click to expand</summary>499 500</details>501-->502 503<!--504### Out-of-Scope Use505 506*List how the model may foreseeably be misused and address what users ought not to do with the model.*507-->508 509## Evaluation510 511### Metrics512 513#### Information Retrieval514 515* Dataset: `dim_768`516* Evaluated with [<code>InformationRetrievalEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator) with these parameters:517  ```json518  {519      "truncate_dim": 768520  }521  ```522 523| Metric              | Value      |524|:--------------------|:-----------|525| cosine_accuracy@1   | 0.04       |526| cosine_accuracy@3   | 0.04       |527| cosine_accuracy@5   | 0.04       |528| cosine_accuracy@10  | 0.06       |529| cosine_precision@1  | 0.04       |530| cosine_precision@3  | 0.04       |531| cosine_precision@5  | 0.04       |532| cosine_precision@10 | 0.03       |533| cosine_recall@1     | 0.008      |534| cosine_recall@3     | 0.024      |535| cosine_recall@5     | 0.04       |536| cosine_recall@10    | 0.06       |537| **cosine_ndcg@10**  | **0.0508** |538| cosine_mrr@10       | 0.0433     |539| cosine_map@100      | 0.0613     |540 541#### Information Retrieval542 543* Dataset: `dim_512`544* Evaluated with [<code>InformationRetrievalEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator) with these parameters:545  ```json546  {547      "truncate_dim": 512548  }549  ```550 551| Metric              | Value    |552|:--------------------|:---------|553| cosine_accuracy@1   | 0.01     |554| cosine_accuracy@3   | 0.01     |555| cosine_accuracy@5   | 0.01     |556| cosine_accuracy@10  | 0.01     |557| cosine_precision@1  | 0.01     |558| cosine_precision@3  | 0.01     |559| cosine_precision@5  | 0.01     |560| cosine_precision@10 | 0.005    |561| cosine_recall@1     | 0.002    |562| cosine_recall@3     | 0.006    |563| cosine_recall@5     | 0.01     |564| cosine_recall@10    | 0.01     |565| **cosine_ndcg@10**  | **0.01** |566| cosine_mrr@10       | 0.01     |567| cosine_map@100      | 0.0193   |568 569#### Information Retrieval570 571* Dataset: `dim_256`572* Evaluated with [<code>InformationRetrievalEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator) with these parameters:573  ```json574  {575      "truncate_dim": 256576  }577  ```578 579| Metric              | Value      |580|:--------------------|:-----------|581| cosine_accuracy@1   | 0.01       |582| cosine_accuracy@3   | 0.01       |583| cosine_accuracy@5   | 0.01       |584| cosine_accuracy@10  | 0.03       |585| cosine_precision@1  | 0.01       |586| cosine_precision@3  | 0.01       |587| cosine_precision@5  | 0.01       |588| cosine_precision@10 | 0.015      |589| cosine_recall@1     | 0.002      |590| cosine_recall@3     | 0.006      |591| cosine_recall@5     | 0.01       |592| cosine_recall@10    | 0.03       |593| **cosine_ndcg@10**  | **0.0208** |594| cosine_mrr@10       | 0.0133     |595| cosine_map@100      | 0.029      |596 597#### Information Retrieval598 599* Dataset: `dim_128`600* Evaluated with [<code>InformationRetrievalEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator) with these parameters:601  ```json602  {603      "truncate_dim": 128604  }605  ```606 607| Metric              | Value    |608|:--------------------|:---------|609| cosine_accuracy@1   | 0.01     |610| cosine_accuracy@3   | 0.01     |611| cosine_accuracy@5   | 0.01     |612| cosine_accuracy@10  | 0.01     |613| cosine_precision@1  | 0.01     |614| cosine_precision@3  | 0.01     |615| cosine_precision@5  | 0.01     |616| cosine_precision@10 | 0.005    |617| cosine_recall@1     | 0.002    |618| cosine_recall@3     | 0.006    |619| cosine_recall@5     | 0.01     |620| cosine_recall@10    | 0.01     |621| **cosine_ndcg@10**  | **0.01** |622| cosine_mrr@10       | 0.01     |623| cosine_map@100      | 0.0275   |624 625#### Information Retrieval626 627* Dataset: `dim_64`628* Evaluated with [<code>InformationRetrievalEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator) with these parameters:629  ```json630  {631      "truncate_dim": 64632  }633  ```634 635| Metric              | Value      |636|:--------------------|:-----------|637| cosine_accuracy@1   | 0.05       |638| cosine_accuracy@3   | 0.05       |639| cosine_accuracy@5   | 0.05       |640| cosine_accuracy@10  | 0.07       |641| cosine_precision@1  | 0.05       |642| cosine_precision@3  | 0.05       |643| cosine_precision@5  | 0.05       |644| cosine_precision@10 | 0.035      |645| cosine_recall@1     | 0.01       |646| cosine_recall@3     | 0.03       |647| cosine_recall@5     | 0.05       |648| cosine_recall@10    | 0.07       |649| **cosine_ndcg@10**  | **0.0608** |650| cosine_mrr@10       | 0.0533     |651| cosine_map@100      | 0.0839     |652 653<!--654## Bias, Risks and Limitations655 656*What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.*657-->658 659<!--660### Recommendations661 662*What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.*663-->664 665## Training Details666 667### Training Dataset668 669#### Unnamed Dataset670 671* Size: 900 training samples672* Columns: <code>anchor</code> and <code>positive</code>673* Approximate statistics based on the first 900 samples:674  |         | anchor                                                                             | positive                                                                             |675  |:--------|:-----------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------|676  | type    | string                                                                             | string                                                                               |677  | details | <ul><li>min: 6 tokens</li><li>mean: 13.86 tokens</li><li>max: 141 tokens</li></ul> | <ul><li>min: 20 tokens</li><li>mean: 253.07 tokens</li><li>max: 512 tokens</li></ul> |678* Samples:679  | anchor                                                 | positive                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |680  |:-------------------------------------------------------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|681  | <code>How to implement LLMCallCompletedEvent?</code>   | <code>class LLMCallCompletedEvent(LLMEventBase):<br>    """Event emitted when a LLM call completes"""<br><br>    type: str = "llm_call_completed"<br>    messages: str \| list[dict[str, Any]] \| None = None<br>    response: Any<br>    call_type: LLMCallType<br>    model: str \| None = None</code>                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |682  | <code>How does get_llm_response work in Python?</code> | <code>def get_llm_response(<br>    llm: LLM \| BaseLLM,<br>    messages: list[LLMMessage],<br>    callbacks: list[TokenCalcHandler],<br>    printer: Printer,<br>    from_task: Task \| None = None,<br>    from_agent: Agent \| LiteAgent \| None = None,<br>    response_model: type[BaseModel] \| None = None,<br>    executor_context: CrewAgentExecutor \| LiteAgent \| None = None,<br>) -> str:<br>    """Call the LLM and return the response, handling any invalid responses.<br><br>    Args:<br>        llm: The LLM instance to call.<br>        messages: The messages to send to the LLM.<br>        callbacks: List of callbacks for the LLM call.<br>        printer: Printer instance for output.<br>        from_task: Optional task context for the LLM call.<br>        from_agent: Optional agent context for the LLM call.<br>        response_model: Optional Pydantic model for structured outputs.<br>        executor_context: Optional executor context for hook invocation.<br><br>    Returns:<br>        The response from the LLM as a string.<br><br>    Raises:<br>        Exception: If an error ...</code> |683  | <code>Example usage of _run</code>                     | <code>def _run(<br>        self,<br>        **kwargs: Any,<br>    ) -> Any:<br>        website_url: str \| None = kwargs.get("website_url", self.website_url)<br>        if website_url is None:<br>            raise ValueError("Website URL must be provided.")<br><br>        page = requests.get(<br>            website_url,<br>            timeout=15,<br>            headers=self.headers,<br>            cookies=self.cookies if self.cookies else {},<br>        )<br><br>        page.encoding = page.apparent_encoding<br>        parsed = BeautifulSoup(page.text, "html.parser")<br><br>        text = "The following text is scraped website content:\n\n"<br>        text += parsed.get_text(" ")<br>        text = re.sub("[ \t]+", " ", text)<br>        return re.sub("\\s+\n\\s+", "\n", text)</code>                                                                                                                                                                                                                                                                                                                      |684* Loss: [<code>MatryoshkaLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#matryoshkaloss) with these parameters:685  ```json686  {687      "loss": "MultipleNegativesRankingLoss",688      "matryoshka_dims": [689          768,690          512,691          256,692          128,693          64694      ],695      "matryoshka_weights": [696          1,697          1,698          1,699          1,700          1701      ],702      "n_dims_per_step": -1703  }704  ```705 706### Training Hyperparameters707#### Non-Default Hyperparameters708 709- `eval_strategy`: steps710- `per_device_train_batch_size`: 4711- `per_device_eval_batch_size`: 4712- `gradient_accumulation_steps`: 32713- `learning_rate`: 2e-05714- `weight_decay`: 0.01715- `num_train_epochs`: 20716- `lr_scheduler_type`: cosine717- `warmup_ratio`: 0.1718- `fp16`: True719- `load_best_model_at_end`: True720- `optim`: adamw_torch721- `batch_sampler`: no_duplicates722 723#### All Hyperparameters724<details><summary>Click to expand</summary>725 726- `overwrite_output_dir`: False727- `do_predict`: False728- `eval_strategy`: steps729- `prediction_loss_only`: True730- `per_device_train_batch_size`: 4731- `per_device_eval_batch_size`: 4732- `per_gpu_train_batch_size`: None733- `per_gpu_eval_batch_size`: None734- `gradient_accumulation_steps`: 32735- `eval_accumulation_steps`: None736- `torch_empty_cache_steps`: None737- `learning_rate`: 2e-05738- `weight_decay`: 0.01739- `adam_beta1`: 0.9740- `adam_beta2`: 0.999741- `adam_epsilon`: 1e-08742- `max_grad_norm`: 1.0743- `num_train_epochs`: 20744- `max_steps`: -1745- `lr_scheduler_type`: cosine746- `lr_scheduler_kwargs`: None747- `warmup_ratio`: 0.1748- `warmup_steps`: 0749- `log_level`: passive750- `log_level_replica`: warning751- `log_on_each_node`: True752- `logging_nan_inf_filter`: True753- `save_safetensors`: True754- `save_on_each_node`: False755- `save_only_model`: False756- `restore_callback_states_from_checkpoint`: False757- `no_cuda`: False758- `use_cpu`: False759- `use_mps_device`: False760- `seed`: 42761- `data_seed`: None762- `jit_mode_eval`: False763- `bf16`: False764- `fp16`: True765- `fp16_opt_level`: O1766- `half_precision_backend`: auto767- `bf16_full_eval`: False768- `fp16_full_eval`: False769- `tf32`: None770- `local_rank`: 0771- `ddp_backend`: None772- `tpu_num_cores`: None773- `tpu_metrics_debug`: False774- `debug`: []775- `dataloader_drop_last`: False776- `dataloader_num_workers`: 0777- `dataloader_prefetch_factor`: None778- `past_index`: -1779- `disable_tqdm`: False780- `remove_unused_columns`: True781- `label_names`: None782- `load_best_model_at_end`: True783- `ignore_data_skip`: False784- `fsdp`: []785- `fsdp_min_num_params`: 0786- `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}787- `fsdp_transformer_layer_cls_to_wrap`: None788- `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}789- `parallelism_config`: None790- `deepspeed`: None791- `label_smoothing_factor`: 0.0792- `optim`: adamw_torch793- `optim_args`: None794- `adafactor`: False795- `group_by_length`: False796- `length_column_name`: length797- `project`: huggingface798- `trackio_space_id`: trackio799- `ddp_find_unused_parameters`: None800- `ddp_bucket_cap_mb`: None801- `ddp_broadcast_buffers`: False802- `dataloader_pin_memory`: True803- `dataloader_persistent_workers`: False804- `skip_memory_metrics`: True805- `use_legacy_prediction_loop`: False806- `push_to_hub`: False807- `resume_from_checkpoint`: None808- `hub_model_id`: None809- `hub_strategy`: every_save810- `hub_private_repo`: None811- `hub_always_push`: False812- `hub_revision`: None813- `gradient_checkpointing`: False814- `gradient_checkpointing_kwargs`: None815- `include_inputs_for_metrics`: False816- `include_for_metrics`: []817- `eval_do_concat_batches`: True818- `fp16_backend`: auto819- `push_to_hub_model_id`: None820- `push_to_hub_organization`: None821- `mp_parameters`: 822- `auto_find_batch_size`: False823- `full_determinism`: False824- `torchdynamo`: None825- `ray_scope`: last826- `ddp_timeout`: 1800827- `torch_compile`: False828- `torch_compile_backend`: None829- `torch_compile_mode`: None830- `include_tokens_per_second`: False831- `include_num_input_tokens_seen`: no832- `neftune_noise_alpha`: None833- `optim_target_modules`: None834- `batch_eval_metrics`: False835- `eval_on_start`: False836- `use_liger_kernel`: False837- `liger_kernel_config`: None838- `eval_use_gather_object`: False839- `average_tokens_across_devices`: True840- `prompts`: None841- `batch_sampler`: no_duplicates842- `multi_dataset_batch_sampler`: proportional843- `router_mapping`: {}844- `learning_rate_mapping`: {}845 846</details>847 848### Training Logs849| Epoch      | Step  | Training Loss | dim_768_cosine_ndcg@10 | dim_512_cosine_ndcg@10 | dim_256_cosine_ndcg@10 | dim_128_cosine_ndcg@10 | dim_64_cosine_ndcg@10 |850|:----------:|:-----:|:-------------:|:----------------------:|:----------------------:|:----------------------:|:----------------------:|:---------------------:|851| **0.9956** | **7** | **-**         | **0.04**               | **0.04**               | **0.03**               | **0.0262**             | **0.0308**            |852| 1.2844     | 10    | 7.098         | -                      | -                      | -                      | -                      | -                     |853| 1.8533     | 14    | -             | 0.0362                 | 0.02                   | 0.0354                 | 0.0154                 | 0.0508                |854| 2.5689     | 20    | 6.5515        | -                      | -                      | -                      | -                      | -                     |855| 2.7111     | 21    | -             | 0.0508                 | 0.01                   | 0.0208                 | 0.01                   | 0.0608                |856 857* The bold row denotes the saved checkpoint.858 859### Framework Versions860- Python: 3.12.12861- Sentence Transformers: 5.2.2862- Transformers: 4.57.6863- PyTorch: 2.9.0+cu126864- Accelerate: 1.12.0865- Datasets: 4.0.0866- Tokenizers: 0.22.2867 868## Citation869 870### BibTeX871 872#### Sentence Transformers873```bibtex874@inproceedings{reimers-2019-sentence-bert,875    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",876    author = "Reimers, Nils and Gurevych, Iryna",877    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",878    month = "11",879    year = "2019",880    publisher = "Association for Computational Linguistics",881    url = "https://arxiv.org/abs/1908.10084",882}883```884 885#### MatryoshkaLoss886```bibtex887@misc{kusupati2024matryoshka,888    title={Matryoshka Representation Learning},889    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},890    year={2024},891    eprint={2205.13147},892    archivePrefix={arXiv},893    primaryClass={cs.LG}894}895```896 897#### MultipleNegativesRankingLoss898```bibtex899@misc{henderson2017efficient,900    title={Efficient Natural Language Response Suggestion for Smart Reply},901    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},902    year={2017},903    eprint={1705.00652},904    archivePrefix={arXiv},905    primaryClass={cs.CL}906}907```908 909<!--910## Glossary911 912*Clearly define terms in order to be accessible across audiences.*913-->914 915<!--916## Model Card Authors917 918*Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.*919-->920 921<!--922## Model Card Contact923 924*Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.*925-->