EmbeddingStudio/query-parser-falcon-7b-instruct
29
1---2library_name: peft3base_model: tiiuae/falcon-7b-instruct4license: apache-2.05language:6- en7pipeline_tag: text-generation8datasets:9- EmbeddingStudio/query-parsing-instructions-falcon10tags:11- search-queries12- instruct-fine-tuned13- search-queries-parser14- zero-shot15- llm16- falcon17inference: false18metrics:19- accuracy20- precision21- recall22- f123---24 25# Model Card for the Query Parser LLM using Falcon-7B-Instruct 26 27EmbeddingStudio is the [open-source framework](https://github.com/EulerSearch/embedding_studio/tree/main), that allows you transform a joint "Embedding Model + Vector DB" into 28a full-cycle search engine: collect clickstream -> improve search experience-> adapt embedding model and repeat out of the box.29 30It's a highly rare case when a company will use unstructured search as is. And by searching `brick red houses san francisco area for april`31user definitely wants to find some houses in San Francisco for a month-long rent in April, and then maybe brick-red houses. 32Unfortunately, for the 15th January 2024 there is no such accurate embedding model. So, companies need to mix structured and unstructured search.33 34The very first step of mixing it - to parse a search query. Usual approaches are:35* Implement a bunch of rules, regexps, or grammar parsers (like [NLTK grammar parser](https://www.nltk.org/howto/grammar.html)).36* Collect search queries and to annotate some dataset for NER task. 37 38It takes some time to do, but at the end you can get controllable and very accurate query parser.39EmbeddingStudio team decided to dive into LLM instruct fine-tuning for `Zero-Shot query parsing` task 40to close the first gap while a company doesn't have any rules and data being collected, or even eliminate exhausted rules implementation, but in the future. 41 42The main idea is to align an LLM to being to parse short search queries knowing just a company market and a schema of search filters. Moreover, being oriented on applied NLP, 43we are trying to serve only light-weight LLMs a.k.a `not heavier than 7B parameters`. 44 45## Model Details46 47### Model Description48 49This is only [Falcon-7B-Instruct](https://huggingface.co/tiiuae/falcon-7b-instruct) aligned to follow instructions like:50```markdown51### System: Master in Query Analysis52### Instruction: Organize queries in JSON, adhere to schema, verify spelling.53#### Category: Logistics and Supply Chain Management54#### Schema: ```[{"Name": "Customer_Ratings", "Representations": [{"Name": "Exact_Rating", "Type": "float", "Examples": [4.5, 3.2, 5.0, "4.5", "Unstructured"]}, {"Name": "Minimum_Rating", "Type": "float", "Examples": [4.0, 3.0, 5.0, "4.5"]}, {"Name": "Star_Rating", "Type": "int", "Examples": [4, 3, 5], "Enum": [1, 2, 3, 4, 5]}]}, {"Name": "Date", "Representations": [{"Name": "Day_Month_Year", "Type": "str", "Examples": ["01.01.2024", "15.06.2023", "31.12.2022", "25.12.2021", "20.07.2024", "15.06.2023"], "Pattern": "dd.mm.YYYY"}, {"Name": "Day_Name", "Type": "str", "Examples": ["Monday", "Wednesday", "Friday", "Thursday", "Monday", "Tuesday"], "Enum": ["Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday", "Sunday"]}]}, {"Name": "Date_Period", "Representations": [{"Name": "Specific_Period", "Type": "str", "Examples": ["01.01.2024 - 31.01.2024", "01.06.2023 - 30.06.2023", "01.12.2022 - 31.12.2022"], "Pattern": "dd.mm.YYYY - dd.mm.YYYY"}, {"Name": "Month", "Type": "str", "Examples": ["January", "June", "December"], "Enum": ["January", "February", "March", "April", "May", "June", "July", "August", "September", "October", "November", "December"]}, {"Name": "Quarter", "Type": "str", "Examples": ["Q1", "Q2", "Q3"], "Enum": ["Q1", "Q2", "Q3", "Q4"]}, {"Name": "Season", "Type": "str", "Examples": ["Winter", "Summer", "Autumn"], "Enum": ["Winter", "Spring", "Summer", "Autumn"]}]}, {"Name": "Destination_Country", "Representations": [{"Name": "Country_Name", "Type": "str", "Examples": ["United States", "Germany", "China"]}, {"Name": "Country_Code", "Type": "str", "Examples": ["US", "DE", "CN"]}, {"Name": "Country_Abbreviation", "Type": "str", "Examples": ["USA", "GER", "CHN"]}]}]```55#### Query: Which logistics companies in the US have a perfect 5.0 rating ?56### Response:57[{"Value": "Which logistics companies in the US have a perfect 5.0 rating?", "Name": "Correct"}, {"Name": "Customer_Ratings.Exact_Rating", "Value": 5.0}, {"Name": "Destination_Country.Country_Code", "Value": "US"}]58```59 60**Important:** Additionally, we are trying to fine-tune the Large Language Model (LLM) to not only parse unstructured search queries but also to correct spelling. 61 62- **Developed by EmbeddingStudio team:**63 * Aleksandr Iudaev [[LinkedIn](https://www.linkedin.com/in/alexanderyudaev/)] [[Email](mailto:alexander@yudaev.ru)]64 * Andrei Kostin [[LinkedIn](https://www.linkedin.com/in/andrey-kostin/)] [[Email](mailto:andreynitsok@gmail.com)]65 * ML Doom [AI Assistant]66- **Funded by EmbeddingStudio team**67- **Model type:** Instruct Fine-Tuned Large Language Model68- **Model task:** Zero-shot search query parsing69- **Language(s) (NLP):** English70- **License:** apache-2.071- **Finetuned from model:** [Falcon-7B-Instruct](https://huggingface.co/tiiuae/falcon-7b-instruct)72- **!Maximal Length Size:** we used 1024 for fine-tuning, this is highly different from the original model `max_seq_length = 2048` 73- **Tuning Epochs:** 3 for now, but will be more later.74 75**Disclaimer:** As a small startup, this direction forms a part of our Minimum Viable Product (MVP). It's more of 76an attempt to test the 'product-market fit' rather than a well-structured scientific endeavor. Once we check it and go with a round, we definitely will:77* Curating a specific dataset for more precise analysis.78* Exploring various approaches and Large Language Models (LLMs) to identify the most effective solution.79* Publishing a detailed paper to ensure our findings and methodologies can be thoroughly reviewed and verified.80 81We acknowledge the complexity involved in utilizing Large Language Models, particularly in the context 82of `Zero-Shot search query parsing` and `AI Alignment`. Given the intricate nature of this technology, we emphasize the importance of rigorous verification.83Until our work is thoroughly reviewed, we recommend being cautious and critical of the results. 84 85### Model Sources86 87- **Repository:** code of inference the model will be [here](https://github.com/EulerSearch/embedding_studio/tree/main)88- **Paper:** Work In Progress89- **Demo:** Work In Progress90 91## Uses92 93We strongly recommend only the direct usage of this fine-tuned version of [Falcon-7B-Instruct](https://huggingface.co/tiiuae/falcon-7b-instruct):94* Zero-shot Search Query Parsing with porived company market name and filters schema 95* Search Query Spell Correction96 97For any other needs the behaviour of the model in unpredictable, please utilize the [original mode](https://huggingface.co/tiiuae/falcon-7b-instruct) or fine-tune your own. 98 99### Instruction format100 101```markdown102### System: Master in Query Analysis103### Instruction: Organize queries in JSON, adhere to schema, verify spelling.104#### Category: {your_company_category}105#### Schema: ```{filters_schema}```106#### Query: {query}107### Response:108```109 110Filters schema is JSON-readable line in the format (we highly recommend you to use it):111List of filters (dict):112* Name - name of filter (better to be meaningful).113* Representations - list of possible filter formats (dict):114 * Name - name of representation (better to be meaningful).115 * Type - python base type (int, float, str, bool).116 * Examples - list of examples.117 * Enum - if a representation is enumeration, provide a list of possible values, LLM should map parsed value into this list.118 * Pattern - if a representation is pattern-like (datetime, regexp, etc.) provide a pattern text in any format.119 120Example:121```json122[{"Name": "Customer_Ratings", "Representations": [{"Name": "Exact_Rating", "Type": "float", "Examples": [4.5, 3.2, 5.0, "4.5", "Unstructured"]}, {"Name": "Minimum_Rating", "Type": "float", "Examples": [4.0, 3.0, 5.0, "4.5"]}, {"Name": "Star_Rating", "Type": "int", "Examples": [4, 3, 5], "Enum": [1, 2, 3, 4, 5]}]}, {"Name": "Date", "Representations": [{"Name": "Day_Month_Year", "Type": "str", "Examples": ["01.01.2024", "15.06.2023", "31.12.2022", "25.12.2021", "20.07.2024", "15.06.2023"], "Pattern": "dd.mm.YYYY"}, {"Name": "Day_Name", "Type": "str", "Examples": ["Monday", "Wednesday", "Friday", "Thursday", "Monday", "Tuesday"], "Enum": ["Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday", "Sunday"]}]}, {"Name": "Date_Period", "Representations": [{"Name": "Specific_Period", "Type": "str", "Examples": ["01.01.2024 - 31.01.2024", "01.06.2023 - 30.06.2023", "01.12.2022 - 31.12.2022"], "Pattern": "dd.mm.YYYY - dd.mm.YYYY"}, {"Name": "Month", "Type": "str", "Examples": ["January", "June", "December"], "Enum": ["January", "February", "March", "April", "May", "June", "July", "August", "September", "October", "November", "December"]}, {"Name": "Quarter", "Type": "str", "Examples": ["Q1", "Q2", "Q3"], "Enum": ["Q1", "Q2", "Q3", "Q4"]}, {"Name": "Season", "Type": "str", "Examples": ["Winter", "Summer", "Autumn"], "Enum": ["Winter", "Spring", "Summer", "Autumn"]}]}, {"Name": "Destination_Country", "Representations": [{"Name": "Country_Name", "Type": "str", "Examples": ["United States", "Germany", "China"]}, {"Name": "Country_Code", "Type": "str", "Examples": ["US", "DE", "CN"]}, {"Name": "Country_Abbreviation", "Type": "str", "Examples": ["USA", "GER", "CHN"]}]}]123```124 125As the result, response will be JSON-readable line in the format:126```json127[{"Value": "Corrected search phrase", "Name": "Correct"}, {"Name": "filter-name.representation", "Value": "some-value"}]128```129 130Field and representation names will be aligned with the provided schema. Example:131```json132[{"Value": "Which logistics companies in the US have a perfect 5.0 rating?", "Name": "Correct"}, {"Name": "Customer_Ratings.Exact_Rating", "Value": 5.0}, {"Name": "Destination_Country.Country_Code", "Value": "US"}]133```134 135 136Used for fine-tuning `system` phrases:137```python138[139 "Expert at Deconstructing Search Queries",140 "Master in Query Analysis",141 "Premier Search Query Interpreter",142 "Advanced Search Query Decoder",143 "Search Query Parsing Genius",144 "Search Query Parsing Wizard",145 "Unrivaled Query Parsing Mechanism",146 "Search Query Parsing Virtuoso",147 "Query Parsing Maestro",148 "Ace of Search Query Structuring"149]150```151 152Used for fine-tuning `instruction` phrases:153```python154[155 "Convert queries to JSON, align with schema, ensure correct spelling.",156 "Analyze and structure queries in JSON, maintain schema, check spelling.",157 "Organize queries in JSON, adhere to schema, verify spelling.",158 "Decode queries to JSON, follow schema, correct spelling.",159 "Parse queries to JSON, match schema, spell correctly.",160 "Transform queries to structured JSON, align with schema and spelling.",161 "Restructure queries in JSON, comply with schema, accurate spelling.",162 "Rearrange queries in JSON, strict schema adherence, maintain spelling.",163 "Harmonize queries with JSON schema, ensure spelling accuracy.",164 "Efficient JSON conversion of queries, schema compliance, correct spelling."165]166```167 168### Direct Use169 170```python171import json172 173from json import JSONDecodeError174 175from transformers import AutoTokenizer, AutoModelForCausalLM176 177INSTRUCTION_TEMPLATE = """178### System: Master in Query Analysis179### Instruction: Organize queries in JSON, adhere to schema, verify spelling.180#### Category: {0}181#### Schema: ```{1}```182#### Query: {2}183### Response:184"""185 186 187def parse(188 query: str, 189 company_category: str, 190 filter_schema: dict,191 model: AutoModelForCausalLM,192 tokenizer: AutoTokenizer193):194 input_text = INSTRUCTION_TEMPLATE.format(195 company_category,196 json.dumps(filter_schema),197 query198 )199 input_ids = tokenizer.encode(input_text, return_tensors='pt')200 201 # Generating text202 output = model.generate(input_ids.to('cuda'), 203 max_new_tokens=1024, 204 do_sample=True, 205 temperature=0.05,206 pad_token_id=50256207 )208 try:209 parsed = json.loads(tokenizer.decode(output[0], skip_special_tokens=True).split('## Response:\n')[-1])210 except JSONDecodeError as e:211 parsed = dict()212 213 return parsed214```215 216## Bias, Risks, and Limitations217 218### Bias219 220Again, this model was fine-tuned for following the zero-shot query parsing instructions. 221So, all ethical biases are inherited by the original model.222 223Model was fine-tuned to be able to work with the unknown company domain and filters schema. But, can be better with the training company categories:224 225Educational Institutions, Job Recruitment Agencies, Banking Services, Investment Services, Insurance Services, Financial Planning and Advisory, Credit Services, Payment Processing, Mortgage and Real Estate Services, Taxation Services, Risk Management and Compliance, Digital and Mobile Banking, Retail Stores (Online and Offline), Automotive Dealerships, Restaurants and Food Delivery Services, Entertainment and Media Platforms, Government Services, Travelers and Consumers, Logistics and Supply Chain Management, Customer Support Services, Market Research Firms, Mobile App Development, Game Development, Cloud Computing Services, Data Analytics and Business Intelligence, Cybersecurity Software, User Interface/User Experience Design, Internet of Things (IoT) Development, Project Management Tools, Version Control Systems, Continuous Integration/Continuous Deployment, Issue Tracking and Bug Reporting, Collaborative Development Environments, Team Communication and Chat Tools, Task and Time Management, Customer Support and Feedback, Cloud-based Development Environments, Image Stock Platforms, Video Hosting and Portals, Social Networks, Professional Social Networks, Dating Apps226 227### Risks and Limitations228 229Known limitations:2301. Can add extra spaces or remove spaces: `1-2` -> `1 - 2`.2312. Can add extra words: `5` -> `5 years`.2323. Can not differentiate between `<>=` and theirs HTML versions `<`, `>`, `&eq;`.2334. Bad with abbreviations.2345. Can add extra `.0` for floats and integers.2356. Can add extra `0` or remove `0` for integers with a char postfix: `10M` -> `1m`.2367. Can hallucinate with integers. For the case like `list of positions exactly 7 openings available` result can be237`{'Name': 'Job_Type.Exact_Match', 'Value': 'Full Time'}`.2388. We fine-tuned this model with max sequence length = 1024, so it may happen that response will not be JSON-readable.239 240The list will be extended in the future.241 242### Recommendations243 2441. We used synthetic data for the first version of this model. So, we suggest you to precisely test this model on your company's domain, even it's in the list.2452. Use meaningful names for filters and theirs representations.2463. Provide examples for each representation.2474. Try to be compact, model was fine-tuned with max sequence length equal 1024.2485. During the generation use greedy strategy with tempertature 0.05.2496. The result will be better if you align a filters schema with a schema type of the training data. 250 251## How to Get Started with the Model252 253Use the code below to get started with the model.254 255```python256MODEL_ID = 'EmbeddingStudio/query-parser-falcon-7b-instruct'257```258 259Initialize tokenizer:260```python261from transformers import AutoTokenizer262 263tokenizer = AutoTokenizer.from_pretrained(264 MODEL_ID, 265 trust_remote_code=True,266 add_prefix_space=True,267 use_fast=False,268)269```270 271Initialize model:272```python273import torch274 275from peft import LoraConfig276from transformers import AutoModelForCausalLM, BitsAndBytesConfig277 278peft_config = LoraConfig(279 lora_alpha=16,280 lora_dropout=0.1,281 r=64,282 bias="none",283 task_type="CAUSAL_LM",284)285 286bnb_config = BitsAndBytesConfig(287 load_in_4bit=True,288 load_4bit_use_double_quant=True,289 bnb_4bit_quant_type="nf4",290 bnb_4bit_compute_dtype=torch.bfloat16,291)292 293device_map = {"": 0}294 295model = AutoModelForCausalLM.from_pretrained(296 MODEL_ID, 297 quantization_config=bnb_config, 298 device_map=device_map, 299 torch_dtype=torch.float16300)301```302 303Use for parsing:304```python305import json306 307from json import JSONDecodeError308 309INSTRUCTION_TEMPLATE = """310### System: Master in Query Analysis311### Instruction: Organize queries in JSON, adhere to schema, verify spelling.312#### Category: {0}313#### Schema: ```{1}```314#### Query: {2}315### Response:316"""317 318 319def parse(320 query: str, 321 company_category: str, 322 filter_schema: dict,323 model: AutoModelForCausalLM,324 tokenizer: AutoTokenizer325):326 input_text = INSTRUCTION_TEMPLATE.format(327 company_category,328 json.dumps(filter_schema),329 query330 )331 input_ids = tokenizer.encode(input_text, return_tensors='pt')332 333 # Generating text334 output = model.generate(input_ids.to('cuda'), 335 max_new_tokens=1024, 336 do_sample=True, 337 temperature=0.05,338 pad_token_id=50256339 )340 try:341 parsed = json.loads(tokenizer.decode(output[0], skip_special_tokens=True).split('## Response:\n')[-1])342 except JSONDecodeError as e:343 parsed = dict()344 345 return parsed346 347category = 'Logistics and Supply Chain Management'348query = 'Which logistics companies in the US have a perfect 5.0 rating ?'349schema = [{"Name": "Customer_Ratings", "Representations": [{"Name": "Exact_Rating", "Type": "float", "Examples": [4.5, 3.2, 5.0, "4.5", "Unstructured"]}, {"Name": "Minimum_Rating", "Type": "float", "Examples": [4.0, 3.0, 5.0, "4.5"]}, {"Name": "Star_Rating", "Type": "int", "Examples": [4, 3, 5], "Enum": [1, 2, 3, 4, 5]}]}, {"Name": "Date", "Representations": [{"Name": "Day_Month_Year", "Type": "str", "Examples": ["01.01.2024", "15.06.2023", "31.12.2022", "25.12.2021", "20.07.2024", "15.06.2023"], "Pattern": "dd.mm.YYYY"}, {"Name": "Day_Name", "Type": "str", "Examples": ["Monday", "Wednesday", "Friday", "Thursday", "Monday", "Tuesday"], "Enum": ["Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday", "Sunday"]}]}, {"Name": "Date_Period", "Representations": [{"Name": "Specific_Period", "Type": "str", "Examples": ["01.01.2024 - 31.01.2024", "01.06.2023 - 30.06.2023", "01.12.2022 - 31.12.2022"], "Pattern": "dd.mm.YYYY - dd.mm.YYYY"}, {"Name": "Month", "Type": "str", "Examples": ["January", "June", "December"], "Enum": ["January", "February", "March", "April", "May", "June", "July", "August", "September", "October", "November", "December"]}, {"Name": "Quarter", "Type": "str", "Examples": ["Q1", "Q2", "Q3"], "Enum": ["Q1", "Q2", "Q3", "Q4"]}, {"Name": "Season", "Type": "str", "Examples": ["Winter", "Summer", "Autumn"], "Enum": ["Winter", "Spring", "Summer", "Autumn"]}]}, {"Name": "Destination_Country", "Representations": [{"Name": "Country_Name", "Type": "str", "Examples": ["United States", "Germany", "China"]}, {"Name": "Country_Code", "Type": "str", "Examples": ["US", "DE", "CN"]}, {"Name": "Country_Abbreviation", "Type": "str", "Examples": ["USA", "GER", "CHN"]}]}]350 351output = parse(query, category, schema)352print(output)353 354# [out]: [{"Value": "Which logistics companies in the US have a perfect 5.0 rating?", "Name": "Correct"}, {"Name": "Customer_Ratings.Exact_Rating", "Value": 5.0}, {"Name": "Destination_Country.Country_Code", "Value": "US"}]355```356 357 358## Training Details359 360### Training Data361 362We used synthetically generated query parsing instructions:363* We generated lists of possible filters for 63 customer categories: 364 * [Raw version of filters dataset](https://huggingface.co/datasets/EmbeddingStudio/synthetic-search-filters-raw)365 * [Split by representations](https://huggingface.co/datasets/EmbeddingStudio/synthetic-search-filters)366* Select randomly up-to 150 possible combinations (1-3 filters in each combination) of filters, the way each filter's representation appears maximum twice.367* For a given category and combination we [generated](https://huggingface.co/datasets/EmbeddingStudio/synthetic-search-queries) with GPT-4 Turbo:368 * 2 search queries and theirs parsed version with unstructured parts.369 * 2 search queries and theirs parsed version without unstructured part. 370* Using filters, queries and parsed version we prepared [72.5k falcon format instruction](EmbeddingStudio/query-parsing-instructions-falcon)371 372**Warning:** EmbeddingStudio team aware you that generated queries **weren't enough curated**, and will be curated later once we finish our product market fit stage. 373 374#### Principles of train / test splitting375 376As we are trying to fine-tune LLM to follow zero-shot query parsing instructions, so we want to test:377* Ability to work well with unseen domain378* Ability to work well with unseen filters379* Ability to work well with unseen queries380 381For these purposes we:3821. We put into test split 5 categories, completely separared from train: `Telecommunication Companies, Legal Services, Enterprise Software Development, Artificial Intelligence and Machine Learning, Documentation and Knowledge Sharing`.3832. Also out of each appearing in train company categories, we put aside / removed one filter and queries related to it.3843. Selected 5% of other queries and put it into test. 385 386#### Filters generation details387 388We used GPT-4 Turbo to generate several possible filters for 63 company categroies. For each filter we also generated some possible representations. For examples filter `Date` can be represented as `dd/mm/YYYY`, `YYYY-mm-dd`, as words `2024 Jan 17`, etc.389 390#### Queries generation details391 392We also used GPT-4 Turbo for generation of search queries and theirs parsed version. Main principles were: 393* If passed schema doesn't contain possible filter, do not generate query itself or a possible filter 394* If a selected representations combination contains enumeration, so we ask to map values in a search query and a parsed version.395* If a selected representations combination contains pattern, so we ask GPT-4 Turbo to be aligned with a pattern396 397#### Instructions generation details398 399For the generation instructions we used following ideas:4001. Zero-Shot query parser should be schema agnostic. Cases like `snake_case, CamelCase, http-headers-like` should not ruin generation process. 4012. Zero-Shot query parser should be spelling errors insensitive.4023. Training instructions should be in the following order:403 * Category404 * Schema405 * Query406 407 So LLM can be used in the following way: just generate embedding of category -> schema part, so inference will be faster.408 409We assume, that `schema agnostic` termin means something wider, like to be able to work not only with JSONs, but also with HTML, Markdown, YAML, etc. We are working on it.410 411So, what was our approach as an attempt to achieve these abilities:4121. For each query we generated a version with a mistake4132. Passed to each parsed version an additional field `Correct`, which contains a corrected version of a search query.4143. For each query we randomly selected and used a case for schema fields and a case for filter and representation names.4154. For each query we additionally generated two instuctions:416 * Where did we remove from a provided schema and parsed version one filter417 * Where did we remove from a provided schema and parsed version all related filters418 419**Warning:** EmbeddingStudio team ask you to curate datasets on your own precisely.420 421### Training Procedure422 4231. Mixed Precision Regime4242. Supervised Fine-Tuning4253. Three epochs with cosine scheduler426 427All details in Training Hyperparameters428 429#### Preprocessing [optional]430 431The preprocessing steps are not detailed in the provided code. Typically, preprocessing involves tokenization, normalization, data augmentation, and handling of special tokens. In this training setup, the tokenizer was configured with `add_prefix_space=True` and `use_fast=False`, which might indicate special considerations for tokenizing certain languages or text formats.432 433#### Training Hyperparameters434| Hyperparameter | Value | Description |435|--------------------------------------|------------------------------|-------------------------------------------------------|436| **Training Regime** | Mixed Precision (bfloat16) | Utilizes bfloat16 for efficient memory usage and training speed. |437| **Model Configuration** | Causal Language Model | Incorporates LoRA (Low-Rank Adaptation) for training efficiency. |438| **Quantization Configuration** | Bits and Bytes (BnB) | Uses settings like `load_in_4bit` and `bnb_4bit_quant_type` for model quantization. |439| **Training Environment** | CUDA-enabled Device | Indicates GPU acceleration for training. |440| **Learning Rate** | 2e-4 | Determines the step size at each iteration while moving toward a minimum of a loss function. |441| **Weight Decay** | 0.001 | Helps in regularizing and preventing overfitting. |442| **Warmup Ratio** | 0.03 | Fraction of total training steps used for the learning rate warmup. |443| **Optimizer** | Paged AdamW (32-bit) | Optimizes the training process with efficient memory usage. |444| **Gradient Accumulation Steps** | 2 | Reduces memory consumption and allows for larger effective batch sizes. |445| **Max Grad Norm** | 0.3 | Maximum norm for the gradients. |446| **LR Scheduler Type** | Cosine | Specifies the learning rate schedule. |447| **PEFT Configurations** | LoraConfig | Details like `lora_alpha`, `lora_dropout`, and `r` for LoRA adaptations. |448| **Training Dataset Segmentation** | Train and Test Sets | Segmentation of the dataset for training and evaluation. |449| **Max Sequence Length** | 1024 | Maximum length of the input sequences. |450 451### Testing Data, Factors & Metrics452 453#### Testing Data454 455All information is provided in [Training Data](#training-data) section.456 457### Factors Influencing Falcon-7B-Instruct Model Performance458 459#### 1. Company Category and Domain Knowledge460- Performance may vary based on the specific company category or domain.461- Enhanced performance in domains specifically trained on, such as Educational Institutions, Banking Services, Logistics, etc.462 463#### 2. Filter Schema Adaptability464- Ability to adapt to various filter schemas.465- Performance in parsing and organizing queries according to different schemas.466 467#### 3. Handling of Spelling and Syntax Errors468- Robustness in handling spelling errors and syntax variations in queries.469 470#### 4. Representation and Type Handling471- Capability to handle different data representations (e.g., date formats, enumerations, patterns).472- Accurate processing of various base types (int, float, str, bool).473 474#### 5. Length and Complexity of Queries475- Impact of the length and complexity of queries on performance.476- Maximum sequence length of 1024 could pose limitations for longer or complex queries.477 478#### 6. Bias and Ethical Considerations479- Inherited ethical biases from the original model.480- Importance of understanding these biases in different contexts.481 482#### 7. Limitations in Fine-Tuning and Data Curation483- Limitations such as extra spaces, handling of abbreviations, etc.484- Influence of the extent of training data curation on model accuracy.485 486#### 8. Specific Use Cases487- Recommended primarily for zero-shot search query parsing and search query spell correction.488- Performance in other use cases might be unpredictable.489 490#### 9. Training Data Quality and Diversity491- Quality and diversity of synthetic training data.492- Influence on the model's effectiveness across different scenarios.493 494 495##### Testing Procedure496 497Results of testing procedure as JSON is provided [here](https://huggingface.co/EmbeddingStudio/query-parser-falcon-7b-instruct/blob/main/falcon-7b-instruct-test.json).498 499This is a list of items, each item is:5001. Predicted parsed query5012. Real parsed query5023. Category503 504#### Metrics505 506#### Metric Overview507 508Our zero-shot search query parsing model is designed to extract structured information from unstructured search queries with high precision. The primary metric for evaluating our model's performance is the True Positive (TP) rate, which is assessed using a specialized token-wise Levenshtein distance. This approach is aligned with our goal to achieve semantic accuracy in parsing user queries.509 510#### True Positives (TP)511 512- **Definition**: A True Positive in our model is counted when the model correctly identifies both the 'Name' and 'Value' in a query, matching the expected results.513- **Measurement Method**: The TP rate is quantified using the `levenshtein_tokenwise` function, which calculates the distance between predicted and actual key-value pairs at a token level. We consider a Levenshtein distance of 0.25 or less as acceptable for matching.514- **Importance**: 515 - **Token-Level Accuracy**: We use token-wise accuracy over traditional character-level Levenshtein distance, which can be overly strict, especially for minor spelling variations. Our token-wise approach prioritizes semantic accuracy.516 - **Relevance to Search Queries**: Accuracy at the token level is more indicative of the model's ability to understand and parse user intent in search queries.517 518#### Generation Strategy519 520- **Approach**: The model generates responses based on input queries with a maximum token length set to 1000, employing a sampling strategy (do_sample=True), and a low temperature setting of 0.05. This controlled randomness in generation ensures a variety of accurate and relevant responses.521- **Impact on TP**: 522 - The low temperature setting directly influences the TP rate by reducing the randomness in the model's predictions. With a lower temperature, the model is more likely to choose the most probable word in a given context, leading to more accurate and consistent outputs. This is particularly crucial in search query parsing, where understanding and interpreting user input with high precision is vital.523 524#### Additional Metrics525 526- **False Positives (FP) and False Negatives (FN)**: These metrics are monitored to provide a comprehensive view of the model's predictive capabilities.527- **Precision, Recall, F1 Score, Accuracy**: These standard metrics complement our TP-focused assessment, providing a rounded picture of the model's performance in various aspects.528 529#### Motivation for Metric Choice530 531- **Alignment with User Intent**: Focusing on token-wise accuracy ensures the model's performance closely mirrors the structure and intent typical in search queries.532- **Robustness Against Query Variations**: This metric approach makes the model adaptable to the varied formulations of real-world search queries.533- **Balancing Precision and Recall**: Our method aims to balance the model's ability not to miss relevant key-value pairs (high recall) while not over-identifying irrelevant ones (high precision).534 535##### Total metrics536 537| Category | Recall | Precision | F1 | Accuracy |538| ------------------------------------------------ | ------ | --------- | ----- | -------- |539| Telecommunication Companies [+] | 0.70 | 0.67 | 0.68 | 0.52 |540| Legal Services [+] | 0.80 | 0.74 | 0.77 | 0.63 |541| Enterprise Software Development [+] | 0.78 | 0.71 | 0.74 | 0.59 |542| Artificial Intelligence and Machine Learning [+] | 0.77 | 0.78 | 0.78 | 0.63 |543| Documentation and Knowledge Sharing [+] | 0.68 | 0.65 | 0.66 | 0.50 |544| Educational Institutions | 0.55 | 0.51 | 0.53 | 0.36 |545| Job Recruitment Agencies | 0.58 | 0.51 | 0.54 | 0.37 |546| Banking Services | 0.73 | 0.81 | 0.76 | 0.62 |547| Investment Services | 0.50 | 0.50 | 0.50 | 0.33 |548| Insurance Services | 0.77 | 0.77 | 0.77 | 0.62 |549| Financial Planning and Advisory | 0.65 | 0.67 | 0.66 | 0.49 |550| Credit Services | 0.60 | 0.65 | 0.63 | 0.45 |551| Payment Processing | 0.79 | 0.74 | 0.76 | 0.62 |552| Mortgage and Real Estate Services | 1.00 | 1.00 | 1.00 | 1.00 |553| Taxation Services | 0.52 | 0.57 | 0.54 | 0.37 |554| Risk Management and Compliance | 1.00 | 0.95 | 0.98 | 0.95 |555| Digital and Mobile Banking | 0.72 | 0.71 | 0.71 | 0.55 |556| Retail Stores (Online and Offline) | 0.96 | 0.87 | 0.92 | 0.85 |557| Automotive Dealerships | 0.52 | 0.53 | 0.53 | 0.36 |558| Restaurants and Food Delivery Services | 0.76 | 0.77 | 0.76 | 0.62 |559| Entertainment and Media Platforms | 0.80 | 0.84 | 0.82 | 0.70 |560| Government Services | 0.58 | 0.65 | 0.61 | 0.44 |561| Travelers and Consumers | 0.89 | 0.89 | 0.89 | 0.80 |562| Logistics and Supply Chain Management | 0.56 | 0.59 | 0.58 | 0.41 |563| Customer Support Services | 0.60 | 0.54 | 0.57 | 0.40 |564| Market Research Firms | 0.52 | 0.49 | 0.51 | 0.34 |565| Mobile App Development | 0.81 | 0.79 | 0.80 | 0.67 |566| Game Development | 0.94 | 0.94 | 0.94 | 0.88 |567| Cloud Computing Services | 0.64 | 0.62 | 0.63 | 0.46 |568| Data Analytics and Business Intelligence | 0.63 | 0.61 | 0.62 | 0.45 |569| Cybersecurity Software | 0.54 | 0.59 | 0.57 | 0.39 |570| User Interface/User Experience Design | 0.63 | 0.64 | 0.63 | 0.46 |571| Internet of Things (IoT) Development | 0.89 | 0.71 | 0.79 | 0.65 |572| Project Management Tools | 0.80 | 0.83 | 0.81 | 0.69 |573| Version Control Systems | 0.77 | 0.73 | 0.75 | 0.60 |574| Continuous Integration/Continuous Deployment | 0.85 | 0.83 | 0.84 | 0.72 |575| Issue Tracking and Bug Reporting | 0.64 | 0.62 | 0.63 | 0.46 |576| Collaborative Development Environments | 0.68 | 0.67 | 0.68 | 0.51 |577| Team Communication and Chat Tools | 0.94 | 0.91 | 0.93 | 0.87 |578| Task and Time Management | 0.78 | 0.78 | 0.78 | 0.64 |579| Customer Support and Feedback | 0.88 | 0.82 | 0.85 | 0.74 |580| Cloud-based Development Environments | 0.81 | 0.81 | 0.81 | 0.68 |581| Image Stock Platforms | 0.88 | 0.85 | 0.87 | 0.76 |582| Video Hosting and Portals | 0.86 | 0.88 | 0.87 | 0.77 |583| Social Networks | 0.60 | 0.57 | 0.59 | 0.41 |584| Professional Social Networks | 0.68 | 0.69 | 0.68 | 0.52 |585| Dating Apps | 0.90 | 0.90 | 0.90 | 0.82 |586| Aggregate | 0.73 | 0.72 | 0.73 | 0.59 |587 588##### Unseen domains metrics589 590| Category | Recall | Precision | F1 | Accuracy |591| ------------------------------------------------ | ------ | --------- | ----- | -------- |592| Telecommunication Companies [+] | 0.70 | 0.67 | 0.68 | 0.52 |593| Legal Services [+] | 0.80 | 0.74 | 0.77 | 0.63 |594| Enterprise Software Development [+] | 0.78 | 0.71 | 0.74 | 0.59 |595| Artificial Intelligence and Machine Learning [+] | 0.77 | 0.78 | 0.78 | 0.63 |596| Documentation and Knowledge Sharing [+] | 0.68 | 0.65 | 0.66 | 0.50 |597| Aggregate | 0.75 | 0.71 | 0.73 | 0.57 |598 599 600## Environmental Impact601 602<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->603 604Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).605 606- **Hardware Type:** NVIDIA Tesla V100607- **Hours used:** 72608- **Cloud Provider:** Google Cloud609- **Compute Region:** us-west-1610- **Carbon Emitted:** 6.48611 612## Technical Specifications613 614### Model Architecture and Objective615 616* Base model: [Falcon-7b-Instruct](https://huggingface.co/tiiuae/falcon-7b-instruct).617* Quantization Configuration: Uses settings like `load_in_4bit` and `bnb_4bit_quant_type` for model quantization.618 619### Compute Infrastructure620 621[To be added]622 623#### Hardware624 625[To be added]626 627#### Software628 629* Python 3.9+630* CUDA 11.7.1631* NVIDIA [Compatible Drivers](https://www.nvidia.com/download/find.aspx)632* Torch 2.0.0633 634## More Information / About us635 636EmbeddingStudio is an innovative open-source framework designed to seamlessly convert a combined637"Embedding Model + Vector DB" into a comprehensive search engine. With built-in functionalities for638clickstream collection, continuous improvement of search experiences, and automatic adaptation of639the embedding model, it offers an out-of-the-box solution for a full-cycle search engine.640 641642 643### Features644 6451. 🔄 Turn your vector database into a full-cycle search engine6462. 🖱️ Collect users feedback like clickstream6473. 🚀 (*) Improve search experience on-the-fly without frustrating wait times6484. 📊 (*) Monitor your search quality6495. 🎯 Improve your embedding model through an iterative metric fine-tuning procedure6506. 🆕 (*) Use the new version of the embedding model for inference651 652(*) - features in development653 654EmbeddingStudio is highly customizable, so you can bring your own:655 6561. Data source6572. Vector database6583. Clickstream database6594. Embedding model660 661For more details visit [GitHub Repo](https://github.com/EulerSearch/embedding_studio/tree/main).662 663## Model Card Authors and Contact664 665* Aleksandr Iudaev [[LinkedIn](https://www.linkedin.com/in/alexanderyudaev/)] [[Email](mailto:alexander@yudaev.ru)]666* Andrei Kostin [[LinkedIn](https://www.linkedin.com/in/andrey-kostin/)] [[Email](mailto:andreynitsok@gmail.com)]667* ML Doom [AI Assistant]668 669### Framework versions670 671- PEFT 0.5.0672- Datasets 2.16.1673- BitsAndBytes 0.41.0674- PyTorch 2.0.0675- Transformers 4.36.2676- TRL 0.7.7